Website Scraper Integration

Configure the Website Scraper to crawl public web pages and index content into your self-healing knowledge base.

What it does

The Website Scraper crawls public website directory structures and parses HTML document trees. It strips out layouts like header navigations, footer scripts, and sidebars, extracting the primary text content to build clean vector indexes. This enables AI agents to reference public-facing FAQs, blog posts, and site information.

Data Flow & Crawler Behavior

The website crawler reads public directories securely:

Data Read: HTML file body structures, headers, body text, inline links, and metadata.
Data Written: None. The crawler is a read-only indexer.
Sync Frequency: Ingestion runs on setup. Recrawling triggers on a user-defined schedule (daily, weekly, monthly) or on-demand via the Source Explorer dashboard.

Scope & Limits

Allowed Content: Publicly reachable HTML text. Web pages protected behind authentication screens or payment gateways are ignored.
Exclusions: Media attachments (videos, sound assets, script tags, canvas widgets) are skipped.
Crawl Limits: Max crawl depth limit of 3 directory levels, with a cap of 1,000 pages per crawl. The scraper strictly respects standard `robots.txt` exclusion protocols. Refer to the Google Robots.txt Specifications.

Setup guide

Estimated setup time: ~4 minutes. Authentication type: None (Public Crawl). Whitelist configuration inside target site firewall may be required if bot protection is active.

Access Source settings

Navigate to Knowledge → Add Source and select Website Scraper.

Configure Crawl settings

Input the target website's root URL (e.g. https://example.com/help). Set the crawl depth limit and add any custom CSS selector query exclusions to block menus or header blocks from indexing.

Execute scrape

Click Start Scrape. The crawler will verify target domain reachable parameters and start parsing links recursively. View live logs in the Source Explorer.

Where it shows up

Source Explorer: Appears as a crawled website resource page. Displays lists of parsed URLs, indexing statistics, block counts, and status indicators.
Citations: AI responses cite URLs, redirecting users directly to the original web page.

Troubleshooting & Common Issues

Zero pages crawled: Verify that your web server is not blocking the crawler user-agent (configured as `VerabaseBot`). Check that the root URL does not contain typos or redirect to broken links.
Stale content: If updates on your website do not show up in agent replies, check your sync scheduler settings or force a manual re-crawl in the Source Explorer.

Frequently asked questions

Can Verabase scrape pages behind a login wall?

No, the standard Website Scraper only crawls public web pages. For authenticated portals, please upload files or use the Webhook Gateway.

How does Verabase handle dynamic JavaScript sites?

The scraper utilizes a headless browser rendering engine to ensure dynamic React, Vue, or Angular content is fully loaded before extraction.

Is there a crawling rate limit?

Yes, to avoid impacting your server's performance, Verabase limits scrape requests to a maximum of 2 requests per second by default.

Related integrations

Ready to scrape your site?

Join the waitlist for early access to the self-healing knowledge platform.

Get early access to Website Scraper