Web scraping

Import content from public websites by scraping a single page or crawling a site through its sitemap to build agent knowledge from help centers and docs.

Import content from public websites to power agent responses. Scrape individual pages or crawl entire documentation sites to build knowledge from help centers, docs, and web resources.

What you can do

• Import a single page, or crawl a site through its sitemap (up to 500 pages) or by following its links (up to 100 pages)
• Scrape JavaScript-rendered and formatted content
• Auto-sync to keep website content up-to-date
• Organize crawled sites with automatic folder structures

Setup

1. Select Website as your source

   When adding documents to a knowledge folder, select Website from the available sources.

2. Enter the URL

   Provide the URL of the webpage you want to import.

3. Enable crawling (optional)

   Toggle crawling on to import pages from the same site, or leave it off to import only the single page.

4. Import content

   Content is scraped and imported from the specified URL.

Learn more about managing knowledge folders

How it works

Browser rendering scrapes website content for accurate extraction:

• Main page content and text.
• Formatted content (headings, lists, paragraphs).
• JavaScript-rendered content.
• Publicly accessible information only.

Single page import

By default, content is imported from only the single URL you provide. This is ideal for:

• Specific documentation pages.
• Help articles.
• FAQ pages.
• Individual blog posts.

Website crawling

Enable crawling to import entire documentation sites automatically. Ravenna looks for the site’s sitemap first, in robots.txt and then as a sitemap.xml file in the URL’s folder and each folder above it.

• With a sitemap: Ravenna imports the pages the sitemap lists, up to 500. Only pages in the same folder as the URL you entered, or below it, are included, so entering https://example.com/docs/ imports the /docs/ pages and skips the rest of the site.
• English pages only: When a sitemap lists the same pages in several languages, such as /hc/en-us/ and /hc/de/, Ravenna imports only the English ones.
• Without a sitemap: Ravenna follows links from the URL you entered, page by page, and stops at 100 pages.
• Same site only: Crawling stays on the exact hostname you entered. Links to other domains and subdomains are skipped.
• Folder structure: Crawled sites are organized under a root folder named after the domain.

Enter the most specific URL that covers what you need, like a help center’s root rather than the company homepage, so the import stays focused and under the page limit.

Requirements

• Website must be publicly accessible (no authentication required).
• Content must be available without login or paywalls.

Managing imported content

After import:

• Archive pages to exclude them from agent responses.
• Delete pages that are no longer needed.

Learn more about managing documents

Auto-sync

When auto-sync is enabled, Ravenna imports the site again with the same URL and crawling setting, so website content stays up to date:

• Pages whose content changed are re-imported.
• On a crawled site, new pages the crawl finds are added to the folder.
• Pages the crawl no longer finds are marked unavailable.
• For a single-page import, the sync fails if the page can’t be scraped.