Web pages

A Web Crawler connection fetches public web pages and indexes their content, so agents can answer from your website, help center or product pages.

Add pages

1. 1Go to Studio → Connections, click Create Connection and choose Web Crawler.
2. 2Enter a Connection Name, for example "Company Website", and a Description.
3. 3Under Website URLs, enter the first address. Click Add URL for each additional page.
4. 4Click Create Connection. Harmona starts fetching and indexing right away.

What gets indexed

• Each URL you add is fetched as one page. Links on that page are not followed, so add every page you want indexed.
• Harmona keeps the main content of the page as text for search.
• A URL that points straight to a document, such as a PDF or DOCX file, is downloaded as a file and processed like an uploaded document.

Keep pages fresh

Harmona fetches all URLs again on your organization's sync interval, once a day by default. To refresh now, open the connection and click Sync Connection. Do this after you publish an important change.

Keep URLs stable. If a page moves, edit the connection with Edit Connection, replace the old address and sync again.

When to use something else

• Docs hosted on ReadMe.com: use a ReadMe Docs source. It syncs every guide, API reference page and changelog post. See Documentation connections.
• Files you already have: upload them to an Internal Files connection. See Files and images.