Website

Copilot allows you to crawl website content to enhance its knowledge and improve responses. By adding a website as a data source, Copilot can reference web-based information while assisting users.

Adding a Website as a Data Source

To integrate a website:

1. Select the Website option.

   Select 'Website' as a data source option in Copilot.

2. Enter the website URL in the input field.

   Provide the URL of the website to be crawled.

3. Choose whether to:

   • Scan Only This Page – Fetch data only from the entered URL.

     Select this option to limit crawling to a single page.

   • Include Nested Pages – Crawl the page and all linked subpages.

     Choose this option to crawl the main page and all linked subpages.

     tip

     Enable Dynamic Content Collection to capture real-time content.

4. Click Fetch to process the website content.

   Click 'Fetch' to start crawling and extracting data from the website.

   Once the website is processed, Copilot will display the extracted links from the site. Each link includes:

   • URL: The specific page crawled.

   • Character Count: The number of characters extracted from the page.

   • Preview Content: A button to view the extracted text.

   • Actions: Navigation to the extracted link & Delete button to remove the link from the data source.

     View and manage extracted links from the website.

5. Use Sitemap for Crawling – Provide a sitemap.xml URL to ensure structured crawling based on predefined paths.

   Add a sitemap.xml URL to enable structured crawling of the website.

6. Click Add Data Source to finalize the process.

   Click 'Add Data Source' to complete website integration.

Setting Up Auto-Sync and Monitoring

Auto-Sync allows Copilot to automatically update and refresh website data at regular intervals, ensuring that the latest information is always available. This feature is useful for keeping indexed content up-to-date without requiring manual intervention.

1. After enabling Auto Sync, choose the desired interval for synchronization. For example, you can select daily, weekly, or other custom timeframes.

   Enable Auto Sync and set the synchronization interval.

2. Confirm your settings to allow the system to process your data source at the chosen intervals, then click Update.

   Click 'Update' to save Auto-Sync settings.

3. If you prefer manual control, click the Train button to sync your data instantly. This will process the latest changes and make them available in Copilot.

   Click 'Train' to manually sync website data.

   tip

   Auto-Sync ensures your data remains up to date without manual effort, but you can always perform a manual sync when needed.

Managing Extracted Links

When you delete an extracted link, the associated webpage data is removed from Copilot's knowledge base. This means Copilot will no longer reference the content from the deleted link when generating responses.

caution

Deleting extracted links permanently removes their data from Copilot's reference. Ensure you want to remove them before proceeding. This action will not restore the character usage spent on crawling the deleted links.

Monitoring Sync Activity

You can access logs in the Training History panel, which maintains a comprehensive record of all past sync activities. This panel allows you to review synchronization details, track changes over time, and identify any errors or issues that may have occurred during the process.

Check training history and monitor sync logs for any issues.

note

The Auto Sync Scheduler will be the same across all data sources.