Crawl a Website
Introduction
The play.ai web embed can be used to crawl a website and answer questions about the content and navigate the user to the relevant pages.
Crawl a website
Enable crawling
On the last page of the create/edit agent flow, check the “Crawl website” checkbox and enter the URL of the website you want to crawl.
Click the 'Crawl' button to start the crawling process
Stay on the page until the crawling process is complete. This can take a while if the website has many pages.
Wait for crawling to complete
Your embed should now be able to answer questions about the crawled website and navigate the user to relevant pages on the site.
Delete a crawled website
Go into the “Knowledge” tab of the create/edit agent flow and click the trash icon next to the website you want to delete under the “Custom Knowledge” section.