Documentation

Hooty gives a website its own assistant, answering from that site’s own content. To do that, our crawler has to be able to read the site.

HootyBot, the crawlerWhat it is, how to identify it, how it behaves, and how to block it if you would rather it stayed away.Allowing it through bot protectionInstructions per product, for when a firewall or anti-DDoS service refuses the crawler.

My crawl failed, what now

The most common reason a crawl fails is that a bot-protection product sits in front of the site and refuses the crawler. Hooty names the product in the error when it can recognise it, and links straight to the matching instructions.

The fix is always the same shape: allow the user agent HootyBot through. Start at Allowing HootyBot through bot protection, or jump to your product in the menu.

Other reasons a crawl reads nothing: the site is rendered entirely in the browser, so there is no text in the HTML for us to read; robots.txt disallows the pages; or the site did not answer at all.

Getting help

For anything about the crawler, including a request that it stop visiting a site, write to bot@hooty.to. We answer the people who run the sites we read.

hooty.to · Last updated 26 July 2026