HootyBot

How to identify it

Every request HootyBot makes carries these headers:

User-Agent

Mozilla/5.0 (compatible; HootyBot/1.0; +https://hooty.to/bot)

From

bot@hooty.to

The product token to use in robots.txt is HootyBot. HootyBot never disguises itself as a browser and never uses a different name.

When it visits

HootyBot only crawls a site after someone sets that site up as a knowledge source in hooty.to. It is not a general-purpose search or training crawler: it does not roam the web looking for new sites, and it does not use the pages it reads to train models.

How it behaves

• It fetches /robots.txt before anything else and follows it per RFC 9309, including * and $ patterns.
• It honours Crawl-delay, waits at least half a second between requests, and fetches one page at a time per site, never in parallel.
• It backs off on 429 and 503 and obeys Retry-After exactly, never retrying sooner than you asked. If you ask for a long wait, it gives up rather than come back early.
• If /robots.txt returns a server error, it does not assume permission. It stops crawling the site entirely until the file can be read again.
• If your server stops answering, it stops trying, rather than retrying into a wall.
• It requests only HTML pages, follows a limited number of them per site, and never submits forms, logs in, or touches carts and checkouts.
• It does not execute JavaScript challenges or attempt to work around bot protection. If your protection blocks it, including a challenge served with a 200 status, it discards the page and reports the block rather than storing the challenge as content.

Blocking HootyBot

Add this to your robots.txt and HootyBot will stop:

User-agent: HootyBot
Disallow: /

To keep it out of one section only:

User-agent: HootyBot
Disallow: /internal/

Changes take effect within an hour. If you need it to stop sooner, email us at bot@hooty.to.

Allowing HootyBot

If you use a WAF or anti-bot service (Cloudflare, Qrator, DDoS-Guard and similar), it may challenge or refuse HootyBot even when your robots.txt permits it. The crawler does not run JavaScript, so it cannot pass a browser challenge. To let your own site be read for your assistant, allow-list the HootyBot user agent in your protection rules. There are step-by-step instructions for each product on the allow-listing page.

Contact

Questions, complaints about crawl rate, or a request to stop: bot@hooty.to. We answer people who run the sites we read.