Crawl, index, answer
Three stages, and what each one does with your pages — including where it stops.
HootyBot reads the site
Nothing is fetched until someone adds a site as a source in the app. HootyBot is not a search or training crawler: it does not roam the web looking for new sites. When it does visit, it requests /robots.txt before anything else and follows it per RFC 9309, including * and $ patterns.
- One page at a time per site, never in parallel, at least half a second apart. Crawl-delay is honoured.
- It backs off on 429 and 503 and obeys Retry-After exactly. If you ask for a long wait, it gives up rather than come back early.
- HTML pages only, and a limited number of them per site. It never submits a form, logs in, or touches a cart or a checkout.
- If /robots.txt returns a server error it does not assume permission — it stops crawling the site until the file can be read again.
Hooty indexes what came back
The text of the pages that were read becomes the material the assistant is allowed to use. Pages you disallowed are not in it. Pages that could not be fetched are not in it. Nothing from anywhere else is in it.
The pages are not used to train models.
The assistant answers from that material
A visitor asks a question in their own words instead of guessing at keywords. The answer is composed from the indexed pages and names the ones it drew on, so it can be checked against the source.
When the material does not cover the question, that is what the visitor is told. Hooty does not fill the gap.
When a crawl reads nothing
The real failure modes, in the order you are likely to meet them.
Bot protection refuses the crawler
The most common one. A WAF or anti-DDoS product challenges HootyBot, and the crawler does not run JavaScript, so it cannot pass a browser check or solve a captcha — and it will not try. A challenge served with a 200 status is discarded rather than stored as content. Where Hooty can recognise the product, it names it in the error.
Allowing HootyBot through bot protectionThe page is assembled in the browser
If your server sends an empty shell and the text only appears once JavaScript has run, there is nothing in it to read. The crawler needs server-rendered HTML, or a prerendered copy.
robots.txt disallows the pages
A disallow is obeyed, including one nobody remembers writing. If a section should be readable, allow HootyBot to it.
The site did not answer
No response at all. HootyBot stops rather than retry into a wall.
Stopping it
Two lines in robots.txt and HootyBot stops:
User-agent: HootyBot
Disallow: /To keep it out of one section only:
User-agent: HootyBot
Disallow: /internal/Changes take effect within an hour. If you need it to stop sooner, write to bot@hooty.to. We answer the people who run the sites we read.
Write about the crawlerSee what your own pages can answer
Add a site in the app and let HootyBot read it.