Which AI crawlers to allow, and the block you probably did not know about
The difference between training crawlers and the ones that fetch your page to answer a live question, and how to check what your site is doing right now.
A number of sites are invisible to AI assistants for a reason that has nothing to do with their content. They are blocking the crawlers, usually without knowing it.
Check this before you spend a month on optimisation.
Two kinds of crawler
The distinction matters and most advice skips it.
Training crawlers collect text to train future models. Their effect is delayed by a model generation and you are not in this year's answers because of them.
Live retrieval crawlers fetch a page right now, because a user asked a question and the assistant decided to search. These decide whether you appear in answers today.
Blocking the first is a legitimate business decision with real arguments on both sides. Blocking the second removes you from the conversation immediately, and almost nobody who does it meant to.
The user agent names change as vendors reorganise them, so rather than memorising a list, look at what your robots.txt actually says and check each unfamiliar agent name against the vendor's own published documentation. Every major AI vendor publishes a page naming their crawlers and what each one is for.
How to check your site in three minutes
1. Open yourdomain.com/robots.txt.
Read it. You are looking for two things: named AI user agents with Disallow rules, and a blanket User-agent: * with Disallow: /.
2. Check your security layer.
Cloudflare, and several WAF and bot management products, have AI crawler blocking as a toggle. It operates independently of robots.txt and it is on by default in some configurations. Log in and look.
3. Check your platform's settings.
Several site builders and SEO plugins added AI blocking options over the last two years, some of them switched on by default in an update. Search your CMS settings for "AI".
4. Check your CDN and hosting rules.
Rate limiting and bot rules can block a crawler without any of the above showing anything.
We have audited sites where the owner spent six months on content while the front door was locked at step two. It is worth the three minutes.
Deciding what to allow
If you want to be recommended: allow the live retrieval crawlers. There is no version of appearing in an answer where the assistant cannot fetch your page.
Training crawlers are a real decision.
Reasons to allow: future models learn your brand, your positioning and your facts. For a small business with no licensing leverage, this is free distribution.
Reasons to block: you are a publisher whose content is the product, you have or want a licensing arrangement, or you have a legal or contractual reason.
For most businesses selling a product or service rather than content, allowing is the straightforward answer.
Blocking everything is a position some publishers take deliberately. It is a choice, not an accident, and if you have made it accidentally you should un-make it.
What blocking does not achieve
Two common misunderstandings.
It does not remove you from what a model already learned. Training data already collected is already collected.
It does not stop people talking about you. If a directory lists you and a forum thread recommends you, an assistant can read those pages and name you without ever fetching your site. You will simply have no control over what it says, because the sources describing you are all third party.
That last point is the strongest practical argument against blanket blocking. Blocking does not make you invisible. It makes you described entirely by other people.
After you have checked
Once the crawlers can reach you, the work is the ordinary work: be present in the third party sources that get read, state your facts in plain text, stay consistent across every profile, and measure whether the answers change.
Measurement is the part to set up now, because you want a baseline from before any of your changes. Twenty buyer questions across ChatGPT, Claude, Gemini and Perplexity, recording who is named and what is cited, weekly. By hand it is about two hours a week; our own Figo runs it from $49 a month.
Either way, take the baseline this week. In two months you will want to know what it looked like before.
Questions people ask
If I block AI crawlers, do I disappear from AI answers?
It depends which ones. Blocking a training crawler affects future models. Blocking the crawler that fetches pages live to answer a question removes you from answers today.
How do I check what my site allows?
Open yourdomain.com/robots.txt in a browser and read it. Look for Disallow rules under any AI related user agent, and for a blanket Disallow under a wildcard.
Should I allow everything?
If you want to be found and recommended, allow the live retrieval crawlers. Training crawlers are a genuine business decision, and publishers with licensing leverage often choose differently from a small business that needs visibility.
Can a firewall block them even if robots.txt allows it?
Yes, and this catches people out. Cloudflare and similar security products have AI bot blocking settings that operate independently of robots.txt.
See it on your own competitors
Figo checks their ads, pages, rankings, reviews and AI answers every week, then tells you what to do in plain words. Set up in two minutes.
Keep reading