Which AI crawlers do the top 1,000 websites block?
Published · Data collected · Viscovery, Pepper Square
The short answer. We read the robots.txt files of the top 1,000 websites in the Tranco ranking and could read 488 of them. Of those 488:
- 79.5% allow every AI crawler. 388 of 488 sites ask none of the 10 crawlers we checked to stay away.
- 20.5% block at least one, and 4.1% (20 sites) block all 10.
- Crawlers that do not serve answers are blocked most often. CCBot, which builds Common Crawl’s archive, is blocked by 17.4%, and the two training crawlers GPTBot and ClaudeBot by 15.2% and 14.8%. The least blocked is OAI-SearchBot, ChatGPT’s search crawler, at 8.4%.
- Of the 74 sites that block GPTBot, 34 (45.9%) leave OAI-SearchBot open: they ask OpenAI not to train on their pages, but can still appear in ChatGPT search answers.
- 19.3% publish an llms.txt file, a plain-text summary of the site written for AI assistants.
Crawler by crawler
The 10 crawlers below are the ones our free AI visibility checker reads. “Answers” crawlers decide whether an assistant can find or read your pages when it answers; “training” crawlers decide whether your pages are used to train models; CCBot builds Common Crawl’s open archive of the web. Google-Extended counts as answers here because Google says it also governs grounding in Gemini’s apps.
| Crawler | What it feeds | Kind | Sites blocking it | Share of 488 |
|---|---|---|---|---|
| CCBot | Common Crawl's open web archive | Archive | 85 | 17.4% |
| GPTBot | OpenAI's model knowledge of your brand | Training | 74 | 15.2% |
| ClaudeBot | Claude's model knowledge of your brand | Training | 72 | 14.8% |
| Google-Extended | Gemini's training and its Search-grounded answers | Answers | 66 | 13.5% |
| PerplexityBot | Perplexity answers and citations | Answers | 57 | 11.7% |
| Applebot-Extended | Apple's model knowledge of your brand | Training | 53 | 10.9% |
| ChatGPT-User | ChatGPT browsing your page on request | Answers | 50 | 10.2% |
| Claude-User | Claude reading your page on request | Answers | 47 | 9.6% |
| Claude-SearchBot | Claude's search results | Answers | 43 | 8.8% |
| OAI-SearchBot | ChatGPT search results | Answers | 41 | 8.4% |
16.2% of sites block at least one answers crawler, and 4.3% block only training or archive crawlers, leaving every answers crawler open.
The top 100
The most-visited sites block more. We could read 52 of the top 100. Of those, 13 (25.0%) block at least one AI crawler and 6 block all 10. With 52 sites counted, read these as a direction rather than a precise rate.
What this means for your site
Blocking a training crawler is a decision about whether your pages train future models, and many sites make it on purpose. Blocking an answers crawler is different: it asks that assistant not to find or read your pages when someone asks it about your category. A robots.txt rule written years ago, or copied from a template, can do the second without anyone choosing it. The free checker reads your robots.txt the same way this study did and names each crawler you block. viscovery.ai blocks none of them.
How we measured
- The list. The top 1,000 domains of Tranco list Y83KG, created 2026-10-03. Tranco is a research ranking that combines five popularity sources over 30 days.
- The requests. On 2026-10-04, one request each for
/robots.txt,/llms.txtand the homepage, sent asViscoveryPreviewBot/1.0with the same engine and 8-second timeout as the free checker (what our crawler does), from a single network in India. - What counted. A site counted if its homepage returned a web page and its robots.txt was readable or absent. An absent robots.txt allows every crawler. 415 counted sites had a robots.txt and 73 had none.
- What did not. 414 domains refused our robots.txt request, failed, did not answer within 8 seconds, or have no web server at all; 98 more returned no web page at the homepage, either because they are content-delivery or API hosts or because the homepage refused our scanner. Neither group is in any percentage.
- “Blocked” means the site’s robots.txt disallows that crawler from the homepage path, resolved the way crawlers do: a group naming the crawler beats the
*group, and the longest matching rule wins. robots.txt is a request, so this measures what sites ask, not what any crawler does. OpenAI notes that robots.txt rules may not apply to ChatGPT-User, because a person starts those fetches. - A snapshot. Sites change their robots.txt, and some answer differently by country or by visitor. These results describe what each site returned to us on 2026-10-04.
Download the data
One row per domain, as CSV: rank, domain, whether it counted, its robots.txt and llms.txt state, and allowed or blocked for each of the 10 crawlers. Excluded domains are included and marked. If you use these figures, please link to this page.