Crawler access
Reads robots.txt and tells you, bot by bot, whether the homepage is allowed, with the exact line that decides it.
Free tool ยท No sign-up
Can AI crawlers read your website?
The AI Visibility Check is a free tool from Awais Consulting. It tests whether AI crawlers such as GPTBot, ClaudeBot and PerplexityBot can reach and read your homepage, by checking robots.txt, fetching the page live as each bot and scoring the result out of 100.
It takes about 30 seconds. No sign-up, and no personal data is stored.
Updated
What it checks
Seven checks, all measured live, all repeatable.
Each result shows its evidence, so you can confirm it yourself in a minute.
Live fetch as each bot
Loads the homepage as a normal browser and then as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot, and compares the answers.
Indexability
Looks for noindex, nosnippet and a canonical tag that points somewhere else.
Readable without JavaScript
Counts the words a crawler can read in the raw HTML, before any script runs.
Business schema
Looks for Organization or LocalBusiness JSON-LD with your name, website, address and phone number.
Answer-first basics
Checks the title, the meta description, a single H1 and a short, direct opening paragraph.
Sitemap
Looks for a readable XML sitemap, declared in robots.txt or at the usual address.
The bots
Thirteen crawlers, and what each one does.
Purposes follow what each company publishes about its own crawler.
| Bot | Company | What it does | Type | Checked by |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Finds and surfaces websites in ChatGPT search. Not used for training. | AI search | robots.txt and live fetch |
| ChatGPT-User | OpenAI | Visits a page when a person asks ChatGPT about it. OpenAI says robots.txt rules may not apply to these visits. | User-triggered | robots.txt |
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI models. | AI training | robots.txt and live fetch |
| Claude-SearchBot | Anthropic | Crawls the web to improve the relevance and accuracy of search results for Claude users. | AI search | robots.txt and live fetch |
| Claude-User | Anthropic | Visits a page when a Claude user asks a question that needs it. | User-triggered | robots.txt |
| ClaudeBot | Anthropic | Collects public web content that may be used to train Anthropic models. | AI training | robots.txt and live fetch |
| PerplexityBot | Perplexity | Helps pages appear in Perplexity search results. Not used to train foundation models. | AI search | robots.txt and live fetch |
| Perplexity-User | Perplexity | Visits a page when a person asks a question. Perplexity says this fetcher generally ignores robots.txt. | User-triggered | robots.txt |
| Googlebot | The main crawler behind Google Search. | Search | robots.txt and live fetch | |
| Google-Extended | A robots.txt token, not a separate crawler. It controls whether Google uses your content for Gemini training and grounding. Google says it does not affect Google Search. | AI training | robots.txt | |
| Bingbot | Microsoft | The crawler behind Bing Search. | Search | robots.txt and live fetch |
| Applebot-Extended | Apple | A robots.txt token that controls whether Apple uses your content to train its AI models. | AI training | robots.txt |
| CCBot | Common Crawl | The crawler behind the Common Crawl open web dataset, which anyone can use, including AI developers. | AI training | robots.txt |
Blocking an AI training bot is a policy choice, so it is shown as information and never lowers the score.
Method
How the score is worked out.
The score is out of 100: crawler access 40, live fetch 20, indexability 10, readable text 10, schema 10, answer-first 5, sitemap 5.
- 1
Step 1
Browser first
I load your homepage like a normal browser. This is the control.
- 2
Step 2
Read robots.txt
I find the group that applies to each bot and the longest matching rule.
- 3
Step 3
Fetch as each bot
I request the homepage with each bot name and compare status and size with the control.
- 4
Step 4
Read the page
I check the HTML, the schema and the sitemap, then rank the fixes by points lost.
Honesty boundary
What can this check not tell you?
I send each bot name from my own server. Some firewalls verify real bots by IP address, so the real bot can be treated differently. A block shown here is a strong signal, not proof. If your site also blocks a normal browser request, the result is marked inconclusive and no bot is blamed. The check looks at the homepage only, and it does not tell you whether an AI assistant mentions your business.
Results are kept for 30 minutes so repeat checks do not load your site again. I keep a one-way hash of the visitor address for an hour to enforce the limit of six checks per hour, and I store no names or emails.
Questions
What do people ask about AI crawlers?
Should I block GPTBot?
It depends on what you want. GPTBot collects content that may be used to train OpenAI models. Blocking it does not remove you from ChatGPT search, which uses a separate bot called OAI-SearchBot. If you want to appear in ChatGPT search answers, allow OAI-SearchBot. Whether to allow training is your choice.
Does blocking Google-Extended hurt my Google rankings?
No. Google states that Google-Extended does not affect the inclusion of a site in Google Search and is not used as a ranking signal. It only controls whether your content is used for Gemini training and grounding.
Why does my result say my firewall blocks a bot?
The check loads your homepage twice: once as a normal browser and once with the bot name. When the browser gets the page and the bot gets an error or a challenge page, something between the bot and your site, usually a firewall, a CDN or a security plugin, treated the bot differently. Some firewalls verify real bots by IP address, so the real bot may be treated differently from this test. Ask your host or developer to review their bot settings.
Is this the same as the Visibility Breakdown?
No. This tool is an instant technical check of whether AI crawlers can reach and read your homepage. The Visibility Breakdown is a free one-page report that I prepare by hand: which businesses Google and AI answers name for your searches, why they win, and what to fix first.
Why is my result marked inconclusive?
Some sites block automated requests from any source, including a normal browser request from this tool. When that happens the check cannot tell whether AI bots are treated differently, so it reports what it could read, such as robots.txt and the sitemap, and gives no score.
Does llms.txt help?
I do not score it. I have not seen proof that an llms.txt file changes whether AI answers cite a site, so the tool only reports whether the file exists.
A clean result is not the same as being chosen.
This check shows whether AI crawlers can read your site. A Visibility Breakdown shows which businesses Google and AI answers name instead of you, and why.
Find the real reason first. Then build only what fixes it.
Or write to awais@awaisconsulting.com