Robots.txt checker for Google and AI crawlers
Type your website. I read its robots.txt, visit the page as each search and AI bot would, and name what blocks them: a rule, a firewall or a tag.
- Reading robots.txt
- Visiting as each bot
- Reading page tags
- Reading llms.txt
Every crawler can read you. Do they recommend you?
I'll ask ChatGPT, Claude, Perplexity and Gemini up to five questions your buyers ask, and show you who they name.
Run free checkFix the blocks first, then see where you stand.
Once they can read you, the next question is whether they name you. The free check shows that in about two minutes.
Run free checkWhat does robots.txt control?
robots.txt is a plain text file at the root of a site. It lists the paths each crawler may fetch. A crawler reads the group that names it, or the general group when none does. When two rules match a path, the longer one wins, and Allow wins a tie.
A missing file means no rules apply. A file the crawler cannot read blocks everything, because the standard treats it as a no. The result says which of those applies to your site.
Are search bots and training bots the same?
Search bots fetch pages to build the index that answers come from, and some fetch a page when a person asks an assistant to open it. Blocking one of them keeps your pages out of those answers.
Training bots collect text to train models. Whether to let them in is your choice. Blocking one does not hide your site from search or from the answers an assistant builds from its index.
Why can a site pass robots.txt and still block AI?
robots.txt only asks. A firewall or a CDN can refuse a crawler whatever the file says, and a tag on the page can keep it out of results. So I read each bot two ways: its rule in robots.txt, and a visit that carries the bot's own user agent string.
I cannot confirm Googlebot or Bingbot from outside. Both check their visits by IP address, so a visit from here does not show what they get. When robots.txt allows them, I say I can't confirm them, and that does not count against the score.
What is llms.txt, and what is it not?
llms.txt is a short Markdown file that sums up a site for language models. Some sites publish one. No major AI engine has confirmed that it uses the file, so I show it as a tip, and it has no effect on the score.
The draft I offer uses only the title and description of your home page. I add no fact that is not already on it, so check the draft before you publish it.
How do I fix each result?
Each problem comes with one next step. Allow the bot in robots.txt and check again. If your CDN's bot settings or a firewall stop a bot, let it through there. If a tag on the page keeps it out, remove the tag. The result also gives you lines to copy into robots.txt, and they keep every path your site already closes.
For the rest of the picture, read how AI decides which companies to name, then ChatGPT SEO and Perplexity SEO. A crawler that can read your site is only the first step.
Questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects pages to train models. ChatGPT's search reads your site through OAI-SearchBot, and ChatGPT-User fetches a page when a person asks it to open a link.
Blocking GPTBot leaves both of those working. Blocking OAI-SearchBot is the block that keeps you out of ChatGPT search answers.
What is OAI-SearchBot?
OAI-SearchBot is the crawler for ChatGPT search. It reads pages to build the index that those answers come from.
It is a search bot, not a training bot, so blocking it keeps your pages out of those answers. It is separate from GPTBot.
Do I need llms.txt?
No. No major AI engine has confirmed that it uses llms.txt, so the file does not change your score here.
Write one only if it helps you describe the site, and keep it to facts that are already on your pages.
Why does my CDN block bots?
A CDN or firewall can refuse a crawler whatever robots.txt says. It usually does this to stop scrapers, through a bot setting that is switched on.
Look for the bot settings in your CDN's dashboard, let the crawler through there, then check again.
How often should I check?
Check after any change to robots.txt, to your CDN or firewall settings, or to the page template that adds tags. Otherwise a check every few weeks is enough to catch a change someone else made.
A second run of the same address within an hour shows the saved result, so it costs you nothing extra.
Does AI name you today?
Free, about two minutes, no login. Up to five buyer questions written from your site, four AI assistants.