Which AI crawlers does your robots.txt let in?
The checker fetches robots.txt, llms.txt and the sitemap, evaluates the rules for the AI user agents that matter in 2026 and quotes the exact line each decision came from.
Enter any URL on the site; the checker walks up to the origin and reads the files from there.
Observations, not verdicts. What you enter is processed for this result and written to a usage journal described in the privacy policy; nothing else is stored.
What is evaluated and how
Three files are fetched from the origin of the URL you enter: /robots.txt, /llms.txt and whichever sitemap robots.txt declares. The fetch is a plain HTTP request from a server in the EU with a named user agent; no browser is started, no script runs.
robots.txt, evaluated per bot
The file is parsed the way RFC 9309 describes. Rules are grouped by User-agent, a bot uses the group that names it and falls back to the * group only when no group does, and within a group the rule with the longest matching path wins, with Allow taking precedence over Disallow of equal length. The checker runs that evaluation for the path you entered and for / against GPTBot, ClaudeBot, Google-Extended and PerplexityBot, plus a handful of others, and shows the result beside the line that produced it. A blanket Disallow: / in the * group with no bot-specific group is the most common reason a site blocks every AI crawler without meaning to, and it is shown as exactly that.
One detail people miss: Google-Extended is not a crawler. It is a token Googlebot reads to decide whether pages may be used for Gemini and Vertex AI training. Blocking it changes nothing about crawling or search ranking, and the report words it that way.
llms.txt, checked against the proposal
llms.txt is a proposed convention from September 2024: a Markdown file at the site root that tells a language model where the useful content is. The checker reads it as Markdown and reports what the proposal asks for. An H1 with the site name is required. An optional blockquote summary may follow. Further content is organised in H2 sections that contain lists of links, each in the form - [title](url): note. A section titled Optional marks links a reader may skip. Findings name what is missing or malformed: no H1, links outside a list, a section without links, or a 200 response that is in fact an HTML page, which usually means a soft 404 rather than a file.
What the checker does not tell you
robots.txt is a request, not an access control. Whether GPTBot honours a Disallow is a matter of OpenAI's published behaviour, which the tool cannot observe; it only tells you what the file asks. It does not read meta robots tags or X-Robots-Tag headers, which also affect crawlers and can contradict the file. It does not evaluate the TDM reservation mechanisms discussed under Article 4 of the DSM directive and § 44b UrhG, and it makes no claim about whether a robots.txt line counts as one. Finally, the presence of an llms.txt is a fact about the site, not evidence that any model reads it: adoption by the large providers is uneven and undocumented, and the report says so rather than implying otherwise.
How to read the result
-
Check the fetch line first
Each of the three files gets a status: found, missing, or answered with something that is not the file. A 200 with an HTML body is flagged; it is not a robots.txt.
-
Read the per-bot table
One row per user agent, allow or disallow for the path you entered and for the root, and the matching line quoted from the file. A row that fell back to the * group says so.
-
Look at the llms.txt findings
Structure findings are listed in the order they appear in the file, each with the line number, so a missing H1 or a link outside a list is quick to fix.
-
Compare with the sitemap
URLs the sitemap declares but the rules disallow for a given bot are listed separately, because that combination is almost always a mistake.
Questions about robots.txt and llms.txt
What is llms.txt?
A proposed convention for a Markdown file at /llms.txt that gives language models a curated map of a site: a title, a short summary and sections of links with notes. It is not a standard, and no major provider has documented that it reads the file.
How do I block GPTBot in robots.txt?
Add a group with User-agent: GPTBot followed by Disallow: /. Because a bot uses its own group when one exists, a later Allow in the * group does not override it.
Does blocking Google-Extended hurt my search ranking?
Google states that Google-Extended controls use for Gemini and Vertex AI training and has no effect on Search. It is not a crawler user agent; Googlebot keeps crawling the same pages.
Is robots.txt legally binding for AI crawlers?
It is a voluntary convention. Whether a robots.txt line is also a valid machine-readable reservation under Article 4 of the DSM directive is a legal question the tool does not answer; it only shows what the file asks.
Why does the tool say my llms.txt is an HTML page?
The server returned status 200 with HTML, which is what many CMSs do for unknown paths. A model that fetches the file gets a rendered not-found page. Configure a real 404 or publish a Markdown file.
Note · This is the crawler view, not the agent view
A crawler reads robots.txt before it fetches. A browser agent that a customer sends to their own account is not a crawler and does not consult the file; it acts as the customer would in a browser. For what such an agent sees on a page, use the agent view tool.
See what an agent sees on the same page
Five browser hours, no card. Open the URL in a real EU-hosted browser and read the tree the agent gets.