We host on Hostinger — get up to 20% off your first plan.Get the link →
Referral link · we may earn a commissionReferral link — we may earn a commission at no extra cost to you.Download a working automation JSON
Can AI answer engines cite your site?
Outreach scripts that get replies
Find the 3 workflows costing you sleep
200+ prompts, ready to ship
12-month posting plan in 60 sec
Brand voice doc in 4 steps
Runway / Pika / Sora / Veo formats
Enter a domain and this tool fetches its real robots.txt, then parses it per AI crawler — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and six more — so you see exactly which ones are allowed, blocked, or partially blocked, and why.
Plus a check for /llms.txt — presence and basic structure. Updated August 2026.
Fetches the real /robots.txt and /llms.txt over a server-side proxy — no fabricated results.
How it works
Type the bare domain or full URL — the tool normalizes it to https://yourdomain.com automatically.
Each of the 10 tracked AI crawlers gets a real ALLOWED / BLOCKED / PARTIAL verdict, which User-agent group matched, and the exact blocked paths if any.
See whether the file exists and whether it has the basic structure (H1, sections, links) the emerging convention expects.
If a bot you actually want indexing/citing your content is blocked, that's a one-line robots.txt fix — or use the llms.txt Generator and AEO Audit tools linked below.
Why robots.txt matters for AEO
A lot of robots.txt files were written years before GPTBot or ClaudeBot existed, then copy-pasted from a boilerplate or a WordPress security plugin that blanket-blocked every unknown crawler as a precaution. That's a reasonable default for scrapers — it's not a reasonable default anymore for a domain that wants to show up when someone asks ChatGPT or Claude about it.
The fix isn't "unblock everything." Google-Extended specifically controls Gemini/AI Overviews training, separate from regular Googlebot indexing — you can keep normal SEO crawling on and still opt a site out of AI training. The point of this tool is visibility into what your file actually says, per bot, so the decision is deliberate.
| Crawler | What it's for |
|---|---|
| GPTBot | OpenAI training crawler |
| OAI-SearchBot | ChatGPT search |
| ChatGPT-User | live browsing plugin |
| ClaudeBot | Anthropic crawler |
| Claude-SearchBot | Anthropic search |
| PerplexityBot | Perplexity search |
| Google-Extended | Gemini/AI Overviews training |
| Bytespider | ByteDance/TikTok crawler |
| CCBot | Common Crawl (feeds many LLM training sets) |
| Applebot-Extended | Apple Intelligence training |
Waseem, building from Bali · info@skynetjoe.com · Updated August 2026
Quick answers
It's a real fetch and parse of your site's actual /robots.txt (and /llms.txt), done over a server-side proxy so the browser can read a cross-origin file. Nothing here is a lookup table or a guess — the ALLOWED/BLOCKED/PARTIAL verdict is computed from the User-agent groups and Allow/Disallow rules actually present in your file.
GPTBot and OAI-SearchBot (OpenAI), ChatGPT-User (ChatGPT's live browsing), ClaudeBot and Claude-SearchBot/anthropic-ai (Anthropic), PerplexityBot, Google-Extended (Gemini/AI Overviews training), Bytespider (ByteDance), CCBot (Common Crawl — feeds many other models' training sets), and Applebot-Extended (Apple Intelligence) are the named user-agents the major AI platforms document publicly for crawling and/or training.
The bot isn't blocked at the root (no blanket Disallow: /), but specific paths are disallowed for it — for example /wp-admin/ or /private/. The tool lists exactly which paths from your real robots.txt triggered that.
If there's no matching User-agent group and no wildcard (User-agent: *) group either, the verdict is ALLOWED — that's how robots.txt actually works: no rule means no restriction. If the file itself doesn't exist or didn't return HTTP 200, that's shown separately as 'No robots.txt'.
No — it's an emerging, non-standardized convention (not an official web standard) that some AI platforms may use to understand a site's structure. This checker reports whether one exists and does a basic structural sanity check (H1, sections, links), not a compliance grade against a spec that doesn't formally exist yet.
Not necessarily — some answer engines browse live (like ChatGPT-User) even when the training crawler (GPTBot) is blocked, and citations can come from crawled/indexed pages elsewhere. But a blocked crawler is still a deliberate signal you should be making on purpose, not by an inherited robots.txt nobody's looked at.
A short AEO fix plan covers robots.txt, llms.txt, and the schema markup that actually gets you cited.
Book a discovery call