ChatGPT, Claude, Perplexity, Google, Copilot, Siri and Meta AI each send their own crawler, and robots.txt decides which of them get in. Enter a domain or any page: we read your robots.txt the way the crawlers do and show, bot by bot, who is allowed — and which blocks cost you answers rather than just training.
17 user agents · OpenAI · Anthropic · Perplexity · Google · Microsoft · Apple · Meta · Common Crawl · ByteDance — no card, no account
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and bingbot fetch your pages so an assistant can show them in an answer and link to them. Blocked, you are simply absent.
GPTBot, ClaudeBot, CCBot and the rest collect pages for training future models. Blocking them is a reasonable choice and does not remove you from today's answers.
Google-Extended only governs Gemini training and grounding; Google says it does not affect Search. AI Overviews and AI Mode come from Googlebot — the one you cannot block without leaving Google.
Read with the RFC 9309 rules: the crawler obeys every group that names it, otherwise the * group; the longest matching rule wins, Allow wins a tie, and * and $ patterns are honoured. Crawler purposes are taken from each vendor's own documentation, checked 6 October 2026. We fetch robots.txt once and cache it for an hour; nothing is stored.