Free check. See if ChatGPT, Claude, Perplexity and Google AI can access your site, in 10 seconds.
AI assistants like ChatGPT, Claude and Perplexity read the web live when they answer questions. If your robots.txt or firewall blocks their search bots, they cannot read your pages, and they are much less likely to cite or recommend you. Access is the first requirement for showing up in AI answers.
AI companies run different bots for different jobs. Search and user bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) fetch pages to answer questions right now; blocking them hurts visibility. Training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent) collect data to train models; blocking them is a business choice and does not directly remove you from AI search answers.
llms.txt is a proposed plain-text file at the root of your site that gives AI assistants a short summary of who you are and links to your most important pages. It is not a standard yet and support varies, but it is quick to add.
Not directly. GPTBot collects training data. ChatGPT search uses OAI-SearchBot and ChatGPT-User, so those are the ones to allow if you want to appear in answers.
The robots.txt part follows the standard matching rules. The live request test uses bot user agents from our server, not the bots' real IP addresses, so firewalls that verify IPs may treat real bots differently.
Googlebot crawls for Google Search, including AI Overviews and AI Mode. Google-Extended only controls whether your content is used to train Gemini models.
It is optional. llms.txt is a proposed standard and AI assistants do not all use it yet, but it is a cheap way to point them to your most important pages.
Many AI crawlers do not run JavaScript. If your homepage shows little text in the raw HTML, render the key content on the server.