agentsiteaudit · Checks · AI agent readiness

AI crawlers and robots.txt: let AI agents read your website

robots.txt is a small file at the top of your website that tells automated visitors which pages they may read. AI companies now send their own visitors, with names such as GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended.

If your robots.txt blocks them, AI assistants cannot read your pages when someone asks about you. Blocking them is a legitimate choice, and some businesses make it on purpose. The problem is when it happens by accident.

In our November 2026 study of businesses selling to AI agents, 9% of the 1,139 websites we checked blocked at least one of the main AI crawlers. See the full study.

Why it matters

More people now ask an AI assistant instead of searching. If the assistant cannot read your site, it may describe you from old or second-hand information, or not mention you at all.

Accidental blocks are common: a rule left over from when the site was being built, a hosting setting, or a security tool that blocks all AI bots by default.

What our check looks for

• Pass: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended may read the page
• Warning: One or more of them blocked

Check ID agents.robots.ai_crawlers · AI agent readiness · Medium priority

How to fix it

Allow everything (the simplest robots.txt)

User-agent: *
Allow: /

Sitemap: https://www.example.co.uk/sitemap.xml

This lets every well-behaved crawler read your site, including the AI ones.

Allow AI search, block AI training

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Some businesses want to appear in AI answers but not have their content used to train models. Our check will show a warning for the blocked ones, because they are blocked; if that is your choice, you can ignore it.

cPanel hosting

public_html/robots.txt

Open File Manager, edit robots.txt in public_html, and remove any Disallow rules for the AI bots you want to allow.

WordPress

Settings > Reading > Search engine visibility

Make sure "Discourage search engines from indexing this site" is not ticked. SEO plugins can also edit robots.txt; check their robots.txt setting if the file seems to change by itself.

Cloudflare

Security settings > Bots / AI Crawl Control

Cloudflare can block AI crawlers at its edge and can also manage your robots.txt for you. Check those settings match what you want, because they override the file on your server.

Common mistakes

• A "Disallow: /" under "User-agent: *" left over from when the site was being built. That blocks every crawler, not just AI ones.
• Blocking all AI bots with a hosting or CDN switch without realising it also stops AI search assistants.
• Confusing the bots: GPTBot is for training, OAI-SearchBot is for ChatGPT search. Blocking one does not block the other.
• Expecting robots.txt to stop bad bots. It only works for crawlers that choose to obey it.

Related checks

• llms.txt: how to add a guide for AI agents to your website
• Tell AI agents who your business is: name and contact details as structured data

Written by Derek Moore, who builds and runs websites for small UK businesses. Last checked 30 October 2026.