robots.txt vs llms.txt: what to allow and why

Training crawlers and retrieval crawlers are different decisions. Confusing them is the most common way sites accidentally opt out of AI search.

There are two kinds of AI crawler and they deserve two different decisions.

Training crawlers such as GPTBot, ClaudeBot, and CCBot collect pages to train future models. Blocking them is a legitimate policy choice with essentially no effect on whether you appear in AI answers today.

Retrieval crawlers such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, and PerplexityBot fetch pages to answer a question someone is asking right now. Block these and you’ve opted out of AI search, usually without meaning to.

Plenty of sites block the second group while trying to block the first, because a blanket rule is easier to write than a careful one.

robots.txt is the first gate

Worth checking explicitly for:

• OAI-SearchBot, ChatGPT-User
• Claude-SearchBot, Claude-User
• PerplexityBot
• Googlebot, Bingbot

If any of these hit Disallow: /, nothing downstream matters until it’s changed.

Allow in robots is not the same as reachable

This one catches people out. Your robots file can say Allow while your edge returns 403 to anything that doesn’t look like a browser. Cloudflare’s bot protection is the usual culprit, and it operates entirely outside robots.txt.

The only reliable test is an actual fetch. If a request from a non-browser user agent gets blocked, your robots file is a statement of intent rather than a description of reality.

llms.txt is documentation, not a gate

The llms.txt spec describes a markdown file at /llms.txt summarising what your site is and which URLs matter. Worth getting right:

• Returns HTTP 200 with real markdown, not your HTML 404 page with a 200 status
• Contains a site name, a plain summary, and a list of key links
• Every URL listed actually resolves

A carefully written llms.txt behind a 403 does nothing. Access before content, every time.

Check yours

Our site audit checks robots.txt rules for Googlebot, Bingbot, OAI-SearchBot, and the main AI crawlers, makes one live request with an AI-crawler user agent to catch firewall blocks, and tests whether your llms.txt exists, says enough to be useful, and links to URLs that load. For the user-triggered agents such as ChatGPT-User and Claude-User, read your robots file by hand for now.