AI robots.txt Rules: Separate Training, Search and User Fetches

An AI-crawler policy has three separate questions: may a provider crawl for training, may its search system discover your pages, and may it retrieve a page when a user asks? The names are similar, but the controls are not interchangeable.

This guide focuses on the exact file and verification steps. For the broader policy choice, use our guide to allowing AI search while restricting training. Neither guide promises a citation, search placement or removal of material already used in a model.

Provider documentation checked on 16 October 2026. The examples below are unexecuted templates for your own website.

Match the token to the purpose

Use the providers' current documentation: OpenAI crawlers, Anthropic crawlers and Google common crawlers. Google-Extended is a robots control token rather than a separate HTTP user-agent string to find in your logs.

Do not treat these names as a complete list of every AI system. Publishers must make an explicit choice for other providers and for future agents.

Preserve the rules your website already needs

Fetch the current file and save a copy before editing:

curl --show-error --silent \
  --connect-timeout 10 --max-time 20 \
  --output robots-before.txt \
  https://example.com/robots.txt

Check the status and body too. A 404 response or an HTML error page is not a valid backup of an existing robots file.

Identify the source of the response: a physical file, a CMS-generated route, a framework deployment or an edge rule. On WordPress, a virtual file can be produced by do_robots(). Editing a plugin setting will not necessarily change a physical file that the web server serves first.

Keep valid sitemap entries and existing restrictions unless you have a specific reason to change them. Copying a short example over a mature site's entire file can remove unrelated rules.

Example: restrict the named training controls

Add or merge the following groups when that matches your policy:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

This sample deliberately does not add a wildcard group or replace the rest of the file. It targets the named controls. Audit your existing search-agent and wildcard groups separately so they do not accidentally contradict the intended search policy.

Under RFC 9309, matching user-agent groups and path rules determine the applicable restrictions. A more-specific group can mean a bot does not use the wildcard group's rules. If you add a bot-specific group that should retain existing path restrictions, include those restrictions in that group's intended policy too.

That detail matters for a site which allows search crawling but wants selected paths excluded. Do not assume a broad wildcard block automatically combines with every specific allow group.

Requests initiated by users need separate thought

Anthropic documents robots controls for Claude-User. OpenAI's ChatGPT-User documentation says robots rules may not apply to certain user-initiated actions. A list of names followed by Disallow: / is therefore not a reliable universal “block every AI access” control.

For private content, use authentication and authorisation. If you need to deny an actual request, use an appropriate access-control layer and understand what it matches. A user-agent name can be imitated, and blocking an IP range can also affect legitimate features. Restrict access deliberately rather than assuming robots.txt is a firewall.

Do not put sensitive paths in a robots file as a secrecy measure: the file itself is public.

Publish at the host's root

The location is /robots.txt on the relevant scheme, hostname and port. example.com, www.example.com and blog.example.com can serve different files. Inventory the hosts that actually publish your content.

In a control panel, identify the correct document root before uploading. For a framework application, use the supported public-file or route mechanism and make sure the file survives deployment.

If a CDN caches the response, verify the public version after release. A correct repository file is not evidence that visitors and crawlers are receiving it.

Check delivery without pretending to be the crawler

Retrieve headers and body with a normal request:

curl --show-error --silent \
  --connect-timeout 10 --max-time 20 \
  --dump-header robots-headers.txt \
  --output robots-body.txt \
  https://example.com/robots.txt

Confirm the intended status, a plain-text response and the exact policy. Look for authentication screens, challenge pages, redirects to unrelated hosts and application HTML returned with a successful status.

A second request with a different user-agent can reveal response variation:

curl --show-error --silent \
  --connect-timeout 10 --max-time 20 \
  --user-agent 'GPTBot' \
  https://example.com/robots.txt

This is only a local header simulation. It does not originate from OpenAI, authenticate a crawler, demonstrate provider compliance or reproduce an IP-based WAF rule. For real visits, inspect access logs and validate crawler identity using the provider's published method.

Diagnose changes with the right evidence

The edited rules do not appear: confirm which layer serves the file, deployment status and caches. A local or admin preview may differ from the public response.

A crawler cannot read the file: inspect response status and security controls. RFC 9309 distinguishes an unavailable file from an unreachable service; a 404 and a server failure are not interchangeable crawler signals.

Search visibility changes: check search-specific groups, wildcard rules, page-level indexing controls and actual crawl errors. Do not attribute a visibility change to a training token without evidence.

No immediate behaviour change: providers can cache policy information and refresh on different schedules. OpenAI currently describes an adjustment period of approximately 24 hours for its search systems after a robots update. That is not an indexing or citation deadline.

Google-Extended does not appear in logs: that is expected for a control token without its own HTTP user-agent string. Inspect the published file and the documented purpose instead of waiting for a request with that exact name.

Record the owner, date and business intent of the policy. Recheck provider documentation when changing it. For Google-specific diagnostics, the robots.txt report can help investigate Google's access; it is not a verifier for every other provider.