Do You Need llms.txt? What Google and Chrome Actually Say

For Google Search, no, you do not need an llms.txt file. Google says in plain words that Search does not use it, and that the file will neither help nor hurt your rankings there.

Here is the twist. Chrome's own Lighthouse tool now has an audit that goes looking for that exact file, and it will flag your site if the request for it ends in a server error.

By the end of this post you will know what each official source says, whether your site is the kind that should bother, and how to check your /llms.txt URL in about three minutes so it never returns a 5xx error.

What llms.txt actually is

An llms.txt file is a plain text file written in Markdown, the simple formatting style where a line starting with # becomes a heading. It sits at the root of your site, at /llms.txt, and gives AI tools a short map of what your site is and where the important pages live.

The idea comes from a proposal by Jeremy Howard on llmstxt.org. The page lists it as published on 3 September 2024 and modified on 10 August 2026, and describes itself as "A proposal to standardise on using an /llms.txt file to provide information to help agents use a website."

Think of it like the welcome sheet a host leaves for a house sitter. The house has many rooms, and the sitter could open every door to work out where things are. The sheet just says: kitchen is here, the important stuff is in these three drawers, ignore the garage unless you need it.

An "agent" here means an AI program that fetches web pages to do a task for someone, like a coding assistant looking up library docs. The proposal says llms.txt "is instead used on demand, when an agent needs information about a topic while assisting a user."

It is also a different job from robots.txt. The proposal puts it this way: "robots.txt lets automated tools know what access to a site is considered acceptable." If you want to control which AI crawlers may visit, that is a robots.txt question, and our guide to robots.txt rules for GPTBot, ClaudeBot and Google-Extended covers it.

The format, according to the proposal

The proposal lists the parts of the file in order. Only one is required.

• An H1 heading (a line starting with # ) with the name of the project or site. The page says "This is the only required section."
• An optional blockquote (a line starting with > ) with a short summary.
• Optional paragraphs or lists with more detail, but no headings in this part.
• Optional H2 sections (lines starting with ## ) that hold "file lists": each item is a Markdown link written as [name](url), optionally followed by a colon and a note.
• By convention, an H2 called "Optional" for secondary links that an agent "can skip when a shorter context is needed."

Heads up: llmstxt.org also says "The llms.txt specification is open for community input." It is a proposal, not a finished, formally adopted standard. Treat it that way.

What Google says, word for word

Google Search Central has a guide on optimizing for generative AI features in Search. At the time of writing (6 November 2026), the page shows "Last updated 2026-07-10 UTC."

In a section listing things you can ignore for Google Search, it says:

You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.

In the same bullet, it adds:

It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them.

The same page also says "There's no requirement to break your content into tiny pieces for AI to better understand it," and "Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add."

So what does Google say actually matters? This line: "To be eligible to be shown in generative AI features on Google Search, a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements."

Google says there is no separate inclusion setting for its generative AI features. To be eligible as a supporting link, a page must be indexed, eligible to appear in Search with a snippet, and meet Search's technical requirements. Search Console reports traffic from these features within the Web search type rather than through a special opt-in control.

Why this matters: if someone sells you an llms.txt file as the way to "rank in AI answers" on Google, Google's own documentation says otherwise. Your effort for Google belongs in indexing and snippets, which Google names as requirements. Our guide on getting cited in ChatGPT, Claude and Google AI answers goes into that side.

Also note what Google does not say. It does not say the file is bad. It says creating one for other services is "completely fine." The message is "we ignore it," not "remove it."

What the Chrome Lighthouse audit checks

Lighthouse is Chrome's site auditing tool, documented on Chrome for Developers. It has an experimental category called Agentic Browsing, and inside it sits an audit named llms.txt.

The Chrome page describes the file as "an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents." It adds that without it, "agents may spend more time crawling the site to understand its high-level structure and primary content."

Here is what the audit does, based on the Chrome documentation:

• It tries to retrieve llms.txt from your site. The scoring page describes the check as looking for "a machine-readable summary at the domain root."
• If retrieving the file causes a server error, Lighthouse flags the page.
• If the server returns a 404 (the "not found" status), the audit is marked Not Applicable, "as providing the file is optional at the moment."
• The suggested fix is to create the file at the root of your site, for example https://example.com/llms.txt, following the llms.txt specification and giving "a concise Markdown summary of your site's purpose and key links."

Read that 404 rule twice, because it is the most useful sentence in this whole post. A missing file is fine. A broken URL is the problem.

Quick tip: a 5xx status (500, 502, 503 and so on) means the server itself failed while answering. A 404 means the server worked fine and simply has no such file. Lighthouse treats those two very differently, and so should you.

Who should bother and who can skip it

Let me walk through a made-up example. Priya runs a small bakery site with a menu, an about page, and opening hours. Her cousin Sam maintains documentation for an open source library with two hundred pages of API reference. (Both people are invented for this example.)

For Priya, the official sources give no reason to rush. Google Search ignores the file. Lighthouse marks a 404 as Not Applicable. Her job is to make sure /llms.txt returns a clean 404 rather than an error, and then get back to baking.

For Sam, the picture changes. The proposal is written with docs in mind: its background section gives the example of "a coding agent" that "fetches a library's documentation to get an API call right." A curated map of his key pages is cheap to write and fits what the proposal describes.

You can probably skip it if:

• Your site is small and your pages are already easy to navigate.
• Your main goal is Google Search, since Google says Search ignores the file.
• Nobody on your team would keep it updated as pages change.
• You would rather spend the time on indexing and snippets, which Google names as requirements.

It may be worth a short file if:

• You publish developer docs, API references, or long technical guides.
• You care about AI agents finding the right page quickly when helping a user.
• You can generate it automatically from your docs, so it does not go stale.
• You have many pages and want to point agents to a short, curated set first, which is how the proposal describes the file's purpose.

Heads up: no official source I checked promises that an llms.txt file will get you more traffic, citations, or visibility in any AI product. I am not going to promise that either. If someone else does, ask them for a primary source.

Check your site in three minutes

You need a terminal with curl, a command line tool for making web requests. Replace example.com with your own domain in every command.

First, ask the server for just the headers of the file. The curl manual describes -I as "Fetch the headers only," using the HTTP HEAD request.

curl -I https://example.com/llms.txt

Look at the first line of the reply. You want HTTP/2 200 (or HTTP/1.1 200) if you have the file, or a 404 if you do not. Anything starting with a 5 is the problem to fix.

Next, get a one-line summary of the status code, the content type, and how many redirects happened. Here -s is silent mode, -o /dev/null throws away the body, -L follows redirects, and -w prints the values you ask for after the transfer.

curl -s -o /dev/null -L -w '%{response_code} %{content_type} %{num_redirects}\n' https://example.com/llms.txt

The curl manual describes response_code as "The numerical response code that was found in the last transfer," content_type as "The Content-Type of the requested document," and num_redirects as the "Number of redirects that were followed in the request."

When I ran this against https://llmstxt.org/llms.txt on 6 November 2026, the result was:

200 text/plain; charset=utf-8 0

That is healthy: status 200, plain text, no redirects.

If you got a 200, look at the first few lines of the file itself to confirm it starts with an H1.

curl -s https://example.com/llms.txt | head -n 5

The very first line should begin with # followed by your site or project name. If you see <!DOCTYPE html> instead, keep reading.

Optional, for documentation sites: here is a minimal file that follows the structure on llmstxt.org. The product "Widgetly" and its URLs are made up.

# Widgetly

> Widgetly is a made-up JavaScript library for drawing simple charts. These links cover setup and the main API.

## Docs

- [Quick start](https://example.com/docs/quickstart.md): Install and draw your first chart
- [API reference](https://example.com/docs/api.md): Every function and option

## Optional

- [Changelog](https://example.com/docs/changelog.md): Release history

Troubleshooting

The URL returns a 5xx error

Symptom: curl -I shows a status like 500, 502, or 503 on the first line. A 5xx status is a server error, which is the case the Lighthouse audit flags.

Cause: something on the server side failed while handling the request. The request might be hitting an application route, a plugin, or a broken rewrite rule instead of a simple static file or a clean "not found."

Fix: check your server or application error log for the time of your request. Either place a real static llms.txt file at the web root, or make sure the path falls through to a normal 404. Then run curl -I again until you see a 200 or a 404.

Redirect loops or a page that is not your file

Symptom: with -L, curl gives up with exit code 47, which the curl manual describes as "Too many redirects." Or the one-line check shows a 200 with text/html as the content type.

Cause: a redirect rule keeps bouncing the request, or your site sends every unknown path to the homepage. In the second case you are getting a web page back, not a Markdown file.

Fix: for loops, run the check with --max-redirs 3 to stop early, then look at each Location: header with curl -I to find where the loop starts. For a homepage served in place of the file, add the real file or make the path return a true 404. Quick tip: no source I checked names one required content type for llms.txt; llmstxt.org's own file was served as text/plain; charset=utf-8 when I tested it.

A 404 that looks like an error

Symptom: a teammate sees 404 on /llms.txt and opens an urgent ticket.

Cause: the file simply does not exist, which is allowed. The Chrome documentation says a 404 makes the audit Not Applicable, because the file is optional at the moment.

Fix: nothing urgent. Close the ticket, or keep it as a low-priority idea if your site is the kind that might benefit from the file.

The file exists but has the wrong structure

Symptom: the file returns 200, but the first line is not a heading. It might start with plain text, a ## heading, or HTML.

Cause: the file was written freehand or copied from somewhere without following the proposal. The proposal says the H1 with the project or site name is "the only required section."

Fix: make the first line a single # heading with your site or project name. After that, add an optional > summary and ## sections with [name](url) link lists, as described on llmstxt.org.

What to watch next

At the time of writing (6 November 2026), the Chrome scoring page says "The Agentic Browsing category and WebMCP support are experimental and based on proposed standards," and that testing the category "requires Chrome 150 or later."

The same page says this category "does not have a weighted average score from 0 to 100." Instead it shows a fractional score, meaning how many readiness checks your site passes. Experimental tools change, so recheck before treating any result as final.

The key phrase in the audit is "optional at the moment." That wording leaves room for change, but the page does not say what might change or when. I will not guess.

The llmstxt.org proposal is still open for community input, and Google's guide was last updated on 10 July 2026. If either page changes its position, the advice in this post changes with it.

If your real concern is AI crawlers rather than AI summaries, that is a separate decision, and our guide on letting AI crawlers in for search while blocking training walks through it.

The one message worth forwarding

Copy this to your developer or marketing lead:

Quick one on llms.txt. Google's Search docs say Search ignores it, so it will not help or hurt our Google rankings. Chrome Lighthouse has an experimental audit that flags the page when retrieving the URL causes a server error; a 404 is marked Not Applicable. Can you run curl -I https://ourdomain.com/llms.txt and confirm it returns 200 or 404, not a 5xx? If we publish docs, a short Markdown file with an H1 and key links is cheap to add later, but it is not a priority for Search.

Takeaway: skip llms.txt for Google, consider it for docs, and never let the URL return a server error.