OptimizationGuy

Answers

Does llms.txt actually do anything?

Short answer: not in the way it's being sold to you. There's enough public data now to say that with a straight face, and I'll show you the numbers rather than ask you to take my word for it.

I put one on this site anyway. I'll explain that at the end, and the reason is not the one you'll hear from someone quoting you a price for it.

What it is

A markdown file at the root of your site — yoursite.com/llms.txt. It lists your most important pages with a one-line description of each, so an AI model handed your address can find the useful parts without crawling everything.

Jeremy Howard proposed it in September 2024. He's a serious person — co-founder of Answer.AI and fast.ai — and the idea is a good one.

What happened next wasn't his doing. My industry attached a story to the file: publish this and you'll show up in ChatGPT. That story was built on speculation that AI platforms would start rewarding the file, not on any platform saying it would. Checking that story is the point of this page.

Two things llms.txt is not. It isn't robots.txt — it controls nothing and blocks nothing. And it isn't the separate practice of publishing plain-text copies of all your pages, which has its own problems and isn't what I'm talking about here.

What Google says, in writing

Google's official guide to optimizing for its generative AI features includes a section on myths — the things you don't need to do. llms.txt is the first item.

The wording is not ambiguous. Google says you don't need special machine-readable files to appear in Google Search or its AI features, because Search doesn't use them, and that creating one will "neither harm nor help" your visibility or rankings, since Search ignores it.

That's not a leak, a rumor, or something someone said at a conference. It's the documentation, published by Google, sitting on their developer site right now.

So if anyone tells you an llms.txt file will help you in Google or in AI Overviews, one of two things is true. They haven't read Google's documentation, or they're counting on the fact that you haven't.

What the server logs say

Google ignoring the file would still leave ChatGPT, Perplexity, Claude, and the rest. So does anything actually read it?

In June 2026, Ahrefs published the largest study on this I've seen — 137,210 domains, checked against real server logs and bot traffic for the month of May.

  • 28% of those domains publish an llms.txt file. Ahrefs notes their own customers skew technical and SEO-aware, so treat that as a ceiling rather than a figure for the web at large.
  • 97% of those files received zero requests. Not from bots. Not from humans. In an entire month, nothing fetched them at all.
  • Of the 3% that got any traffic, the largest single category of visitor was SEO audit tools, at 21.7%.
  • AI retrieval bots — the crawlers that fetch pages to answer live questions inside AI search products — accounted for 1.1%.

Read that last one twice. The bots that determine whether an AI answer mentions your business are barely touching the file being sold as the way to get an AI answer to mention your business.

And one finding closes the door entirely. Ahrefs looked at every request to an llms.txt path that came back 404 — sites without the file — and found that not one of those requests came from an AI bot. AI systems don't go hunting for a file that isn't there. There's no knock at your door that you're failing to answer.

The funny part

The largest audience for llms.txt files, in the best data we have, is the SEO industry checking whether people have llms.txt files.

Add up the audit crawlers, the AI-readiness scoring tools, and the dedicated scanners and directories built solely to catalogue these files, and you get a whole economy formed around measuring adoption of a standard whose readers never showed up.

The detail that says it best: in that same dataset, Slack's link-preview bot fetched llms.txt files more often than Perplexity's crawler did. Slack. Because people in my line of work were pasting links to each other's llms.txt files in chat, and the preview bot dutifully went and got them.

That's not a market. That's a room full of people admiring the wallpaper.

Google arguing with Google

Here's what keeps the confusion alive, and it's worth understanding because you'll run into it.

Google Search says skip the file. Days after that guidance went up, the Chrome team shipped an llms.txt check inside Lighthouse's experimental agentic browsing audits, with documentation explaining that without the file, agents may spend more time crawling a site to work out its structure.

Same company, opposite advice. It isn't really a contradiction, and the resolution is the whole point of this page: these are two different surfaces. Showing up in search results is one thing. Being easy for a browser agent to read while it tries to complete a task on your site is another. Google Search doesn't use the file. A future agent might.

One correction while we're here, because this gets misreported. If your site returns a 404 for llms.txt, that Lighthouse audit is marked Not Applicable — not failed. Providing the file is optional, and the audit only flags when the server throws an actual error trying to serve it. Nobody is docking your score for not having one.

The one audience that's real

Buried in the numbers is a genuine exception. Among AI bots, coding agents were the biggest fetchers of these files — Anthropic's Claude Code out-fetched every AI search bot, every AI assistant, and every training crawler.

That lines up with what John Mueller at Google said when he was asked about the whole thing publicly. Coding tools can read ordinary HTML perfectly well, he pointed out, so a simplified copy is a convenience that saves them some tokens — not a requirement. His verdict for everyone else was blunt: for non-developer sites, he doesn't think it makes much sense, even with more agent traffic coming in the future. He added that every site has more important things to do than prepare for a situation that may or may not arrive.

So if you publish developer documentation and your customers point coding agents at it, llms.txt earns its keep today. If you install gutters, it does not.

The risk nobody brings up

One thing in that study stuck with me. The largest single research crawler in the dataset identified itself as prompt-injection-survey.

Somebody is systematically studying llms.txt as a way to feed instructions to AI agents that are designed to trust it. Think about the shape of that. It's a file whose entire purpose is to be read by a machine that will then act on what it read. Of course that's a security problem waiting to happen.

That's not a reason to skip the file. It's a reason to treat it like code rather than like marketing copy: keep it in version control, know exactly who can change it, keep it to plain links and descriptions with nothing instruction-shaped in it, and actually read anything your website platform generates on your behalf. Mine is built automatically from the site's own page list every time the site is built, so it can't drift out of date and there's nowhere for a stray instruction to hide.

So why did I add one?

Not for rankings. Not for AI visibility. Both would be claims I can't support with evidence, and not making those claims is roughly the entire premise of this business.

Three honest reasons:

It costs nothing. Mine generates itself from the same page list that produces the sitemap. No ongoing work, nothing to keep in sync by hand, no maintenance to forget about in eight months.

If the agentic web shows up, I'm already there. Google keeps saying the future of search is agentic. If agents end up as the layer between people and businesses, this file is already in place and already accurate. That's a lottery ticket. I'd rather hold a free one than not.

I'd rather be the guy who tested it than the guy with an opinion about it.

What I did not add is the companion file, llms-full.txt, which holds a copy of a site's entire text in one download. For a documentation site that's genuinely useful. For a small business site it's a clean, tidy copy of everything you've written, gift-wrapped for anyone who wants it. The math doesn't work.

The sentence to keep: If someone quotes you a price for adding an llms.txt file to your website, you're being charged for ten minutes of work and a rumor.

That's where it stands in August 2026. If it changes — if OpenAI or Anthropic or Perplexity says their retrieval systems read the file — I'll come back and update this page. Until then I'd rather tell you it does nothing and be right, than tell you it does something and sell you the fix.

Sources

Start with a free AI Visibility Check.

Where you show up in ChatGPT — and where you don't…

Free AI Visibility Check