Blue Crane Blue Crane We Lift. You Build. Log in Try the demo

HomeArticles › Getting found by AI

Getting found by AI

llms.txt and Markdown for agents, in plain language

Published
AuthorTrevor, Founder
Reading time7 min
Written inHamilton, Ontario

llms.txt is a plain-text index at the root of a website that tells an AI assistant what the site is and where its important pages are; Markdown for agents is serving each page as clean Markdown, either when a tool asks for it or at a .md address, so an assistant reads the content in a tenth of the bytes. Both make a site cheaper and more accurate for a machine to read once it has found you. Neither makes it rank: Google says in writing that no AI text file or markup is needed to appear in its AI features, and nobody who sells these as a ranking trick is telling the truth.

What is llms.txt?

A proposal, not a standard, published at llmstxt.org: a Markdown file at /llms.txt that opens with the site's name, a one-paragraph summary, and a list of links to the pages an assistant would need, each with a line saying what it is. The idea is that a web page is a poor thing to hand a language model, full of navigation, scripts and repetition, and a short curated index is a better starting point. Blue Crane's own is at bluecraneworks.ca/llms.txt and says, in a few hundred words, what the company does, what it charges and where each product lives.

Its job is accuracy. When an assistant is asked about a business and reads that file, it gets the facts the business chose, in the words the business chose, instead of whatever it could scrape from a footer. A stale llms.txt is therefore worse than none: it looks authoritative and is wrong. Ours is regenerated with the site so it cannot drift.

What is Markdown for agents?

Two separate mechanisms that people talk about as one. The first is ordinary web plumbing: a browser asking for a page sends a header saying what formats it accepts, and an AI tool can send Accept: text/markdown. A server that notices can return the same page as Markdown instead of HTML. That is content negotiation as defined in RFC 9110, the HTTP standard, and the Markdown media type is registered in RFC 7763. The second mechanism needs no header at all: the same page is also reachable at a .md address, so /pricing has a twin at /pricing.md, which is the convention the llms.txt proposal suggests.

Why both? Because the tools split. The coding assistants, Claude Code, Copilot, Cursor, send the header; the consumer assistants people actually ask about local businesses, ChatGPT and Perplexity among them, do not, according to the field tracking at acceptmarkdown.com. For that larger group the only clean thing you can give them is a .md address that a person, a sitemap or an llms.txt hands over. Cloudflare sells the first half as a paid feature; its documentation describes exactly that, negotiation on the Accept header, and does not create the .md addresses. Blue Crane builds both halves on every managed site, for free, and measured the result on this domain: the homepage is 213 kilobytes as HTML and about 11 as Markdown, 95 percent smaller for a tool that asks.

Do these files help a site rank or get cited?

No, and it is important to say so plainly because they are sold as if they did. Google's guide to AI features says that to appear in AI Overviews and AI Mode "you don't need to create new machine readable files, AI text files, or markup", and that there are no additional requirements beyond being indexed and snippet-eligible. Bing's guidelines carry the general limit: GEO "does not guarantee grounding or citations." What the files change is what happens after an assistant has already decided to read you: fewer tokens, less noise, and your facts in your words. That is worth having. It is not a ranking.

What can go wrong?

Three things, each of which we have seen live. A Markdown copy that says something different from the page, an old price or a dropped paragraph, is the version the assistant will quote back to a customer, and serving different content to different readers is also the definition of cloaking, which Google penalises. A server that negotiates on the header without telling caches it did so can hand one visitor's Markdown to the next visitor's browser; the fix is one response header and it is not optional. And an llms.txt written once and never updated turns into a confident lie about the business. The rule for all three is the same: generate the files from the same source as the pages, on every deploy, so they cannot disagree.

Where do these sit in the order of work?

Last. First the crawlers have to be allowed in and actually get in, the words have to be in the HTML, and nothing on the page may forbid citation; the five-minute check covers that floor and it decides everything. Then the site has to be worth quoting and has to agree with the business listing. Then, and only then, llms.txt and Markdown make a readable site cheap to read. On a Blue Crane managed site all of it is built in from day one; on a site you already have, the one-time AI Visibility Pass adds the same files and proves them.

Want this checked and fixed on your site?

The AI Visibility Pass runs every check in this article against your live site, fixes what it finds, and hands you the proof. Fixed quote, written guarantee. On a managed website it is included from launch.

See the AI Visibility Pass Managed websites, $100 a month