What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers what they may not access; it's a set of rules. llms.txt does the opposite: it points language models to your best content and presents it cleanly. One restricts; the other recommends.

Is llms.txt the same as a sitemap?

No. A sitemap lists every URL for search-engine crawlers in XML. llms.txt is a short, curated, human-readable markdown file highlighting only your most important pages, written for language models to ingest cleanly. They complement each other.

Do I actually need an llms.txt file?

It's an emerging standard, not yet universally read by AI engines, so it's optional today. But it costs little to add, can't hurt, and almost no sites in most niches have one, so it's a low-risk first-mover move if AI traffic matters to you.

← All articles

What is `llms.txt`, and do you actually need one?

By Bryan Shelley, Founder · Search & AI

Published June 11, 2026 · Updated June 11, 2026

llms.txt is a short markdown file at the root of your site that points AI models to your most important pages and presents them cleanly: a sitemap written for language models instead of search crawlers. It’s new, it’s simple, and almost nobody has one yet. Here’s whether that should be you.

What it actually is

It lives at yourdomain.com/llms.txt and it’s plain markdown. A title, a sentence about what the site is, then a curated list of links to your key pages with a short note on each. No navigation, no ads, no cookie banner, just the signal. The idea: when a language model wants to understand your site, a normal web page buries the useful part under layout and clutter. A clean markdown map hands it the substance directly.

How it’s different from the files you already know

vs robots.txt: robots.txt is a list of don’ts: what crawlers may not touch. llms.txt is the opposite: a list of here’s the good stuff. One restricts, one recommends.
vs sitemap.xml: a sitemap is every URL, in XML, for search engines. llms.txt is a short, curated, human-readable shortlist of your best pages, for AI. The sitemap is the phone book; llms.txt is the “start here” card.

They don’t compete. Keep your sitemap and robots.txt; llms.txt sits alongside them.

What goes in it

The convention is markdown, roughly like this:

An # H1 with your site or product name.
A one-line > blockquote summary of what you do.
## Sections grouping your key links, each as [Page title](url): one-line description.

Some sites also publish llms-full.txt: the same idea but with the full text of those pages inlined, for models that want everything in one fetch. Start with the short version.

Do you actually need one?

Honest answer: not yet, strictly. It’s an emerging standard and not every AI engine reads it today. So this isn’t a must-fix like a broken noindex. But the math is friendly: it takes minutes, it can’t hurt anything, and in most niches none of your competitors have one. If AI traffic matters to your category, it’s a cheap first-mover move. If it doesn’t, skip it and fix your crawlability and content first; those pay off no matter what.

How to make one in a few minutes

List your 5–15 most important pages: the ones you’d want an AI to read to describe you accurately.
Write the markdown: title, one-line summary, then grouped links with a short note each.
Save it as llms.txt and serve it at your domain root, so yourdomain.com/llms.txt loads as plain text.
Keep it current when your key pages change; a stale map is worse than none.

Quick answers

llms.txt vs robots.txt? robots.txt restricts crawlers; llms.txt recommends your best pages to AI.

Is it the same as a sitemap? No. A sitemap lists every URL in XML; llms.txt is a short curated markdown shortlist for AI.

Do I need one? Optional today, but cheap and low-risk: a first-mover move if AI traffic matters to you.

Check it with GreenLight

GreenLight’s AI Search (GEO) check looks for an llms.txt (along with AI-bot access, structured data, and question-style headings) and tells you in plain language what’s there, what’s missing, and the exact thing to add.

Run a free check

What is llms.txt, and do you actually need one?