Insights

llms.txt and AI crawlers: what they are and what they are not

robots.txt for AI crawlers and the llms.txt proposal: what they are, what they can do and what they cannot do for a brand's presence in AI.

robots.txt is the file a website uses to tell crawlers, including those run by AI companies, which parts of the site they may access. llms.txt is a proposed convention: a Markdown file at the site root that summarizes the site for language models. Neither guarantees that a brand will appear in AI answers: robots.txt is a request that well-behaved crawlers follow, and llms.txt is not an official standard, with support that varies from one provider to the next.

Key takeaways

  • robots.txt says which crawlers are invited in. Blocking AI crawlers by mistake can keep your content from being used in answers.
  • Some crawlers collect data to train models; others fetch pages to answer a user’s question. Decide on each kind separately.
  • llms.txt is a proposal: low-cost, optional, and no guarantee of anything.
  • Crawler behavior changes over time, so check each provider’s official documentation.

What does robots.txt have to do with AI?

robots.txt sits at the root of a website and holds rules by crawler name (user agent): which crawlers may enter, and which sections they should stay out of. AI companies publish the names they use so that site owners can set specific rules for them. Some you are likely to come across:

  • GPTBot
  • OAI-SearchBot
  • ChatGPT-User
  • ClaudeBot
  • PerplexityBot
  • Google-Extended

The syntax is the same as for any other robots.txt rule:

User-agent: <crawler name>
Disallow: /a-folder-you-want-crawlers-to-skip/

Each provider describes what each name is for in its own documentation. The list and the behavior can change, so do not rely on any article, this one included, in place of the original documentation.

How do training crawlers differ from answer crawlers?

Broadly, there are two main purposes:

  • Collecting data to train models. The content becomes learning material for later versions of a model.
  • Fetching pages to answer a question. When a user asks something, the system finds and reads relevant pages to compose its answer, often with sources attached.

The distinction matters when you make decisions. A blanket rule against all AI crawlers can unintentionally remove a website from retrieval-based answers, when all the business really wanted was to limit the use of its content for training. Which kinds to allow or block is each organization’s own business decision; the thing to avoid is blocking without knowing what you are blocking.

What can robots.txt do, and what can’t it do?

  • It is a request, not an enforcement mechanism. Well-behaved crawlers follow it; the file does not technically prevent access. Content that needs protecting should sit behind a login or access controls.
  • Allowed does not mean used. Opening the door to crawlers is only a precondition. Whether content gets picked still depends on its quality and relevance. See how AI chooses its sources.
  • It is not the only barrier. A firewall, a bot-protection service or server configuration can also block AI crawlers even when robots.txt allows them.

What is llms.txt?

llms.txt is a proposed convention: place a Markdown file at the root of a website that summarizes the site for language models. It typically contains the site’s name, a short introduction and a list of links to the most important pages, each with a brief description. The idea is to hand a model a compact, readable map instead of leaving it to work through the full site structure on its own.

What is llms.txt not?

  • Not an official standard. For now it is a proposal.
  • Not supported by every AI provider. Adoption varies, and you should not assume that any particular AI assistant will read the file.
  • Not a guarantee. Having an llms.txt does not mean being cited or mentioned more often.
  • Not a substitute for robots.txt. llms.txt does not control crawler access.
  • Not a substitute for good content. A tidy summary cannot make up for pages that lack specific information.

Should you create an llms.txt?

You can, as long as you treat it as a small, optional task. The cost is low: one text file, updated when the site structure changes. If you do it, keep the file short and accurate, link only to canonical pages, and make sure it stays consistent with what is really on the website. Do not expect this file alone to produce a measurable change, and do not put it ahead of more fundamental work.

What should you check first?

  1. Open robots.txt and look for rules that block AI crawlers unintentionally.
  2. Decide a policy for each kind of crawler based on the providers’ documentation.
  3. Check your firewall and bot-protection settings.
  4. Make sure the main content is readable as text without a login.
  5. Consider llms.txt once the steps above are done.
  6. Review periodically.

Accessibility is only the first layer. For AI to understand a brand correctly and have a reason to cite it, the brand also needs an entity foundation, citable content and presence across many sources. You can read about the G.E.O framework or GEONGON’s Entity & Schema service.

Related articles

Where does your brand stand in AI answers today?

Start with a conversation and a first look at how visible your brand is across AI platforms.

Get an AI Visibility audit