AI Agents & Crawlers

Serve markdown to AI agents by content negotiation, declare crawler rules and content signals in robots.txt, and publish an API catalog.

Two audiences read your site: people, and the agents that answer questions about it. The agent surface is driven by config.agents and composes with the config.indexable master switch described in Marketing & SEO.

What is served

URL	Content type	Who reads it
Any marketing URL with `Accept: text/markdown`	`text/markdown`	Claude Code, Cursor, OpenCode
The same URL with `.md` appended	`text/markdown`	Codex, Gemini CLI, Copilot, Windsurf, and humans pasting links
`/llms.txt`	`text/plain`	An index of the above
`/robots.txt`	`text/plain`	Crawlers
`/.well-known/api-catalog`	`application/linkset+json`	API discovery (RFC 9727)

Markdown negotiation is the only item here with measured consumption. A page served as markdown costs an agent roughly 1% of the tokens the rendered HTML does, and Claude Code skips its summariser entirely on a markdown response under 100k characters.

The two markdown shapes

Both serve the same rendition and both ship on by default.

Accept negotiation keeps one canonical URL. proxy.ts rewrites the request to app/md/[locale]/[[...slug]], which answers with Vary: Accept so a shared cache never hands markdown to a browser.
The .md suffix is a second URL per page (/contact.md, and /index.md for the home page). Four of the seven common agents ignore Accept entirely, and a person can paste a .md URL into any chat.

Renditions are built from the same config and translations the React page renders - never scraped from rendered HTML. Content documents and legal pages serve their own markdown source; /, /contact and /download are generated from pricingConfig, the landing.* translation keys, and the landing data modules in lib/landing/. The home rendition carries the plans and one-time products, because the home page does. Every configured section's listing joins the static routes in llms.txt.

Routes are derived from feature flags, so a build with an empty content section or the desktop app switched off never advertises a page it does not serve.

Config

packages/config/src/index.ts

agents: {
  markdown: true,      // markdown renditions + /llms.txt
  apiCatalog: true,    // /.well-known/api-catalog
  crawlers: {
    namedGroups: true, // write named User-agent groups at all
    search: true,      // OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot…
    userFetch: true,   // ChatGPT-User, Claude-User, Perplexity-User…
    training: true,    // GPTBot, ClaudeBot, Google-Extended, CCBot…
    contentSignals: { search: "yes", aiInput: "yes", aiTrain: "unset" }
  }
}

Set a crawler group to false to write Disallow: / for those bots. Content Signals take "yes", "no", or "unset" (the token is omitted, which states no preference).

Defaults allow all three purposes. Blocking search and live-fetch bots costs the citations that send readers back, and a boilerplate should not reserve training rights on every project's behalf. aiTrain is deliberately unstated - set it to "no" to assert an EU DSM Article 4 text-and-data-mining reservation.

What crawlers actually do

Be clear-eyed about which of these are enforcement and which are requests.

Item	Reality
Markdown negotiation	Observed. Agents send the header and act on the response.
`robots.txt` crawler groups	A declaration. OpenAI, Perplexity and Meta all document user-initiated fetches that may ignore it; Anthropic's `Claude-User` claims compliance.
Content Signals	A declaration with no known crawler consumer. Its value is the legal reservation it records, not enforcement.
API catalog	A published standard with almost no deployment. Cheap, correct, and points at a real generated spec.

Googlebot and bingbot are deliberately left in the wildcard group: both feed search and AI answers, so no robots.txt rule can separate them. Naming a bot also takes it out of User-agent: * entirely (RFC 9309), which is why every global Disallow is repeated inside each named group.

Deliberately not shipped

Item	Why
A2A agent card	No A2A endpoint exists; a card would advertise a capability you do not have.
MCP server card	The current proposal moved away from the path scanners probe. Your MCP server's OAuth discovery (RFC 8414 / 9728) already ships and is required by the MCP spec.
Web Bot Auth	Nothing for a site to implement - it is verification at your CDN or WAF, not application code.
DNS-AID	DNS zone configuration, not application code, and an unadopted draft.
Agentic commerce (ACP, AP2, UCP, x402)	Retail cart protocols with no subscription primitive.

llms.txt is included because it is nearly free, not because it works: studies of it find almost no bot fetches, and no crawler even probes for the file. Serve the negotiation; treat the index as a courtesy.

What is served

The two markdown shapes

Config

What crawlers actually do

Deliberately not shipped

On this page