In the last eighteen months, a small pile of new well-known files has appeared at the root of ambitious websites: llms.txt, auth.md, agents.json, an OpenAPI spec, an MCP manifest. Some are emerging conventions with real adoption. Some are one platform's proposal that hasn't gone anywhere yet. Here's the map we built while actually implementing this stack for tailyx.ai — not a roundup of what's theoretically possible.
Engineering TailyX — Part 2 of 4
Previous: How We Built an MCP Server in Ruby on Rails · Next: Why We Built an AI Qualification Engine Instead of Another Chatbot
Treat AI discoverability as a stack, not a single file. Each layer answers a different question. There's no universally established sequence across every agent platform — different clients discover capabilities differently — but this is the mental model we use for reasoning about the layers and the order we built them in:
robots.txt → "Am I allowed to fetch this at all?"
llms.txt → "What is this site, in agent-readable terms?"
auth.md → "How do I authenticate if I want to call something?"
openapi.json → "What REST endpoints exist and what do they expect?"
MCP manifest → "What tools exist, and where's the server?"
submission → "Register this surface with a platform (ChatGPT Apps, etc.)"An agent that stops at layer one learns what it can't touch. An agent that reaches the MCP manifest can transact with you. Most sites today stop at layer one. That gap is the opportunity — and, from a defensive angle, the reason we treated this stack as worth the engineering time rather than a marketing checkbox.
Decades old, universally respected by well-behaved crawlers, and most sites have never updated it for the newer category of agent crawler distinct from search crawlers. This is where you explicitly allow or disallow specific agent user-agents. It's the one file on this list every site already has and almost none have actually reviewed for this purpose — which is exactly where we started.
The closest thing to a genuine emerging standard here: a plain markdown file at the site root giving a structured, agent-readable summary of what the site is and where the important content lives. Not ratified by any standards body. Actually read by a meaningful volume of AI products in production today, which puts it ahead of several "official-sounding" proposals that aren't.
The test we use for "does this file matter": is anything actually reading it in production, or is it a spec someone published and hoped would catch on? llms.txt passes. Several others on this list don't yet.
There's no accepted standard yet for "here's how an agent authenticates against my API," which is a strange gap given the energy poured into discovery. We published our own auth.md at tailyx.ai with verified endpoints and the auth flow an agent needs. Not a standard — a stopgap that solves the actual problem until one exists, written in the same plain, structured, agent-parseable spirit as llms.txt rather than a PDF or a developer-portal login wall.
OpenAPI predates the current wave by a decade and isn't an "AI discovery file" in the sense the others are — but it's become load-bearing for agent tooling anyway, because a well-formed spec is often the fastest way for a platform to auto-generate tool definitions from an existing REST surface. If you already maintain one for human developer docs, you're most of the way to agent-readiness on the REST side for free.
This is where it's genuinely unsettled. Several platforms have proposed their own manifest formats — variants on "agents.json," MCP server cards, and similar — describing capabilities in a platform-specific shape. We track these but don't treat any single one as authoritative, because the honest answer in mid-2026 is that this layer hasn't consolidated. Our approach: implement the ones with real platform backing (an MCP manifest tied to an actual deployed server, in our case), and hold off on speculative formats until adoption signals which one wins.
These files aren't independent — they should point at each other. Our llms.txt references where the OpenAPI spec and MCP manifest live. auth.md is referenced from both, so an agent that discovers the API surface can immediately find how to authenticate, rather than discovering three unconnected islands. Treat this stack the way you'd treat internal linking on a website: the value compounds when the pieces reference each other.
We used an external agent-readiness score as a check on our own progress through this stack, rather than trusting our own judgment about what "done" looks like — useful as a forcing function if you want a number to track instead of a checklist to eyeball.
llms.txt and auth.md — doing discovery first would have let us test the MCP manifest against a discovery layer that already worked, instead of debugging both at once.Current as of July 2026. This is one of the fastest-moving parts of the stack described in this series — expect some of these proposals to consolidate or be superseded within months.