# Agent-Ready Web Stack The set of formats and protocols that let an AI agent reach, read and use a website. ## Read layer - **Markdown for agents (Cloudflare, Feb 2026).** An agent sends `Accept: text/markdown`. Cloudflare converts the HTML and serves Markdown. Beta on paid plans.^[https://thenewstack.io/cloudflares-markdown-for-agents-automatically-make-websites-more-aifriendly/] - Cloudflare claims about 80% fewer tokens. One test cut a page from 16,180 to 3,150 tokens. - Claude Code and OpenCode already send the header. - **llms.txt.** A Markdown index of key pages at the site root. Cheap to add. No proof that major models use it. - **Token budgets.** Osmani suggests quick starts under ~15k tokens and API references under ~25k. Put the answer first.^[https://addyosmani.com/blog/agentic-engine-optimization/] - **AGENTS.md / skill.md.** Files that tell coding agents how to use a product. ## Act layer - **WebMCP (Google + Microsoft, W3C).** A site exposes structured tools to an in-browser agent through `navigator.modelContext`. - Chrome origin trial, Chrome 149–162, ends 30 March 2027. Edge trial runs to the same date. - Firefox has an off-by-default prototype. Safari opposes it.^[https://venturebeat.com/infrastructure/google-chrome-ships-webmcp-in-early-preview-turning-every-website-into-a] - **NLWeb (Microsoft, 2025).** Open project. Each site instance acts as an MCP server with a natural-language interface. Low visible adoption in 2026. - **Remote MCP servers.** A product exposes its API to agents directly. Peec, Conductor and Profound all ship MCP or ChatGPT apps. See [[Connector Layer and MCP]]. ## Access and trust layer - **robots.txt** controls named AI crawlers. About 15% of AI fetches ignored a disallow in H1 2026 (TollBit).^[https://ppc.land/15-of-ai-page-fetchers-in-europe-reached-disallowed-urls-tollbit-finds/] - **Web Bot Auth (Cloudflare, IETF draft).** An agent signs each request (RFC 9421). The site verifies the agent, not a spoofable user-agent string. Visa TAP and Mastercard Agent Pay build on it. - **Pay per crawl (Cloudflare).** Unpaid crawlers get HTTP 402 with a price. Still closed beta. Cloudflare plans a move to pay-per-use.^[https://pressgazette.co.uk/platforms/cloudflare-says-bot-blocking-is-fueling-publisher-ai-deals/] > [!question] Tension > Blocking or charging crawlers protects publishers. It can also remove a brand from answers. Brands and publishers want opposite settings. ---- tags: #boat #ai links: [[Agent Engine Optimisation MOC]] [[Agentic Commerce Protocols]] [[CLI vs MCP]]