How to build dual-layer machine discoverability using llms-full.txt and Model Context Protocol (MCP) server endpoints to serve autonomous enterprise agents.
The web is experiencing a tectonic transition from human-browsed hypermedia to machine-orchestrated agentic ecosystems. Autonomous AI agents—ranging from developer assistants like Claude Code and Cursor to enterprise workflow automations—require instant, deterministic access to site architectures, API schemas, product documentation, and domain knowledge graphs without the overhead of client-side JavaScript rendering.
While the standard /llms.txt file functions as a high-level table of contents for LLMs, forward-thinking organizations are now deploying comprehensive `/llms-full.txt` files. An llms-full.txt specification aggregates complete, curated Markdown documentation into a single structured stream, enriched with precise YAML frontmatter metadata, OpenAPI contract summaries, and typed entity schemas.
The next frontier beyond static markdown files is the integration of Model Context Protocol (MCP) tool endpoints. MCP, an open protocol developed for AI-to-software interoperability, allows websites to expose secure, rate-limited JSON-RPC tool endpoints directly to AI agents. Rather than scraping dynamic UI states, an AI agent interacting with an MCP-enabled domain can query live catalog availability, execute real-time site search, or fetch structured technical specs through native function calling.
To implement this dual architecture effectively, web architects should configure reverse proxy rules (e.g., in Nginx, Cloudflare Workers, or Next.js middleware) that detect AI crawler User-Agents (such as ClaudeBot, GPTBot, and PerplexityBot) and serve clean, pre-compiled Markdown streams with zero redirect latency, accompanied by Cache-Control headers optimized for agentic ingestion.
Scanasite automatically analyzes and validates your domain’s /llms.txt and /llms-full.txt files against official schema guidelines. Our automated crawler validates token efficiency, verifies Markdown heading hierarchies, and tests machine-readable endpoint response times to ensure enterprise-grade agent readiness.