Your site is already being crawled by AI agents — maybe not the way you expect
If you run an indie product, a solo SaaS, or a small team site, something changed in 2025 and 2026. Autonomous AI agents — the kind that research, compare, and even transact on behalf of users — began arriving at websites in significant numbers. Adobe Analytics measured a 4,700 percent year-over-year jump in generative-AI traffic to US shopping sites by mid-2025, and HUMAN Security later reported AI agent browser traffic up nearly 8,000 percent year-over-year. These are not human visits. These are machines reading your pages, evaluating whether your product is worth recommending, and sometimes acting on it.
The uncomfortable part: most sites were built for humans, not agents. Agents do not render JavaScript single-page apps the way a browser does. They do not click through cookie banners. They do not fill out CAPTCHAs. They time out quickly and move on. If your product is discoverable only through a long signup flow or buried behind a dashboard, an agent will likely skip it entirely and recommend a competitor that exposes its capabilities more clearly.
This is what people now mean by AI agent discoverability: the idea that a website must present machine-readable signals — protocols, files, headers, and structured data — so that autonomous software can find your product, understand what it does, and decide whether to recommend or use it. It is not the same as SEO. SEO optimizes for human searchers who read snippets and click links. Agent discoverability optimizes for software that consumes structured content, calls endpoints, and makes decisions without you in the loop.
The five signals agents actually look for
Research from multiple agent-readiness frameworks — including AgentReady, Cloudflare, and Postman — converges on five practical signal categories. Think of these as the minimum checklist your site should cover before assuming agents can reach you.
Discovery. Can an agent find your main pages and know which ones matter? The basic signals here are straightforward. A robots.txt file that explicitly names agent user-agents like GPTBot, ClaudeBot, and PerplexityBot tells agents which crawlers you allow. A /sitemap.xml gives agents a quick map of your important URLs instead of forcing them to crawl blindly. Adding /llms.txt — a markdown index at your site root that summarizes what your product is and links to the pages agents should read first — is now considered a SHOULD by most standards. It is one of the easiest high-leverage changes you can make.
Capability exposure. Once found, can an agent actually use your product? For sites that expose APIs or tools, this means publishing an OpenAPI spec or hosting an MCP server at a known path. The Model Context Protocol has become the dominant pattern for letting agents call your functionality. Many frameworks now serve MCP server cards at /.well-known/mcp/server-card.json, which lets an agent discover your tools with a single fetch. If your product does not have an API surface, consider whether your core data or workflow could be exposed as a skill or endpoint.
Content accessibility. Agents read differently than humans. A 12-second page load that a human tolerates is a hard timeout for an agent making decisions across dozens of sites. Support for Markdown negotiation — returning compressed markdown responses when an agent requests them — can reduce payload size dramatically. Prerendering or server-side rendering matters because agents generally cannot execute JavaScript to see your content. If your pages are mostly dynamic JS with no fallback, agents see empty shells.
Trust and authentication. Agents need to know what they are allowed to do. Standards like Web Bot Auth and OAuth metadata in /.well-known/ help agents understand your authentication model. For sites handling payments or sensitive data, protocols like x402 and agentic resource discovery files (/.well-known/ai-catalog.json) are emerging as the way products prove they are safe and legitimate for agents to interact with.
Commerce readiness. If your product involves transactions, agents are now expected to complete purchases, compare pricing, and handle payments autonomously. This requires exposing pricing in machine-readable formats, supporting agent payment protocols, and making sure your checkout flow does not depend on human-only interactions like CAPTCHAs or complex form fills.
What this means for a solo founder with limited time
The full agent-readiness spec includes dozens of requirements with conditional MUST, SHOULD, and MAY qualifiers. You do not need to implement all of them. The practical question is: what moves the needle most for the least effort?
Start with the lowest-friction discovery signals. A properly configured robots.txt and a clean sitemap take minutes. A well-written /llms.txt file — one paragraph describing what your product does, five to ten links to your most important pages, and a note about how agents can reach your API if you have one — can be drafted in an afternoon. These three items alone put you ahead of the vast majority of sites. The top 100 websites currently average around 55 percent agent readiness, and nearly all of them fail basic content negotiation checks, according to 2026 audits.
Next, decide whether your product has an API surface worth exposing. If you already use Postman, Zuplo, or any API gateway, you likely have endpoints that agents could call. The gap is usually not in the code — it is in the documentation and error messages. Postman research found that 89 percent of developers use AI daily, but only 24 percent design APIs with agents in mind. The problem is that agents break when error responses are vague, schemas are undocumented, or endpoint naming follows database logic instead of user intent. A human developer can infer what POST /orders means from context. An agent needs the description in the spec to tell it exactly what parameters to send and what each error code means.
If you are not ready to expose an API, focus on making your content legible. Server-rendered pages, clear HTML structure, and /llms.txt will serve most indie sites better than chasing MCP servers or agent payment protocols right now. Those commerce and capability layers matter more for platforms, marketplaces, and products where agents are already completing transactions on behalf of users.
The trade-offs and what to watch
Adding agent signals is not free. Publishing an MCP server means you are exposing functionality that autonomous software can call — which raises questions about rate limits, access control, and what actions you are comfortable letting agents perform without human confirmation. Many founders start by exposing read-only endpoints and data access before considering write operations or payments.
There is also a timing question. The agentic web is growing fast, but it is still a parallel surface layered on top of the human web, not a replacement for it. Agent traffic is real and growing — Cloudflare documented tens of thousands of agent requests hitting individual sites within days of enabling Markdown for Agents — but it has not yet displaced human traffic for most indie products. The pragmatic position is to ship the discovery layer now because the work is lightweight and irreversible, while holding off on deep API exposure until you understand what your users actually want agents to do on their behalf.
Another thing to consider: agent readiness is measurable, not self-asserted. Multiple independent scanners now audit sites against standards like AgentReady and test whether agents can actually reach your surfaces. Self-assessment badges are everywhere and none of them agree. The useful signal is whether an independent check confirms your site is legible to the agents already browsing it.
When to go further and when to wait
Go deeper if your product is a platform, API, marketplace, or tool that users already ask to integrate with other services. In those cases, agents are the next natural extension of your integrations ecosystem. Expose an MCP server, publish an OpenAPI spec, and make sure your error handling is agent-friendly — clear messages that tell a machine exactly what went wrong and how to recover.
Wait and focus on basics if your product is primarily a marketing site, a content property, or a simple SaaS with a conventional signup-and-login flow. Get the sitemap, robots.txt, and llms.txt right first. Those three items are the foundation. Everything else builds on top of them.
The founders who treat agent discoverability as optional will find their products bypassed in recommendation engines and agent-assisted purchasing flows. The founders who ship the basics this quarter will be legible to the agents that are already browsing, and they will be ready when agent-mediated traffic becomes the norm rather than the exception.
FAQ
Is agent discoverability the same as SEO? No. SEO optimizes for human searchers reading snippets. Agent discoverability provides structured, machine-readable signals that autonomous software consumes directly. They share some tactics — like good sitemaps and clear content — but the intent and execution differ significantly.
Do I need an MCP server to be agent-ready? Not necessarily. MCP is one of several protocols agents use. If your product has no API surface, a well-structured /llms.txt and proper robots.txt may be enough for agents to read and cite your content. MCP matters most when you want agents to call your functionality, not just read about it.
Will adding agent signals hurt my human visitors? No. These signals are invisible to human browsers. A /llms.txt file, a sitemap, and agent-aware robots rules do not change how humans experience your site. They simply add parallel surfaces that software can consume.
How much traffic from agents should I expect? It varies widely by product type and visibility. Some sites reported tens of thousands of agent requests within days of enabling Markdown negotiation. Others saw negligible agent traffic. The growth trajectory is steep, but current volumes depend heavily on whether your product lives in a category agents actively shop or research — like SaaS tools, e-commerce, and developer platforms.
