Writing

Agents don't read your homepage. They probe it.

What agents and the scanners that grade them actually do when they meet a product. The one HTML response that failed three checks, the clients that discover auth two different ways, the surface rule we hold every tool to, and the half of an agent-readiness audit we chose to ignore.

We cannot tell you what an agent prefers. We can tell you what agents and the scanners that grade them did when they met our sites, because every failure left a status code behind. The pattern across all of it is short: an agent judges you on the first machine request it sends, it sends that request to places you did not design for, and it believes whatever comes back.

We took a set of content sites from a readiness score of 68 to 100, a product site from 76, and this blog to 98. Here is what moved the numbers, and what we refused to move.

68 → 100
Content sites, readiness score
one external scanner; the fixes were shared code across every site
76
Product site, first scan
the same scanner; half its misses were deliberate
3
Checks failed by one response
an HTML 405 where a scanner expected JSON-RPC
~1%
Homepage text, share of HTML
most of the page was inlined framework payload

Answer every machine request in a machine shape

The single most valuable finding on the product site was one page. The path a person would guess for our MCP server was a human guide to connecting, a marketing page. The real server lived on its own subdomain and answered correctly. But a scanner only probes the main domain, and a POST to the guide answered 405 with an HTML body. That one response failed three checks: server found but connection failed, resources not exposed, response not valid JSON.

The fix was not to move the guide. A GET still renders the page for people. A POST now answers with valid JSON-RPC that names the real endpoint. Same URL, one answer per kind of caller.

The content sites had the same disease in a quieter form:

Give them the text, not the page

The homepage of the product site was about one percent visible text by weight. Most of the HTML was inlined framework payload, the data the client needs to hydrate. An agent fetching the page gets that payload. It is fixable only by shrinking the payload, not by writing more copy.

So every page that matters has a Markdown twin: the same path with .md, or the same path with Accept: text/markdown. An llms.txt lists them, the sitemap and robots file are derived from the same list, and each HTML page advertises its twin. This blog does the same.

The rule that matters most on a twin is what it leaves out. The pricing twin reads prices from the same plan records the page uses, and a plan with no record behind it is omitted, never printed at zero. A zero price in a machine-readable file is the failure that would mislead a buyer's agent, so a test pins it.

Two clients, two ways to find your auth

Our MCP server is a standard OAuth resource server. A request without a token gets 401 with a WWW-Authenticate header pointing at the protected-resource metadata, which names the authorization server, which offers dynamic client registration. Nothing branches on which client is calling. And still only some clients could connect.

Client behaviourWhat it asks forResult before the fix
Challenge firstsends a request, reads the metadata URL off the 401worked
Probe firstderives the metadata URL from the spec itself404, no authorization server

The spec forms the metadata URL by inserting the well-known segment between the host and the resource's path. We served only the bare well-known path, which the spec treats as a fallback. Challenge-first clients never noticed. Probe-first clients asked for the path-suffixed URL and got nothing. The fix serves the metadata at the real resource path, from one shared document so the two URLs cannot drift, and 404s every other suffix, because answering broadly would advertise our authorization server as guarding someone else's resource.

The lesson generalises: test with a client that reads the spec, not only with the clients you use.

The agent does what a user can do, and nothing more

The surface rule we hold every tool to: an external agent may do exactly what a signed-in person can do in the interface, and no more. It sounds obvious. Auditing against it removed real things:

Two patterns came out of it. Enforcement lives in the mutation's own schema, not in the tool registry, because a second transport once forwarded arguments without checking the registry. And when an agent needs a narrower view of a read the interface also uses, add a sibling built for the agent and point the tool at it, rather than changing the shared function.

The same rule applies to behaviour, not just permissions. The agent surface runs the same pre-send safety and coaching check as the chat in the browser, one mechanism on both. We tried letting the model discover tools on demand instead. In the browser chat, where the pre-send check loads the tool, 30 of 30 news questions reached it. Over MCP with on-demand discovery alone, 0 of 6 did: the model never searched for the tool and used a general web search instead. Every tool you mount costs something, but a tool the model never finds costs more.

No keys on disk

The command-line client we are building has no API keys. It is a first-party OAuth client: loopback with PKCE on a machine with a browser, a device grant with a code to type on one without. Tokens are short-lived and bound to one audience.

The reason is specific to agents. A static key on disk, or in an agent's environment, works for whoever copies it, and a key in an agent's environment is exactly what a prompt-injected agent can be talked into printing. A short-lived token bound to one resource limits what a leaked one can do.

The half of the audit we ignored

Half of the product site's misses scored a public REST API we deliberately do not publish. Agents reach the product through MCP; the old REST action route answers 410 Gone and points at the MCP server, and a test asserts the published spec contains no REST paths. Inventing a CRUD API to satisfy a scanner would reverse a product decision to fix a number.

We also declined to add rate-limit headers to the public machine-readable files. They are cached at the edge and unlimited, so announcing a quota would be a machine-readable lie. Where a limit is real, on the read-only API and the MCP endpoint of a news site, a firewall rule enforces it and the headers say so.

A readiness scanner is a good linter. Treat its report the way you treat a linter's: fix what is wrong, and write down why you are ignoring the rest.

A checklist from the status codes

If you have run a readiness scanner on your own product, I would like to know which of its checks you chose to ignore, and why.