138 database queries to show a home page
Closing the loop on latency in a chat product. The harness, the eight causes, what each fix measured, and why counts beat milliseconds when the agent is doing the work.
The chat product I work on got slow in September. Not broken, just slower every week, in the way that nobody notices on the day it happens. When I finally measured it, the home page was making 138 database queries and firing 13 server actions, one after another, to render a screen that is mostly a text box.
This is the story of closing that loop: measure, fix, measure again, with coding agents doing most of the fixing. The numbers are from a local production build on a throttled phone profile unless they say otherwise, and I'll explain why that matters.
First, a harness
The first thing we built was not a fix. It was a script that loads eight routes in a real browser, desktop and a mid-range phone profile with 4x CPU throttling and a slow network, cold and warm, three runs each, and writes a table.
It records the usual web vitals, but the columns that turned out to matter were the ones a browser does not report: how many server actions fired during the load, how many Postgres statements the load caused, how many kilobytes of JavaScript arrived, and how many requests. Those are counts. Counts do not care that the database is on localhost.
That last point is the whole method. A local rig is flatter than production in three ways: the database answers in a tenth of a millisecond, the desktop profile has no network latency, and there is no analytics or support widget. So a local millisecond figure means little. But a query count, an action count and a byte count are the same locally as in production, and in production every one of them is paid for with a round trip. We optimised the counts and let production collect the milliseconds.
Eight causes
With the harness in place, the agents went looking. Each cause below became one commit, measured before and after with the same script.
| Cause | What it cost | Fix and measured delta |
|---|---|---|
| Every HTML request rendered the whole tree twice. A framework prefetch option started a second render per document and inlined its payload. | Twice the queries and server CPU; a second 275 KB blob in every HTML response | Option off app-wide, kept on one marketing route. HTML 580–616 → 310–334 KB; session reads per render 2 → 1 |
| Server actions after hydration run one at a time per client. Home fired 12 or 13; a thread view 15 or 16. | About 70 ms per action on the phone profile, serialised | First reads served from the server render into the client cache. Actions 13 → 9, then seeded entirely in a later pass |
| Every authenticated request read the session, the user and the signing keys, and signed a token for a header nothing read. | 57 of a thread view's 183 queries | Header off; session read once per request with a short encrypted cookie cache. Key reads 20 → 0; auth queries per cold home load 20 → 6 |
| The sidebar logo linked home with default prefetch, so every other page pulled home's chat bundle. | About 1 MB gzipped on every non-home page | Prefetch on hover or focus only. Non-home JS 1,837 → 807 KB; requests 106 → 70 |
| A help menu imported navigation links from a module that also carried the product catalogue as JSON. | 808 KB of JSON in the root layout's client chunks | Links split into their own module; the catalogue is server-only. −143 KB gzipped on every route |
| The chat bundle loaded every renderer eagerly: code highlighting, diagrams, maths, and all 43 card types. | Hundreds of KB a reader may never use | Highlighting and diagrams on first use (−190 KB), maths only for messages that contain it (−80 KB), cards via dynamic import (−220 KB) |
| Four TrueType fonts preloaded on every page; one was 563 KB. | Bytes and a late paint | Lossless WOFF2. −91 KB per cold load; phone LCP −208 ms on home |
| One feature flag was read up to three times per render, each a remote call. | About 70 ms per read, p50 | Read once per request. Calls 3 → 1 |
Nothing in that table is exotic. Every row is a known performance smell. The point is that none of them had been visible, because nothing had counted them.
What the phone saw
Mobile, cold load, medians of three. The home page and the main thread view are the heavy ones because they carry the chat.
| Route | JS KB gz | LCP ms | TBT ms | Settled ms | Actions | DB queries |
|---|---|---|---|---|---|---|
| Home | 1971 → 1340 | 3900 → 2736 | 663 → 414 | 4285 → 2968 | 13 → 9 | 138 → 96 |
| Thread view | 1975 → 1343 | 3916 → 2764 | 676 → 473 | 4490 → 3246 | 16 → 12 | 199 → 154 |
| Inbox | 950 → 808 | 2144 → 1856 | 243 → 242 | 2454 → 1966 | 7 → 4 | 93 → 74 |
| Canvas view | 2011 → 1360 | 2264 → 1704 | 539 → 113 | 2648 → 1885 | 6 → 4 | 126 → 106 |
Warm loads changed more than the table shows: the transfer became the HTML only, and hydration on the phone dropped from about 1,090 ms to about 315 ms on the two chat routes.
Then the live numbers
The local harness decides what to fix. It does not tell you what a person feels. For that we wrote a second, much smaller script that hits any origin, including the deployed test environment, and logs first byte, time to the composer, time to content, and every slow request. Every performance claim in a pull request since has had its output pasted in.
Two live measurements from the same weeks, on the deployed test environment:
- Inbox rows appeared at 1.43 to 1.97 seconds, with 4 serialised server actions before the first row. After moving those reads into the server render: 0.96 to 1.10 seconds and 0 actions before rows.
- First byte on the restored good build was 0.12 to 0.19 seconds; usable at 1.4 to 2.0 seconds. Those became the reference numbers every later change had to meet.
And one of them became a test. "The inbox tab makes no server calls on load" is asserted in the browser test suite, so the regression that started all this cannot come back quietly.
The regression that started it
I should say how we got to 138 queries, because it is the part most relevant to anyone working with agents.
A week earlier, an agent had moved the first-paint data reads from the server to the client: 140 reads across 53 files, each a server action fired after hydration. Asked later why the app was slow, it described that as the established pattern of the codebase. It was its own change. I wrote more about the decision rules that came out of that in the guardrails series. The performance lesson is simpler: a pattern that is convenient for the agent, fetch what you need where you need it, is a pattern that serialises round trips. Server components load the data, the client reads it. We wrote that down as the default and have not moved off it.
The other latency: the agent's own turns
Page loads were half of it. The chat is an agent, and an agent turn has its own budget. We measured 68 turns on the test environment over four days.
| Tools used in the turn | Turns | Median | 90th percentile |
|---|---|---|---|
| 0 | 30 | 5.4 s | 8.1 s |
| 1 | 23 | 4.7 s | 12.2 s |
| 2 | 7 | 10.5 s | 31.1 s |
| 4 to 5 | 4 | 21 to 24 s | 27 to 33 s |
A reply that calls no tool still took 5.4 seconds. Tool execution was not the cost; the tools themselves ran in 20 to 800 ms. The cost was model steps, and what each step carried. Every root step sent at least 33,662 input tokens before any conversation: 35 tool definitions at about 15,800 tokens, mostly their input schemas, plus about 11,000 tokens of instructions. Only 49 of 115 root steps hit the prompt cache, and the first step of a turn almost never did.
Direct calls with the real compiled prompt made the trade-offs concrete: trimming the fixed prompt from 35K to about 8K tokens saved roughly 1.8 seconds on every uncached step; a cache hit saved about 1 second and most of the step's cost. That is where the coach post comes from. Per-turn guidance appended as ephemeral context kept cache reuse at 94 to 97 percent; the same text in system scope dropped it to zero.
One traced turn is worth the whole table. A status question took 32.8 seconds. The answer was known at second 10. The agent then queued follow-up suggestions and waited on a subagent to produce them before replying, so the person waited another 23 seconds for chips they had not asked for. The fix was an ordering rule: answer first, suggestions after, never hold the reply on them. And 14 of the 68 turns were return visits where the agent spent a full root step, about 35K uncached tokens, to decide it had nothing to say. That decision now happens in code.
What closed the loop
- A harness that counts. Queries, actions, bytes, requests. Milliseconds locally are noise; counts are truth.
- One cause, one commit, one measured delta. The table above is the pull request log.
- A live instrument for the deployed environment, and reference numbers every change must meet.
- A regression test for the invariant that mattered: zero server calls before the inbox renders.
- Restore the last good build before debugging forward. Old deployments re-alias in seconds.
- Measure the agent's turns like pages. Tokens before the conversation, cache hits per step, and when the answer was actually known versus when it was delivered.
What remains is honest too: the chat client and its streaming renderer are still the biggest chunks on first load, the analytics library is 169 KB that we have not yet deferred, and the real-network cost of the remaining action chain has only been sized from the test environment, not from production users. That is the next loop.