Why browser agents fail in production
Published
The demo works. It always works. Someone films a thirty-second clip of an agent logging into a portal, downloading an invoice and filing it, and the clip is honest — that really happened. Then the same task runs four hundred times a day for a month and the picture changes completely.
What follows is a taxonomy of what actually goes wrong, drawn from watching a lot of runs fail. It is organised by failure mode rather than by component, because the useful question when something breaks at three in the morning is not "which layer is this" but "what class of problem am I looking at, and does it recur".
The single most important property of this list: four of the eight modes are silent. They do not raise. They do not exit non-zero. A monitoring setup built around process exit codes will report a healthy system for weeks while it produces nothing of value.
The taxonomy
1. Selector drift — loud, frequent, well understood
The classic. A button moves, a class name is regenerated by a build, a wrapper div appears, and a selector that has worked for eight months stops matching.
This is the failure everybody plans for, which is why it is the least interesting one on the list. It fails immediately and visibly. Your retry logic catches it, your alerting fires, someone fixes the selector. The cost is real but it is a maintenance cost, not a correctness risk.
It is worth saying plainly that agents genuinely help here, and this is the strongest honest argument for them. A model reading the page fresh each turn does not care that the class name changed. That is not marketing — it is the one place where the non-determinism buys something concrete.
2. Timing and readiness — loud, frequent, badly diagnosed
The element exists in the DOM but is not yet interactive. The click lands on the overlay that is mid-fade. The page has fired its load event but the framework has not finished hydrating, so the handler you are trying to trigger does not exist yet.
Diagnosed badly because the symptom — an element was not clickable — points at the element, and the cause is a race. The tell is intermittency: a step that fails perhaps one run in twelve, always on the slower runs, and never when you step through it by hand.
The fix is almost never a longer sleep. It is asserting on the condition you actually care about:
// Fragile: passes as soon as the node exists.
await page.waitForSelector('#submit');
await page.click('#submit');
// Better: waits for the state that makes the click meaningful.
await page.locator('#submit').click(); // auto-waits for actionability
await expect(page.locator('#result')).toBeVisible(); // asserts the effect, not the cause
The second version fails on the outcome, which means the failure message tells you what did not happen rather than what was not found.
3. Session and authentication expiry — silent, then loud, at the worst time
A session cookie expires. A refresh token rotates. The portal invalidates the session because it saw a login from another address. The next request returns 200 and the login page.
This one is nasty because the HTTP status is fine and the page renders correctly. It is a successful response to a request nobody wanted to make. If your extraction runs after this point, it extracts the login page, and what it writes downstream is structurally valid and semantically empty.
Two things help. First, assert on a post-authentication marker before doing anything that depends on being authenticated — the presence of an account name, a logout link, anything the anonymous page cannot show. Second, treat session lifetime as a known quantity rather than a surprise: if the portal expires sessions after twenty minutes, a task that takes twenty-five minutes has a bug that has not happened yet.
4. Silent empty extraction — silent, common, expensive
The selector matched. The element was there. The text content was an empty string, because the value loads asynchronously and you read it before it arrived, or because the field is genuinely empty for this record and nobody decided what that should mean.
This is the failure mode that produces the phone call six weeks later asking why the numbers are wrong.
The defence is a validation layer that is separate from the extraction layer and refuses to let structurally-valid nonsense through:
const record = await extract(page);
// Shape, not presence. A populated string is not evidence of anything.
if (!record.invoiceNumber?.match(/^[A-Z]{2}-\d{6}$/)) {
throw new ExtractionError('invoiceNumber failed format check', record);
}
if (record.lineItems.length === 0) {
throw new ExtractionError('no line items on an invoice', record);
}
const sum = record.lineItems.reduce((t, l) => t + l.amount, 0);
if (Math.abs(sum - record.total) > 0.01) {
throw new ExtractionError('total does not reconcile with lines', record);
}
Notice that the last check is the one that catches the widest class of problems, and it is not a check on any single field. Cross-field consistency is where extraction errors reveal themselves.
5. Blocked or rate-limited — loud or silent depending on the site
You are making requests faster than the site expects, from an address range with a poor reputation, or in a pattern that does not look like a person. The response is a 429, an interstitial, a challenge page, or — the silent variant — a normal-looking page with less data on it than a browser would see.
The important reframe: this is a reliability problem before it is anything else. The instinct in this industry is to reach for proxy rotation, which treats being blocked as a network-layer problem to be routed around. It is usually a behaviour problem. Halving the request rate resolves a large fraction of these, costs nothing to try, and is the first thing to test precisely because it is cheap.
Where you have an actual relationship with the site — a supplier portal, a customer's own system, a partner API — the durable answer is an allowlist entry or a declared agent identity rather than a disguise. Web Bot Auth and the signed-agent schemes several large networks now support exist precisely so that a legitimate automated client can be recognised deliberately instead of inferred from behaviour. That is a direction of travel worth building toward, because evasion is a maintenance commitment that only ever gets more expensive. We have written more on what detection actually measures.
6. Non-deterministic path divergence — silent, agent-specific
Two runs of the same task take different routes. Both are legitimate. One of them clicks "Download all" and the other downloads each invoice individually, and the second takes eleven times longer and costs eleven times more.
Neither run failed. Your success metric says 100%. Your bill says something else.
The metric that surfaces this is turn count per task, tracked as a distribution rather than an average. A task whose median is 8 turns and whose 95th percentile is 60 is not one task; it is two tasks wearing the same name, and the expensive one is worth understanding.
7. Prompt injection through page content — silent, agent-specific, security-relevant
The page contains text that the model reads as instruction. It might be a support ticket someone filed, a product review, a filename in a directory listing, or a hidden element placed there deliberately.
This is not a hypothetical. Any workflow where an agent reads content authored by someone other than you has this exposure, and the exposure is proportional to what the agent is allowed to do next. It deserves more than a paragraph, and it gets one in the piece on securing browser agents.
For the purposes of this taxonomy the point is narrower: injection presents as a successful run. The agent did something. It reported completion. Nothing raised.
8. Resource exhaustion in the browser process — loud, but late
Long-running sessions accumulate. Detached DOM nodes, event listeners that were never removed, an ever-growing heap in a single-page application that was designed for a human who would eventually close the tab. The browser gets slower, then unresponsive, then dies.
This is a failure of the deployment model more than of the automation. A session-per-task model with an enforced ceiling on both duration and memory converts a slow degradation into a clean, attributable failure. That is a strictly better outcome: a task that failed is recoverable, a fleet that is quietly getting slower is not.
The pattern
Lay the eight out and the shape is clear:
| Mode | Signal | Recurs? | Found by |
|---|---|---|---|
| Selector drift | Loud | Constantly | Exit code |
| Timing and readiness | Loud, intermittent | Constantly | Flake rate |
| Session expiry | Silent | Predictably | Post-auth assertion |
| Silent empty extraction | Silent | Constantly | Field fill rate |
| Blocked or rate-limited | Either | Under load | Response-shape check |
| Path divergence | Silent | Always | Turn-count distribution |
| Prompt injection | Silent | Rarely, badly | Action audit |
| Resource exhaustion | Loud, late | Under duration | Session limits |
Every silent mode in that table is caught by a data check, and none is caught by a process check. That is the single most useful thing on this page. Monitoring built on "did the process exit cleanly" is blind to five of eight rows.
What to measure instead
Four signals, in the order they earn their keep:
Field fill rate, per field, over time. Not "did we get a record" but "what fraction of records had a populated, valid invoiceNumber this week versus last". Silent extraction failures show up here days before anyone notices the totals are wrong. Alert on the derivative, not the absolute value — a field that was always 60% populated is fine; one that dropped from 98% to 60% on Tuesday is an incident.
Turn count distribution per task type. Median, 95th percentile, and the count above your ceiling. This is your cost control and your divergence detector in one number, and it is close to free to collect.
Time to first meaningful assertion. How long from session start to the first check that proves you are where you meant to be, authenticated as who you meant to be. Rising values here predict blocking and rate-limiting before the blocking itself becomes visible.
Replay availability for failed runs. Not a metric — a capability. The difference between a five-minute diagnosis and a two-day one is whether you can watch what the agent saw. If your platform discards that on failure, you have optimised storage at the direct expense of the thing you need most on the worst day.
A harness that makes the silent modes loud
None of the above requires a platform. It requires four decisions, and they are worth making explicitly rather than inheriting by default.
Decide what "done" means, in data. Every task needs a completion predicate that is checkable without trusting the agent's own report. "The agent said it filed the invoice" is not a completion signal; "a row exists in the ledger with this invoice number and today's date" is. This sounds obvious and is skipped constantly, because the agent's self-report is right there and free.
Give every run a budget, and enforce it in three dimensions. Wall-clock duration, turn count and token spend. A run that exceeds any of them should be killed, not warned about. The reason is mode 6: a diverged run does not know it has diverged, and left alone it will spend your whole month's budget being confidently wrong in an expensive way.
Separate the retry decision from the retry. Not every failure should be retried, and retrying the wrong class is how a rate-limit problem becomes an outage. A useful split:
| Class | Retry? | Why |
|---|---|---|
| Timing, readiness | Yes, immediately | The race will usually go the other way |
| Selector drift | No | It will fail identically until someone changes something |
| Session expiry | Yes, after re-authenticating | The retry is only meaningful with a fresh session |
| Blocked or rate-limited | Yes, with a long backoff | An immediate retry makes it worse, measurably |
| Validation failure | No, route to review | The data is wrong; a second attempt produces wrong data twice |
| Resource exhaustion | Yes, in a fresh session | The process is the problem, not the task |
The row that surprises people is validation failure. The instinct is to retry because a retry is cheap, but a record that failed a cross-field consistency check failed for a reason, and the reason is very rarely transient.
Keep the evidence, and keep it addressable. For each run: the final URL, a timestamp, the raw fragment each extracted field came from, and a replay if you have one. This is a small amount of storage and it is the entire difference between answering a question about a number and shrugging at it. It also happens to be what a regulated process needs anyway, so the work is not wasted even where nobody has asked for it yet.
Where agents help and where they do not
Because the taxonomy cuts both ways, and a piece that only listed problems would be misleading about the trade.
Agents materially reduce mode 1, and that is the whole case for them. Selector drift is the dominant maintenance cost of scripted automation and a model reading the page fresh is genuinely robust to it. If you maintain automation against a hundred portals that each redesign on their own schedule, this alone can be decisive.
Agents leave modes 2, 3, 5 and 8 roughly unchanged. Readiness races, session expiry, blocking and memory growth are properties of the browser and the network, not of who is deciding what to click. A model does not wait more correctly than an auto-waiting locator does.
Agents introduce modes 6 and 7 outright. Path divergence and prompt injection have no equivalent in scripted automation, and both are silent, which means they are exactly the kind of problem that a team migrating from scripts will not have instrumentation for.
So the honest summary is: agents trade a loud, frequent, well-understood failure for two silent, rarer, less-understood ones. That is often a good trade — a loud failure at high frequency is genuinely expensive — but it is a trade, and it is worth making with the instrumentation for the new modes already in place rather than added after the first incident.
The uncomfortable conclusion
Most teams shipping browser automation are running monitoring designed for HTTP services against a system whose dominant failure modes are silent and semantic. The process exited zero. The dashboard is green. The data has been wrong since the fourteenth.
The fix is not more infrastructure. It is moving the assertions from the process to the output, and accepting that in this domain a successful run and a correct run are genuinely different claims that need to be checked separately.
If you want the layer underneath to stop being one of your variables, that is what we do — sessions with enforced limits, replays kept long enough to be useful, and no evasion features to complicate the picture. But the measurement discipline above matters more than the vendor, and it is worth adopting whether or not you ever talk to us.