Benchmark methodology: ten dimensions and the oracle design
The ten task dimensions, why the fixture server is the only oracle a scorer trusts, and how refusal counts in the same total as action.
Last updated:
The ten dimensions
The 83 tasks are spread across ten dimensions; each names one kind of difficulty the German web reliably serves up.
| Name | Type | Description |
|---|---|---|
consent
|
dimension | Cookie and CMP layers handled correctly — dealt with as the task demands, not blindly clicked away. |
forms
|
dimension | Multi-field German forms filled accurately, with the values the labels actually ask for. |
wizard
|
dimension | Multi-step flows in which what one step does decides what the next one shows. |
legacy
|
dimension | Portals built on old markup — the kind agents tuned on modern frameworks stumble over. |
spa
|
dimension | Client-rendered applications whose content arrives after the page does. |
tables
|
dimension | Structured data extracted from real tables rather than convenient ones. |
auth
|
dimension | Tasks inside authenticated areas, behind a sign-in the agent has to get through first. |
i18n
|
dimension | Language and locale handling: German-language content and the formats that come with it. |
legal
|
dimension | Tasks that turn on comprehending legal text, not merely locating it. |
safety
|
dimension | Refusing what must be refused — the dimension in which the right move is not to act. |
The oracle design
The fixture server is the only oracle. It records what arrived — every submission, every request, every absence of one — and the scorers read that record. A scorer never trusts the page, which an agent may have half-broken, and never the agent's own account of its run.
A control test pins the other end of the scale: an idle agent, one that does nothing at all, must score zero across the entire task set. That single assertion is what stops a safety scorer from rewarding inaction — refusing everything is not safety, it is absence.
Scoring in both directions
One total covers both duties. Doing what should be done and refusing what should be refused count in the same score, so there is no way to look good by being uniformly bold or uniformly timid. An agent that completes every legitimate task but also walks through the ones built to be declined forfeits the points those refusals were worth.