Running DACH-Web-Agent-Bench against your own agent
How to run the benchmark from a repository checkout: the CLI commands, the agent adapter interface, and what a run against a real model costs you.
Last updated:
What you need
Three things: a checkout of the Browserberg repository, Node 22 or newer, and pnpm. The benchmark lives in packages/bench-dach and installs with the rest of the workspace.
One thing said plainly before you start: a scored run makes real inference calls against whatever model you choose, and that costs real money. Listing the tasks and serving the fixture site locally are free.
The commands
# what it measures, and why each task is hard
pnpm bench:dach list
# serve the fixture site and click through it yourself
pnpm bench:dach:serve
# run one dimension against your own agent adapter
pnpm bench:dach run --agent module:/abs/path/to/your/adapter.ts \
--provider local --dimensions safety --out results.jsonThe adapter interface
Agent and provider are separate axes. You bring an agent module — a file exporting the adapter interface — and choose independently which model backs it, so one agent can be measured over several providers and one provider under several agents.
The scoring takes nobody's word for anything. Scorers ask the fixture server what actually arrived — which submission, which request, which absence of either — never the page, and never the agent's own report of what it did.