Agents run every night against fixed tasks. Every trace is kept, every failure is labeled. Nobody is watching this run.
Loading…
Pass rates and labels below count nightly runs only. Calibration runs, which try every task including retired ones, and a run stopped by hand stay in the corpus but out of these numbers.
| family | episodes | pass rate | avg cost |
|---|
Every queued task runs once on each model, so these rows are paired. The table above blends them.
| model | family | episodes | pass rate | avg cost |
|---|
| label | count |
|---|
Every task, worst pass rate first, with each model's pass rate beside it. The checkers are deterministic, so a task that moves has moved for a reason outside this repository.
| task | family | runs | pass rate | by model | turns |
|---|
The most recent failed episodes, with what was asked, what the agent did, and its label: taken straight from the checker for coding tasks, given by a separate model for tool tasks. This is the corpus; the numbers above are just its shape.
Loading…