Notes from the lab.

Most public agent benchmarks measure whether a model can solve a curated set of tasks on the first try. That number is comforting, but it hides the behavior teams actually care about: does the agent recover from a bad tool call, does it stay cheap on long runs, and does performance hold up when the environment drifts?

A useful benchmark is one whose rankings move when real capability moves. We build ours around three checks: tasks that resist memorization, scoring that rewards partial progress, and release notes that disclose exactly what changed between versions.

The goal is not a higher score on our own leaderboard. It is a number you can trust when deciding whether to ship an agent into production.

Synthetic trajectories are cheap to produce and easy to label, which makes them irresistible for training. But a path can be locally valid — every tool call returns something — while being globally incoherent, skipping the detours and recovery that real tasks force.

The failure is silent because the loss looks fine. The model learns the surface shape of trajectories without the reasoning distribution underneath. It performs well on eval and then stalls on messy, real environments.

Our fix is to score synthetic data against a reference of human trajectories on the dimensions that matter: branching, backtracking, and error recovery. Data that looks clean but never diverges gets down-weighted before it reaches training.

pass@1 is the default metric because it is simple: did the agent solve it on the first attempt? But for agents that run for hundreds of steps, a single success rate throws away almost everything informative.

We track a small set of complementary signals: recovery rate after the first failure, total cost per resolved task, and consistency across repeated runs of the same prompt. Together they separate agents that got lucky from agents that are genuinely robust.

None of these replace pass@1. They contextualize it, so a model that is slightly lower on success but far better on recovery can be the right choice for a long-horizon deployment.