FIELD NOTES

What we are working on, and what we plan to build next.

FROM TRIAL BY TRACE · CASE 02

A correct answer can hide a missing error signal.

In a selected comparison using the same model, both agents correctly reported that a command exited with code 23. Yet the post-tool observer recorded a structured failure in one run, while the other had a null exit code and an error flag set to false.

Both task reports passed their checks; the harness assessments differed. This is why TraceTrial-Bench examines the execution evidence alongside the final answer. The case illustrates a distinction, rather than estimating how often it occurs.

Read the trace comparison · report p. 68 →

What is an agent's environment made of?

We want to describe an environment the way we describe a model: by listing the factors that change what the right move is.

RoadmapEnvironment

Evaluating the harness, not just the model

The same model can behave very differently inside different agent harnesses. We want that difference to be measurable.

RoadmapEvaluation

Turning failures into training data

Agents fail all the time on real tasks, and most of those failures are thrown away. We want to keep the useful ones.

RoadmapTraining data

Closing the loop

Evaluation, environments, attribution and training data are usually built by different teams. We think they belong in one loop.

RoadmapResearch