TRACELITE AI / RESEARCH

Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

Zhiyong Cao*, Hengjie Zhu*, Yongxin Wang*, Xuchen Li*, Jintao Li, Shuxia Liu, Dunqiang Liu, Bo Wang, Kairan Zhang, Cong Lin, JiangDi Yu, Siyuan Liu, Sijia Yao, Yi-Liang Wei, Lingxu Ma, Aijia Wang, Anxin Chen, Zixian Wei, Xiang Ruan, Ruizhi Xu, Ruqian Shi, Baihua Xiao✉, Sicheng Zhou✉

* Core contributor✉ Corresponding author

TraceLite AI

arXiv preprint · arXiv:2610.32285v1 ·

A successful task does not always mean a reliable harness.

Abstract

LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses.

To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness.

We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level.

Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.

Author abstract · arXiv v1, 23 October 2026

Why measure the harness?

Task outcomes describe the complete agent system. They do not establish whether context, tool results, or lifecycle behavior remained faithful during execution.

How task outcomes can hide harness failures; trace evidence distinguishes model and harness behavior.
TraceTrial-Bench looks beyond the final answer: native execution traces reveal what the model saw, what the harness delivered, and what actually happened.
95
Mechanism-oriented tasks
7
Reliability dimensions
4 × 9
Harnesses × models
3,420
Scored executions

From execution traces to reliable evidence

A benchmark for the system around the model.

Four-stage framework: task design, native execution and tracing, multi-source evidence acquisition, and failure attribution.
Framework overview. Native harness behavior is preserved; criterion-specific evidence connects each verdict to its execution record.
01

Trace native behavior

Collect execution records without replacing native context policies, tool transport, or session behavior.

02

Evaluate two layers

Score task deliverables and harness obligations separately, using executable checks and criterion-specific evidence.

03

Attribute with evidence

Review declared mechanism chains connecting harness shortfalls, observable distortions, and agent consequences.

Candidate-link activation identifies a path for review. It does not, by itself, establish causality.

Seven dimensions. 95 targeted tasks.

Mechanism-oriented tasks with executable or state-based ground truth, preserving equivalent task semantics across native harnesses.

C

Context

Information preservation, isolation, and recovery.

15 tasks
E

Execution

Process semantics and faithful result delivery.

27 tasks
G

Governance

Permissions, policies, and security boundaries.

9 tasks
L

Lifecycle

Session transitions, delegation, and state recovery.

19 tasks
O

Observability

Faithful diagnostic and failure channels.

7 tasks
T

Tools

Tool interfaces, discovery, and transport fidelity.

17 tasks
V

Verification

Completion claims versus authoritative outcomes.

1 task

40 mechanism-level properties · 46 single-turn, 45 fixed multi-turn, and 4 simulator-driven tasks. Coverage is uneven; Verification contains one task.

What is inside a benchmark task?

Every scenario pairs a concrete deliverable with a specific harness responsibility.

01

Instructions & interaction

Each task fixes its user prompt or turn sequence, required actions, working boundaries, and expected deliverables. Single-turn, fixed multi-turn, and simulator-driven tasks exercise different interaction conditions.

02

Initial state & interfaces

The task package specifies workspace files, controlled services, native and task-specific tools, and the mechanism trigger—such as oversized output, timeout, or a session handoff.

03

Independent ground truth

Private truth, simulator ledgers, process outcomes, and executable verifiers check what actually happened. Private oracle state stays outside the agent-writable workspace.

04

Resource & execution contract

Resolved configurations retain time, turn, and tool-call limits alongside runtime requirements. Adapters preserve task semantics while keeping each harness’s native behavior.

05

Two score sheets

Agent criteria check actions and deliverables. Harness criteria check the declared obligation at an explicit observation boundary, using its own evidence selectors.

06

Inspectable evidence

Trace IR, native records, artifact differences, and environment probes retain provenance. Verdicts remain connected to the evidence used to reach them.

Example: recover 12 escalation codes from one ticket export. The agent must use bounded requests against the same snapshot; an independent ledger and verifier compare the issued codes, delivered tool results, and final JSON mapping.

Inspect the output-recovery case →

How an evaluation runs

Fixed task contracts, native harness execution, and evidence-grounded scoring.

01

Prepare & isolate

Task-scoped Docker containers establish the initial workspace and declared limits. Admission checks cover container lifecycle, archive integrity, trace scoreability, applicable probes, and private oracles.

02

Execute & collect

Native adapters run Claude Code, Codex, DeepSeek Harness, and OpenClaw. Trajectory tracing and multi-source evidence capture execution events, artifacts, environment state, and recorded resource usage.

03

Verify & diagnose

Deterministic checks are fixed first. Semantic Agent and Harness criteria are then evaluated with their respective evidence; final semantic evaluations use DeepSeek-V4-Flash. Candidate mechanism links undergo separate evidence review before propagation claims.

Configured task ceilings range from 120–5,400 seconds, 1–120 user-input turns, and 5–300 task-tool calls. These are allowed limits, not observed usage. Each model–harness block contains the same 95 tasks.

Task success is only part of the story

In this selected evaluation, different harnesses lead on task success and Harness-criterion satisfaction.

Experimental conditions

The arXiv v1 results use 95 tasks × 9 model configurations × 4 native harnesses: Claude Code, Codex, DeepSeek Harness, and OpenClaw. The archive retains one selected execution per cell, including recovery replacements: 3,420 runs in total and 855 matched model–task cells per harness.

The nine model groups are DeepSeek-V4-Pro, Gemini 3.7 Flash, GLM-5.3, GPT-5.6-Sol, Grok-4.6, HY3, Kimi K3, Opus 5, and Qwen3.8-Max. All final semantic evaluations use deepseek-v4-flash; scoring budgets vary across runs. These are descriptive results for the archived configurations.

Paper: experimental protocol, Section 4

31.1%

1,065 / 3,420 runs achieve task success despite at least one failed Harness criterion

8.4%

286 / 3,420 runs pass every criterion in both the Agent and Harness layers

12.2 pts

Harness Macro ranges from 59.8% to 72.0% across the four harnesses

Overall scores for nine model groups and four harnesses. Claude Code leads harness Overall at 73.5%.
Overall Score gives equal weight to Agent Score and Harness Macro. Model bars average across four harnesses; harness bars average across nine models.
Aggregate results · 855 matched model–task cells per harness
HarnessAgent PassAgent ScoreHarness MacroCleanOverall
Claude Code44.075.971.29.173.5
Codex35.870.759.82.865.2
DeepSeek Harness40.073.368.89.871.0
OpenClaw38.271.772.011.771.8

All values are percentages. Agent Pass requires every Agent criterion; Clean requires every criterion in both layers. Macro averages per-run fractions.

01

Agent Score & Agent Pass

Agent Score averages each run’s fraction of passed Agent criteria. Agent Pass counts only runs that satisfy every Agent criterion; a high partial score does not imply a fully completed task.

02

Harness Macro & Micro

Macro gives each run equal weight by averaging its passed-criterion fraction. Micro pools all Harness criteria, so tasks with longer score sheets contribute more items.

03

Overall & Clean

Overall equally weights Agent Score and Harness Macro. Clean requires every criterion in both layers to pass. These measures describe different aspects of the same selected runs.

Per-run Overall = 50 × (passed Agent criteria / total Agent criteria) + 50 × (passed Harness criteria / total Harness criteria). The two layers use distinct score sheets, but the outcome-conditioned treatment of unexercised Harness criteria means they are not statistically independent.

Explore the full leaderboard →

Cost and overall performance across model–harness configurations.
Performance and recorded cost. Estimates use frozen model-specific prices and selected executions, excluding Judge calls and discarded runs.
Reliability scores by dimension, four criterion-verdict states, and candidate-link incidence.
Reliability analysis across the 3,420-run grid. Automatically activated candidate links are distinct from evidence-validated causal propagation.

Look inside the trace

Five paired case studies. The same model and task, seen through different native harnesses.

Can the agent recover what was truncated?

The selected Opus 5 traces compare whether visible output preserves the information needed to retrieve missing content, alongside the observed recovery outcome.

Output recovery paired execution trajectories
Output recovery · Selected trajectories from the manuscript appendix. Fractions are passed criteria / total criteria for each run.

Correct reports can hide different failure states.

Both agents can recognize failure in text while structured error and exit-code fields tell different stories.

Failure signals paired execution trajectories
Failure signals · Selected trajectories from the manuscript appendix. Fractions are passed criteria / total criteria for each run.

Dispatch accepted is not completion delivered.

Asynchronous delegation requires preserving the distinction between an accepted child task and a result actually delivered to the parent.

Child handoff paired execution trajectories
Child handoff · Selected trajectories from the manuscript appendix. Fractions are passed criteria / total criteria for each run.

Did the end-of-file signal reach the process?

Native terminal traces reveal whether EOF delivery and the resulting task state agree.

Terminal EOF paired execution trajectories
Terminal EOF · Selected trajectories from the manuscript appendix. Fractions are passed criteria / total criteria for each run.

Correct final values. Repeated actions.

With Qwen3.8-Max, session-handoff traces distinguish correct final values from compliance with an exactly-once interaction rule.

Exactly once paired execution trajectories
Exactly once · Selected trajectories from the manuscript appendix. Fractions are passed criteria / total criteria for each run.

Cases explain mechanisms; they do not estimate failure frequency or establish controlled causal effects. Click any figure to enlarge.

Limitations

The targeted task suite has uneven mechanism coverage, including only one Verification task. One selected execution is retained for each task–model–harness combination, so these results do not estimate repeated-run stability and provide limited evidence about deployment workloads.

Task revisions, runtime settings, recovery conditions, and Judge budgets vary across the archive. These differences limit causal claims about individual harness design choices; runtime durations are not a controlled throughput comparison.

A failed Harness criterion means satisfaction was not established under the declared evidence contract; it does not automatically confirm a defect. Candidate links require further evidence review before failure propagation is established. The study does not estimate corpus-level Judge agreement, evidence-selector recall, candidate-link precision, or attribution accuracy.

Paper: interpretation limits and limitations, Sections 5.7 and 6.1

Inspect, compare, reproduce

Follow a result back to its task, configuration, and execution evidence.

01

Inspect a task

Start with the prompt, initial workspace, resource limits, verifier, and the two score sheets. Confirm which mechanism is triggered and which observation establishes the verdict.

02

Compare configurations

Use the leaderboard to compare harness aggregates or model–harness pairs. Keep task population and metric definitions aligned; inspect the paired cases to understand concrete mechanisms.

03

Recompute from evidence

Use the archived task and resolved run configuration together with traces, artifacts, and final criterion verdicts. Final combined-score files supply score authority; a root execution-status label alone does not.

Paper license: CC BY-NC-SA 4.0 · arXiv v1

BibTeX

@misc{cao2026trialbytrace,
  title = {Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution},
  author = {Cao, Zhiyong and
            Zhu, Hengjie and
            Wang, Yongxin and
            Li, Xuchen and
            Li, Jintao and
            Liu, Shuxia and
            Liu, Dunqiang and
            Wang, Bo and
            Zhang, Kairan and
            Lin, Cong and
            Yu, JiangDi and
            Liu, Siyuan and
            Yao, Sijia and
            Wei, Yi-Liang and
            Ma, Lingxu and
            Wang, Aijia and
            Chen, Anxin and
            Wei, Zixian and
            Ruan, Xiang and
            Xu, Ruizhi and
            Shi, Ruqian and
            Xiao, Baihua and
            Zhou, Sicheng},
  year = {2026},
  eprint = {2610.32285},
  archivePrefix = {arXiv},
  primaryClass = {cs.AI},
  url = {https://arxiv.org/abs/2610.32285}
}