Agent evaluation lab

TraceLite.ai

Traceable evaluation infrastructure for AI agents. Capture real trajectories, replay failures, and turn benchmark runs into evidence your team can inspect.

View GitHub repo Explore the system

Trace Full agent trajectory capture

Replay Failure reproduction and audit

Verify Case-level scoring and evidence

What we ship

Evaluation pieces for agents that need to be trusted, not just demoed.

TraceLite.ai is built around reproducible runs: the task, the tools, the agent actions, the final artifact, and the verifier all stay connected.

GitHub ->

01Trace layer

Trajectory capture

Record prompts, tool calls, file changes, outputs, and intermediate states so every agent result has a readable execution trail.

Repository Workflow

02Benchmark layer

Case-driven evaluation

Run targeted cases that expose planning gaps, tool misuse, lifecycle failures, and final-artifact mismatches in real workflows.

Cases Thesis

03Verifier layer

Evidence-backed scoring

Separate clean completion from correct completion by validating the final artifact, the path taken, and the contract the task required.

Verifier Contact

Thesis

Agent reliability starts with traces you can argue from.

Pass rates are too flat on their own. A useful evaluation system should explain what happened, where the run diverged, and whether the agent satisfied the real contract of the task.

Behavior needs context, not screenshots.

Tool calls, hidden assumptions, environment state, and final artifacts belong in the same evidence record.

Benchmarks should fail for the right reason.

A discriminative case catches semantic mistakes even when the agent exits cleanly and produces a plausible-looking result.

Evaluation should be repeatable by another team.

Tasks, harness configuration, scoring logic, and expected evidence should be portable enough for independent reproduction.

Workflow

From one run to an inspectable evaluation record.

The site is static, but the product story is concrete: agents enter a harness, traces are captured, verifiers judge the result, and failures become reusable cases.

$ tracelite run cases/lifecycle-orchestration

agent: openclaw-runtime

trace: captured tool calls and artifacts

replay: deterministic environment snapshot

verifier: contract mismatch detected

evidence: saved as reusable benchmark case

Run the agent in a controlled harness.

Fix the task state, tool surface, and expected contract before measuring behavior.

Capture the whole trajectory.

Keep the reasoning-adjacent evidence that matters: calls, outputs, files, and final deliverables.

Score with explicit verifiers.

Judge correctness against the task contract rather than relying on whether the run looked smooth.

Promote failures into cases.

Convert real bad runs into reusable tests for regression, model comparison, and harness design.

Build evaluation around evidence.

Use the repository as the public home for TraceLite.ai while the benchmark cases and runtime pieces mature.

Open GitHub