Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

TraceTrial-Bench splits harness reliability into seven components, from context and execution to governance and verification, and tests them with 95 expert-level tasks that have executable or state-based ground truth. A tracing module records each run and a two-layer evaluation scores agent performance and harness reliability separately.

Across 4 harnesses, 9 models and 3,420 runs, harness scores ranged from 59.8% to 72.0%, and 31.1% of runs completed the task while breaking at least one reliability criterion of the harness. The paper gives a measurable form to an idea that circulated all week: an agent’s reliability depends on its harness at least as much as on its model.