00Field guide / agent evaluation
Coding Agent Evaluation That Measures Accepted Outcomes
An evaluation should measure whether an independently checked result satisfies a locked task, not whether the agent announced completion. Quality, latency, cost, reliability, and authority are related, but they are not interchangeable.
01Question
Define the decision the evaluation is supposed to support.
A benchmark for model research, a procurement comparison, a routing policy, and an internal release gate need different evidence. Begin with the decision: whether a candidate system is safe to shadow, whether parallel specialists reduce accepted-outcome latency, or whether a provider meets a task and cost boundary.
Then lock the task, acceptance catalog, model and provider identifiers, tool policy, token and cost ceilings, retry policy, delivery route, and evaluator versions before execution. Changing those after seeing an output converts a measurement into a demonstration.
- Decision stated before metric selection
- Task and acceptance locked
- Provider, model, tools, and limits bound
- No outcome-dependent rule changes
02Endpoint
Measure accepted outcomes, not self-reported completion.
A model can finish quickly and still leave build errors, inaccessible interactions, insecure dependencies, or behavior outside the brief. The evaluation endpoint should be an independently assessed candidate. Graph completion time can still be measured, but it must not be presented as time to an accepted result.
Retain each gate verdict, including failures and blocks. If repair is allowed, count the attempts and time. If no retry is allowed, preserve the first outcome. A system that hides failed runs or reports only its best sample cannot support a reliability claim.
- Independent acceptance catalog
- Every attempt retained
- Completion and acceptance timed separately
- Failure and block reasons preserved
03Metrics
Keep quality, latency, cost, and operational risk separate.
A useful record includes provider-reported input, cached-input, reasoning, and output tokens where available; quote-accounted currency cost; wall-clock stages; gate results; retries; and the exact source and environment identities. Cost should not be called provider-reported if the provider only reports tokens and the platform applies a bound price quote.
Aggregates need enough samples and an explicit statistical method. Medians, intervals, and failure rates should name their population. A single live run can prove that the end-to-end mechanism invoked a provider and retained a verdict; it cannot establish benchmark leadership or general reliability.
- Provider usage distinguished from quote-accounted cost
- Stage latency distinguished from total latency
- Quality and safety gates reported individually
- Sample size and uncertainty shown beside aggregates
04Integrity
Make the run reproducible without leaking prompts, code, or credentials.
Plans, quotes, authorization, preflight, execution, candidate trees, and gate outputs should form a verified digest chain. The materializer must reject a one-byte mismatch before applying the candidate. Replaying the record should not require contacting the provider again.
Public evidence needs a privacy boundary. Account aliases and policy states can be retained; raw credentials cannot. Customer material should not move through a sharing-enabled account. A fixture may prove the verifier works, but it must never be published as a live provider outcome.
- Offline lineage verification
- Exact candidate reconstruction
- Secret-free evidence
- Fixtures clearly separated from live runs
05Interpretation
Publish what the artifact proves, and no more.
Every result needs limitations: task count, task type, model, provider, environment, retry policy, customer-data status, evaluator coverage, and any open human gate. Negative results are useful when the protocol is sound; they reveal a failure boundary without requiring the system to retest until it passes.
The strongest evaluation program grows in stages: deterministic fixtures, controlled specialist comparisons, genuine provider runs, repeated task sets, design-partner acceptance, and external review. Each stage adds authority. None should borrow the credibility of a stage that has not happened.
- Limitations are part of the result
- Single runs stay single-run evidence
- Negative outcomes remain publishable
- Broader claims wait for broader samples
Design partners
Evaluate the system you will actually trust with work.
Measure accepted outcomes on locked tasks, and keep fixture results and real results apart. TraceAxiom Verify is a GitHub Action, in development and not installable yet, that runs a pull request’s tests on base and head, checks its claims, and posts one verdict with its reasons.
Join the waitlist