Measure Coding-Agent Speedup

01Comparison

Compare the same task under the same acceptance rules.

Freeze the starting repository revision, requested behavior, acceptance checks, allowed tools, model configuration and resource budget for each workflow. Record the baseline version and measurement date. A parallel workflow that gets a smaller task, extra hidden assistance or weaker checks answers a different question.

Choose the task group before running the comparison. Keep development tasks separate from held-out evaluation tasks. Repeat matched runs where feasible and vary their order so one configuration does not consistently receive a warm cache or an unusually quiet machine. Record differences in cache state, queue conditions and reviewer availability; do not remove inconvenient pairs after seeing their outcomes.

02Clock

Define the start and stop events before measuring.

For an end-to-end user measurement, start when the eligible task is submitted and stop when its exact candidate receives the required independent acceptance. Include provisioning, queue waits, authoring, integration, repair, verification and review wait within that boundary. A narrower engine-only timing can be useful too, but give it a separate name and disclose its exclusions.

Use elapsed durations from a suitable clock and retain the underlying event records. When components run on different hosts, record clock synchronization and uncertainty before subtracting their timestamps. Sum only intervals that do not overlap. Two workers running for twelve minutes at the same time contribute twelve minutes of elapsed authoring time and twenty-four worker-minutes of work. Those are different quantities.

• Task identity and starting revision
• Start event, acceptance event and elapsed-time method
• Attempt and repair links for the same task
• Overlapping work, wait time and clock uncertainty

03Worked example

Twice as fast at authoring can mean only 1.09× faster overall.

Assume both workflows complete the same illustrative task and satisfy identical acceptance requirements. For this simplified timeline, stages do not overlap. The sequential workflow takes 6 minutes to prepare, 24 to author, 12 to integrate, 18 to verify and 12 for review through acceptance: 72 minutes total.

The parallel workflow takes 6 minutes to prepare, 12 of elapsed authoring time, 18 to integrate, 18 to verify and 12 for review through acceptance: 66 minutes total. Authoring time is halved, but extra integration work consumes part of the saving.

Define elapsed-time speedup as baseline time divided by candidate-workflow time. Here it is 72 / 66, approximately 1.09×. The elapsed-time reduction is (72 − 66) / 72, approximately 8.3%. State the formula: a speedup ratio and a percentage reduction are not interchangeable.

Now suppose a required repair adds 8 minutes before acceptance in the parallel workflow. Its total becomes 74 minutes. The ratio is 72 / 74, approximately 0.97×, and elapsed time is approximately 2.8% longer than the baseline. Keep the repair inside the same task record and clock. This example supplies arithmetic, not a prediction about which workflow will win.

04Incomplete outcomes

A failed or capped task has no completed acceptance time.

If a task reaches a 120-minute cap without acceptance, record its 120 minutes of observed work, its terminal reason and acceptance time as unavailable. Do not label 120 minutes as time to an accepted result. Preserve the task in the predeclared cohort, including its consumed resources and any partial verification evidence.

Report the full task inventory and accepted-result rate beside timing summaries. For example, if a baseline accepts four of five tasks and a parallel workflow accepts three of five, a faster median among the parallel workflow’s successful tasks cannot establish an overall delivery advantage by itself. Their success subsets differ.

Show matched-task results and state exactly which observations enter each summary. A comparison restricted to tasks accepted by both workflows is conditional on that subset; retain the excluded failures and explain the restriction. Predeclare how capped and pending outcomes will be analyzed, and leave an advantage claim unresolved when the evidence does not support it.

05Tradeoffs

Keep human attention and total usage next to elapsed time.

Measure active review, coordination and correction minutes separately from elapsed review time. A reviewer waiting for an artifact is different from a reviewer actively repairing a patch. Include founder or operator assistance even when nobody invoices for it. Record what work the reviewer actually performed.

Track model and tool usage, sandbox compute, storage and verification costs across every attempt. More concurrent workers may reduce elapsed time while consuming more total resources. A workflow that is a little faster but needs substantially more review or has a lower accepted-result rate may be unsuitable for the intended team. Set worthwhile-effect and unacceptable-tradeoff thresholds before interpreting the comparison.

06Review record

Make the claimed improvement reproducible and bounded.

Retain the task manifest, configuration versions, candidate revisions, complete attempt history, event times, acceptance records, usage inputs and analysis method. Bind each verdict to the candidate it actually checked. An old passing report does not establish acceptance for a repaired revision. Record exclusions, missing data and unfavorable outcomes alongside the reported result.

Use the multi-agent development guide for dependency and integration design, the evaluation guide to define the cohort, and the cost-per-accepted-change guide for the resource calculation. Repeat comparisons on the task families that matter to the user before extending a claim beyond the tested group.

This guide explains an evaluation method and reports no measured TraceAxiom speed figure. A timing result from a controlled fixture does not establish performance on real user repositories.