Chapter 29

Measure Accepted Changes, Not the Number of Answers

An agent's metric must reflect the benefit to the developer and the risk to the repository: was the patch accepted, how much rework was needed, was the scope preserved, did the checks pass, and how much human time went into acceptance.

Seven metrics and what each shows are worth keeping together.

Metric · How to count · What it shows

Acceptance rate - Tasks with an accepted patch / completed tasks of the same type; Practical usefulness of the result
Rework turns - Number of fix cycles after the first review; Quality of the plan and context
Scope precision - Expected changed files vs actual; Change discipline
Verification completeness - Acceptance checks with an artifact or exit code; Provability of the result
Engineer review time - Active time until accept or reject; Real attention saved
Escalation quality - Stops for a correct reason / all necessary stops; How well the workflow handles uncertainty
Post-merge defects - Regressions tied to an accepted agent patch; Quality after local checks

So the metrics rest on facts rather than memory, every run saves a minimal audit record.

JSON

{
  "taskId": "discount-142",
  "contractHash": "sha256:...",
  "product": "codex|claude-code|cursor",
  "surface": "cli|ide|app|cloud",
  "startRevision": "git-sha",
  "allowedWritePaths": [
    "src/discount.js"
  ],
  "commands": [
    {
      "argv": [
        "node",
        "--test",
        "tests/discount.test.js"
      ],
      "exitCode": 0,
      "timedOut": false
    }
  ],
  "changedFiles": [
    "src/discount.js"
  ],
  "independentReview": "pass",
  "accepted": true,
  "residualRisks": []
}

Comparing fairly takes discipline: group tasks by class (bug fix, test writing, refactor, review), fix one environment and the same acceptance checks, don't mix evaluation with production secrets, compare accepted outcomes rather than the beauty of the final message, and break failures down by cause - context, plan, tool, environment, permission, model, or oracle.

Productivity can't be derived from tokens. Token count and wall time are useful for cost and capacity but don't tell you whether the agent saved engineer time or created risk. Tie usage to an accepted, verified result, or the metric measures activity, not benefit.

The metrics are in place. What remains is to assemble it all into a rollout order - autonomy is added in stages, not in one setting.

Measure Accepted Changes, Not the Number of Answers

Links