Chapter 29

Pilot on Real, Expensive Tasks

A multi-agent architecture must win on a representative task bank, not on one specially picked demo. A pilot plan enforces discipline: gather 30-100 representative tasks from the real backlog and production failures; fix a single-agent or workflow baseline; define the outcome graders before tuning the team; run the variants on the same inputs and environments; collect cost, wall time, success, conflict, and human rework; analyze the failed traces and classify the causes; keep multi-agent only for segments with a stable gain.

The adoption decision is a gate in code, not an impression.

TypeScript

export function compareArchitectures(
  single: ArchitectureMetrics,
  multi: ArchitectureMetrics
): AdoptionDecision {
  const reasons: string[] = [];
  const qualityGain = multi.successRate - single.successRate;
  const latencyGain = single.medianLatencyMs - multi.medianLatencyMs;
  const costRatio = multi.costPerAcceptedTask / single.costPerAcceptedTask;

  if (qualityGain >= 0.05) {
    reasons.push(`success rate +${(qualityGain * 100).toFixed(1)} pp`);
  }
  if (latencyGain > 0) {
    reasons.push(`median latency -${latencyGain} ms`);
  }
  if (multi.humanReworkMinutes < single.humanReworkMinutes) {
    reasons.push("less human rework");
  }
  if (multi.conflictRate > 0.02) {
    reasons.push("conflict rate exceeds 2%");
  }
  if (costRatio > 2) {
    reasons.push(`cost ratio ${costRatio.toFixed(2)}x`);
  }

  const adopt =
    (qualityGain >= 0.05 || latencyGain > 0) &&
    multi.conflictRate <= 0.02 &&
    costRatio <= 2 &&
    multi.humanReworkMinutes <= single.humanReworkMinutes;

  return { adopt, reasons };
}

And it's checked by tests on a positive and a negative scenario - so an expensive team with conflicts is honestly rejected.

TypeScript

import assert from "node:assert/strict";
import { test } from "node:test";
import { compareArchitectures } from "../src/evaluation.js";

test("adopts multi-agent only when measured gain pays for coordination", () => {
  const decision = compareArchitectures(
    { successRate: 0.75, medianLatencyMs: 6000, costPerAcceptedTask: 1, conflictRate: 0, humanReworkMinutes: 20 },
    { successRate: 0.84, medianLatencyMs: 4200, costPerAcceptedTask: 1.8, conflictRate: 0.01, humanReworkMinutes: 12 }
  );
  assert.equal(decision.adopt, true);
});

test("rejects a costly team with excessive conflicts", () => {
  const decision = compareArchitectures(
    { successRate: 0.8, medianLatencyMs: 5000, costPerAcceptedTask: 1, conflictRate: 0, humanReworkMinutes: 10 },
    { successRate: 0.83, medianLatencyMs: 4600, costPerAcceptedTask: 2.4, conflictRate: 0.08, humanReworkMinutes: 18 }
  );
  assert.equal(decision.adopt, false);
  assert.ok(decision.reasons.some((reason) => reason.includes("conflict")));
});

Anthropic recommends combining code-based, model-based, and human graders; evals give a baseline for latency, token usage, cost per task, and errors, while production monitoring then catches distribution drift but doesn't replace pre-launch evaluation. And choose KPIs so they don't reward noise.

Bad KPI · Why it misleads · Replacement

Number of agent messages - Rewards noise; Accepted outcomes
Tasks started - Doesn't show completion; Success rate by an external gate
Price of one call - Ignores retries and rejected runs; Cost per accepted task
Subjective speed - A parallel UI looks fast; End-to-end wall-clock latency

Don't carry over a vendor benchmark. A research system's internal result doesn't prove a gain on your coding, support, or operations workload. Use it as grounds to test a hypothesis, not as a ready business case.

Usefulness is proven on data. The last step is to roll out autonomy in stages, not with one switch.

Pilot on Real, Expensive Tasks

Links