Chapter 5

State the Hypothesis as a Gain, Not a Number of Agents

The pilot must answer one question: did the system get better than the single-agent baseline in outcome, latency, and the cost of an accepted result. A bad hypothesis is "four specialists reason more deeply." A good one is "three independent checks will cut median wall-clock time by 25 percent without the conflict rate rising above 2 percent and without doubling cost per accepted release."

Choose metrics so they don't substitute usefulness with activity.

Metric · What it measures · What it doesn't substitute for

Task success rate - The share of accepted outcomes; The beauty of the text and the agent's self-assessment
Cost per accepted task - The full price of a successful result; The price of one model call
Wall-clock latency - Time from input to gate; The sum of durations of parallel calls
Conflict rate - The share of overlapping or rejected writes; The number of messages between agents
Human rework - Time to fix and assemble results; Time spent watching a pretty demonstration

A decision across several metrics is convenient to express as code, not an impression.

TypeScript

export interface ArchitectureMetrics {
  readonly successRate: number;
  readonly medianLatencyMs: number;
  readonly costPerAcceptedTask: number;
  readonly conflictRate: number;
  readonly humanReworkMinutes: number;
}

export interface AdoptionDecision {
  readonly adopt: boolean;
  readonly reasons: readonly string[];
}

export function compareArchitectures(
  single: ArchitectureMetrics,
  multi: ArchitectureMetrics
): AdoptionDecision {
  const reasons: string[] = [];
  const qualityGain = multi.successRate - single.successRate;
  const latencyGain = single.medianLatencyMs - multi.medianLatencyMs;
  const costRatio = multi.costPerAcceptedTask / single.costPerAcceptedTask;

  if (qualityGain >= 0.05) {
    reasons.push(`success rate +${(qualityGain * 100).toFixed(1)} pp`);
  }
  if (latencyGain > 0) {
    reasons.push(`median latency -${latencyGain} ms`);
  }
  if (multi.humanReworkMinutes < single.humanReworkMinutes) {
    reasons.push("less human rework");
  }
  if (multi.conflictRate > 0.02) {
    reasons.push("conflict rate exceeds 2%");
  }
  if (costRatio > 2) {
    reasons.push(`cost ratio ${costRatio.toFixed(2)}x`);
  }

  const adopt =
    (qualityGain >= 0.05 || latencyGain > 0) &&
    multi.conflictRate <= 0.02 &&
    costRatio <= 2 &&
    multi.humanReworkMinutes <= single.humanReworkMinutes;

  return { adopt, reasons };
}

Compare architectures on the same task set, with the same inputs and one external gate. A multi-agent run's result can't be called a success just because the supervisor said "done."

The normal pilot outcome. The single agent may win. That's not a research failure but savings on production complexity. Keep the multi-agent variant only for classes of tasks where it proved an advantage.

The hypothesis is set. If the split is justified, the next choice is the topology - and it's set by where the decision is made.

Knowledge check

What proves the usefulness of the multi-agent variant?

State the Hypothesis as a Gain, Not a Number of Agents

Links