Chapter 24

Evaluate Outcome and Interaction Separately

The quality of each worker doesn't guarantee the quality of the system. Evals must check the outcome, routing, tool use, cost, conflicts, and stopping. For one test case several graders are useful: a code-based one checks schema, state, and artifacts; a model-based rubric assesses open-ended synthesis; a human reviewer calibrates disputed criteria and finds unexpected failure modes. No single type covers everything.

Layer · Check · Example

Outcome - Did the external state change correctly; A release report is approved only with all evidence
Artifact - Schema, completeness, provenance; A finding contains a source and severity
Trajectory - Routes, tools, attempts, forbidden calls; The reviewer didn't edit the source artifact
Economics - Tokens, calls, wall time, accepted cost; Cost per accepted task didn't exceed the gate
Human - Usefulness and hidden rework; Engineer time to assemble the result

In the companion project the outcome and the order of branches are checked by ordinary tests, not by trust in the supervisor's message.

TypeScript


test("runs independent branches and an evidence-gated review", async () => {
  const report = await new Orchestrator({
    tasks: releaseTasks,
    handlers: createReleaseHandlers(),
    limits,
    maxConcurrency: 3
  }).run("success");

  assert.equal(report.status, "completed");
  assert.equal(report.taskStatus.review, "completed");
  assert.ok(report.artifacts.some((artifact) => artifact.key === "report/release.json"));
  assert.equal(report.evidence.at(-1)?.name, "independent-review");
  assert.equal(report.evidence.at(-1)?.passed, true);
});

test("parallel branches start after plan and before review", async () => {
  const report = await new Orchestrator({
    tasks: releaseTasks,
    handlers: createReleaseHandlers(),
    limits,
    maxConcurrency: 3
  }).run("ordering");
  const completed = report.events
    .filter((record) => record.event.type === "task.completed")
    .map((record) => record.event.type === "task.completed" ? record.event.taskId : "");
  assert.equal(completed[0], "plan");
  assert.equal(completed.at(-1), "review");
  assert.deepEqual(new Set(completed.slice(1, 4)), new Set(["tests", "security", "docs"]));
});

test("retries one transient worker failure within task budget", async () => {
  const report = await new Orchestrator({
    tasks: releaseTasks,
    handlers: createReleaseHandlers(["security:security"]),
    limits,
    maxConcurrency: 3
  }).run("retry");
  assert.equal(report.status, "completed");

Keep the capability and regression suites separate: the capability set holds hard tasks and shows a new topology's potential, the regression set protects already-working scenarios and should pass almost always. On every change of model, prompt, tool description, or routing policy, run both.

Read traces, not just the aggregate score. The average score reports that something got worse; a sampled transcript and trace show why. Without looking at trajectories it's easy to optimize the judge rather than the useful process.

Evaluation is separated from execution. So it survives a change of SDK, the control plane is built independently of the model provider.

Evaluate Outcome and Interaction Separately

Links