RU
Get in touch
Chapter 17

Turn Tests Into Feedback, Not a Final Ritual

The cycle is reliable when the agent sees a red baseline, changes the code, gets a green focused test, and then checks a wider set. This distinguishes fixing the cause from a random green result. In the companion project the whole cycle runs through an end-to-end demo: the fixture must reproduce the defect before the patch, or the demonstration fails.

TypeScript
    validatePlan(repositoryBefore.root, plan, policy);
    const baseline = await runCommand(temporaryRoot, testCommand, policy);
    if (baseline.code === 0) {
      throw new Error("The demonstration fixture must reproduce the bug before the patch");
    }
    const operation: TextReplacement = {
      path: "src/discount.js",
      expected: "  return subtotal * (1 - percent);",
      replacement: "  return subtotal * (1 - percent / 100);"
    };
    await applyTextReplacement(repositoryBefore.root, operation);
    const verification = await runCommand(temporaryRoot, testCommand, policy);
    const repositoryAfter = await indexRepository(temporaryRoot, { maxFiles: 100, maxBytesPerFile: 128_000 });
    const snapshotAfter = await snapshotRepository(repositoryAfter);
    const changed = changedFiles(snapshotBefore, snapshotAfter);
    const review = reviewChange(contract, changed, [verification]);
    if (!review.passed) {
      throw new Error(`Review gate failed: ${review.findings.join("; ")}`);
    }
    const fixedSource = await readFile(join(temporaryRoot, "src", "discount.js"), "utf8");
    const output = {
      repositoryFiles: repositoryBefore.entries.length,
      promptSections: ["Goal", "Context", "Allowed write scope", "Constraints", "Procedure", "Done when", "Final report"],
      planSteps: plan.steps.map((step) => step.id),
      baselineExitCode: baseline.code,
      finalExitCode: verification.code,
      changedFiles: review.changedFiles,
      reviewPassed: review.passed,
      selectedSurfaceForRepeatableWorkflow: chooseGuidanceSurface({
        repeats: true,
        isWorkflow: true,
        mustBeDeterministic: false,
        isObjectiveMergeGate: false
      }),
      fixedLinePresent: fixedSource.includes("percent / 100")
    };
    process.stdout.write(`${JSON.stringify(output, null, 2)}\n`);
    if (!prompt.includes("# Done when")) {
      throw new Error("Prompt contract is incomplete");

Behind the code stand five signals, and skipping any makes "green" unreliable: reproduction (the test fails before the patch for the expected reason), focused verification (the same test passes after), regression envelope (neighboring tests, typecheck, and build caught no side effect), diff review (only the expected files changed), and evidence report (commands, exit codes, and meaningful output lines saved).

How to react to a check failure - and how not to - is best kept in view.

Check failure · Wrong reaction · Correct next step

  • The test doesn't reproduce the defect - Change the code anyway; Refine the fixture, input, and expected behavior
  • Unrelated tests are already red - Declare them a result of the patch; Fix the baseline and separate the new failures
  • The test hangs - Wait without a limit; Timeout, log, and analyze the resource or deadlock
  • Output is truncated - Treat code 0 as enough for diagnosis; Save the artifact or narrow the command
  • The UI test is green - Don't look at the result; Check the screenshot and browser console

The evidence is set in the prompt itself: Claude Code advises giving the agent a check it can run, Codex recommends stating Done when, Cursor includes verification steps in the plan. This is the common foundation of the three products.

Don't weaken the oracle. Removing an assertion, a broad snapshot update, and raising a timeout can make the pipeline green without fixing behavior. Any test change must explain why the prior expectation was wrong - otherwise you're fixing the indicator, not the defect.

Tests give a signal to the executor itself. But final readiness must be checked by someone else - in a fresh context.

Links