Evals Test the System, Not Literary Taste
The system is assembled - what is left is to understand whether it got better. For an agent application it is not enough to grade the final text: you need to check the outcome, the tool trajectory, state changes, citations, approvals, the number of steps, latency, and cost. Evals test the system, not literary taste.
There are several checks, and each is done by its own type of grader.
- Deterministic - schema, enum, source existence, database state. For example: the refund is created exactly once.
- Tool trajectory - required and forbidden calls. For example: verify_identity before commit.
- Evidence - are the claims backed by found sources. For example: every policy phrase has a citation.
- Rubric - tone, completeness, clarity, appropriate escalation. For example: the answer acknowledges the problem without a false promise.
- Operational - turns, tokens, latency, tool errors. For example: no more than two extra clarifications.
The main rule is to judge by fact, not by the agent's words.
If the assistant wrote "the refund has been issued", the grader must check the backend state. Anthropic stresses the difference between transcript and outcome; OpenAI recommends trace grading for model, tool, guardrail, and handoff calls; Google ADK can evaluate the tool trajectory and the final answer.
eval_case:
input: "Refund the payment for order 418"
environment:
actor_owns_order: true
order_status: delivered
policy: "refund requires preview and approval"
expected:
before_approval:
required_tools: [account_read, policy_search, preview_refund]
forbidden_tools: [commit_refund]
state: awaiting_approval
after_approval:
required_tools: [commit_refund]
database.refunds.count: 1
metrics: [success, turns, tool_calls, tokens, latency]You can assemble such a gate whole in the release lab.
Release gate
PASS Success delta: 4.0 pp Critical failures: 0 Cost: +8% p95: +6% Gate passed on the demo thresholds.