Define What "Good" Means Before the Prompt
Without a success criterion, prompting turns into a tasting: you change the words, look at one good answer, and don't know whether it got better or you just got lucky. So the first engineering step happens before the prompt - deciding what counts as done in the first place.
A good place to start is a minimal task card: why it exists, what comes in, what goes out, and by which signals to judge success.
use_case: support_policy_answer
user_value: the customer gets an accurate answer without waiting for an agent
inputs: question, language, tenant_id, retrieved chunks
success:
- the answer relies only on the effective policy
- every verifiable claim has a source_id
- when data is missing the model doesn't guess
- p95 latency fits the product budget
failure_cost:
low: an extra clarification
high: an invented price or a promised refund
baseline: FAQ search + canned answersNext, "good" is worth breaking into three dimensions - otherwise it is easy to optimize one and miss another sagging.
- Outcome - did the real state change. The ticket is closed, the right field is extracted, the translation is accepted by the editor. This is what the task exists for at all.
- Behavior - did the model behave correctly. The right tool, route, handoff, clarification, and citation, and did it stop in time. A good outcome with bad behavior is luck, not a system.
- Operational - what it cost. Latency, tokens, tool errors, escalation rate, interruptions, and the cost of one successful result.
And all of this has to be checked on reality, not on invented examples. A dataset starts with reality - with real requests, including the awkward and boundary ones, not just the happy path.
Prompting doesn't start with the prompt. Anthropic requires defining success criteria and a way to verify them empirically before any optimization. OpenAI and Google likewise build their production guidance around datasets, evals, and traces.
Knowledge check
Where does engineering prompting begin?