Template

The eval set

The golden case sheet for an AI feature: the input, what a good answer looks like, how you'd grade it, and the real failure it came from.

When you reach for it

Before you ship, change the prompt for, or swap the model behind any AI feature.

1
Input
The exact thing the model receives, captured from a real case rather than invented.
What it asks
What's the real input, word for word, that this case is testing?
Filled in
User message to Sona's budgeting assistant: "why did you tell me I overspent on dining when I only ate out twice this month"
How it goes wrong
Writing a clean, well-formed input because it's easier to reason about, when the case that actually broke in production was messy, half a sentence, mixing English and transliterated Hindi. Golden cases have to include the ugly real ones.
2
Expected behaviour
What a correct answer would do, described concretely enough that two different people would grade it the same way.
What it asks
What should the model's response actually do here, not just what tone should it take?
Filled in
It should check the user's actual dining transactions for the month, name the specific merchants and amounts counted as "dining," and if the category includes something the user wouldn't call dining (a grocery delivery miscategorised), say so rather than defending the total.
How it goes wrong
Writing the expected behaviour as a vibe, like "should be helpful and accurate." That can't be graded consistently by two different people, which means it can't be graded consistently by a model either.
3
Rubric
Turns the expected behaviour into a scored checklist, so grading doesn't depend on who's grading.
What it asks
Break the expected behaviour into 3 to 5 pass or fail checks.
Filled in
1. Names the specific transactions counted as dining. 2. Total matches what's actually in the user's ledger for that category. 3. Does not restate the user's question back as the entire answer. 4. Offers to recategorise if a transaction looks miscategorised.
How it goes wrong
A rubric with one item, "is the answer correct," graded holistically. That's exactly the shape an LLM judge is bad at: a single holistic score rewards a confident, well-written wrong answer over a shorter, hedged right one.
4
The failure it came from
Ties every golden case back to something that actually broke, so the eval set grows from reality instead of imagination.
What it asks
What real incident, ticket, or bad output is this case testing for? Link it.
Filled in
From ticket 4021: user was told they overspent on dining based on a $340 grocery delivery order auto-tagged "dining" because the merchant name contained the word "kitchen." This case checks that the model catches likely miscategorisation instead of trusting the tag.
How it goes wrong
Writing eval cases from imagination, "what might a user ask," instead of from real failures. Imagined cases test what you already thought of. Real failures test what you didn't.

Take the whole thing

Paste into your doc

# Eval set: [feature]

## Case [number]

**Input** (verbatim, including the messy ones):


**Expected behaviour** (concrete enough that two people would agree):


**Rubric** (binary checks, not one score):
1.
2.
3.
4.

**Source of this case** (ticket, transcript, or incident link):


**Last run result**: pass / fail
**Model or prompt version tested**:

Rules for using it

Every case traces back to a real transcript, ticket, or incident. No invented cases in the first pass.
The rubric has multiple binary checks, never one holistic score.
Include the messy, malformed, real-world inputs, not the clean ones you'd write yourself.
Re-run the whole set after every prompt or model change, not just the cases you think are affected.

Draft it with a model

Useful for the first pass. The judgement is still yours.

Prompt

I'm building an eval set for an AI feature. I'll paste a real transcript or ticket where the model got something wrong. Turn it into a golden case: (1) the exact input as it actually occurred, don't clean it up, (2) expected behaviour described concretely enough that two graders would agree, (3) a rubric of 3 to 5 binary pass or fail checks derived from that expected behaviour, not one holistic score. Flag if my expected behaviour description is too vague to grade consistently. Here's the failure: [paste the real transcript, ticket, or bad output]

What to catch

Used later as the grader, an LLM judge tends to prefer longer, more hedged, more agreeable answers, and it can prefer its own writing style over a plainer correct one. Spot check its grading against a human on at least 10 percent of cases, especially the ones it scores as borderline.

Taught in

Level 05, where you build one of these against a real brief.

Open the level

More templates

The eval set

Input

Expected behaviour

Rubric

The failure it came from

Take the whole thing

The PRD that survives contact with engineering

The signal log

The opportunity tree, on one page