Skip to content
PMcademy
Template

The experiment brief

The hypothesis, the metric, the sample, and the rule for stopping, all written before the test runs, not after.

When you reach for it

Before any A/B test or pilot goes live, so the goalposts can't move once the data shows up.

  1. 1

    Hypothesis

    States what you believe and why, so success has a definition before there's data to argue about.

    What it asks

    If we do X, we believe Y will happen, because Z. Write it in that shape.

    Filled in

    If we add a progress bar to onboarding that stays visibly incomplete until a bank is linked, we believe bank-linking within 24 hours will rise, because Priya's tickets show users don't realise linking is required. They think the app works without it.

    How it goes wrong

    Writing the hypothesis after glancing at early results, so it quietly becomes whatever the data already shows. Timestamp the brief and don't touch the hypothesis line again.

  2. 2

    Metric and guardrail

    The one number that decides the test, plus the number you're watching to confirm nothing else broke.

    What it asks

    What's the single primary metric, and what's the guardrail metric that must not get worse?

    Filled in

    Primary: percent of new signups who link a bank within 24 hours. Guardrail: overall onboarding completion must not drop more than 2 points, in case the progress bar makes people feel pressured and quit instead.

    How it goes wrong

    Picking a metric only the new feature could move, so there's no way to tell if you helped or just shifted behaviour, and no guardrail, so a quiet regression elsewhere goes unnoticed.

  3. 3

    Sample and duration

    Decides in advance how long you'll wait and how many people you need, so a promising trend on day 3 doesn't get called early.

    What it asks

    What percent of traffic, and for how long, based on your normal weekly signup volume?

    Filled in

    50 percent of new signups, for 14 days. At roughly 800 new signups a week, that's about 1,600 people in the test, enough to detect a 5 point shift in linking rate at the confidence we use for a decision this size.

    How it goes wrong

    Peeking daily and stopping the moment the number looks good. Write the stop date in the brief before launch and hold it, even when day 4 looks great.

  4. 4

    Stopping rule

    What ends the test besides the calendar, especially if something goes wrong.

    What it asks

    What result, at what point, ends this early, in either direction?

    Filled in

    If the guardrail drops more than 2 points at any daily check, stop immediately regardless of the primary metric. If the primary metric shows no movement either way by day 10, let it run to day 14 rather than extending further.

    How it goes wrong

    Having no rule for stopping early on harm, so a broken variant runs the full two weeks because nobody wants to be the one who calls it.

  5. 5

    What would change your mind

    Names the result that would make you walk away from this idea, written before you're attached to it.

    What it asks

    What result means you don't ship this, rather than iterating on it?

    Filled in

    If linking rate within 24 hours moves less than 2 points either way and the guardrail is flat, we conclude the problem isn't awareness, and we don't ship the progress bar or try a louder version of it.

    How it goes wrong

    Only ever writing down what would confirm the idea, so any result gets read as "promising, let's iterate," and the idea never actually dies no matter what the data says.

Take the whole thing

Paste into your doc
# Experiment brief

## Hypothesis
If we [change], we believe [metric] will [move], because [evidence].


## Metric and guardrail
Primary metric:
Guardrail metric (must not worsen):


## Sample and duration
Percent of traffic:
Duration:
Expected sample size:


## Stopping rule
Stop early if:
Otherwise, run to:


## What would change our mind
Result that means we don't ship this:
More templates