Did it actually work?
Unit 5 of 8 · Lesson 4 of 4Index
The verification fix shipped. Completion went up twelve points and everyone wants to celebrate. Then you look at the calendar: it was payday week, and marketing ran a campaign.
Before and after cannot show cause
Payday, a campaign, a holiday. Anything that changed that week could have moved the number.
Half of new users get the fix and half do not, at the same time. Same week, same payday, same campaign. The only difference is the fix.
A few dozen signups can swing either way by luck. Work out how many you need before you start.
Write down the result you would need to see, before you look. Otherwise every number starts to look like a win.
How Microsoft Bing handled it
- The situation
An engineer at Bing suggested a small change to how ad headlines were displayed. It sat in the backlog for months because nobody thought it mattered.
- What they did
When someone finally ran it as a controlled experiment, revenue rose by about 12 percent, so much that the team first suspected a bug. The result held.
- Why it matters to you
Opinions about what will work are often wrong in both directions. A fair split is how you find out.
Doing it with AI, and where it breaks
Ask a model to attack your test design: what could make the two groups differ other than the fix, and what is the sample size formula for the effect you care about.
Models will happily compute a sample size from numbers they made up. Give it your baseline and the smallest change worth acting on, and check the result with a calculator you trust.
Here is my A/B test plan: who is split and how, the primary metric, the counter metric, and the run length. List every way the two groups could differ other than the change being tested. Then write the sample size formula per group for a baseline rate and a minimum detectable effect, without calculating it.
Build it
A one page test plan for the verification fix, with the win written down before launch.
Google SheetsA sample size calculator
The worked solution
Stuck, or want to compare?
There is a worked solution. Try the build first. Reading it before you have attempted anything is the fastest way to learn nothing.
Checkpoint
If you did the build, these take two minutes. If you cannot answer one of them, that is the part to go back to.
Moving on marks this lesson complete. Finish the build first, it is the part that counts.