A metric that cannot be gamed
You propose increasing weekly active users by 20 percent. Your engineer says, deadpan, “I can do that by Thursday with a push notification saying your account has a problem”. He is not wrong.
Every metric has an exploit. Find yours before someone else does.
The number and its guardrail
- Primary: Faster completion
- Counter: Error rate
- Exploit: Skipping checks
Before you commit to a number, spend five minutes actively trying to cheat it. If you can think of a way, so can a team under quarterly pressure, and they will, without meaning to.
Then pair it with a counter metric. Speed with error rate. Engagement with unsubscribes. Conversion with refunds. The counter metric is what stops the primary metric from eating the product.
And define it precisely enough that two analysts would compute it identically. “Active user” is not a definition. “A user who completed at least one transaction categorisation in the last seven days” is.
A metric is a compact definition of what you are trying to change. Its counter-metric protects you from celebrating a result that makes the overall product worse.
Use your chosen course project throughout. The additional examples below are fictional practice cases; transfer the method to your own evidence.
Write the measurement contract
Specify the event, eligible population, counting unit, time window, and exclusions. “Activation improved” is ambiguous until two people can independently calculate the same rate. Decide how duplicate events, test accounts, and incomplete observation windows are handled before looking at the result.
Choose a counter that can catch harm
Pair the desired change with a plausible adverse effect. Faster support responses need a quality check; more notifications need a measure of unwanted interruption. A counter-metric is useful when it could change the launch decision, not merely decorate the dashboard. Explain what you would investigate or stop if it worsens.
Keep a decision attached
Name the owner and review date. Separate normal variation, measurement errors, and product effects. Do not treat an arbitrary threshold as scientifically established. For a small project, a manual review of failures may be more useful than an impressive-looking statistical claim based on very few observations.
A useful win has boundaries
- Outcome: What improves?
- Counter: What could worsen?
- Decision: Review both together
How Facebook handled it
Facebook's growth team needed one number that predicted whether a new user would stay.
They found that users who connected with seven friends within ten days retained dramatically better, and made that the growth team's single target. It worked because it was a leading indicator of real value rather than a proxy for it. You cannot fake having seven friends who post things you care about.
The best metrics measure the moment a user gets value, not the moment they touch your product. Sessions can be inflated. Value delivered usually cannot.
Doing it with AI, and where it breaks
Ask a model for ten candidate metrics, then for each one, how a cynical team would hit the target without helping a single user.
It is bad at knowing which exploit is likely at your company. It will rank a theoretical abuse above the boring one that actually happens, like the team quietly redefining what counts as active.
Your AI workbench
Start with your own notes or clearly labelled practice data. Remove private details before sharing. Replace the placeholders, run the prompt in your chosen AI tool, and keep the output beside its source.
Audit this metric definition [paste] for ambiguous population, numerator, denominator, window, deduplication, and exclusions. Propose a counter-metric tied to a concrete failure mode. Return a sample calculation using explicitly fictional rows and list decisions the metric cannot support.
Before you use the output
- Two people could reproduce the calculation
- The counter detects a plausible harm
- Thresholds are identified as proposed or established
Stuck? Try this next
Ask AI to create an edge-case table: duplicate, late event, cancelled account, and missing timestamp. Decide the treatment yourself and add it to the definition.
Keep a brief AI log: input used, useful output, what you checked, and what you rejected. The decision remains yours.
Build it
One primary metric, one counter metric, and a written exploit analysis for both.
The worked solution
Stuck, or want to compare?
There is a worked solution. Try the build first. Reading it before you have attempted anything is the fastest way to learn nothing.
Checkpoint
If you did the build, these take two minutes. If you cannot answer one of them, that is the part to go back to.
Moving on marks this lesson complete. Finish the build first, it is the part that counts.