← All decks

Experiments & Metrics Patterns

Measure honestly and learn what actually works.

Methods for measuring a product or service honestly and learning what actually works. For product teams, analysts and growth leads, the cards cover defining success, collecting data you can trust, designing and reading A/B tests, avoiding classic statistical traps, reasoning about cause without randomising, building dashboards people act on and making testing a team habit.

56 cards · 8 suits · 5 recipes · 2 missions

What's inside

• Define7Is it clear what success means here?
• Instrument7Can you trust the data you collect?
• Design7Will this test give a clear answer?
• Read7Are you reading the result without fooling yourself?
• Traps7Have you ruled out the classic traps?
• Cause7Can you show cause without a randomised test?
• Show7Does your dashboard lead anyone to act?
• Habit7Is the team learning from every test, even losses?

A few cards

Define

One North Star

Choose one number that rises only when customers get real value, and let every team aim at it.

Design

Falsifiable Hypothesis

State what you expect, for whom and by how much, so the test is able to prove you wrong.

Traps

Peeking

Look at a running test as often as you like, but don't stop it the moment it first crosses the line.

One card in full

Show

Question-First Dashboard

Build the dashboard from the questions, not the data.

Most dashboards grow by accretion: someone asks for a chart, it gets added, nobody removes it. Six months later there are forty tiles and no one can say which decision any of them informs.

How to use it

1. Name the audience: one role or team, not "everyone".
2. Ask them for the five to ten questions they need answered each week, in their own words, and the decision each one feeds.
3. For each question, choose the one chart or number that answers it, with its comparison (Compared To What).
4. Arrange the page in the order people ask the questions, broadest first (Overview First).
5. Delete every chart that answers no listed question, or move it to an exploration page.
6. After a quarter, check what people actually open, and retire what nobody does.

Example

A hypothetical school leadership team replaces a 30-chart attendance dashboard with one page built on four questions: which year groups are slipping, which pupils have crossed the persistent-absence line, did last month's intervention help, and what needs action this week.

Watch out

• Asking "what data do you want?" gets a wish list. Ask "what will you decide?" instead.
• Parking one-off analyses on the dashboard. They belong in a report.

Then make it a habit with a Weekly Metric Review.

Recipes

• Dashboard Rescue: Rebuild the dashboard around the questions people actually ask, compute each number once, put a comparison and a normal range beside it, mark known events, and review it weekly.
• Did The Launch Work?: Rule out the ordinary explanations, draw what causes what, allow for bounce-back and self-selection, then estimate the effect against a comparison group or a built twin.
• Launch A Clean Test: State a hypothesis that can fail, choose the metric that decides it, randomise and size the test properly, and write what you will do with each outcome before it starts.
• Metrics From Scratch: Pick one number that tracks customer value, break it into team-sized inputs, check the fast ones really predict it, fence off what must not suffer, and pin every metric and event down in writing.
• Read A Result Honestly: Check the split and anything too good to be true first, then read the interval, its practical size, the number of comparisons, how the effect changes over time and what a flat result rules out.

Missions

• Audit a result: from a test readout or a chart someone is excited about to a verdict on whether it holds, with the checks run (split, peeking, multiple comparisons, novelty, mix shift, cause) and the claim the evidence supports.
• Design an experiment: from "let's just A/B test it" to a one-page brief with hypothesis, primary and guardrail metrics, randomisation unit, minimum detectable effect, sample size, duration and the decision rule, written before launch.

Related decks

Business Finance Patterns

Read the numbers behind every business decision.

48 cards

Pricing Patterns

Set prices that reflect what buyers value.

49 cards

Economics Patterns

See the trade-offs and incentives behind everyday choices.

56 cards

← All decks