What this lesson covers
How do I know a change worked, and wasn't just luck?
- Hypotheses and test design
- A/B and multivariate tests
- Sample size and significance in plain language
- Testing with small numbers: before-and-after periods, holdout weeks, geographic splits and qualitative signals
- Conversion rate optimization (CRO)
- Learning from failed tests
Why this matters now
Placeholder for the lesson's opening story: a vivid, real moment where a marketer or small-business owner meets this exact problem.
You can now
Design a test, read the result honestly and learn something even when the variant loses.
For your own business or employer: Improve results step by step without fooling yourself with luck.
Industry case studies
Industry Case
Tepary Kitchen
Local restaurant · Local and offline
Case update 8.2
Module 8 of 8 in this case
Was it the offer or the weather?
In October, Tuesdays come alive. Covers jump from about 70 to about 95 a night. Miguel is ready to call the campaign a triumph.
Rosa isn't so sure. October is when the heat breaks and the winter residents come back. Maybe Tuesdays would have filled up anyway.
That's the problem with small numbers. A restaurant gets one Tuesday a week, not a million website visitors, and the weather, a football game or a school holiday can swing a single night by twenty guests. A classic A/B test can't split one dining room in two.
So they design a test that fits a small business: holdout weeks. For eight weeks, the Tuesday ads and club texts run every other week. Everything else stays the same: same menu, same staff, same $26 table.
The results (illustrative): Tuesdays with the campaign averaged 101 covers. Tuesdays without it averaged 84. The same eight Tuesdays last year averaged 72.
Now the story is honest. Cooler weather and returning residents explain about 12 extra guests a night. The campaign explains about 17 more on top of that. It's working, just not as dramatically as the first glance suggested.
The servers fill in the rest. When they ask what brought guests in, the answers are “I got the text” and “my neighbor posted about it.” The numbers say how much. The guests say why.
Artifact added
Test log: holdout weeks, a year-over-year baseline and what the servers heard
Industry Case
Bajada Plumbing & Drain
Local home service · Demand capture
Case update 8.2
Module 8 of 8 in this case
Before and after the night line
Bajada can't split its customers in half and leave one half on voicemail. So Carmen compares before and after: after-hours jobs booked in the eight weeks before the answering service started, against the eight weeks after.
Before: about 6 a month, mostly next-morning callbacks. After: about 31. And because summer brings more emergencies anyway, she checks the same weeks last year too: about 7. The season didn't make the difference. Answering the phone did.
The landing page test follows the same rule for small numbers: test big changes, not button colors. The page with the call button at the top gets about 9 calls per 100 visitors. The old page got about 4.
Artifact added
Test log: a before-and-after comparison with last year as the baseline, and a landing page test
Industry Case
Ocotillo & Ash
DTC fashion · Demand creation
Case update 8.2
Module 8 of 8 in this case
Decide what “won” means first
Lena has enough traffic for a real A/B test. Half of her visitors see the new product page with the fit guide and size quiz. The other half see the old page. The test runs four full weeks.
Before it starts, she writes down the one metric that decides it: returns, not clicks.
The conversion rate barely moves: 2.4% against 2.2%, too close to call. But returns from the new page are 13% against 21% from the old one. That difference is large enough, and steady enough week after week, to trust. The fit guide goes on every product page.
Artifact added
Test log: the fit guide A/B test, measured by returns
Listen
Podcast · Move at the Speed of Culture
When Data Says Yes but Culture Says No
Listen for what to do when a test result and your judgment disagree, and how a better-designed test could settle it.
More from Move at the Speed of Culture →
Listen online: moveatthespeedofculture.com/when-data-says-yes-but-culture-says-no
Key terms
- A/B test: An experiment that shows two versions, A (the control) and B (the variant), to similar groups at the same time and compares results.
- Before-and-after test: Comparing results from a period before a change with a period after it. It is easy to run but can be fooled by seasonality and other changes.
- Conversion rate: The percentage of visitors or recipients who complete a desired action.
- Conversion rate optimization (CRO): The systematic process of increasing the percentage of visitors who convert, through research, hypotheses and testing.
- Failed test: A test in which the variant didn't beat the control. It still produces learning about what customers value.
- Geographic split test: A test that runs a change in some locations but not in similar others, then compares results. Also called a geo test.
- Holdout: A group or period deliberately left out of a campaign or change, so you can compare it with the group that got it. A holdout week pauses a campaign for a week to see what happens.
- Hypothesis: A testable prediction of what a change will do and why, such as "Adding reviews near the button will raise bookings because visitors need reassurance."
- Multivariate test: An experiment that tests several elements, and combinations of them, at once. It needs much more traffic than an A/B test.
- Qualitative signals: Non-numeric evidence, such as customer comments, call recordings, session recordings and survey answers, that helps explain why people behave as they do.
- Sample size: The number of visitors or people in a test. Too small a sample makes results unreliable.
- Statistical significance: A measure of how unlikely it is that a test result happened by chance alone. It doesn't tell you how big or important the effect is.
- Test design: Planning a test: what to change, what to measure, who sees each version, how long it runs and what result counts as a win.