Sales went up the week of your mailing. Was it the mailing? The weather? A competitor’s bad week? Without a comparison group you can’t know. PizzaSMARTS built the comparison into every campaign with one simple rule: odd and even house numbers.
From the 1997 manual
“By applying our marketing to only the odd section of a mailing, we have a ready made control group for comparison. For example, we could direct a mailing to our odd numbered non-customers. If sales from the odd numbered group go up by 10% and at the same time sales from our even numbered non-customers go down by 5%. We can attribute a 15% point spread to the marketing effort.”
Mailing Quantity Report, Executing Plan
Why it works
Odd and even addresses are mixed together on every street. Same neighborhoods, same weather, same competitors, same week. The only difference between the two groups is your marketing. So whatever the odd houses did that the even houses didn’t is the effect of the campaign, and nothing else.
The manual applied the same test to door hangers: hang the odd-numbered houses and skip the even ones. If the odd group adds 15 new customers while the even group loses 5, “you soon realize that doorhanging really does make a difference that is worth the effort.”
Today this is called a holdout or control group, and the largest direct marketers in the world run their campaigns this way. Most pizzerias still don’t.
How to do it today
- Pick the target group. Fading customers, first-timers, a neighborhood of non-customers: whoever the campaign is for.
- Split it at random. For mail and door hangers, odd and even house numbers still work perfectly. For email and text lists, split the list at random. Half and half gives the clearest answer; a smaller holdout works if you can’t bear to leave half your customers out.
- Market to one half only. Same offer, same timing, nothing to the other half.
- Compare the change, not the totals. After the campaign, compare the two groups on the number the campaign was meant to move: orders, new customers, rescue rate.
- Work out the cost per result. Divide what the campaign cost by the extra customers it produced. That number lets you compare campaigns against each other.
Worked example (hypothetical numbers)
You send a win-back offer to 200 fading customers with odd house numbers and nothing to the 200 with even numbers. Next month 46 of the odd group order again, and 24 of the even group come back on their own. The campaign produced 22 extra customers, not 46. If it cost $110 to send, that’s $5 per customer won back.
What to measure
The difference between the marketed group and the held-out group, on the one number the campaign was meant to move. Then the campaign’s cost divided by that difference.
Holdouts tell you whether a campaign worked. Before you spend on a big campaign at all, you can find out which offer will work best: probe first.