DECISION SYSTEMS TOOLKIT · EXAMPLE 01

Does the add-on actually reduce churn?

Three add-on services. The same test. Three different decisions.

Worked example · proportion comparison Retention measured at a fixed 12-month point

Retention after being offered the add-on — not after buying it. Solid = offered · Dashed = control

A · priority support

n per arm	2,400
12-mo retention	70.0% / 73.0%
Difference	−3.0 pp
95% CI	−5.6 to −0.4
p-value	0.021

Don't roll it out Real, and the entire interval sits below zero.

B · guided onboarding

n per arm	1,800
12-mo retention	79.0% / 73.0%
Difference	+6.0 pp
95% CI	+3.2 to +8.8
p-value	< 0.001

Roll it out Even the low end clears the switching cost.

C · extended warranty

n per arm	700
12-mo retention	75.4% / 73.0%
Difference	+2.4 pp
95% CI	−2.2 to +7.0
p-value	0.30

Not a retention decision The gap looks real, but the interval crosses zero. Decide this one on margin.

A/B TEST TOOLKIT · WHICH TAB

Mean comparison

Proportion comparison

Sample size

Statistical power

Retained-or-not at a fixed month is a proportion, so all three calls above came out of the second tab. A curve isn't a test — a rate at one point is. Use Sample size before you start, to find out whether the difference you care about is even detectable at the traffic you have.

gracege.com/tools/ab-test-calculator →

Figures in this example are illustrative and generated for teaching purposes. They do not describe any real company.
This tool is provided for reference only and does not constitute professional statistical or business advice.