A/B testing is the most borrowed ritual in marketing, and the only one you can refute with a calculator.
Here’s the arithmetic nobody runs before installing the testing tool. Your landing page converts at 3%. You want to detect a 10% improvement, to 3.3%, which would be a genuinely good result. At standard statistical settings, the kind every testing tool defaults to, you need roughly fifty thousand visitors per variant. A hundred thousand visitors total, to detect a result that good, and most changes aren’t that good.
Now bring your traffic. At 10,000 visitors a month, that test runs for ten months. During those ten months: seasonality shifts, you change the pricing, a competitor launches, your traffic mix moves from a launch spike to organic, and you, being human, peek at the dashboard weekly and stop the test the first time it flatters you. The sample was poisoned by month two. The test will never finish. The arithmetic never allowed it to.
Booking and Google run thousands of experiments because their preconditions exist: millions of sessions, where fifty thousand per variant is an afternoon. The ritual is rational where it was born. Copied to a small product, it produces something worse than no data: false data with a confidence interval, decisions feeling scientific at sample sizes where the dashboard’s wiggle is weather.
That’s the second half nobody tells you: at small samples, randomness dominates. The 30% Tuesday jump means nothing. The variant that’s “winning” at n=400 is a coin flip wearing a chart. You weren’t reading signal. You were reading tea leaves with error bars.
So what replaces testing at your scale? Deciding. Ship the version you believe in, fully, to everyone. Watch the numbers that can’t lie at any sample size: revenue, refunds, support emails saying “I don’t get it.” Talk to ten customers; at your size, ten is a meaningful fraction of everybody. Opinionated changes plus honest listening outperforms simulated science, and it ships ten months sooner.
Run the calculator before you ever run a test. It’s the one experiment that finishes instantly.