The peeking problem

What optional stopping breaks

A fixed-horizon significance test makes one promise: choose the sample size in advance, look once when it is reached, and under the null hypothesis the test crosses its threshold with probability α — 5% at the conventional 0.05 level. Every additional look is another draw against that threshold, and stopping at the first flattering crossing keeps exactly the errors. Johari, Koomen, Pekelis and Walsh documented the practice at industrial scale in 2017: commercial A/B platforms displayed continuously updating significance figures, users stopped experiments on them, and realized false-positive rates ran far above the nominal level.

The inflation is structural, not a matter of insufficient willpower. Under a true null, a test statistic recomputed after every new observation wanders; the law of the iterated logarithm guarantees that, monitored long enough, it eventually crosses any fixed threshold. A procedure that stops on the first crossing therefore converts noise into a declared winner with probability that grows toward certainty as monitoring continues.

In cold outbound, the peek is the workflow

Cold-email dashboards update with every send and every reply, and the decisions that matter — which angle receives tomorrow’s rationed sends, which sequence gets paused — are made mid-flight rather than at a preplanned n. A fixed-n test is the wrong instrument for this setting by construction: it is valid only at a look nobody actually waits for.

One major sequencing platform’s published statistics guidance (Instantly, 2026) illustrates the current division of labor: it advises targeting 95% confidence with at least 1,000 recipients per variant and running significance through external calculators, while the dashboard itself carries no significance gate. The stopping decision — the step on which the guarantee depends — is left with the person watching the numbers move.

The repair is a different guarantee, not more discipline

Sequential analysis replaces the single-look promise with a time-uniform one. Johari, Pekelis and Walsh formalized always-valid inference for this setting in Operations Research 70(3), 2022: p-values and confidence intervals constructed so the stated error rate holds at every stopping time simultaneously, whatever rule — impatience included — decides when to stop. Under such an instrument, checking daily costs nothing, because every look was already paid for in the construction.

The price appears somewhere honest instead. Time-uniform intervals are wider than fixed-horizon intervals at the same sample size, so the floors for detecting small effects rise. The trade is visible rather than hidden: the analyst gives up some narrowness to keep a guarantee that survives the way people actually behave around live numbers.

How RevenueOS uses this

RevenueOS treats the peeking problem as a design constraint, not a training problem. Its proof engine reports anytime-valid confidence sequences at α = 0.1 — 90% coverage at every stopping time — so a partner who reads the band every morning spends none of the error budget by looking. A claim closes only when the band excludes zero, and it stays open, on the record, until it does; null results publish on the same ledger as wins.

The cost of peek-proof inference is priced upfront rather than discovered mid-pilot: the minimum-n table on the RevenueOS methods page (generated 2026-07-03 from 200 seeded Monte-Carlo runs per grid point) puts the floor for qualifying a 1.5× arm lift on a 2% positive-reply base at 86,496 randomized rows under its stated design assumptions. RevenueOS is at design-partner stage as of July 2026 and claims no customer results; the guarantee on offer is the statistical one.

Related terms

Anytime-valid confidence sequence · Minimum detectable effect · Positive reply rate