Anytime-valid confidence sequence

Time-uniform coverage, stated precisely

A classical confidence interval makes one promise: fix the sample size n in advance, compute the interval once, act on nothing in between, and it contains the true value with probability at least 1 − α. Every clause is load-bearing. Monitor the estimate as data arrive and stop when it looks favorable, and the realized error rate can run well above the α on the label — the interval was never built to survive a data-dependent stopping time.

A confidence sequence replaces that per-n promise with a single trajectory-wide one: with probability at least 1 − α, the band contains the truth at every observation count t simultaneously. Because coverage is uniform over time, any stopping rule is legal — stop at the first crossing, run longer, or read it every morning. Howard, Ramdas, McAuliffe and Sekhon put this on general footing in 2021 with nonparametric, nonasymptotic constructions that hold over the whole trajectory. The price is width: at any given n, an anytime-valid band is wider than the classical interval, and it narrows only as evidence accrues. That is the honest trade — width paid up front in exchange for the right to look.

The betting construction

The variant that executes in practice comes from Waudby-Smith and Ramdas (2024): estimate the mean of bounded observations by betting against it. For each candidate mean m, a hedged pair of capital processes places wagers — one leg betting the truth sits above m, one betting below — with bet sizes chosen only from data already seen. If m really is the true mean, wealth cannot grow fast: by Ville’s inequality the capital exceeds 1/α with probability at most α. So a candidate is eliminated from the confidence set exactly when the capital wagered against it exceeds 1/α — at α = 0.1, a tenfold gain. The candidates still standing at time t form the interval, and the interval is valid at every t at once.

Early on, almost every candidate survives and the band is wide; as observations accumulate, capital compounds against the false candidates and the set tightens. Nothing in the elimination rule references a planned sample size — which is precisely why reading the band mid-stream costs nothing.

How RevenueOS uses this

RevenueOS runs this betting construction on its causal-lift claims at α = 0.1 — a 90% anytime-valid band on the difference a message program caused, computed from per-lead treatment-versus-holdout scores rescaled into [0, 1]. A candidate lift is eliminated when the hedged capital betting against it exceeds 1/α = 10; the band the survivors form starts wide, narrows as leads resolve, and keeps its coverage however often an agency, a client, or RevenueOS itself refreshes the page. A claim closes only when the band excludes zero, and it stays open — honestly — until it does; the /methods page carries that commitment in writing.

The time-uniform theory of Howard, Ramdas, McAuliffe and Sekhon (2021) is the anchor; the betting engine of Waudby-Smith and Ramdas (2024) is the executing variant. What the design costs in width, RevenueOS prices openly: the minimum-n table on /methods (generated 2026-07-03 from 200 seeded Monte-Carlo runs per grid point, at the same α = 0.1) states how many randomized leads a claim needs before a band at a realistic effect size τ can close at all.

Related terms

The peeking problem · Causal lift · Minimum detectable effect