Methods · revised 2026-07-11 · every figure carries its source
The math, in writing
Why can’t a dashboard tell you which email angle works?
Because outbound metrics are checked continuously. A classical significance test assumes you look once, at a preplanned sample size; a dashboard is looked at every day. Checking a running experiment until it looks good — the peeking problem — inflates false positives far past the promised rate, so an angle that merely got lucky for a week gets scaled as a winner.
What is an anytime-valid confidence sequence?
A confidence sequence is a confidence interval you may legally read at any time. RevenueOS uses anytime-valid sequences at α = 0.1: a band that starts wide, narrows as evidence accrues, and keeps its 90% coverage guarantee no matter how often anyone checks it. A claim closes only when the band excludes zero — and stays open, honestly, until it does.
How many leads does a claim need before it closes?
The floors below come from simulating the shipped proof engine — two hundred Monte-Carlo runs per grid point — never from a textbook power formula. Panel A prices program incrementality against a never-mailed holdout; Panel B prices arm-versus-arm lift, where small effects on small bases are honestly expensive. Rows count leads randomized into the proof and holdout cells, not emails sent.
Generated 2026-07-03 · α = 0.1 · design assumed: 10% holdout / 10% proof cell (the 25%/25% design accrues rows 3.0× faster at the same sends budget). “≥” marks a lower bound where the floor never crossed inside the simulated grid.
| Cell | Rung | Planned rates (holdout → proof) | τ | Rows n (w=τ, median [p25–p75]) | Rows n (w=2τ, bare) | 3K sends/mo | 5K sends/mo | 10K sends/mo | 20K sends/mo |
|---|---|---|---|---|---|---|---|---|---|
| A1 | positive-reply | 0.1% → 0.64% (conservative (Sales.co "1 in 157" ≈ 0.64%/lead — see sources note)) | 0.54% | 21,928 [21,928–21,928] | 5,484 | 164 | 98.7 | 49.3 | 24.7 |
| A2 | positive-reply | 0.1% → 1.2% (mid (1.2%/lead)) | 1.1% | 9,260 [9,260–9,260] | 2,316 | 69.4 | 41.7 | 20.8 | 10.4 |
| A3 | positive-reply | 0.1% → 2.2% (healthy (2.2%/lead)) | 2.1% | 4,476 [4,476–4,476] | 1,120 | 33.6 | 20.1 | 10.1 | 5.0 |
| A4 | meeting | 0.05% → 0.48% (mid program × 40% booking (0.48%/lead)) | 0.43% | 24,776 [24,776–24,776] | 6,196 | 186 | 111 | 55.7 | 27.9 |
| A5 | meeting | 0.05% → 0.88% (healthy program × 40% booking (0.88%/lead)) | 0.83% | 11,648 [11,648–11,648] | 2,912 | 87.4 | 52.4 | 26.2 | 13.1 |
| Cell | Rung | Planned rates (holdout → proof) | τ | Rows n (w=τ, median [p25–p75]) | Rows n (w=2τ, bare) | 3K sends/mo | 5K sends/mo | 10K sends/mo | 20K sends/mo |
|---|---|---|---|---|---|---|---|---|---|
| B1 | positive-reply | 2% → 2.5% (1.25× lift on 2% base) | 0.5% | ≥250,000 [≥250,000–≥250,000] | 77,904 | ≥1,875 | ≥1,125 | ≥563 | ≥281 |
| B2 | positive-reply | 2% → 3% (1.5× lift on 2% base) | 1% | 86,496 [86,496–86,496] | 10,812 | 649 | 389 | 195 | 97.3 |
| B3 | positive-reply | 2% → 4% (2× lift on 2% base) | 2% | 12,944 [12,944–25,888] | 3,236 | 97.1 | 58.2 | 29.1 | 14.6 |
| B4 | meeting | 0.8% → 1% (1.25× lift on 0.8% base (2% reply × 40%)) | 0.2% | ≥250,000 [≥250,000–≥250,000] | 194,784 | ≥1,875 | ≥1,125 | ≥563 | ≥281 |
| B5 | meeting | 0.8% → 1.2% (1.5× lift on 0.8% base) | 0.4% | 216,360 [216,360–216,360] | 27,048 | 1,623 | 974 | 487 | 243 |
| B6 | meeting | 0.8% → 1.6% (2× lift on 0.8% base) | 0.8% | 64,848 [32,424–64,848] | 8,108 | 486 | 292 | 146 | 73.0 |
What is a firewalled proof cohort?
Before a program sends anything, RevenueOS randomly assigns a slice of its leads to a holdout that is never mailed and a proof cell whose arms rotate uniformly. The optimizer never touches either cell, so the comparison stays causal: whatever difference appears between proof and holdout was caused by the sending, not selected by the algorithm that benefits from looking good.
Why do null results publish?
A proof layer that only reports wins is an advertising layer. When a claim’s band never excludes zero, RevenueOS publishes the null on the same ledger, with the same recomputable arithmetic — because an agency deciding where five thousand rationed sends go needs to know which angles did nothing just as much as which ones worked.
Where do the numbers come from?
The statistical machinery is the published literature on time-uniform confidence sequences and betting-based estimation; the market rates are dated 2026 benchmark publications, cited below with their disagreements noted rather than smoothed over. Every figure on this page carries its source, and the table’s generator script is named so the arithmetic can be recomputed rather than believed.
- Howard, Ramdas, McAuliffe & Sekhon — “Time-uniform, nonparametric, nonasymptotic confidence sequences”, Annals of Statistics 49(2), 2021
- Waudby-Smith & Ramdas — “Estimating means of bounded random variables by betting”, Journal of the Royal Statistical Society: Series B
- Instantly — Cold Email Benchmark Report 2026
- Prospeo — cold-email reply-rate study (live-verified 2026-07-03)
- Generator: backend/scripts/proof/minimum-n-table.ts — seeded, deterministic, 200 Monte-Carlo runs per grid point. The table above is drift-locked to its output.
RevenueOS — the proof layer for cold outbound · the first entry · the pilot, on one page