White's Reality Check & Hansen SPA: Did Your Best Strategy Actually Beat Luck?

White's Reality Check is a statistical test that asks whether the best-performing strategy in a set of candidates genuinely outperforms a benchmark, or whether its lead is what you would expect from luck given how many strategies you tried. Introduced by Halbert White in 2000, it is a data-snooping test: it explicitly accounts for the fact that you selected the winner from many contenders, which is precisely the selection bias that makes single-strategy p-values meaningless after a big search.

The mechanism is a bootstrap. You take the performance series of every variant, resample the data many times under the null hypothesis that no strategy beats the benchmark, and each time record the maximum performance statistic across all variants. That distribution of maxima -- the best result you would see repeatedly under pure luck -- becomes your yardstick. Your actual observed best is then compared against it to produce a p-value. Crucially, the test bootstraps the max across the whole family at once, so it correctly captures the inflation we describe in [the multiple-testing haircut](/learn/multiple-testing-haircut-trading). This is the same integrity-first question our free [Backtest Health Check](/backtest) is designed around: not 'how good is the equity curve' but 'how much of it survives luck'. For the broader context, see [why most backtests fail](/learn/why-most-backtests-fail).

The engine of both tests is a bootstrap of the maximum performance statistic. Start with each strategy's series of excess performance over the benchmark -- for example, per-period return differences. The null hypothesis is that the best strategy has no genuine edge: its true expected outperformance is zero or negative. To simulate that world, you resample the historical series using a block bootstrap (Politis and Romano's stationary bootstrap is the standard choice, because it preserves autocorrelation by resampling contiguous blocks rather than single points -- randomizing point-by-point would destroy the serial structure that matters in return data).

On each of thousands of resamples, you recompute every strategy's average outperformance and take the largest one. Collecting those thousands of maxima gives you the sampling distribution of 'the best result achievable by luck alone across this many correlated strategies.' Your real observed best outperformance is then placed against that distribution. The p-value is the fraction of bootstrap maxima that meet or exceed your actual best. A p-value of 0.02 means only 2% of luck-only worlds produced a champion as good as yours; a p-value of 0.60 means most luck-only worlds beat you, so your winner is unremarkable.

A minimal, look-ahead-clean sketch of the per-resample step:

# excess[t, k] = strategy k's outperformance over benchmark at time t
# resample contiguous blocks of rows (stationary bootstrap), then:
boot_means = excess[block_rows, :].mean(axis=0)  # mean per strategy
boot_max = boot_means.max()                       # best-of-many under null

No future rows leak into any past computation; each resample uses only its sampled block indices.

Hansen's Superior Predictive Ability (SPA) test, introduced by Peter Reinhard Hansen in 2005, is a refinement of White's Reality Check that fixes a known weakness. White's test can lose power when your candidate set is padded with obviously poor strategies: those irrelevant losers pull the null distribution around and make it harder to detect a genuinely good winner. In plain terms, stuffing your search with junk can hide a real edge under the Reality Check.

Hansen addresses this in two ways. First, he studentizes the performance statistics -- scaling each by its own variability -- so noisy and stable strategies are compared on fairer footing. Second, his SPA framework down-weights strategies that perform so poorly they are clearly not contenders, so they stop distorting the benchmark distribution. The result is a test that is typically more powerful than the Reality Check at the same size, meaning it is better at recognizing a truly superior strategy when one exists, while still controlling false positives from data snooping. In practice, treat Hansen SPA as the stronger default and White's Reality Check as the intuitive, easier-to-explain sibling. Both share the same honest premise: the champion of a large search must be judged against the best luck can produce, not against a naive single-strategy threshold. Neither speaks to whether the strategy will earn money -- only to whether its historical edge is distinguishable from noise.

A significant Reality Check or SPA p-value means one specific thing: your best strategy's historical outperformance is larger than what this many correlated strategies would typically produce by luck alone. That is genuinely valuable -- it is the difference between 'the winner of my sweep' and 'a winner that beat the sweep-adjusted luck bar.' It is a robustness signal about the past.

Here is what it does not mean, and each caveat matters. It does not mean the strategy is profitable, or will be -- statistical superiority over a benchmark on historical data is not a forecast of returns, and nothing in this test estimates future performance. It does not mean the edge will persist; markets change, and a past-tense result carries no guarantee. It does not license you to have snooped elsewhere: if you first eyeballed results, discarded ugly variants, or tuned before assembling the candidate set, your effective trial count is larger than what you fed the test, and the p-value is optimistic. It does not fix a broken backtest -- [look-ahead bias](/learn/look-ahead-bias), survivorship gaps, or unrealistic costs will sail straight through, because the bootstrap faithfully reproduces whatever bias is baked into your series. And it is not a substitute for out-of-sample evidence; a clean p-value plus [walk-forward analysis](/learn/walk-forward-analysis) and honest [in-sample vs out-of-sample](/learn/in-sample-vs-out-of-sample) separation is far stronger than any one number. Read the p-value as 'this passed a data-snooping check,' never as 'this makes money.'

Use the Reality Check and SPA as a gate before you celebrate a winning strategy, and feed them an honest candidate set. The most common failure is silent pre-filtering: you run 400 variants, discard the 380 that looked bad, and submit only the 20 survivors to the test. That understates the true number of trials and inflates your significance. Instead, include the full family you actually searched over, ugly variants and all -- Hansen SPA is specifically built to tolerate poor candidates without losing power, so there is no honest reason to hide them.

A sound sequence: assemble every variant you tried; compute each one's outperformance over a fair benchmark; run a stationary block bootstrap of the max statistic (Reality Check) and its studentized SPA counterpart; report both p-values alongside the winner's raw statistics. Then corroborate rather than conclude -- pair the p-value with a [multiple-testing haircut](/learn/multiple-testing-haircut-trading) on the Sharpe, a [Deflated Sharpe Ratio](/learn/deflated-sharpe-ratio), path stress via [Monte Carlo simulation](/learn/monte-carlo-simulation-backtests), and a check that you even have [enough trades](/learn/how-many-trades-is-enough) for the statistics to mean anything. Our free [Backtest Health Check](/backtest) and [Overfitting Check](/backtest/overfitting) are built to surface exactly these selection risks. Throughout, hold the frame: these tests defend integrity and robustness. They are educational tools, not financial advice, and they never claim or imply that any strategy will be profitable.