Probability of Backtest Overfitting (PBO): The Number Every Optimizer Should Report

The Probability of Backtest Overfitting (PBO) is the estimated probability that the strategy variant you selected as best in-sample will perform below the median of your candidates out-of-sample. It was formalised by Bailey, Borwein, Lopez de Prado, and Zhu, and it is computed with a procedure called Combinatorially Symmetric Cross-Validation (CSCV). In plain terms, PBO measures how much your selection is driven by luck. If picking the in-sample winner tells you nothing about out-of-sample rank, the winner falls below median roughly half the time and PBO approaches 0.5.

The reason PBO deserves to be a headline number is that it directly targets the act of selection, which is where most overfitting actually happens. You didn't overfit by running one backtest; you overfit by running many and keeping the best-looking one. Any single backtest can be honest and still be the luckiest of a thousand. PBO asks: across all the ways I could have split my data, how often does 'best here' fail to mean 'good there'? A PBO near or above 0.5 says your optimizer is a random winner generator. For the conceptual background on why selecting the best of many is dangerous, read [overfitting and curve-fitting explained](/learn/overfitting-curve-fitting-explained) first. PBO is the number that puts a value on that danger.

CSCV starts from a matrix, not a single equity curve. You lay out the per-period returns of every strategy variant you tried, one column per variant, one row per time period. That matrix is the honest record of your optimization: everything you tested, not just what you kept.

The procedure then splits the rows (the time periods) into an even number of equal blocks and considers every way to assign half the blocks to in-sample and the other half to out-of-sample. For each split, it finds the variant with the best in-sample performance, then checks that same variant's rank out-of-sample. It records the out-of-sample rank as a relative position, and computes the logit of that rank. Do this across all the symmetric combinations of blocks and you get a distribution of logits. PBO is simply the fraction of splits where the in-sample best landed below the out-of-sample median.

Two design choices make CSCV trustworthy. First, it is symmetric: because it uses every complementary combination, in-sample and out-of-sample are treated identically and no single arbitrary split can flatter you. Second, it evaluates rank, not raw performance, so it is robust to the overall level of returns and focuses purely on whether selection carries over. It is a close cousin of the split logic in [in-sample versus out-of-sample](/learn/in-sample-vs-out-of-sample), generalised from one split to all of them. The cost is combinatorial: the number of splits grows fast with the number of blocks, so CSCV is usually run with a modest block count.

Read PBO as a probability of self-deception, and read it alongside its companions rather than in isolation. The rough interpretation below is directional, not a certification.

| PBO | What it suggests | |---|---| | Near 0 | In-sample winner tends to hold up out-of-sample; selection appears to carry information | | Around 0.5 | Selection is essentially a coin flip; the best in-sample variant is a lucky draw | | Above 0.5 | In-sample winners systematically underperform out-of-sample; selection is actively misleading |

A low PBO is necessary but not sufficient. It tells you your selection process is not obviously fitting noise; it does not tell you the strategy is good, robust to the next regime, or free of look-ahead bias. You can have a clean PBO on a strategy that still fails live because the whole dataset shared a bias, or because transaction costs were understated. PBO is one gate, not the wall.

The CSCV paper also reports the performance-degradation slope (how much out-of-sample performance drops as a function of in-sample performance) and the probability of loss out-of-sample. Read those together. A strategy with acceptable PBO but a steep degradation slope is telling you that in-sample gains evaporate out-of-sample, which is its own warning. And notice the deliberate framing throughout: every one of these is a statement about robustness and consistency of ranking, never a promise about returns.

PBO attacks selection; the other honest tools attack related but distinct failures, and you want the set. The Deflated Sharpe Ratio (Bailey and Lopez de Prado) adjusts a Sharpe estimate for the number of trials, non-normal returns, and sample length, so it discounts a headline Sharpe that looks good only because you tried a hundred configurations; see [the deflated Sharpe ratio](/learn/deflated-sharpe-ratio). White's Reality Check and Hansen's Superior Predictive Ability test ask a formal question about a whole universe of strategies: is the best one better than a benchmark once you account for the fact that you searched? Harvey and Liu's work on multiple-testing haircuts applies the same discipline to reported Sharpe ratios, shrinking them for the number of tests run.

The unifying idea across all of them is that the number of things you tried changes the meaning of the thing you kept. PBO expresses that idea as a probability of overfit selection; the Deflated Sharpe expresses it as a haircut on a statistic; Reality Check and SPA express it as a corrected hypothesis test. They are complementary, and a serious optimization report shows more than one. Walk-forward analysis, covered in [walk-forward analysis](/learn/walk-forward-analysis), is the procedural cousin: it enforces out-of-sample discipline over time rather than measuring it after the fact. Use them together. Each closes a door the others leave open.

Producing a PBO requires the one artifact most people throw away: the returns matrix of every variant you tested. If you optimized fifty parameter sets, keep the per-period return series of all fifty, not just the winner. Without the full matrix there is nothing for CSCV to work on, because PBO is a statement about the whole search, not the survivor. This is the single most common blocker, and it is a discipline problem more than a maths problem: log every trial as you go.

With the matrix in hand, feed it to a CSCV routine, pick an even block count that keeps the combinatorics tractable, and report three things together: the PBO itself, the performance-degradation slope, and the out-of-sample probability of loss. Report the block count you used, because it is a parameter and readers should see it. Then state plainly what the number does and does not license: a low PBO supports the claim that your selection is not obviously noise-fitting, and licenses nothing about future returns.

If you would rather not implement CSCV yourself, our free [Overfitting Check at /backtest/overfitting](/backtest/overfitting) takes a returns matrix and reports PBO alongside the companion diagnostics, and the broader [Backtest Health Check](/backtest) folds it into a wider integrity scan. Both are quality-assurance tools: they are built to catch the backtest that lies, not to tell you a strategy will make money. This article is educational and is not financial advice.