Backtest Overfitting: Track the Search behind the Winning Result

Look beyond the final parameter set

A strategy can use correctly timed data and still be selected because it happened to fit one historical sample. Trying many windows, thresholds, instruments, filters and start dates gives chance patterns multiple opportunities to look convincing. Manual edits count as research choices even when no optimizer recorded them.

The primary paper The Probability of Backtest Overfitting examines selection and out-of-sample degradation in investment backtests. Its formal framework is more detailed than a single pass/fail rule. This reference uses a simple original probability example to explain why the search history belongs in a review.

Calculate the effect of repeated opportunities

Assume, only for illustration, that each of 20 independent tests has a 5% chance of producing a false positive under a no-effect model. The chance that none does so is 0.95 to the twentieth power, approximately 35.85%. The chance of at least one is therefore about 64.15%.

Actual strategy variants are usually dependent and their selection rules differ, so 64.15% is not an estimate of your strategy's overfitting probability. The example demonstrates a narrower point: the interpretation of one attractive result changes when it was chosen from many opportunities.

Keep an experiment ledger

Record the hypothesis before the run, dataset version, date range, costs, fill model, parameters, objective and result. Preserve failed and abandoned variants. If a researcher tries five entry rules, four exit rules and three universe filters in all combinations, that is 60 configurations before considering further tuning, even if the final report names only one.

Also log informal visual selection. Choosing the most attractive equity curve after looking at many charts uses the same historical outcome information as a coded ranking rule. The ledger should explain why a configuration was selected and what would count as evidence against the underlying explanation.

Protect evaluation data from repeated reuse

Separate development from later evaluation using a documented chronology. When a supposedly untouched period is inspected and the strategy is changed in response, that period has contributed to development. Renaming it “validation” does not restore its independence. A new evaluation period or appropriately designed resampling procedure may be required for the next claim.

Small samples and regime changes limit any evaluation. Stable behavior across a reasonable neighborhood of parameters can be informative, but searching for that neighborhood after observing every result is another selection step. Include realistic costs and execution constraints, since an optimizer can otherwise exploit an unrealistic simulator.

For copy-trading research, the chosen source account, start date and allocation rule also form part of the search. An appealing historical account curve cannot establish future performance without considering how it was selected and what comparable alternatives were discarded.