Why Lottery Backtesting Can Fool You

A backtest asks how a rule would have performed on recorded past data. It can be useful for debugging and exploration, but a good historical score does not automatically establish predictive ability. If the rule was selected because it fit that history, the score partly measures the search that found it.

Lottery prediction claims are especially vulnerable because a large collection of random-looking data offers many patterns to choose from after the fact.

The training result is not a forecast Suppose you select the six most frequent labels in the last 100 draws and then report how often those labels appeared in those same 100 draws. The procedure is guaranteed to select a high-count set. That is its definition, not evidence that it anticipates another draw.

A future test must occur after the selection rule has been fixed. Our hot/cold experiment makes the distinction visible by using fresh test draws that played no role in ranking the labels.

The test set can become training data indirectly A researcher tries ten windows, then five thresholds, then several models, repeatedly checking which does best on the same evaluation period. Even if the code never calls that period “training”, its outcomes have influenced the final selection.

Holding out a dataset only works while it remains genuinely untouched by model choice. Repeated consultation spends that independence. A further clean evaluation is needed to assess the selected procedure.

Leakage can be subtle A feature may summarize the full dataset, including future draws. A normalization step may use future observations. A rolling count may accidentally include the draw it is supposed to predict. A random train/test split may not match the chronological forecasting problem.

The remedy is to define the time at which each piece of information would have been available and enforce that boundary in the code. A model should not receive information that the real predictor would not yet know.

Match the metric to the claim Average main-number matches, any-prize frequency, jackpot matches and net return are different outcomes. A method that improves a descriptive partial-match statistic has not thereby demonstrated profitable jackpot prediction. Costs and payout rules introduce additional questions.

Under independent uniform k-of-n draws, any k-label set selected from past information has expected k²/n main matches against the next draw. This provides a clear baseline. A claim of improvement should quantify its uncertainty and account for all the rules searched.

A stronger prospective design Publish the rule or freeze it with a verifiable timestamp. Define eligible draws, the evaluation horizon, comparison baseline and analysis before outcomes arrive. Preserve every prediction, including failures. Avoid changing the stopping point because a temporary result looks impressive.

Even a well-run synthetic backtest only checks behaviour under the simulated model. It does not establish a real-world defect in a lottery mechanism. The Lab is explicit about that boundary. Its purpose is to demonstrate how honest evaluation works and why historical pattern hunting, however sophisticated its software, cannot substitute for fresh evidence.