Backtest Overfitting: An Interactive Example

What is backtest overfitting? And why should I care?

Graphic on left shows an investment strategy (in blue) making steady profit while the underlying trading instrument (in green) gyrates in price. This website is designed to illustrate that this type of "optimal" investment strategy is all too easy to produce if enough variations are tried, yet it is typically ineffective moving forward, a consequence of backtest overfitting. Graphic on right shows the same investment strategy performing poorly on a different sample of the same trading instrument, which demonstrates that an overfit investment strategy is quite likely to fail in real world investing.

By checking this box, you acknowledge that this web site is a demonstration of a mathematical concept known as "backtest overfitting." The software and material on this site should NOT be interpreted as a recommendation to buy or sell any security or securities; as applicable to the specific investment needs of any particular individual or organization; or as applicable to the forecast of future market prices or trends. Option 1: Repeat the example above Please be patient! The program may take up to two minutes to display the results page after clicking Go! button Option 2: Try with random parameters Please be patient! The program may take up to two minutes to display the results page after clicking Go! button This "quick-start" option uses random (system generated) values for all of the inputs available in Option 3. Option 3: Try with your own parameters You can enter values in one or more of the text boxes below. A more detailed explanation of what each box does is provided in the tutorial document below. Enter values for these parameters and press Go! button Maximum Holding Period an integer; ranges from 5 to 20. The number of days a stock is held before it is sold if no other exits (e.g., the stop loss) have been triggered. Maximum Stop Loss an integer %; ranges from 10 to 40. The percentage loss from the initial entry at which point the stock will be sold. Do not enter the '%' sign. Sample Length an integer; ranges from 1000 to 2000. The number of days simulated. The sample data considered as the daily closing prices. Standard Deviation an integer; the standard deviation of random numbers used to generate daily price changes; any positive integer could be entered; we recommend 1 or 2 Seed a seed for the pseudorandom numbers used for In Sample Data; it can be any positive integer; enter a nonpositive integer to use the current time as the seed Please be patient! The program may take up to two minutes to display the results page after clicking Go! button Option 4: Try with stock market data You can enter values in one or more of the text boxes below. Enter values for these parameters and press Go! button Maximum Holding Period an integer; ranges from 5 to 20. The number of days a stock is held before it is sold if no other exits (e.g., the stop loss) have been triggered. Maximum Stop Loss an integer %; ranges from 10 to 40. The percentage loss from the initial entry at which point the stock will be sold. Do not enter the '%' sign. Sample Length an integer; ranges from 1000 to 12000. The number of days simulated. We suggest between 5000-6000 as it makes almost equal length data on IS and OOS sides. The sample data considered as the daily closing prices. Please be patient! The program may take up to two minutes to display the results page after clicking Go! button

Default values: If you do not enter a value or enter a value that is outside the range mentioned above, a default value will be used. The default values are: maximum holding period = 7; stop loss = 10; sample length = 1000; and standard deviation = 1.

About execution time: The values for maximum holding period, stop loss and sample length significantly affect the number of iterations performed by the program; the larger these values are, the longer the program will run.

A few recent articles explaining the backtest overfitting in the financial press:

Related documents:

Tutorial of the online tool:

Questions or comments:

Credits:

David H. Bailey, Jonathan M. Borwein, Marcos Lopez de Prado, and Qiji Jim Zhu Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance, Notices of American Mathematical Society , May 2014, pg. 458-471.

We gratefully acknowledge the helpful comments and suggestions from colleagues and friends in shaping this web site. In particular, the suggestions from Mr. David Witkin of StatisTrade, Bin Dong of LBNL, and Beytullah Yildiz of Turkey were extensive and very helpful in improving readability of the web pages. Thanks!