What are the Risks of Backtesting Investment and Trading Strategies

Over twenty years ago, someone asked me "how do you know your trend systems will be profitable if market conditions aren't the same as they have been?" From that point, I started backtesting technical indicators to develop trend signals and build complete trading systems that determine what to buy or sell, when, how much we should risk in each position, and when to exit a loser, laggard, and a winner.

The truth is, we can never "know" anything about a future that hasn't yet existed, but through a rigorous scientific statistical approach, we can develop trading systems and investment programs with mathematical basis for believing they'll be robust. But still, backtesting comes with a disclaimer for good reason: 

Backtest Backtested Disclosure

Backtested performance does not represent actual performance and should not be interpreted as an indication of such performance. Actual performance for client accounts may be materially lower than that of the index portfolios. Backtested performance results have certain inherent limitations. Such results do not represent the impact that material economic and market factors might have on an investment adviser's decision-making process if the adviser were actually managing client money. Backtested performance also differs from actual performance because it is achieved through the retroactive application of model portfolios designed with the benefit of hindsight. As a result, the models theoretically may be changed from time to time and the effect on performance results could be either favorable or unfavorable.

Backtesting investment and trading strategies can be a useful tool for evaluating the potential profitability of a trading strategy, but there are also some risks involved that should be considered:

1. Overfitting: Backtesting can lead to overfitting, where a strategy is optimized to perform well in historical data but does not perform well in the future. This can happen when the strategy is too complex or too reliant on specific past market conditions.

2. Data quality: Backtesting requires accurate historical data, and if the data is incomplete or inaccurate, it can lead to incorrect results.

3. Survivorship bias: Backtesting can be biased towards successful strategies that have survived in the market, while ignoring unsuccessful strategies that have been abandoned. This can lead to an overestimation of the profitability of a strategy.

4. Transaction costs: Backtesting must account for transaction costs such as brokerage fees, slippage, and taxes, which can significantly impact the profitability of a strategy and how its reduced by the expense of trading the system.

5. Model assumptions: Backtesting relies on certain assumptions about market behavior and the effectiveness of trading signals, and if those assumptions are incorrect, it can lead to inaccurate results. Or, the future may not look like the past period tested.

6. Curve fitting: Backtesting can lead to curve fitting, where a strategy is optimized to perform well on a specific set of historical data, but performs poorly on other data sets. Past performance is never a guarantee of future results, but the more data minded the backtest, the less likely it can adapt and evolve with new data it hasn't yet seen.

Overall, backtesting can be a useful tool for evaluating trading strategies to develop trading systems, but it should be used with caution and in conjunction with other forms of analysis and testing to minimize the risks involved.