Backtesting Trading Strategies Everything You Need to Know

Backtesting Benefits

There are five major benefits to backtesting a given strategy. Below I cover each one.

1. Helps avoid bad strategies

Backtesting a strategy before risking capital can help avoid losing strategies that stand zero chance to make a profit and will perform poorly in live trading.

2. Paper trading  

Backtesting can help a trader visualize what following a trading strategy would be like. Finding a strategy that matches your personality can be a benefit to long-term success. Utilizing a risk-free demo account to conduct paper trading to gain confidence is an excellent idea before risking money.

3. Evaluate different market conditions

Backtesting enables traders to analyze a strategy on different market conditions from different historical data periods. How does this strategy do during the 2008 Great Financial Crisis? How does it do during a strong bull run from 2013 to 2015? How did it do during the Covid drop of 2020 or some future market conditions we do not know yet?

Additionally, intraday strategies can be evaluated on different days of the week or during different time windows.

4. Set proper expectations

Backtest results can help traders understand how profit and loss expectations should be set and engage in smarter money management. If a strategy earned $10,000 per 100 shares per year, then expecting a strategy to make $1,000,000 on 100 shares next year would be unreasonable investment research.

Moreover, noticing a steep drawdown during a backtest can help a trader properly fund a strategy prior to going live.

I spoke more about setting proper expectations here: Monte Carlo Analysis and Drawdowns

5. Compare Strategies

Not all traders have unlimited capital and must choose between strategies. Backtests can help traders understand where and when their capital is best put to use. Spread bets, algo trading, futures, stock gappers, trading cfds, etc. can all be compared objectively.

Backtesting Pitfalls and Common Mistakes

Poor backtest results often indicate a strategy unworthy of live trading. However, good backtest results are not a greenlight to live trade. There are a handful of backtesting pitfalls we must watch. Some backtests may be too good to be true!

Overfitting or Curve Fitting Backtesting Risk

Over optimization or curve fitting is finding a trading strategy that too closely fits the historical data and will fail to adapt to new data. The strategy essentially learns the nuances of the historical data during the backtest but fails on new data when the exact nuances do not play out exactly as they did in the previous time period.

Overfitting is the largest risk system traders face. Overoptimizing parameters to improve backtest results can lead to great test results and good emotions, but horrible live trading performance.

There are many advanced techniques to overcome this pitfall which I wrote about here:  Curve Fitting – Build Alpha 

Look Ahead Bias

Another backtesting pitfall, look ahead bias, is using data that was not known at the time of the historical trade. A tragic example of look ahead bias in a multi-timeframe strategy would be using the daily close to make a trading decision midday. At the time of the trade, midday, it is impossible to know the close of the same day.

These look ahead bias risks are often negated and removed with proper data cleaning or professional grade automated backtesting software.

Survivorship Bias

Running a backtest on only currently traded symbols introduces too much optimism in the backtest results as it can avoid buying stocks that eventually went bankrupt or delisted. This phenomenon is known as survivorship bias.

Much like look ahead bias, survivorship bias deals with underlying historical data issues. Many data providers only have symbols currently listed on exchanges. A proper backtest should consider all symbols that were available at all historical dates and times. This includes stocks that went bankrupt, merged, went private or delisted.

Additionally, choosing to run a backtest on only the best stocks also introduces survivorship bias. The best stocks cannot be known ahead of time and undoubtedly leads to overly optimistic results. If we could identify AAPL or GOOGL ten years ago then your personal finance situation would be significantly different.

We must be unbiased in our symbol universe selection to have truly reliable backtest results. Ask your data provider if they provide listed and delisted securities!

Ignoring Transaction Costs

Backtesting a trading strategy without transaction costs such as slippage or commissions can mislead traders into believing unrealistic backtesting results thus depleting their accounts in live trading.

Backtest results can show an average trade of $10 or $15 but in real markets where slippage and commissions are present the average trade may be negative.

Ignoring trading costs during backtesting can find unrealistic and unobtainable results. Always include transaction costs in your backtests for reliable results!

Backtesting vs. Forward Testing

Forward testing is creating a trading strategy and allowing it to trade on live data on a paper or simulated account. Some forward test in a real account with small size. Many traders view forward testing as a necessary component of moving a strategy from the research stage to the live trading phase.

Backtesting is great for quickly identifying the viability of a trading strategy. Forward testing creates the same hypothetical results but on new data. Forward testing takes more time but is done on the most recent data.

There is no need for a trader to choose one or the other as both can have their place in a proper strategy pipeline; however, forward testing is often not necessary if statistical significance can be obtained during the backtesting phase and rigorous robustness testing is applied.

Backtesting Portfolio

In financial markets, backtesting single trading strategies is a tremendous achievement and necessary for trading system developers. However, backtesting a portfolio of strategies goes one step farther.

Ray Dalio, founder of Bridgewater – the world’s largest Hedge Fund, released a compelling graphic demonstrating the math behind combining multiple strategies together.

The graphic shows the more uncorrelated strategies one can combine, the lower the probability of losing money in a given year. Wow! This epiphany is a true key to algo trading success. I wrote about it here: Algorithmic Trading Complete Guide.

That being said, it is important to have the ability or software to backtest portfolios of strategies not just single strategies. This was a key driver behind Build Alpha’s Portfolio Mode upgrade.

Advanced Backtesting Techniques

Due to the various pitfalls and brilliant minds in quantitative finance, various advanced techniques have been created to identify more reliable backtest results which hopefully lead to more sustainable live trading performance.

Out of Sample Testing (OOS)

Out of Sample testing is withholding some historical data for future testing – typically some percentage. First, develop and tweak your trading strategy on the In-Sample data. Once you have achieved desirable backtest results you can test your strategy on the withheld out of sample data.

Ideally, performance results should be similar in both the in-sample and out-of-sample data periods. If trading performance fails on the out of sample data, then we likely avoided an overfit or poor strategy.

I wrote more exclusively about this test here: Out of Sample Testing

Randomized Out of Sample Testing

Selecting the proper out of sample period is a challenge to creating reliable backtest results. What if you select an out of sample period that contains only a bull market (like the image above)? The out of sample test may be easy to pass. Randomized out of sample testing can help avoid this.

Randomized Out of Sample testing randomly selects non-congruent dates from your historical data set to be part of the out of sample period.

The trader can create 1,000 or more randomized out of sample periods then compare results on all of them. This helps overcome randomly selecting data points from a bull market only. Learn more about Randomized Out of Sample.

Lying Backtests. Are backtests enough?

Let’s discuss a simple case study that uses intermarket signals from the Japanese Yen to trade US 30-year bond futures. The strategy has only 2 rules for entry and produces fairly stable in-sample returns for being so simple (2003-2012 backtesting period).

Graph shown shows results based trading 1 contract per signal for demonstration purposes

The intermarket strategy performed quite well on the unseen, out of sample data (2012-2016 OOS). The ability for a strategy to continue to perform on unseen data is paramount and mission critical for many system traders and strategy architects.

Graph shown shows results based on 1 contract per signal for demonstration purposes

However, can we trust this backtest? In Build Alpha, there are many validation and robustness tests that can add confidence to your system development and backtesting process.

There is a special test called the “Noise Test” which creates 1,000 new price series by adding and subtracting varying amounts of noise (read volatility) from the historical data. The test then “re-trades” the strategy on the new 1,000 price series.

The idea is that if a strategy is overfit then it is overfit to the noise of the original data and is not trading the underlying signal we hope it is. Changing the noise will reduce performance if the strategy is overfit to the noise of the original data. Ideally, we want to see a strategy continue to perform well on the new, noise-adjusted price series.

This “Noise Test” can help prevent a backtest from lying to us and hopefully prevent us from wasting further research time or taking a weak strategy live! You can see that our original backtest in blue is significantly better than the 1,000 “Noise Test” results. This is worrisome and most likely a sign of an overfit strategy aka a lying backtest that will surely fail on live data.

Viewing this simple strategy’s performance since 2016 confirms the value of the Noise Test. The strategy has been in a nosedive since going “live”, but a smart trader would have never taken it live – no matter how pretty the backtest – due to the failed Noise Test.

Any kind of trading – manual or discretionary – should be subjected to these rigorous tests before getting real money allocations. If not, you may be in for a surprise from a lying backtest or unreproducible hypothetical returns!

To read the full article, please check out The Noise Test and Lying Backtests; additionally, I wrote about the Noise Test here: Noise Testing

Best Backtesting Software

Build Alpha is the best backtesting software available. To manually backtest trading strategies is a time consuming and error prone process. Build Alpha enables the trader to test any and all ideas in a few clicks and requires no coding or programming. Build Alpha also offers the most advanced validation and robustness tests to help identify lying backtests as this has been the focal point of my professional trading career.

Build Alpha’s genetic algorithm can take thousands of input signals, exit ideas, risk management rules, and more to find the best trading strategies for you. The robustness checks and tests give the ability to analyze any backtest result in depth before risking real capital. Portfolio mode allows for traders to combine strategies and backtest portfolios. To learn more head over to the Build Alpha homepage or sign up to see the demo.

Backtesting Need to Knows

• Backtesting applies a trading strategy on historical data to view hypothetical results
• Need data, a trading idea, software or coding to implement a backtest
• Can backtest all financial instruments, any time period, any market condition
• Use to identify poor strategies ahead of risking capital. A fast way of paper trading
• Backtesting performance metrics can compare strategies or run scenario analysis
• Positive backtest results are not a greenlight to trade a strategy; more testing required
• Build Alpha makes automated backtesting and validation possible with no coding

Summary

Many traders implement trading strategies without testing them which often leads to depleted retail investor accounts. Backtesting enables traders to visualize historical performance of any trading strategy prior to risking real capital. Traders only need market data and a trading idea to conduct a proper backtest. A backtest can be used to weed out poor strategies, identify those worthy of future research, compare competing strategies, and more. Backtesting is a vital tool in any serious trader’s repertoire but is not a be all, end all to print money. Backtesting is a great first step, but other validation methods are needed before one should risk real capital.

Author