Educational: The truth about backtesting
I recently started to read a number of finance books and watching a lot of YouTube channels specifically on the topic of chaos theory and backtesting. I am going to start a series here where I share what I am learning and today we are going to delve into the world of backtesting.
"Those who fail to learn history are condemned to repeat it" - George Santayana
Many would agree that the above statement is truth. However one of the first thing you learn in trading/finance is that "Past performance does not predict future results". This statement is on every website, every video and every article. Everyone arms themselves with this statement and then proceeds to base the information they provide on past performance/data.
Well if this statement is truth, what is the alternative to using past performance/data? Financial reports, economic forecast, technical analysis, price action, footprint chart are all based on past performance/data. The truth is that the statement is made primarily as a legal protection against mislead investors who may believe that because something happened in the past it is guaranteed to happen again. But does that mean that there is no truth to the statement ? Well not exactly.
"Hindsight is 20/20" When looking at market data of the past it is very easy to see clear patterns in market behavior that seemingly repeat over and over again, creating the illusion that if you map an investing/trading strategy on this data it is very likely to performance well in the future. But this is not truth at all. In order to understand why it is not true please try to imagine the activity below.
Imagine you are a racecar driver, and you're brought to a new racing track that no one has every driven on . You are task with setting the best time record on the track. You got in your car, did your best and set a record of 10 minutes to complete the track. After the race you now have a map of the track and know what turns to expect and all the details of the road. Lets say you share this information to the next driver. He now attempts to set a record on the track and he beats your record by 2 minutes and the new record is now 8 minutes. The second driver now shares what he has learnt about the track to the third driver. The third driver takes the information from driver 1 and driver 2 and modifies his car to have the right tire's, right suspension, right aerodynamics etc. in order to have to best performance on the track. The third driver goes on the track and sets a record of 5 minutes. Completely outperforming driver 1 and driver 2. Driver 3 is now the champion of the track.
Below is a simple visualization of what is happening in terms of information that each driver had on their first run of the track.
Driver 3 is now dominating on the track and thinks he is unbeatable. Some time has past and driver 3 is brought to a new track that no one has raced on with his modified car from the first track. He attempts to race the track and sets a record of 12 minutes. He thinks he has done well. But driver 2 attempts the track and sets a record of 7 minutes out performing driver 3
Why didn't driver 3 outperform the other two drivers on the new track with his modified car? Well, what has happened here is that driver 3 has modified his car to the specifics of the first track as best as he can but when he went on another track that has different variables his car was not able to perform as well as it did before. Maybe the terrain was too rough or maybe the corners were too tight on this new track. Regardless of the reason, the modifications made to the car ended up working against him. In backtesting this is called "overfitting".
Overfitting in backtesting is when a strategy has been developed too precisely on past data causing the strategy to "fit to the curve" and there is now a risk of it not performing in the future once the market behavior changes.
Above we have a demonstration of how overfitting works. Where we looked at some data in the past and then we created a strategy that would have performance well on that data. Based on this information it now seems like we have a very profitable system but lets see what happens when me use this strategy on future data without adjusting our strategy.
In the image above we can see that the same strategy is no longer working in the new market environment. There are periods where it does perform but overall it is far worst. Now you might wonder, why is this? If the strategy worked on the market in the past why wont it work in the future. This is because the market is unpredictable. People often say the market is random, it is not random, what it is, is unpredictable and that is very different from being random. I will be doing another publication on chaos theory in the markets which will go into this topic much deeper, but for now, know that the reason it doesn't work is because the market is unpredictable.
When looking at past data if you take a bunch of variables and brute force your way into finding a strategy you will eventually find a strategy that "fits to the curve". You should not be using backtesting as a way to necessarily find a edge but it should be used as a way to validate your edge.
So with knowing this information how do we combat overfitting of data and how to properly backtest a system. When backtesting you need to deploy your backtesting using out of sample and in sample data. And what that means is that you will break apart your data into clusters. Where your "in sample" data is where you design your strategy and parameters and your "out of sample" data is where you will apply your strategy to see if it works. For example lets say you are looking at 10 years of data that you want to backtest. Take 3 years of that 10 and develop your strategy. Then with the strategy you developed which performs well on your "in sample" data you will now use it to test the other 7 years, of your "out of sample" data. What this does is prevent you from overfitting to the curve by basing your strategy on the entire data set.
Below is a visualization of this
What this does allow you to create a robust strategy that can be forward tested on future data.
Once again this is me sharing some of the knowledge I have attained. I am sure there are programmers here on tradingview that could possible explain this better or in more details. I welcome discussion and any criticism regarding my publication. Thank you for taking the time to read and I will update this publication in the future if needed.
"Those who fail to learn history are condemned to repeat it" - George Santayana
Many would agree that the above statement is truth. However one of the first thing you learn in trading/finance is that "Past performance does not predict future results". This statement is on every website, every video and every article. Everyone arms themselves with this statement and then proceeds to base the information they provide on past performance/data.
Well if this statement is truth, what is the alternative to using past performance/data? Financial reports, economic forecast, technical analysis, price action, footprint chart are all based on past performance/data. The truth is that the statement is made primarily as a legal protection against mislead investors who may believe that because something happened in the past it is guaranteed to happen again. But does that mean that there is no truth to the statement ? Well not exactly.
"Hindsight is 20/20" When looking at market data of the past it is very easy to see clear patterns in market behavior that seemingly repeat over and over again, creating the illusion that if you map an investing/trading strategy on this data it is very likely to performance well in the future. But this is not truth at all. In order to understand why it is not true please try to imagine the activity below.
Imagine you are a racecar driver, and you're brought to a new racing track that no one has every driven on . You are task with setting the best time record on the track. You got in your car, did your best and set a record of 10 minutes to complete the track. After the race you now have a map of the track and know what turns to expect and all the details of the road. Lets say you share this information to the next driver. He now attempts to set a record on the track and he beats your record by 2 minutes and the new record is now 8 minutes. The second driver now shares what he has learnt about the track to the third driver. The third driver takes the information from driver 1 and driver 2 and modifies his car to have the right tire's, right suspension, right aerodynamics etc. in order to have to best performance on the track. The third driver goes on the track and sets a record of 5 minutes. Completely outperforming driver 1 and driver 2. Driver 3 is now the champion of the track.
Below is a simple visualization of what is happening in terms of information that each driver had on their first run of the track.
Driver 3 is now dominating on the track and thinks he is unbeatable. Some time has past and driver 3 is brought to a new track that no one has raced on with his modified car from the first track. He attempts to race the track and sets a record of 12 minutes. He thinks he has done well. But driver 2 attempts the track and sets a record of 7 minutes out performing driver 3
Why didn't driver 3 outperform the other two drivers on the new track with his modified car? Well, what has happened here is that driver 3 has modified his car to the specifics of the first track as best as he can but when he went on another track that has different variables his car was not able to perform as well as it did before. Maybe the terrain was too rough or maybe the corners were too tight on this new track. Regardless of the reason, the modifications made to the car ended up working against him. In backtesting this is called "overfitting".
Overfitting in backtesting is when a strategy has been developed too precisely on past data causing the strategy to "fit to the curve" and there is now a risk of it not performing in the future once the market behavior changes.
Above we have a demonstration of how overfitting works. Where we looked at some data in the past and then we created a strategy that would have performance well on that data. Based on this information it now seems like we have a very profitable system but lets see what happens when me use this strategy on future data without adjusting our strategy.
In the image above we can see that the same strategy is no longer working in the new market environment. There are periods where it does perform but overall it is far worst. Now you might wonder, why is this? If the strategy worked on the market in the past why wont it work in the future. This is because the market is unpredictable. People often say the market is random, it is not random, what it is, is unpredictable and that is very different from being random. I will be doing another publication on chaos theory in the markets which will go into this topic much deeper, but for now, know that the reason it doesn't work is because the market is unpredictable.
When looking at past data if you take a bunch of variables and brute force your way into finding a strategy you will eventually find a strategy that "fits to the curve". You should not be using backtesting as a way to necessarily find a edge but it should be used as a way to validate your edge.
So with knowing this information how do we combat overfitting of data and how to properly backtest a system. When backtesting you need to deploy your backtesting using out of sample and in sample data. And what that means is that you will break apart your data into clusters. Where your "in sample" data is where you design your strategy and parameters and your "out of sample" data is where you will apply your strategy to see if it works. For example lets say you are looking at 10 years of data that you want to backtest. Take 3 years of that 10 and develop your strategy. Then with the strategy you developed which performs well on your "in sample" data you will now use it to test the other 7 years, of your "out of sample" data. What this does is prevent you from overfitting to the curve by basing your strategy on the entire data set.
Below is a visualization of this
What this does allow you to create a robust strategy that can be forward tested on future data.
Once again this is me sharing some of the knowledge I have attained. I am sure there are programmers here on tradingview that could possible explain this better or in more details. I welcome discussion and any criticism regarding my publication. Thank you for taking the time to read and I will update this publication in the future if needed.
C Nicholas Downie
Related publications
Disclaimer
The information and publications are not meant to be, and do not constitute, financial, investment, trading, or other types of advice or recommendations supplied or endorsed by TradingView. Read more in the Terms of Use.