3 Ways to Test Your Investment Strategy
James Picerno | Aug 05, 2023 10:01
There are three basic ways to test an investment strategy. Each has its own unique set of pros and cons, but only one is practical.h2 1. Out-Of-Sample Testing With Real Money/h2
The best approach is to develop a strategy and then run it with real money out of sample for at least 3 to 5 years. Longer is even better. That’s the gold standard, but that takes time, and so there are obvious limitations. Be careful not to confuse this version of out-of-sample testing with its pseudo-out-of-sample cousin, which uses a portion of historical data to build a model and then tests it on the remaining unused “out of sample” historical numbers. Useful, but no substitute for the genuine article.
h2 2. Paper Trading/h2The weakest alternative is to develop a strategy and paper trade to decide if it passes the smell test. Relatively easy and quick, but here too, there are clear challenges, namely, the transition from theory to empirical usually brings many surprises.
h2 3. Backtesting/h2The best (or should we say the least worst) alternative is to backtest a strategy. The idea here is that you can have the best of both worlds: a rough approximation in the here and now of how a strategy would have fared if implemented at some point decades ago. Alas, this is no silver bullet either since no backtest can flawlessly tell you how a strategy will perform in the years ahead. But short of acquiring the powers to see into the future, it’s the best that mere mortals can do.
Indeed, the key advantage to historical backtesting: you don’t have to wait years to determine if a strategy is a winner or a dog. Another plus: you’re not totally reliant on theory for assessing how the future may unfold.
The critical issue, of course, is designing a backtest that comes close to replicating the real world through a historical lens. Easier said than done. A poorly designed backtest is usually worse than simply making guesstimates. That’s a crucial hazard since there are more ways to delude yourself with backtests than there are techniques for developing a robust test.
In fact, building a useful backtest is a delicate dance of art and science. Ideally, you’ll run many techniques, recognizing that developing useful backtest data and assessing it accurately and objectively is a bit like the tale of blind men trying to describe an elephant. Approximating the truth requires combining multiple descriptions and perspectives.
In short, there are no silver bullets for building a solid backtest. Quite a bit of successfully building and evaluating historical simulations is avoiding rookie mistakes. One error I see a lot is using a single time window to do the heavy lifting.
Get The App
For instance, backtesting a strategy that looks impressive over a 2000-2023 sample period may be misleading because it relies heavily on sidestepping much of the 2008-2009 financial crash. But it’s problematic if, after taking out that period or using a post-2009 start date, the strategy falls apart.
h2 The Best Way to Backtest a Strategy/h2There are many ways to guard against this pitfall, including my favorite technique: assembling a backtest using rolling-forward start dates and then assessing all the time-window results for determining the strategy’s stability (or lack thereof) through time.
As an example, consider a simple 60%/40% stock/bond portfolio that’s rebalanced to the target weights at the end of each calendar year. We’ll use SPDR® S&P 500 (NYSE:SPY) and iShares Core U.S. Aggregate Bond ETF (NYSE:AGG). This toy example begins with a Jan. 1, 2016 start date and calculates the annualized return through Aug. 2, 2023, via daily numbers.
The analytics calculates the full period return using a Jan. 2, 2016 start date and then uses a Jan. 3 start date, and so on. The goal is to aggregate all the annualized returns for each time window and evaluate the distribution, as shown in the chart below.
The main takeaway: the performance is heavily skewed toward a moderately positive result. The interquartile range of returns: is 3.6% to 6.8%, shown by the two blue lines, with a median of 6.2% (red line). Deciding if this is acceptable or not is a bigger question. The point, for now, is that we’re not relying on one time window, which may be deeply flawed for one reason or another.
If this was a robust rolling-forward backtest, we’d use a much earlier start date. We’d also run a batter of other analytics before making a final judgment. But as a first step for deciding whether it’s wise to go deeper or look elsewhere, this is a relatively painless, quick, and useful test. By contrast, a backtest that uses one start date may be faulty in the extreme.
Trading in financial instruments and/or cryptocurrencies involves high risks including the risk of losing some, or all, of your investment amount, and may not be suitable for all investors. Prices of cryptocurrencies are extremely volatile and may be affected by external factors such as financial, regulatory or political events. Trading on margin increases the financial risks.
Before deciding to trade in financial instrument or cryptocurrencies you should be fully informed of the risks and costs associated with trading the financial markets, carefully consider your investment objectives, level of experience, and risk appetite, and seek professional advice where needed.
Fusion Media would like to remind you that the data contained in this website is not necessarily real-time nor accurate. The data and prices on the website are not necessarily provided by any market or exchange, but may be provided by market makers, and so prices may not be accurate and may differ from the actual price at any given market, meaning prices are indicative and not appropriate for trading purposes. Fusion Media and any provider of the data contained in this website will not accept liability for any loss or damage as a result of your trading, or your reliance on the information contained within this website.
It is prohibited to use, store, reproduce, display, modify, transmit or distribute the data contained in this website without the explicit prior written permission of Fusion Media and/or the data provider. All intellectual property rights are reserved by the providers and/or the exchange providing the data contained in this website.
Fusion Media may be compensated by the advertisers that appear on the website, based on your interaction with the advertisements or advertisers.