Nobody ever lost money in a Spreadsheet

After decades of experience with trading systems and CTA’s, we are comfortable with the phrase (Past Performance is not necessarily indicative of future results), as well as disclaimers for hypothetical performance due to back testing. We’ll be the first to recognize the use of back testing on top of the mandatory disclaimers due to the back testing. But what irks us, is when other advisors/investors aren’t held to the same standard, pretending that almost every other investment they look at is not back tested… Take this nice quotation from MorningStar’s Samuel Lee:

“In a real sense, any investor who observes a historical pattern has engaged in back-testing.”

To be fair (and we agree), the article goes on to say that true back testing done correctly, requires quite a bit more than just looking at historical patterns.  But after reading a recent Barron’s blog post, it seems that back testing (by our definition) isn’t just for managed futures and other alternative investments anymore.  It seems that back testing is being applied to the $1.7 trillion ETF market. And even if back testing is done correctly (which is a difficult task to accomplish), there are still some pitfalls.

The issues regarding back testing are real and at times hard to avoid. Nearly all trading strategies including systematic and discretionary traders (both those traded by individual investors and professionals) rely pretty heavily on back tested performance to validate their models and/or trading ideas.  The fundamental problem being that who would ever create a trading strategy that didn’t hold up over its initial testing period? MorningStar provides a good example.

“Last year, Vanguard published a study on ETFs tracking back-tested indexes. The authors, Joel M. Dickson, Sachin Padmawar, and Sarah Hammer (1), looked at a sample of equity indexes with at least five years of back-tested history and five years of live performance. In the five years prior to index live date, the indexes averaged 12.25% excess returns above U.S. equity market; five years after live date, they averaged negative 0.26%. In other words, most ETF index back-tests are garbage.”

Of course, the issue with this is that even the most cautious system developer/coder/writer can unintentionally fit the parameters of their system to what the market(s) have done in the past; creating a system that looks great in testing, but in reality has little to no chance of holding up in the future. This is what we in the industry call “Curve Fitting”. And, while our experience has taught us to be cognizant of this bias and ask pertinent questions to flush it out during our due diligence,  we wonder if everyday stock investors realize that many of the latest and greatest ETF innovations also have the potential for being curve fit to the market. Take Savita Subramanian and three coauthors of Bank of America Merrill Lynch’s quantitative strategists.

“Given our quantitative roots, we are sympathetic to the fact that back tests are often used as an input into making investment decisions. But past returns, as we all know, do not predict the future. And we think backtested results may be particularly problematic today. Very little fundamental data for US equities extends back more than 30 years, but the last 30 years were a period generally accompanied by two related phenomena: increasingly easy monetary policy and falling interest rates. In particular, the wave of liquidity and stimulus provided in the wake of the Tech Bubble coincided with unprecedented levels of credit expansion, rising asset correlations and record earnings volatility.”

The catch is that advisors and ETF marketers cannot show investors hypothetical performance when promoting a new ETF to the client…but of course, Wall St. being Wall St. you know there is a workaround of sorts. A workaround so obvious that most investors probably won’t even catch it at first…the potential for back test bias is in the actual index the ETF is attempting to mimic.  See approximately 90% of all ETF’s are “passive” investments aka they attempt to model an index like the S&P 500 or a single market like Gold. And what Wall St. and ETF creators have figured out is that they create an index to mimic almost any investment strategy… An index that CAN be shown to potential investors for marketing purposes (with the proper disclaimers of course).  An index that is…you got it, back tested (emphasis theirs).

IN THE SEARCH FOR THE NEW and different indexes that will power a new and different ETF, back-testing plays a critical role. Index providers, including S&P, Dow Jones, MSCI, Russell, Zacks and others can index just about anything. You want to rank the companies in the S&P 500 by earnings growth, then take the 50 top firms and weight them equally? Weight them by market capitalization? Go long the top 50 and short the bottom 50? They can build it. And then they back-test it. Indexes that look good in hindsight have a shot to become ETFs. Those that don’t, don’t.

That approach allows fund companies to cherry-pick, in order to launch funds that can appear rosy. The problem with that, says Joel Dickson, senior ETF strategist at Vanguard, is that there’s no guarantee that the parameters and rules that would define an index you would create today would have been the same ones you would have used, say, five or 10 years ago.

“In 2008, you might construct an index that was tilted more toward value factors,” Dickson says. “And the back-test would show terrific outperformance. But in 2001, you would never have tilted toward those value variables in the same way.”

Nor has hypothetical past performance been an indicator of future success. Vanguard’s research shows that the average back-tested index outperformed the market by 10.3% per year in the five years prior to an ETF’s launch, then underperformed by 1% a year in the subsequent five years.

Now the authors quoted here aren’t making any claims of fraudulent activity going on in the ETF creation process. In fact we have had many conversations with smart people on the ETF side who are working on some great, innovative products and genuinely have their clients’ best interests at heart.  All the authors/we are saying is that investors need to be aware that the ETF creation process is a form of financial engineering and that just because an ETF is a passive investment doesn’t mean that tough questions don’t need to be asked.

5 comments

  1. […] Just because it is an ETF doesn’t mean it shouldn’t get significant scrutiny.  (Attain Capital) […]

  2. SUCCESS
    The scenario goes something like this:
    Newly emerging CTA with stellar performance and non-correlation.
    1. As AUM grows CTA finds that slippage increases in smaller commodities markets. Market allocations are revised on a case-by-case determination. Future performance including correlations incrementally drift from the prior “track record” and back testing process.
    2. Institutional investors shortly become the 800 pound gorilla. Large gorillas do not like volatility. CTA modifies leverage selectively on volatile markets. Performance decelerates away from the prior track record and back testing.
    3. Performance compresses because of migration from volatile markets and reduction of leverage. CTA does the math and finds that 2% management fee on $250 million is more than 20% performance fee on $50 million at a decompressed ROI 15 % performance. CTA focuses harder on volatility control.
    4. Future performance is further divorced from prior track record.
    Disclosure?

  3. When did the ETF market grow to $7.8 trillion (2nd paragraph)? I feel like Rip van Winkle – I must have been sleeping.

  4. […] The Attain Capital blog had a good post up last month on the “hard issues” facing anyone dealing with […]

  5. […] Here is a good article on some backtesting pitfalls as they apply to ETFs. […]

Write a Comment

Disclaimer
The performance data displayed herein is compiled from various sources, including BarclayHedge, RCM's own estimates of performance based on account managed by advisors on its books, and reports directly from the advisors. These performance figures should not be relied on independent of the individual advisor's disclosure document, which has important information regarding the method of calculation used, whether or not the performance includes proprietary results, and other important footnotes on the advisor's track record.

Benchmark index performance is for the constituents of that index only, and does not represent the entire universe of possible investments within that asset class. And further, that there can be limitations and biases to indices such as survivorship, self reporting, and instant history.

Managed futures accounts can subject to substantial charges for management and advisory fees. The numbers within this website include all such fees, but it may be necessary for those accounts that are subject to these charges to make substantial trading profits in the future to avoid depletion or exhaustion of their assets.

Investors interested in investing with a managed futures program (excepting those programs which are offered exclusively to qualified eligible persons as that term is defined by CFTC regulation 4.7) will be required to receive and sign off on a disclosure document in compliance with certain CFT rules The disclosure documents contains a complete description of the principal risk factors and each fee to be charged to your account by the CTA, as well as the composite performance of accounts under the CTA's management over at least the most recent five years. Investor interested in investing in any of the programs on this website are urged to carefully read these disclosure documents, including, but not limited to the performance information, before investing in any such programs.

Those investors who are qualified eligible persons as that term is defined by CFTC regulation 4.7 and interested in investing in a program exempt from having to provide a disclosure document and considered by the regulations to be sophisticated enough to understand the risks and be able to interpret the accuracy and completeness of any performance information on their own.

RCM receives a portion of the commodity brokerage commissions you pay in connection with your futures trading and/or a portion of the interest income (if any) earned on an account's assets. The listed manager may also pay RCM a portion of the fees they receive from accounts introduced to them by RCM.

See the full terms of use and risk disclaimer here.

Disclaimer
The performance data displayed herein is compiled from various sources, including BarclayHedge, RCM's own estimates of performance based on account managed by advisors on its books, and reports directly from the advisors. These performance figures should not be relied on independent of the individual advisor's disclosure document, which has important information regarding the method of calculation used, whether or not the performance includes proprietary results, and other important footnotes on the advisor's track record.

Benchmark index performance is for the constituents of that index only, and does not represent the entire universe of possible investments within that asset class. And further, that there can be limitations and biases to indices such as survivorship, self reporting, and instant history.

Managed futures accounts can subject to substantial charges for management and advisory fees. The numbers within this website include all such fees, but it may be necessary for those accounts that are subject to these charges to make substantial trading profits in the future to avoid depletion or exhaustion of their assets.

Investors interested in investing with a managed futures program (excepting those programs which are offered exclusively to qualified eligible persons as that term is defined by CFTC regulation 4.7) will be required to receive and sign off on a disclosure document in compliance with certain CFT rules The disclosure documents contains a complete description of the principal risk factors and each fee to be charged to your account by the CTA, as well as the composite performance of accounts under the CTA's management over at least the most recent five years. Investor interested in investing in any of the programs on this website are urged to carefully read these disclosure documents, including, but not limited to the performance information, before investing in any such programs.

Those investors who are qualified eligible persons as that term is defined by CFTC regulation 4.7 and interested in investing in a program exempt from having to provide a disclosure document and considered by the regulations to be sophisticated enough to understand the risks and be able to interpret the accuracy and completeness of any performance information on their own.

RCM receives a portion of the commodity brokerage commissions you pay in connection with your futures trading and/or a portion of the interest income (if any) earned on an account's assets. The listed manager may also pay RCM a portion of the fees they receive from accounts introduced to them by RCM.

See the full terms of use and risk disclaimer here.