How Morningstar can Fix their Rankings

In today’s world of Theo Epstein’s using advanced statistical metrics to arrive at the best players to win a world championship, you would think there’s just as much thought given to everyone’s favorite mutual fund metrics – the Morningstar rating. After all, there’s billions and billions of dollars basing their decisions to invest in this mutual fund over that based on those rankings. Only, they don’t do such a good job of helping find the best players for your future team. At least that’s the claims The Wall Street Journal are making about Morningstar’s ranking system this week.

The article gives example after example of pensions and financial advisors investing into funds with a five star rating from Mornginstar, only to see the performance dwindle afterwards. There are quotations of financial advisors admitting to recommending funds to clients purely based on the star rating; there are charts showing funds not staying at five stars over time; and there are charts showing investors pulling money from low rated funds while performance is actually improving. Here are some interesting bits of information from the long form article:

Of funds awarded a coveted five-star overall rating, only 12% did well enough over the next five years to earn a top rating for that period; 10% performed so poorly they were branded with a rock-bottom one-star rating.

Morningstar groups funds into categories based on their investing style or area, more than 100 groups in all. It compares funds not to all other funds, nor to the overall market, but to other funds with the same investment focus. The top 10% of funds in each group receive five stars, the bottom 10% get one, and the rest get two, three or four stars.

The ratings don’t reflect raw performance, but performance adjusted for funds’ degree of risk. To make that calculation, Morningstar uses an algorithm Mr. Mansueto devised that reflects the variation in funds’ month-to-month returns.

To Morningstar’s defense there is quotation after quotation from Morningstar ensuring that these rankings shouldn’t be used as an end all be all but as a way to narrow down a list of potential investments (despite the fact that there’s story after story suggesting that’s the what financial advisors use it for). The entire premise of the Wall Street Journal Investigation is to say that Morningstar’s system is bad at predicting how mutual funds will perform in the future. To that, we say duh. They can only rank funds based on past performance. If their statistical data could predict futures results, everyone would be rich. But alas, it seems the star ranking matters more to an investor and advisor than the returns themselves. Example A:

MorningStart five star funds

So, is it Morningstar’s fault for not being explicitly clear that these stars are just a starting point when looking to invest, or is it the financial advisors and investors fault for using the stars as the only due diligence before investing? We say both. Morningstar clearly knew what was happening despite their warnings, and built their brand as the place billion and billions of dollars go to find an investment. However, Financial Advisors are also the blame, with it clear from the numbers that they should have done better due diligence. Ultimately, they trusted a brand to do their jobs for them.

Morningstar doesn’t have a crystal ball, even if they claim their top star programs “likelier to outperform in the future.” After all, past performance is not necessarily indicative of future results. And doing these types of rankings are hard. As we say in our Managed Futures Rankings:

It’s trickier than it looks. Put too much emphasis on returns, and you penalize those who control risk. Too much emphasis on experience and you penalize a potential new star. Too much reliance on the present and you discount the past, too much on the past and you discount the present, and so on.

You see, if not for any rankings, you would have these advisors likely just going to the best performing fund last year. The rankings aren’t all the predictive, but neither is the de facto ranking of performance only. It would be interesting to see the Morningstar rankings alongside a simplistic look at persistence among the top quartile funds, or how the top ranked five star funds do against their benchmarks.  And would be great to see Morningstar sort of Epstein their rankings to incorporate more data and more time frames. Here’s how we do that in our ranking methodology:

We measure the programs across different metrics related to return, risk, correlation levels, and length of track record. Next, we time-weight the numerous statistics, evaluating each metric across 1, 3, 5, and 10 year time periods in addition to the full length of the program since its inception. This focus on varying time frames ensures that great returns far back in a program’s track record don’t skew their ranking, and, likewise, that newer programs that haven’t “lived through tough times” don’t dominate the rankings.

But rankings are only as good as their ability to identify funds/programs which can outperform their benchmarks – be it on returns or risk, or risk adjusted returns. So how have our rankings done in that regard?  Here’s a look at an equal weighted portfolio of the Top 15 as displayed in our February 2014 rankings, and how it has performed against the benchmark BarclayHedge CTA Index over the 3 years following those rankings release:

Managed Futures Rankings(Disclaimer: Past performance is not necessarily indicative of future results).

So what do you say Morningstar… should we partner up on a new, more advanced, way to do your rankings? Two Chicago companies working together to meet the criticisms levied by the WSJ piece and launch a new ranking methodology to meet the new statistical/quant age we now live in. Helping advisors instead of hiding behind the disclaimer that these aren’t really supposed to be used to base allocation decisions on. Call us at 855-726-0060 to talk. And check out how we do it by downloading our Managed Futures Rankings here.

The performance data displayed herein is compiled from various sources, including BarclayHedge, and reports directly from the advisors. These performance figures should not be relied on independent of the individual advisor's disclosure document, which has important information regarding the method of calculation used, whether or not the performance includes proprietary results, and other important footnotes on the advisor's track record.

Benchmark index performance is for the constituents of that index only, and does not represent the entire universe of possible investments within that asset class. And further, that there can be limitations and biases to indices such as survivorship, self reporting, and instant history.

Managed futures accounts can subject to substantial charges for management and advisory fees. The numbers within this website include all such fees, but it may be necessary for those accounts that are subject to these charges to make substantial trading profits in the future to avoid depletion or exhaustion of their assets.

Investors interested in investing with a managed futures program (excepting those programs which are offered exclusively to qualified eligible persons as that term is defined by CFTC regulation 4.7) will be required to receive and sign off on a disclosure document in compliance with certain CFT rules The disclosure documents contains a complete description of the principal risk factors and each fee to be charged to your account by the CTA, as well as the composite performance of accounts under the CTA's management over at least the most recent five years. Investor interested in investing in any of the programs on this website are urged to carefully read these disclosure documents, including, but not limited to the performance information, before investing in any such programs.

Those investors who are qualified eligible persons as that term is defined by CFTC regulation 4.7 and interested in investing in a program exempt from having to provide a disclosure document and considered by the regulations to be sophisticated enough to understand the risks and be able to interpret the accuracy and completeness of any performance information on their own.

RCM receives a portion of the commodity brokerage commissions you pay in connection with your futures trading and/or a portion of the interest income (if any) earned on an account's assets. The listed manager may also pay RCM a portion of the fees they receive from accounts introduced to them by RCM.

See the full terms of use and risk disclaimer here.