If you’ve been following the stock market a little, Jim Cramer cannot be a total stranger. He’s been holding his daily show, Mad Money, for more than 10 years. With an opinion on almost every stock, the guy is a polarizing figure.

His past performances have also been a subject for debate and the focus of many articles. Unfortunately, most of those analyses are either outdated or fall short in terms of rigorous investigation. An astonishing example in this CBS news article on Jim Cramer wrongly predicting four different stocks:

Being a numbers guy, I couldn’t resist calculating the odds of making four sell recommendations on what ends up being the four best performers out of 749 different stocks. Can we have a drum roll? The odds are 1 in 13.1 billion. […] Thus, picking the four best performers as stocks to sell is the next closest thing to being statistically impossible.

The odds are actually 1 out of 16. But not only is the math totally wrong, the process in itself is very disputable. The journalist arbitrarily choose a four best performers criteria. But what about the ten best, or the top 15 ? No mention of those. They probably did not make Cramer look as bad as the top 4. If CBS likes statistics, what are the chances that you can ridicule anyone who’s making countless recommendations by carefully picking some criteria ? Quite high I suppose.

At cresus.io, we believe in numbers and we’ve decided to put Cramer to the test in a detailed and unbiased analysis of a year of Mad Money stock picking. The article will span across different posts, to make sure we can address each aspect thoroughly.

Dataset

The historical stock picking data is publicly available on Mad Money — CNBC.com. With an average of 20 recommendations per show and a past horizon of a year, the stock picking corpus is rich of 4115 picks. All picks of this study lie in a timeframe between 10 June 2014 and 9 June 2015.

Each pick is assigned to a segment, informing on the context of the recommendation. Two of them stand out in terms of volume. The discussed segment is Cramer’s opportunity to talk about stocks of his choosing. The lightning round is where viewers of the show can call Cramer directly and seek advice on any ticker.

If you were to consider a strategy of buying a fixed amount worth of a stock that just got a positive review from Cramer, and to sell it all on a negative mention. You would end up with the following portfolio. AAPL is the king of the hill with an impressive 58 consecutive recommendations, accounting for almost 13% of the total value.

Yearly performance

As a first experiment, let’s imagine a faithful Cramer follower cautiously listening to his guru, with a similar strategy to the one described earlier except for the selling part.

He starts with a fixed capital fully invested in a safe index, the S&P 500. Any time the guru discusses a stock as being a buy, he sells some S&P 500 shares to invest a fix amount in the recommended stock. 100$ for a buy rating, 200$ for a strong-buy rating. Let’s see how well did our Cramer enthusiast do over the last year.

As time passes, the Mad Money portfolio starts to diverge from the S&P index. The investor’s portfolio progressively moves away from an all S&P investment and gets more exposition to Cramer’s recommendations. He’s doing quite well up until the end of June 2015, finishing the year just above the index. However, it should be noted that a real-world example would suffer from the cost incurred by the high number of transactions over the year.

As an element of comparison, the Anti-MadMoney portfolio is presented. It follows a strategy where the investor only buy stocks that get a sell-rating by Cramer. As you can see, a disaster in terms of performance. This could hint to a possible shorting strategy.

Lightning Round

During the lightning round, Cramer receives live calls from auditors. They get to ask about any stock they like, small cap, large company, it does not matter. They seek advice and Cramer always has some. But how do these recommendations pan out in the future? Let’s first start by having a look at the monthly performance (in percent) of each stock asked in the lightning round, and arrange those into a histogram.

Predicting stocks in the lightning round is a risky business. Even if a good amount of the 1-month performance lies around 0, the standard deviation is as high as 13%. As you can see on the plot, a gaussian distribution seems like a good fit to model the probability distribution of the 1-month performance. This means that if you were to flip a coin any time an auditor asks about a stock, that’s the kind of distribution you would get from your recommendations. A good Guru should be able to move the mean of the distribution to the right, while maintaining or reducing the standard deviation. That’s what the following graph is highlighting. Each stock that receives a strong-buy rating from Cramer gets accounted for in the green curve. Each stock that gets a strong-sell rating goes into the red curve. The blue curve is the element of comparison and account for all stocks.

Two takes from this graph. The first one, is a certain risk aversion from Cramer. The stock that he classifies as strong-buy get a significantly lower standard deviation (8.57%). He is also able to move the mean performance above the 0% threshold, beating our previous coin-toss experiment. We also have to give him some credits on his strong-sell recommendations, that end up having a poor average performance.

Just for fun, let’s have a look at the outliers in those distribution. Here are is a small compilation of top-3, flop-3 strong-buy recommendations from Cramer:

Symbol Date Performance CY 11 November 2014 38.78% UVE 17 October 2014 37.43% LCI 2 October 2014 36.11% […] SD 11 September 2014 -26.90% FSLR 11 September 2014 -27.76% CLDN 20 March 2015 -36.81%

To be continued…

We have only scratched the surface of the analysis of Cramer 2014-2015 performance. A year worth of show represent a very rich corpus, from which we can gain good insights on what happens behind the scenes. Many more questions still need to be answered.

So far, we’ve seen that the challenge of stock picking is a hard one. The market is volatile and unpredictable. In no way one can judge Cramer on a few picking mistakes. Only a bulk analysis can shed some light. Our fictional Cramer-aficionado had a pretty good year listening to his guru. But his return is too close from the S&P baseline to draw any definite conclusions.

Cramer’s biggest merit is that he actually has an opinion. His clear-cut takes on the market generate debate and make Mad Money a great source of information for any investor, as long as he can keep a critical stance.

Whether positive or not, Cramer, with his hundred of thousands of viewers, probably has an influence on the market. If you want to be notified when a stock you’re following appears in Mad Money. Create your portfolio on cresus.io and get daily insights about your investment. It’s free.