Chartbreakers: How spammers are gaming the podcast charts

How did an unknown show about real estate beat out Serial, Dr. Death, and The Daily?



Something weird has been happening on the Apple podcast charts lately.

As we wrote in our last post, How to Break into the Top Ten, the Apple Podcasts US top chart is “the closest thing to the Billboard Top 100 in the podcast world.” The chart started looking really unusual on October 1st, when unknown shows made up most of the Top 100, but was reported fixed a few days later.



Case closed, right? Not quite. In this post, we'll look closely at one podcast—Bulletproof Real Estate—as an example of how charts can be manipulated, and I'll examine the follow-on effects that manipulation can have in the podcast ecosystem. [1]

Bulletproof Real Estate: A swift rise to the top

Bulletproof Real Estate with Andy Dane Carter launched in November 2017. You can check out a recent episode here. Though it's been around for a year, the show didn't appear on the charts until September 30th of this year, when it ranked #218 on the US All Podcasts chart and started climbing.

The show's host, Andy Dane Carter, noticed how well his podcast was doing. The morning of October 1st, he tweeted, “My #podcast went viral and made @itunes Top 10”. Around the same time, Iman Jalali from the Court Junkie podcast noted that Carter touted his rankings at a conference that weekend.

The show finally hit #1 for the first time on Sunday, October 7th. Here's what the chart looked like with Bulletproof at the top:

For comparison, here's how it looked just one week earlier—Serial was at #1, followed by familiar popular shows: Dr. Death, Happy Face, Uncover, and The Joe Rogan Experience.

Things really changed over the course of that week. While Bulletproof Real Estate was at the top of the chart, Joe Rogan dropped to #76, the lowest rank we've ever seen. Serial was in the lower reaches of the Top 10 after sitting at #1 for most of the last few weeks.



Bulletproof Real Estate stayed at #1 for a day or so. Then, at 6pm on October 8th, Apple took action, Bulletproof Real Estate started dropping down the charts, and over the next few hours, the charts went back to "normal," with Serial and Joe Rogan back on top.

Subscription suspicion

What happened? Did Bulletproof Real Estate “go viral” as its host suggested? Did Apple make some kind of mistake with their chart algorithm? Or was something else going on?



Though only Apple knows exactly how these charts work, Rob Walch from Libsyn has said that they are “100% about the total number of new subscribers in the past 7 days, with a weighted average for the last 24, 48, and 72 hours.” [2]



Apple does provide information on which podcasts are related to each other by subscriptions. Every podcast page on iTunes or in the Apple Podcasts app includes a section called “Listeners Also Subscribed To,” which looks like this for Serial:

The recommendations are pretty straightforward: People who subscribe to Serial also subscribed to Up and Vanished and Dirty John. Nothing surprising here.



Let's take a look at the same recommendations for listeners of Bulletproof Real Estate:

Notice anything? The top 3 recommendations are all from WARRIOR EMPIRE, a podcast network that also showed unusual movements around the same time. Now, if we look at the recommendations for listeners of the fifth podcast in that list, According to Sources, we see something interesting:

The shows recommend each other, plus the same 5 other podcasts. It seems really unlikely that the same listeners are listening to all of these shows, and catapulting them all to the top of charts. What's really going on here?



To find out, I took a closer look at how these shows are related. First, I grabbed Bulletproof Real Estate's recommended podcasts, and then their recommendations, to create a map of their relationships:



Every box on the graph represents a podcast, and every arrow represents a recommendation from the “Listeners Also Subscribed To” section.



We can see that many of these podcasts recommend each other, with many connections to shows like The Man Made and Are You Hearing This?, along with many shows helpfully titled in ALL CAPS from WARRIOR EMPIRE, like LIFE WITH THE CHAPMANS.



But does interconnectedness really mean that something fishy is going on? Maybe listeners of WARRIOR EMPIRE shows are an unusually loyal bunch, and subscribe to all shows from the network. Or perhaps people interested in Why That's Funny, a show that describes itself as “two guys from New Jersey just hanging out,” have a sincere interest in Bulletproof Real Estate.

The shady bunch

To find out, I wanted to see how this cluster of podcasts related to other top shows on the charts, like Serial and Joe Rogan. I grabbed them all for the top 50 podcasts and made another network graph:

Again, every box on the graph represents a podcast, and every arrow represents a recommendation. The chart easily breaks into four clusters, and we can draw some quick conclusions from them.



First, there's one “main cluster” that includes most popular shows. You can see some natural sub-clusters—for example, one sub-cluster around Joe Rogan includes similar talk shows; another around Someone Knows Something includes true crime shows.



Clusters 1, 2, and 3 are completely disconnected from the main cluster. There are zero recommendations in common between them. Bulletproof Real Estate lives in Cluster 1. You can see by the density of connections that the isolated clusters also have many more connections between the shows than even the most popular sections of the main cluster.



The isolated clusters are highly interconnected, but with very different subject matter. For example, Breaking the Underdog Curse for Chiropractors is related via subscriptions to many podcasts from both Clusters 1 & 2, but has little in common with them in terms of subject matter. The same goes for shows like Winning with Shopify, an ecommerce podcast, and This is Hot Bowga, "home of THE greatest hunting podcast ever created," in Cluster 3.



So, what can we conclude from this network graph? Here's my take:



If the podcast charts are based on subscription velocity, it's highly likely that some or all of the podcasts in the isolated clusters have artificial subscriptions. We can't be certain that every podcast in each isolated cluster is subject to manipulation. But the combination of highly interconnected shows, plus isolation from the “main cluster” of popular shows, plus disparate subject matter suggests that something deeply strange is happening across a sizable number of podcasts. It's been suggested that click farms are used to artificially boost subscriber counts. [6]

How to spot a fake

If the charts feature shows that don't belong, then how do you spot a fake? It turns out that popular podcasts have many things in common that are both easy to see and difficult to manipulate:



Popular episodes

Truly popular podcasts have episodes on the top episode charts. In fact, the Apple Podcasts top episode charts usually include multiple episodes from the most popular podcasts like Serial and The Joe Rogan Experience. The highest rank any episode of Bulletproof Real Estate achieved, even while it sat at #1 of the Top Podcasts chart, was #271. Compare that with Serial, which at the time of writing has 5 episodes from its latest season in the top 50.



Popular on multiple charts and in multiple countries

Most popular shows are not popular just in the US, or just on a single chart—they're popular around the world, and are also popular on their associated genre charts. For example, The Argument from the New York Times launched on October 4th, and currently appears on 53 separate country and genre charts. Bulletproof Real Estate appears on just 5 charts.

Many ratings and reviews

Popular podcasts usually get ratings and reviews in a few predictable patterns: we see big bursts for new launches and new seasons. For example, Dr. Death made waves for gathering so many reviews upon its launch. Most shows then gradually gather more ratings and reviews over time.

Bulletproof Real Estate has only gotten a few more ratings since reaching #1, going from 43 to 53 over more than a week, and has just 6 reviews.



Popular on multiple podcast players

Non-Apple podcast players like Castbox, Breaker and Podcast Addict show statistics on how many of their users subscribed or listened to a particular show or episode. As of Friday, Bulletproof Real Estate had just 4 subscribers on PodcastAddict. In contrast, The Argument, despite only being live for a week, had 172 subscribers on Podcast Addict. [3]

What now?

Clearly, someone at Apple is aware that there's an issue and has taken steps to fix it. So should we still worry about the charts?



While manipulation may be less prevalent than it was last week before Apple took action, it looks like it's getting worse once again. Bulletproof Real Estate dropped to #95 on October 6th, but since then has slowly climbed back up to #20. And Entrepreneurship Stories 4️⃣Inspiration, whose logo features Chuck Norris sporting shutter shades, is currently sitting at #5 on the chart, having whipsawed in and out of the top 400 many times.



Should we care if the Apple charts are being manipulated? As Nick Quah summarized on Twitter, “Lots of folks have long operated under the pretense that the charts are unreliable,” and “a good deal of podcast discovery isn’t happening on the charts anyway.” There are reasons for concern, as Quah went on to write in last week's Hot Pod:

An Apple Podcast Charts that doesn't work the way it should might have limited consequences for bigger shows and publishers...but it disproportionately impacts smaller or newer teams... Even if the actual number of people checking out the charts is generally low, it's still a meaningful number to that class of show.



Are substantial decisions actually made based on the charts? Probably not, as far as major podcast publishers, advertisers, and various business folk are concerned... But the ones who do make decisions based on them are the ones the industry needs: those dipping their toes into podcasting waters for the first time.

I agree with Quah that the charts are an entry point for people new to podcasts, like the 74% of Americans who don't listen to podcasts on a monthly basis [4]. Like it or not, most people listen to podcasts through Apple's app, and the charts are one of the only discovery tools available to them.



Moreover, Apple isn't just one player among many in the podcasting ecosystem—the Apple Podcasts directory is the closest thing to a centralized podcast database that this industry has. Plenty of apps like Overcast and Castbox (and Chartable, of course) rely heavily on Apple's directory. Castbox's discovery feature actually mirrors the charts exactly, which means that ranking highly on Apple's charts can drive subscribers and plays to podcasts on Castbox. For example, bulletproof Real Estate managed to reach over 8,000 subscribers on Castbox within a week of hitting the Apple charts. [3]



Castbox represents just under 3% of global downloads compared to Apple's 64% [5], yet it still drove thousands of subscribers to Bulletproof Real Estate. That means reaching the top of Apple's charts must deliver even more subscribers. More subscribers means more downloads, and because podcasts sell ads based on download numbers, that means there's a significant monetary incentive for podcasters to game the charts.



Ultimately, as long as Apple dominates podcast consumption, the Apple charts will continue to play an outsize role. And as long as they drive downloads, there are huge incentives for podcasters to game them. James Cridland sums it up nicely in the conclusion to his editorial:

... An actual “chart” is sorely needed. A way to compare your own product with someone else’s is a powerful creative stimulus — just look at follower counts on Twitter. Podcasting, particularly, needs a chart.



The question is whether we can get a truly global replacement for the Apple Podcast Chart — one less susceptible to being cheated, one more representative of the industry as a whole, and one that is just as accessible for passion podcasters as for VC-backed podcast publishers.

We're interested in building the next generation of podcast charts. Want to join us? Please get in touch—email info@chartable.com or follow us @ChartableDotCom on Twitter.

If you liked this post, please consider sharing it on: Facebook · Twitter · LinkedIn · Via email