At Opensignal we have always prided ourselves on our ability to measure what the consumer experiences on mobile networks . We measure the real-world speeds consumers see on their smartphones, rather than the highly optimized connections typically reported. We track how often real users have access to 4G signals in the places they frequent, rather than using complicated models to predict coverage. We run our latency tests not to specialty test nodes, but to the servers from which consumers seek their content. Opensignal's philosophy has always been that the only metrics that matter are the ones that measure the true consumer experience. The rest are just ephemera.

Real-world speed and real-world availability are very useful for gauging the power and reach of an operator's 4G services, but speed and availability are still relatively abstract concepts. Consumers don't consume speed. They don't occupy themselves with availability. Instead, they're surfing the mobile internet, chatting on social media and streaming video. That's why we've decided to take another step forward in our approach. In addition to tracking the underlying metrics of the mobile networks, we've begun analyzing how the consumer experiences the apps and services they use on their mobile phones. We're kicking off this new forward-looking approach with an analysis of video experience. It seeks to answer a simple but enormously relevant question for modern mobile consumers: How good or bad does video render on my operator's network?

A first-of-its-kind measurement in the mobile industry, Opensignal's video experience metric is derived from an International Telecommunication Union (ITU)-based approach for measuring video quality. Our tests sample video at multiple resolutions accessed from multiple content providers, and they weigh three main criteria: the load time before the video begins playing, the stalling rate characterized by stops and stutters in the video playback, and the level of picture resolution. We measure video experience on a scale from 0 to 100 — the higher the score, the better the video experience. Furthermore, we divide those scores into ranges to determine their rating. A score that falls within 75-100 is Excellent, 65-75 is Very Good, 55-65 is Good, 40-55 is Fair and 0-40 is poor. An Excellent score is a big achievement indeed, meaning fast load times and practically non-existent stalling at all resolutions. As we descend down the ratings, loading times get longer and we encounter more stops and stutters in the video stream. At our worst rating, Poor, users don't have a good experience at all with very long waits for the video to start and frequent stalling even at the lowest resolutions. (For more details on our video experience metric see this blog post.)

For our analysis we examined 69 countries spread throughout the globe to see how they stacked up in video experience. We also ranked those same 69 countries in overall download speed and then plotted speed against video experience to see if we could define a relationship between the might of a country's mobile broadband networks and the quality of video customers see. We discovered that the relationship is a complicated one. Where mobile broadband connections are slow, speed has a big impact on video experience. But at faster levels, speed has little relation to the quality of video streaming. The countries with the fastest speed don't necessarily offer the best video experience.

In search of the perfect mobile video stream

As you can see from our overall video experience chart, no country falls into the Excellent (75-100) category for average video quality. An Excellent score would indicate that a country's operators are able to consistently support high-resolution video with fast loading times and nearly non-existent stalling, across their networks. 4G networks have come a long way since first introduced nine years ago, but the fact that no country in our list was able to meet the highest benchmark for video quality shows just how far mobile technologies need to progress.

Even the countries with the most sophisticated LTE networks had to settle for a Very Good (65-75) rating in our video measurements, and many countries renowned for their 4G services didn't even make that list. The Czech Republic topped our chart with an overall video experience score of 68.5, and only 10 other countries joined it among the video experience elite with a score of Very Good. Most of them were in central and northern Europe, while the United Arab Emirates and Singapore represented the Middle East and Asia. Few users in these countries would find fault with the video quality they received as a Very Good score generally means fast loading times and only occasional stalling. What primarily distinguishes a Very Good score from an Excellent score is less consistency in the video quality from different video sources and at different resolutions.

The vast majority of the 69 countries we examined fell into a relatively narrow range of scores between 40 and 65, earning them either Fair or Good ratings. So what does that mean exactly? It means for much of the world, the typical mobile video experience leaves something to be desired. Video load times are sluggish; stops and stutters mid-stream are common to varying degrees; and connections often have trouble coping with higher-resolution formats, especially for those countries in the Fair category. In general, European countries tended to rank higher than their counterparts in the Americas, while Asian and Middle Eastern countries are scattered throughout the rankings.

Finally, three countries, India, Iran and the Philippines, fell below the Fair threshold into Poor territory (0-40), where the typical consumer experience is characterized by frequent stalling during video playback and long loading times even for low-resolution video.

Why speed isn't everything

For the second part of our analysis, we calculated overall download speeds in the 69 countries we examined to compare against our video experience metric. Overall download speed takes into account both average 3G and 4G speeds and the availability of each type of network connection, and is intended to show the typical download speeds consumers experience on an everyday basis. As you can see from the chart, there is a large amount of variation in the overall speed scores of different countries — some pushed the 40 Mbps barrier, while others languished below 10 Mbps. But you'll also notice the rankings in overall speed are much different than the rankings for video experience, especially at the top end of the chart.

Things get really interesting when we plot overall speed against overall video experience in our Full Spectrum of Mobile Video chart. Here we clearly see there is some relationship between average speeds and video quality as our video experience scores generally improve the faster speeds get, but it's not an exact correlation. South Korea is the prime example. With an overall download speed of 45.6 Mbps, it was the fastest of the 69 countries by quite some margin. Yet South Korea was well short of the top mark in video experience. It didn't even make the cut of 11 countries in our Very Good ratings. Rather it landed in the Good tier, right alongside several countries with nowhere near the sheer mobile broadband might of the East Asian 4G powerhouse. Kuwait's average download speed was a mere 14.7 Mbps, but it was nearly level with South Korea in video experience. Clearly a good video experience is dependent on more than a lightning-fast mobile connection.

Where we see the biggest correlation between speed and video quality is among the countries with slower mobile connections. Nearly all of the countries that had Fair video ratings fell within a tight cluster below the 14 Mbps overall speed mark, and in this group a small increase in overall speed often coincided with a sharp climb up the video experience scale. At the lower end of the table, speed clearly is a big factor governing the quality of a video stream, and in those slower countries even a small boost in speed can have a big impact on video experience.

But right around the 15-Mbps threshold, our plot curve starts leveling off and the distribution of countries along that curve becomes far more random. That means the tight relationship between video experience and download speed becomes much more loose as average connection speeds climb higher. Beyond the 20-Mbps threshold, speed had relatively little say in the matter of how good video experience actually was in our measurements. Of the 11 countries that scored Very Good ratings, their overall download speeds ranged from 21 Mbps to 40 Mbps. The country with the best video experience score in this analysis, the Czech Republic, didn't even make the top 10 list in speed.

So if raw average throughput isn't determining the quality of experience in faster countries, then what is? Latency, which measures the response time of a network, has a big impact on our video experience metric, as the longer a device waits to hear back from the video server, the longer it takes for the video to load and begin playing. Consistency of connection speed is also an important factor. A super-fast connection isn't necessary to stream video over mobile networks, but a video needs relatively consistent throughput to avoid stalling. Thus a network that delivers a 50-Mbps connection one second and a 2 Mbps connection the following second is likely to provide an inferior overall video experience than a network that can maintain a constant 20 Mbps connection over a long duration of time. Maintaining consistent speeds isn't an easy thing for a mobile operator due to the nature of cellular networking. Unlike wireline broadband, where subscribers often have a dedicated line to the network, mobile networks operate through shared capacity. As the number of users and data demands change on any given cell, the mobile network is constantly redividing and redistributing its available capacity, which can create large fluctuations in individual connection speed from one moment to the next.

Video experience can also be heavily impacted by operator policy. Many operators globally use video optimization technologies to restrict the level of video resolution their customers can access on their phones. As our tests sample video at different resolutions, any downgrading of video quality — say from HD to SD — would have an impact on our scores.

The U.S. is a prime example of such policies at work. In our scatter plot chart, the U.S. is an outlier. It lands in the middle of the Fair range of video experience with a score of 46.8, but its average overall download speed of 16.5 Mbps suggests it should have scored much higher in video experience. Almost every other country with an overall speed of 16 Mbps or greater earned a Good video experience rating in our measurements. The explanation lies in the evolving nature of data plans in the United States. Unlimited plans are gaining huge ground, and to prevent their networks from becoming overloaded with video traffic, operators have put streaming restrictions on their different tiers of unlimited plans. For many users on unlimited plans in the U.S., the highest resolution video they can stream over a mobile connection is 480p.

So it stands to reason that if an operator lifts restrictions on video resolution, then its video experience scores would improve, right? Not necessarily. Depending on the type of video, a 720p stream can consume twice as much or more data than a 480p stream. And as video now accounts for the majority of all mobile internet traffic, a doubling of the gross tonnage of video consumption would have a major impact on any operator's network. More traffic leads to congestion, and congestion not only impacts overall speeds available to consumers but can also lead to inconsistent connections and poorer latencies — all of which have a bearing on video experience. Ironically, if U.S. operators were to lift video restrictions on their unlimited plans, it’s possible that the typical video experience in the U.S. might suffer more.

The future of mobile video

Speed has been the focal point of the mobile industry ever since the first LTE network came online nearly a decade ago. But speed is just an abstract measurement, not an indication of how consumers actually experience the internet or applications on their mobile phones. Speed certainly has a bearing on mobile video experience, but as this report shows, it's not the sole factor determining the quality of video we see in countries with sophisticated LTE services. We would argue that it's time for the industry to turn its attention away from raw speed and focus on the combination of factors that go into ensuring a good mobile internet experience. Video is the ideal place to start as it depends on so many elements: the availability of 4G connections, the latency of the network and the consistency of connections, as well as the individual data-shaping policies of operators.

Judged by the user experience rather than rote measurements, the mobile industry still has work to do when it comes to video. Countries with extremely powerful LTE networks in terms of download and upload speeds, availability and coverage, aren't always providing the best video experience. And no country we've analyzed has managed to achieve the highest level of video streaming quality. If operators want to boost the video experience they offer their wireless customers, building faster and faster LTE-Advanced networks isn't necessarily the answer.

It might well be the case that the best quality video experience may not come from a 4G network. 5G is often touted for its gigabit speeds, but more important for video are 5G's purported low latencies and high-density network architecture, which would respectively cut down on video load times and create the resilient connections necessary for consistent video playback. There's no question that 4G revolutionized mobile video, but in order for mobile video to reach its full potential, we may need to wait for 5G.