Over the last few years, Netflix, YouTube and others have been pushing Media Source Extensions (AKA MSE) as an HTML5 video API for JavaScript (and Dash). This has been a significant development for HTML5 video, after years of domination by Flash players. Now, this year, Javascript (JS) is driving the delivery of the majority of the video on the web. Why did this finally happen and why is it important?

You can say it happened because Flash is finally dying. Well, that’s partially true. But even without Flash, you don’t necessarily need JS. Many websites use HTML5 video tags with an MP4 or WebM source. It’s fast, it’s easy and no extra code is needed. So why do we need JS?

The answer is summarized in one word - control. Video applications can no longer rely on generic implementations. They need finer grained control to deliver carefully-crafted video experiences on every device, on any network.

Here are the 5 main benefits of the switch to JS-based delivery of video:

1. Cross platform consistency

Many content providers broadcast on a variety of platforms -- desktop, iOS, Android, Roku, Apple TV, PS4 and more. Without JS, they would have to support the “native” implementation of each platform, which has a significant amount of overhead. Some mobile devices are notorious for their HLS implementation for instance. Not only that, even every browser version can be different, forcing developers to play nice on all of the different creatures. Video.js is an interesting example of a player that took it one step further -- they built the player functionality in JS even when Flash is responsible for the decoding.

2. Robust, updatable, Adaptive Bit Rate algorithm

Adaptive bit rate (ABR) is now the standard for premium content delivery. But not too long ago it was about to be baked into the browsers. Luckily Google pushed hard to build a JS API and implement the minimally-required layer in the browser. Aaron Colwell explained that to me a few years ago and he said that he designed the MSE API to be as flexible as possible. His perspective came from the pain he experienced while working at REAL Networks on hard-to-update ABRs.

You want to provide different experience for mobile devices? Easy. Deliver a different JS for mobile. You found a bug in the rendition switching? Don’t panic. It can be replaced immediately since the latest JS is used each time. It’s the web you know. And while we’re using the power of the web, we can easily create A/B tests on the ABR algorithm to see what works best.



3. Buffer management

With JS and MSE, you take full control of the buffer. You get to decide what, when and how much video to buffer. You can set a buffer of 5 seconds, or a buffer of 30 seconds, based on your bandwidth policies. One can even change the buffer size dynamically based on the user agent, network speed or even based on whether a user is a premium subscriber. These are pure applicative decisions that a browser can’t make.

You can also choose to load content in a nonlinear manner, which makes sense for some use cases. For example, let’s say you have annotations on the seek bar, or a button to fast forward 5 min ahead. With JS you can relatively easily build logic that prefetches these video segments so the user can play them instantly.

4. Multiple slots, multiple servers

Native implementations usually load the video sequentially from the source (server). With JS and MSE, a developer can create multiple connections to the server, which can sometimes increase the download speed. More interestingly, multiple servers or CDNs can now be used. For example, a video player can use two sources, “fast-and-rich” and "slow-and-cheap". JS will load the first bits fast to ensure a very good user experience. Once enough seconds are buffered, it can then load the following segments from the slow server and save some dollars on the CDN.

5. Use of other protocols - WebRTC, WebSockets or a combination

This is where the fun begins. By using JS, not only can the video player use multiple sources at the same time, it can also use sources that are not HTTP. WebSockets, WebRTC, Filesystem API, you name it.

It’s even possible to use HTTP (CDN) for the first segments and P2P for the rest. That’s how our Peer5 solution operates. We build a JS client that knows what segments to fetch from the server and what segments can be delivered from peers. It’s flexible and configurable so the content provider do whatever they want, with the idea that they want to deliver the best user experience.

The video world is finally moving to Javascript delivery. And thankfully, it is not just because of Flash losing steam. There are some substantive technical reasons why it is a good thing.

Interesting reads:

dash.js, an MPEG dash open source project with ABR - https://github.com/Dash-Industry-Forum/dash.js/wiki

Building an MSE player http://blog.wirewax.com/building-a-media-source-html5-player

Fantastic overview from JW about the state of HTML5 http://www.jwplayer.com/html5/

EME https://en.wikipedia.org/wiki/Encrypted-Media-Extensions