At the beginning of this year, we officially turned off Flash support for VHS, the New York Times video player. We now use HTML5 video technology for all video playback on desktop and mobile web browsers. Flash was a very powerful and popular technology in its day but it has waned over the years as browsers have embraced the open standard of HTML5. Throughout the second half of 2015, Chrome, Firefox and Safari also began blocking the Flash plugin from automatically loading content unless users gave their permission. In order to continue providing a quality video experience for our viewers, switching exclusively to HTML5 video became necessary.

Background

Video has become a critical part of storytelling at The New York Times, and it has seen tremendous growth over the past few years. With this growth, as well as a new company-wide focus on video, we decided to build our own video player last year rather than continuing to use third-party solutions. There were many reasons for the move, but primarily we wanted to own the entire video experience in-house in order to focus on our needs and values. Performance, premium video quality and ability to customize the experience were our primary concerns. With video used in both articles and in unique, custom experiences, we have a diverse set of requirements. We wanted to give the newsroom a flexible, extensible player with a core foundation so they could focus on creating the experience. With this aim, we built a JavaScript wrapper around a video element that could either be an HTML5 video element or a Flash OSMF video object. For extensibility, we included a light, event-driven plugin system.

Why Flash?

When we first developed the player, Flash was still the dominant technology for publishers. This was largely because most of the ad inventory was still in Flash. VPAID ads were almost exclusively created with Flash, and advertisers preferred to deliver their ads in a VPAID wrapper so they could add their own tracking.

Another reason we needed Flash was to play legacy content. Most of our older content (stretching back to 2006 on our earlier website) was encoded with On2 VP6, which requires Flash.

Lastly, we needed Flash in order to support our live content for breaking news and live events. Our live feeds are delivered using HLS Adaptive Streaming, so for browsers that don’t natively support HLS, we used the Flashls project to support HLS via a Flash OSMF player.

For newly encoded video that wasn’t ad-supported and not live, we were free to use HTML5 video. This allowed stories that weren’t ad-supported to take advantage of the performance we were getting from HTML5 video while we could still support the business. These HTML5 experiences include our more visual story experiences where video enhances a larger feature story like in “Ten Years After Katrina” or the video for “Bieber, Diplo and Skrillex Make a Hit.” In these instances, we found HTML5 video more performant and easier to work with for the customization the newsroom needed.

How We Got There

As we saw the industry move toward HTML5 video, we knew we wanted to get ahead of it. The browsers blocking Flash really hastened our efforts. We needed to support legacy, VPAID and live content with an HTML5 player to give our users a seamless video experience.

Legacy Content

To support our legacy content, we re-encoded and republished all of our old content, creating both MP4 and WebM renditions with different resolutions and bitrates. We also created HLS renditions with six different levels, enabling us to serve our on-demand videos with adaptive streaming. This was a large effort that we were only able to do with the support of the business.

HTML5 VPAID

To support HTML5 VPAID content, we updated our ad integration and made our player VPAID 2.0 compliant for HTML5. Then we approached our ad vendors and requested HTML5 VPAID ads to start the certification process. We had some hiccups along the way, but the vendors worked with us. We both made adjustments to our code to make everything work as required to give users a good ad experience.

We needed to be able to resize and move the VPAID ads so we wouldn’t get any collision between our controls and any VPAID controls within the ad. The volume and mute controls didn’t work across the board; however, this was fixed.

We also ran into playback state synchronization issues with some HTML5 VPAID ads. If a user paused or played the ad with the in-ad controls but the ad didn’t trigger the AdPaused and AdPlaying VPAID events, then the player didn’t accurately reflect the state. The VPAID specification only required those events to be called in response to method calls to pauseAd and resumeAd:

VPAID 2.0 Spec Excerpt – Section 3.3.17 Page 31 – “The AdPaused and AdPlaying events are sent in response to the pauseAd() and resumeAd() method calls, respectively, to confirm that the ad has either paused or is playing.”

Since the ad can be paused or resumed from various vectors in addition to the pauseAd() and resumeAd() functions, this leaves no way to ensure synchronization of the playback state between the player and the advertisement. We recommended to the IAB Video Group that usage of the AdPaused and AdPlaying events should be extended. The events should be triggered when the playback state of the ad changes — regardless of what function initiated the change of state. This way, a video player can confidently maintain playback state with a VPAID ad. We’re hoping that change makes it into the next version of the specification to improve the ad experience everywhere.

All the vendors were HTML5 VPAID certified, and we are now trafficking HTML5 VPAID content from Sizmek, DoubleClick and Innovid.

Live

For live support in HTML5, we investigated moving to MPEG-Dash by integrating with ShakaPlayer or Dash.js, but then HLS.js was released. HLS.js allowed us to use HLS in any browser that supports MediaSource Extensions. HLS.js integration with VHS was smooth and took less than a week. With HLS.js, we didn’t have to change our current live feed to support HLS adaptive streaming. We’re still exploring whether MPEG-Dash will improve the video experience in browsers that don’t support HLS natively. For now, using HLS.js allows us to retire Flash support sooner.

The Future

From here, we want to expand our support of adaptive streaming. We’ll widen our usage of HLS.js to our on-demand content library on browsers that do not support HLS natively. That will enable us to take advantage of adaptive streaming on all our content, in all platforms and improve video quality and performance. We also have plans to expand and enhance our plugin system so we can extract all our business logic and open source our core player. We’re excited to keep improving the player and can’t wait to see what new experiences are built with it next.

If you’re a senior iOS engineer, come work with us to extend The New York Times video experience.