Introduction

Wi-Fi CERTIFIED Miracast* is a solution for seamlessly and wirelessly displaying multimedia between devices [1]. This solution has a number of advantages that make it excel compared to other wireless display technologies.

Miracast is natively supported on Windows* and Android*, but until now Chrome OS* has been missing Miracast support. Engineers in Intel’s Open Source Technology Center (OTC) have done the work described in this article which helps to enable Miracast on Chrome OS.

What makes Miracast special?

The popular wireless display solutions on the market today include Chromecast*, Miracast, Intel® WiDi technology (Miracast compatible), and Apple Airplay*. While all of these technologies provide more or less the same functionality, Miracast does have some advantages when compared to the others:

Miracast is a popular, industry-wide solution, so the technology works well across various OEM devices, regardless of brand. Miracast connections are easy to set up and use because devices choose the appropriate connection settings automatically [1]. Miracast enables devices to communicate with each other without connecting to a local network. This is a big advantage to users as it greatly simplifies setting up a wireless display session. Miracast is natively supported on Windows and Android [1].

These features made it highly beneficial for Chrome OS devices to begin supporting Miracast-compatible wireless display as well.

Enabling Miracast

Let’s review some of the foundational technologies that Miracast uses so we can better understand what we need to implement to enable Miracast on a Chrome OS device.

Miracast is the name for the certification program operated by Wi-Fi Alliance*. Devices that pass this certification testing can be referred to as Miracast devices. Miracast certification is based on the Wi-Fi Alliance Wi-Fi Display specification [1]. Wi-Fi Display* is the name of the technology behind the Miracast solution.

This means that Miracast devices must implement the functionality described in the Wi-Fi Display specification for a given device type: media source, media sink or both1.

Chrome OS devices naturally play the media source role, given that Chrome OS is usually installed on mobile devices with relatively small screens. In this article we focus on the implementation of Wi-Fi Display media source device functionality.

Before the Wi-Fi Display media source can start streaming content to a Wi-Fi Display media sink, the media source device passes through several stages, illustrated on the figure below:

At first, the media source device discovers the list of available media sink devices and allows the user to choose a media sink to connect to. Typically, Miracast relies on Wi-Fi Direct technology2 to enable one of the greatest Miracast features: easy connection setup.

Wi-Fi Direct technology allows devices to directly connect to each other without the need for a Wi-Fi access point. To connect the devices, you only need to either push a button, enter a PIN, or tap two NFC-capable devices together. Wi-Fi Direct allows media source and media sink devices to discover each other and provides the underlying device-to-device connectivity for Miracast [1].

After the Wi-Fi Direct connection is successfully established and authenticated, the media source and media sink devices negotiate which supported audio and video formats are the most appropriate to stream media content for both devices.

While the Wi-Fi Display implementation pieces in Chromium are cross-platform and can run on any platform that can run Chromium3, the networking functionality that provides discovery, connection, and watch operations to media sink devices does depend on the underlying platform.

Chromium just contains the generic DisplaySourceConnectionDelegate interface, which should be implemented on each platform accessing system or middleware networking components available there. As a result, at the time of writing, the Chrome OS platform does not enable Wi-Fi Direct. Therefore, the DisplaySourceConnectionDelegate interface remains unimplemented, but is an area for future work.

Miracast uses an RTSP-based protocol for capabilities negotiations and other session controlling functionality. Control messages are sent between the media source and media sink devices over a TCP connection.

After that, the devices can start streaming media content. Audio and video content must be encoded using some of the supported encoders listed below, multiplexed into an MPEG transport stream, then packetized to RTP packets and sent over the UDP socket to the media sink device.

For video, Miracast supports the ITU-T H.264 video codec (Advanced Video Coding [AVC]) for high-definition video. It supports the Constrained Baseline Profile (CBP) and the Miracast-specific Constrained High Profile (CHP), at levels ranging from 3.1 to 4.2. Supported display resolutions include common Consumer Electronics Association (CEA) formats, Video Electronics Standards Association (VESA) formats, and handheld formats [1].

For audio, Miracast supports a number of Linear Pulse-Code Modulation (LPCM), Advanced Audio Coding (AAC), and Dolby Advanced Codec 3 (AC3) modes [1].

Enabling secondary display technologies on Chromium-based platforms

The Chromium browser is such a major component of the Chrome OS platform [4] that it actually stands behind everything that users interact with. This includes Web applications, extensions, and browser functionality itself. So before we discuss any further details of Wi-Fi Display implementation on Chrome OS, it is necessary to both describe the generic mechanisms and frameworks available in Chromium for enabling any secondary display technologies, and to understand the requirements and constraints that must be applied to the intended Wi-Fi Display implementation.

Presentation API and Media Router framework

W3C Presentation API aims to make presentation displays, such as projectors or connected TVs, available to the Web. The Presentation API takes both wired displays connected via HDMI, DVI, or a similar connection and wireless technologies like Miracast, Chromecast, DLNA, or AirPlay into account [2].

This is very high level API that allows a website to check if a secondary display is available and if there is, present a given URL there. Refer to the examples section of the specification to see the API usage.

Presentation API does not provide any data about the available secondary displays. For example, how many secondary displays are available on the system, how they are connected to the device, or which of them should present the presentation URL are all responsibilities that a user agent like a browser should take care of.

MediaRouter is the name of a component in Chrome that implements Presentation API. It also enables browser users to cast the content of a browser tab (including tabs that aren’t visible) or the whole desktop to a secondary display, as shown in the figure below.

The Chrome Media Router is a browser-resident service that serves as a media-protocol-agnostic platform for parties interested in media routing. It matches clients that need to render content outside the browser (media sources) with devices and endpoints capable of rendering that content (media sinks) [3].

The Chrome Media Router itself does not directly interact with media sinks. Instead, on Chrome OS and desktop platforms, it delegates these requests and responses to a media route provider in the Media Router Extension [3].

The Media Router extension is a Chrome extension that is a part of the Media Router framework. The Media Router extension manages discovery of and network interaction with individual media sinks and uses some chrome.* platform APIs, such as chrome.dial, chrome.cast.channel, and chrome.mdns to implement network level interaction. Media Router Extension contains set of Media Route Providers. Each provider is a JavaScript* bundle that knows how to find and communicate with a specific type of media sink (for example, Chromecast, DIAL, etc.) [3]. Media content is accessible to providers via the MediaStream interface.

The Chrome Media Router framework is a generic way for Chrome browser UI or Presentation API to access a secondary display technology on different platforms, including Chrome OS. To provide the best user experience, the Wi-Fi Display implementation on Chrome OS must be compatible with Media Router. It should be possible to create a Wi-Fi Display Media Router Provider.

chrome.displaySource API

As mentioned above, the Wi-Fi Display implementation on Chrome OS must be compatible with the Media Router Framework. This means that the implemented functionality must be exposed to the Media Router extension as a Chrome extension API, and that this API must accept media content from MediaStream objects.

For this reason our team introduced the new chrome.displaySource extension API. Even though this API was designed primarily for Wi-Fi Display media source functionality, it does not have any Wi-Fi Display dependencies. So it is a generic API that can also be used for enabling any other secondary display technologies.

The chrome.displaySource extension API can be used both by the Media Router extension (via a dedicated provider) or by any other extension, just the same as other Chrome extension APIs.

The code snippet below finds a list of available media sink devices and starts a display session with the first media sink in the list if the list is not empty:

function StartWiFiDisplaySession(video, audio) { chrome.displaySource.getAvailableSinks(sinks => { if (sinks.length == 0) { console.log("No available sink devices are found."); return; } let sink = sinks[0]; let session_info = { sinkId: sink.id, videoTrack: video, audioTrack: audio, authenticationInfo: { method: "PBC" }}; function onStarted() { if (chrome.runtime.lastError) console.log("Session to sink " + sink.name + " has failed to start: " + chrome.runtime.lastError.message); else console.log("Session to sink " + sink.name + " has started."); } chrome.displaySource.startSession(session_info, onStarted); }); }

As you can see, the code is very simple. It calls chrome.displaySource.getAvailableSinks() to get the initial list of available media sink devices. Every media sink has a unique ID that can be used to start or terminate a display session as well as a human-readable name to present to the user.

The session_info dictionary is fulfilled with the data that is necessary to start a display session. This information includes:

ID of the media sink device. Two MediaStreamTrack objects representing audio and video content. Authentication information. With Wi-Fi Display, it describes which Wi-Fi Protected setup approach should be applied to establish a connection.

Finally chrome.displaySource.startSession() is called with the fulfilled session_info. With that, the given media content is rendered on the selected remote screen!

The given MediaStreamTrack objects could be obtained from multiple sources, like navigator.getUserMedia, chrome.desktopCapture, chrome.tabCapture, or HTMLMediaElement APIs. This provides a lot of opportunities for the client extension to provide something more interesting to the user than simple screen sharing!

The Chromium repo contains a demo extension that casts the contents of a browser tab to a media sink device selected by a user from the list. It gets the captured tab media stream via the chrome.tabCapture API. The Media Router extension should use chrome.displaySource API in a similar way.

Wi-Fi Display implementation in Chromium

Our team implemented the essential part of Wi-Fi display media source functionality in Chromium behind the chrome.displaySource extension API. There is a Chromium bug for enabling Miracast on Chrome OS that we used to track patches.

The main implementation blocks are shown in the figure below:

The main blocks include implementation of chrome.displaySource bindings, Wi-Fi Display session management, and the whole media pipeline.

Chromium has multiprocess architecture with sandboxed renderer processes providing better security and robustness. Considering this, we implemented all the business logic within the renderer process and only left raw sockets in the browser process for networking communication with media sink devices.

Code location

The code is located in Chromium repo at extensions/renderer/api/display_source and extensions/browser/api/display_source. The Wi-Fi Display backend is build conditionally if the enable_wifi_display build flag is enabled (it is disabled by default).

Wi-Fi Display session management

Implementing Wi-Fi Display session management was one of the most difficult parts of this project. After a TCP connection is established between the Wi-Fi Display media sink and media source devices, they start exchanging RTSP messages to negotiate the appropriate audio and video formats and also to manage the display session when it is established.

Message parsing and interpretation is very complex and requires writing a lot of code that definitely should not be the part of the Chromium browser and instead should be in a self-contained library. Unfortunately there was no such open source library available, so our team contributed to Wireless Display Software For Linux OS (WDS) and created the libwds library.

The libwds library implements a Wi-Fi Display dialect of RTSP that includes the parser, actual negotiation logic for the media sink and media source, and the related data structures. It is not tied to any specific connection manager, media framework, or main loop [5]. It exposes several simple interfaces for the embedder to implement.

Wi-Fi Display implementation in Chromium uses libwds, which is imported as a third-party component on Chrome OS and Linux.

Media pipeline

Audio and video content is obtained from the MediaStreamTrack objects passed to the displaySource.startSession() method. The row media frames are encoded, multiplexed, and packetized within the render process of an extension that uses the chrome.displaySource API.

The prepared RTP packets are passed to the browser process, which sends them to the Wi-Fi Display media sink device over a UDP socket.

Our Wi-Fi Display implementation reuses the H.264 video encoder and LPCM audio that are already present in Chromium. Chromium uses both the accelerated H.264 video encoder (via VAAPI) and the software encoder provided by the openH264 library, which is imported as a third-party component.

Our team wrote the code for audio/video multiplexing and packetization functionality from scratch.

Future work

Media sink discovery and connect

As mentioned above, the Wi-Fi Display implementation pieces in Chromium are cross-platform and can run on any platform that can run Chromium3.

However, the networking functionality that provides discovery, connection, and watch operations to media sink devices does depend on the underlying platform. Chromium just contains the generic DisplaySourceConnectionDelegate interface, which should be implemented on each platform accessing system or middleware networking components available there.

At the time of writing, the Chrome OS platform does not enable Wi-Fi Direct. Therefore, the DisplaySourceConnectionDelegate interface remains unimplemented, but is an area for future work.

Conclusion

The crucial part of the Wi-Fi Display media source functionality (except Wi-Fi Direct networking) is now available in Chromium. The code is open source and can be used as a baseline by OEMs who are willing to enable Miracast on Chromebooks*.

Introducing the chrome.displaySource extension API as an entry point for Wi-Fi Display media source functionality and our future work to integrate it into the Media Router framework produces very flexible setup that fits various clients. Web sites can access Wi-Fi Display functionality via Presentation API, Chrome users can access via Media Route UI, and extensions and web apps can use chrome.displaySource API directly.

Footnotes

There are generally two device types used in wireless display technologies. The device that provides the media content is usually called the media source. The media source is usually a mobile device, like a smartphone, tablet, or laptop. The device that presents the media content is usually called the media sink. The media sink is usually a device with a larger screen than the media source device. For example, a smart TV or a TV with a linked adapter that supports the display technology. Miracast also allows TDLS, but the vast majority of Miracast devices actually use Wi-Fi Direct. Those platforms where Chrome extensions are enabled.

References

[1] http://www.wi-fi.org/discover-wi-fi/wi-fi-certified-miracast

[2] https://www.w3.org/TR/presentation-api

[3] http://www.chromium.org/developers/design-documents/media-router

[4] https://www.chromium.org/chromium-os/chromiumos-design-docs/software-architecture

[5] https://github.com/01org/wds