Microsoft’s HoloLens and their range of VR-style headsets are all Windows Mixed Reality headsets. How can this be? One is an augmented reality headset and the others are virtual reality headsets! We’d previously seen the term mixed reality used to define a certain type of AR… has it changed? I asked around and found the answer. Here’s what Windows Mixed Reality is and how it fits in with virtual reality and augmented reality.

It all started with a spectrum

When discussing “mixed reality”, it all comes back to a 1994 research paper by Paul Milgram and Fumio Kishino entitled “A Taxonomy of Mixed Reality Visual Displays”. In that paper, the first appearance of the term “mixed reality” appeared and referred to part of “the virtuality continuum” (later also called the Reality-Virtuality (RV) continuum). Here’s a diagram I put together of that continuum with some extra notes:

This spectrum basically shows a range of environments, from our reality with no virtual elements to a totally virtual environment with none of our reality visible. The space in between when real world elements start to mix with virtual ones to some degree or another is “mixed reality” in their original definition.

“The most straightforward way to view a Mixed Reality environment, therefore, is one in which real world and virtual world objects are presented together within a single display, that is, anywhere between the extrema of the virtuality continuum.” — Paul Milgram and Fumio Kishino, “A Taxonomy of Mixed Reality Visual Displays“

Within “mixed reality”, Paul Milgram and Fumio Kishino have two different extremes — “augmented reality” and “augmented virtuality”.

Augmented reality (AR)

This is where we bring in some virtual elements into our real world. Heads up displays (HUDs) such as DAQRI’s HUD for cars, Google Glass, Snapchat filters, Pokemon GO’s AR view, mobile AR apps and Microsoft’s HoloLens are “augmented reality” of varying degrees.

Augmented virtuality (AV)

This is a term that isn’t often used, but involves environments where a virtual world has some real world elements within it. The one most recent example of this that comes to mind was Intel’s “Merged Reality” technology (which was recently discontinued). Their technology aimed to allow people to bring their real hands or real people in the real world into their VR experience:

Please, let’s not call anything “merged reality” or “augmented virtuality”… it’ll cause way too much confusion. Virtual reality, augmented reality and mixed reality are already plenty of terms for the public!

As Tom Painter, the founder and director of Bigman in Brighton, explained consisely at the VR Brighton Meetup (he starts to speak around 52 mins in):

“Reality on one side, VR on the other side. Everything in the middle is mixed reality as far as I’m concerned.” — Tom Painter (VR Brighton Meetup)

That makes total sense to me! But wait… the latest Windows Mixed Reality headsets are all VR headsets. It’s been pretty strongly agreed in the definition that VR doesn’t count… right? Mark Billinghurst, in his article explaining mixed reality, is pretty clear that VR is not mixed reality:

“Put simply, the VR experience viewed in a VR head mounted display that doesn’t show part of the real world isn’t a MR experience.” — Mark Billinghurst

So, how does “Windows Mixed Reality” fit into all this?

Windows Mixed Reality 101

Microsoft’s concept of “mixed reality” is not one single grade of device. It is a spectrum/range of devices that go from augmented reality (such as the Microsoft HoloLens) all the way to virtual reality (such as their “Windows Mixed Reality” VR headsets) — covering the various degrees of integration with the real world. The one part that is a bit unexpected from Microsoft is the inclusion of VR headsets within the “mixed reality” definition. I do think there’s an explanation for it, which we’ll get to very soon.

Prior to the Windows Mixed Reality headsets, “mixed reality” began to be used as a description for the style of augmented reality that interacted with real world objects (e.g. a character that would run along a real world table, falling off if it reached the edge). I covered that differentiation back in 2016. David Francis, one of my favourite people to ask about all things AR, believes initially this is how Microsoft were using the term:

“Microsoft just wanted to try to describe occlusion of AR by real world objects, using their sensor array, by calling it mixed reality […] then they tried to extend it to their VR headsets, perpetuating the misunderstanding that its a catch all phrase for AR + VR” — David Francis

The AR on the Microsoft HoloLens with creatures bursting out of real world walls and such was seen as a clear example of “mixed reality” with the occlusion that David talks about. Magic Leap, another company building an AR headset of their own, were also following suit with the same definition. Today, however, this isn’t what is meant by mixed reality when it comes to Windows Mixed Reality — this occlusion of AR by real world objects is just one type of mixed reality under their definition.

Microsoft appear to have snuck in “virtual reality” to fit within their “mixed reality spectrum” on their page on the topic, with augmented reality on the left and virtual reality on the right:

How are their Windows Mixed Reality headsets really “mixed reality”?

That misunderstanding David mentioned that “mixed reality” is now VR + AR is definitely growing. Earlier this year, The Next Web published a whole article entitled “Microsoft insists on calling AR and VR ‘Mixed’ Reality. Maybe we should too.”. I think that while it might have been solely a marketing department decision at Microsoft, it could also have been a decision made for more concrete reasons. After asking around and navigating a lot of understandably confused people in the industry who were just as uncertain as I was, I think I’ve settled on an answer for how “mixed reality” comes into play with their latest VR headsets.

An article in June 2017 by Fast Company explained that Alex Kipman, the man leading the charge on mixed reality at Microsoft, explained “mixed reality” like so:

“He brings up “mixed reality” (MR), Microsoft’s term for tech that mixes real-world with computer-generated imagery and will, some day, according to Kipman, seamlessly blend augmented and virtual reality. He says that one of the most exciting features of MR is its potential to unleash “displacement superpowers” onto the real world.” — “Is Mixed Reality The Future Of Computing?”, Fast Company

It is the blending of realities

“Mixed reality” is their attempt to seamlessly blend augmented and virtual reality together. At a glance, that sounds just like “augmented reality” and doesn’t explain their VR headsets at all. Until you see their end goal isn’t typical VR. Their goal with the Windows Mixed Reality headsets is to be able to generate virtual worlds that seamlessly blend nicely with the real world. Got a desk in your room? That desk could turn into a virtual desk in a virtual world (you don’t see your desk — you see a virtual one in its place). This is possible with their Windows Mixed Reality headsets as they use the same RGBD sensor array from the Microsoft HoloLens. Windows 10 runs behind it all and aims to be one and the same on the Windows Mixed Reality headsets as it is on the HoloLens. This video shows it incredibly well:

But… the HoloLens and their VR-style headsets aren’t the same!

Yep. They’ve got different capabilities that are on different parts of that mixed reality spectrum. The Windows Mixed Reality VR headsets are largely focused on being able to use the sensors to bring in elements from the real world into VR, while the HoloLens is trying to bring in virtual elements into the real world in a more typical AR sense. VR/AR Unity Developer, Peter Liang, pointed out to me that the Windows Mixed Reality headsets are closer to the “augmented virtuality” concept:

“On the reality-virtuality continuum, it really should be called augmented virtuality, but I think they’re aware that there’s more than enough acronyms floating around for the public to keep track of, though labelling it mixed reality really doesn’t help either.” — Peter Liang

The best way to visualise it came to me when Sampson Crooks, Jr., a VR/AR/MR Program Lead at Tata Consultancy Services, mentioned the spatial mapping mesh of the physical environment which that HoloLens maps out. Imagine that grid but in VR where it can use that grid to bring some of that into the virtual world instead:

“The biggest difference between AR and MR is spatial awareness which has to do with the inside out tracking that is used in the HoloLens and the Mixed Reality headsets. It is more apparent with the HoloLens when using spatial mapping to create a mesh of the physical environment that interacts with virtual content using colliders and rigid body physics.” — Sampson Crooks, Jr.

Microsoft does define a difference between the two types of headset. They call their HoloLens-style headsets “Holographic Devices” and their VR-style headsets “Immersive Devices”. There are more details on Microsoft’s explanation of the differences between the two and more over on the Microsoft website.

I think “mixed reality” makes sense

After looking into it, I think the term “mixed reality” genuinely does make sense. It isn’t VR. When people are using virtual reality with no sense of their surroundings, there’s no mixed reality there. That’s VR. Plain, simple, incredibly-thrilling-and-wonderful-in-its-own-right VR. Maybe it would have been easier to call everything VR. Or stick to the VR/AR terms alone. It does, to an extent, make sense to want to differentiate the capability of doing both to different degrees.

The HoloLens is a form of mixed reality and one which I hope develops and grows alongside a range of AR technology out there to bring us to a better future. That third confusing area of mixed reality, the Windows Mixed Reality headsets — that is indeed mixed reality too, but only when the experience is pulling in the real world into the virtual one in some way or another. Those headsets can be used for plain old VR too, and my guess is most of the time in the near future, they will be. I’m excited to see that augmented virtuality-style VR in action though. I, for one, would quite enjoy turning my home office into a totally virtual space that I could theme entirely differently at a whim. That’d be pretty neat.

A huge thank you to the many people in the industry who gave their thoughts on Windows Mixed Reality and how it all fit together — you definitely helped me clarify things!

Know other emerging tech enthusiasts who might want to read this too? Please like and share this post with them!