The Ghosts of Mars

When Microsoft announced HoloLens at the beginning of 2015 one image from the press kit that didn’t grab the attention it deserved was of NASA scientists collaborating remotely. The Jet Propulsion Laboratory (JPL) worked with Microsoft to create OnSight, a tool that uses ‘holographic computing’ to enable scientists around the world to meet in a 3D simulation of the Martian environment, to plan the operation of NASA’s Curiosity Rover. That is, from the vantage point of standing beside it, virtually, on Mars.

JPL scientists collaborate remotely using OnSight

In the B-roll video footage it’s clear to see the phantom outline of the remote scientist — John — watching his local colleague as he moves from a desk to the shared projection of the Martian soil. Not that we can see John’s eyes exactly, but his attention is easily inferred from the motion of his ghostly head while his gaze is emphasized by a dotted line. Of course the demo is likely just a mock-up using post-production effects but it’s nevertheless representative of the most exciting thing in Augmented Reality,… sorry, Holographic Computing that no-one is talking about; people. Or rather, the spatial representation of remote colleagues in a shared (or merged), 3D digital environment.

Now, here’s where I deviate slightly from the hymn-sheet. The reason is Microsoft would apparently have us refer to all digital overlays seen through their headset as ‘holograms’. But for the sake of clarity in this essay I need another term to distinguish between the projection of non-humanoid and humanoid forms. The problem is with the preconceptions — in popular culture holograms are typically depicted as auras, projected from the floor or a wall, that enjoy a limited range of positional movement. Or none at all. But the point of this article is to consider the properties of humanoid forms that have unrestricted and continuous freedom of movement throughout, and indeed through, the environment. So I’ll use the label ‘ghosts’ to disambiguate. Ghosts are humanoid holograms, seen in outline, that move freely through the holographic space.

Why don’t I use ‘avatar’ instead of 'ghost’? Because of its long-term use in computer gaming, and association with stylized characters for fictional narratives, controlled with gamepads. More digital puppets than actual avatars (like the movie). Whereas ‘ghosts’, here, are motion-tracked and body-mapped stand-ins or surrogates (yes, like the movie).

My hope is that thinking about holographic humanoids as ghosts will give you a different perspective on the possibilities for holographic computing, beyond what is normally portrayed in fiction. That is, with the honorable exception of Dennō Coil, a truly visionary science fiction anime “depicting a near future where semi-immersive augmented reality (AR) technology has just begun to enter the mainstream”. In that TV series the boundary between the real and the virtual world was porous to such a disconcerting degree that it took viewers quite a time to figure out which domain a creature belonged to. The beings populating the world of Dennō Coil were ghost-like in their movement and passage between realms. They existed everywhere and nowhere. (Please do watch this video to better grasp the rest of this article).

Denno Coil

Putting the presence into Telepresence

So, what about an OnSight for the rest of us? A workplace that works more like Dennō Coil than Cubicle Farm. In contrast to separation by partition, can the holographic office convince us that colleagues are alongside us, at all times, regardless of where they actually are, physically? What would that workspace look like and why might we want it? To answer that, lets start by looking at the changing nature of the labour market, as Loic Le Meur sees it —

“The Millenials want flexible work hours, they expect to be mobile and work from home/office/cafes at will, from anywhere really. They often work outside of normal business hours and prefer to freelance, work on flexible hours and collaborate online. 3 billion new minds are coming online within the next 10 years, many will learn and work exclusively online and never be a full-time employee of a corporation.”

[ASIDE: And speaking of millennials — in an article entitled ‘Why remote work isn’t just for millennials’, Susan Tenby lists Second Life among her favourite tools for remote workers — “for synchronous meetings with lots of people”. If you never understood the appeal of that virtual world platform (virtual reality without the goggles), Susan’s liking for it should make a lot more sense by the end of this article.]

In the words of Stephane Kasriel, CEO of Upwork, “You can totally see the emerging trend of people who don’t want a full-time job, period. And the best people really don’t, because they have a choice.” That begs the question — and it’s not a new one, I know — will work cease to be defined by an office? Or even by a place? Fast Company has more figures on the rise of the freelancer economy

“As of May 2015, 15.5 million people in the U.S. were self-employed, according to the Bureau of Labor Statistics — an increase of roughly 1 million since May 2014. That number is expected to keep growing at a steady clip. By 2020, a separate study estimates that more than 40% of the American workforce, or 60 million people, will be independent workers — freelancers, contractors, and temporary employees.”

The corollary of that being that —

“It’s not uncommon for independent workers to feel isolated. But the rise of co-working spaces in top urban centers is changing that, offering freelancers unprecedented support and resources. Co-working spaces are providing more than just a sense of community that comes from working around others. WeWork, for instance… has raised the bar in part by focusing on creating a collaborative ambiance you’d find at any cutting-edge startup.”

There’s an apparent paradox at play here. People want the freedom of working independently but don’t want to lose the commonwealth of togetherness. Co-working seems like the cure for now but it’s no panacea. Just a stopgap. While many independents miss the social ambiance of the office, few miss the many downsides of attending shared spaces that offset the benefits of professional freedom. Commuting for one! Distractions for another. And on and on. As Micah Baldwin succinctly puts it — “Spending some time in close proximity to other people, but not having to interact with them is probably sufficient for most freelancers”. How about having the best of both worlds? What if you could be among colleagues without being with them? You know, like a ghost.

The crux of the problem is that the real world is too binary. You’re either here or there. In a single place at a time. It’s black and white. While in contrast the digital world is… well, more analog. It’s many shades of grey. You can be ‘in’ a place to varying degrees and in multiple places at once. Yes, like a ghost.

However, it’s true to say that the boundaries of real world communication are more permeable. We move in and out of earshot, and view, all the time. Conversations change continuously according to who and how many are within vocal range. And often fork into (multiple) overlapping dialogues as groups expand. It’s organic and analog. Voices fade in and out as a function of physical movement, projection of speech both in volume and direction, and the shape, placement and material properties of objects in the space.

These are valuable properties and nuances of communication that we’ve failed to replicate in any meaningful way online. We can’t simply collapse the multi faceted communications fabric of meatspace into the narrow thread of cyberspace. Or the 2D portal of a video call.

We all know that video-conferencing has failed. Miserably. I long thought the failure was just a matter of time; time for Moore’s Law to work its magic. If the sound was surround enough, the video big enough, the lighting bright enough and the cameras… well, if they could somehow bend light to convey actual eye contact, then we’d surely feel as if we were in the same room, together. Surely, right? But then I got to actually experience a Cisco Telepresence system, with the very best of everything and… it was thoroughly disappointing. I still felt absolutely separated from the people at the other side of the call. The other side… I didn’t think about that until I wrote it down just now but it neatly highlights the real issue. Video-conferencing will always have a here and a there. Instead of just a here, as it needs to be.

And I’m far from alone in my disappointment with ‘videophones’. Paul Watson, former CTO at Storyful, laments -

“Video conferencing is woeful. Before you even get the call connected is painful, then the lag, drop outs, echoes, hiss, volume, looking at a ceiling, etc. 2D video is not going to make people feel like they are here… I reckon Microsoft’s Hololens is the way, until we have neural connections. Ditch physical screens all together.”

Paul identifies the key problem with video — regardless of how good it is it’ll never deliver a sense of social presence or co-presence. Photo-realism is not what we crave in ambient communication with distant colleagues. Spatial realism is. And the corporeal form, even in outline, with the unmistakable fluidity of human body language. Our animal brain is hardwired to sense when another creature is in ‘our’ space. And a flat reflection in the watering hole doesn’t fool us.

But what does fool us is Virtual Reality… sorry, Holographic Computing. Chief Scientist at Oculus, Michael Abrash explains how —

[Virtual Reality creates] a powerful sense of what’s known as presence… the sense of being someplace else while in virtual reality; many people feel as if they’ve been teleported. Presence is distinct from immersion, which merely means that you feel surrounded by the image of the virtual world; presence means that you feel like you’re in the virtual world. Presence is an incredibly powerful sensation, and it’s unique to VR. There’s no way to create it in any other medium. Most people find it to be kind of magical, and we think that once people have experienced presence, they’ll want it badly.”

So, while high-end video-conferencing may eventually deliver a sense of immersion it’ll never evoke a true sense of presence. That is to say, the photorealism of other people in video doesn’t compare even to matchstick men in VR. Not convinced? Take a moment to observe the giddy reaction to the social presence offered by Oculus’ Toybox demo, even with the most basic, ghostly avatars —

Oculus Toybox demo

Then listen to the Voices of VR podcast, to hear the wild delight of others who tried the same demo. As host Kent Bye emphasizes -

“It was a watershed moment for so many developers to be able to experience social and emotional presence with another person within virtual reality. It became less about the technology and tech specs, and more about the experience of playing, having fun, and connecting to another human in ways that were never possible before. This Toybox demo felt like a real turning point and “Aha!” moment for a lot of VR developers to see how compelling social experiences in VR are going to be.”

Paul Watson’s comments on video-conferencing were particularly interesting to me because I knew he had previously explored ways to bring ambient (always-on, background) presence to his distributed team. One solution he investigated was Sqwiggle — “a tool for instant tap on the shoulder remote discussions”. Or “an always-on online workplace for your remote team to work together throughout the day and feel more connected”. But we understand now why products like Sqwiggle will never quite achieve that ambition while relying on video.

Furthermore webcams are unsuited to ambient presence because we have an instinctive unease at being spied upon, essentially through a peep hole. It’s a natural lizard brain reaction. Cameras are often misaligned, switched off and inconsistent placed so you can never quite be sure that someone isn’t actually staring at you. And that’s uncomfortable. Your only guide is a grid of postage stamp faces on a flat screen that you can’t really monitor in your subconscious.

In normal life we usually know when we’re being stared at (e.g. on public transport) — we’re hard-wired to ‘feel’ it because we have full spatial awareness of the orientation of people’s heads, what angle they’re held at, what direction their eyes are facing in. We can scan a room full of people with a dart of our eyes and immediately detect who’s looking in our direction. Video-conferencing doesn’t guarantee the conveyance of such clues. So the strong possibility that people are actually looking at you while it doesn’t appear they’re looking feels creepy. It feels off.

In addition there’s the nagging fear of forgetting that someone can see and hear you. When we don’t sense the physical presence of people, even in our periphery, we forget that microphones and webcams are switched on. That wont so easily happen when we have holographic ghosts around us. Even when they’re positioned behind us the spatial audio from a holographic headset reminds us with the sound of breathing, rustling and so on.

Holographic ghosts may be intangible but will take us across a cognitive threshold that video never can, and finally put the presence into telepresence.