on Jan 30, 2014

Augmented Intimacy via Digital Surrogates:

Think, Oculus Rift and VR on steroids – This is “Augmented Intimacy”. To be able to see and interact with your partner right in your living-room: The future of LDRs (long distance relationships).

There is a more serious – bordering on taboo- use of this tech for a person’s digital surrogate, but mentioning it would be a major spoiler for those interested in reading about Memories with Maya.

A quick 101:

– An iphone camera “maps” the room (in the book – Dan and Krish use a kinect like device)

– The “camera’s” position is located and any 3D object can be locked to it – in this case the Digital Surrogate of another person – augments the live camera view at correct depth.

– Wearing a See-through visor (Vuzix makes a Digital see-thru as well as announcing an optical see-thru version) a wearer can see another person in the room. *update* there’s now Hololens and MagicLeap.

The book has more of the rest….

The still shots of the final AR scene above, does not do justice to the coolness of seeing her dancing in the room – live.

Dirrogate [Digital Surrogate] case scenarios:

The aim of the experiments are to gain insights into whether we might evolve into accepting Digital beings – in social interaction scenarios – the way we do with humans.

– Can a grieving person who’s lost their lover/ partner find solace interacting with a Digital re-incarnation, perhaps to help during the grieving process?

– Can a person who may be disfigured, maimed, and for other reasons NOT able to have intimate relations with another human, find a way out via their Digital Proxy?

– Can two human partners – separated due to War (Soldiers), work, or bed ridden – have an LDR via Digital Surrogates?

Create your Digital Surrogate in 2 minutes:

The video above shows the Kinect with software developed at the USC.

Next Steps:

Occlusion mapping – One chapter in the book describes how a real-time 3D map is created from the Stereoscopic 3D input of the ‘Fisheye’ – the fictional version of a Kinect 2. For the demo above, if a 3D map of the room is built, a Kitchen counter (3D model) can be set to occlude correctly, parts of the 3D model of the Digital Surrogate – should she walk behind the counter.

The Oculus Rift, with a camera, either DIY or if designed by Oculus (as an add-on) can convert the VR visor into a Digital See-thru AR solution. Digital see-thru – uses a camera to supply a live feed of the environment.

Digital see-thru – uses a camera to supply a live feed of the environment. IK and AI to drive the 3D model: i.e. she could “sit” on a chair, with IK correctly taking care of skeletal deformations of the 3D model. Proximity triggers and simple AI will know when to trigger IK.

More complex AI can drive TOD (time of day) routines or “Frames” in AI speak, for the Digital Surrogate if indeed the Digital Surrogate is not being steered in real time by a human operator [tele-travelling]

Using the Metaio Sdk in the Unity3D authoring environment, rudimentary or moderately complex AI can be realized.

The Unity3D authoring environment is also known for allowing Kinect 2/Unity3D realtime control of motor function of a 3D avatar – [Dirrogate] using skeletal tracking – idea seeding for VR and AR developers

Ari Shapiro’s Smart Body system can be integrated into the Unity3D environment for Dirrogate behavior control / AI routine automation. This is currently beyond my scope / skill-set, and will need collaboration with beings of higher capability.

Lastly:

Oculus Rift, powered by the Unity3D game engine and running an Augmented Reality SDK from Metaio or (Qualcomm’s) Vuforia, should be able to produce a finished marketable product – “Augmented Intimacy”

In the quest for Digital Surrogates..soon we’ll have this level of detail on mobile devices, as in the Nvidia Tegra video below:

Update: (April 5th 2014)



Current / ToDo

– Improve Skin renderer for mobile devices.

– Need more robust AR tracking for human sized visualization.

– AI script system needed for T.O.D routine.

– Port to Oculus Rift (AR/VR hybrid).

– Google’s Project Tango would be great to map the living room *update* We now have Microsoft’s Hololens and MagicLeap’s Magic Leap one

(I need help from AI experts, bone riggers and C# coders)

Memories With Maya is a “low-level” sci-fi book, i.e the science is so hard – it can be implemented within the next few years.

A special thanks to METAIO (now part of Apple) for furthering AR. A nod of acknowledgement to Amire, who got me interested in AR all those years ago.