Overview

To pursue the development of the Haro3D™ library, a driver was developed to interface LabVIEW with the virtual reality headset Rift Development Kit 2 from Oculus (https://www.oculus.com/rift/). To demonstrate the potential of the new driver, an application was developed that also uses the Kinect V2 from Microsoft.

The first version of the Haro3D™ library was presented earlier (https://decibel.ni.com/content/docs/DOC-40832) in this same community (National Instruments Developer community). The current application is provided to obtain a feedback from the community about LabVIEW applications related to virtual reality, the Oculus Rift headset, the Kinect, and the 3D picture tools. Comments related to any other subject are also welcome.

The spaceship application is kind of a simple game that consists to pilot a small spaceship around a planet very similar to the earth. There are two moons orbiting the earth, one very similar to the actual moon, and the other one very similar to Jupiter. Realism of gravitational effects was not a concern here.

The spaceship orbits around the planet at a constant angular speed but its roll and its altitude can be controlled. The roll and altitude are controlled through the Kinect using gestures. The view of the pilot inside the spaceship is rendered in stereo in the Oculus Rift headset.

The application is provided as an executable only running on LabVIEW 2014 64-bit (update: now available as LabVIEW 2014 32-bit). The new Haro3d™ library should be released soon and will probably contain an example similar to this application.

Update: There is way to do pin cushion correction using the "Render to image" property nodes of 3D picture controls, and 2 additional 3D picture controls. See: https://decibel.ni.com/content/docs/DOC-42762.

Description

The Oculus Rift DK2

The Rift can work in two modes: the Direct mode or the Extend Desktop mode. The Direct mode requires to stream the video directly to the Rift. This mode is therefore not accessible to the 3D picture tools offered by LabVIEW. The Extend Desktop mode makes the Rift look like an extra monitor on Windows. This latter mode can be used by LabVIEW without any special driver.

The Oculus Rift DK2 is equipped with a high-performance positional tracker. The positional tracker uses 40 infrared LEDs that are integrated into the headset. The infrared LEDs are invisible to the naked eye but can be seen on images acquired by most CMOS camera that are sensitive to the near-infrared. The image below shows some of the 40 LEDs used to track the Rift headset position and orientation. For more information abou how the tracker works, see this nice document: http://doc-ok.org/?p=1095.

A DLL was developed to obtain the position and orientation information from the Rift headset from within LabVIEW.

The Oculus Rift positional tracker was positioned on top of the Kinect V2 (shown in the picture below) in a manner to make its orientation approximately match the orientation of the coordinate system of the Kinect.

The position information from the Oculus Rift tracker is not used in the application. The position information from the Kinect is used instead. The head position as provided by the Kinect is used as the position of the eyes (after including the offsets associated with the inter-pupillary distance). The orientation of the view is determined by the orientation provided by the Oculus Rift tracker. In addition, the Kinect provides information for the positions of 25 body joints, making it possible for the user to see the hands and feet of his avatar, corresponding to his own hands and feet. The figure below shows an example of the images for each eye of the user, including his hands. Notice the slight differences between the two images.

The two stereoscopic images in this application are rendered for an inter-pupillary distance of 64 mm. Pincushion and chromatic distortions are not compensated for (Update: see update in overview). It does not appear possible to make those compensations using the current LabVIEW tools. However, it is not a major concern for the current application because the virtual world produced here is not an attempt to reproduce a normal human environment.

A sitting position for the user is strongly recommended because of possible disorientation that the user might experiment with the use of the Oculus Rift headset. In some cases, the Kinect has difficulties to recognize gestures probably because of the sitting position but also because of the presence of the headset.

The user should be sitting at a distance of at least 2 meters (6 feet) from the Kinect (see figure below). When the hands and arms are low, the Kinect tends to not recognize the presence of the user. This is an issue when the user is making moves that are recorded over some time (like grabbing up or down to change altitude) because the loss of the user tracking might result in the reset of the hand tracking over time. One solution is to keep the hands extended and at or above shoulder level for the altitude change gestures.

The user can turn his head and look around. He can even look at the back of the spaceship, the Rift tracker can follow the orientation up to 180° head turns and beyond. The Kinect tracks the position of the body the spaceship is positioned relative to the head position. It is therefore not possible for the user to “exit” the spaceship. If the user moves around in the real world, the spaceship will follow in the virtual world, adding to the possible disorientation sensation of the user.

With a Kinect present, the user can make the spaceship roll to its left or right by using the lasso hands (closed hand with the middle and index fingers extended) on each side of his body. Circular movements of the two hands in the clockwise direction makes the spaceship roll to its right, and counterclockwise movement make it rolls to the left. The movement changes the rolling rate, not the position of the hands. The user can open both hands to stop the rolling at the current angle at any moment. The rolling rate is capped when using the Kinect but not with the keyboard.

The user can also change the altitude of the spaceship. The altitude is displayed in real time on the control panel of the spaceship. The altitude is increased when using the Kinect by closing the right hand and moving the hand down (pulling the sky down). Closing the right hand and moving the hand up decreases the altitude. Each gesture need to be clearly separated in time (at least 2 seconds) because the Kinect will interpret movement on the way up as well as those on the way down. Also, it is better to keep the left hand closed during altitude changes because both open hands are interpreted as a different gesture (stop rolling).

If the Rift headset or the Kinect, or both, are not present, the application can still be operated using a keyboard.

For more details about how to use the application, including how to use a keyboard, look at the manual.

The application is offered as an executable and the installer can be downloaded from the current page. Read the installation instructions to determine the version you need to download. The manual on how to operate the application can also be downloaded from the current page.

A video giving details about the application and showing real time rendering can be seen at: http://youtu.be/m8EfFS9zkYM

Requirements

Minimum Requirements

Operating system: Windows 7 or 8 64-bit.

RAM: 4 GB of RAM at a minimum. The application alone requires approximately 1 GB of available RAM.

Works on a low level graphic card but the higher the grade of the graphic card, the better.

One monitor with 1920x1080 resolution is preferable if the Oculus Rift is not used.

Requirements for use of the Kinect and Oculus Rift

Kinect

Kinect V2 connected and configured to run on the computer

For the Kinect V2 requirements, see: http://msdn.microsoft.com/en-us/library/dn782036.aspx

Oculus Rift DK2

Installation

Application

Select and install the version of the application to be installed. The Microsoft Kinect V2 requires Windows 8 64-bit or later. To run the application on Windows 7 (obviously without Kinect), install the “no Kinect” version of the Spaceship application. To run the standard Spaceship application on Windows 8 64-bit or later when no Kinect is present, the Kinect runtime 2.0 must be downloaded and installed, or simply install the “no Kinect” version of the Spaceship application.

If not already present, install LabVIEW Run-Time Engine 2014 (64-bit): http://www.ni.com/download/labview-run-time-engine-2014-sp1/5197/en/ (update: a 32-bit version is now available for download).

If not already present, install Visual C++ 2013 redistributable x64 (vcredist_x64.exe): http://www.microsoft.com/en-us/download/details.aspx?id=40784 (update: you must install the 32-bit version (vcredist_x86.exe) if using the LabVIEW Run-Time Engine 2014 (32-bit)).

Kinect V2

The Microsoft Kinect V2 requires Windows 8 64-bit or later.

Upon connecting the Kinect for Windows V2 for the first time to your computer, the drivers should be downloaded through Windows Update. If the computer is not connected to the internet, download and install the Kinect runtime 2.0 or later ( http://www.microsoft.com/en-us/download/details.aspx?id=44559 ).

Oculus Rift DK2

Install the Oculus Rift runtime: https://developer.oculus.com/downloads/#version=pc-0.5.0.1-beta (application was tested with runtime versions 0.4.4.1 and 0.5.0.1)

Configure the Oculus Rift and its software as per Oculus’ instructions.

After installation of the Oculus Rift runtime, and connection, the headset must be configured to use the Extend Desktop mode. This change can be done using the Oculus configuration utility and selecting “Rift Display Mode”. The window shown below will appear.

Select “Extend Desktop to the HMD”.

Windows should then detect the presence of a second monitor. Open the monitor configuration utility on your computer. Notice that the Rift must be the second monitor (unless it is the only monitor).

Depending on the configuration of the computer, the headset might be configured as a portrait monitor. It is then necessary to change its configuration to “Landscape (flipped)” (as shown in the figure below).

Video

Watch a video of the application at: http://youtu.be/m8EfFS9zkYM

Summary

This application is only an illustration of the potential of the combination of the Oculus Rift, the Kinect, and LabVIEW. The use of the Kinect to control an application with a virtual reality headset is very beneficial because the user does not have to attempt to interact with elements outside the virtual world, like a keyboard or a mouse for example.

Obviously, LabVIEW is not a platform for the development of games or virtual reality applications for large audiences. For rendering of situations more similar to normal human environments, the lack of lens compensation could become a major issue when using the Rift for some period of time. In any case, the capabilities of LabVIEW to quickly generate and render 3D objects are worth further exploration for virtual reality, especially for industrial and scientific applications.