The Scenarios

There are three scenarios that we explored:

Multiple real-world users can interact with a Virtual user through their mobile devices. (Many-to-one)

A Teacher & Classroom scenario where a teacher can control the student’s immersive experience. (One-to-many)

A training simulator where a user interacts with a custom controller to simulate control of a virtual machine. (Simulation)

Space Assault

In this scenario, a user plays the virtual role of an astronaut surrounded by alien creatures. Each alien is actually a person in the real world who can blast the astronaut using a web-based mobile interface on their phone. The astronaut’s only line of defense is a shield that they have to block the alien’s beam.

This game utilizes web sockets to communicate between each real world user’s phone and the virtual user’s headset. The bi-directional communication is near instantaneous and shared among all of the users (real and virtual). Scores and status are updated in real time on all devices. In the virtual user’s headset, the Unity application utilizes spatial audio to alert the player of the real-world user’s blast.

Solar system explorer

This scenario supposes a teacher walking through a lesson with students. At the appropriate time, the students enter the virtual world and are given more in-depth instruction in virtual reality. As the teacher describes specific planets, they select the planet on their phone. This action causes all of the student’s views to focus on the teacher-specified planet.

This guided educational experience offers a new and engaging way for students to learn together, at the direction of the teacher. While they can still explore their virtual reality individually, the teacher ultimately controls their progress and focus throughout the lesson.

ExcavatVR

To explore ways that a user in virtual reality could interact with the real world on an individual basis, we decided to explore how a custom physical controller could influence an object in a virtual world. Driving and controlling an excavator from virtual reality easily won the most support.

To accomplish this, we wired a series of buttons to an Arduino and connected that to a Raspberry Pi. With some custom software, we were able to send button presses through the cloud to the headset which then controls the excavator. While the controls don’t reflect an actual excavator’s controls, this concept could be tailored to training experiences, allowing a user to learn how to control a complex or dangerous piece of equipment in VR before venturing out into the real world.