WebXR hand controllers for Three.js

I created VRController with my colleagues at Google’s Data Arts Team (DAT) in early 2017 to make supporting 3DoF and 6DoF hand controllers in WebVR a snap. (We then used VRController to power Dance Tonite, an ever-changing virtual reality dance collaboration between LCD Soundsystem and their fans; directed by Jonathan Puckey and Roel Wouters.) VRController supports hand controllers for Oculus Rift + Touch, HTC Vive, Windows Mixed Reality, Google Daydream, Samsung GearVR, and similar devices.

Just include VRController.js in your Three.js-based WebXR project and call THREE.VRController.update() from within your animation loop. When VRController discovers an available hand controller via the Gamepad API it will emit a global event labeled 'vr controller connected' and pass a reference to the controller instance through that event. That controller instance is actually an extended THREE.Object3D which means you can add it to your scene, attach objects and models to it, and so on.

Are you trying to track down a bug or just peek under the hood? Enter THREE.VRController.inspect() into your JavaScript console for a full overview of what’s connected, what buttons and axes are available—and their current state.

Attaching visuals to the controllers

Ok, but how do we use VRController in Space Rocks? Let’s begin with the visuals that we’ll attach to each. Rather than create a spaceship hull that encompasses the player’s body, I decided to make the player’s arms themselves the means of both propulsion and asteroid destruction. Each arm hull is composed of a few simple shapes which I begin to construct near line 115 in /scripts/player.js :

/* LEFT RIGHT

lives score

♥ ♥ ♥ 12345

╭───────╮ cannon ╭───────╮

│ ▪ │ cannon pointlight │ ▪ │

├───────┤ ├───────┤

│ │ │ │

│ │ hull │ │

│ │ │ │

│ │ │ │

├───────┤ engine ├───────┤

│ │ │ │

╲_____╱ ╲_____╱

↓ engine exhaust ↓ */

const hull = new THREE.Mesh(



new THREE.CylinderGeometry( player.arms.radii,

player.arms.radii,

0.2,

7

),

new THREE.MeshPhongMaterial({ color: 0x999999,

specular: 0xCCCCCC,

shininess: 70

})

…

Once we’re done building the models for the player’s arms we can store them as player.arms.left and player.arms.right respectively. With these pieces ready, let’s listen for the 'vr controller connected' event near line 516 in /scripts/player.js :

addEventListener( 'vr controller connected', function( event ){ const controller = event.detail

controller.standingMatrix =

M.three.renderer.vr.getStandingMatrix()

controller.head = M.three.camera

M.three.scene.add( controller )

…

If supporting an array of hand controller models from different manufacturers were simple then at this point we could reliably query controller.getHandedness() to see if we need to attach the visual model for player.arms.left or player.arms.right . But simple it is not. In some instances certain controllers will occasionally return an empty handedness string upon connection. Vive controllers can actually swap hands after some initial controller movement based on their relative position to the headset — which is a very neat feature, but does complicate things. To handle all of these behaviors we’ll first ask for a valid handedness string, if there is one we can immediately attach a relevant arm model, but either way we’ll listen for subsequent 'hand changed' events:

let side = controller.getHandedness()

if( side === 'left' || side === 'right' ) attachArm( side )

controller.addEventListener( 'hand changed', function( event ){ side = event.hand

attachArm( side )

})

Our attachArm function near line 563 in /scripts/player.js assigns a few convenience variables for internal use, but it is effectively the same as this simplified version:

const attachArm = function( side ){ controller.add( player.arms[ side ])

}

So at this point we’ve got visual models for both arms, are listening for available controllers, and as soon as those controllers report either 'left' or 'right' handedness we’ll attach the visual model to that controller instance. Phew! That was a lot! But what about interaction?

Handling buttons

VRController wraps the Web Gamepad API which handles the discovery and polling of gamepad-like devices. A Gamepad instance has standard properties —one of which is a buttons array. Each object within the buttons array has three properties: value <Number> , touched <Boolean> , and pressed <Boolean> . This is very useful information! But it can be confusing when trying to support various controller models from different manufacturers that were made for different purposes. For example, what if we decide to add 3DoF support to Space Rocks? We still want our cannon to fire off photon bolts when a user hits a button — so let’s look at supporting Vive controllers (6DoF) alongside Daydream controllers (3DoF) as a test case.

The Vive has several buttons, and the one that seems best suited for a shooting action is the trigger. A Vive controller’s trigger happens to be buttons[1] in the Gamepad instance’s buttons array. Let’s say we have a Gamepad instance called gamepad representing the Vive’s controller; we could poll for the pressed value of buttons[1] within our update loop. If its value is true, we run our cannon fire routine.

myUpdateLoop(){ if( gamepad.buttons[ 1 ].pressed ) fire()

}

But we’re not using the raw Gamepad API — we’re using VRController — and that calls for a slightly different syntax:

myUpdateLoop(){ if( controller.getButton( 1 ).isPressed ) fire()

}

VRController provides explicit support for several popular controller models by way of meticulous code comments about how the device functions and proper names as strings for each button. This means we can re-write the above as:

myUpdateLoop(){ if( controller.getButton( 'trigger' ).isPressed ) fire()

}

Look at how that almost reads like a real sentence! But maybe we only want to call our fire routine when the trigger is initially pressed and not for every single frame afterward that our user continues to hold the trigger before releasing. VRController handles this for us by emitting button events on the controller instance. Instead of polling in our update loop as above, we can try this:

controller.addEventListener( 'trigger press began', fire )

Ok, now we’re really getting somewhere, right? Daydream, however, does not have a trigger button. In fact, it only has one single button — and that’s the thumbpad! So for Daydream we’d either need to poll for gamepad.buttons[0].pressed , poll a VRController instance for controller.getButton( 'thumbpad' ).pressed , or listen for the 'thumbpad press began' event. And now we have a new problem: If we’re just listening for thumbpad presses that could mean both the Daydream’s thumpad press (intentional) and Vive’s thumbpad press (unintentional). I suppose we could check the gamepad.id string or controller.style string within the thumbpad event listener and only call fire() if it’s a Daydream controller, but that’s not so elegant is it?

This is where VRController’s last bit of button magic comes into play — the concept of a primary button. For supported controllers like Vive or Daydream that’s pretty easy to determine. Vive has a trigger and it feels like the primary button. Daydream only has a single button so it is the primary button by default. (For unsupported / unknown controllers VRController will make an educated guess as to what should be labeled primary.) So we can replace any of the above with this single line:

controller.addEventListener( 'primary press began', fire )

VRController in Space Rocks

In Space Rocks our controller’s grip buttons fire up the engines. We catch the 'grip touch began' controller event on line 693 of /scripts/player.js and set the engine rotation in motion — the user can see their engine exhaust ports spin as they are propelled forward.

controller.addEventListener( 'grip touch began', function(){ engine.rotationVelocity = 0.15

})

That’s neat, but where’s the code to actually move the player? Because engine thrust needs to be applied continuously as long as the player is holding the grips — and not just upon the initial 'touch began' event — I decided to use the controller’s updateCallback option. Just assign a function to this property and it will be executed at the end of each VRController.update() call. Here on line 666 of /scripts/player.js we can see this in action: