Modern smartphones have helped shed a light on the power of user interfaces that are driven by gesture and touch. It’s increasingly clear that touch will play a prominent role in the future of computing, but there are still challenges that make it difficult to bring the advantages of touch-enablement to conventional desktop form factors.

Depth cameras and 3D position trackers seem to offer a particularly promising route to ubiquitous touch interaction. The latest generation of the technology is still far from delivering the Minority Report ideal, but it shows a lot of potential.

Last year, Ars reviewed the Leap Motion controller, a device that uses high-precision motion capture and finger tracking to provide gesture input on any standard desktop computer. The Leap Motion controller, which plugs into a standard USB port, relies on built-in cameras and infrared LEDs to capture and analyze finger and hand movement. The Leap Motion software processes the image data, translating it into gestures and touch events.

Leap Motion provides an extensive SDK that makes it easy for developers to incorporate support for the device into their own applications. It supports several different programming languages across a number of platforms—in addition to native desktop software, Leap Motion also offers a JavaScript library that can be used to build Leap-compatible websites that work in conventional Web browsers.

Working with the Leap Motion SDK is very easy and rewarding. It abstracts away much of the controller’s complexity, exposing the hand and finger tracking data through high-level APIs. In this tutorial, we’ll describe how to build front-end Web applications that take advantage of Leap Motion tracking. We'll start by showing how to render finger points on an HTML Canvas element before demonstrating how to use the Pixi.js graphics library to make a simple 2D game with Leap Motion controls.

Leaping ahead

The Leap Motion JavaScript library relies on WebSockets to expose data from the controller to the user’s Web browser. The WebSockets standard was designed to allow JavaScript code running in a webpage to establish a persistent connection to a remote server, a feature that's typically used to build browser-based chat clients and other real-time Web applications.

When a user sets up his or her Leap Motion device and installs the accompanying software and drivers, one of the built-in software components included in the installation is a lightweight WebSocket server that runs in the background on the user’s computer. The data captured by the Leap Motion controller is pumped through the WebSocket server, making it easy to consume within a Web browser without requiring special browser plugins. The Leap Motion JavaScript library connects to the local WebSocket server, captures the data, and wraps it with some simple APIs that are very easy to use.

To get started, let’s create a webpage that loads the Leap Motion JavaScript library, obtains data from the device, and logs some of that data in the browser’s debug console:

<html> <head> <script src="http://js.leapmotion.com/leap-0.4.2.js"></script> </head> <body> </body> <script type="text/javascript"> Leap.loop(function(frame) { if (frame.pointables.length > 0) console.log(frame.pointables); }); </script> </html>

In the head element, there is a script tag that downloads the Leap Motion JavaScript library from the company’s CDN. Leap Motion provides a minified version for production use and a non-minified version for development purposes. We're using the latter in this case because it will make it easier to step through the code if we need to use the browser’s JavaScript debugger. You can visit the Leap Motion website to find the URL for both versions.

In the second script tag, which is beneath the page body , we use the Leap.loop method to capture data from the device. The Leap Motion drivers emit “frames” of data, which are processed snapshots of the controller’s video stream. The software produces roughly 30 frames every second, providing applications with a constant stream of information. The anonymous function that is passed into the loop will execute every time a new frame is available.

The Leap Motion APIs make it easy to determine the position of hands, fingers, and tools. A “tool” is a long implement, such as a pencil, that the user holds in the air. In Leap Motion parlance, the generic term “pointable” is used to describe something that is either a tool or a finger. The frame object has a property called pointables that exposes an array of pointable objects that are visible in the frame.

The example above outputs the array of pointables to the console in each frame. If you examine the pointable objects that it pushes to the console, you can see the various properties that are used to expose information about the pointable length and width, the spatial coordinates of the pointable tip, and the rate at which the pointable tip is moving.

Computing the normalized finger position

Using the pointable data, we'll make a simple demo that allows the user to control the position of an element on the screen by waving a finger in the air. The first step is obtaining the position of the fingertip. The following example shows how to obtain the raw coordinates of a single pointable:

Leap.loop(function(frame) { if (frame.pointables.length > 0) { var position = frame.pointables[0].tipPosition; console.log("X: " + position[0] + " Y: " + position[1]); } });

In many cases, you will want to take advantage of the Leap Motion software’s auto-stabilization rather than using the raw tipPosition data. The Leap Motion controller detects movement with such a high degree of precision that it will pick up the nearly imperceptible shaking that occurs when a person is holding his or her hand still in the air. There is an alternate property called stabilizedTipPosition that gives you the same data but filters out the subtle shaking. Now that we have the physical coordinates of the user’s finger, we want to correlate that with a position inside the browser window.

In Leap Motion terminology, the large virtual space tracked by the controller is called the interaction box. The interactionBox property of the frame object has a number of methods and properties that provide relevant information about the interaction box and its dimensions. It has a convenient normalization method, which converts the raw spatial coordinates of a physical point into equivalent values that represent the point’s relative position within the interaction box.

The normalized position is useful because it will make it possible for us to take the finger’s raw position in space and map it to an equivalent point within the application window. The normalized coordinates are given in the format of floating point values between 0 and 1. All you have to do to get the corresponding point in the browser window is multiply the X and Y values by the width and height of the target space:

Leap.loop(function(frame) { if (frame.pointables.length > 0) { var position = frame.pointables[0].stabilizedTipPosition; var normalized = frame.interactionBox.normalizePoint(position); var x = window.innerWidth * normalized[0]; var y = window.innerHeight * (1 - normalized[1]); console.log("X: " + x + " Y: " + y); } });

Drawing the finger position on the screen

Using the normalized position, it is now possible to draw the finger position on the webpage. To keep things simple, we'll start by showing you how to do that with a DOM element. Simply create a div with absolute position and then set the top and left properties to the X and Y coordinates that we get out of the normalization function in the Leap Motion loop:

<html> <head> <script src="http://js.leapmotion.com/leap-0.4.2.js"></script> <style type="text/css"> #position { width: 25px; height: 25px; position: absolute; background-color: blue; } </style> </head> <body> <div id="position"></div> </body> <script type="text/javascript"> Leap.loop(function(frame) { if (frame.pointables.length > 0) { var position = frame.pointables[0].stabilizedTipPosition; var normalized = frame.interactionBox.normalizePoint(position); var element = document.getElementById("position"); element.style.left = window.innerWidth * normalized[0]; element.style.top = window.innerHeight * (1 - normalized[1]); } }); </script> </html>

As the user moves his or her finger in the air over the Leap Motion controller, the div element on the screen will follow. In the examples so far, we’ve only followed one finger—using the first element of the pointables array. In the next example, we'll iterate over every element in the array in order to handle each finger. And this time, we'll paint the finger position on an HTML5 Canvas instead of using a DOM element:

<html> <head> <script src="http://js.leapmotion.com/leap-0.4.2.js"></script> </head> <body> <canvas id="canvas" width="800" height="600"></canvas> </body> <script type="text/javascript"> var canvas = document.getElementById("canvas"); var ctx = canvas.getContext("2d"); Leap.loop({frameEventName: "animationFrame"}, function(frame) { ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height); frame.pointables.forEach(function(pointable) { var position = pointable.stabilizedTipPosition; var normalized = frame.interactionBox.normalizePoint(position); var x = ctx.canvas.width * normalized[0]; var y = ctx.canvas.height * (1 - normalized[1]); ctx.beginPath(); ctx.rect(x, y, 20, 20); ctx.fill(); }); }); </script> </html>

The example above uses the forEach method to iterate over the pointable items. It identifies the normalized position of each one and then paints it on the screen as a rectangle. The clearRect method is called for each frame, ensuring that the squares drawn for the previous frame are erased.

There is one other new feature introduced in the example: the frameEventName option. By default, the Leap.loop callback will be invoked for every frame emitted from the Leap Motion controller. Setting the frameEventName option to the value “animationFrame” will change that behavior, making it so that Leap.loop is tied to the browser’s draw cycle. It will use the browser’s requestAnimationFrame API, ensuring that the callback is evoked only when the browser is ready to draw.

Detecting Gestures

In addition to finger position, the Leap Motion SDK also identifies several common hand gestures, including swipes and taps. The Leap Motion frame object has a gestures property that exposes an array of gestures detected within the frame. The following example shows how to iterate over swipe gestures, normalize their start and end position, and then draw them on the canvas:

var canvas = document.getElementById("canvas"); var ctx = canvas.getContext("2d"); var options = { enableGestures: true, frameEventName: "animationFrame" }; Leap.loop(options, function(frame) { ctx.clearRect(0, 0, ctx.canvas.width, ctx.canvas.height); frame.gestures.forEach(function(gesture) { if (gesture.type != "swipe") return; var start = frame.interactionBox.normalizePoint(gesture.startPosition); var end = frame.interactionBox.normalizePoint(gesture.position); var startX = ctx.canvas.width * start[0]; var startY = ctx.canvas.width * (1 - start[1]); var endX = ctx.canvas.width * end[0]; var endY = ctx.canvas.width * (1 - end[1]); ctx.beginPath(); ctx.moveTo(startX, startY); ctx.lineTo(endX, endY); ctx.stroke(); }); });

The Leap Motion SDK doesn’t expose gestures by default. To get gesture data, the enableGestures property must be set to true in the option object that is passed into the Leap.loop method. If the option isn’t set, the frame.gestures array will simply be empty.

Each gesture has a type property that describes the nature of the gesture. In the forEach loop, start by using a conditional expression to ignore the gestures that aren’t swipes. Next, use the interactionBox to normalize the gesture’s start and end position. Finally, use standard canvas drawing APIs to paint a line that goes from the start position to the end position.