Towards declarative touch interactions

January 2015 by Ralph Thomas January 2015 This page outlines Slalom, a system with prototype implementation for concisely describing and implementing a class of touch interactions. Various examples demonstrate Slalom's versatility. Slalom grew out of my previous article on how to use physics simulations to build touch-driven user interfaces. When writing that, I observed that I was writing much of the same code across the examples and that this repetitive, often inflexible code plagues most touch device engineering. Most touch interactions on today's tablets and smartphones are implemented by writing new code to handle every touch movement, which computes various transforms and updates the application's view tree or the browser's document object model (DOM). The coder needs a reasonable understanding of Newtonian physics to create something that feels good or natural to use. As a result, most apps use only the few touch interactions provided by the operating system: scroll and tap. Many touch interactions involve the user's dragging a visual object such as a panel, a photo, an icon, a window or any other structural element of the user interface (and the object's moving synchronously under the touch point) and imparting momentum to that object, such that when released the object keeps moving with some momentum. The touch handler that you write will impose some constraints on how and where an object can move, such as constricting it to a single axis and clamping the distance it can travel. If you're implementing a complex interaction, then every time you iterate your idea and want to change something, you have to rewrite a lot of manual, very detailed code. What if we came up with a system where we just specify how something moves and what the constraints are? Slalom Slalom combines the Cassowary linear constraints solver with a minimal physics engine to solve a class of touch interactions declaratively. It has the following components: Linear Constraints are used to place objects relative to each other and to restrict how they can move. Slalom uses the Cassowary linear constraint solver which has found a niche in 2D-layout recently (it's the solver used by Apple's AutoLayout and Grid Style Sheets). If we think of a touch interaction as a slot-car game, then the linear constraints let us define the track on which the slot-cars will travel.



For example, this is how we'd use linear constraints to position an image that is wider than its parent view, whose size is often the size of the screen (in essence, you're making a small viewport onto a large photo, showing just a part of the image): scaledPhotoWidth = (parentHeight / photoHeight) * photoWidth photo.y = 0 photo.bottom = parentHeight photo.right = photo.x + scaledPhotoWidth A Manipulator listens for touch events and updates a Cassowary variable in response. The manipulator knows how to use the momentum to create an animation when a touch gesture ends. In our slot-car metaphor, the manipulator is both the hand-held car controller (with accelerator) and car.



To make our image draggable, we would create a manipulator for it like this: Drag the photo left and right

// Drags in x on the parentElement will change the photo's x coordinate. context.addManipulator(new Manipulator(photo.x, parentElement, 'x')) Motion Constraints place limits on how far an object can move or where it can stop. Motion constraints are similar to linear constraints, except they're enforced by physics simulations (typically springs) so that when a variable being controlled by a manipulator violates a motion constraint the object bounces or undertracks the finger. Motion constraints would be like a rubberband stretched across our slot-car track or like a speed bump on the track.



We can add motion constraints to our image example like this: Drag the photo left and right and release it

// Prevent the photo's left edge from going right of the parent's left edge: // photo.x <= 0 context.addMotionConstraint(new MotionConstraint(photo.x, '<=', 0)) // Prevent the photo's right edge from going to the left of the parent's right edge: // photo.right >= parentWidth context.addMotionConstraint(new MotionConstraint(photo.right, '>=', parentWidth)) Check out the source code for this example on GitHub. Now you can drag and flick the image without its vanishing into the distance. The above Slalom example describes full-momentum scrolling with just one manipulator and two motion constraints. It handles all of the tricky edge cases, such as when the user starts dragging while the photo is already bouncing on one of the edges, or when the user imparts momentum while in the overdrag. In systems where there are multiple manipulators, Slalom identifies the correct manipulator that is causing a motion constraint violation and applies feedback to it. More examples Here are some more examples with links to the source. All require significantly less code and the code is cleaner using Slalom than when using the imperative alternative. Vertical scrolling lists

Drag these lists vertically





Linear constraints

The list items are 40 pixels tall: panel[i].bottom = panel[i].y + 40 The list items are stacked vertically in a column, with a 10-pixel gap: panel[i].y = panel[i-1].bottom + 10 Motion constraints

The first list item can't have a vertical position more than 0 pixels from the origin: panel[first].y <= 0 spring The last list item's bottom position can't be higher than the parent height: panel[last].bottom >= parentHeight spring Variation (list on right)

In the list variation on the right, we just prevent the boxes from leaving the parent by imposing extra linear constraints: panel[i] >= panel[i-1].y + 3 panel[i] <= panel[i+1].y - 3 panel[first].y >= 0 panel[last].bottom <= parentHeight Now our panels stack with a padding of 3 pixels, and the first and last panels don't leave the parent. The list items are 40 pixels tall:The list items are stacked vertically in a column, with a 10-pixel gap:The first list item can't have a vertical position more than 0 pixels from the origin:The last list item's bottom position can't be higher than the parent height:In the list variation on the right, we just prevent the boxes from leaving the parent by imposing extra linear constraints:Now our panels stack with a padding of 3 pixels, and the first and last panels don't leave the parent. Check out the source code for this example on GitHub.

Twitter for iPad The next example reproduces the old Twitter for iPad panels interface. The second variant introduces some of the non-default properties of motion constraints.

Drag the panels horizontally and release to impart velocity Notice how these panels animate to be either open or closed

Linear constraints

The left edge of each panel can't go past the left edge of the panel that came before, plus 10 pixels. panel[i].x > panel[i-1].x + 10 The left edge of each panel can't go past the right edge of the panel that came before. panel[i].x < panel[i-1].right Motion constraints

The first panel's left edge is constrained to the left edge, which is 0 pixels. This is a springy motion constraint. Here the motion constraint applies a spring, but you could easily make a motion constraint that makes the panel rebound or that completely stops the motion. panel[first].left = 0 spring Variation (lower example)

The second variation adds a motion constraint on the gap between two cards: either the gap is 10 pixels or it's the full panel width, 250 pixels. We also specify that the motion constraint shouldn't be enforced while dragging (overdragCoefficient: 0) and that this motion constraint's animation should keep running even when the constraint is satisfied, which is handy when using underdamped springs. This is the "captive" option.



First, define the gap using a linear constraint: gap = panel[i].x - panel[i-1].x Then add a motion constraint on the gap using the "or" operator: gap = 10px || 250px spring {overdragCoefficient: 0, captive: true} Check out the source code for this example on GitHub. The left edge of each panel can't go past the left edge of the panel that came before, plus 10 pixels.The left edge of each panel can't go past the right edge of the panel that came before.The first panel's left edge is constrained to the left edge, which is 0 pixels. This is a springy motion constraint. Here the motion constraint applies a spring, but you could easily make a motion constraint that makes the panel rebound or that completely stops the motion.The second variation adds a motion constraint on the gap between two cards: either the gap is 10 pixels or it's the full panel width, 250 pixels. We also specify that the motion constraint shouldn't be enforced while dragging (overdragCoefficient: 0) and that this motion constraint's animation should keep running even when the constraint is satisfied, which is handy when using underdamped springs. This is the "captive" option.First, define the gap using a linear constraint:Then add a motion constraint on the gap using the "or" operator:

Animation using gravity A manipulator doesn't have to use a friction simulation for the animation it creates after you let go of a drag. Here's an example using gravity instead:

Toss the heavy box up

Motion constraints

We only have one motion constraint here, which is that the bottom of the box can't go outside the parent. It's enforced by a slightly underdamped spring (the default) giving our heavy box a soft landing: box.bottom >= parentHeight spring Check out the source code for this example on GitHub. We only have one motion constraint here, which is that the bottom of the box can't go outside the parent. It's enforced by a slightly underdamped spring (the default) giving our heavy box a soft landing:

Photo viewer Another really useful operator for motion constraints is modulo, "%". Here we use a motion constraint that isn't enforced during dragging to ensure that our photo viewer animates to show only one photo. This actually uses a variation of modulo called "adjacent modulo" to prevent the user from flying past many photos with one big high-velocity flick.

Drag the photos horizontally

Motion constraints

We have three motion constraints in this system. The first and last photos can't go away from the edge of the parent (otherwise you could scroll to before the first photo or to after the last photo): photo[first].x <= 0 spring photo[last].x >= parentWidth spring Also, the scroll position must be a multiple of the photo width (plus padding): scrollPosition %= parentWidth + padding { overdragCoefficient: 0, captive: true } The scroll position is a variable that's added to every photo's x coordinate. We have three motion constraints in this system. The first and last photos can't go away from the edge of the parent (otherwise you could scroll to before the first photo or to after the last photo):Also, the scroll position must be a multiple of the photo width (plus padding):The scroll position is a variable that's added to every photo's x coordinate. Check out the source code for this example on GitHub.

Scaling like Facebook Paper In Slalom, we're not limited to just translating objects. If we add a manipulator to the y-coordinate of a box and constrain the bottom of the box, then we can grow and shrink the box. Notice that we're expressing motion constraints on variables that are somehow related to scale but aren't the scale variable. We can do this because Slalom discovers the relationship between two variables by inspecting the Cassowary simplex tableau—this is a huge advantage over a numerical constraint solver where it would be harder to find how two variables relate to each other. Currently the manipulator consumes all drag events. If it passed on unused drag deltas then we could add a second manipulator to translate the box horizontally, creating something like Facebook's Paper UI.

Drag the image upwards to grow it



Linear constraints

Constrain the aspect ratio of the box (aspect = parentWidth / parentHeight): box.width = box.height * aspect Relate the height to the scale: box.height = scale * parentHeight Pin the box to the bottom of the screen: box.bottom = parentHeight Center the box horizontally: (box.x + box.width/2) = parentWidth / 2 Motion constraints

The box's width must be greater than 150 pixels. We're really expressing a constraint on the minimum scale, but because we can get the coefficients out of Cassowary, we can write this in terms of the box's width. box.width >= 150 spring The box's top mustn't go above the top of the screen. Again, this is really a scale constraint (the scale can't be more than 1.0), but we're expressing it in terms of box.y because that's more natural. box.y >= 0 spring Check out the source code for this example on GitHub. We manipulate the box's y-coordinate variable for vertical drags.Constrain the aspect ratio of the box (aspect = parentWidth / parentHeight):Relate the height to the scale:Pin the box to the bottom of the screen:Center the box horizontally:The box's width must be greater than 150 pixels. We're really expressing a constraint on the minimum scale, but because we can get the coefficients out of Cassowary, we can write this in terms of the box's width.The box's top mustn't go above the top of the screen. Again, this is really a scale constraint (the scale can't be more than 1.0), but we're expressing it in terms of box.y because that's more natural.

Note that we could tweak the constraints further. For instance, we could make the box go off the bottom when it gets too small instead of shrinking further. So we have a lot of flexibility with this kind of system. Also note that there's slip! If you drag from the middle of the box, you'll see that the part of the image you grabbed slips out from under your finger. That's because the manipulator is just operating on the box's y coordinate. If we wanted to avoid slip then we'd have to create a new variable when the finger goes down, relate it to y (that is, fingery = box.y - 123 * scale, based on the current scale and finger start position) and then manipulate fingery instead of box.y. This isn't very hard, but the current manipulator code doesn't support it. iOS notification and control centers This next example is an implementation of the iOS notification center and control center UIs. The notification center is actually very interesting because it uses a different physical model when coming down (positive gravity) than when going up (anti-gravity). It looks weird if it slows down while going upwards. I was able to do this by creating a custom motion constraint operator that changed its physics model depending on the end point. (For the purposes of this paper, I opted not to implement the arrow buttons within the UIs but it should be straightforward to do so, either outside Slalom or by extending Slalom.)

Drag down from top to show notification drawer; drag up from bottom to show control center

Meetings You will be in meetings from dawn til dusk today. A chill breeze blows from the East. Your body starts to shiver, for no mere mortal can resist the ...You will be in meetings from dawn til dusk today.

iOS application switcher This example demonstrates a way to make the iOS application switcher using Slalom. There are two manipulators: one which controls the first app's x-coordinate, and one which controls the first icon's x-coordinate. The icons are offset by half the scroll offset of the apps, so dragging the icons (small squares at bottom) moves the apps (large rectangles at top) twice as fast. Even though we have two manipulators, we still need only two motion constraints. The real iOS app switcher applies some non-linear translation to the icons (based on velocity?) which I am unable to represent with linear constraints.

Drag horizontally on the "apps" or "icons"

Windows overscroll In Windows 7, Microsoft implemented touch overscroll by translating the whole window rather than just the content. We can use linear constraints to make this happen too, and we need only one motion constraint (the window should stay where it is) to bounce things back.

Drag the image vertically to scroll

Google Maps details transition This example implements the complex transitions that occur when the user drags upward within a Google Maps location view. The linear constraints are fairly straightforward, while the motion constraints that prevent the UI's stopping half-way open reflect the complexity of the transitions. The motion constraint is predicated on multiple variables, so I wrote a custom operator to handle it; I'd like to be able to express this more elegantly though, possibly with some kind of a Range type.

Drag upwards on the blue info bar to see more information Grassy park with tennis/handball courts, horseshoe pits, playgrounds, splash pad & a fenced dog run. 600 East Meadow Drive, Palo Alto, CA 94306 Open 24 hours More Info... Suggest an edit Street View 26 Photos Add a photo... Add a photo... Review Summary Reviews Mitchell Park 4.5 stars 5 minutes

Future directions We can use Slalom to describe a wide variety of touch interactions. There's a lot left to do, however, some involving further research: Non-linear constraints are needed for some interfaces (one example is the Chrome for Android tab switcher—the tabs move less when they're at the top) and these can't be expressed with Cassowary. For simple "external" variables with no dependencies it would suffice to provide a non-linear mapping function (and inverse function).

Manipulators recognize only one gesture currently. I plan to extend the Manipulator class or subdivide it so that it's possible to manipulate multiple variables with one gesture. This would allow the above scaling example to become more like Facebook Paper.

Various parts of Slalom need better explanation, including the captive flag on the Motion Constraints class.

Currently there are no state or conditional constraints. To tackle more ambitious interactions, we need a notion of state that enables and modifies constraints, similar to how class names can be used to change CSS properties. Grid Style Sheets handle some of this and manage to avoid feedback loops.

Currently there is no transition support. Often you want to use a tap to change some value (for example, the iOS notification panel and control center should hide when you tap the arrow buttons). It's also common to want to transition from a resting position to where a touch point (which is moving) will be.

Slalom could be ported to one of the mobile platforms to ensure it integrates well.

A parser could be written for linear and motion constraints, once their syntax is defined. I want to do a native port first to identify any syntax or parser changes needed to support integration with a traditional toolkit.