Direct manipulation using a trackpad. With videos and prototypes!

Update (08/10/2015): Called it!

The new Apple TV is eternally imminent. People like me are waiting eagerly for it, since the current Apple TV UI hasn’t gotten any sort of relevant refresh in a long while, and it leaves a lot to be desired. I got tired of waiting. I’ve had this idea for a while now, and I finally decided to sit down and take a stab at it.

Every time Apple comes out with a new product category, it usually brings along a new interaction method. Here’s my idea for an interaction method on a remote, big screen.

Direct Manipulation

Remember what was great about the original iPhone? You could slide your finger across the slab of glass and the pixels below it would respond as naturally as technologically possible to your actions. You could pinch photos to zoom them and scroll lists like never before.

“You had me at scrolling”

If there’s one great thing about iOS’s UI (and there are many) it’s UIScrollView. It embodies the principles of natural, direct manipulation of objects on the screen with a very responsive and smart scroll experience and very natural-feeling inertia calculations, with bounces and everything.

Let’s go back to the current state of the Apple TV interface. Clicks is the name of the game. There’s 4 directional buttons and a center button on the physical remote, and there’s a Remote app on the iPhone that does its best to translate taps and swipes into discrete clicks — the only input method the current Apple TV can understand. It’s actually the Remote app that inspired this whole thing.

Wouldn’t it be great if instead of trying to turn finger gestures into artificial clicks, Remote acted more like a trackpad — directly responding to the position of your finger on the tracking surface? Selection in a complex button layout — such as the on-screen keyboard —would be much more usable if you could make the cursor move in any direction, instantly. What if, instead of relying on an app for this, the new Apple TV remote control had a smooth, clickable (or force-touch enabled) trackpad, that allowed for these behaviors without having to look at a second screen?

Well, that’s the whole idea. I thought it would be worth a shot, so I built a prototype of the interaction model that you can try using an iPhone and AirPlay. I don’t know if it’s the definitive answer for Apple TV interaction, but I found it to be better than the current state of things.

(This interface paradigm has been done before, but it’s only available to TimeWarner Cable subscribers, so you know it won’t ever have the global reach the Apple TV has. I probably won’t ever touch it, since I’m from Chile. Still, this is a fun exercise that shows the untapped potential living within the Apple TV today, and you can try it out for yourself, if you’re a developer.)

The Remote

This new interface is to be driven by a new physical remote. This remote would have a large tracking surface, plus some other buttons that would aid navigation, like so:

The proposed remote next to its current incarnation for reference

The big delimited area would be a ‘click-pad’, a trackpad you could click, not unlike the MacBook’s. It’s wider, to allow for comfortable finger tracking, making room for 3 buttons below the pad instead of just two, breaking down ‘MENU’ to a clearer pair of Back and Home buttons. It even has less buttons than the current one — 4 versus 7. I’m not an industrial designer, this is just a simple proposal based on the interaction model.

The Prototype

This is an interaction demo, built for the iPhone and AirPlay, that aims to show how a remote like the one described above might behave while controlling an Apple TV. It’s not particularly pretty and doesn’t do stuff. It simulates some current Apple TV interface patterns to evaluate whether or not direct manipulation works better than discrete clicks. (I think it does.)

I know you just want to see it in action, so here it is:

If you want to try it out for yourself, you need to be a registered Apple developer in order to compile apps to your devices, an iPhone to run the app, and an Apple TV to AirPlay to. You can grab the code on Github. Make sure you enable AirPlay Mirroring from your device to your Apple TV before you run the app. Trackpad clicks are simulated by lifting your finger and tapping on the tracking surface — on an actual device, you wouldn’t need to lift your finger to click.

Although these demos are simple, you can see the potential for more complex interactions using edge gestures, for example. Here’s what I thought a Now Playing screen could look like with a sort of Control Center popping up from the bottom, invoked using an edge gesture: