One of the most interesting parts of the Nintendo Switch has gone nearly unused by game developers, including Nintendo, for the first year of the system’s existence. A mind-blowing demonstration yesterday, however, showed that the little sensor on the bottom of the right Joy-Con is more capable than we ever imagined.




Specifically, it can be harnessed to transmit simple video feeds to the Switch screen or even detect heat sources. The sensor’s capabilities have been hiding in plain sight for a year, and are about to be used to power an impressive array of gaming experiences.

Nintendo shows off the IR camera sensor at a Switch showcase event in Tokyo in January 2017.


Nintendo first showed off the sensor at a pre-launch Switch event in January 2017. There, the system’s producer, Yoshiaki Koizumi, pointed out that the right Joy-Con’s “motion IR camera” could be used to detect the shape and distance of something held in front of it. He showed that the sensor could distinguish the shapes of a hand making gestures in the game Rock, Paper, Scissors.



The motion IR camera was used in the March launch game 1,2, Switch for an eating contest challenge. With the sensor pointed at the player’s face, the system could track the opening and closing of a player’s mouth.

1,2 Switch is the rare Switch game that already uses the right Joy-Con’s IR motionsensor.

The sensor wasn’t used for many other 2017 games. It’s understandable, given that it is positioned to point at the the floor or at a player’s torso when they’re holding the Joy-Con. A Resident Evil re-release seemed to use it to track a gesture for reloading a gun, but otherwise, it hasn’t seemed to be good for much.


But the Joy-Con (R)‘s humble sensor is getting its day in the sun with the Nintendo Labo project, which combines the various parts of the Switch console with folded cardboard. Many of the Labo builds involve holstering the right Joy-Con so that it points inside a cardboard piano or backpack or other enclosed space, where it can then detect the movements of specially marked stickers, sending signals back to the Switch to, say, emit different piano sounds or register different motions in a robot punching game.



Image via Nintendo’s trailer for its Labo project.


What’s more extraordinary, and what Nintendo has barely hinted at up until now, is that the “motion IR camera” can indeed function as a camera. A Labo game that positions the Joy-Cons on the side of a skittering cardboard car is set up so that the sensor takes transmits a live, infrared feed of what it is pointed at to the Switch’s screen.

Look inside the green square on the Switch’s screen and you’ll see a live feed of what— or in this case, who—the sensor is pointing at.


In that same car mode, the IR sensor can be switched into a heat-sensing view, which it allows it to display which sources of heat it can see.




And when the car is set to auto-drive, that heat-sensing mode, combined with vibrations of the Joy-Con controllers, can be harnessed to make the car move toward any heat source. That could be your hand, or your dog, or your fireplace, so watch out next Christmas morning.

Leading the heat-sensor while the Nintendo Labo contraption is in auto-drive mode. It’s chasing the heat source.


More chasing of the heat-source, but going in circles this time, possibly due to a hitch in how the system is calculating how much vibration it needs in each Joy-Con to move the car toward my hand. In another demo, a Nintendo presenter was able to use his hand to drive a similar car in a long, straight line.

Suddenly, the mind boggles at what Nintendo can do with a camera that’s been built into every Nintendo Switch, but it’s clear that Nintendo has already been thinking of some fun stuff. After all, the initial trailer for Labo showed the right Joy-Con being used as the lens of a big cardboard camera. That may seem structurally redundant, and it’s not like the IR motion camera is going to be shooting high-res shots, but there’s still some pretty cool potential here:




Image via Nintendo’s trailer for its Labo project.

It’s enough to make you wonder what other secrets are buried in the Switch.