The world of NES emulation hasn't been all that exciting since the late '90s, when NESticle provided "good enough" emulation accuracy and stability for any NES game out there (though there has been a lot of subsequent work to get that final bit of true emulation accuracy). So it was a bit of a surprise this week to stumble across a new NES emulator that provides a genuinely new perspective on decades-old games by rendering them in three dimensions.

The 3DNES project, as the name implies, extends the 2D sprites of the NES into the Z axis, letting players rotate the camera around to see the sides and back of the formerly flat sprites. This isn't just a conversion of every pixel into a uniform voxel, either. In a game like Super Mario Bros., for instance, 3DNES converts pipes to into cylindrical 3D models, with bulging piranha plants embedded in the center. In Mega Man, ladders remain in the background while wall-hugging enemies are accurately placed on the sides of thick blocks.

The emulator's developer, who goes by the handle "geod" online, has been posting videos of the work-in-progress emulator for months. This week, he finally took his work public, posting a playable beta version that runs through the Unity Web player and can load arbitrary (and definitely legally obtained) ROMs from the cloud.

Searching for shapes

The beta is still quite rough (and it only works in Firefox for the time being), but the fact that it works at all is pretty impressive. How exactly does an emulator take the flat 2D images produced by the NES and intelligently convert them into a 3D model?

Your first assumption might be that geod has hard-coded specific algorithms for each supported game, training the emulator in how to detect specific objects and create good-looking 3D models for them. But in online postings, geod insists that his algorithm, while "still far from perfect and there are still many things to improve," is generalized enough to work on any NES game. "Everything is calculated automatically in runtime. Nothing is done manually," he wrote on emutalk.

The core, geod writes, is a simple shape-detection algorithm, which looks at each part of the 2D image generated by the emulated PPU and determines what kind of 3D object it should be. "Basically, game objects contain full of blocks and cylinders [sic]," geod wrote on TASVideos. "The pipe in SMB is determined to be a cylinder like many other game objects. There is not any specific code for any game."

Geod went in to a little more detail on the algorithm in an NESDev thread, saying, "The rendering mechanism is not just based on [an] 8x8 [pixel sprite] tile, but based on "shape"—[a] meaningful collection of consecutive tiles... Consequently a 2D tile is not always mapping to a '3D tile.' For example a 2D solid tile can be used in the introduction cube at the begin of SMB but [could] also be used in grass or cloud. The corresponding 3D tile in each case is completely different."

"Shape recognition—like any recognition problem—is hard and does not have an ultimate answer," geod continued. "On NES system, fortunately, we can choose color pattern as the main factor for shape clustering, it works for 90% case [sic]. It's still an open problem for improvement though."

The current algorithm definitely could use some improvement, as it works better on certain games than on others—geod notes that Super Mario Bros. works particularly well because it's the main game used for internal testing. Some games are internally inconsistent; in a quick test of Mike Tyson's Punch-Out!!, characters like Glass Joe and Doc had a somewhat believable (if angular) 3D shape, while Little Mac appeared like a thick, woodcut block without any real 3D definition.

Going forward, geod says he plans to release his emulator as a Unity3D-powered executable, which should be a bit more stable than a Web-based plugin. He also wrote about specific improvements he's working on for the 3D detection engine.

"In the future, more semantic info will be collected from [the] 2D scene, more predefined patterns will be added, more rules will be created to make the emu more robust," he wrote. "And when the emu becomes mature enough, the optimization for a specific game is as simple as values assigning for a set of coefficient. That's how I see things. ... You say it right there will never be a perfect solution for this kind of problem but [a] better solution is always possible."