Better testing leads to better games

I felt like a ghost inside the Matrix.

I was standing inside a representation of Waltz of the Wizard by Icelandic developer Aldin Dynamics, but I wasn’t in the game itself. Instead, I was watching someone else play the game from within a sort of pointillistic version of the environment, studying what they do. I was standing in an HTC Vive, watching a virtual version of a virtual world.

I accomplished this with program that Aldin is hoping will help lead to better virtual reality games.

"Since the early Development Kit 1 days it's been really apparent how important it is to run users through testing," Hrafn Thorisson, CEO and co-founder of Aldin, told Polygon. "There's no way to fully understand the user experience with what you designed without running people through it and [seeing] what they do ... how they feel, how they interact with your world, how they break your programming. It's key for virtual reality to really see how it impacts your users."

Developers, as of this writing, need to shoot video of players trying their game if they want to rewind and study how people interact with the game. You can capture the game itself, but depending on your game that might limit what you can see to a first-person view. Aldin was frustrated by how hard it was to learn about how people played its games in a scientific way.

"We just wanted to make a tool to help us do that," Thorisson told me when I visited the company's office in Iceland. "So we came up with this, which is called Ghostline."

A history in VR

Aldin Dynamics was founded in 2013, and released Trials of the Rift Drifter in the same year. It was one of the first Rift games to find a wide audience on the Oculus Share platform, and made the team members minor celebrities in the nascent virtual reality scene.

The history of Aldin part 1 Trials of the Rift Drifter July 2013 Trials of the Rift Drifter was a very early experience released for the Development Kit 1, the very first piece of virtual reality hardware Oculus released to developers so they could start creating games. "Back then VR development was very community driven, meaning that both sharing information and getting feedback from the community was a huge part of what made the development so much fun and rewarding," Gunnar Valgardsson, co-founder and CTO of Aldin told Polygon. "The work was understandably very experimental at the time, but from the very beginning we were interested in giving people the sense of having experienced something new," he continued. "Something unique that would only be possible in VR. For instance in Trial of the Rift Drifter you are placed in a strange trial scene where a group of ominous judges asks you questions and you answer by shaking or nodding your head." It sounds basic now, but at the time the sense of being someplace else, and being able to interact with that world through head movement, helped set the company apart. The demo opened a lot of doors. "The reception was absolutely phenomenal," Thorisson said. "It was played by a majority of the VR community and we’re eternally grateful to the support we got from early VR adopters. Shortly after the initial release one of the co-founders of Oculus got in touch and invited us to have it featured for the launch of Oculus Share. That was the start of our formal relationship with Oculus, which continues to this day."

More releases and experiments followed, as the hardware improved and more platforms, like the PlayStation VR, were announced. "The speed of the growth in the technology itself has been incredible," Gunnar Valgardsson, the other co-founder and CTO of the company said.

What the team members learned was that it was common to engage in "sprints" during VR development and then test each virtual world by manual observation of how people played the game. Sometimes they would film people playing their games and try to figure out what the players were feeling by watching the footage.

"It was a nightmare to work with, and it really hinders what you can design, and how interactive you can make an experience," Thorisson said.

The more a game responds to you, the more testing is required to make sure players can't easily break the interactions. Suddenly the things that make your game feel alive become a limiting factor.

Thorisson and Valgardsson became friends in college when they noticed they were both spending a lot of time using the creation tools in Second Life, and in 2013 they were planning on starting a software company together to work on creating new kinds of games using AI. They had long been interested in VR, but the existing hardware was both expensive and low quality.

The Oculus Rift Kickstarter changed everything for them.

"The day our Rift DK1 kit arrived was a defining moment for us," Valgardsson said. "Here was the most powerful medium in history being born and to us it felt like a blank canvas — an opportunity to define a new industry. We decided to dedicate our careers to creating new realities."

Aldin Dynamics is a relatively young company — Thorisson and Valgardsson both seem endearingly childlike and enthusiastic in person — but it already has an impressive track record in VR since its founding in 2013.

It has released multiple games and demos across the different generations of the Oculus Rift and Gear VR, along with projects coming to the HTC Vive and PlayStation VR. The team's office is filled with VR hardware, with multiple generations of development kits from all the major platforms on display. It's working on content and experiences across multiple platforms, but it's also working on making the process of developing for VR, while analyzing player behavior in your games, to be as useful as possible the developer.

How to test for emotion

Ghostline is a drag and drop plugin aimed at VR developers game that allows you to record and track how people play your game during play tests. You can watch someone play in the first-person view, or from the top-down or using a free-roaming camera.

Part 2: Asunder (2013, DK1 and DK2 ) "Asunder was one of the first story-driven cinematic VR experiences, letting you feel like a character in an interactive movie," Thorisson said. "We were learning to leverage VR’s unique strengths to drive a story forward, together with characters that responded to your actions." The game reacted to you, and you could react back using head motions. Characters knew when you were looking at them, and you felt like you were in the middle of the story instead of just watching it. It was also something rare for the time: A commercial release. "Considering the size of the early VR adopter market in 2013, the sales surpassed our expectations and we topped 1,000 copies sold in the 12th week from launch," Valgardsson said. "The price was premium for a short experience, but we were glad to see the early adopters appreciating the hard work and details we put into the production. The positive feedback was amazing and to this day we get regular requests for updates from people saying Asunder is one of their most memorable VR experiences." Asunder also served as the inspiration behind Ghostline. Making a world that reacted to the player wasn't easy, and testing for every possible way someone could play it proved tricky. "After its release we studied dozens of gameplay videos to broaden our understanding of VR design and to improve our skills," Thorisson. "It was clear that user testing would be vital for all our VR products — and manual observation and videos were extremely time consuming and inefficient. That’s how the concept of Ghostline was born; as an in-house tool to understand our users, helping us build more engaging and polished VR content."

You can even, as I did, step inside the testing environment and watch through VR.

The plugin tracks the position of objects and presents you with a version of your game or experience that looks like it was made from a large number of points of light. Each demo you "record" only takes up a few megabytes of storage, since Ghostine is only recording positional data. You're left with a stylized version of your program, as seen by the plugin, which is good enough when judging how people interact. After all, you know how your own game looks.

It's amazing how much emotion you can intuit by watching even a stylized version of the player from within a world of Ghostline. If someone tries to grab something multiple times but can't, their body language changes. People hold their body differently when a scene or item they're holding delights them. You can't see facial features through the tool, but I could tell exactly how the player was feeling. It's fascinating.

"In VR it’s not just about analytics, it’s about how the player felt when she was doing it," Thorisson said. "So we can look at it and see the body language of the person. Was she really interested when she picked put he crossbow? And did she have trouble figuring out how to work it? All of this, this is what we see as our enabler for doing more complex and more convincing virtual worlds that really impact the users in the way that we want."

The tool also allows you to look at each object in the world individually, and the players' interaction with those objects. By selecting an object in the game world (in the case of Waltz of the Wizard you could select the spyglass, for instance) you then see a timeline that shows all the time the player interacted with that object and jump to that moment in the demo. If you want to watch 10 people playing and jump straight to the moment they pick up a certain item and make sure each and every one of them knows how to work it? You can do so quickly and easily.

By looking at how many times players interact with certain objects, and then watching their body language once they try to use that object, you can fine tune everything from where it's placed in the world to how it works.

Part 3: Twisted Realms (2015, Gear VR) "Having partnered up with Oculus Studios and worked side by side on releasing Twisted Realms was a great experience," Thorisson said. Both men also seemed a bit frustrated by the current limitations of mobile VR. "For now we think mobile is a good introduction to VR, but from what we’ve seen it always leaves people wanting more," Thorisson said. "Usage will increase as the devices get lighter, grow in processing power and upgrade body-tracking capabilities. It’s going to take some time, but untethered is of course where we all want to see VR going before long."

You may select an object and notice that players aren't picking it up until the end of the demo, and figure out how to make it more prominent in the space. Or you can see if an object is repeatedly picked up, only for the player to put it back down and forget about it if they couldn't figure out what it was supposed to do. You can track how many times an item was looked at or picked up, when the player did so and then jump straight to the moment and watch them use the object via a standard monitor or from VR itself.

Ghostline made it possible for Aldin to look at large numbers of people playing through its games and jump to every moment those players tried a particular spell, for instance. They could figure out how long it took players to learn that spell, and then watch them use it and see if it made the player happy or frustrated them.

"It started as an in-house need for us, because it was delaying and slowing down our development process that we didn’t have a good way to jump into record and compare and see and answer questions about our designs," Thorrison said.

"That’s the way it started, but two years ago we saw these launches; they weren’t going to be like rocket launches when they get deployed, these headset launches," he continued. "We were a content company, and we were faced with that knowledge that this won’t take off instantly. It’s going to be a gradual process, and the content we want to develop, it’s really complex, it’s really difficult to make these living worlds. We need better tools to do that. So that’s where this came from initially."

As the team members began to use the tool themselves, they started to realize what they had was a commercial product, although they were a bit hesitant to talk pricing. "We are establishing early partnerships for trials of the initial version; from there we’ll refine its most valuable features and expand," Thorisson said.

"We want to release Ghostline as soon as possible because there’s such a need for a tool like this," he continued. "The industry is constantly prototyping completely new kinds of VR content with no precedents for what works or a way to measure how it engages users. Our hope is that Ghostline will help cut VR production costs and accelerate content development across the industry."

Waltz of the Wizard is coming soon for the HTC Vive, and it almost feels like a proof of concept for the power of Ghostline. It's a toy just as much as its a game, with many objects in the world you can pick up and play with. It rewards experimentation, and it feels like you can do everything you want to do, as if the software is bending itself to be comfortable.

The game offers you a selection of toys with which to play, and there are so many little ways to be creative and test the boundaries of VR. Waltz of the Wizard does a great job of making you feel powerful one minute and fragile the next. I've visited Aldin Dynamics every year during CCP's Fanfest to see what they're working on, and many of their experiments have been interesting, but Waltz of the Wizard is the most polished, and the most confident.

Part 4: Waltz of the Wizard (May 31, HTC Vive) We've been playing Waltz of the Wizard from early previews to a version of the software that's close to finished (the video included in this story is from an earlier version of the game) and the change is remarkable. The interactions feel natural, the included spells and toys are fun to play with and easy to understand, and scenes last long enough to create tension but not so long they become frustrating. You mix potions, play with the items around you and the whole thing feels magical. Be careful! That desk isn't real.

That confidence comes from testing, and the testing has been made much easier due to Ghostline. The more you can watch people play your game, and drill down to how they're using each individual part, the more you can make the illusion of your world's reality effective.

Aldin has spent years of iterating and working on virtual worlds and the tools needed to create and perfect them, and these two programs are some of the most impressive uses of VR I've seen yet, both as a player and someone who pays attention to the development challenges of the medium.

Trying to create worlds that acknowledge your presence is a harder task in VR than many people assume, and the ability to run a large number of tests, learn from them and iterate rapidly is a huge advantage during development.

There's no way to know how people will act inside your world until you put them in it, after all. "Humans," Thorisson said, "are pretty much the most unpredictable thing in the world."