Ultimately, according to Schuh, it's up to the developer how much they use the feedback from the User Research group. And 343, he says, is one of the most active teams involved.

When Schuh talks about User Research testing making games better, he says it with confidence. There's no doubt in his mind that it significantly improves games to use data to smooth out experiences.

Not every experience wants to be smooth, however. Creative types often cite a desire not to get too much outside input on their ideas. The directors of the recent Indie Game: The Movie documentary bookended their film with a quote from Braid designer Jonathan Blow — the only central quote to appear in the movie twice — highlighting this divide.

"[Big companies] create highly polished things that serve as large of an audience as possible," he said. "The way that you do that is by filing off all the bumps on something; if there's a sharp corner, you make sure that's not going to hurt anybody if they bump into it or whatever. That creation of this highly glossy commercial product is the opposite of making something personal."

Blow's comments refer less literally than they do creatively, and few would claim that Halo is trying to be a game about one person's vision, but his overall point carries over — too much external input can sometimes push developers into areas they would prefer not to go. Changing the geometry of a level to make it more user friendly, for instance, may make it less attractive. Like the thicker taco shell, fixing a problem doesn't always create a better product.

Schuh and the designers at 343 make a clear distinction here, saying they're trying to perfect a creative vision rather than change its substance, noting that ultimately the design choices come down to the design team. They use the phrase "confirming intent" to point out that this process is not about challenging creative ideas so much as discovering whether those ideas resonate.

"It comes down to what the intent is," says Holmes. "There are moments where you want the player to feel that sense of being a little bit lost and disoriented. And then it's more about finding the balance of, 'How long do you want them to maintain that sensation before giving them the information that they need to drive forward?' So a lot of it just comes down to testing against intent. There's no magic set of metrics where you're like, 'OK, you've just got to tick off these boxes and then you've got yourself a great experience and a great level.' You go in with a very clear intention, but then you're testing to see that you are actually achieving that intention."

Ultimately, according to Schuh, it's up to the developer how much they use the feedback from the User Research group. And 343, he says, is one of the most active teams involved.

"This was my first experience really working with this sophisticated user research process and group, and it's been a godsend," says Holmes. "I've worked in the past at companies that put a lot of effort into testing at the concept level — doing focus testing, and trying to figure out, 'OK, is this idea resonating?' But not really having a formalized process for testing how players experience the game and whether you're getting the clarity you want."

"One of the big reasons I came to work here was this department," says lead designer Scott Warner. "I'm not joking when I say, top three reasons I came to 343, one was user research.

"A lot of people would say, 'Well we'll watch people play and see how things go,' but there [are] ways of doing this [that work better than others]. And Microsoft has a really long history of doing this kind of testing, not only through games but also obviously through applications and [everything else they produce]."