The fireman

"There's been days at Xbox where it's felt similar to those days back in the fire department. It's definitely a higher-stress job when things aren't going as well as you hope." — Derek Ingalls

Derek Ingalls is a quiet, reserved individual. He carries himself like a gunfighter, looking more hard laborer than hardware engineer. He has the hands of a man who's used them and the eyes of someone more used to looking for trouble around him than inside of a computer. When he walks into the room and sits down, you say to yourself, "This is a man who's seen things and survived." And that's no coincidence. Before he came to work at Microsoft in 1997, Ingalls was a fireman.

"I’m like, ‘Nobody’s going to die today. I know what that looks like. Calm down.’"

"When I was running Exchange internally," says Ingalls, "if Bill [Gate]'s Exchange server was down, there were people running up and down the halls like the building was on fire. I'm like, 'Nobody's going to die today. I know what that looks like. Calm down.'"

One of the hardest lessons the Xbox team learned in its first five years was the value of what's called "change management," the process of keeping track of changes. It sounds simple, but with such a large and growing service that impacted multiple moving parts at Microsoft and elsewhere, the Xbox team often found itself playing catch-up with its own activity and fighting problems it had caused itself.

When Microsoft needed someone to step into Xbox and start running it like a business in 2008, it tapped Ingalls, an expert in change management who was, at the time, running Exchange. Ingalls likens the job to making sure the toilets flush.

"As long as they flush, everybody’s happy, but nobody’s going to pat you on the back for it."

"As long as they flush, everybody's happy," Ingalls says, "but nobody's going to pat you on the back for it. But in the event that they don't, people make a lot of noise about it."

Ingalls's first week on the job as Xbox Live's operations general manager was right before the launch of Grand Theft Auto 4.

"I don't know if you remember," says Neustadter, "but the CEOs of Rockstar made a public comment, something along the lines of, 'We'll be so popular we're going to burn Xbox Live down.' So we were very determined that that wasn't going to happen."

With the chaos of Christmas the year before still fresh in its mind, the Operations team was talking about adding server capacity to accommodate GTA4, while the facilities team was talking about problems it was already having with overheating in the server farms. Ingalls, present for both conversations, realized that not only did Xbox Live have a problem, nobody else knew it.

"I'm watching this and trying to put the two conversations together. ... And I'm doing the math," Ingalls says. "'Wait a minute, if we're having power and space and cooling problems, and there's this big game coming, and the numbers you're talking about ... are we in trouble?'"

Ingalls was an unlikely Microsoft engineer. His twin brother Dustin already worked at the company while Ingalls was with the fire department. The two would have lunch and Ingalls would tell his brother about surviving a bad fire or watching a child die, and Dustin would tell him he needed to find another job. Eventually Ingalls took a break. Redmond FD has a long-established sabbatical policy. Firefighters are encouraged to take a two-year leave, after which they can return to their exact same job at their exact same salary, with no loss of benefits or loss of seniority. Most who take it come back. Ingalls took it and went to Microsoft.

Ingalls believes his fireman training prepared him well for running critical services at Microsoft, especially in knowing how to prepare for accidents — and prevent them.

"It's just unacceptable to ever be surprised," says Ingalls. "It's great for the business to say, 'Hey, wow, business is booming and we didn't realize how great it was going to be!' But when you get down to it, we know how many consoles are going out the door. We know how many Live subscriptions we're selling. We know how many game discs we're printing. We know the code that we wrote. We know how it runs in test and in stress. I think there's no excuse, ever, to be surprised."

Overseeing Xbox Live Operations, Ingalls focused on prevention, building up processes and training XOC staff to handle crises efficiently and — most importantly — consistently. And he hired a team of engineers to work separately from Xbox Live Operations to ensure that the training held — and remained fresh.

The one problem with prevention is that it can breed inattention. If a fire never starts, for example, the people who put out fires might forget how. That's why firefighters train by setting fire to abandoned houses. For Xbox Live, Ingalls needed a way to burn down the house without waiting for someone to do it for him. So he started doing it himself.

Ingalls oversees a team at Xbox called "Team Chaos." Their job: Inject faults into Xbox Live so that the XOC team can find and repair them. Ingalls believes this keeps the XOC team in shape and ready for when something happens that it didn't cause itself.

"We just don't tell anybody, and we let 'em go," Ingalls says. "We grade ourselves on how quickly we identify and shut down what's going on."

During his first week on the Xbox team, while putting the pieces together regarding the impending doom of the GTA4 launch, Ingalls also quickly realized there was no written process for anything at Xbox Live. For any given event, Operations Center employees would line up behind Neustadter, a solution would pour out of his brain and then people would do what he told them. It worked, to a point, but for a large and growing service like Xbox Live to have its entire operation residing in one person was potentially catastrophic.

Ingalls moved quickly, established procedures, separated existing processes from their habitat in Neustadter's brain and began a methodical, but rapid, process of splitting Xbox Live into multiple server farms with its now legendary geographical redundancy. The result: a dramatic increase in uptime and a reduction in overall outages. Xbox Live would now be ready for whatever Microsoft had in store for it. Perhaps surprisingly, the person most receptive to the changes was Neustadter himself.

"[Ingalls' arrival] was great for me, because Derek is great at all the things I don't like to do," Neustadter says. "Deep down, I am a technical person, and I'm happiest on a whiteboard designing stuff and trying to solve cool technical problems to let games do new stuff. ... That's what really jazzes me about the job."

In 2008, the Operations team was around 50 people, and managing these people was Neustadter's full-time job. No time for doing technical work, no time to go hands-on with games. It was sucking the enjoyment out of his day and he was happy to let go of running the XOC and get back to working with the creators on making games. Since Ingalls' arrival, Neustadter has spent the majority of his time working with the teams behind Halo, Call of Duty, Gears of War and others to make sure their games work as well as they can on Xbox Live and that Live has the tools it needs to support them. He's gone back to being a game engineer, and he couldn't be happier.

"That's something I hadn't gotten to do a lot of since probably Halo 2," Neustadter says, "because I was busy doing these other things. ... That's part of the reason why I'm still here after all this time. I get to work on really cool problems every day. ... I spend my day sitting here solving problems for Call of Duty, or right now Titanfall, Sunset Overdrive, Destiny, whatever's coming next. Then I get to go home and play them as a customer."

For Neustadter, the evolution of the Xbox Live team just makes sense.

"We went from being a startup to being a big business," he says. "Used to be we could kind of get away with doing whatever the heck we wanted, because we were this little skunkworks project. We now get mentioned during SEC quarterly conference calls about the performance of our business. ... We have had to grow up."

In May of 2008, when Grand Theft Auto 4 launched into the world, it broke all standing records for first-day game sales, moving more than 3.6 million copies in its first 24 hours, selling over a million more copies its first day than Call of Duty 2 sold in its entire lifetime.

Xbox Live did not crash.

Time to penis "Hacking on Call of Duty started to get totally out of control around [2008], with people hacking the game and custom lobbies and all that kind of stuff," says ATG Account Manager Sam Charchian. "We built this really tight integration between [Infinity Ward's] anti-hack team and ours inside, because Microsoft has their own, what they call the Xbox Live Enforcement team. We had their anti-hack/enforcement team come spend a day with ours and talk about how they could work together to hopefully defeat these people. "One of the funny things that came out of it ... we're meeting with the Xbox Live enforcement team, and [there was] a woman on the team who's in her early 20s. She was explaining that she spends a good part of her week going into Uno games with the camera on and basically just waiting for dudes who will see a woman playing Uno with her camera on and expose themselves. "I was like ... 'Really? How often do you actually have dudes, you know, whip it out?' And she said, 'Eh, my time to penis is about 14 minutes on the average.' I mean, she would sit on there and just ban these guys when she ran into them, but apparently it was a serious problem. Part of what makes it extra sad is that it's Uno, which you think would be a family-friendly game for people to play."

New Xbox

"A lot of folks don't like change. That's when the people who wouldn't normally be talking come out and say, 'Oh, I hate this.' They just don't understand what's going on. When you look back, it makes perfect sense, but at that moment, not everybody really gets it." — Larry Hryb

The sweeping changes to Xbox and Xbox Live were not only promoted by the growth of the service, but also by movement at some of the highest levels at Microsoft. By the end of 2007, Allard, the visionary leader whose dream of a "Snow Crash box" to unite the world had spawned Xbox and Xbox Live in the first place, had moved on to oversee the production of Microsoft's next big thing: the ill-fated music player Zune.

Microsoft's Marc Whitten had stepped in to replace Allard, with Allard's protege (and sometime critic), Boyd Multerer, filling in on the idea side and Executive Producer Ben Kilgore managing production. Together the three men would oversee the next phase of Xbox Live and eventually move the entire Xbox team away from its offsite headquarters at Millennium E back to the Microsoft mothership in Redmond.

"The fun period for Xbox was definitely over at that point. That’s what everyone thought."

"The fun period for Xbox was definitely over at that point," says Ingalls. "That's what everyone thought. For the first couple weeks, there was an amazing amount of trepidation. ... Teams get really passionate about the work that they do and the people they work with. All the big leaders have a posse, and so everybody's always like, 'What is that going to mean? What is that like?'"

Xbox had, at one time, been an experiment. No one, not even the Xbox team itself, knew for sure that it would succeed. The structure required the team to create something completely new in a completely different category because Microsoft was a culture that could encourage people to break barriers and not be afraid to fall on their faces. In other words, a culture completely unlike the entrenched, politics-heavy corporate culture at Microsoft. Now that Xbox had become a success, it was time to rein that counterculture in.

"I had an awesome time at Microsoft," says Charter, "but as the Xbox division got bigger and more successful, it became far more buttoned up and corporate than it was when we were over on the redheaded stepchild campus all by ourselves, just spending Bill's money and doing stuff that everyone would live to regret. The more corporate it became, the more it was like, 'I don't really fit in here.'"

Externally, the Xbox team was pushing toward what would be the first major update to Xbox Live since the launch of the Xbox 360. Dubbed "The New Xbox Experience," or NXE, the update would dramatically alter almost every aspect of how users interfaced with the service — and how they would use the console itself. It would be a major test of both Whitten's new priorities and the concept of a long-lived console.

At the time, few people outside of Microsoft knew just how long the Xbox 360 was intended to last. Game consoles typically had a four- or five-year lifespan. In 2008, Xbox 360 was in its third year of life, at which point most observers would have expected to begin hearing about a new console. What they heard instead was about an an update to Xbox Live.

"There was this process internally at Microsoft called strat review," says Whitten, "where each year, every part of the company would come in and pitch their strategy. As you can imagine, it's lots of PowerPoints and lots of Excel spreadsheets and all this stuff. I had been thinking a lot about this idea of, 'Well, what does it mean to really open up Xbox Live and to make it more visual, to help people discover content faster, to add new functionality?' And so instead of a PowerPoint and an Excel spreadsheet, I had the team build a video, a sort of two-minute video of what Xbox Live could look like. I also wrote the review, a magazine review article. ... I would show it to people and they'd say, 'Yeah, but it's not a new console.' And I'd say, 'Exactly.'"

The core of the NXE was a change in what's called the UI, or user interface. The original Xbox Live used a UI called Blades. Each type of activity on Xbox Live had its own blade, and users switched between blades using buttons on the controller.

"At the time we thought [Blades] was the most amazing UI in the world," says Kilgore. "We used to just sit there going back and forth between the blades and high-fiving each other. ... It's crazy what you do at 2 a.m. after working 100 straight hours. Sometimes you might lose a little perspective."

As cool as Blades may have been, Kilgore and others at Microsoft worried that the UI didn't showcase all that Live could do. And the more Live was able to do, and the more users were able to do with it, the less powerful Blades proved to be.

"You didn't have this sense [with Blades] that you were part of something bigger," Kilgore says. "What we wanted to do with NXE was to go and create the sense that you're a part of something bigger. Show you what people are doing. Show you all the different ways that you could be online and participating. ... We want to be Disneyland, not Coney Island, [and] NXE was the map of the park."

Microsoft had been sending a survey to users logging out of Xbox Live. One question: Why are you leaving? The answer, more often than not, was: "There's nothing to do."

"Then we'd say, 'What if you could do this?'" Whitten says. "They'd say, 'Well, that would be cool.' And of course they could do that on Xbox Live; they just didn't know it."

The NXE would change all of that. It would introduce a more streamlined interface, similar to the Windows OS of the time. It would bring some of Live's deeper functions, like the marketplace and messaging, to the forefront, showing users a hint of everything Live had to offer whenever they turned on their Xbox.

Most importantly, NXE would help Microsoft realize the possibilities of a console that could reinvent itself using the exact same hardware — and extend the lifecycle of the Xbox 360 for another unprecedented seven years. And it would introduce a video streaming service that would revolutionize the way most gamers used their Xbox consoles forever.