I recently moderated a webinar in which Sam Guckenheimer, a Microsoft product owner and group product planner, discussed their development journey to DevOps. It was a “lessons learned” presentation that any company looking to benefit from DevOps can use.

To sum it up: The first decade of doing agile development uncovered some key areas. First, your development team needs to be autonomous, yet aligned with the business in terms of goals and quality. Second, managing technical debt is important, as debt interferes with quality and poses a potential risk. Third, monitor the flow of customer value; that is, make sure you listen to the customer to minimize reworking code. Make sure the stakeholder is satisfied and has input, from idea to production.

(Related: How to plug into DevOps)

These are largely viewed today as essential to agile development. But Guckenheimer said the Microsoft developer division found four more “essentials” to moving from agile development into DevOps.

He started with the backlog, saying it is merely a set of hypotheses and beliefs rather than a form of requirements. The hypotheses need to be turned into experiments, which provide the data that can either substantiate or refute the hypotheses. “Learning leads to the next set of hypotheses. It’s not a random walk; it’s not a set guesswork; it’s a set of rigorous tests of beliefs using data to match against the beliefs,” he said.

Guckenheimer shared this story: “We had a signup experience for Visual Studio Team Services. We asked for your identity, we asked you to set up a project, and then we had all these other tiles that informed you about what else you could do. And we were not at all happy with the conversion rate through this screen. So we set up an experiment where we diverted some of the traffic to a new experience. We had a Web version and we have an experience where you sign up through the Visual Studio IDE. In the IDE, we had an instant 7x bump on the new experience for signups from the IDE. On the Web, we had about a 50% bump. But we discovered by the comparison to the IDE that…we were pumping a bunch more traffic to the Web, essentially routing a large volume of unqualified leads in. We want to turn off that extra traffic, and when we did, we get to a 3x improvement on the Web. And of course we went to the new experience based on those results.”

We could do that, he said, because we gather evidence in production. That’s the next essential habit. He said the Microsoft team gathered business-related data (conversions), but also gathered data for troubleshooting, as well as performance metrics, live-site issues that affect end users, and the number of times needed to communicate and mitigate issues. Precise alerting is key to fast issue detection on live sites, he noted, but it’s something people struggle with. You need a production-first mindset.

“We had these Tier 1 people in our service delivery team who were triaging lots of alerts,” said Guckenheimer. “They were looking at 40 or 50 alerts and trying to find which one is the real source of the problem. Based on that, they would contact the responsible individual, the developer on the feature crew who was responsible. He needs to be on the bridge in five minutes if it’s work hours, and 15 minutes if it’s outside of work hours.