This is a text transcription of the slides from the "Windows: a software engineering odyssey" talk given on Microsoft culture by Mark Lucovsky in 2000. This is hosted here because I wanted to link to the slides, but the only formats available online were powerpoint and slide-per-page HTML where each page is basically a screenshot of a powerpoint slide. If you're looking for something on current Microsoft culture, try these links.

Agenda

History of NT

Design Goals/Culture

NT 3.1 vs. Win2k

The next 10 years

NT timeline: first 10 years

2/89: Coding begins

7/93: NT 3.1 ships

9/94: NT 3.5 ships

5/95: NT 3.51 ships

7/96: NT 4.0 ships

12/99: NT 5.0 a.k.a. Windows 2000 ships

Unix timeline: first 20 years

69: coding begins

71: first edition -- PDP 11/20

73: fourth edition -- rewritten in C

75: fifth edition -- leaves Bell Labs, basis for BSD 1.x

79 -- one of the best

82 System III

84 4.2 BSD

89 SVR4 unification of Xenix, BSD, System V NT development begins



History of NT

Team forms 11/89

Six guys from DEC

One guy from MS

Built from the ground up Advanced PC OS Designed for desktop & server Secure, scalable, SMP design All new code

Schedule: 18 months (only missed our date by 3 years)

History of NT, cont.

Initial effort targeted at Intel i860 code-named N10, hence the name NT which doubled as N-Ten and New Technology

Most dev done on i860 simulator running OS/2 1.2

Microsoft built a single board i860 computer code-named Dazzle, including the supporting chipset; ran full kernel, memory management, etc. on the machine

Compiler came from Metaware with weekly UUCP updates sent to my Sun-4/200

MS wrote a PE/Coff linker and a graphical cross debugger

Design longevity

OS code has a long lifetime

You have to base your OS on solid design principles

You have to set goals; not everything can be at the top of the list

You have to design for evolution in hardware, usage patterns, etc.

Only way to succeed is to base your design on a solid architectural foundation

Development environments never get enough attention

Goal setting

First job was to establish high level goals Portability: ability to target more than one processor, avoid assembler, abstract away machine dependencies. Purposely started the i386 port very late to avoid falling into a typical Microsoft x86 centric design Reliability: nothing should be able to crash the OS. Anything that crashes the OS is a bug. Very radical thinking inside MS considering Win16 was co-operative multi-tasking in a single address space, and OS/2 had similar attributes with respect to memory isolation Extensibility: ability to extend OS over time Compatability: with DOS, OS/2, POSIX, or other popular runtimes; this is the foundation work that allowed us to invent windows two years into NT OS/2 development performance: all of the above are more important than raw speed!



NS OS/2 design workbook

Design of executive captured in functional specs

Written by engineers, for engineers

Every functional interface was defined and reviewed

Small teams can do this efficiently Making this process scale is an almost impossible challenge Senior developers are inundated with spec reviews and the value of their feedback becomes meaningless You have to spread review duties broadly and everyone must share the culture



Developing a culture

To scale a dev team, you need to establish a culture Common way of evaluating designs, making tradeoffs, etc. Common way of developing code and reacting to problems (build breaks, critical bugs, etc.) Common way of establishing ownership of problems

Goal setting can be the foundation for the culture

Keeping culture alive as a team grows is a huge challenge

The NT culture

Portability, reliability, security, and extensibility ingrained as the teams top priority Every decision was made in the context of these design goals

Everyone owns all the code, so whenever something is busted anyone has a right and a duty to fix it Works in small groups (< 150 people) where people cover for each other Fails miserably in large groups

Sloppiness is not tolerated Great idea, but very difficult to nurture as group grows Abuse and intimidation gets way out of control; can't keep calling people stupid and except them to listen

A successful culture has to accept that mistakes will happen

NT 3.1 vs. Windows 2000

Dev teams

Source control

Process management

Serialized development

Defects

Development team

NT 3.1 Starts small (6), slowly grows to 200 people NT culture was commonly understood by all

Windows 2000 Mass assimilation of other teams into the NT team NT 4.0 had 800 developers, Windows 2000 had 1400 Original NT culture practiced by the old timers in the group, but keeping the culture alive was difficult due to growth, physical seperation, etc. Diluted culture leads to conflict Accountability: I don't "own" the code that is busted, see Mark! reliability vs. new features 64-bit portability vs. new features



Source control system (NT 3.1)

Internally developed, maintained by a non-NT tools team No branch capability, but not needed for small team

10-12 well isolated source "projects", 6M LOC

Informal project seperation worked well minimal obscure source level dependencies

Small hard drive could easily hold entire source tree

Developer could easily stay in sync with changes made to the system

Source control system (Windows 2000)

Windows team takes ownership of source control system, which is on life support

Branch capability sorely needed, tree copies used as substitutes, so merging is a nightmare

180 source "projects", 29M LOC

No project seperation, reaching "up and over" was very common as developers tried to minimize what they had to carry on their machines to get their jobs done

Full source base required about 50Gb of disk space

To keep a machine in sync was a huge chore (1 week to set up, 2 hours per day to sync)

Process management (NT 3.1)

Safe sync period in effect for 4 hours each day; all other times, the rule is check-in when ready

Build lab syncs suring morning safe sync period, which starts a complete build Build breaks are corrected manually during the build process (1-2 breaks were normal)

Complete build time is 5 hours on 486/50

Build is boot tested with some very minimal testing before rrelease to stress testing Defects corrected with incremental build fixed

4pm, stress testing on ~100 machines begins

Process management (Windows 2000)

Developers not allowed to change source tree without explict, email/written permission Build lab manually approves each check-in using a combination of email, web, and a bug tracking database

Build lab approves about 100 changes each day and manually issues the appropriate sync and build commands Build breaks are corrected manually; when they occur, all further build processing is halted A developer that mistypes a build instruction can stop the build lab, which stops over 5000 people

Complete build time is 8 hours on 4-way PIII Xeon 550 with 50Gb disk and 512k cache

Build is boot tested and assuming we get a boot, extensive baseline testing begins Testing is a mostly manual, semi-automated process Defects occuring in the boot or test phase must be corrected before the build is "released" for stress testing

4pm, stress testong on ~1000 machines begins

Team size

Product Devs Testers NT 3.1 200 140 NT 3.5 300 230 NT 3.51 450 325 NT 4.0 800 700 Win2k 1400 1700

Serialized Development

The model from NT 3.1 to 2000

All developers on team check in to a single main line branch

Master build lab syncs to main branch and builds releases from that branch

Checked in defect affects everyone waiting for results

Defect rates and serialization

Compile time or run time bugs that occur in a dev's office only affect that dev

Once a defect is checked in, the number of people affected by the defect increases

Best devs are going to check in a runetime or compile time mistake at least twice a year

Best devs will be able to code with a checked in compile time or run time break very quicky (20 minutes end-to-end)

As the code base gets larger, and as the team gets larger, these numbers typically double

Defect rates data

With serialized development Good, small, teams operate efficiently Even the absolute best large teams are always broken and always serialized





Product Team # Defects/dev-yr Fix time / defect Defects / day Total fix time NT 3.1 200 2 20m 1 20m NT 3.5 300 2 25m 1.6 41m NT 3.51 450 2 30m 2.5 1.2h NT 4.0 800 3 35m 6.6 3.8h Win2k 1400 4 40m 15.3 10.2h

Dev environment summary

NT 3.1 Fast and loose; lots of fun & energy Few barriers to getting work done Defects serialized as parts of the process, but didn't stop the whole machine; minimal downtime

Windows 2000 Source control system bursting at the seams Excessive process management serialized the entire dev process; 1 defect stops 1400 devs, 5000 team members Resource required to build a complete instance of NT were excessive, giving few developers a way to be sucessful



Focused fixes

Source control

Source code restructuring

Make the large team work like a set of small teams Windows is already organized into reasonable sized dev teams Goal is to allow these teams to work as a team when contributing source code changes rather than as a group of individuals that happen to work for the same VP Parallel development, team level independence

Automated builds

Source control system

New system identified 3/99 (SourceDepot)

Native branch support

Scalable high speed client-server architecture

New machine setup 3 hours vs. 1 week

Normal sync 5 minutes vs. 2 hours

Transition to SourceDepot done on live Win2k code base

Hand built SLM -> SourceDepot migration system allowed us to keep in sync with the old system while transitioning to SourceDepot without changing the code layout.

Source code restructuring

16 depots for covering each major area of source code

Organization is focused on: Minimizing cross project dependencies to reduce defect rate Sizing projects to compile in a reasonable about of time To build a project, all you need is the code for that project and that public/root project Cross project sharing is explicit



New tree layout

The new tree layout features Root project houses public 15 additional projects hang off the root No nested projects All projects build independently Cross project dependencies resolved via public, public/internal usnig checked in interfaces



Team level independence

Each team determines its own check-in policy, enable rapid, frequent check ins

Teams are isolated from mistakes by other teams When errors occur, only the tema causing the error is affected A build, boot, or test break only affects a small subset of the product group

Each team has their own view of the source tree, their own mini build lab, and builds and entire installable build

Any developer with adequate resources can easily duplicate a mini build lab Build and release a completely installable Windows system

Teams integrate their changes into the "main" trunk one at a time, so there is a high degree of accountability when something goes wrong in "main"

Build breaks will happen, but they are easily localized to the branch level, not the main product codeline

Teams are isolated from mistakes made by other teams When errors occur, they affect smaller teams A build, boot, or test break only affects a small subset of the Windows development team

Each team has their own view of the source tree and their own mini buikld lab Each team's lab is enlisted in all projects and builds all projects Each team needs resources able to build an NT system

Each team's build lab builds, tests, and mini-bvt's a complete standalone system

Automated builds

Build lab runs 100% hands off

10am and 10pm full sync and full build Build failures are auto detected and mailed to the team Sucessful builds are automatically released with automatic notification to the team

Each VBL can build: 4 platforms (x86 fre/chk, ia64 fre/chk) = 8 builkds/day, 56/week No manual steps at all 7 VBLs in Win2k group Majority of builds work, but failures when they occur are isolated to a single team



Productivity gains