Philip Greenspun founded ArsDigita Corporation and was its CEO from inception until it reached $20 million/year in revenue. Currently he flies airplanes and helicopters and teaches electrical engineering and computer science at the Massachusetts Institute of Technology.

Software engineering is different.

Software engineering is different because only the best people significantly contribute to achievement. Traditional management texts assume a distribution of talent among the workers. Each worker is contributing something useful and the challenge is to get each one to perform at his or her maximum potential. In the same factory, the best worker may produce two or three times as much as the average, but all the workers are contributing. In software engineering a good programmer is at least 10 times more productive than an average programmer (Brooks 1995). If a product is being developed rapidly, the average programmers will consume nearly their entire work day just in reading and understanding the new code generated by the good programmers. Thus the challenges of a software engineering manager first and foremost are (1) creating a work environment where good programmers will be satisfied enough to stay, and (2) creating a system via which average programmers can become good. In an ideal software engineering organization, there are still some average-quality people but these should be viewed as being apprenticed to the best people and being taught as fast as possible.

Software engineering is different because people at all levels of the organization perceive themselves to be equally intelligent. Consensus-style management can perhaps work when there is a gradient of perceived ability. Given enough time, the less able workers will follow the lead of the more able workers. One of the paradoxes of software engineering is that people with bad ideas and low productivity often think of themselves as supremely capable. They are the last people whom one can expect to fall in line with a good strategy developed by someone else. As for the good programmers who are in fact supremely capable, there is no reason to expect consensus to form among them. Each programmer thinks his or her idea about what to build and how to build it is the best. (See Kruger and Dunning's "Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments" for more on this topic.)

Software engineering is different because a leaf-node worker is more expert than any manager, even when the manager is a great engineer, in at least the small portion of the system that the leaf-node worker has personally built. This makes it difficult for a manager to engage in a technical argument with a worker. It becomes nearly impossible when the manager's technical skills are weak. The worker can spin castles of complexity in the air and come up with impressive-to-the-MBA excuses for why it has to be done a certain way or on a certain schedule.

Software engineering is different because the organization can't afford to lose the individual productivity of the best people by pushing them into management. A truly great programmer may generate 10 times as much business value as a merely good programmer. Can the organization afford to take someone who can do the work of 100 average programmers and push him or her into a pure management role? Probably not. Can the organization afford to put people with weak technical skills into management roles? Probably not. Once you give Joe MBA a title and ask him to coordinate eventually he will be making decisions that have engineering implications. Thus many of the best programmers are eventually forced at least to assume project leadership and mentoring responsibilites. Since they are still expected to produce designs, software, documentation, and journal articles, the danger is that the new manager will become glued to his or her screen and never look up to see how the project team is doing.

Software engineering is different because measurement is notoriously difficult. The world is full of products that failed due to overly complex and tasteless designs. Yet all of these designs were considered tasteful by their architects. Systems that experts evaluated and found wanting, such as the Unix operating system (1970), eventually proved to have great utility. It is a bit easier to count up the lines of code per day produced by a programmer but if the project was not very tightly specified originally, how do you know whether or not these lines of code are useful?

At this point a skeptical reader might be thinking that, while software engineering is different from line production work or any other endeavor with a manufacturing division of labor, there are similarities with research and development, management consulting, and financial analysis. This is certainly true but there aren't too many interesting books on how to reliably produce results in these fields (one is referenced in the "More" section below).

Ideas to Steal

people don't do what they are told

all performers get the right consequences every day

small, immediate, certain consequences are better than large future uncertain ones

positive reinforcement is more effective than negative reinforcement

ownership leads to high productivity

people don't do what they are told

Bringing out the Best in People

A corollary to this principle is that people do what you reward them to do, not what you hope they will do. Often, when you look at what is truly rewarded in an organization, you find it is different than what you think is rewarded. Do the managers have an engineering background? If not, they'll probably be unable to perceive when a programmer is accomplishing nothing. So the programmer who does nothing gets a paycheck at the end of the month. Having thus been rewarded for doing nothing, the programmer tries it again the next month...

all performers get the right consequences every day

small, immediate, certain consequences are better than large, future, uncertain ones

positive reinforcement is more effective than negative reinforcement

We thus proved the management adage that a deadline just gives someone an excuse to procrastinate and do nothing until the very end.

Graduate school at MIT is different. We want the students to do research, write up their results, publish them in journals, and graduate with a reasonably interesting PhD thesis. If a student finishes some research, the most effective faculty advisors immediately provide positive reinforcement by paying attention, helping design the next experiment, helping to draft a paper outline. If the student finishes a write-up, he or she is positively reinforced by being sent to a conference to present it. If the student finishes a PhD thesis, he or she is positively reinforced by being given a 3-7X pay raise.

The lesson from MIT? Negative reinforcement can work if the organization is extremely tightly managed, if the consequences are small and immediate (usually a problem set is due every week and only represents a part of the final grade), and if the goal is to make sure that everyone comes up to a reasonable level. However, the worldwide fame of MIT rests on research achievements by graduate students. This innovation is mostly supported by positive reinforcement.

ownership leads to high productivity

As an example, in the early days of ArsDigita we had only a handful of customers: America Online, Environmental Defense Fund, Hewlett-Packard, Levi Strauss, Oracle Corporation, and Siemens. We had only a handful of programmers as well and hence the easiest way to divide the work was to give a programmer total responsibility for one project. The programmer owned that customer. If the project went well and the customer wrote us a big check, we gave nearly all of the money directly to the programmer. If any project had gone poorly and we'd been fired by a customer, we would not have had to think very hard to figure out who was responsible (fortunately this never happened while I was running the company!). People worked insanely hard to make their projects successful and their clients happy. More importantly, the programmer who did an entire project by him or herself learned enough to train new people, lead a larger project, etc.

After we grew beyond the 40-person mark, pressures to dilute the ownership aspects of our organization grew. We wanted to grow rapidly--nobody wants to buy enterprise software from a small company, even if the software happens to be open source. As our reputation grew, customers came to us with larger projects. We believed that many of our developers were too junior to handle complete responsibility for these large projects. Our costs went up because we had to coordinate the diffused responsibility. In the summer of 2000, when we had 200 or so employees, one of our clients was unhappy. It took a week just to arrange a meeting among the five managers who bore collective responsibility for the project! Meanwhile, individual productivity fell. It was taking more programmer-months and more calendar months to get things done. On weekend mornings you could walk naked through an entire floor of our headquarters building without fear of embarrassment.

(At the time of this writing, there is a proposal on the table to consolidate some of the separate management pyramids, thus taking us back closer to our original structure.)

Building and keeping a good software engineering team

What attracts good programmers? Traditionally the best programmers seek the most challenging problems. They want to work in an organization that is trying to build something important. Programmers have huge and fragile egos. If they are somehow assigned to a trivial problem and that is their only possible task, they may spend six months coming up with a bewildering architecture more complex than the Windows 2000 operating system, merely so that they can show their friends and colleagues what a tough nut they are trying to crack. Another source of ego-gratification for programmers is to have other programmers admiring their work. Open-source software projects thus have a big recruiting advantage over closed-source software companies.

What kind of working environment is necessary for programmer satisfaction? Good programmers want to achieve and therefore removing barriers to achievement is the most important step that one can take in creating an effective working environment. Programmers dread elaborate process, endless meetings, and layers of marketing approval before a product can be shipped. Ideally it would be possible to conceive a product on Friday evening, set up the development environment Friday night, write code on Saturday and Sunday, test on Sunday night, and ship on Monday morning. Maintaining this kind of freedom is a serious challenge as a company grows and its products become more complex. Successful companies such as Oracle Corporation burden their marketing departments with overlapping products rather than stifle programmer initiative. For example, during most of the late 1990s there were at least three different Web servers that you could buy from Oracle, each one backed up by a document explaining why it was the one true path toward database-backed Web site glory.

A good physical working environment is essential. Great programmers get a lot of positive reinforcement from their work itself. They write some code and immediately can see it dance. That keeps them at work for hours that, while they would not impress a taxi driver in Singapore or a factory worker in Guangzhou, will surprise many American business people. When we hired an architect to lay out the interior of ArsDigita's first building in Cambridge he surveyed the programmers and came back shaking his head: "I've never seen any group of people who spend so many hours continuously sitting at their desks."

From a business point of view, long hours by programmers are a key to profitability. Suppose that a programmer needs to spend 25 hours per week keeping current with new technology, getting coordinated with other programmers, contributing to documentation and thought leadership pieces, and comprehending the structures of the systems being extended. Under this assumption, a programmer who works 55 hours per week will produce twice as much code as one who works 40 hours per week. In The Mythical Man-Month , the only great book ever written on software engineering, Fred Brooks concludes that no software product should be designed by more than two people. He argues that a program designed by more than two people might be more complete but it will never be easy to understand because it will not be as consistent as something designed by fewer people. This means that if you want to follow the best practices of the industry in terms of design and architecture, the only way to improve speed to market is to have the same people working longer hours. Finally there is the common sense notion that the smaller the team the less management overhead. A product is going to get out the door much faster if it is built by 4 people working 70-hour weeks (180 productive programmer-hours per week, after subtracting for 25 hours of coordination and structure comprehension time) than if by 12 people working 40-hour weeks (the same net of 180 hours per week). The 12-person team will inevitably require additional managers and all-day meetings to stay coordinated.

Your business success will depend on the extent to which programmers essentially live at your office. For this to be a common choice, your office had better be nicer than the average programmer's home. There are two ways to achieve this result. One is to hire programmers who live in extremely shabby apartments. The other is to create a nice office. Microsoft understands this. In the early 1990s they did radio spots with John Cleese as a spokesman. One of the main points of the ad was to ridicule the cheap open-plan offices in which programmers were traditionally housed and promote the fact that at Microsoft each developer gets a plush personal office.

How can an office be nicer than one's home? Let's consider the following dimensions:

social

physical comfort

aesthetic

entertainment

attractive

Social

A social place will never be friendly if it is trapped behind a high wall of security. It ought to be very easy for a programmer to invite a friend over. If programmers are comfortable meeting their friends at the office it greatly increases the likelihood of friends recruiting friends.

An open office plan contributes to making the work environment stronger on the social dimension.

Physical Comfort

One horrible mistake that we made was letting our architect design the workstations. Each programmer was given a 6'x2' desk, 12 square feet total. Two 21" monitors took up so much depth that there wasn't even room for a keyboard. Immediately we had to toss our monitors and get flat panels (cost about $400,000 extra). IBM had a better architect for its Santa Theresa facility: Gerald McCue. He found that each worker needed 100 square feet of dedicated space and 30 square feet of work surface. McCue also found that programmers needed noise isolation from enclosed offices or high partitions but personally we think this rule is worth breaking in a dotcom world where a team has to work fast and in sync. Better to manage noise by spreading desks apart a bit so that there are fewer programmers in a given area and therefore fewer conversations, fewer telephones, and more opportunities for sound to be absorbed before reaching someone's ear.

Aesthetic

Entertainment

The average home cannot accomodate a pinball machine. An office can. The average home can have video games, which are very popular with young programmers, but not people with whom to play. The average home cannot have a grand piano but almost any office can.

Attractive

dog-friendly policy

grand piano

climbing wall

indoor garden

aquarium

koi pond

exercise room with fancy machines

pinball machine

Change of Venue

What does it take to let the entire team pick up and work somewhere else for awhile? A beach house or a ski house within a two-hour drive of their main office. It is kind of expensive for an individual to rent a vacation house year-around, equip it with a DSL line or cable modem, and pack it with enough desks and computers for a team to work. But if you've got a group of 30 programmers and get a house large enough for 6 or so to sleep and work, the cost is manageable. In the winter, a programming team can disappear for a week, ski every morning and work all afternoon and evening. In the summer, a team can spend a week looking out at the ocean... while typing most of the time. It costs more than not having the beach house but a lot less than having employees go off on their own to have fun every weekend and not work.

Turning average programmers into good programmers

How does one create a good programmer? Raw materials are important. You want someone with a strong computer science education, a high IQ, and an ability to communicate effectively in oral presentations and in writing. But without the right experience, such a person will never be more than an average quality programmer.

These principles are important in building up someone's programming skills:

A person won't become proficient at something until he or she has done it many times. In other words., if you want someone to be really good at building a software system, he or she will have to have built 10 or more systems of that type. A person won't retain proficiency at a task unless he or she has at one time learned to perform that task very rapidly. Learning research demonstrates that the skills of people who become accurate but not fast deteriorate much sooner than the skills of people who become both accurate and fast. Technology shifts force a programmer to go through bursts of learning every year or two.

Learning to design and build software systems requires that the programmer design and build software systems. These can be smaller subprojects for internal or external customers, standalone software system for non-profit organizations, or demonstration systems to be written up and distributed to other programmers. A particularly effective option that is only available in the Web world is to build and launch a free public service. See http://remindme.arsdigita.com and http://towzone.arsdigita.com for examples of one-evening training projects.

Whatever the training task, the pace must be ruthlessly brisk. The learner should be expected to build at the same pace as an experienced developer. The difference between the learner and the wizard is that you expect the learner to make a lot of mistakes. The system as built may be awkward or not handle error cases properly. That's okay. Training research shows that if you get speed now you can get quality later. But if you don't get speed you will never get quality in the long run. We practice this technique in 6.916, Software Engineering for Web Applications, our course at MIT. Each student builds five database-backed Web applications during the 13-week semester. The first few that they build, during the course of the problem sets, are not necessarily elegant or optimal, but by the end of the semester they've become remarkably proficient, especially when you consider that each student is taking three or four other classes.

If you see one of your best people walking out the door at 6:00 pm, try to think why you haven't challenged that person with an interesting project. If you see one of your average programmers walking out the door at 6:00 pm, recognize that this person is not developing into a good programmer. An average programmer's productivity will never be significant in a group of good programmers. If you care about profits, you must either come up with a new training program for the person or figure out the best way to terminate his or her employment with your organization.

Still not convinced? Take a look at the Japanese "code factory" circa 1990. These precisely organized large organizations where each person had his role, however small, were supposed to overtake the American approach where small teams of craftsmen worked in a comparatively disorganized manner. The factory approach sometimes produces acceptable corporate IT solutions but for innovation and successful product development, the craft approach has been overwhelmingly vindicated.

Turning good programmers into good managers

The classic problem situation at ArsDigita is a manager getting lost in his or her own work and failing to review a subordinate's efforts for two or three months. When the review occurs, inevitably the subordinate has either been working on the wrong thing in the wrong way or hasn't been sufficiently productive. At this point the manager is really angry. Three months of calendar time and money have been wasted. But should the manager be angry with the employee? If the manager had reviewed the subordinate every week, the company would never have been at risk of losing more than one week of time and money.

Our solution is to decouple responsibility for review from responsibility for scheduling review. We use administrative assistants to ensure that each manager is scheduled to look at every subordinate's work at least once per week, more frequently in the case of junior employees. It has proven remarkably more effective when a neutral third-party is responsible for scheduling than when people with incentives to shirk are responsible for scheduling.

Management by Consensus Considered Harmful

Software engineering companies will tend to have a fairly flat distribution of intelligence. The 22-year-old Stanford CS punk that was just hired will be just as smart as the 30-year-old lead engineer who will be just as smart as the 40-year-old CEO. Within a company's technical team, the raw IQ differences are even smaller. If each member of the team were playing the Bach Partitas and Sonatas for Solo Violin, the wrong notes, shaky intonation, and bad phrasing would make it pretty obvious to the novices that they needed to take advice from the experts. But because software quality is tough to measure and software quantity is seldom measured, the novices in a software engineering group are able to think of themselves as experts.

What would be wrong with a completely egalitarian software engineering group? Maybe the entire team really is at the same level of ability. And suppose that somehow the challenge of getting everyone to attack the same problem had been surmounted. Remember what Fred Brooks said in The Mythical Man-Month : high quality systems must be architected by no more than two people.

Getting design input from leaf-node engineers is important for having a good product design. But at the end of the day nobody should be confused as to who is providing leadership. There is an irreducible amount of Engineer A imposing his or her design on Engineer B. This can lead to some harsh-sounding words and bruised feelings. Microsoft is not the self-esteem company, at least if you believe Playboy magazine's interview with Bill Gates: "We hear you're brusque at times, that you won't hesitate to tell someone their idea is the stupidest thing you've ever heard. It's been called management by embarrassment challenging employees and even leaving some in tears." Truly elite organizations can be far worse than Microsoft. Ask a group of surgical interns and residents how much respect they get from the surgeons. Go into a world-class biology department and ask the grad students and post-docs about their treatment at the hands of the professors.

Wherever You Go, There You Are

Consider the average working programmer's life:

surfing USENET

surfing Slashdot

reading docs

questioning colleagues

writing specs and designs

writing docs

writing code

testing code

fixing bugs

filing bug reports on others' code

attending meetings

helping sales and marketing staff

(For comparison to the grad student life, see http://philip.greenspun.com/humor/graduate-student-emotion-check-list.)

Characterizing this person's productivity is going to require more than one number. But if we don't do it, days or weeks could slip by without the programmer realizing that his or her achievement levels are plummeting. In a company with disorganized or technically clueless managers, the programmer's supervisory chain won't notice the lack of achievement either.

Production of documentation and code is generally measurable by reference to the company's version control system. Bugs filed and fixed are easily tallied by looking at the company's ticket/bug tracking system (see http://www.arsdigita.com/doc/ticket for a description of our favorite open-source ticket/bug tracker). The softer stuff can still be quantified but it will have to be done by humans filling out forms.

Ideally the programmer will get daily feedback, which is kept private unless the individual elects to publicize it. Performance in each sanctioned area of activity will be marked up and scored with a weight. The programmer can then see if his or her crude achievement level is going up or down.

Summary

Start by attracting a good core team, perhaps by setting up an organization that enables each engineer to excel along the axes defined in http://philip.greenspun.com/ancient-history/professionalism-for-software-engineers. Provide a productive working environment and a physical environment that is better than the average programmer's house. Provide daily positive reinforcement. Provide daily feedback showing the programmer more or less exactly what he or she has accomplished, plus a graph for the preceding few months showing the trend. Aim to install a feeling of ownership in each worker.

More