Why I hate programming competitions

NOTE

I wrote this article a long time ago (in 2001, if memory serves), and frankly I'm not that interested in discussing it anymore. These days I'd rather write programs and teach programming languages than write opinion pieces. The article accurately reflects my opinions as of the time I wrote it, but may not accurately reflect my current opinions. I didn't post this article to reddit.com or to any other site. I have no interest in drumming up traffic to this page. I've received some surprisingly hostile emails from people who disagree with me. To those people: please realize that this is a rant, an opinion piece. If you don't agree with me, feel free to write your own opinion piece. But name-calling and trolling just makes you look bad and undercuts your own arguments. I won't respond to trolls, even if they also have some good points to make. If you can't talk civilly, talk to someone else. That said, I welcome constructive comments.

The Original Essay

As someone who loves programming and cares very deeply about teaching programming to undergraduates, I would like to express my opinions on why programming competitions are (for the most part) a bad thing, and on what I'd rather see in place of them that might serve the same end, but would more accurately reflect the bigger picture of what it means to be a good programmer. In the interest of full disclosure, I should add that I've never participated in a programming competition myself. If I did, I probably would not do very well, since I consider myself a rather slow programmer (more on this below). So maybe this is all just sour grapes -- you be the judge. I'll use the ACM (Association for Computing Machinery) programming contest as a typical example of the problems inherent in programming contests, because it's very well-known and extremely prestigious and competitive.

Almost all programming competitions have the same format. A number of relatively short problems are posed to the contestants. They have to come up with algorithms and working code that implements those algorithms. The test of the resulting code is whether it works at all, and if so, how fast it is. There is a time limit which is generally quite short. And that's all.

Here are my objections to (most) programming competitions as they currently exist:

Fast programming isn't the same as good programming

Gerry Sussman, the famous MIT computer scientist and co-author of the classic programming textbook Structure and Interpretation of Computer Programs ("SICP" for short), once told me that good programmers tend to fall into two categories:

People who can write mostly-correct code very fast. People who can write absolutely correct code more slowly.

Programming competitions only reward the first category of programmers, so at best, they are tapping into the skill sets of only some of the good programmers. Far be it for me to suggest that the ability to crank out mostly-working code at light-speed is a useless ability. In some jobs (e.g. writing ASP-type code for a web-based startup company) it may be the only thing that matters in the short term. But think about it: most code, if it works at all and performs some useful task, is going to be around for a long time. After a while, it is going to be modified, probably by many different people. The original programmer will probably have long since disappeared, and the original code will have mutated beyond recognizability. The advantage of having mostly-correct code that has been churned out in record time will diminish in comparison to the advantage of having code that is good from other standpoints (which I discuss below).

Also, mostly-correct isn't the same as completely correct. Bullet-proofing code may not be glamorous and may not be considered important in programming competitions, but it is very very important in the real world, where any unchecked assumptions about the input to a program are likely to be violated sooner or later.

In addition to all this, I have a visceral objection to the notion that fast mostly-correct programming is a good measure of how good a programmer is. I think of programming as an art form, and art is not measured in lines of code per hour. I have often said that programming competitions make as much sense as novel-writing competitions. [Monty Python have a very funny routine about a novel-writing competition which beautifully expresses the absurdity of such a concept.] I can just imagine a sculpture competition in Renaissance Italy: "OK, Michaelangelo, you have five hours to make a sculpture of this naked woman. If you're not done in five hours, you lose!" Some people would disagree with this characterization on the grounds that programming should be thought of as a science, not an art. Fair enough. "OK, Albert Einstein, you have to derive the equations for general relativity in the next five hours. If you take longer than that, you lose!" Maybe programming is more like engineering. "OK, Wright Brothers, you have to design and test a working airplane in five hours, or you lose!" It's just not a rational measure of programming ability.

But by far the biggest problem with programming competitions is that the criteria they use to judge programmer competence are so limited. This leads me to my next topic...

There is more to programming than algorithms and speed

Computer algorithms are a fascinating topic. A good algorithm is a thing of beauty, much like a work of art or a mathematical theorem. In addition, fast algorithms are not only beautiful but incredibly useful. However, there is a common misconception that algorithms and speed are the be-all and end-all of programming. This is simply not true and never has been true. In my experience, non-obvious algorithms tend to make up only a small portion of any large programming project (much less than 10%, and often less than 1% of the total code). Algorithms are important because that small amount of code often dominates the entire running time of the program, but there's much more to writing good code than just writing good algorithmic code.

But before we get to those other aspects, let's talk about what a good algorithm really is. Many programmers don't appreciate the distinction between asymptotic efficiency and micro-optimization. Many inexperienced programmers are obsessed with micro-optimization, because they want to be "37337 h4x0rs" (elite hackers), and they think this is how to do it. Such programmers invariably consider C to be the ultimate computer language, because it's fast and it provides more fine-grained bit-level control over the program's data structures and execution than most other languages. They live for clever hacks involving packing data structures into bit vectors, using left- and right-shifts for multiplication, maybe doing a little in-line assembly coding, etc. etc. There's nothing inherently wrong with this, but if you really care about efficiency you have to be concerned first and foremost with the asymptotically most efficient algorithm, something that most programmers know little about. I won't go into this here, but if you're interested, read Knuth or Cormen et. al. Programming competitions that attempt to measure skill in generating algorithms should therefore use large data sets in order to test for algorithms that scale well. Looking over the ACM problems, I see some that use explicitly small data sets and some that don't. In the real world, large data sets are a fact of life, and micro-optimization on small data sets is not nearly as important.

That said, it is certainly true that many programming competition problems depend on finding an algorithm that has optimal asyptotic behavior, versus a more naive solution that might e.g. require exponential time. So I consider this a minor problem.

Now to the core of my argument. Good programmers are people who write good programs. Let's look at various aspects of this skill.

Algorithmic efficiency This is what programming competitions (partially) measure and which is discussed above. Even within this narrow domain, though, there are problems with standard competitions. I look at most competitions as a "guess the best algorithm" contest. Note the word "guess". There generally is not nearly enough time to prove that a particular algorithm is optimal (which would make for a much more interesting competition BTW), so you have to use your intuition and hope for the best. Many, many times the best algorithm will hinge on knowledge of some obscure factoid of mathematics (e.g. a closed-form representation for computing the nth fibonacci number), which, though interesting and relevant, hardly can be claimed to represent extreme skill in programming. If all the problems are like this, then programming competitions should be run by the math department. By the way, I wonder how many interesting theorems in mathematics were proved in less than an hour -- I'd guess not many. Good design This involves issues like picking the right set of abstractions for the program, minimizing coupling between conceptually different parts of the program, etc. and is discussed at great length in SICP and in other books. Most programming competitions don't measure this at all, because they involve tiny programming problems, and design only becomes important for large programming projects. In those projects it tends to totally dominate the picture; I'd much rather work on a well-designed large program with poor algorithms (which I could fix) than a poorly-designed large program with fast algorithms (which I could probably never fix). The irony is that many, perhaps most, of the programmers who have the skill set needed to win programming competitions (the ability to write mostly-correct fast algorithms quickly) have little or no design sense. And yet, according to the ACM, the most august body in computer science, these people are "the best programmers"! This sends a message that the ability to design a program well is not an important skill, and explains a lot about why we have to live with bad software which keeps crashing all the time. For instance, I once had a student who (I was told) had won tens of thousands of dollars in programming competitions. He wanted to write a program to play the game of Othello. I didn't like his solution because his graphical user interface (GUI) code was completely mixed up with the Othello-solving code (a classic beginner's design mistake). When I pointed this out to him, he couldn't understand what I was talking about and argued with me (maybe he thought that his competition wins indicated that he knew more about programming than I did). I pointed out that if the language he was using ever changed their standard graphical user interface (which can and does happen), then he'd have to rewrite all his code, even the parts that only dealt with board solving, which should be unaffected. He still wasn't convinced. "There is only one GUI library for this language", he said, "and what if there never is another one?" He was too lazy to rewrite his code to do the right thing, which wouldn't have been necessary if he'd known something about design to start with. Maintainability It's not enough to write a good program. Any good program is going to be extended, usually in ways that the original author could not foresee. Thus, one aspect of good design is to make the design easy to maintain, which may manifest itself in a number of ways. One very simple (but usually overlooked) way to keep code maintainable is to write good comments and keep them up-to-date as the code evolves. Good formatting and style help a lot. Choosing the right abstractions and creating the right abstraction barriers (again, see SICP) help enormously. Refactoring the design at both large and small scales is crucial to keeping code robust (see Fowler's book Refactoring for many examples of this), because a design that works now may not work well a year from now when the original mission of the program has been extended in three different ways by three different programmers with three different visions of what the software should become. Programming competitions never deal with this crucial aspect of real-world programming skill. Good documentation A good program is only as good as its documentation. Even if your program kicks mighty ass, if there is no documentation nobody is ever going to be able to use it. Most "I-write-fast-algorithms-quickly" programmers not only can't write documentation to save their lives, they can't even spell. Perhaps in recognition of this, they tend not to write any comments in their code; a hallmark of this kind of programmer is twenty pages of dense code with no comments at all (perhaps a reflection of an overdose of Jolt Cola). Typically they also like to put multiple lines of code on one very long line with as few spaces as possible, presumably because the effort of hitting the return key or the space bar distracts them from the stream-of-consciousness flow which is essential to their thinking process. The problems start when these programmers leave for another job, and then someone else is stuck with the task of adding new features to the code base which is completely undocumented. Really good programmers typically document code both in comments within the code and in user-level (and developer-level) documentation. This is yet another skill that is never tested in programming competitions. In fact, it's selected against , because the time it takes to write comments detracts from the time it takes to solve the problem. So programming competitions foster bad habits. Note : To give a full description of good program commenting, formatting and documentation practice would go far beyond the scope of this document. Good programmers vary in the way they approach this problem, and it is also dependent on the programming language being used. For instance, code written in assembly language, by sheer necessity, has to contain proportionally more comments than code written in a high-level language such as Scheme (C falls in between). Good choices of variable and function names can obviate the need for some comments as well.

Exceptions to the rule

I have to say that I've seen some programming competitions that did not fit the stereotype I've described above. There was a programming competition put on by the Software Carpentry web site that involved developing a replacement for the standard Unix program build tools ( make , autoconf , automake , libtool ). This competition was heavily oriented towards good design, and the time limit was substantial (several weeks if I recall correctly). Of all the programming competitions I've heard of, this one was the best model of what I think a programming competition should really be like (if we need competitions at all, which we don't -- see below).

Another competition I like is the International Functional Programming Competition (IFPC). Even though this competition usually has a rigid time limit (one weekend), the purpose of the competition is mainly to showcase the ability of lesser known functional programming languages rather than to pit programmer against programmer, and the cash rewards are negligible (it's mainly about bragging rights, both for the programmer and for the language implementation being used).

Yet another competition I like very much is the International Obfuscated C Code Competition (IOCCC), whose goal is mainly humorous. Nevertheless, some astonishingly beautiful (albeit warped) programs have been created for the IOCCC, and the programmers are, in my opinion, real artists. There is a similar competition for perl programmers, and similar comments apply.

Despite all of this, I have never entered a programming competition and probably never will, because I don't like programming with a stopwatch.

Cooperation, not competition, is the correct metaphor for programming

My final object to programming competitions is that they represent the wrong way to think about programming. Programming is not a sport like tennis or basketball, where one player/team "wins" and another player/team "loses". The free software and open source programmers of the world have shown the power of large groups of programmers working together to solve problems. In order to do this, the programmers involved have to develop all of the skills needed to write really good programs, not just the ability to write algorithms quickly. Design, documentation, maintainability, abstraction, etc. are all important. Therefore, my advice to programmers who want to improve their skill: skip the programming competitions and start a free software/open source project of your own. If you do that, you have a chance of becoming a truly good programmer, not just a glorified code-grinder.

What about the ACM contest?

I have nothing against the ACM (in fact, I think it's a fine organization), and I'm sure the intentions behind the programming contest are good. If one accepts the notion that programming contests are worthwhile (which I don't, for the reasons given above), then what should be done about this contest?

The ACM contest rules reward two things:

getting as many problems right as possible;

assuming you got a problem right, solving it in the smallest amount of time possible.

The fact sheet for the contest states that all problems (ten for the 2005 contest) must be solved in five hours (!) by teams of three programmers. And the solutions also have to be as fast as possible. So this is definitely a "guess the best algorithm quickly" kind of programming contest.

I can think of a few things that the ACM could do to make their contest more representative of real programming skill:

Divide the programming contest into two parts. One part will be the algorithm part, which would be similar to the current contest. The second is a design contest, where the task is a larger-scale problem and the goal is to come up with the most elegant, powerful, extensible, and maintainable design. Extend the time limits to one week for the algorithm part and one month for the design part. Judge algorithm solutions on a variety of criteria, including: Fastest solution.

Most scalable solution (programmatically scalable, not just algorithmically scalable).

Most elegant solution.

Clearest solution. Award special prizes for algorithm solutions that come with mathematical proofs of correctness (and optimality, if that's relevant).

My advice to budding programmers is not to worry about programming competitions. However, I do recommend that they download the problems for the ACM competition (and other competitions e.g. TopCoder) and work through them on their own time without any time pressure. Aside from being fun, this will undoubtedly improve their (algorithm-related) programming skills.

Epilogue: July 2006

I haven't read through this article in a long time, but I find that I still feel substantially the same way. One person pointed out that the ACM problems have very comprehensive test cases, so to imply that they reward "mostly-correct" programs is inaccurate. No argument there. I was really thinking more about the mentality that programming competitions with short time limits foster. Also, if your notion of "correctness" includes things like good design, maintainability, readability etc. (which it certainly doesn't in the ACM contest) then my point stands as is.

There was a recent (2005) ICFP contest that explicitly targeted good design by requiring first a solution to one problem, and then a solution to an extension of the first problem. That's a cool idea -- it's just like working in a real job, where your pointy-haired boss changes his mind every three days ("How hard would it be to merge our stock simulation software with Google and Mediawiki?"). To do well in this competition, you'd want to have a very flexible design up front, which is something most competitions don't address at all.

I still feel, however, that the best programmers aren't the ones who do the best in competitions but are the ones who write the best programs. I'm reminded about a comment Frank Zappa said about guitarists, to the effect that all the flashy guitarists who can play 1000 notes a minute were simply turning music into a gymnastic competition. And these guys aren't making the best music: for instance, Yngvie Malmsteen or Steve Vai (who can play rings around most guitarists) haven't written a lot of great songs, while someone like Kurt Cobain (who only had the most basic guitar skills) could hardly write a bad song. So I'd like to see less competition amongst programmers (except the kind of friendly bragging-rights or for-education or for-the-beauty-of-it style of competition you see in the ICFP and IOCCC contests) and a greater appreciation of what the art of programming is really all about. And I still think that solving programming competition problems is a great way to improve your skill -- provided that you take your time. I still hate programming with a time limit.

Acknowledgments

References

Abelson, H., Sussman, G. J., and Sussman, J. Structure and Interpretation of Computer Programs MIT Press, 1996. Knuth, Donald Ervin. The Art of Computer Programming, 3rd. Ed. Addison-Wesley, 1997, 3 volumes. Cormen, T. H., Leiserson, C. E., Rivest, R. L., Stein, C. L., Introduction to Algorithms, 2nd. Ed. MIT Press, 2001. Fowler, M. Refactoring: Improving the Design of Existing Code Addison-Wesley, 1999. The ACM programming contest home page. You can download the competition problem sets here. The ICFP contest home page

Go back to my home page. Last updated August 28, 2007