I wanted to bring together some ideas from my posts Because Unanswered Problems Are Always Hard and Respecting The Programmer.

Lisp deserves its place as the first high-level programming language. There were other languages of the time which let programmers work at a higher level than they had before, but they did it by putting higher level concepts into the language. Lisp was the first that made it possible to add higher level concepts. It placed power in the hands of the programmer, instead of leaving it in the hands of the language designer.

So if it was so hot, why aren't we all coding Lisp now? Those were days when resources were much more limited, but that didn't stop Lisp, and it still should have been ready when the computing resources became available. Lisp should have been poised to take over the world (and a lot of people thought it would). There's lots of different theories about why this didn't happen, but I'll offer one more.

Lisp makes good programmers really productive, more than they could be in another language. Paul Graham talks about this in Beating the Averages. He made great software and sold it for a bundle to Yahoo. But now it's been reimplemented in C++_. Why, oh why?

It's easy to blame stupid people for this sort of thing, except that it keeps happening over and over. Metaprogramming is powerful, and was central to Viaweb (20-25% of the code, according to Graham). I think this is an example of Common Lisp's fatal flaw (and since Common Lisp is the standard bearer for all Lisps, it is Lisp's fatal flaw).

Lisp metaprogramming is hard. All metaprogramming is hard. But if you can make due with, say, 75% less code due to metaprogramming, and the code is only 100% harder to understand, the net result is that the code is half as hard to understand in its entirety.

Except Lisp metaprogramming scales wrong. For a really good Lisp programmer, the result may be a lot better than 0.25 x 2.00 (i.e., 25% of the code, 200% as hard to understand, end result is 0.50 times as difficult). But for someone who doesn't know Lisp it's like 0.25 x 10.00 (25% of the code, 1000% as hard to understand = 2.5 times as hard, net; and often it's worse than that), because even after they learn Lisp, they have to learn the macros, and for many (most?) people macros are going to be really really hard.

Quoting from Respecting The Programmer:

We talk about how Python is also easy to learn and maintain, and that's still true, but it doesn't mean that it evens out the differences in productivity between programmers. In fact, quite the opposite, it makes those most productive programmers more productive and more valuable. That Python is easy to learn and maintain means that there's less risk in using a highly skilled, highly productive programmer. In other languages you risk being left with a program that only another highly skilled programmer can maintain, and that's less likely to occur with Python (for an appropriate definition of "highly skilled").

The risk I'm talking about is exactly the trap that the Viaweb code fell into. Maybe the Yahoo manager that moved to C++ was still dumb, but dumb managers happen. Don't create a situation where a dumb manager invalidates your entire implementation. And you have to reward every programmer, no matter what experience they bring in; you should reward the experienced programmer most of all, because why else become experienced? But if you punish the beginner, it's hard to make friends. Code made up of 20% macros is a punishing experience even for an experienced programmer.

So I think it's really important that we approach metaprogramming with caution. I think Guido has been right to resist macros. Not because they are necessarily wrong, but because we haven't yet figured out how to do them right. And maybe we never will, maybe source code simply isn't the right level for abstractions. I think it's good that Python doesn't do tail-call elimination; it seems like a nice feature, but in subtle ways it makes the language harder. And I think continuations could lead to bad things. There are wrong paths on the road to higher-level programming. (Though in some ways I also feel the opposite: tell us not to shoot ourselves in the foot, and expect us to listen, don't assume we'll abuse every feature that exists; there's always a tension in these design choices.)