Two years ago I landed a job at ITA Software, which Paul Graham called one of the “ten or twenty places where hackers most want to work,” along with Google. And one of the interesting things about ITA is that the majority of their software is written in Lisp.

Now before starting there, my experience with Lisp was about the same as most CS grads. I’d used it as an undergrad in a programming languages course, as well as an AI course on natural language processing. And my thoughts about it were pretty standard: it’s a functional language, so if you’re going to use it you’re best off doing recursion and avoiding assignment statements; and dynamic typing and garbage collection make it slow.

Boy, was I wrong. It turns out, Lisp is a lot more practical than that, and we use it for the search engine that powers Orbitz.com, Kayak.com, and most of the U.S. airline websites, among others. The main things I’ve learned are:

It’s not a functional language, it’s a multi-paradigm language. You don’t need Monads to sequence instructions, and a lot of forms automatically sequence their bodies (“implicit progn”). Our code is mostly imperative, we use lots of state and assignments, and we only use recursion rarely, where it’s clearly better than iteration. G_Morgan put it very well. Lisp is the only dynamic language (that I’m aware of) that has optional typing for efficiency. You can tell it the type of any expression, and it will assume the result is that type, thus skipping a potentially expensive check and branch. Sprinkling a few type declarations here and there, along with Lisp’s type inference, can result in compiled code equivalent to optimized C code. Lisp’s macros are great. They can do arbitrary computation at compile time. The C++ crowd has rediscovered this with template metaprogramming in the last few years, but the C++ version is awkward. It’s Turing complete, but the syntax is reallyawkward. Not only is it a pure functional language, but it doesn’t even have loops. No loops! Can you imagine programming without a loop? If you’ve done template metaprogramming, you don’t have to imagine, you’ve lived it. But macros are useful for a lot more. You can use them to bind variables, to provide clean, easy access to values, much in the way that eachXXX withXXX closures are used in Groovy. And they can even be used to develop little languages.

So why hasn’t Lisp caught on more? Well, in a way it has: a lot of the innovation in programming languages has come out of Lisp. Dynamic typing, syntax support for useful data types (only Lists in Lisp), automatic garbage collection, hell, even the if statement appeared in Lisp first, and only made it into other languages once found to be useful. Object Oriented programming is the only major exception that I can think of, mostly because when you have macros, you don’t really need it.

But Lisp’s syntax is a little off-putting. On computers of the ’50s, ’60s, ’70s and ’80s (yes, Lisp has been around that long), Lisp was big and slow. But more importantly, most people’s exposure to Lisp is as an example of a functional language. And functional programming requires you to think like a mathematician, which is unnatural for a lot of people.

I’m waiting for other languages to discover the second and third points above: a dynamic language that cares about efficiency, and a macro capability that’s as powerful as Lisp’s. When I first discovered Groovy, with it’s Java compatibility and optional typing, I thought it was the holy grail of an efficient dynamic language. But sadly it’s creators subscribe to the mainstream view of “dynamically typed languages should be slow, statically typed are fast.” Which is a shame, as it has a lot to recommend it, the syntax and semantics are pretty close to my ideal language. But they seem to have a mental block on annotations or keywords for optional typing or for turning off Groovy’s extreme dynamism when the programmer doesn’t want it anyway.

But Cython looks promising. It has optional typing for efficiency, makes it easy to integrate/recode in C, and there’s currently a lot of debate about compile time transforms and code execution that is a step towards Lisp macros. Let’s hope…