Thank you for your comment

 Why Racket? Why Lisp?

 Beau­tiful Racket and Prac­tical Typog­raphy were made possible by a publishing system called Pollen. I created Pollen with the Racket program­ming language. Racket is a descen­dant of Scheme, which in turn is a descen­dant of Lisp.

 So while Racket is not Lisp (in the specific Common Lisp sense), it is a Lisp (in the familial sense). Its core ideas—and core virtues—are shared with Lisp. So talking about Racket means talking about Lisp.

 In prac­tical program­ming projects, Lisps are rare. Racket espe­cially so. Thus, before I orig­i­nally embarked on my Lisp adven­ture, I wanted to under­stand the costs & bene­fits of using a Lisp. Why do Lisps have such a great repu­ta­tion, yet few users? Was I seeing some­thing everyone else missed? Or did they know some­thing I didn’t? To find out, I read what­ever I could find about Lisps, including Paul Graham’s Hackers & Painters and Peter Seibel’s Prac­tical Common Lisp. (OK, parts. It’s a big book.)

 What I found was plenty of Lisp flat­tery from expert Lisp program­mers. (Also plenty of Lisp kvetchery from its detrac­tors.) What I didn’t find were simple, persua­sive argu­ments in its favor. So here’s why Racket was the right tool for me, and what I see as the prac­tical virtues of Lisps in general.

 I didn’t study computer science in college (though I was a math major for two years, before switching to design). I’ve never held an offi­cial job as a programmer. Rather, program­ming has been a secondary skill I’ve used in my work as a web designer, type designer, and writer.

 But in the last few years, I’ve spent an increasing amount of my time program­ming. This program­ming gener­ates income. So by the simplest defi­n­i­tion—does the skill make you money?—I suppose I qualify as a profes­sional programmer. And since most of my program­ming efforts are in Racket, I qualify as a profes­sional Racket programmer.

 Mind you, I’m not claiming that I’m an expert programmer. Among the Racket commu­nity, which is laden with computer-science Ph­Ds & profes­sors, I (have no choice but to) embrace my rela­tive inex­pe­ri­ence. Hence the title of my talk at Racket­Con 2014: Like a Blind Squirrel in a Ferrari.

 Yet despite my limi­ta­tions as a programmer, with Racket I’ve been able to render bigger ideas into programs more quickly, and with fewer bugs, than any language I’ve used before (and there have been many—BASIC, C, C++, Perl, Java, Java­Script, Python, and others). Since I haven’t gotten a brain trans­plant recently, there must be some­thing special about Racket as a language.

 Lisp is a language most program­mers have heard of, for two reasons. First, it’s one of the oldest computer languages, in use since 1958. Second, it’s accrued a repu­ta­tion as a language for brainiacs. Orig­i­nally this repu­ta­tion arose from its asso­ci­a­tion with the field of arti­fi­cial intel­li­gence. Since then, this repu­ta­tion has been main­tained by peri­odic endorse­ments from respected program­mers (latterly, Eric Raymond and Paul Graham) and the enduring fame of the text­book used in intro­duc­tory computer-science courses at MIT, Struc­ture and Inter­pre­ta­tion of Computer Programs (which uses Scheme, and that one I did read start to finish).

 But as main­stream program­ming tools, Lisp and its descen­dants have been largely ignored. Popu­larity of program­ming languages is tricky to measure, but here’s a simple proxy—let’s count the number of projects currently hosted on Git­Hub. One could quibble about the accu­racy of this method, except that the results aren’t even close:

  Language Git­Hub projects JavaScript 7,488,087 Java 6,570,575 Python 4,017,958 PHP 2,123,489 Ruby 1,699,590 Clojure 64,239 Lisp 17,989 Scheme 12,412 Racket 11,725

 The last four languages are Lisps, and together account for only 106,365 projects. Racket itself only accounts for a small frac­tion of this small frac­tion.

 Popular program­ming languages aren’t neces­sarily good—look what’s at the top of that list. + [Java­Script] has a lot of stupid in it … The good parts of [Java­Script] go back to Scheme and Self.

—Brendan Eich, here and here But unpop­ular languages often have fatal flaws that prevent wider adop­tion. As I was consid­ering languages, Racket had a lot to recom­mend it. But was there a fatal flaw I was over­looking? And by commit­ting to a Lisp, would I be painting myself into a corner? I wanted to under­stand the risks and bene­fits.

 I said above that Lisp flat­tery is easy to find. The problem with Lisp flat­tery is that it makes sense only to expe­ri­enced Lisp program­mers. To others—espe­cially those who are trying to decide whether to learn and use a Lisp—it just comes across as unsub­stan­ti­ated hoodoo.

 For example, in his essay How to Become a Hacker, Eric Raymond says “Lisp is worth learning for … the profound enlight­en­ment expe­ri­ence you will have when you finally get it. That expe­ri­ence will make you a better programmer for the rest of your days, even if you never actu­ally use Lisp itself a lot.” Unfor­tu­nately Raymond doesn’t follow up this claim by describing the “enlight­en­ment expe­ri­ence,” nor why it’s “profound,” nor how it will improve your program­ming skills gener­ally.

 To be fair, Raymond’s essay is not focused on Lisp. But compare Beating the Aver­ages, by Paul Graham, which is. Graham starts off by citing Raymond’s compli­ment to Lisp and seems ready to make the claim concrete.

 Instead, he breaks it into smaller chunks of flat­tery. “We knew Lisp was a really good language for writing soft­ware quickly.” Because of what char­ac­ter­is­tics? He doesn’t say, but then describes Lisp as his “secret weapon”. OK, so what’s the secret? He says “program­ming languages vary in power”. Fine, but what exactly makes Lisp more powerful?

 Graham offers one concrete example: Lisp’s macro facility, which he describes as its ability to make “programs that write programs”. After four years using a Lisp language, I’d agree with Graham that macros are great when you need them. But for someone new to Lisp languages, they’re not neces­sarily a bread-and-butter benefit.

 I was hopeful when I opened Peter Seibel’s Prac­tical Common Lisp and saw that the intro­duc­tion was subti­tled “Why Lisp?” Yes, tell me! Seibel echoes Graham’s claim: “You’ll get more done, faster, using [Lisp] than you would using pretty much any other language.” OK, but how? Seibel wonders whether “I like Lisp because of some quirk in the way my brain is wired. It could even be genetic, since my dad has it too.” That’s not encour­aging to those of us outside your family. Ulti­mately, he sums up the appeal of Lisp by describing it as “the program­mable program­ming language”. But I’ve never used a program­mable program­ming language. Why should I start?

 And by the way, when do I get the speed and power you keep promising?

 In short—what’s in it for me, now?

 This is the funda­mental ques­tion that Lisp advo­cates have to answer for new users. But more often, it’s side­stepped. I’m not picking on Raymond or Graham or Seibel. They’re excel­lent writers. As program­mers, they’re way out of my league. As I learn more about Lisps, I return to these arti­cles and they make more sense.

 But these arti­cles are also emblem­atic of a general weak­ness of messaging about Lisp. I say that not as a ranking member of the Lisp commu­nity, but rather as someone who spent a lot of time seeking an answer to that funda­mental ques­tion. I never got it.

 Seibel is passing the buck when he says that to under­stand the bene­fits of Lisp, “you’re going to have to learn some Lisp and see for your­self”. Sure, this method works—using Racket for a few months finally made the bene­fits of Lisp clear to me. But it also required an invest­ment of about 100–200 hours. + For more on the perils of taxing reader patience, see why does typog­raphy matter in Prac­tical Typog­raphy.

 That’s asking too much. If Lisp languages are so great, then it should be possible to summa­rize their bene­fits in concise, prac­tical terms. It should be possible to demon­strate the power of Lisp in one hour, not 100. If Lisp advo­cates refuse to do this, then we shouldn’t be surprised when these languages remain stuck near the bottom of the charts.

 In a word, expres­sive­ness: the measure of how easy it is to put your ideas into code. For instance, an expres­sive language like Racket lets you write the “Hello world” program like this:

 "Hello world" 1 "Hello world"

 Whereas a less expres­sive language—I won’t name names—requires this:

 public class HelloWorld {

public static void main(String[] args) {

System.out.println("Hello world");

}

} 1 2 3 4 5 public class HelloWorld { public static void main ( String [] args ) { System . out . println ( "Hello world" ); } }

 Conci­sion is valu­able, but expres­sive­ness also embodies other qual­i­ties: preci­sion, read­ability, flex­i­bility, poten­tial for gener­al­iza­tion.



 Compared to other languages, Lisps are tremen­dously expres­sive. Like the over­pow­ered Japanese motor­cycle I once owned, they go where you want, very quickly, with a minimum of input. If you’ve ridden a motor­cycle, then you know what I mean. If you haven’t, good news—Lisps are cheaper and safer.

 Here’s my ranking of the language features that offered the most imme­diate value to me, when I was a programmer new to the Lisp world. For each, I’ve noted whether it’s a feature of Racket specif­i­cally, or Lisps gener­ally.

 Every­thing is an expres­sion. [Lisps] Most program­ming languages are a combi­na­tion of two distinct ingre­di­ents: expres­sions (things that are eval­u­ated to produce a value) and state­ments (things that denote an action). For instance, in Python, x = 42 is a state­ment, and x + 42 is an expres­sion.  State­ments and expres­sions are distinct because while expres­sions can be natu­rally nested with each other, state­ments and expres­sions cannot. For instance, in Python, this is a valid expres­sion:  42 + 101 1 42 + 101  And so is this, substi­tuting another expres­sion for the right-hand value:  42 + (100 + 100) 1 42 + ( 100 + 100 )  But this is not valid:  42 + if 1 < 0:

100

else:

200 1 2 3 4 42 + if 1 < 0 : 100 else : 200  Why? Because in Python, a stan­dard if condi­tional is a state­ment, and can only be used in certain posi­tions.  By making every­thing an expres­sion, however, Lisps remove this limi­ta­tion. Since expres­sions are nestable, anything in the language can be combined with nearly anything else. For instance, because an if condi­tional is an expres­sion, you can use it in place of a value:  (+ 42 (if (< 1 0) 100 200)) 1 ( + 42 ( if ( < 1 0 ) 100 200 ))  “But wait! Python has a ternary condi­tional expres­sion!” It doesn’t change the essen­tial point, but OK—you can indeed write this:  42 + (100 if 1 < 0 else 200) 1 42 + ( 100 if 1 < 0 else 200 )  But now suppose we want to use a condi­tional in place of the oper­ator, not the right-hand value. In a Lisp, because every­thing is an expres­sion—including the oper­ator itself—this is easy:  ((if (< 1 0) + *) 42 100) 1 (( if ( < 1 0 ) + * ) 42 100 )  But if you try the same thing in Python, it will raise a syntax error:  42 (+ if 1 < 0 else *) 100 1 42 ( + if 1 < 0 else * ) 100  Why? Because Python oper­a­tors are not expres­sions.  This is a synthetic example. The point is not that you’d neces­sarily want to do this, but that Lisps permit it. You don’t run into the syntactic guardrails that are lurking in other languages. As a programmer, this simpli­fies your work, because every­thing snaps together easily. It also expands your possi­bil­i­ties, because you can combine parts of the language in unusual ways if you feel like it.  It’s similar to the basic idea behind Legos. Other building sets offer special­ized pieces that can only fit together certain ways. But by sharing uniform measure­ments, Lego bricks offer maximum possi­bil­i­ties for combi­na­tions. This ends up being more flex­ible & more fun.  So it is with an expres­sion-based language. If you find this idea exciting, congrat­u­la­tions—you might be a Lisp programmer. (If you find this idea weird and scary, this is a good moment to bail out.)  Every expres­sion is either a single value or a list. [Lisps] Single values are things like numbers and strings and hash tables. (In Lisps, they’re some­times called atoms.) That part is no big deal.  The list part, however, is a big deal. In a language like Python, the list is one data type within the language. But in Lisps, the list is more like an orga­nizing prin­ciple for every­thing that happens. + The name “Lisp” is an abbre­vi­a­tion for “list processing”. So yes, you can use the list as a data type. But a func­tion call is also a list. In fact, the source code for the func­tion is a list. Actu­ally, the rest of the program is too. Lists are every­where. + The fancy CS term for this prop­erty is homoiconicity.  The bene­fits of lists are similar to that of expres­sions. By bringing more of the language into a consis­tent form, more possi­bil­i­ties arise for how pieces can be combined and manip­u­lated.  Seibel describes Lisp as a tool for getting “more done, faster”. Here, you can start to see why this is so. Lisp languages are immensely flex­ible and permis­sive in how their pieces can be connected. This means that the way you think about a program­ming problem can be quite close to the way you actu­ally program it. (This is also why Lisps have tradi­tion­ally excelled for proto­types and exploratory work.)  To be fair, getting the most out of a Lisp means learning to think more in the Lisp idiom of lists and expres­sions. For that reason, I agree with Seibel—trying it your­self is the best way to be convinced of the bene­fits. As you get a feel for lists and expres­sions, it does pay increasing divi­dends throughout the language. You see how tiny lines of code can produce epic amounts of work. You also start to appre­ciate that even in a well-designed language like Python, you’re spending a lot of time shaping your ideas to fit its limi­ta­tions, like shaving an invis­ible yak.  Func­tional program­ming. [Lisps] Yes, I know that other languages offer func­tional-program­ming features, and that Lisps aren’t consid­ered pure func­tional languages. But many program­mers haven’t been exposed to this idiom, and thus tend to under­rate its bene­fits. I know I was in that cate­gory.  Func­tional program­ming doesn’t mean program­ming with func­tions. Every­body does that. Func­tional program­ming refers to a stricter style where func­tions receive certain data as input, process only that data, and return a result. In func­tional program­ming, func­tions avoid two habits common in other languages: muta­tion (= changing data in-place rather than returning a value) and relying on state (= extra context that’s not provided as input, for instance global vari­ables).  “Wait—I love state and muta­tion. Why would you take them away?” Because they’re false friends. They contra­dict the essen­tial concept of a func­tion, which is to encap­su­late data and algo­rithms. When a func­tion relies on state or muta­tion, it’s oper­ating outside those bound­aries. There­fore, you either take on an increasing house­keeping burden to keep track of how func­tions affect each other, or watch your program sink into a swamp of myste­rious, compli­cated bugs.  Program­ming in a func­tional style takes more effort at the outset. But it encour­ages you to struc­ture the program in a clean, compart­men­tal­ized way. This pays off imme­di­ately in programs that are easier to test and debug. It’s also more likely to lead to reusable compo­nents, since func­tions are truly inde­pen­dent.  This bite-the-bullet aspect of func­tional program­ming is another reason why you can get “more done, faster” with a Lisp. The differ­ence between proto­type and produc­tion code often ends up being small, because you don’t take as many short­cuts at the start. The program grows and evolves more smoothly because it’s easy to change one part without causing ripple effects else­where.  Macros. [Racket] Some Rack­e­teers quibble with this term, prefer­ring syntax trans­formers, because a Racket macro can be more sophis­ti­cated than the usual Common Lisp macro.  A macro in Common Lisp is a func­tion that runs at compile time, accepting symbols as input and injecting them into a template to produce new code.  Macros in Racket, on the other hand, rely on the concept of hygiene. They can handle Common Lisp-style macros, but also more elab­o­rate syntax rearrange­ments.  But forget that—what’s in it for you? As a programmer, you end up getting two bites at the apple every time you run a file: Racket runs the macros (which alter the source code), and then the source code itself.  Unlike some­thing like the C preprocessor, which is basi­cally a sepa­rate mini-language, Racket macros are them­selves Racket func­tions that give you access to every­thing in Racket. Like lists and expres­sions, macros add another layer of expres­sive possi­bil­i­ties.  Create new program­ming languages. [Racket] When I first read that Racket could be used to create new languages, I had two thoughts—are they serious? and would I really want to do that? The answers were yes and oh hell yes.  Between expres­sions, lists, and macros, Racket gives you a huge amount of semantic flex­i­bility. But on top of that, it also adds syntactic flex­i­bility, in that you can define a reader that converts surface syntax into stan­dard Racket S-expres­sions, and an expander that deter­mines the meaning of these S-expres­sions. + Paul Graham’s program­ming language Arc, a dialect of Lisp, was built on top of Racket.  You can use this facility to make special­ized dialects of Racket. Or imple­ment earlier languages. Or create entirely new languages with their own rules. You can use any of these languages within Dr­Racket to code new projects. (These special languages are some­times called domain-specific languages, or DSLs.) Scribble is a DSL based on Racket; Pollen is a set of DSLs based on Scribble.  If you’re like most program­mers, you’ve never had a tool for making a new language, so you’ve not consid­ered it a real­istic approach to a problem. And you won’t need it all the time. But when you do, it is awesome, in both the new and old senses of that word.  (The sequel to this piece—Why language-oriented program­ming? Why Racket?—is devoted to this topic.)  Libraries & docu­men­ta­tion. [Racket] This might not look like a compet­i­tive differ­en­tiator—doesn’t every program­ming language have libraries & docu­men­ta­tion?  Yes, but prob­ably not like this. As a conse­quence of being used in research settings for many years—Racket’s core devel­op­ment team is made of computer-science profes­sors—Racket’s libraries & docs are more like a trans­mis­sion from a highly evolved alien intel­li­gence.  You get the essen­tials, of course: web server, JSON, XML, drawing, foreign-func­tion inter­face, and so on. Then you notice pack­ages you maybe didn’t expect: GUI appli­ca­tion frame­work, math plot­ting, package-distri­bu­tion system, unit tester. Beyond that, your face starts to melt a little bit: seman­tics engi­neering? Futures visu­al­izer?  I won’t pretend to know what all this shit does. A lot of it is over my head. But I like that. Each week I use Racket, I end up exploring a new part of the library, and learning some­thing new. As opposed to other languages that seem to kill brain cells on contact (= pretty much anything named *Script, I find).  This learning is only possible because of Racket’s truly outstanding docu­men­ta­tion. It’s vast, thor­ough, precise, and approach­able. See for your­self. + If you don’t like the typog­raphy and layout of the docs, blame me.  Dr­Racket. [Racket] Yes, I know how to use a command line. But Racket includes a cross-plat­form graph­ical IDE called Dr­Racket that’s pretty great. Dr­Racket lets you edit, run, and debug Racket source files (or any other language based on Racket—see item #9 on this list.)  No, it doesn’t have the Ginsu-level search-and-replace facil­i­ties of some­thing like Sublime Text. But it does have helpful editing features opti­mized for Racket code (for instance, you can right-click on a symbol name and rename it throughout the file, or jump from a func­tion to its docu­men­ta­tion).  More­over, the command line within Dr­Racket doesn’t just show plain text—it can show stacked frac­tions, draw­ings, math plots, and other unex­pected guests. If your command line does all that, by all means keep using it.  X-expres­sions. [Racket] This choice is some­what biased by my work with Racket, which mostly involves docu­ment processing and type­set­ting. But related topics arise in most web program­ming. An X-expres­sion is a special native data struc­ture that Lisps use to repre­sent HTML and other XML-ish data.  Well, not “special” in a Lispy sense—keeping with the usual policy, an X-expres­sion is just another list—but special in the sense that other program­ming languages don’t have it. Usually your choice is to repre­sent HTML either as a string or as a full XML tree. A string is wrong because it doesn’t capture the struc­ture of the HTML, as defined by its tags and attrib­utes. An XML tree shows this struc­ture, but conceals the sequen­tial nature of the data elements, and is unwieldy to work with.  An X-expres­sion ends up being an ideal hybrid between a string and a tree. More­over, because it’s just another list-based expres­sion in the language, you have a lot of options for processing it. Trans­lating an X-expres­sion to or from a text repre­sen­ta­tion using angle brackets is trivial and fast. (Details.)  Given the close kinship between XML-ish data struc­tures and Lisp languages, I have no expla­na­tion why, during the Internet era, they’ve not been paired more often. They’re like peanut butter and jelly.  Scribble. [Racket] Pollen wouldn’t have been possible without Scribble, so for me, this has been the stone-cold killer feature of Racket. But that won’t be true for everyone, so I’m moving it down the list. + Scribble was orig­i­nally created to serve as Racket’s docu­men­ta­tion language (a job it does well).  Scribble is a dialect of Racket that inverts the ordi­nary rela­tion­ship of plain text and code: rather than embed­ding text strings within source, a Scribble docu­ment consists of code expres­sions embedded within plain text.  “So it’s like an HTML template language.” Yes, in the sense that a template language allows code to be embedded in text. But also no, because a template language is usually a pidgin version of a real program­ming language. Scribble, by contrast, lets you invoke any Racket code simply by adding a command char­acter to the front. In keeping with the theme already estab­lished, this approach is both simpler (because there’s almost nothing new to learn) and more powerful (because you can invoke anything in Racket).  In its combi­na­tion of text and code, Scribble has more kinship with La­TeX. While it doesn’t have the type­set­ting facil­i­ties of La­TeX, the program­ming facil­i­ties are much better.  Oppor­tu­ni­ties to partic­i­pate. [Racket] In theory, open-source soft­ware projects create the oppor­tu­nity for groups of devel­opers to join together and make better things in collab­o­ra­tion than they could sepa­rately.  In prac­tice, I’ve found that they sort into a bimodal distri­bu­tion: over here, the under­doc­u­mented solo projects that sputter along fitfully (if at all); over there, the mature, popular projects that can be intim­i­dating for new contrib­u­tors.  As an open-source project, Racket is posi­tioned at a happy medium. The core devel­op­ment team has been working together for years, and the commits remain fast & furious. But they’re friendly scien­tists, not Shire-dwelling egotists, and remain recep­tive to improve­ments across the whole system. If you have a better idea, they’ll listen; if you code it up to their stan­dards and make a pull request, they’ll take it.

 The point of this list has been to tell you about the posi­tives. That doesn’t mean there aren’t nega­tives. The small pool of Racket program­mers means that when you hit a pothole, it’s possible no one’s ever seen your problem (= the inverse of Linus’s Law). If I wanted to hire a Racket programmer, the options would be few.

 Still, why shouldn’t I be enthu­si­astic? What I’ve been able to accom­plish so far with Racket has been tremen­dously useful, educa­tional, and fun—the most fun I’ve had in 25+ years of program­ming.

 If you think I sound like a fanboy or cult member, I can live with that. But those are people whose enthu­siasm is dispro­por­tionate to reality. Here, I’ve tried to stay out of the clouds (and the weeds) and explain the concrete, prac­tical features that have made Racket such a plea­sure in my own work.

 As always, your mileage may vary. But if I persuade a few people to down­load Racket and try it, I’ll be happy. In fact, if you try it and don’t like it, I invite you to contact me, because I’m always curious to hear dissenting opin­ions.

 I will end by taking on the big kahuna—

 I won’t claim I’ve reached the top of the moun­tain. But I can tell you what the view looks like so far.

 There’s a sense in which Lisp and its descen­dants are more than program­ming languages. They’re tools in the broader intel­lec­tual inquiry into the theory of compu­ta­tion. Lisp’s inventor, John Mc­Carthy, orig­i­nally consid­ered Lisp a “way of describing computable func­tions much neater than the Turing machines,” adapting the nota­tion of lambda calculus to do so. Racket, like­wise, has grown out of scien­tific research and explo­ration.

 The theory of compu­ta­tion is just one of many great scien­tific discov­eries in the last 100 years. But I don’t get to use quantum mechanics or rela­tivity or DNA sequencing in my daily work. When I’m program­ming, however, I’m using compu­ta­tion.

 Racket, as a Lisp dialect, has many prac­tical bene­fits. But it also opens a window onto a vast theo­ret­ical world that under­lies every­thing we can do with programs. I’m not a brainiac computer scien­tist. But some days, through that window, I can start to see a bit of what they see—some math, some science, a lot of truth, and more than a little beauty and mystery.

 Paul Graham calls Lisp a “secret weapon”. I would clarify: Lisp itself isn’t the secret weapon. Rather, you are—because a Lisp language offers you the chance to discover your poten­tial as a programmer and a thinker, and thereby raise your expec­ta­tions for what you can accom­plish.

 If that’s not a step toward enlight­en­ment, I don’t know what is.