Lisping at JPL

Copyright (c) 2002 by Ron Garret (f.k.a. Erann Gat), all rights reserved.

This is the story of the rise and fall of Lisp at the Jet Propulsion Lab as told from my personal (and highly biased) point of view. I am not writing in my official capacity as an employee of JPL, nor am I in any way representing the official position of JPL. (This will become rather obvious shortly.)

1988-1991 - The Robotics Years

JPL at the time was in the early planning stages for a Mars rover mission called Mars Rover Sample Return (MRSR). In those days space missions were Big, with a capital B. The MRSR rover was to weigh nearly a ton. The mission budget was going to be in the billions of dollars (which was typical in those days).

Against this backdrop I went to work for a fellow named David Miller, who also happened to be my thesis advisor. Dave had the then-radical idea of using small rovers to explore planets instead of big ones. In 1988 that was a tough sell. Very few people believed that a small rover could do anything useful. (Many still don't.)

Using some creatively acquired R&D funding, Dave hired Colin Angle (then a grad student working for Rod Brooks at MIT, now CEO of IS Robotics) as a summer student. Colin built a small robot named Tooth, which stood in very stark contrast to the 2000-pound Robby, which was the testbed for the MRSR mission.

At the time it was more or less taken for granted that AI work was done in Lisp. C++ barely existed. Perl was brand new. Java was years away. Spacecraft were mostly programmed in assembler, or, if you were really being radical, Ada.

Robby had two Motorola 68020 processors running vxWorks, each with (if memory serves) 8 MB of RAM. (This was considered a huge amount of RAM in those days. In fact, the Robby work was often criticized for, among other things, being too memory-hungry to be of any use.) Tooth, by contrast, had two Motorola 68hc11 8-bit microcontrollers each with 256 bytes of RAM and 2k bytes of EEPROM. (In later robots we used 6811's with a whopping 32k bytes of RAM.)

(As an indication of just how fast and how radically attitudes can change, the other day I heard an engineer complain that "you can't do anything with only 128 megabytes.")

Both Robby and Tooth were programmed using Lisp. On Robby we actually ran Lisp on-board. The Lisp we used was T 3.1 which was ported from a Sun3 to vxWorks with a great deal of help from Jim Firby, who came to JPL from Yale. Jim also wrote a T-in-common-lisp compatibility package called Common T. Using Common T we could do code development on a Macintosh (using Macintosh Common Lisp and a Robbie simulator, also written by Jim) and then run the resulting code directly on Robby with no changes.

Tooth's processors didn't have nearly enough RAM to run Lisp directly [1] so instead we used a custom-designed compiler written in Lisp to generate 6811 code. At first we used Rod Brooks's subsumption compiler, but later I decided I didn't like the constraints imposed by the subsumption architecture and wrote my own for a language called ALFA [2]. ALFA was subsequently used to program a whole series of rovers, the Rocky series, which eventually led to the Sojourner rover on the Mars Pathfinder mission. (Sojourner had an Intel 8085 processor with 1 MB of bank-switched RAM. It was programmed in C. More on this decision later.)

Tooth, Robby, and the Rocky series were among the most capable robots of their day. Robby was the first robot ever to navigate autonomously in an outdoor environment using stereo vision as its only sensor. (NOTE: The stereo vision code was written in C by Larry Matthies.) Tooth was only the second robot to do an indoor object-collection task, the other being Herbert, which was developed a year or two earlier in by Jonathan Connell working in Rod Brooks's mobot lab at MIT. (But Tooth was vastly more reliable than Herbert.) The Rocky robots were the first ever microrovers to operate successfully in rough-terrain environments. In 1990, Rocky III demonstrated a fully autonomous traverse and sample collection mission, a capability that has not been reproduced to my knowledge in the twelve years since.

The period between 1988 and 1991 was amazingly productive for autonomous mobile robot work at JPL. It was also, unfortunately, politically turbulent. Dave Miller's group was, alas, not part of the organizational structure that had the "official" charter for doing robotics research at JPL. The result was a bloody turf battle, whose eventual result was the dissolution of the Robotic Intelligence group, and the departure of nearly all of its members (and, probably, the fact that Sojourner was programmed in C).

1992-1993 - Miscellaneous stories

Also in 1993 I used MCL to help generate a code patch for the Gallileo magnetometer. The magnetometer had an RCA1802 processor, 2k each of RAM and ROM, and was programmed in Forth using a development system that ran on a long-since-decommissioned Apple II. The instrument had developed a bad memory byte right in the middle of the code. The code needed to be patched to not use this bad byte. The magnetometer team had originally estimated that resurrecting the development environment and generating the code patch would take so long that they were not even going to attempt it. Using Lisp I wrote from scratch a Forth development environment for the instrument (including a simulator for the hardware) and used it to generate the patch. The whole project took just under 3 months of part-time work.

1994-1999 - Remote Agent

We used four different Common Lisps in the course of the Remote Agent project: MCL, Allegro, Harlequin, and CLisp. These ran in various combinations on three different operating systems: MacOS, SunOS, and vxWorks. Harlequin was the Lisp that eventually flew on the spacecraft. Most of the ground development was done in MCL and Allegro. (CLisp was also ported to vxWorks, and probably would have been the flight Lisp but for the fact that it lacked threads.) We moved code effortlessly back and forth among these systems.

The Remote Agent software, running on a custom port of Harlequin Common Lisp, flew aboard Deep Space 1 (DS1), the first mission of NASA's New Millennium program. Remote Agent controlled DS1 for two days in May of 1999. During that time we were able to debug and fix a race condition that had not shown up during ground testing. (Debugging a program running on a $100M piece of hardware that is 100 million miles away is an interesting experience. Having a read-eval-print loop running on the spacecraft proved invaluable in finding and fixing the problem. The story of the Remote Agent bug is an interesting one in and of itself.)

The Remote Agent was subsequently named "NASA Software of the Year".

Warning!

1999 - MDS

The New Millennium missions were supposed to be the flagships for NASA's new "better, faster, and cheaper" philosophy, which meant that we were given a budget that was impossibly small, and a schedule that was impossibly tight. When the inevitable schedule and budget overruns began the project needed a scapegoat. The crucial turning point was a major review with about 200 people attending, including many of JPL's top managers. At one point the software integration engineer was giving his presentation and listing all of the things that were going wrong. Someone (I don't know who) interrupted him and asked if he could change only one thing to make things better what would it be. His answer was: get rid of Lisp [3].

That one event was pretty much the end of Lisp at JPL. The Remote Agent was downgraded from the mainline flight software to a flight experiment (now renamed RAX). It still flew, but it only controlled the spacecraft for two days.

I tried to resurrect Lisp on my next project (the JPL Mission Data System or MDS) but the damage was done. In an attempt to address one of the major objections to Lisp, that it was too big, I hired Gary Byers, who wrote the compiler for Macintosh Common Lisp (MCL) to port MCL to vxWorks. (Along the way he also produced ports for Linux and Solaris.) The MCL image was only 2 MB (compared to 16 or so for Harlequin), but it turned out not to matter. Lisp was dead, at least at JPL. After two years Gary realized that his work was never going to be used by anyone and he too left JPL. A few months later I followed him and went to work at Google. (The work Gary did on the Linux port eventually found its way into OpenMCL so it was not a total loss for the world.)

There were at least two other major Lisp developments at JPL: Mark James wrote a system called SHARP (Spacecraft Health Automated Reasoning Prototype) that diagnosed hardware problems on the Voyager spacecraft, and Curt Eggemeyer wrote a planner called Plan-IT that was used for ground planning on a number of missions. There were many others as well. All are long forgotten.

2000-2001 - Google

One of the reasons I stayed at JPL for twelve years was that I was appalled at what the software industry had become. The management world has tried to develop software engineering processes that allow people to be plugged into them like interchangeable components. The "interface specification" for these "components" usually involves a list of tools in which an engineer has received "training." (I really detest the use of the word "training" in relation to professional activities. Training is what you do to dogs. What you should be doing with people is educating them, not training them. There is a big, big difference.)

To my mind, the hallmark of the interchangeable component model of software engineers is Java. Without going into too many details, I'll just say that having programmed in Lisp the shortcomings of Java are glaringly obvious, and programming in Java means a life of continual and unremitting pain. So I vowed I would never be a Java programmer, which pretty much shut me out of 90% of all software engineering jobs in the late 90's. This was OK since I was managing to put together a reasonably successful career as a researcher. But after Remote Agent I found myself more and more frustrated, and the opportunity to work at Google just happened to coincide with a local frustration maximum.

One of the reasons I decided to go work for Google was that they were not using Java. So of course you can guess what my first assignment was: lead the inaugural Java development at the company, what eventually became Google AdWords. Thank God I had a junior engineer working for me who actually knew something about Java and didn't mind it so much. In the ancient tradition of senior-junior relationships, he did all the work, and I took all the credit. (Well, not quite -- I did write the billing system, including a pretty wizzy security system that keeps the credit card numbers secure even against dishonest employees. But Jeremy wrote the Lion's share of AdWords version 1.)

I did try to introduce Lisp to Google. Having had some experience selling Lisp at JPL I got all my ducks in a row, had a cool demo going, showed it to all the other members of the ads team, and had them all convinced that this was a good idea. The only thing left was to get approval from the VP of engineering. The conversation went something like this:

Me: I'd like to talk to you about something... Him: Let me guess - you want to use Smalltalk. Me: Er, no... Him: Lisp? Me: Right. Him: No way.

2001-2004 Your tax dollars at work

But I digress.

Commentary

The situation is particularly ironic because the argument that has been advanced for discarding Lisp in favor of C++ (and now for Java) is that JPL should use "industry best practice." The problem with this argument is twofold: first, we're confusing best practice with standard practice. The two are not the same. And second, we're assuming that best (or even standard) practice is an invariant with respect to the task, that the best way to write a word processor is also the best way to write a spacecacraft control system. It isn't.

It is incredibly frustrating watching all this happen. My job today (I am now working on software verification and validation) is to solve problems that can be traced directly back to the use of purely imperative langauges with poorly defined semantics like C and C++. (The situation is a little better with Java, but not much.) But, of course, the obvious solution -- to use non-imperative languages with well defined semantics like Lisp -- is not an option. I can't even say the word Lisp without cementing my reputation as a crazy lunatic who thinks Lisp is the Answer to Everything. So I keep my mouth shut (mostly) and watch helplessly as millions of tax dollars get wasted. (I would pin some hope on a wave of grass-roots outrage over this blatant waste of money coming to the rescue, but, alas, on the scale of outrageous waste that happens routinely in government endeavors this is barely a blip.)

In the words of Elton John: It's sad. So sad. It's a sad, sad situation. My best hope at this point is that the dotcom crash will do to Java what AI winter did to Lisp, and we may eventually emerge from "dotcom winter" into a saner world. But I wouldn't bet on it.

NOTES:

[2] ALFA was an acronym that stood for A Language For Action. My plan was to eventually design a language that would be called BETA, which would stand for Better Even Than ALFA. But I never got around to it.

[3] This begs the question of why he said this. The reason he gave at the time was that most of his time was being taken up dealing with multi-language integration issues. However, this was belied by the following fact: shortly before the review, I met with the integration engineer and offered to help him with any Lisp-related issues he was encountering. He replied that there weren't any that I could help with. So while there were issues that arose from the fact that Lisp had to interoperate with C, I do not believe that a good-faith effort was made to address those issues.

Postscript: Many of the multi-language integration headaches were caused by the interprocess communication system that allowed Lisp and C to communicate. The IPC relied on a central server (written in C) which crashed regularly. Getting rid of Lisp did in fact alleviate those problems (because the unreliable IPC was no longer necessary). It is nonetheless supremely ironic that the demise of Lisp at JPL was ultimately due in no small measure to the unreliability of a C program.