CORECURSIVE #076

LISP in Space

With Ron Garret

LISP in Space

Have you ever had a unique approach to a problem and been excited to use it, but you’re met with skepticism?

Today’s story: what happens if you take someone who’s passionate about LISP and put them in an organization where that’s just not how they write software.

Today’s story is about getting LISP into space.

The year is 1988. The USSR still existed. Ronald Reagan was president of the United States. And Ron, under his thesis advisor, was working on a prototype for the first Mars Rover.

Transcript

Note: This podcast is designed to be heard. If you are able, we strongly encourage you to listen to the audio, which includes emphasis that’s not on the page

Introduction

Adam: Hi, this is CoRecursive, and I’m Adam Gordon Bell. Each episode is the story of a piece of software being built.

Have you ever had a unique approach to a problem and been excited to use it, but you meet with skepticism. That’s today’s story. What happens if you take someone who’s passionate about a certain approach and put them in an organization where that’s just not the way things are done? Today is the story of getting LISP into space.

You know LISP, right? The language with all the parentheses and the macros and the treating code as data. Well, my guest today is a big fan, Ron Garrett. At the time this story starts, he’s wrapping up his PhD in robotics and AI. And the AI that we’re talking about here is not machine learning, it’s AI meaning autonomy, like autonomous robots for self-driving cars. We’re going back in time.

Mars Rover Prototypes

FANG, the Futuristic Autonomous Navigation Gizmo.
"He was actually a Cybermotion K2A." - Ron
(Courtesy of Ron Garret)

The year is 1988. The USSR still existed. Ronald Reagan was president of the United States and Ron under his thesis advisor was working on a prototype for a very iconic NASA project.

Ron: My name is Ron Garrett.

Our goal was to develop the software that would allow NASA to send an autonomous rover to Mars. The challenge with operating on Mars, well, there are lots of challenges to operate on Mars, but one of them is that there’s a 40 minute round trip light time delay. And so you can’t just remotely operate a rover. The time lag is much, much, much, much too long. So it needs a certain level of autonomy because there’s going to be 40 minutes between when you tell it to do something and when you find out what it actually did as a result of what you told it. And so your productivity is going to improve the more that the rover can get done on each one of those cycles. So we were trying to push the level of autonomy so that it could go further on a single command cycle. And that was the goal.

Adam: This all happened at the Jet Propulsion Lab, the JPL in Los Angeles, where Ron had a fairly standard cubicle.

Ron: But around the corner, there was an electronic shop where I would sometimes go when we needed to build some hardware, which back in those days we often did have to build our own custom hardware. And a lot of the robots that we built were indoor robots and we just ran them around the office. But every now and then, we would go outside into, there’s a dry river bed called the Arroyo Seco, which literally means dry riverbed that is adjacent to the lab. And on occasion, we would go out there to run robots outdoors because the environment there is actually very similar to the environment that you get on Mars. And on a couple of occasions, we would actually go out to Death Valley and do experiments out there. And that was a lot of fun.

The Robots

Capture The Flag
Tooth.
"Fun fact: Tooth was built by Collin Angle who went on to become the founder and CEO of iRobot corporation, makers of the Roomba." - Ron
(Courtesy of Ron Garret)

Adam: How big are these robots?

Ron: So the smallest one was about the size of a shoebox, I guess. I should actually start out by telling you about a medium size one that we had, which was about the size of maybe half of a standard refrigerator, big and heavy. And if it ever ran into something, it could do some serious damage. And so we called it FANG, which was an acronym for Futuristic Autonomous Navigation Gizmo. And so the little shoebox sized one, we called Tooth.

Adam: Oh, I see.

Ron: And then there were a couple of outdoor robots. The first one was this giant six wheeled behemoth that used truck tires. It was the size of an SUV and it called Robbie. And then there were a series of smaller rovers, which eventually turned into the first actual Mars rovers, the Sojourner rover. So a series of prototypes for that rover called Rocky, the Rocky series. So there was Rocky 1, 2, 3, 4.

Robbie The Robot (Credit: NASA)

Using LISP

Rocky IV, the last prototype before the Sojourner mission, seen here in the Arroyo Seco next to JPL.
(Courtesy of Ron Garret)

Adam: Sojourner would be the first Mars rover. And we know now it would like land on Mars in 1997 and be considered a great success. But nine years before then in 1988, rovers were just a research project. They were exploring what was possible and Ron’s team’s approach and their choice of programming languages, which was the LISP, it came right out of the academic research they were doing. One potential problem though is that using LISP really wasn’t how software was built at NASA.

Ron: There was quite a bit of prejudice against LISP because it was weird and unfamiliar, and it had this strange garbage collection technology that you just never knew when it would just stop your process dead in its tracks. And that can be catastrophic when you’re dealing in an embedded environment where you have to respond in real-time. And also people thought that LISP was very memory hungry. But back in those days when eight megabytes of RAM was considered a huge amount of RAM. And there were people saying like, yo, you’re sure your research machines have eight megabytes, but you’ll never fly machines that big. Of course, nowadays a hundred mega megabytes is normal. And of course, this is all foreseeable because space qualified hardware follows Moore’s law. Just like everything else, it just lags behind a couple of years. So I found all this to be very frustrating.

Adam: I get this. I’ve been this person before who thinks he has a great way to solve a problem, but hits a lot of barriers with getting people to adopt it. But Ron is talking about running LISP on embedded robots. And to do that, I think you really need a trick.

Ron: On the larger robots that had a few megabytes of RAM, we actually ran LISP on the robot. So the actual code that controlled the robot was written in LISP. Now, the other way which LISP was used for the little robots was designing languages specifically for programming robots and compiling those languages down to embedded code that would run on small processors.

Adam: As an outsider, this seems like a really cool approach. You could get down in the weeds of the robot hardware and try to write autonomous code at that level. But instead, Ron is raising the abstraction. He’s writing a custom language that can best specify the problem and then compiling that down to the actual hardware. It’s a neat trick where every problem becomes a compiler problem. It sounds like it will be a lot of work in most programming languages, but Ron’s using LISP, which he’s been doing since the mid eighties.

Ron: So you have to keep in mind that there is no Java, there is no Python, there is no JavaScript, there is no C++. There is Pascal and C and basic and machine code, and that’s pretty much it in terms of popular languages. And to get anything done in any of those languages is just really, really hard. And this by contrast gives you all of this high level abstraction that you can just use without having to worry about it. And so you don’t have to worry about memory management, you don’t have to worry about dangling pointers. It’s just so much faster and easier to get things done when the language you’re using provides you with some of these high level abstractions. And in a world where the only language that has that is LISP, knowing LISP really is like a superpower. It just blew everything else out of the water back in those days.

Testing Robbie

Ron Garret, Robbie and Rocky III at an event in Washington DC.
(More from event in Bonus Section.) (Courtesy of Ron Garret)

Adam: So Ron was a true believer, but he was also a talented developer and the stakes were high because NASA has a rival team also working on Mars rover prototypes. Ron’s team has to get Robbie and FANG self-driving if they want a chance at their approach being used on the real rover.

Ron: And so we had this development environment where we would develop the code off board on Macintoshes in an environment that had a simulator for the robot, and we get things mostly working and then we’d run the same code on the robot to see if it worked on the real robot. And the reason for that was that running experiments on the real robot, especially on Robbie, was very expensive in terms of time. To all be carefully planned, we had to load Robbie onto a flatbed truck and drive him out into the Arroyo and unload him and get everything set up and all the support equipment. And it took a couple of hours to get set up and then we’d be out there for most of the day running experiments and then we had to pack everything up and put him away again at the end of the day.

So that was a big deal. And so running the code in simulation helped us make more effective use of that time.

Adam: Ron’s group and the work they did was groundbreaking in many ways.

Ron: The goal was to get these rovers to move around in natural environments completely autonomously. So you would tell it, I want to go over there. So Robbie had a pair of stereo cameras. He was actually the first robot in the world to navigate using stereo vision. So in a way, Robbie was the precursor to a lot of the autonomous cars that we have nowadays. They don’t use stereo vision, they mostly use LIDAR, but some of them do use vision processing in order to do obstacle avoidance. So Robbie was the first robot to do this. And then there was the actual navigation code, which took 3D stereo data and the goal location and planned a path and executed it. And that was what I worked on.

Adam: So you’d take it out to wherever and put some location and then you kind of see what happens and…

Ron: Right, exactly. And you have to keep in mind that Robbie moved very slowly. So yeah, he’s big and heavy, but he’s moving it less than one mile an hour.

Adam: Okay.

Ron: So he’s not really that dangerous. If he starts to do something wrong, there’s plenty of time to push the panic button, but you do have to be watching him constantly because he’s a big, expensive one off piece of research equipment and he is very easily capable of driving up onto a rock that would tip him over and that would probably destroy him if he fell over. It never happened, thankfully. But yeah, we had to be watching him all the time, but you didn’t have to worry about him going rogue and killing somebody.

Adam: Developing real-time software for robots in the 1980s, that can be a frustrating endeavor.

Ron: It was cutting edge research code, so it was buggy and it crashed and things went wrong. We never had any catastrophes, thankfully. I think FANG once went rogue and put a hole in the drywall in the office. But that was the worst because FANG w as easily capable of moving fast enough to do some damage. So thankfully, he never hurt anybody.

Politics

Rocky III.
"The white thing on the mast is an electronic compass. The NCC-366R is an inside joke. Our group number was 366. Putting that number in a prominent location was a way of flaunting the fact that this robot belonged to us and not the group we were competing with." - Ron
(Courtesy of Ron Garret)

Adam: Robbie, Rocky, Tooth and FANG, they were some of the most capable self-driving robots at the time. A lot of self navigation first happened on this team. But while Ron was heads down working on the code, things with the team started to go sideways.

Ron: And telling this story would be a little touchy because it was to some extent political. I was young and naive and had my head down in the code and I was vaguely aware that there was all these political battles being fought, but I didn’t really know the details and I didn’t understand how political battles are fought. I’ve learned a lot about that since then, but at the time I was pretty oblivious.

Adam: You see, there was a turf war between Ron’s researchy robotics group which was trying to push autonomy and that other group.

Ron: So their approach was not to just give it a goal and have the rover figure out how to get there, but to have an operator describe an exact path for the rover to follow, which it would then do. They have some level of autonomy, but there is an operator who’s sitting at a console where he has a 3D view of the terrain and describing exact path for the rover to follow on each cycle.

Adam: Building a Mars rover prototype of any kind is, I’m sure, super challenging, but this other team’s approach where they’re describing an exact path, it’s a smaller problem than marking a destination and having the robot navigate to that goal, which is good, but also when it was deployed, there would be some challenges.

Ron: You don’t know whether you’ve succeeded for the better part of an hour after you’ve sent it to command. So you’re sitting there bitting your fingernails hoping that you didn’t screw something up.

Adam: The operational burden of this approach is really high, whereas a totally self-driving approach has a much lower operations cost. In fact, if the robots could be self driven, potentially more and smaller robots could be sent. I mean, they would have to be very reliable, but each small robot could head out exploring on its own and then more could be done with less. But there is a political problem.

Ron: There’s decades of operational experience that into the current ways of doing things. And it’s very labor intensive and developing a technology whose goal is to put these people out of work doesn’t immediately endear you to the people whose livelihoods depend on doing this work. There’s this whole infrastructure of people whose livelihood depends on this technology not existing.

Sojourner on Mars

Adam: So don’t lose hope because the story continues. This LISP code is going into space, but it did not make it into the first Mars rover, Sojourner. The low autonomy operator driven solution won the day, but that might not be a bad thing.

Ron: Even today, I’m not entirely convinced that NASA made the wrong decision. And I base that just on what they’ve actually managed to accomplish. I look at the pictures that come back from these rovers and the science that comes back from these rovers, and it just boggles my mind and blows me away. It impresses me so much that I can barely find the words. Here I am sitting at my desk in my home, looking at photographs, high resolution, color photographs from another planet. That’s just such a spectacular thing to be alive at a time in human history when that is possible.

Adam: Do you feel like it makes it real, like seeing these pictures? Does it seem more like a real place?

Ron: Oh yeah. The photos coming back today are just incredibly high quality and give you a real sense of the place. And one of the things that I find most striking is how similar it is to places on earth and in particular, how similar it is to the places in Death Valley, where we ran some of these early tests. And in particular, it can really, at least gives me a sense of what it would like to actually be there and walking around the there, and all this has me very convinced that I never want to go there. I’ve spent time in Death Valley and these places that are just like there except much more benign because you don’t actually have to be wearing a space suit to survive. I can tell you from semi firsthand experience that colonizing Mars is a pipe dream.

Adam: Really?

Ron: Yeah. We really need to take care of this planet that we have because no matter how bad things get here, it’s still going to be better here. It’s still going to be easier to terraform earth and to terraform Mars.

Adam: So Sojourner was a success, just without the LISP. The code it shipped with was written in C and the autonomy was very restricted. Ron’s group disbanded and many of them left the JPL, though not Ron. And a couple years later, it seemed like there was a new opportunity.

NASA’s New Millennium Program

Website for Deep Space 1

Ron: There were some people who thought that maybe some of this autonomy technology could be used for actual spacecraft, flying in space, and that some of the theories of economies of scale could be applied there. And in particular, NASA got a new director around that time. He was an advocate of this idea of trying to pursue economies of scale in order to reduce mission costs. And he launched a pilot program called the New Millennium series. It was a series of missions, technology demonstration missions with the explicit goal of trying to demonstrate technologies that could be deployed at scale in order to reduce operating costs. And the autonomy technology that we had started developing for rovers was repurposed for that in a project called the Remote Agent that was done in collaboration with the Ames Research Center and some work that people were doing there in planners and diagnostic systems.

There were three or four, depending on how you count, major components of this thing that were developed at JPL and Ames. And one of them came from Carnegie Mellon that were all kind of all together to make the Remote Agent.

Deep Space 1 and The Remote Agent

JPL Deep Space 1 Promotional Image

Adam: It was called the Remote Agent because it was going to have agency. Just like the rover prototypes, it would have a goal and take actions to accomplish that goal. But this time it was a flight controller and not a rover controller.

Ron: So the New Millennium emissions were divided into two categories. There were the deep space missions which went out of earth orbits and the EO missions that were in earth orbit. And we were slated to fly on the first deep space mission, DS1, Deep Space 1, which was going to go to rendezvous with a comet and an asteroid. And they had a whole bunch of other cutting edge technologies that were going to fly on the same mission and get demonstrated in the same mission. And originally we were slated to control the entire mission with the Remote Agent.

Project Management Problems

Adam: But unfortunately problems occurred before things got that far.

Ron: And this, again, is where it gets a little bit a touchy. And I have to be very careful here because I really don’t want to offend anybody who was on the team because everybody on the team was really good and really smart and worked incredibly hard. And the problems were more managerial than technical. So we were four different teams from three different institutions that had all been brought together to build the system and nobody was really in charge of designing and integrating the final system. It was just allowed to kind of happen organically. And so there was a lot of negotiating and cajoling and frankly infighting and arguing. And on the technological side, in particular, there was one component that the three of the four components were written in LISP and one of the four or components was written in C. And the component that was written in C early on kept crashing because when you write in C, you have to worry about all these low level details and this is just a thing that happens when you write in C.

And they eventually got that all worked out, but while things were crashing, that slowed down development because you’d be running an experiment and suddenly it would crash and you had to reboot and reload everything and that cost you 10 minutes and multiply that by a couple of dozen times a day and you’re running behind schedule.

Adam: So then I assume everybody learned that you shouldn’t use C.

Ron: Well, you would think that would be the lesson that they would learn, but no, that lesson was not learned. Nowadays, the flight control software, I believe, is written in C and they have to code it very carefully and use a lot of testing and analysis tools in order to get it to work. And they have managed to make it work and they’ve managed to make it reliable, but it’s a tremendous amount of effort. And unfortunately at the time, I was not very skilled at playing politics and so I was a little bit more blunt about this than I should have been. I ended up off a lot of people and I think that had probably as much to do with the ultimate failure of the project as anything else because all these old timers who knew how to run spacecraft just didn’t like me and didn’t want to have this obnoxious brat around.

Adam: Sure. Ron’s certainty probably ruffled some feathers at the JPL, but it was the time to get these four systems integrated that actually caused the most delays and got this autonomous system downgraded to a flight experiment. Instead of controlling Deep Space 1 for the whole trip, it would control it for just three days. But still three days of controlling a spacecraft is no simple matter. And so the various teams worked hard to make sure the software was as solid as it could be.

The Executive

Deep Space 1 being assembled (Credit: Nasa)

Adam: Ron’s portion was called the executive.

Ron: So it was the software that decided, okay, this is what we’re going to do next in response to the input data and the possibility of various contingencies. It was the thing that decided what the spacecraft was going to do moment to moment. And it was programmed using another one of these custom languages that was written in LISP. And funnily enough, I can’t remember the name of the language now, which is really weird because that was like my baby. Why can’t I not remember its name? And this language was designed in such a way that at like LISP, the structure of language would prevent you from writing code that had certain kinds of issues. And the biggest issue that it was designed to prevent was the kinds of problems that you get when you have multi-threaded code, in other words, code that does more than one thing at the same time. That kind of code is notoriously difficult to manage.

There are all kinds of subtle problems that can arise that are sneaky and un-intuitive and can lead to problems like race conditions and deadlocks. And so this language was designed so that the very structure of the language prevented you from encountering these kinds of difficulties. And the code was even subjected to a formal analysis that we had a formal proof that the safety in variance that this language was designed to maintain were actually maintained.

Adam: Oh wow.

Ron: So we had a formal proof and of course, we had extensive ground testing. It was tested for days and days and days in simulation. And when I say simulation, this was running on a copy of the flight hardware and the only thing that was being simulated was that we weren’t actually in space, but all of the hardware was the same hardware or copy of the hardware that was actually going to fly. And so we had a formal proof of correctness and we had extensive ground testing with no problems.

So we were very confident that it was going to work and it didn’t work.

Deep Space Failure

Deep Space Network Dishes in Canberra, Australia (Credit Nasa)

This was like the second day of the three day experiment. So people were sleeping during this time, but not very much. And my memory of the details here are fairly fuzzy, but it was an all hands on deck situation because the scenario had been designed with safeguards in place so that the odds of actually losing the spacecraft were very low, even if the Remote Agents screwed up.

But the fact that it screwed up was still orange alert, if not red alert.

Adam: It stopped making decisions or it just freezes?

Ron: There was a time at which it was supposed to do something and that time came and went and it didn’t do the thing it was supposed to do. And alarm bells rang and-

Adam: Where was this?

Ron: It was out in space. It was 150 million miles away. It was an hour round trip lifetime.

Debugging Code in Space

Adam: Whoa. So now it’s time to debug a production incident, but this isn’t code running on a server in the cloud, this code is one light hour round trip time away. This is code running in deep space and to fix the problem, there’s Ron who wrote the programming language, and then there’s this guy that he hired to help him do the spacecraft code.

Ron: So I hired the guy in part so that I could push the coding responsibilities off onto him and kind of back away from the process because I could tell that I wasn’t really being very effective. And I was young and foolish and a very, very bad manager at that point. And I did not do a very good job of handing this task off to him. I basically just said, here’s what needs to be done. Go do it.

Adam: And he did do it. He got things built, but now this code that’s been proven deadlock free seems to be frozen 150 million miles from home. And so it’s time for Ron to jump back in and take control.

Ron: This guy was frustrated with me. He’d really been in the trenches for months doing my dirty work for me. That was all hands on deck and so I was swooping back in as a white knight to try and fix the situation again the way I had so skillfully done so many times before. The guy told me to go F myself, which came as a surprise to me because I didn’t realize that things had reached that level of tension. And so I think in retrospect, his reaction was very understandable, but at the time it took me completely by surprise.

Adam: Emotions are running high and it’s time to debug this issue. The more time passes, the more off target Deep Space 1 is.

Sending S-Expressions

Ron: We had no idea what was going on. All we knew was that this telemetry that it was supposed to send back hadn’t been received and now we had to decide how to proceed. And everything that we did when we decided to do something, we would do it and then we would sit around and wait an hour for the result.

Adam: So by the time you knew that it didn’t make that turn, that was already an hour ago.

Ron: Well, it was a half an hour ago because it’s a half hour out and a half hour back. Things get really weird on how you reckon these things, right? So yeah, effectively, it was an hour ago.

Adam: How does this work? How do you communicate with it?

Ron: That’s a good question. So you’re familiar with Python, right?

Adam: I am. Yeah.

Ron: So when you’re programming a Python, you have a REPL, a read–eval–print loop. So you can program Python interactively in addition to just putting code in a file and running it like a regular program. Well, LISP pioneered that. So in this, the normal way of interacting with LISP was through a REPL and this was the first language to have that feature. And it was the only language to have that feature for decades. And that’s one of the reasons why it was such a big lever back in the early days, but this was REPL so we had a REPL running on the spacecraft and we could interact with the spacecraft through that REPL. Now accessing that REPL was not just a matter of sitting down at a terminal typing at it because to communicate with that REPL, you had to go through the Deep Space network and deal with this hour long round trip light time.

So what the process actually looked like is we’d sit around in a conference room and scratch our heads and argue and try to decide what commands to send it, and finally come to a consensus on what batch of commands that we should send. And those then went through a review process that consisted of a number of layers of management, all of whom had to sign off on it. And once they signed off, it was handed off to an actual operator who was trained in interacting with the Deep Space network. And they would sit down at the console that was connected to the Deep Space network and they would type in these commands and then push the big red button to send those commands up to the spacecraft. So it goes from the console in a building at JPL through a dedicated hardwired network to one of the Deep Space network antennas. And there are three of these around the world so that you have coverage of the entire sky.

And these are huge 70 meter antennas, just enormous pieces of infrastructure. And I don’t actually know the details of what happens at the station. I don’t know whether there’s another human in the loop at that point or whether it’s all automated, but eventually the signal goes out of this huge 70 meter antenna and goes flying through space at the speed of light where it’s received by the antenna on the spacecraft and goes through this very elaborate system on the spacecraft where the signal is decoded and turned into bits. And eventually those bits get fed into the LISP system in exactly the same as it would’ve been if you’d been sitting there physically with the terminal typing at it. And it was a very elaborate and formal process that involved people wearing these headsets that you see like in the movies, going through this very formal process, that we are now going to send this command to the spacecraft and somebody else saying, yes, you are go to send the command to the spacecraft.

And then finally the guy who was sitting at the console would push the big red button and then we’d sit around and wait for the results. And then we’d look at the results and do the whole thing over again.

Adam: That’s crazy. You’re sending like a big tree of a LISP expression or what…

Ron: Yeah. We were sending up S-Expressions.

Adam: What do you tell it? Are you telling it to reload? It’s actively running. What do you do?

Ron: So the first thing that we told it was send us a back trace. Send us a dump of the system state, of all the processes that are running. The processes that are waiting for something to happen, what are they waiting for? So that was the first thing we did. That information came back and we looked at it and then it was actually almost immediately obvious what was going wrong because there was this one process that was waiting for something that should have already happened. And so that was a big clue as to what was going wrong. What turned out that the problem was that there was, in fact, a race condition, which was supposed to have been impossible.

The Proof and the Assumptions

Adam: It was proved impossible, right?

Ron: It was proved impossible, but the proof relied on an assumption. And the assumption was that when you were programming this thing, you were only using the constructs in the language. So the proof said, if you use the constructs in this specially design language, then you will not get deadlocks. So it’s kind of like nowadays you have rust where you have a safe part of the language and an unsafe part of the language. Well, we basically had that too, but we didn’t take that explicit. But what we hadn’t anticipated was that there was a guy who had been tasked to write some of the code-

Adam: This is the F off guy that Ron hired to build the software using his language.

Ron: … and he had to make it do a particular thing and he couldn’t figure out how to make it do that using the constructs in the language so he just called a lower level construct in LISP in order to get it to work. And that did an end run around the safety guarantees of the language. And this was a failure on my part. I had not made this sufficiently clear to the guy that I hired to write this code who was working under my supervision. Basically just said, here’s what needs to be done. Go do it. And he did, but because I hadn’t impressed on him that it… In my own defense, I’ll say it was not really in the forefront of my mind that it needed to be said. I thought this guy would just figure it out. And that was obviously in retrospect an unrealistic expectation. I didn’t make it sufficiently clear that he wasn’t supposed to do this. And so he did it, not his fault, my fault, and that’s what was the cause of the bug.

Fixing It

Adam: Do you kill something or restart something or?

Ron: So then we had to decide whether to try to salvage this session by injecting some event to un-wedge it, which we actually did. So we actually managed to un-wedge it and have it continue through the scenario.

Adam: So manually issuing the event worked. Thanks to the magic of LISP and the mind blowing idea of having a live REPL on a spacecraft, the mission was saved. And also, I feel like you have to think of the odds here. This code was tested extensively and it never hit a deadlock. If you ran this mission a hundred times, it may have only deadlocked that first time and never again. Deadlocks are artifacts of timing and obviously in test scenarios, the exact timing that exposed this deadlock never happened. The odds that this failure happened are really low, but the mission only happened once and it did deadlock. So for people outside the team, it certainly doesn’t look like a success.

The Impact of the Deadlock

Ron: We didn’t lose the spacecraft and we did accomplish all of the mission objectives. So technically it was a success, but the development process was so painful and fraught with difficulty and again, there were politics. So despite the fact that we actually did manage to get it to work, the autonomy project was canceled after that and it never flew again.

Adam: This whole process, as you can imagine, was very frustrating for Ron.

Ron: Frustrated is understating it quite a bit. It was extremely stressful and ultimately it led to me quitting. So I got more and more marginalized in my role and it became clearer and clearer to me that fewer and fewer people wanted to work with me and there was more and more pressure to collaborate.

Enter Google

Adam: Around this time, while Ron was at a record low, he discovered this new search engine called Google.

Ron: I was reading a Usenet newsgroup, in fact, comp line LISP. And somebody answered some obscure technical question, which I don’t remember what the question was anymore, but they, gave this answer and then followed up saying, thank God for Google. And I was like, what the heck is Google? And so I did what one did in those days when one encountered something that was unfamiliar, was pull up my Netscape navigator and type in . And sure enough, it was a search engine kind of like AltaVista, except that after just five minutes of noodling around with it was, it was obvious to me that this was light years ahead of anything else that existed at the time. And it was so good and so fast that my jaw was just on the floor saying, how the heck did they do that? And at the bottom of the page, there was this link saying we're hiring. So in a fit of what Alan Greenspan would call irrational exuberance, I dashed off a resume and 15 minutes later, my phone rang.

Is LISP Worth It?

Adam: So Ron left the JPL and started working at this tiny company called Google. But since this is CoRecursive and not the space exploration podcast, I wanted to find out now that it’s years and years later, is Ron still a LISP evangelist?

Ron: One of the things that I learned is that LISP has an advantage for me because it happens to be a good impedance match to my mindset. One of the reasons that it has good impedance match to my mindset is because I started using it when I was in high school. And so I’ve just been steeped in LISPness for 40 years or more. But for somebody else, it’s really weird. All those parenthesis and the weird prefix notation, that can seem very foreign and especially in common LISP, that has a lot of historical baggage and a lot of weird corner cases and potential nasty surprises. And so productivity really has a lot more to do with matching your tools to your mindset than finding the one true tool because there is not a one true tool because different people have different mindsets and different approaches and different things that they resonate with.

The one thing that list does give you is a better understanding of how things work at their core. And with that understanding, it becomes easier to learn other things. But in terms of actually getting things done, it has a lot more to do with how well the tool fits your particular mindset than the actual merits of the tool.

Outro

Adam: A big thank you to Ron Garrett for being on the show. A link to his website is in the show notes. And if you want to hear more about Ron’s time at Google during the dot-com boom, then you’re in luck. I’ve released it as a bonus episode. So if you go to the supporters link in the show notes at corecursive.com/supporters, you can access the episode right now. Will this little startup, Google, make it through the dot-com crash of the 2000s. Will Ron convince them about the beauty of LISP? You don’t have to wait to find out. It’s right now available for supporters. And until next time, thank you so much for listening.

Bonus Content From Ron

Ron: This photos were taken at an event in Washington DC. I don’t recall the details but I think it was done in conjunction with the National Air and Space Museum. It was a Big Deal. Transporting all that equipment across the country, especially Robbie, was a non-trivial undertaking.

"That’s me on the right standing in front of Robbie. Rocky III can be see in the lower left, and above him are Rajiv Desai and Robert Ivlev, two other members of the team. " - Ron (Courtesy of Ron Garret)
"This is me and Rajiv Desai at that same event. The orange-colored soil is a sandbox set up to look like Mars." - Ron (Courtesy of Ron Garret)
"Rocky III in the sandbox at the same event." - Ron (Courtesy of Ron Garret)
"This is me showing off Rocky III to the crowd. " - Ron (Courtesy of Ron Garret)
"Rocky IV on a mock-up landing vehicle with an engineering model Surveyor lunar lander in the background (so actual space hardware, not a model). This is at a high-profile demonstration event in the arroyo. " - Ron (Courtesy of Ron Garret)
Support CoRecursive

Hello,
I make CoRecursive because I love it when someone shares the details behind some project, some bug, or some incident with me.

No other podcast was telling stories quite like I wanted to hear.

Right now this is all done by just me and I love doing it, but it's also exhausting.

Recommending the show to others and contributing to this patreon are the biggest things you can do to help out.

Whatever you can do to help, I truly appreciate it!

Thanks! Adam Gordon Bell

Audio Player
back 15
forward 60s
00:00
00:00
38:06

LISP in Space