Intro
Adam: Hi, this is CoRecursive, and I’m Adam Gordon Bell. Today, we’re taking a look into the world of software design and architecture.
Do you remember the swine flu, H1N1? Back during that time, around 2009, I was working at a software company. I was sitting at my desk enclosed by kind of these blue pastel cubicle walls in a large gymnasium-like open office when one of the more senior devs, Cedric, comes over to the area and he has kind of a mischievous smile, which isn’t good, and he says, “I need to show you something.” He asked me to pull up a branch in TFS and find a specific VBScript file, VBScript being Microsoft’s answer to PHP. “Go to line 307,” he says, and, you know, on line 307, it says something like “TODO: Figure out the logic for this part.”
It’s not even a code comment. It’s just plain text right on the page. And it broke the page. The page wouldn’t render. And yeah, that was my handiwork. And it was in a branch that we were going to release. Cedric thought it was kind of funny.
You see, mistakes like that happened more than they should have back then, and that was such a minor one that it was humorous. This was the chaotic world we lived in. We had no code reviews. We were all moving super fast. We had a shared branch that we would use per release, everybody pushing code to it and then testing it and releasing it.
It was a digital Wild West where we moved fast, but it was a wonder that anything worked at all.
Later, I joined a company with a code review process, and man, that was a wake-up call. I hated it at first. “I know what I’m doing, just let me do it.” But after a little bit, I saw the value.
The worst code issues are never these small comments or little bugs, right? They’re the features that are slightly misdesigned that you end up stuck with for years. And if you can catch some of those in code review, if there’s a forcing function for people to discuss how it’s working, tricky conversations can happen.
You can discuss the trade-offs. And the software improves, and the team does as well, hopefully.
That process is kind of what today’s about.
Greg Wilson
Greg: All right, one last test. Voice is coming through.
Adam: Yeah, sounds good.
Greg: Welcome to WKFI. Gonna spin some old vinyl for you this evening, ladies and gentlemen.
Adam: That’s Greg Wilson, and he’s not a DJ. He’s an accomplished software developer and teacher. He’s taught scientists software engineering. He’s authored many influential books, one of them being Beautiful Code, a pioneering work born from his frustration with the field. It’s sort of what we’re talking about today because Greg’s been on a crusade to elevate software design from the mire of mindless feature building that I was doing back in the swine flu days to a form as respected and revered as architecture itself.
You see, once you start reviewing code, talking about potential solutions, and as soon as it’s beyond the level of linting and code formatting, you need a language for discussing approaches and giving us a language and a repertoire of software design has been one of Greg’s goals.
But before we dive into Greg’s approach to software design and software architecture, let’s rewind. Let’s go back to the 80s when Greg was an undergrad at Queen’s University and he didn’t want to be a programmer at all.
Soldering Iron Burn
Greg: I was doing electrical engineering as an undergrad for reasons we don’t need to go into.
And to be honest, I’m actually quite clumsy. I have a scar on my right hand from picking up a soldering iron by the hot end twice in one hour.
And the second time I had my hand under the cold water tap, the lab tech came over and put his arm around my shoulders and asked me if I’d considered switching into software.
And at the time, I thought he was doing me a favor. Looking back, I realize the next semester was the course on power transmission lines with 50,000-volt transformers and things like that in the lab. And I think he just didn’t want to have to clean up a mess that I would probably have made.
Computer Lab Flow
Adam: So Greg didn’t become a human lightbulb. He switched to programming and he fell for it pretty hard.
Greg: I was in the basement of Jeffrey Hall at Queen’s University. A cold, snowy day in Kingston, Ontario in 1982. I was sitting in front of a green on black text terminal using a dumbed-down version of VI, and I was writing Pascal to play three-dimensional tic-tac-toe.
Adam: This was a programming language’s course assignment. Greg was still in engineering but he was starting to transition to software.
Greg: And it was my first encounter with recursion and with searching trees and backtracking, and all of that. And there’s that moment when you can feel the pieces clicking together. You’ve been looking at all this stuff scattered on the table and you realize that this goes here and this fits here and this other thing is going to tuck in over here and all of a sudden the jigsaw is starting to take shape and you can see what the picture is going to be and every next piece you can see where it belongs.
Because I’d gotten so deep into solving the problem that I’d completely lost track of the fact that it’s now 3:30 or 4 o’clock in the afternoon. I haven’t eaten for eight hours. I’m getting a headache, but it was the first time I’d lost myself in solving a problem, and it was a rush. I won’t say bewildered, but I realized that like this was one of the few moments in my life when I realized I’d had an experience that was going to change me.
This was something I’d never experienced before. I didn’t decide there and then that I was going to be a programmer, but the moment has stuck with me.
Adam: What was that feeling like?
Greg: Honestly, years later, the first time I went scuba diving in warm water down in the Caribbean, I had that same feeling. And I’ve had it sometimes when you’re out on a bike. Like, you get on a hill and as long as you can keep your balance, you can go as fast as you want because gravity is doing all of the work, but you’ve got to do the steering.
You’ve got to keep your balance. You’ve got to think about whether you can make this corner, or whether there’s going to be sand on the road and you’re going to skid out. But it’s such a rush. I’ve had it occasionally playing music. I used to play tenor sax. And there will be times when you’re on the beat, and you’re with the band, and everything is just clicking together.
And I, you know, people talk about flow. But there really is something special that happens there sometimes.
HPC
Adam: I know this feeling. Probably you do too. Sometimes I feel like my life’s just been about chasing it. It was the same for Greg. He was hooked. He finished his undergrad, and then he went off to Scotland to do a master’s and a PhD in computer science, where he ended up working at a high-performance computer lab at the University of Edinburgh.
Greg: Lots of physicists building lots of code that I never really understood. Doing very complicated quantum things, but at the beginning and end, somebody’s got to read all the files in and write all the files out. Somebody’s got to checkpoint the computation in case the machine falls over. These days, that kind of work is called research software engineering, or RSE.
You know, the analogy is the astronomer who builds telescopes but doesn’t look through them. You need to understand the problem. But you’re not the one who’s actually doing the science. You’re doing the support for the science. So I spent six years doing that in what is now the Edinburgh Parallel Computing Centre.
Not Growing As A Dev
Adam: Greg got a PhD, and presumably he learned a lot of computer science, but really the thing Greg loved was falling into a programming project. That was the thing, right? But it wasn’t clear how to get better at building software.
Greg: There were Usenet news groups, but there wasn’t anything like Stack Overflow. There wasn’t anything like today’s online community. Most people who were learning programming, it was either through direct contact, or you’d go and you’d buy a book and hope that it would teach you more than just the syntax of the programming language.
I picked up bits and pieces, but by the time I left in 1992, I knew enough to know that there had to be a better way. I felt like I was banging my head against a wall or banging a wall against my head some days,
Everything felt like it was taking a lot longer than it should. And it was a lot harder than it should be. And that I was doing the same thing over and over again and not getting better.
Adam: Greg knew this feeling, because he had been here before as a saxophone player.
Greg: I’d go months where I could tell that I wasn’t any better than I had been three months ago, six months ago, that for whatever reason I was stuck, I’d plateaued. And years later, I learned some of the science behind learning and teaching. It’s hard for people to break out of that on their own because they don’t know what they don’t know.
There’s this story from, I don’t know, decades and decades ago. A man gets shipwrecked on an island and plays chess against himself for 10 years. And at the end of it, he’s the world’s best chess player because he’s had nothing else to do. And it’s false. It doesn’t work that way.
You need other people to knock you out of your ruts, to show you things that you yourself would never have invented or discovered. And there’s a handful of people who can bootstrap themselves in isolation. But it is such an extraordinarily rare thing. That almost all the time, you’ll advance some way on your own, and then you’ll hit this point where you need somebody to say, “Look over there, try this. Here’s another perspective on it,” and I didn’t really have anybody in Edinburgh who was doing that for me.”
Unix Trilogy
Adam: But one day, something did happen, and it propelled him to new levels of skill.
Greg: I guess it would have been 1986 or 1987 somebody handed me a copy of the book, The Unix Programming Environment, right? One of that trilogy that Brian Kernighan co-authored with other people at Bell Labs.
And it’s not just about how to use Unix. In fact, that’s almost incidental. What it’s about is how to think about solving programming problems by putting together pipelines of simple tools, each of which does one job. And I’d been using Unix for six or seven years at this point, but I’d never understood that there was a philosophy behind it, that there was a coherent design. Everything is a file, even devices or files, even processes or files, they can all be connected like this.
There were half a dozen ideas that just fit together and then you could do almost anything with them. There aren’t nearly as many examples of that kind of conscious design thinking in computing as we would like to believe.
I actually believe very strongly that the major reason for the success of Unix was those books. Because after you’d read the C programming language, the Unix programming environment, elements of programming style, you couldn’t help but see the world the way they saw it. Right? It’s like calculus. Once you’ve seen it, every problem in physics is a calculus problem.
You can’t unsee that way of looking at the world.
IBM Company Culture
Adam: So Greg wanted more breakthroughs like that. He wanted to get better at his craft. Not just be a good research engineer, or a good computer scientist, but to be a great software developer. And so he headed back to Canada, and he got a job at IBM. And IBM was a big deal in the world of software.
Greg: I was lucky enough to spend a year working under a guy named Bill O’Farrell. And so this would be ‘95, ‘96. C++ is coming on strong. And he’d been doing object-oriented programming since before most people knew the term. He understood not just how to write C++ but how to think about breaking your problem up into pieces that could be represented by a class hierarchy where the instances of that hierarchy are interacting in predictable ways.
Adam: Do you have an example?
Greg: Oh, sure. So we were working on a library for handling different kinds of concurrency at the threading level, at the multiprocess level. And he came up with a really elegant way to use multiple inheritance, so that any particular actor in the system was inheriting some behavior from one parent and some from another.
And when you compose them, they would fit together and give you something that would actually run. And thinking about the two, or in fact, three class hierarchies that you build so that any actual object is inheriting from three different parents. And one of them is giving it methods A, B, and C. One of it’s giving it methods P, Q, and R.
But the P, Q, R methods are using the A, B, C methods and vice versa. And they fit together, so there’s a contract between these. It’s not a composition relationship. It’s actually all married together in one class. It was elegant.
Adam: But IBM was struggling. Making the job a bit tough. So Greg joined a startup. Working on an early single sign-on tool.
90s Start-Up
Greg: It was a group that had worked together in a couple of previous companies. That’s where I actually learned most of what I know about software engineering of a building product, because I was finally with people who knew how to play as a team, play as an orchestra, whatever analogy you want to use.
There was a backlog and it was regularly groomed. It was the first time I’d ever met a product manager. And this guy was going through and talking to users and figuring out how to do the sort of 90-degree pivot from what are the users complaining about to what features do we need to build?
And it was the first time I saw somebody who had a systematic process for doing that for identifying gaps in the market. What is worth building? What might sell? Coordinating a team that when I joined it was 10 people grew to the high 30s, by the time the dot com crash came. and,
As with those months that I spent working with Bill O’Farrell, it was like, “Oh, somebody else has thought this through. Somebody knows how to do this and I can learn from them.” And I think a lot of this can’t be taught through books, because every learner’s questions are going to be different.
I could talk at you for hours about how to ride a bicycle. It wouldn’t make any difference to your ability to ride a bicycle. There are some things you have to do in order to learn.
Because there’s so many special cases. There’s so many times where the right answer is, “It depends,” but that doesn’t scale. And it’s gatekeeping. If you’re growing up in a small town in Western Canada, when are you going to bump into people like this?
Adam: Greg is a teacher at heart, and for a certain type of person, when there’s important knowledge that’s hard to teach, that looks a lot like an opportunity.
The Teaching Idea
Greg: I got interested in whether it would be possible to teach the things that I’d learned systematically and at scale, which meant in a classroom. Early 2000s, University of Toronto asked me if I’d be willing to teach a course on software architecture.
Adam: Definitionally, software architecture describes a piece of software’s structure and its behavior, its design, its tradeoffs, and how those things might affect maintenance or scalability or whatever.
These are the type of things he had learned slowly at IBM and at the startup. These are the type of things that Greg wanted students to learn. He didn’t want students to have to get just the perfect job where these skills would be presented via osmosis. He wanted to give people a path to learn these things.
Greg: What I hoped I would accomplish was giving 20-year-olds the lessons that I hadn’t had as a 20-year-old. I was trying to figure out some of the stuff on my own. And what’s worse, when I was 20, I wasn’t even trying to figure it out because I didn’t know there was a thing to figure out. I didn’t know software design was, was even a thing.
But the idea that there’s some sort of higher-level thinking that can go into what are the pieces, how do they fit together? I’d never seen that. It wasn’t something that was taught in classes, at least not the classes I was doing. And wind forward 20 years to the early 2000s.
Now I understood that this is a thing, and I’d done some of it and seen some of it. Okay. Can we pass that on? Can we teach that explicitly so that newcomers aren’t in the same situation that I had been in at their age?
Adam: So Greg started with a literature search. And then he collected textbooks that other universities were using for software architecture courses.
Greg: I had exactly 12 books with the word software architecture in their title sitting on my shelf. Let’s say that each of those books is 300 pages long.
So that’s on the order of three or four thousand pages. The thing I did count was how many of them actually described the designs of actual software systems. There was less than 30 pages, less than 1 percent of those pages describing real systems.
Adam: That’s wild. What were they, what were they doing?
Greg: Well, there was a lot of UML and design patterns,
Adam: Yeah, that makes sense.
Greg: But it makes sense if the books are being written by non-practitioners, by people who have jumped straight to the theory but don’t yet have a repertoire to base that theory on. And there’s a lot of that, right? One of the reasons the original work on design patterns was so impactful was that Erich Gamma and others had spent years building real software systems.
So they’re drawing on things that they’ve done out in the real world for their patterns. One of the few good books after that on design patterns was called “Design Patterns in Ruby.” But every single line of code in that book was taken from the Ruby standard library.
He only described patterns by saying, not only is this real code, but it’s real code that I didn’t write that’s in the standard library. And if I can’t find an example in the standard library, this isn’t a pattern you need to learn.
Adam: This is what Greg was looking to teach, right? Look at the great works of those who’ve built real software. See the decisions they’ve made, the trade-offs they embraced. But he wasn’t finding that. And so he broadened his search. Meanwhile, he’s teaching programming at the university and he’s noticing some gaps there too.
The students are missing some key practical skills.
Practical Skills & Lab Work
Greg: I have seen students who are getting A’s in senior undergrad courses spend hours, well into the wee hours of the morning, trying to debug something when other people in the same class with lower grades were solving the same problem in 15 minutes because they knew how to use the debugger in VS Code.
There’s very little correlation between grade and tool proficiency. I think there are people who have gotten through undergrad by just working harder with primitive tools.
There are other people who, because they’re sitting beside the right person, whatever it is, have stumbled over a better tool. And as a result, they’re able to get more done in less time.
Adam: This is another gap, right? Many students, successful students, just don’t have the skills they need to be successful software developers. Greg wants to bridge this gap.
But even though Greg was ostensibly teaching software engineering, right?
Practical skills. The university wasn’t interested that much in practical skills. This was supposed to be the science of computing.
Yet, in some fields, the approach is different. In some fields, practical skills are key.
Greg: If you do geology or mechanical engineering, of course, there are lab exams. You have to prove that you know how to use the lab equipment.
You have to be able to prepare cell cultures to get a degree in biology. You’re just expected to be proficient with the tools so that you can do the science.
Can we screen record students solving particular problems? And they submit the screen recording and the TA watches it at 10x and gives them a grade on how proficient they are with the tool.
Okay, so here’s another thing that’s missing from undergrad curriculum. I don’t know any university anywhere that has an undergrad course on debugging.
And yet, professional programmers spend 40 to 60 percent of their time debugging, and novices spend 90 or more percent of their time debugging.
I write five lines, and now I have to figure out why it’s not working. And we never teach that, which is nuts.
And again, I don’t know how to teach debugging in a book because it’s all about the how, not the what.
It’s riding a bicycle. It’s a process, rather than a thing like the syntax of Python.
Adam: Greg did try to address this. He found someone who could teach debugging.
Greg: Mike Conley, who works at Mozilla on Firefox, does a webcast every Wednesday called “The Joy of Code,” in which he picks a bug or an open issue in Firefox that he hasn’t seen before and just dives in. Right. And just live casts as he tries to figure out, is this actually a problem? What is the actual problem?
Why did this ever work in the first place? How do I track it down to its roots? Like he’s showing people how you do what I spend most of my programming life doing, which is not writing code. It’s spelunking. It’s cave diving, right? It’s climbing without the safety harness, and then going, “Oh, why did this ever work in the first place?”
We don’t teach this.
Adam: But Greg could teach this. And so he brought in Mike for live code reviews of the Mozilla code base in class. I assume many students wouldn’t catch every detail. I certainly wouldn’t. But they’d see how to think through a code change.
What is the thought process?
But still, Greg needed a great curriculum for this software architecture course. So he searched far and wide. He wanted students to see beyond the code, to understand the design, the big picture. He needed to teach them design.
Looking To Architecture
Greg: I was reading a lot at the time trying to figure this out, and there are disciplines that have a long history of teaching design. And this is one of the things that frustrated me.
In your first year in an architecture program at any university I’ve ever heard of, you will look at literally hundreds of buildings. Right? You’ll look at photographs, you will look at floor plans, you will look at sketches, you will read notes about the design, you will learn about the history of how those designs have evolved.
I don’t know of any English language university that offers a course in which you read the great programs of history and take them apart and figure out why were they built this way and what is this way in the first place.
Adam: What do you learn from? Like, if I show you a Frank Lloyd Wright house, what do you get out of it?
Greg: The first thing you learn is what a musician would call a bag of tricks, right?
Most musicians have a bunch of licks or riffs, patterns that they will play, and hundreds or even thousands of these little things. So they’re not thinking in terms of notes or even in terms of chords, they’re thinking in terms of passages, they’ve got larger pieces that they can assemble in the same way that you as a programmer
would look at a data structure and say, “Oh, that’s a doubly nested loop.” So you’re thinking at a higher level than somebody who’s encountering loops for the first time and having to step through atom by atom. You’re thinking in terms of molecules or organs or entire organisms.
So you can search the space of possible designs much faster and much more efficiently. Because you’re searching in bigger steps, you’re seeing a bigger picture. And the same is true in architecture. People who have not just looked at enough buildings, but analyzed enough buildings, have their attention drawn to particular features.
When the time comes for them to design a building, they’ll say, okay, well, we’re going to want an atrium. They’re not going to start by saying, well, we need to have a door with some glass in it and an open space where we can put a reception desk and like they’re not thinking it through from first principles.
They’ve got a much larger set of pieces that they can pick from and assemble so they can do their design far faster and far more efficiently. Hmm, a repertoire, yeah, a repertoire. There you go, right? The other benefit of it. There’s a thing called a studio class in architecture. There’s, there’s a dozen of us, right?
And the assignment is design a bus shelter. Here’s a ground plan. Here’s where the roads are. Here’s a couple of constraints, like how much you’re allowed to spend or the weather conditions or things like that. Design a bus shelter. So we all design bus shelters and the prof goes eeny, meeny, miny, moe and picks you.
And you go up to the front and you spend a few minutes presenting your design for the bus shelter. The other students in the class then critique your design. Now it’s not the same as criticize, but they give you feedback. Why did you decide it this way? Have you thought about this alternative? What about this? What about that? Right?
The prof then gives all of those students feedback on their critiques. He doesn’t talk about your bus shelter. He’s trying to teach the rest of the class how to do the equivalent of a code review. How do you give people feedback? What do you look for? I don’t know any English language university that explicitly teaches students how to do code reviews.
Adam: It turns out there is research into teaching students how to critique code, though it’s not phrased as MFA style group critique instead, it’s framed as a way to outsource grading.
Greg: If you’ve got an undergrad intro to programming class with 2000 students in it spread across 10 sections, you’re going to have a squad of grad students who are marking those assignments. If you give any particular assignment to two different grad students, they’re not always going to agree on the grade.
But you hope they’re going to be pretty close. And you can actually measure how close they are statistically. It’s called inter-rater agreement. How close do you and I agree on whether this is an 8 or a 9?
So the question is, can you get the first year students to grade each other’s assignments, as well as the grad level TAs are grading those assignments.
Adam: It turns out you can, right? You give them a grading scheme, points off for poor variable names, points off for confusing structure and so on.
Greg: The grade the undergrad gets is how closely their review matches the one that the prof did of that code.
So it turns out that after three iterations of this, each of which takes about half an hour, the undergrads can grade one another’s Intro Python assignments, as well as the TAs were grading them. Right? So you can teach people how to do reviews.
Adam: To Greg, it became somewhat obvious that this type of reading comprehension of programs was a great place to start. That’s how we develop a taste for good software. Reading code matters.
Greg: You can teach musicians how to listen to music. Right? That’s a large part of an undergrad music degree, is what should you be hearing in this song? What are the interesting parts? How is this put together? So that when the time comes for you to do it, you’ll have a larger repertoire of ideas that you can draw on. Graphic designers do this all the time, anybody who’s designing a UI is drawing on this enormous vocabulary of specialized terms and references and ideas. They’re not just picking colors and fonts.
Beautiful Code
Adam: The world of graphic design, it’s a fascinating comparison. Like architecture, it’s all about feedback and criticism and refinement. And they’ve got this language, they have a way to talk about what looks good. They can point to examples. They can see what works and what doesn’t. What’s sleek, what’s awkward. That’s what Greg wanted for software design. He was after a library of examples to pull from, and he was frustrated that it didn’t exist.
Greg: And it annoyed me so much that in the spring of 2006, I hunted around on the internet. I got email addresses for 300 people and I sent mail to them one at a time. I spent a whole weekend, a long weekend actually, firing off emails. Hi, my name is Greg Wilson and I’m putting together a book for O’Reilly called Beautiful Code. Now I hadn’t told O’Reilly this yet, but I was pretty sure that we could connect the dots, right? What I want from you is a chapter. Pick the most beautiful piece of software you’ve ever seen, and just write me a few pages about what makes it beautiful.
Adam: What Greg needed was examples to put in front of students to build up their taste. And to him, emailing practitioners was a great way to avoid a very specific trap because he needed diverse and real world examples.
Greg: I didn’t want to go out and pick them myself. I wanted people who had built real code to come back and show me things that I would never have stumbled over by myself.
Adam: The list of authors in Beautiful Code reads like an early 2000s software design hall of fame. Brian Cantrell on the Solaris kernel, Carl Fogel on Subversion, Michael Feathers on building a testing framework and so on and on for 33 chapters.
Greg: Another contributor wrote about a set of Fortran libraries called BLAS, the Basic Linear Algebra Subprograms. And his argument was, this code is now almost 30 years old and is in daily use and is the workhorse underneath almost every piece of scientific software on the planet. As far as he was concerned, something is beautiful if it’s been so useful for so long. The beauty comes from the utility, and this has proven its beauty by being useful.
Adam: Yeah, the farmer who loves his old tractor that has just been going and going.
Greg: Classic Jeeps from the 1940s and 1950s that will not die. The machines back then, man, they were ugly and they were loud and they polluted. But, I actually watched people put chains and haul a Jeep out of a river where it had been sitting for a couple of years, take it into the shop, and a week later it’s back on the road. We had people writing about just a whole variety of things, and it was a great book. I mean, it was a really enjoyable book. We raised almost 100,000 for Amnesty International because rather than divvying up author royalties 36 ways, we just… Send the check to Amnesty. But it wasn’t any use for my intended audience. Undergrad students couldn’t make sense of it because there were too many different programming languages, too many different levels, too many different perspectives. You had to have already been working in the field and have seen these issues to appreciate it.
Adam: That’s where software architecture and talking about beauty seems the most challenging, right? A beautiful building, you can go over all the details of why it was built that way, what the constraints were, but also you can see it all at once. A software design needs thought. It’s like a dense poem. You need to mull it over. You need context to see that beauty. So anyhow, by then, Greg was in his third semester of teaching this architecture course, and he sort of thought this gap. It was just too large.
Greg: And I told the department to cancel the course. This would have been 2003, it was pretty clear after three and a half years that academia wasn’t for me.
A Hunger for More
Adam: But all was not lost. This book became a hit, and even me, the “no code review” me from back in 2009, even though there wasn’t much design talk at work, I did have a copy of this book, and I did read most of it. And so did many others.
Greg: I do remember being completely convinced by this exercise that there was a hunger for this kind of thing.
That,
I think, Scott Hanselman talks about dark matter developers.
Have you ever heard this phrase?
Adam: I think so. Just they’re, they’re not visible, I guess, or something.
Greg: Most of the matter in the universe is invisible to us. It doesn’t radiate as far as we can tell; all it does is interact with gravity. We can tell it’s there, but we can’t see it. And Hanselman’s point 20 years ago is that 90 plus percent of programmers never post a question on Stack Overflow, much less answer one.
They don’t have public GitHub repos. They don’t have personal websites. The few percent that are radiating information are atypical.
But they’re the only ones we can see. So Beautiful Code convinced me that there was a hunger for learning about software design among dark matter developers.
It wasn’t just the few who show up to conferences and have blogs.
Teaching Research Scientists
Adam: So although his approach to teaching via real-world examples didn’t take off in academia, it seems to have been soaked up by real-world software engineers. And maybe that’s been Greg’s biggest impact on the world. And then he created more books, books that followed it. Because he kept going with this idea.
And we’ll get to that shortly.
But actually, there’s another side to Greg’s journey, another adventure in teaching. And in some ways, it’s where his impact on coding has been the largest. It started back when Greg was a research engineer, back when he was in Scotland. He was reading Brian Kernighan’s books, and he was thinking about that magical place in New Jersey.
He’s thinking about Bell Labs.
Greg: I think we sometimes mythologize a few places in moments: Bletchley Park, Bell Labs, Xerox PARC. But on the other hand, for the space of a few years, a few people working together can change the course of an entire industry, right?
They really did push us in a whole new direction. I don’t think they could possibly have realized at the time just how big an impact they were having, but man, most of the programming I’ve done over 43 years has basically been footnotes to what they did at Bell Labs in the 1970s.
Adam: Greg had had this Unix programming epiphany, and it frustrated him seeing smart scientists miss out on all these tools. A lot of research involves crunching data, and researchers were being held back by a skills issue. So he spread the word.
Greg: I corralled a bunch of people and we wrote a couple of journal articles basically saying, look, scientists are taking weeks or months to do things that they could do in hours or days. Because nobody’s shown them that there’s a better way.
There’s a tool to do what you’re doing by hand. You can write macros for editors, right? Learn how to do that. Regular expressions are useful when you’re dealing with messy scientific data files. Here’s how they work. None of this is rocket science to me, but it’s unfair to expect somebody who has spent years learning about quantum chemistry to somehow magically know about command line pattern matching tools.
Adam: Greg actually went to a chemistry professor and he told her, you know, the chemistry department needs to start teaching data processing. They need to start teaching computer programming and not just chemistry undergrads, but geology students and, and others as well.
Greg: And the professor I was talking to finally lost her temper and said, okay, so what do we take out?
Do we teach them less organic chemistry or less thermodynamics? And she was right. Every undergrad science and engineering program is bursting at the seams and they’re still only scratching the surface. You’ve got four years, you can’t teach people more than 1 percent of modern geology in four years.
And now I’m telling you that we’re going to take away 10 or 15 percent of that to teach them programming? I understand the pushback, and I think they should learn the programming. But I understand why people are saying, great, it’s not enough to say we should add X. You’ve got to tell me what to take out.
It’s like budgeting discussions in government. Everybody’s got something they want to spend money on. The hard part is what are we not going to do so that you can have your road or your school or your hospital or your tax cut? That’s where the rubber really hits the road.
Adam: So Greg refined his thoughts, maybe not undergrad classes, but what about programming boot camps for grad students? He jotted down his ideas and the IEEE published them.
Greg: And John Reynders, who was running the advanced computing lab at Los Alamos, reached out to me and said, put up or shut up. If you think that teaching this stuff will make a difference, I’ll give you a couple of chances to do it. You put together a workshop, you come here, you teach my physicists these things, and we’ll see if it makes a difference.
I got to tell you, you kind of feel like the elevator dropping underneath you a little too fast when somebody says, all right, all this griping you’ve been doing, show me.
So yeah, you know, you think you can make a better chili, there’s the kitchen.
Los Alamos
Adam: Los Alamos is no joke, right? This is the location of the Manhattan Project. And Greg’s basically told them, you know, they’d be better at their job of particle physics or weapons research or weather simulation if they knew some Python, if they knew some command line tools. But I mean, he sort of had a point.
And so he and Brent Gorda went down there to teach a workshop.
Greg: You got a classroom with a bunch of people who are physics, geology, chemistry, pure math. Some of them my dad’s age. Some of them quite skeptical because if you’ve worked in a large institution for any number of years, you’ve seen a lot of bandwagons roll through. And anytime your boss says, here’s this training course, your first reaction is going to be…
If I just sit here quietly, eventually this will just pass me by and I can get back to work. But there were a lot of people then who really did want to improve their skills so that they could get more done in less time and with less pain. They, a lot of them, they would have had somebody rooming with an undergrad who did computer science.
So you can see that they’re somehow able to do this stuff with a couple of lines on the keyboard that’s taking you forever to do in whatever WYSIWYG tool you’ve got or using Fortran and compiled code. So you’re open to the idea that better is possible. And that’s often the hardest thing with learning, is to convince people that improvement is actually possible.
Adam: The first thing Greg would do is teach the basics of Python. These researchers were used to working in Fortran. This was a big change. One memorable group of three have been coding together since the 1960s.
Greg: They had a code base that was over a million lines of Fortran, and they weren’t allowed to tell me what it simulated, but they could tell me that what it simulated was really, really loud. Right?
Adam: So Greg did his workshop at Los Alamos, and he did it a couple of times, and he refined it.
Greg: This was and is a struggle. If you’ve only got a little bit of time, what do you teach? What’s going to have the most impact on this audience? I eventually realized that what I was doing was the equivalent of teaching first aid.
My sister in law is a paramedic here in Ontario. And their approach to care is based on the same science as what you get from your GP if you walked into their office. But they’re coming at it from a very different angle. My sister in law will say that she does everything a doctor does except she does it in a ditch in the rain. And… She’s okay, if need be, with not being in sanitary conditions. Her job is to make sure you don’t die and keep you stable in the ride back to the hospital when you become somebody else’s problem. And what you teach somebody who’s going to be working under those conditions is very different from what you teach somebody who’s working in a well-funded hospital where they’ve got a supply closet right there with everything that’s ever been invented and they’ve got a reliable power supply and they’ve got three people who can step in if they need a spare pair of hands.
Figuring out which bits of what you and I do on a daily basis are worth teaching to a data scientist or to a physicist or to an economist is a computer science departments don’t think about because that’s not their audience.
Adam: A lot of this, like First Aid, was basics, but very crucial. How to use source control, how to use Python for quick data scripts, instead of Fortran. Turns out, if you’re a meteorologist doing weather simulations, some software engineering, some shell scripting, a little bit of Unix philosophy, can be quite empowering.
So the workshops were a hit. And soon other research institutions wandered in.
A Simulation Break Through
Greg: Not long after that, I taught one of these workshops on my own at NCAR, National Center for Atmospheric Research in Boulder, Colorado. And partway through the first afternoon, I just, as an aside, mentioned profilers.
And one of the people in the course came up to me at the break and said, what is this? And I explained that you can actually, with a couple of compiler flags, you can get the program to record where it’s spending its time so that you can see what you need to optimize. They’ve been banging their head against this problem, manually inserting timing calls and logging stuff for weeks, trying to get the data they needed to improve the performance of this really big piece of weather modeling code, and I had just told them the compiler has that as an option.
And so they just left my class and they came back about an hour later. And they’re like, you’ve just saved me a month.
Adam: By introducing these researchers to the power of profilers, he didn’t just save them a month of work, right? He gave them new skills. And skills compound across all of your future work.
Greg’s powerful insight was that software engineering skills could expedite research, potentially accelerating scientific discovery itself.
The ripple effect of this knowledge transfer is immeasurable, and it probably helped increase our understanding of the world. And it also helped, in a small way, I guess, to make Python a fixture of scientific computing.
But yeah, Greg eventually left this behind, right? He did the IBM thing, he did the startup thing. But this idea never left him, this idea of computing as first aid.
He sometimes thought of it like teaching people to build sawhorses.
Greg: I, I don’t know if you’ve ever done any serious carpentry, but you see a house builder show up. The first thing he does is he makes himself a couple of sawhorses, right?
Five pieces of wood, nail them together, five pieces of wood, nail them together. You got a couple of sawhorses. You’re probably going to leave them behind when you’re done the job, right?
There’s a lot of sawhorses in computing.
Adam: And then, after his startup time, and after his somewhat failed attempts to teach software architecture, he thought maybe he should revisit that. Because it had had a big impact, and he did like working with scientists.
Scientists & Shell Programming
Greg: It’s more fun to work with scientists than it is to work with many or even most of the people you will bump into in Silicon Valley.
Because scientists are motivated by the problem they’re trying to solve. You don’t become a chemist or a physicist unless you love chemistry or physics.
Adam: And so Greg took his workshop curriculum, he put it out there under an open source license, and he started a non-profit, Software Carpentry, and just started out on the road teaching grad students how to build sawhorses.
Greg: So many grad students had… Seen a bit of programming in first year, typically intro to Java course that had nothing to do with the kind of problems that they were going to tackle.
Maybe had pushed around a bit of SAS or SPSS in a stats course. And now here they are in grad school and there’s a terabyte of data sitting between them and their thesis. So teach them a bit about the Unix shell. So that they can start. Doing things with scripts instead of with GUI tools, so that they can automate those things.
Teach them a bit of Python and later a bit of R, so that they understand that they should break their code up into functions that they can reuse in the test. Teach them version control, teach them how to write tests. Fairly simple things, but life-changing for a lot of the participants.
Adam: Compared to his original goal of raising the level of discussions around software design and architecture, pushing the field forward, compared to all that, this knowledge he taught grad students was quite basic.
But a lot of times, like First Aid, the basics can be life-changing.
Greg: I was teaching a workshop, once, and teaching people the basics of the Unix shell,
And one of the things that we show is how to write a three-line shell script that’s basically a for loop for each file in this directory, process it the following way. And uh, one of the women in the class started to cry.
She was a grad student I found out at the coffee break when I, when I spoke to her about this, that every evening for about a month, she’d been coming into the lab and she’d go to the first machine, type in the command to process file number one, which would take about 20 minutes, then go to the next machine, type in the command to process file two, right?
Six machines, get them all running, sit there and knit. When they were finished, she’d do a second batch. When they were finished, she’d do a third, and then she’d go home. She had over a thousand files that she had to process, and she was treating it the way you would treat cell cultures or cell assays, is you just gotta put in the hours.
She wasn’t crying because she had just realized that she could have written a three-line shell script and leave it run over the weekend and it would be done. She was crying because… She had just realized from the comments that the two guys sitting beside her were making to each other that they had known for a month what she was doing, that they knew how to solve it, and that they hadn’t told her because they thought it was funny.
Software Carpentry
Adam: This was another part of the whole teaching thing, right? When you teach knowledge openly you knock down barriers. You knock down gates. No one can hoard knowledge out of spite or misogyny or whatever.
And Greg’s teaching was hands-on, no slides. Just him and his computer showing people how to do things. It’s really the kind of stuff you would have seen if you’d been sitting next to him back when he worked at the high-performance computing place. But now, it’s in a classroom, and he’s breaking it down step by step.
Greg: For most workshop participants, the single biggest thing they got out of the workshop was the belief that this was something they could do. For so many people, their earlier experiences of programming had been so off-putting, so confuseling and belittling, that they had just come away believing that whatever the gene is, I don’t have it.
And that’s why I teach. It’s so empowering for people to give them the tools so that they can go and build their own sawhorses, so they can put up their own drywall, so they don’t have to wait for a contractor to become available to fix a leaky faucet. We called it software carpentry because it isn’t software engineering. Most of us aren’t designing a hydroelectric dam, but we should all know how to do a few home repairs because it puts the power back in our hands. And teaching people how to design software means that they can finish that and then go back to whatever the original problem was.
And I don’t know who first said it, but like the only reliable way to be a 10x developer is to go and find 10 people and make them each twice as productive.
The Architecture of Open Source Applications
Adam: So Greg helped advance the careers of many scientists, and probably their research too, by teaching them Python and UNIX and tool usage. Thank you very much. And on the software design and architecture front, Greg is still trying to push the ball forward.
He put together the architecture of open source applications in 2011. He did volume two of it in 2012.
He wrote Making Software: What Really Works in 2010. He did a book Software Design by Example: In JavaScript targeting students. That was in 2022. And then he did Software by Example in Python, which just came out in 2023. So he’s trying to push this forward,
but Greg still thinks we have a long ways to go before we can talk about software design effectively as a field.
But he’s trying, right? He doesn’t want others to waste 10 years figuring it out alone. And he’s ready to share what he learned from Bill at IBM, from his startup time. He wants to pass it on. He wants to make it simple.
But most of all, I think he wants us to be able to talk about the approaches we use in various real-world software, right?
Why is LLVM more modular than GCC? What’s the trade-off there? What are the design trade-offs of Matplotlib or of RStudio?
Can we actually talk about the trade-offs between, you know, having services and having monoliths instead of just dunking on one being better than the other?
Greg: If we put the right examples in front of people and draw their attention to them, they don’t have to spend years wandering around in the forest wondering if there’s a road somewhere, right?
That feels worthwhile.
Adam: These books, the architecture of open source application series, now four volumes strong, they’re catching on. They start discussions, chapter by chapter, of what makes for a great design, what a real-world codebase looks like, what the constraints are, how you trade them off. They give us the vocabulary we need to talk about these things.
Greg: If you listen to two undergrad architecture students having a conversation about a bus shelter, they’re talking at a level so much more refined than the level that even experienced software designers can talk about software. Because they’ve got centuries of insight and critique and vocabulary to draw on.
They’ve got the terms and the ideas. Industrial design is an interesting field here. It was consciously and deliberately created in the 1910s, 1920s, 1930s. People like Loewy were trying to convince society as a whole that mass-produced artifacts can be considered aesthetically. There was this feeling then that if it’s mass-produced, it might be useful and cheap, but it’s not art.
And people like Raymond Loewy were saying, no, no, there’s another category here. There are things that we mass-produce, but are nevertheless beautiful. And they won. People today will take iPhones seriously. They will take the design of a set of speakers seriously from an aesthetic point of view, not just an engineering point of view.
But it took several decades of essays and arguments and public talks and university courses and creating a profession to get society to believe that handmade and beautiful weren’t necessarily correlates of each other. And if Beautiful Code and the architecture of software, open source applications, and other things that I’ve worked on are one small step towards starting a conversation about us being able to talk about the aesthetics of software in the way that the UX designers we work with can talk about the aesthetics of an interface, then that was worth doing. But that discussion hasn’t even started yet.
You know, design patterns from the early 1990s are still as far as we’ve gone in terms of having a vocabulary for talking about why Nginx is more elegant internally than Apache.
Adam: Part of the problem Greg suspects is that there’s sort of an implied hierarchy about what’s valued in our field.
Pink Collar Jobs
Greg: Programmers look down on UX design, right? They look down on graphic design. And yet, those are the people that we work with most closely who could start to tell us how to take design seriously and how to look at it, talk about it, and think about it.
There have been essays recently, right? About why programmers don’t take CSS seriously. And the answer is because design is seen as a feminized profession. You take a look at a startup. The programmers are guys. The designers are women. And the social status that comes with male-female distinctions, pink collar jobs, things like that, designers are over there writing sonnets to each other. And we programmers bang rocks. Bang, bang. Bang rocks. Good. Right?
Adam: All is not lost though. Beautiful Code might have been too advanced for undergrads, had too much implied context, but you know, I think it’s really the discussions that Greg sparked among his book’s contributors that really pushed the field forward. Because that was the first step to a world where we can discuss software design at a higher level.
And that’s Greg’s legacy. By championing the use of real-world examples in software education, he’s bridging the gap between theory and practice. Giving a glimpse of a world where, you know, in PRs we can have real discussions. Drawing on common examples of real-world software and the trade-offs that they have.
That would be progress, because there was a time when even talking about design, when talking about code being beautiful, might lead to eye rolls.
Here’s an example. Back when Greg sent out those 300 emails for Beautiful Code, the first response he got back was from Rob Pike. And if you don’t know Rob, right, he worked on UTF, he worked on UNIX, he worked on Go and Plan 9.
He’s exactly the type of person that Greg wants us to learn from. Whatever Rob knows, can he teach us?
But here’s the gap in our language or in culture, because what Rob said to Greg was, he’d never seen any Beautiful Code. Beautiful Code wasn’t a thing.
Greg was, I assume, saddened by this, but he shared his response with Brian Kernighan. And Brian wrote the first chapter of Beautiful Code about some code written by Rob Pike. It was this simple regex matcher, written in C. Brian spent a chapter talking about it.
You see, Rob spent time polishing that code, until, according to Brian, it’s some of the most beautiful code he’s ever seen. And if you read the chapter, you get to see how Brian thinks about design and what makes that code beautiful.
Greg: You look at it and you go, well, of course it has to be like that. Right? Like that’s elegant. That’s really cleanly put together. There’s no wasted motion, there’s no wasted parts. It does exactly what it’s supposed to.
I couldn’t add anything, I couldn’t take anything away.
Adam: And also, by writing that chapter, Brian was in effect saying, You’re wrong, Rob.
Code can be beautiful. And you’re actually one of the greats.
And here’s the thing. Here’s the thing that kind of blows me away. I’m about to get a little excited. Because in his analysis, he sounds a lot like what had blown Greg away back at IBM all those years ago. Right? Once you consider the solution, you see the beauty in it. Because beautiful code doesn’t hit you at a glance like a beautiful sunset.
It requires thought. You work to understand the code, you dig in, you consider the various alternatives, and you realize: If I had solved this problem 100 times, if I had solved it in 10 different ways, this solution would be the clearest. This solution is the cleanest.
Outro
That was the show! Big thanks to Greg. Find him at third-bit.com. Find him on LinkedIn. Find him @gvwilson on Mastodon. Or check out his books and the 30 other things that he’s constantly up to. He’s still working with scientists. He’s still trying to push forward the field. Amazing person.
And if you like the podcast, you might like my newsletter where I cover similar topics, or you might like to follow me on Twitter @AdamGordonBell, where I try to share behind the scenes details.
And for true fans, the best thing you can do to support the podcast is go to corecursive.com/supporters. There’s a link in the show notes and join as a podcast supporter. You’ll receive access to bonus episodes. Lately, what I’ve been doing is kind of reflecting on the themes in the show and how it applies to me and the things that I’m going through in my life.
One of the supporters, Patrick C.L., loyal supporter, thank you, Patrick, said, “I really liked the format you’ve been doing with the bonus episodes where you expand on things from the main episode and reflect them upon your own experiences.” So if that sounds interesting, or if you just want to show your support, please check out the supporters community.
And until next time, thank you so much for listening.
Sticher
Overcast
Apr 02, 2024
57
min