Introduction
Adam: Did you ever wonder how open source software got started?
When I first got a computer, I used to buy shareware games from a local computer store. The discs were in this spinning carousel thing. Like you’d see books in at the airport. And they all had this picture of a friendly cartoon wizard on them. And then the title of the game. They were $5 for a disc, and the first one I got was Duke Nukem. That was shareware. But my friend, Eric, his computer had a CD-ROM and you could fit a lot more games on a CD-ROM. And eventually he got a CD for something called Slackware, which was a Linux operating system. This was a complete replacement for the operating system of your computer. And this was in the days when Microsoft was charging serious money for Windows 95, and this was totally free.
Not only was it free, but according to Eric’s uncle, it was so much more powerful than Windows. This would be my introduction to Linux. Well, it would’ve been, but Eric never managed to get it installed and I didn’t even have a CD-ROM on my computer. So I just actually stayed with my shareware games.
Oh yeah. This is CoRecursive and I’m Adam Gordon Bell. Each episode, a guest shares the story of a piece of software being built. And yeah, I always wondered who built these Linux distributions. Did you know that 96% of the top million servers run Linux and many of those are using a Debian based distribution.
Debian is one of the oldest Linux distributions and it’s maintained and supported totally by a group of volunteers. I mean, this is what blew my mind as a kid. When I heard about the idea of open source and it still kind of does. That there are these people working hard on a voluntary basis to build software that powers important parts of the world.
It’s almost like if important parts of our electrical grid were built by a volunteer group of electrical engineers, who we’re just really passionate about what they did. That would be amazing, right? I mean, I know open source isn’t quite like that, but it kind of is, and it is an amazing thing.
So my guest today is this guy:
Joey
Joey: Yeah, hi. I’m Joey Hess, I’m kind of a crowdfunded free software developer.
Adam: And I can’t see you now, often I have a video call, but describe your surroundings.
Joey: Well, I’m in my office right here at home. I live in a kind of a cabin, you can say a cabin and you get the right idea, in the woods here in Tennessee, just surrounded by trees.
Adam: Joey dedicated a large part of his adult life to working on Debian, and he’s going to give us a peek into that world. He had times worked for years without pay to push Debian forward, and then something changed and he decided to leave of it all behind.
So that’s, today’s story. An insider view of Debian. The good and the bad. And it’s almost all good. It’s a story that’s about open source software, but it’s also about community and about teamwork. Joey’s introduction to Linux started a similar way to my story with trying to install Slackware. He was at Cornell staying in residence and someone lent them 15 floppy discs.
Joey: I got it installed, and I’m sure I ran into problems, I probably destroyed, five or 10 systems before I figured out what not to do. I remember at some point, I moved on to Red Hat and I had to email somebody at Red Hat just to figure out which floppy driver I needed and that was actually the first time I talked to somebody in the computer community who was actually a developer.
Adam: That developer was Donnie Barnes, Red Hat employee number two. Red Hat would go on to become a giant company and eventually be acquired by IBM. But at that point they were small and Donnie told Joey what driver he needed to get things working, which is great customer service.
Joey: It’s amazing customer service, especially since I wasn’t paying for it.
The Internet in 1995
Adam: Yeah, exactly. Once you had Linux running on your machine, what did you do with it?
Joey: Well, mostly what I did with Linux was what I had been doing with Windows, which was building websites and CGI scripts. This was back in like 95, 94-95. And I had done that on Windows and quickly found that Windows was not suited for it back then, so I just moved all that stuff over to Linux and kept on with that path. Yeah and I started writing command line utilities and things too, but a lot of it was web focused.
Adam: What websites did you build?
Joey: Oh, a bunch of things that have been lost to history more or less. There was this thing called the personalized Shakespearean insult list. Which was just funny around the net, it was a list of, several lists of words. You pick one from each and you get your insult. So I made a web version of that, but you would type in your name and it would say the pox ridden, whatever, cock’s wing. I don’t know, I don’t remember them, which was very briefly a big deal because there wasn’t really much going on, on the web. So having a website that insulted you was novel.
Adam: So were these running on a server in your dorm room or…
Joey: It was running on my desktop in my dorm room, 24/7.
Adam: People would’ve struggled to put up websites right? But you were hooked right up. You had a good connection to the internet. You had Linux running, you were set.
Joey: Yeah, I had pretty much everything that you could possibly need to put up a website in 1995.
Adam: That’s awesome. I don’t know… What did the internet look like in 1995? That’s pre my time of internet access.
Joey: Yeah. I mean, it was very text based, with occasional images and pretty sparse content when you get right down to it. You really wouldn’t spend a lot of… You would browse around the web, browsing of course was much different back then, because you would jump around a lot from site to site and end up somewhere random. But, it wasn’t… there wasn’t enough really, you couldn’t have a conversation with someone on the web, for example, that was all limited to other mediums.
Adam: I don’t know what year this was, but there was the Netscape site of the day or something? Do you remember this?
Joey: Oh yeah. So yeah, there would be a site of the day because, wow, there’s a new website.
Adam: Yeah.
Joey: My Shakespearean insult thing was probably on, I don’t know if it was the Netscape one, but whatever came before that.
Adam: And your little computer handled that like the being the site of the day,
Joey: You know, it was probably a few thousand people who looked at it. So, but I mean, I was, the web quickly grew, this was 94. It had been around for a few years and it was well on its inflection point upward at that point. So that was a brief period in time.
Enter Debian
Adam: It’s wild that Joey’s dorm room computer and some small script he wrote, could handle being the site of the day for the whole internet. The internet was smaller then but it was growing quickly. But anyways, one of the problems of switching your computer to Linux in the nineties was there was very few computer games. Joey thought he could address that by finding free games and packaging them up to run on Red Hat Linux. But while Red Hat Linux was free, Red Hat was a commercial business and they had no interest in accepting game contributions or any contributions from outsiders really.
Joey: I was, well, that seems kind of silly. And then I was, oh, here’s this Debian, which somebody showed me in a user’s group meeting. Well, I’m going to switch over to Debian, that way I can make packages and they’ll land right in the distribution.
Adam: Debian had a different model than Red Hat. It was built in the open by volunteers who coordinated on email mailing lists and on Internet Relay Chat. And if the licensing was right, anyone could add a package to Debian. Which meant that now all the Debian users could easily install and run that software.
Joey: So I pretty much became a Debian user to become a Debian developer. I was, 20 or whatever, 18, 19, something like that. I was exploring all the Linux games, which there weren’t very many, but there were various Tetris and stuff like that. So I was, well let’s just package all these up, that way when somebody and installs Linux, they can play Tetris.
Adam: And then did you start socializing with people in Debian, were you on IRC? Did you make friends or…
Joey: Yeah, I was on the Debian mailing list. I assume I was on IRC, though I can’t really remember it, and it was all remote and I don’t know if making friends is the right word for what happens when you’re on the technical mailing list with other people, but you’re collaborating and you’re getting to know them.
Off To Berkley
Adam: Joey finishes school and gets a job in the Bay Area.
Joey: I was running a place in Berkeley and Bruce Perens showed up at my door. Bruce Perens was the leader of the Debbie Project at the time. He also lived in Berkeley, up in the Hills and he was, I’m going to take you on a tour of the Bay Area. I had only ever, been in Berkeley briefly.
Adam: How did he know that you were in the area? Did you…
Joey: I guess, I must have came up on IRC or something? I don’t remember.
Adam: And then that was the first time you met him when he showed up at your door, rang your door bell or?
Joey: Absolutely, yeah.
Adam: And what did he show you?
Joey: I remember he drove, we were driving down, what was it, the 580 or whatever, I can’t remember now. I haven’t been in the Bay Area in a while, but anyway, we were driving down. He’s well, this is the Stanford Particle Accelerator we just crossed, and then we went down into San Jose, we probably went to Fry’s or something, it was basically just a Bay Area tour. We didn’t really stop at a lot of institutions of computers in the Bay Area, but he did probably point out here’s this company and that company and that kind of thing. And that was kind of the first person in Debian, probably the first real free software in person interaction outside of maybe a user group or something, it was pretty neat.
Adam: It’s kind of amazing.
Joey: I totally agree. I mean, all I was doing was packaging up Tetris for Debian, and here’s the Debian Project leader. Hey, let me show you around, again kind of like the Donnie Barnes thing, there’s a openness there, which is I think pretty incredible.
Adam: And did you feel like this is a place for me?
Joey: Yeah. I mean, I certainly felt, I got sucked into the Bay Area to a decent extent, although, I was spending most of my time sitting in front of a computer, so how much can you really be sucked into it? But, there were Linux user’s group meetings in the Bay Area probably every other week or so. So I started to go to those and I met a lot of other people who were, users and developers and people and companies and all that stuff in the Bay Area that way.
VA Linux
Adam: So Joey’s going to these Linux meetups and meanwhile, the internet is growing. The dotcom boom is underway and all these crazy buy cat food on the web startups need web servers to run their websites on. And so Joey’s Debian experience is pretty valuable and he got recruited by a company called VA Linux.
Joey: So VA wasn’t a Linux distribution. They were actually a hardware vendor. So their thing was, they were selling rack mount computers with Linux on it, and then would support it for people. They were kind of trying to pull in a collection of people from different free software communities that were all involved in Linux. So they had X developers and they had me as a Debian developer and they had this, just a whole bunch of collection of talent. So they could then turn around and say to somebody, well, look, you need this. We’ve probably got somebody who knows about that or knows somebody who can help you with that or that kind of thing. They would occasionally complain to me that it was too hard to install Debian, and I’d be, yeah it’s too hard to install Debian, I’m thinking about it. But mostly my day job was just to go into work, work on whatever I was working with Debian that day, and then go home and probably do the same thing.
Adam: So it seems like you had a soft landing in the Bay Area. You got there, you met the head of the Debian Project, and then you got a job where you just got to do whatever you wanted.
Joey: Yeah, I don’t think you can really have a better experience than that. I’m pretty sure that I wouldn’t have even went to work for a different company if it hadn’t been something like that, I was pretty skeptical about just, going to some IPO and selling cat food or whatever was a big deal at the time.
Adam: Oh yeah, because, set the stage, right? This is…
Joey: Yeah, this is 1998, 1999, and the dotcom boom is ramping up. And that’s why, of course, VA was selling all these rack mount computers because everybody needed lots of them.
Adam: So were they growing at some astronomical rate as you were there or?
Joey: Yeah, I think I was probably employee number, I don’t know, 30 or something. And then there were probably, within a year there were 200 or 400 or I don’t know how many employees, plus not even counting all the fulfillment this was just, in the office. So it ramped up pretty quick. It had an IPO after I’d been there for maybe a year and a half, which I think is still on record as the biggest IPO in history or maybe the second biggest IPO.
Adam: Oh wow.
Joey: Yeah, as far as the opening price or the opening increase or something.
Adam: You had stocks.
Joey: I had a little bit, but they did have a generous friends and family program, which was nice. I was able to get my dad some stock at least.
Adam: So you had unvested stock options.
Joey: Yeah, sure.
Adam: How much were they worth?
Joey: I don’t know. A couple million dollars at some point or other I’m sure. I really tried not to pay any attention to it because you know, it was clear that I was never going to see that money.
Adam: Was it clear though, did you know at the time?
Joey: Well, I mean, I can look at the trajectory line on the graph and make a guess as well as the next guy.
Adam: I don’t think I would feel it like this. Okay, I have several million dollars once this vests, and yes, the stock market is starting to go down, but maybe it’ll be back by the time I vest, maybe it’ll be even higher. That’s what I would’ve told myself. But Joey, he has a different focus,
The Debian Installer
Joey: All that was kind of in a way, a big distraction from what I was really interested in.
Adam: Which was, packaging or what?
Joey: Well, when I first joined Debian, I did package up, 70 or 80 games or whatever. But then the first thing that I did is, wow, it’s really hard to package software in Debian, how can I make this easier? And so I went off and built something called debhelper, which made it easy to make Debian packages. Of course then people are, whoa, here’s this thing that makes it much easier to do our work. So everybody in Debian start switching to debhelper. So within a year or so, probably 50% of the distribution was built with debhelper. But still, I mean it… debhelper was a fairly simple program. And then I went to VA, while VA was talking to me about how hard Debian was to install. So I’m, well, I should try to make Debian easier to install.
Adam: At this point the Debian community really starts growing.
Joey: It was around a 100 or 200 people for a while. It got bigger during the dotcom boom, and probably at some point went up to more like a thousand total people. It was a whole lot of hobbyist stuff and college students and, occasional somebody in the industry who was using it or whatever. But yeah, it was a very, very wide array of different people, and very few of them were paid to work on it like I was. I was very much the anomaly at that point.
Adam: Did you rise in stature? Like if there’s a leader board of important people in Debian, is it like…
Joey: Yeah, I guess so, like I said, everybody was using debhelper and so that meant that everybody had, ideas for how to make debhelper better. And I was the person they came to them with those ideas. So I was constantly talking to pretty much everybody in the project about their pet peeve and then trying to get them to develop a patch or developing a patch for it. There was a group of people who had done a whole lot of very hard and important work to build Debian, building package managers and coming up with the entire idea of a Linux distribution and things like that. So, I was certainly in that group.
Adam: Being an important Debian maintainer, and also being a tool creator for Debian it’s a hugely impactful role. And now Joey was planning to write a new installer that would improve everything. But first disaster strikes.
Joey: The dotcom crash happened, VA imploded.
Adam: Oh no.
Moving Back Home
Joey: I decided it was time to leave before I got stuck doing something that wasn’t just, go develop Debian all day, we don’t care. Because, it was clear that eventually the money would run out and it would be more like, well we really need you to go install Debian all day on these machines here that we’re going to ship out tomorrow.
Adam: Nice, yeah.
Joey: Or something like that. So I decided to leave, I bailed and move back to Tennessee where I grew up.
Adam: This was a common story of that era, the bubble burst and people left the Bay Area to go back home and look for new work. But Joey didn’t look for new work. I mean Debian was still there and there was a lot to do. Recently the Debian project leader, Jonathan Carter said Debian is a bottomless pit of problems. It happens often that our lives are completely consumed by Debian. For Joey, the thing that would consume him was his new idea for a Debian-Installer. So he found a way to live with very few distractions and at a low cost.
Joey: I was really lucky that I was able to move in to this little cabin down in a valley that was basically free for me to stay at for as long as I wanted to which, how many people have that opportunity? But we had friends who owned the place. It was kind of a ex hippy commune and it was nice to have somebody there to mow the yard and stuff. I’m used to just working on whatever I want to work on. And somehow I had money, so I guess I’ll just keep doing it right.
Adam: The thing he wanted to work on was a new Debian-Installer. He had come up with a design for it and now he just needed long periods of uninterrupted work, to work on it. And the place in the woods was perfect for this. Although it was more of a shack, than a house.
Joey: It, it was like a one room cabin with a stove, with a wood stove in the middle and a unheated kitchen in another building, it was pretty rough. It was fun though. That was some of the… I was really, I was, I really enjoy living this way, it’s such a change from being in the Bay Area.
Adam: That would be a big change to go from living in the Bay Area to in a one room.
Joey: Yeah, the hauling water jugs down, half mile.
Adam: Oh wow. Well, I mean, what spoke to you about that process?
Joey: I mean, it’s kind of finding a way to be almost bored because if you’re bored, then you’re going to think about something, and it’s not going to be like what’s on TV or where should I go to dinner or something it’s, it might be something interesting. So it’s partly that, and it’s partly that I enjoy sitting down in the morning or probably at, 12:30 actually opening up my laptop and just working and not being interrupted all day long.
Adam: Mm-hmm (affirmative).
Joey: And just entering a flow state and staying in it, pretty much continuously I find that I really get good work done that way. It’s just, it’s almost an order of magnitude improvement. And so, I much prefer to work that way a lot of the time. And so having, being out in a quiet place really facilitates that.
And I find that it doesn’t really… I don’t miss out on a lot because when I go into town, I enjoy that then, when I’m in the city, I enjoy the city as the city, but I’m not, I may be getting some work done in a coffee shop, in Cambridge or something. But you know, it’s always, it’s a temporary thing. And I enjoy coming back to calm and just, re-centering and just, having the space to work on what I want to work on at home.
Adam: So Joey spent around six years in that one room cabin, he worked a lot on Debian-Installer and got paid for it when he could. And a couple times a year, he went to user groups.
Deb Conf
Adam: Eventually Debian started having a conference for developers, Debian Developers Conference, Deb Conf. Deb Conf gave Joey a chance to see the world.
Joey: Deb Conf moves around every year to a different country. So I ended up visiting, I probably went to 10 or 12 or something, Deb Confs, all in different countries. So I mean, Deb Conf often would be, a small amount of talks and a whole lot of hanging out and socializing and also trying to get probably far too ambitious amounts of coding done. And often people would be up all night doing it. In fact, I’ve done that more than once at Deb Conf and other Debian get togethers. I remember one time in Spain, I was up all night in, I think Cáceres or someplace trying to convert a massive subversion repository into a thousand Git repositories and it kept failing. And it was just all night long with somebody else alternating, things like that. Eventually I realized that I really wasn’t doing good work at these conferences, so I kind of stopped even trying to get anything technical done.
Adam: Giving up on technical work meant he could just focus on socializing and bonding with other Debian developers. And he has a lot of good memories from these times.
Joey: Sleeping in Norway on the floor of, I think it was a grade school gymnasium with basically half the Debian project, because we were all poor back then. This was all we could afford for the accommodations for the conference. A few people brought tents and most people just had a sleeping bag or something.
Adam: The year after Norway, the conference was in Brazil and it really stands out in Joey’s mind.
Joey: Because I went over a month before the conference with some Debian folks and we met… We were staying with local Debian users and we traveled around and yeah, I really enjoyed that. I remember at one point in Brazil, I sat down for breakfast, and next to somebody who I had never met before and it turned out and it was Jim Gettys, who was basically one of the main developers of the X Window system back in the eighties. I had no idea. He was not involved in Debian, he had just come down for the conference probably because he was, this is a cool conference in a cool country.
Adam: In 2011, the conference was in Bosnia.
Joey: Well, me and a bunch of friends from the UK ended up driving with three cars full of people all the way across Europe, over to Bosnia. They were kind of reenacting top gear, like zooming down the Autobahn with three automobiles, all trying to pass one another.
Adam: That’s awesome.
Joey: Yeah. And all communicating by radio.
Adam: Oh, you guys had like little walkie-talkie radios?
Joey: Yeah, yeah. Or something.
Adam: I think these occasional adventures that the Debian core people went on, it was a big part of what bonded them together. Everyone was excited to see each other. And when they got together, big ideas started knocking around.
Joey: Yeah. There were a lot of things that started just as a conversation at a conference somewhere. A big one that I remember is the idea of source dependencies, which when Debian started, you would have the source to build the package, but it wouldn’t declare what it depended on to build it. You know, originally, Debian and all Linux it just got bootstrapped through some trial and error process, so they ended up with the compiler that worked and well what should you do then? Well, you ship it. And throughout my time in Debian there was a lot of going back and fixing, shortcuts or just, rethinking a problem and having to do a whole lot of work to change it. It’s a whole lot of what Debian’s about really, or probably any Linux distribution is about, continually making these big changes that touch a lot of different stuff.
Adam: These big projects to refactor Debian, and the fact that these improvements would affect millions and millions of machines. That was a big motivating factor for Joey, but they would also would be why he ultimately left Debian. But I think I’m getting ahead of myself. I mean, Joey’s life still revolved around Debian at this point.
The Impact of Open-Source Contributions
Joey: I was doing Debian stuff pretty much full time for about 10 years, maybe a little bit longer, somewhere in there. I mean, I did have other projects that also happened in those times, but a whole lot of time was spent reading mailing lists, arguing with people on mailing list, going off and coding, answering bug reports, trying to prepare releases. It was pretty much a full time job often unpaid for many years.
Adam: I mean, that’s intense, I guess, you must have enjoyed it.
Joey: Oh yeah, yeah. I mean, I enjoyed it because you can see that people were using it, you would make some improvement and then see your users either complain about it or hopefully not even notice it happened, but you could see signs that it had or, you would just do something because it really needed to happen to allow other things to happen. Yeah I was really happy to put that time in, although it did eventually become too much for me to deal with. I was just, but for many years I was very happy with that being my thing.
Adam: And was it like the impact of your work that you liked? You felt, I’m improving this for a lot of people or?
Joey: Yeah, it was a lot about the impact. You know, like I’ve mentioned a couple of times when somebody from the free software community had a big impact on me with something that, was small, but why would they bother to do that? And I’ve tried to do the same thing here and there though I’m probably not anywhere near as good at it than some of those people. I would also try to fix step helper bugs, especially within an hour because I knew that it was blocking somebody else from getting their work done. And I would release a new version with one bug fix in it because I’m, well, let’s get it out. And I think that kind of thing did, people were like, hey, that’s cool, he’s responsive. He’s, fixing it really quickly and getting out to me really quickly.
Adam: But despite all this enthusiasm, Joey did start to get frustrated with Debian.
Joey:
I had noticed for years that it was too hard to make large changes in Debian. The first one that I noticed was, it was a completely ridiculous thing. Debian was shipping documentation in /usr/doc, but the file system standards said it should be in usr/share/doc.
It’s like nothing, the easiest change in the world, guess how long it took for every single piece of software Debian to be updated? It took, I believe six years and a lot of work.
Adam: Six years is so much time and all they were trying to do was have everybody move from putting something in one place to putting it in another. But I do get why this would take a long time. I mean, Debian is a volunteer project and people always, probably had something more important to do. And this happens at big companies too, if there’s a low priority, big initiative, sometimes it can take years to get complete.
Joey: Yeah. I bet this is not unique to Debian at all. It’s just that it’s all out in the open at Debian right? So, and also, Debian is probably the largest free software project I’ve contributed to. And so you’re going to see, the edge cases there, and it’s much easier to contribute to something with 10 maintainers.
Off-Grid Living
Adam: Somewhere in the midst of all this, Joey had to leave his one room cabin, but the concentrated focus he got there was so important to him for achieving big projects, like the new Debian-Installer, and he wanted a way to recapture that.
Joey: I heard about this place that was off grid and the people who lived there were getting ready to move to Africa for a number of years and were looking for a caretaker. And I was, well, okay, I can go back out there and, kind of get back to where I was earlier.
Adam: The place was much bigger than his old one room cabin, but in other ways it was a big challenge.
Joey: It had basically no power whatsoever and no internet and also no running water. So yeah, nothing.
KickStarter
Adam: When he moved to the cabin, Joey was paying bills by doing Debian work for a company that needed it. But then he quit his job and raised some money on Kickstarter.
Joey: It was a point where even a minor thing like a person writing a piece of software that most people have never heard of, once it got to, a certain number of contributions was suddenly trending on Kickstarter probably or nearly trending.
Adam: What was your goal? What did you ask for and what did you promise?
Joey: I think I probably asked for $20,000 or something, it really wasn’t a lot of money, but I was just thinking in terms of, I’d like to work on this for a year and see where it goes.
Adam: You had already adapted to kind of stretching out your savings so that you could do this kind of work?
Joey: Yes, but I’ve been really lucky in that way that I haven’t been dragged down by, the cost of living in some city. And I can’t imagine it working if I had been living in the Bay Area or really any major city.
The Idyllic Rural Life?
Adam: Around this time is when I first heard of Joey. And the thing that caught my interest about him was he looked like this platonic ideal of a hardcore software developer. He was just working on what he cared about and living out in the woods. At the time, everybody was talking about entrepreneurship and startups and how you can work crazy hard as a software developer and make a whole bunch of money, and then you’d be set for life. And here was Joey, and he had been through the first bubble and it seemed like he had cracked the code. He said, you’re looking at the numerator how much money you need to do what you want, but I’m looking at the denominator. I’ve just decreased my cost of living. I made the Zen move, so instead of hitting a big score, I can just do what I want right now. At least this was my impression from the outside.
So that’s really the question, I wanted to ask Joey: Was this idyllic life you’ve built in a cabin in the woods as great as it looks?
Joey: There’s always a backstory that might not live up to the romanticism. But I certainly do feel very lucky that I do have a lot of ability to take some time and just think about an idea and then be, okay, I’m going to go spend whatever amount of time it ends up taking, because it’s worth doing this. It’s hills around me here and a few mountains in the distance, but dense forest and I’m kind of down an oval bowl with basically completely isolated from whatever’s going on, except for whatever noise might filter up from the distant road a mile away.
And yeah, it’s a very calm and peaceful place. And for me that’s more of just a background thing, I just know that, I can sit down and work for five hours and that’s a really nice thing. And I know that if I need a break, I can go and easily take a walk and refresh my mind. And so yeah, it’s the little things really that make living in a rural place nice. I wish more people have that ability and I feel very, very lucky to have it right now. Who knows how long it will continue.
Adam: It sounds magical to me, but I’ve also never had to haul my own water.
systemd
Adam:
Anyway, hauling water was less of a pain to Joey than this thing called systemd. systemd is an init system. It’s basically the thing that starts up everything else on your computer. It’s not really important to the discussion, how it works, but systemd was new and most people in Debian wanted to adopt it. But most just wasn’t enough.
Joey:
Debian had this big fight internally about systemd because it raises people’s temperatures. There was a lot of fighting about it, a lot of technical arguing. Debian has all this, constitutional framework and a technical committee that can break log jams and that kind of thing. And all that got dragged into it. I mean, it took several years really for all that to work its way out. I was perfectly happy with systemd myself. I did empathize with people who didn’t like it for whatever reason, but it was clear to me that every Linux distribution except for one or two would be switching to systemd eventually.
And I felt that I was pretty good at prognosticating, this kind of thing. So I was, if Debian can’t make the obvious decision without all this much drama, it’s too much for me. I don’t want to be drugged through that for every decision. I would rather things move quickly rather than slowly.
Adam: Joey had been contributing to Debian for 18 years at this point. And for most of those years, it was a pretty major part of his life. But he was stressed and he felt like the way the community was structured and the Debian constitution, they were making this decision process harder than it needed to be.
Joey: It was more about taking a step back from the decision and saying, hey, this isn’t really healthy. It’s not healthy for a project to have this hard a time making a decision. Part of it is that Debian is a large project and Debian tries to operate by consensus, which is great, except, consensus are never going to be a hundred percent. There’s always going to be somebody who’s unhappy with it. And it’s just kind of structurally hard for it to move quickly.
Leaving Debian
Adam: So Joey sent an email to the Debian mailing list. It was a bit angry in tone :
It’s been abundantly clear to me that this is no longer the project I originally joined in 1996. We’ve made some good things and I wish everyone well, but I’m out.
He also took shots at the governance model:
If I have one regret for my 18 years in Debian, it’s that when the Debian constitution was originally proposed, despite seeming dubious, I had neglected to speak out. It’s clear to me now that it’s a toxic document that has slowly but surely led Debian in very unhealthy directions.
And the type of news organizations that track operating systems, they wrote headline about this like “More systemd drama as long time, Debian developer quits”. That was info world, or “What’s wrong with Debian” from Dev watch.
The problem with a totally open project is that all the drama ends up right in the open. And you might expect based on that, a backlash from the people within Debian, maybe they would close ranks and shun him, but it wasn’t like that at all.
Attending Your Own Funeral
Joey: It was a lot like if you imagine you went to your own funeral and everybody was saying good things about you, it was one of those days. It was really nice. I’ll treasure, those emails, and I really miss a lot of people in the project, which is the shame because, it’s just not going to be the same when you’re not there working with them every day. But at the same time it was probably also a lot like leaving some company that you’ve worked at for 15 years.
Adam: Yeah.
Joey: And you have a lot of friends there, but yeah.
Adam: What do people say that you found touching?
Joey: You know, just a lot of heartfelt, well wishes and thanks for what you did and that kind of thing, nothing. It was more of just hearing from all these people who, they were friends of mine, but you don’t really know until you do something that’s almost guaranteed to piss people off and instead they turn around and say, oh, thank you. We appreciate your work. And you know.
Adam: When the conference time comes around, do you feel, do you miss it? Do you think back to that?
Joey: Yes. Oh, I do. Yeah. I’ve tried to find another conference. That’s that tight, but it’s hard. I’m sure there are a lot of them out there, but there really was something special about Debian, Debian really is a family in a lot of ways with the good and the bad that comes from being a family right? So, I’ve searched for quite a while for conferences like that. And I still miss them. And every year when they have it, I’m like, aww, I wish I was there.
Adam: Have you gone back?
Joey: I did go back. I went back and spoke actually, it was a good experience, but not one that I think I would want to do very often, because it was also a painful experience because, nice as it was to see everybody, they were all talking about stuff that really was no concern of mine anymore. And so, and then you realize, well, how much of this friendship is just, sharing a technical context.
Adam: Yeah.
Joey: Yeah.
Adam: I’ve thought about this a lot before and not with Debian though with, I worked at a place and I forget the word they used. Oh, they were trying to build a culture. They’re oh, we have to have this culture right? And I had a problem with this word because I wasn’t sure there’s like such thing as culture, but I tried to make an argument to them that it’s actually, it’s about shared experiences.
A culture isn’t something that you hand down or that it appears, it’s more just, people who may have, it’s like people go to war and they might be in a platoon with people who they shared nothing with who have very different views, but they’re, they’re bonded by the horrible experience.
Joey:
Right. Yeah, I think you’re right. I think, especially since I was in Debian as it grew up a whole lot, there were a lot of horrible experiences. And went one way or another things like moving that usr/doc around or having to go back and add source dependencies to everything.
Adam: Yeah.
Joey: Yeah, definitely.
Adam: The people were there with you and yeah.
Joey: Yeah.
On the Outside Looking In
Adam: Yeah. And I know, I don’t know if you’ve ever left a job, you have a great job, you leave for whatever reason. And you have kind of a notice period and it’s super weird. It sounds like what you’re saying about the going back to the conference, because you were so tight and everything, whatever. But once you give your notice is kind of, there’s kind of everybody kind of stepping away kind of process right? And all of a sudden you’re still there, but it’s like you’re not, it’s…
Joey: Mm-hmm (affirmative) Right. Yeah. I mean, it was, you were definitely on the outside looking in, even though not intentionally, they weren’t snubbing you. It was just what can you really contribute to this discussion?
Finding Community
Adam: If Joey’s story were a movie then I think that as he left that last Debian conference, there would be sort of a montage. It would show the time he got a tour of the Bay Area and the time he was working in his cabin on the new design for the Debian-Installer, or staying up late at a Debian conference, working on some project and sleeping on the floors of gymnasiums. But then the montage would end and Joey would drive off into the next part of his life.
Adam: Debian was this great thing for you and this great community, what’s out there now. If you’re pointing somebody towards a community to grow in?
Joey: I honestly have been mulling over that exact question for a year and I haven’t come up with the answers. So I would like to know, I know people who are involved in developing guix, which is nix like distribution, they’re clearly involved in that kind of thing. They have that same kind of feeling that I had back working on Debian. There’s clearly somewhat in various language communities like the Haskell community has a community feel to it where, we all have a lot of libraries that we collaborate on and there are concepts that flow through the community and things like that.
But I do feel that something a little bit is missing that was there when I was younger, maybe, but I would like to know where it is.
Outro
Adam: That was the show. A big thanks to Joey for being on the show. Thanks for sharing so much about your life. And thanks for all the great open source software that you’ve created.
If you want to know more about Joey, you can go to his website, joeyh.name. And if this is your first time listening, be sure to subscribe to the podcast. The episodes come out monthly.
And if you want more than that, you can subscribe to my Patreon page. Next up on Patreon will be some bonus questions with Joey that I didn’t get to here. Here we mainly focused on Debian, but Joey also built Git Annex and some cool solutions to off the grid living using FRP and Haskell. We’ll talk about why he likes Haskell so much and other stuff that didn’t fit into this story. So until next time, thank you so much for listening.
Sticher
Overcast
Apr 02, 2024
57
min