Apple 2001 With David Shayer - CoRecursive Podcast

Intro

Adam: Today on CoRecursive:

David: So I was in my office one day, and my boss’s boss came in. And he said that, “I have a special project. And I want you to work on it. There’s some guys that are going to come over from the US government, and we’re going to help them build some kind of device.”

And I said, “What do you mean?”

And he said, “Oh, I don’t know anything more than that.” And he said, “Your boss doesn’t know anything about this. He just knows you’re working on some special project. Don’t talk to him only talk to me.”

Adam: Didn’t that make you suspicious?

David: Apple had a lot of special projects that would go on at various times. There were people who would have their office doors closed, and whenever you would go and see if they wanted to go to lunch, they would actually have black cloth they would put over whatever they were working on. This was not that uncommon.

Adam: Hello and welcome to CoRecursive. I’m Adam Gordon Bell. Each episode of CoRecursive, someone shares the fascinating story behind some piece of software being built.

Today on the show, we have David Shayer. He worked at Apple for 14 years, and he really has a wild experience to share. He’s going to give us the insider view of what it was like for him at Apple during the 2000s, roughly between 2001 to 2015, when Apple transformed into the powerhouse that it is today. There will be special projects. Some of them top secret. There’ll be government agents. But also just building file systems and working on really short timelines, and having development plans upended by Steve Jobs. And also, just generally working hard under high pressure.

The Poker Game

Adam: The story starts in the summer of 2001.

David: I was an independent developer writing Mac software, and selling it to companies that wanted things published and didn’t have enough engineers. And there was an Apple poker game that Apple engineers ran on the side. The poker game originally took place actually at Apple in a conference room. There was one conference room with had a nice round table that was just the perfect size. So everyone would show up at 7:00 PM. And most of the people were Apple employees, but there was a Google employee and a Microsoft employee. And a couple independent people like me.

Adam: Do you just buzz in or something? Somebody would come get you?

David: Yeah, you’d call someone. You’d say, “Hey, I’m downstairs.” And someone would come and get you and take you up.

I got myself invited to that poker game and would go every week and lose my 20 bucks. So they’d keep inviting me back. And would keep my ears open because you could find out all kinds of interesting stuff about what was happening at the company. And you could get tech support occasionally. You’d say, “Hey, I’m having trouble. This call doesn’t work the way the documentation says it’s supposed to work.” And someone at the poker game would usually say, “Oh yeah, I wrote that. The documentation is wrong. You need to make this other call first, and then it’ll work.” And this would save me days of messing around.

It was the cheapest, most reliable form of tech support available. And it was fun. I like these guys. They were friendly. And one of the things I was doing at that time was working for Symantec, working on Norton Utilities, which was a disk repair program. And so I was deep in the Mac file system and repairing broken catalogs and such. And I got a call from one of the guys at the poker game that said, Apple wants to hire someone who knows how to write a Mac file system.

It’s a specialty right there. Aren’t a lot of people who know file systems well, and I’ve been doing it for years. And so he said, “Call this guy, Tony Fadell,” who I’d never heard of, “and tell him you want to interview for this open position.”

So Tony knew I was going to call. I called Tony and said, “Hey, I hear you need a guy who can write file systems.” So I came in to interview.

Adam: What did you think going in? Were you nervous? Were you hopeful?

David: This was in 2001, and the Dot-com bubble had just collapsed in the stock market, which meant most of the companies in Silicon Valley that had been giving me work had either gone bankrupt, or even if they weren’t bankrupt, they had canceled everything that wasn’t actually earning the money that day. And so all my contracts disappeared within a couple of months. I had no work. It was the first time since getting out of college that I didn’t have any work. Nobody was answering my phone calls at any of these companies.

In fact, most of the people I know had been laid off. So it was pretty bleak. So when I heard there was a job, I’m like, “Hell yeah, I’m taking that.” And then when I heard it was to write a file system, that was perfect.

The Mac file system was called HFS+, which was proprietary to Apple. And there weren’t a lot of people who knew a lot about it. Probably half of the people who knew how it worked, worked at Apple already. So obviously, I knew that those guys were all busy, or else they would have just assigned them to work on this project. And the other people they knew about it, there were probably couple dozen out in the world, not all of them lived in the US, most of them were doing other stuff. So I knew that there wasn’t a lot of competition for this rather particular skill they needed.

So I came in to interview, and Tony said, “Can you write a file system?” I said yes. And he said, “Do you know everything there is to know about Mac file systems?”

I said, “Yes, pretty much. I’ve been working on them for years, and I know an awful lot about it.” And then there was a pause, and I realized he didn’t know anything about file systems. So he didn’t know what else to ask. He was an electrical engineer, a hardware guy. And he was really smart and really good at what he did, but he wasn’t a software expert. So basically just said, “Okay, can you start on Monday?”

And I said, “Sure. What are we building?”

And he said, “You don’t need to know.”

The First Day

Adam: So David heads into his job not knowing what he’s going to be working on. But he does know that the guy that hired him was an electrical engineer, and that he’s to be working for the hardware division of Apple.

David: So Monday I show up at Apple, and they show me to a cube. And there’s a development board. And the development board looks like the motherboard from a desktop PC that had been taken out and put on my desk. Because it has all the components out on the board. And it’s got a test point so you can hook up oscilloscopes and logic analyzers and stuff, if it’s not working right, and diagnose the problem. It’s got big power supply off on the side to power. It’s got a big, full-size desktop, hard disk hooked up on the side.

This is pretty much the way all new computers are developed. They start out like this so that in the early days, hardware engineers can hook test equipment up and see what’s not working. And then as you get into production, you get the tiny little board which ends up in your laptop. So it’s a foot by a foot. And it’s got a bunch of discrete components on it. And the SOC with the dual ARM cores in the middle, and hardware test points all around the outside of the board.

Adam: Apple’s computers at this time were based on the PowerPC CPU, but this machine spread out all over David’s desk has an ARM CPU, and he’s given a Windows computer. This is really strange, but he knows what he was brought in to do: to get a file system up and running on it.

David: So I use my Windows machine with the ARM development tools, because someone at the beginning decided we were going to use the Windows version and not the Linux version of the ARM development tools. And you would compile the operating system in C++. You would load it on across a little serial interface, and then you would run it. And you would debug it with this little debugger that went across a serial debugger connection, called JTAG, which stands for something. I don’t know what it stands for.

Adam: That’s crazy. I would have been scared if I came into a new job, and they had what appeared to be a motherboard hooked up to a serial cable, which you don’t even see those anymore. And then an oscilloscope or whatever. I would have been worried that I got the wrong job or something.

David: Well, I didn’t have an oscilloscope in my office because I’m not a hardware engineer. I wouldn’t know what to do with it. But we had the same boards that the hardware engineers had because that was the only board that was in production. At some point, the production line gets up, and they start doing test run. So every week they do a test run, and they produce a couple dozen boards. And they get shipped from China back here for us to play with them and see if they’re working. And they don’t want this stuff just sitting out unprotected where people can see it.

So they make what Apple calls a stealth case, which is just a big plastic box. So our stealth case was so ugly. It looked like they went into an old Russian medical equipment leftover warehouse and just took some plastic boxes, and stuck it in. It was horrible. They said basically, “ You’re writing the file system for this thing. You don’t need to know what’s on the disc. Just make it work.”

Building the Team

Adam: So David dives into making it work. And they start to build up a team.

David: I was the second software engineer hired. They were hiring quickly, but still, this was not a big team. They’ll probably a dozen people total. When I started a couple of hardware engineers, a mechanical engineer, product manager. It wasn’t a big team. By the time we shipped at the end of the year, there were probably two dozen people. So it was a small team.

And at the first meeting that I went to, they said, “We’re going to ship by the end of the year.” And I thought it was a joke because I thought nobody can ship brand new product with new hardware and new software in six months. And I was about to laugh because I literally thought it was a joke. And I noticed no one else is laughing. So I’m like, “Oh, this is what I signed up for.”

So we were working literally 80 and 90 hours a week, seven days a week-

Adam: Oh, wow.

David: … till midnight or later to get this thing out. I remember I was out on a Sunday night in September with my wife at a concert, and at 10:00 PM, my boss calls me at the concert on my cell phone and says, “There’s a bug, and we think it’s in your software. And you needed to come in here because when the guy who is going to call this code comes in Monday morning, it needs to work.” So I told my wife when this concert’s over, “I’ll drive you home, and I’m driving into work.”

And I went and spent Sunday night in work fixing it so that Monday morning it was ready for the guy who was going to be the customer of this API.

The Development Loop

Adam: The operating system for this mysterious project wasn’t being made from scratch. The lower level parts of the OS had been licensed from a third party.

David: Apple’s file system was called HFS+. And the people when we bought the lower level of the operating system from obviously didn’t know what that was, hadn’t supported it. Because Apple was not taking over the world back in 2001. So they didn’t see any reasons to support Apple’s file system. They supported FAT32 because that was what most people use back then.

I took the HFS+ code from the Macintosh operating system. The first problem I had is that the Mac, at that time, was a big Endian processor, and the ARM chip was a little Endian processor. And the file system has all these data structures that are written out to disk. And they’re written out big Endian. So we had to add code to reverse all those as they come in and as they come out.

Adam: Big Endian and little Endian is the order that numbers are written in binary?

David: Right. When you take a number, an integer or a long, it’s the order that the bytes are actually in, in memory. In big Endian, the most significant bit is at the lowest byte number. And in little Endian is the reverse. This was the first time that I was aware of Apple had shipped a little Endian processor. And so I had to put in code to handle that, which is not difficult. It’s mostly just very detail-oriented. It’s a matter of finding every single data structure in the file system. And of course, the file system has variable size data structures. It’s not just simple data structures. A lot of them are variable size.

And the development loop was terrible, though, because the ARM development tools were slow, and loading your code across the serial cable was slow. And the debugger was terrible. So how big this field is depends on these other fields that have come before it. So there’s a lot of calculation, and there are various places where I’d miss something. And then it would crash, and then you have to go back and figure out, “Okay, what did I miss?”

Adam: We already have rules for how we write down numbers. Why not follow the lead of humans? Follow how decimal works?

David: I think the hardware engineers told me once that you use one less gate if you do it little Endian.

You use one less gate, if you do it little ENDIAN rather than big ENDIAN and so Intel and I guess arm chose little ENDIAN because it used slightly less hardware. Fortunately, when the TCP/IP was developed for the internet, they picked the standards and my memory is that all stuff sent across TCP/IP is big ENDIAN.

Adam: Yeah, that’s the one that makes sense.

David: Yes, it’s the one that makes sense. If you just take a hex editor and you look at the memory just in straight hex, big ENDIAN is the one that looks like you expect. And little ENDIAN is the one who like, okay, this byte actually goes on the left of that byte and then you’re doing all these corrections in your head.

Adam: I have to be honest. I’ve never had to use a hex editor in any sort of software development task, but David has.

Project Secrecy

Adam: And so as he works through this kind of monotonous task of mapping the bite orders back and forth from one style to another, the secrecy on the project just continues.

David: We weren’t on the main campus. We were a quarter mile away in a accessory building, which mostly had people who did backend work at Apple. The people who ran the computers that did accounting and logistics and shipping and stuff like that. So they just took a part of a floor in that building, they put up security doors so those guys couldn’t come into our area and they put us in there. They were so paranoid, they told us when you go to lunch because we would go to the main building with the main cafeteria at lunch. They said, you can’t all walk together, right? You can go in groups of two and three, but all of you can’t go to lunch together because we don’t want the other Apple employees to look and say like, Oh, who are these people? We’ve never seen them before. What are they working on?

Adam: That’s crazy.

David: Yeah. They were serious about it.

Adam: It seems like a vague vibe of paranoia that I’m getting across from your stories.

David: I wouldn’t call it paranoia, but Apple definitely has a lot of secrecy. I mean, people have always been interested in what Apple is doing, right? There’s all these websites which are printing rumors about what’s going to be in the next iPhone or what’s been added to the next version of the operating system and other companies don’t have this, right? I mean, I’m not aware of any websites which print rumors about what’s coming in the next Dell laptop.

Adam: Yeah.

David: So Apple likes to show something and have it be the first time you’ve ever seen it because they know you get a huge punch from showing something and you’re like, “Wow! That’s really cool.” Like when Steve Jobs showed the first iPod, nobody had seen one and people were like, “Wow, that is a really cool thing.” Whereas, if it had been leaked and everyone kind of knew it was coming, then he shows it. You’re like, “Oh yeah, that thing I already knew about it.” And so they work really hard to get that. And teams don’t talk about what they’re working on. When you go to lunch with friends who work on different teams, you don’t ask them, what are you doing? Because then they’re just going to have to tell you, like I can’t say and that’s impolite. So you just to talk about something else.

Adam: Eventually David did figure out what they were building.

David: I talked to enough people and I realized like, okay, we’re building a music player because people are worrying about how are we going to put music files on it? And how are the headphones going to work?

Adam: That’s kind of a giveaway.

David: Yeah. So the iPod was announced, I think at the end of November in 2001, and it shipped a couple of weeks later early December. So the timing was terrible. 9/11 had happened in September of 2001, the U.S. was in a recession between 9/11 and the dot com bubble bursting. The U.S. was in a recession and Apple releases this $400 music player. Remember this is 20 years ago. So this is probably like a $600 music player now based on inflation and everyone’s looking at this going, what are you guys thinking? And we didn’t really sell many until the first quarter of 2002. And we sold a couple hundred thousand of them, I think.

I remember thinking I better start looking for another job because this project obviously is doomed. So this is why nobody would hire me for marketing because obviously I had no idea what’s going to be popular and what’s not because the Pod went on to be a huge success. But in the beginning, nobody knew it was going to be a huge success. Marketing would come by every few weeks and they’d show us a little clip of some basketball star was seen getting off the bus and he had the white earbud. So we’re like, “Oh cool! Some celebrity is using our products. Isn’t this great?” Now of course, Apple sells a million iPhones a day.

Adam: It’s crazy. So I got, I think the third generation of the classic one, I remember it seemed expensive. I used it a lot. I found it very revolutionary at the time. But in today’s terms, it just played music, but a lot came from this, right?

David: Yeah, I think it did two key things for Apple. It showed people that Apple could do cool little devices. And so people stopped thinking of Apple as the company that sells those weird computers that nobody uses except the guy in the graphics department and people started buying little iPods and saying, “Oh, Apple makes cool stuff. I kind of like this, right?” So it changed Apple’s image. And it got Apple a lot of skill at making small miniaturized devices because that hardware team that made the iPod ended up doing a lot of work on the iPhone because they knew how to make tiny little devices. And Apple also learned how to write software for small embedded devices too.

When the first iPhone came out, it pretty hard to figure out how are we going to fit Unix? How are you going to fit that into the hardware that was in the original iPhone? It was quite a challenge. I mean, there was some talk about whether we should just expand the iPod OS and there was a bake-off as they call it where they did a couple of different OSs. There was a little team of a couple of guys who got Linux running on early iPhone hardware. There was a team from the Mac group that got a cut down version of the Mac operating system running on the early iPhone hardware. They showed all this stuff to Steve and to the senior vice president of software engineering. And they ended up picking the one that was based on the Mac OS. And that’s what eventually turned into iOS that we use now.

Steve Jobs Evaluation

Adam: As the iPod sold more units, it started attracting more attention and David’s teams work started being directly evaluated by Steve Jobs.

David: There was a period in the early 2000s when Apple was mainly known as the iPod company, before the iPhone shipped. And Steve was intensely interested in the software, but of course, mostly the user interface level software, right? I mean, he didn’t care about the file system, as long as it worked. It wasn’t his interest. He wanted to know what do the apps look like on the screen? I was in the group that wrote all those apps. My boss would have a weekly meeting with Steve and he’d go and show him what we had worked on. And usually he had questions like okay, there’s several different ways we can go working on this photo’s app or this notes app, what should we do? And sometimes Steve would say, Oh, this is terrible. This is good. You should do more of that.

But often Steve would do something totally different. I remember my boss would come back from his meeting with Steve. We’d all be huddled around his office, waiting for him to come back to find out what did Steve say? Because this determined the direction we were going in for the next week. He had gone in expecting Steve to say, did he like these screens we had done for the new player screen that showed you what song was playing. And instead Steve had spent the entire meeting going into the preferences panel and just tearing it all up and saying, I don’t like this. I don’t like that. I want you to rewrite all the preferences. And the preferences had already been shipping for two years. We thought it was all settled.

My boss come back. He’d be like, well, guess what we’re doing this week? We’re rewriting all the preferences, I guess we’ll see if we can get some closure on these other items next week.

Adam: That’s awesome.

David: So that was actually a pretty common occurrence, is Steve just going off on something completely unrelated because that’s what caught his eye. The other thing was that you did not want your demo to crash when Steve was playing with it, because remember this is months before we ship, right? And so we were building alpha builds of the operating system and we would get like a half dozen iPods and each would be loaded with slightly different build to show off some different possibility of what we could do. And my boss would go into the meeting with all these different iPods and there was kind of what we called, the path he was supposed to go down, which is like what user interface items you were supposed to touch to show off.

You usually get 30 seconds in before Steve would grab the iPod out of my boss’s hand and would just start clicking around to play with it. And of course that’s the worst thing ever because half of the stuff isn’t hooked up yet and Steve’s going to hit something and then it’s going to crash. And then Steve just decides that you’re all a bunch of idiots because your software crashes. So we hated it when that happened.

Disk Formatting

Adam: Throughout these years of working on the iPod, David got to work on everything from device drivers to UI, to secret government assignments, more on not very soon, but his favorite thing to work on was always the file system.

David: There is one piece of software that I am still proud of. And it’s something that converts the file systems between HFS and FAT. When the iPod shipped originally, it only worked on macs and it used the HFS plus file system, which was Apple’s file system. And then at some point they said most of the world uses Windows. And so we want to be able to sell a Windows version too. And so we added support for FAT 32 so that you could hook the iPod up to a Windows. We shipped a Mac iPods and Windows iPods and you’d go into an Apple store and you’d say, I want a Mac iPod or Windows iPod. And they had different skus and the hardware was identical. There was no difference. The difference was whether the disk was formatted for a Mac or for a Windows machine. And the marketing people came back and they said, this is great because we’re selling on both platforms, but we don’t want to have to keep track of two skus when the hardware is identical. So can you make one which is going to work on either, right? I said, that sounds simple. We’ll just reformat the disk to be either FAT or HFS, depending on what you hook it up to. And they said, Oh, one more thing though, we’re going to do a marketing deal with some record companies and we’re going to preload some songs onto the iPod. They’re going to pay us and we’re going to preload whatever they’re promoting this season and you can’t erase that. I said, Oh, this makes it a lot more complicated now.

And so all the iPods would ship from the factory. I think they all shipped FAT. And the first time you plugged it into a computer, if you plug it into a Windows computer, it would say, that’s great. It’s on a Windows computer just going to stay FAT. If you plugged it into a Mac, the first time you plugged it in, it would say, “Oh, it’s a Mac. I need to reformat to HFS.” And so I wrote code, which would take the iPod, which is basically a computer, right? It’s got a processor and a disk and all that. And while you were booted from the hard disk, it would reformat the hard disk from FAT to HFS plus without losing any data. So all the files are maintained. If you crash at any point or runs out of power or someone does a hard reset at any point, you don’t lose anything.

Adam: Revert.

David: So what I did is I would look for unused blocks in the FAT disk and I would write the HFS data structures there. And then at the final right, was the right that changed the pointers at the beginning of the disc. So instead of pointing to the FAT data structures, they pointed to the HFS data structures. And if it crashed during that one, right, then you are screwed. But that’s very unlikely, right? As a single disk write of one block is very fast. So it goes from being a FAT disk, which has a bunch of weird data structures that don’t make any sense written in empty blocks, to being an HFS disk, which now has a bunch of weird data structures, which don’t make any sense written in unused blocks.

Guess what? The marketing department never actually closed the deal with the record companies. They never used this feature. I mean, they use the feature that reformatted the disk, right? So there’s only one skew. You’d buy an…

I’m out of the disk, right? So there’s only one skew. You’d buy an iPod and the first time you’d plug it in, it would reformat itself as FAT or HFS, depending on what computer you plugged it into. But they never used the feature that I spent all that time on, which saved any data that was already on the disk, because they never did a deal with the record company for that.

Adam: That’s hilarious. I like your strategy though, right? It’s not a good state the whole way through the conversion.

David: Yes. I had to do that. I mean, one of the things I learned going from, you know, writing software where if you were selling a few thousand copies a month, that was considered good sales, to working for Apple, where you’re selling millions is that when you’re selling millions and millions of units, every single possible thing that could go wrong is going to happen for someone. And so you have to make it absolutely bulletproof, if there’s any way that something can fail thousands of people are going to hit that failure point and that’s not acceptable.

Dev in a Hardware World

Adam: David loved that working on the iPod meant that he could literally be full stack from boot loaders and device drivers to use their interface. But working in a hardware organization also had its challenges.

David: Apple has separate hardware and software divisions. So most of the software engineers work for the software engineering division, but hardware obviously always has a little bit of software in it, right? There’s always a firmware chip or something. And working for a hardware division, like we were an iPod, means that the top people are all hardware engineers, right? The VP of iPod was a hardware engineer, not a software engineer. And usually it’s fine. It just means like they don’t know all the details of how we do our work. Whereas they do know all the details of how the hardware engineers under them do their work because they are hardware engineers, but also occasionally would cause problems.

We were on these really tight schedules, working hard to get a new version of the iPod out every year. You know, Apple shipped a new one for Christmas every year, just like they do now for phones.

Adam: Mm-hmm (affirmative).

David: And if you worked on a product where you’re going really fast, trying to get stuff out the door, you know, sometimes you don’t always do the perfect design. Sometimes you kind of hack something together because it works and you think, “I’ll come back later and I’ll clean this up.” Because this isn’t great, but, you know, I can do a better job later, but right now I have to get this done so we can ship. And after a couple of years of that you have code, which is pretty messed up and hard to support.

Adam: Yeah.

David: And that’s when you want to refactor and we’d go to our boss and we’d say, “This code to support how menus are laid out, it’s terrible, has all these bugs, we should fix it.” And my boss would say, “Okay, I’ll try and put some time into the schedule.” And then it would go upstairs and the hardware engineers who actually ran the whole division, they would be like, “ What’s this slack time? You’re taken a couple of weeks out of the schedule to do this refactoring crap. What’s that?” They thought we basically wanted to screw off and do nothing for a couple of weeks so they would cancel that. And they would be like, “No, no. We’re going to add some more features that we can sell.”

We had a big blow up a couple of years later where a lot of people were having a lot of problems getting the code ready, getting it stable, things weren’t working, it coincided with a new development tool that we’d been promised not coming in on time. So, that made us late. And the code was all just years of accumulated crap that we hadn’t had time to clean up.

We ended up just working seven days a week for months getting an out the door. I know one engineer worked 34 days in a row without a day off.

Adam: Oh, wow.

David: We’d also hired a bunch of people so the team had grown and because of the way Apple accounting worked, the budget for bonuses had been set at the beginning of the fiscal year. But by the end of the fiscal year, there were a bunch more people. But the budget for bonuses hadn’t grown. So everyone got what we considered to be a small bonus for this super hard work. And at the end, after we shipped the product, a third of the engineers quit. This was in the mid-2000s. And so that finally opened up management’s eyes that you can’t keep doing this because if you do it again and a third of the engineers quit next year, you’re not going to have enough engineers to put out product again.

Adam: Right.

David: So, that’s when they finally realized that, okay, you guys, aren’t just making up this stuff about we want some time to refactor our code. This is a real thing. This is something software engineers do is they take a couple of weeks off, they pick something that’s really ugly and unmaintainable, and they rewrite it to make it better. So we finally got some time to do that.

The Top Secret Project

Adam: In the midst of all these hectic schedules for yearly iPod releases, David’s knowledge of file systems led him to get a very strange assignment.

David: Yeah. So, I was in my office one day and the director of iPod software, who was my boss’s boss, came in and he said, “I have a special project and I want you to work on it. There’s some guys that are going to come over from the US government and we’re going to help them build some kind of device.” And I said, “What do you mean?” And he said, “Oh, I don’t know anything more than that. You’ll get a call tomorrow.” And he said, “Your boss doesn’t know anything about this. He just knows you’re working on some special project. Don’t talk to him, only talk to me.”

Adam: Did that not make you suspicious?

David: Apple had a lot of special projects that would go on at various times, there were labs on the floor that were locked, and my badge would not open them in theory you don’t know what’s going on. Sometimes you do know what’s going on in them, but you’re not supposed to so you don’t talk about it.

So my boss’s boss says, “I want you to work on this secret project.” And the next day two guys show up, Paul and Matthew, and go into a conference room with them and they said they’re actually working for Bechtel, which is a large US defense contractor. They’re not actual government employees, Bechtel has the contract from the government to do this.

Adam: What did they look like?

David: They were just regular engineers. They were probably about 30, just normal looking guys. They were very good engineers. I got them an office. I requisitioned an office. I got them a badge so they could come into the building. I told them what they needed in order to be able to compile and develop our code. So, they needed a Windows machine. They needed to license the ARM development software from ARM. They needed to buy some iPods to mess with because Apple wasn’t providing any material support. Apple is just providing a place to do the work and they were providing me to help them. And I gave them a copy of the source code, burned on a CD, but I told them, you cannot take it out of the building. Once you compile it, you can take the object code on your working iPods out of the building, but you can’t take the source code out.

Adam: So, like, you tell them day one, “Okay, you’re going to need this machine and whatever.” And then what? They come in with a computer the next day and the shopping cart full of iPods they bought down at the Apple store?

David: Pretty much, yeah. I don’t think it was the next day, I think it took them a couple days. They requisitioned it, Bechtel bought it for them, and they came back with this stuff and we set it up.

Building and learning your way around the source code for iPod was kind of complex. And so my first job was just showing them this is where things are in the source code, this is how to run the compiler, which was fairly complicated. And we had a pretty deep build script. Once they got that going then they said what they wanted. They wanted to add some hardware to do something, which they wouldn’t tell me what it was. They wanted a way to turn this on and off.

Adam: How would they add hardware?

David: I never saw that part. I’m not a hardware engineer. I was never in the office when they had any of that out. So I don’t know what it looked like or how they did it.

Adam: Okay. Speculate. They pop open and like solder something in? I don’t know, what’s the process.

David: Or possibly they have something connected on the outside. I don’t know.

Adam: And then how do they wire that into like the operating system?

David: So that was the part that I helped with. They wanted a way to turn it on and off. So we picked one of the menus in preferences, which was really deep, one of the deepest menus you could go down, and we added a menu item, which had some really boring innocuous name like equalizer mode on off or something. Something dumb.

Adam: Yeah.

David: And I show them how to add a menu item. I showed them how to detect when it had been selected. And then they wanted a place to store their data on the disk that wouldn’t be detected so I showed them how to add another partition. They wanted it to be completely innocuous. You could plug it into a Mac or a PC, iTunes would pop up, would put new music on it, just like a real iPod and nothing strange would happen. iTunes would never say, “Hey, what’s this extra thing.” Right?

Adam: Yeah.

David: So we played with it a little bit until we found something totally innocuous that nobody noticed it was there. And then I showed them how, in the operating system, to get access to this partition and to write to it, and then they’d say, “Thank you very much,” and I was out of the office, they would close the door, and they would get to work.

Adam: How would they get the data off so it doesn’t show up when you plug it into your computer?

David: USB supports a bunch of different protocols. And MSC is one of the protocols for just hooking up a standard USB hard disk. If you just buy a cheap USB hard disk from Amazon and plug it into your laptop, that’s protocol, it’s running. And the iPod did that too. And the iPod also had some extra stuff we’d added into that protocol. iTunes would send a special proprietary command and if it got an answer then it knew that it was talking to an iPod, and if it didn’t get an answer, it knew that, “Oh, this is just some random hard disk that was plugged in.”

Adam: Oh, so they could conceivably do the same thing. So there’s just something they’re going to send. And it’s like, knock three times and it’s like, “Oh, here’s your extra data.”

David: Yes, exactly. Presumably they had a tool that they wrote, which ran on their desktop computer, which would talk to the iPad and knew what to look for.

Adam: Where in the government were they supposed to work from?

David: So Bechtel had a contract with the Department of Energy. And I know this because they gave me business cards and the business cards said they worked for Bechtel in a division that had a contract with the Department of Energy. And the US Department of Energy manages the US nuclear arsenal. So, my guess is they were making some sort of secret Geiger counter and they wanted to be able to take it around and look for, I don’t know, people making dirty bombs maybe? But not have it looked like a Geiger counter. If someone’s walking down the street with something that’s clicking, like a Geiger counter, you’re like, “Uh-oh. What’s going on here?”

Adam: I looked it up. The Department of Energy in Nevada has the Nevada National Security Site. They have some role to play in making sure that nuclear weapons don’t spread around the world.

David: Yeah. I could see that. I mean, if you wanted to go and check if there’s any signs of radiation in some place where you’re not officially supposed to be checking, you’re on a tourist visa in Tehran or something, this would be a very useful device. If you’re arrested with a Geiger counter you’re in trouble, but if someone looks at your iPod and the iPod works and it plays music and it’s totally normal, you’d be much safer, right?

Adam: Yeah, totally. So, they built this and then?

David: They built this thing, you know, it took a couple of months and they were done and they left and I never found out what happened to it.

The interesting thing is, as far as I know, Apple had no written agreement. This was all handshake under the table agreement. There were only four people at the company who knew about it. And as far as I know, we’re the only people who knew that this happened. There was nothing written down inside Apple, no email.

Adam: And I assume, because of, like, this not talking about what’s going on culture at Apple itself, did you keep the story to yourself?

What’s going on, culture at Apple itself? Did you keep the story to yourself for a long time? And you were like, “By the way, I think I worked on a secret nuclear detection device?”

David: Yes. Well, the other thing is, if this thing is still in use, you don’t want to let you know whoever our adversaries are know that, “Hey, this is one thing that we’ve done that you could look for to find out if there are any spies in your city.” But of course at this point, iPods are so obsolete you would draw more attention to yourself trying to use an iPod now.

Adam: If I want to verify this story, there’s nobody who in fact can verify it.

David: So, actually, Tony Fadell, who was the Vice President of iPod, when the story first was printed, Tony read it. And he Tweeted, “Yes, this really happened.” It’s quite possible there was some other Apple engineer helping them with the other parts that I never knew about because I didn’t have any reason to know about it.

Adam: I’ve included a link to Tony’s versions of events on the episode page. He doesn’t say a lot, but he does back up the story. I guess enough time has passed that people are now comfortable sharing the details. I think that’s also the reason David’s sharing about working on the original iPod.

The Profession Has Changed

Adam:
David actually loves the products that Apple builds. He’s been a fan of Apple for a long time. One of his first computers was the Steve Wozniak designed Apple II.

David: I had a deal with a teacher. They had got a few Apple IIs and they didn’t have any software. And since I was known as kind of the kid at school who knew how to program, the teacher let me take one home for several months on the condition that I wrote a program that she wanted. And so it was a program to teach you basically how to balance a checkbook. And so that was how I earned the ability to have this Apple II at home for, I guess, a semester, a year, something like that was writing the software and giving it to the school.

Adam: If you’re going to teach somebody starting today how to program, do you get them an old Apple II?

David: In a way it’s harder to learn now because the programs that you see are so much better when I started to learn GUIs didn’t exist and everything was command line, and writing a command line program is simple. You just print stuff out and you read in what people type, and there’s no other interface, and so that’s not very hard. And so it was easy for me to write programs that looked as good as the programs on the computer I was using. And so in a couple months I thought, “Hey, I’m a programmer.”

When my kids wanted to learn to program when they were in high school they were using these great video games on the Xbox that look amazing. You can’t just sit down and write something like that. When I tried to show them how to write some simple software, they look at that, and it’s so far removed from the stuff that they see it’s like someone building a doll house compared to a real house. They can just see that it’s not the same and they look at how much they’re going to have to learn to get to write what they want. And it’s discouraging.

I’m on a mailing list with a bunch of engineers and this is actually a common topic, is as people’s kids get to a certain age, the kid is like, “Hey dad, how do I write a program?” I think JavaScript’s good because you can do things to show up in the web browser, most web browsers have a pretty good debugger in them. You can do some graphics.

Adam: Yeah. For approachability JavaScript’s nice. It’s right there. One thing that’s different about your stories of software development from my experiences of software development is just this kind of low level working with hardware and trying to like figure things out. Do you think that is missing from how people build software today?

David: I remember reading somewhere that Leonardo Vinci was probably the last human who could know all of human knowledge. Human knowledge has expanded so much that you could spend your entire life learning things and you would never come close to learning all the things that we have to know right now. And software is kind of expanded like that too. When I started programming when I was a kid, there weren’t that many languages, there weren’t that many processors, graphics barely existed. It was possible to learn most of what there was to know. In the iPod I knew most of the software that ran in it from the bottom to the top. The only thing I really didn’t understand was the mathematics inside the audio codec, which converted MP3 files into audio wave forms. And now I just don’t think anyone can do that.

I mean, you have full stack developers, which means they know how to write an iPhone app or an Android app and they know how to write some backend code that runs on a server and they understand the protocols that go in between, but there’s so much more that they don’t deal with. They probably don’t know how the TCP/IP stack works. They probably don’t understand the memory management inside. They probably don’t understand all the device drivers or the graphics package that puts everything on the screen. On the back end, they probably don’t understand the operating system that runs on the server.

Adam: They don’t understand like how to debug hardware with JTAG.

David: I mean, it’s great that there’s so many more parts of software to get interested in. When I started the web didn’t exist, and now the web itself is such a large specialty you can spend your entire career just working on web stuff and never get into any other part of software, and it’s enormous and complex. But it’s completely different than writing a bootloader that knows how to talk to the registers in the hardware chips on the mother board.

Adam: Or changing the file system from big-endian to little-endian. With the podcast, I’m on a bit of a theme lately of kind of going down the stack, and you are definitely on that theme. Somebody said, “Hey, the benefit of digging into the details and getting low level is the stack actually bottoms out. You actually hit hardware and you’re at the end. In many other directions, you can just keep going forever. You can start to look into something and never get back to where you started.”

David: Yeah, that’s true. At a certain point, if you’re at the bottom, if you’re talking to registers on chips that are on the motherboard, if you’re looking at the assembly code that’s being generated by the compiler, that is the hardware level. But on the other hand, there is stuff below that too. If you talk to people at Intel who code the microcode inside the processor, they’re even below the assembly line.

Adam: Yeah, no, that’s very true. If I spend my life writing Python code, but I’m a very dedicated software developer, I mean, do you think that my knowledge is too superficial? Am I missing something?

David: So I actually have changed my opinion about this. I was at Apple for 18 years and I interviewed a lot of people who were coming in applying to work at the company. And we used to do a big part of the interview checking to see how well you understood memory management and pointers and stuff like that, because that was all kind of key information to see are you going to be able to work in this environment? But as we moved into languages like Objective-C and then Swift, where the pointers are really hidden, I realized I shouldn’t worry about whether people understand all the details.

I have mixed feelings about that, because then you work with people who can’t read assembly language and you think, “Oh, they’re missing something.” But on the other hand, they don’t need to read assembly language because our tools have gotten so much better. So I have to admit it’s a step forward.

Adam: This is an interesting perspective. One thing I’ve learned from hearing all these varying opinions is that the context that you work in really matters. I’m not sure there’s black or white answers. Game programmers value different things from embedded programmers, who value different things from operating system writers, et cetera.

So that was the show. You can find some more details on this, including Tony Faddell backing up the story linked to on the episode page, which you can find in the show notes in your podcast player. There’s also a full transcript up with nice headings and links, and you can jump to different parts of the episode. I’ve been working hard to get professional transcripts and nice episode pages up for all of the podcasts, all the back episodes, and I’m finally done. A lot of people like to actually read podcast episodes, which I found a bit odd at first, but I’ve been coming around to it. So I tried to make these pages really nice and readable and shareable, and they have nice links so you can jump to certain sections. If you haven’t seen what the new episode pages look like, take a look. They’re also a great way to share an episode. Hint, hint.

Oh, one more thing. If you, or someone you know worked on some interesting and prominent or important software project in the past, and you want to share the story, spill the beans, let me know. Email me at adam@corecursive.com, put story and title there somewhere because I get a lot of strange email and that’ll help me filter it. Until next time, thank you so much for listening.