Introduction
Adam: Hello. This is CoRecursive, and I’m Adam Gordon Bell. Each episode is the story of a piece of software being built.
Once, around 2012, I was working on this web application that I had inherited and I didn’t like the way the front end of it was written. It looked old, and it used some tech that I didn’t like. Don McKay, my frequent podcast guest, joined the team at this time, and we decided to just rewrite that front end.
We moved all the UI to Bootstrap, and we made some other changes. It turned out it was a lot more work than we thought, but it was exciting. We threw ourselves into that project, and we got it done. We got it done because it was our project. We took it on, and we told people we would get it done.
I share this because I feel like I get what it’s like for a group of people to get excited about something and give it all they got.
That’s what today is about. What could you accomplish if your teammates were all excited and determined to hit some project timelines? Today, I’ve got somebody from the Android team.
Coffee and Four Foot Tables
Adam: Yeah. Why don’t you tell me your name and what you do?
Chet: I am Chet Haase. I am a software engineer and have been for many years now. I currently work on the Android team at Google where I’ve been since about 2010 working on graphic software, animation, performance, whatever.
Adam: Chet is here to share the story of Android, the mobile operating system that powers the majority of phones worldwide. Let’s start on Chet’s first day when he was given a four-foot-wide desk in a large room full of other desks to start his work at.
Chet: That’s your space. There is no bookshelf next to it. There’s no wall for you to hang stuff on. There are cube walls, but it’s not high. You can see everybody else in the room. You are all part of that shared space there, which can be a bit much. They break up the space a little bit so that at least you’re not hearing someone on the speakerphone several rows away. It does help create an atmosphere where you do feel like you are working with the other people around you as opposed to everybody’s in their own office doing their own thing.
Adam: When Chet joined, they were working on Honeycomb, which would be followed by ICS, Ice Cream Sandwich, and then Jelly Bean. Android releases, by the way, have dessert-based internal code names. From that, you might guess that the Android team had a passion for desserts, but when Chet joined the team, it was a different passion that jumped out.
Chet: There were so many ways to get awesome coffee in the building, which I found a little bit weird until I understood how hard people were working, and then it kind of made a little bit more sense. But there was really good drip coffee sitting there that you could just pick up or there was an espresso machine. There was a Clover machine there. There was a pod thing or whatever.
I remember walking in one time. I walked up to the espresso machine or maybe I was just getting drip coffee, I don’t know. There was a guy using the espresso machine who … I don’t know if you’ve used these real espresso machines. You basically have a grinder to grind the things into the little dispenser, and then you tamp it down. There’s a specialized tool for pressing this thing. You’re pressing it just exactly right so that you get the perfect extraction, all that stuff.
So the guy walked up to the espresso machine, and he pulls his own metal tamper out of a holster in his belt. He had a specialized coffee tamper, and that kind of defined the culture to me. These people are really serious about their coffee, that they’re going to bring their own specialized equipment to use the already overspecialized equipment that is sitting right in front of them.
Adam: That’s awesome. Was that in a little kitchenette off to the side?
Chet: Yeah. it was just a little, we call them MKs, micro kitchen where all the snacks and drinks are.
Adam: In my mind, if you’re making a coffee that fancy, you don’t just go back to your desk and code away. You sit and appreciate it or-
Chet: No. The break was actually making the coffee. Having the coffee, that’s just something you do while you’re coding.
Bacon Sundays
Adam: Chet learned pretty quickly that this group was different than other software groups he had worked with in the past.
Chet: The thing that I remember the most of the time was hitting the ground running really fast and really hard. Everybody was fairly serious and focused on what they were doing. They weren’t just hanging out. They weren’t getting the coffee and sitting in the room and chatting away. It was like, no, everybody was working. But also the hours. I tend to be a morning person just because I evolved that way given some of the commutes that I’ve had over the years.
I was one of the few people that would be there early, but everybody else was there late. On those days when I would stay late as well, there was a lot of other people putting in those late hours or working when they’re getting home. You can see them checking in code whenever, on the weekends as well.
Adam: Towards the end of the Honeycomb release, Chet was introduced to Bacon Sundays.
Chet: For the few weeks coming up to a release when we really needed to hit the deadline and we had too many bugs and the release needed to be nailed down so that we could ship it, then everybody was encouraged, not made to, but encouraged to show up on Sunday so that we could pull together. There would be other people around to collaborate with to nail the bugs, to do the work that needed to be done. And then they would provide a catered breakfast.
In the café, people would show up around 10:00. They’d grab some nice food in the café, and then they’d make their way up to their desk and get to work.
Adam: They were working hard because they were fighting for their right to exist. Android is huge now. But in 2010, that wasn’t the case.
Chet: People were buying Android phones, but not hugely. We were one of the smartphones, but we were a solid fourth or fifth place in terms of the platforms that were out there selling and doing well. So it wasn’t really obvious that we were going to stick around. So the more and the faster that we could do our work, the more chance we had to actually stick around and do more of that.
It was more work harder, work more hours than I had put into any job prior to that. My main memory was I’m on the team and, all of a sudden, I’m writing code. I’m checking in stuff as fast as I can because the team felt, at that time, the more we could do faster, the more chance we had to actually survive. So that’s my lasting memory was that first couple of years was a serious amount of work.
Success Was Unlikely
Adam: In those early years, the odds were against Android. It was just a small team building a mobile operating system as fast as they could. At the very beginning, they weren’t even part of Google. They were a startup. I know the story because Chet spent four years writing a book, Androids: The Team That Built the Android Operating System. It’s a super good book and we’re going to be drawing on it here, but you should probably read it. Anyways, let’s start at the beginning.
Chet: The goal of the startup Android was to provide an open source operating system. They wanted to do was basically level the playing field, that they provide an open operating system. Then it allows other manufacturers, not just the company that acquired them, but everybody to take part in this ecosystem and use this common platform across all of those different devices.
Adam: At the time, in 2005, there was two models for phone operating systems. One was the RIM BlackBerry model where you sell a phone and then also create the operating system.
Chet: Windows had the other model where they did an operating system that they would then license. HTC or one of these manufacturers would build devices using an operating system, a platform provided by Microsoft, and then Microsoft would get a cut of each handset. But HTC or these manufacturers could not see the code. They didn’t really necessarily know where it was going in the future. So, to some extent, it was kind of a black box, but it was a black box that they had to port to their devices, which was a little bit tough.
It was a black box that they had to pay for, and the margins can be quite small in these devices. So what Android wanted to do was provide an open source thing. It’s going to be free to everybody and everybody can see the code. So it’s going to be easier to port. It’s going to be common across all these devices, so you can have applications working on different phones. There is no licensing fee, so it’s easy for people to get into.
Google’s Goals
Adam: That was the goal of the original Android startup. But Google’s goal was different and based on some predictions about the future.
Chet: It was clear given the numbers at that time. You could look at the graphs of phones being sold in the world and PCs being sold. Google, at that time, was totally focused on PCs. Everybody accesses the web through a browser on their desktop, laptop, desktop, workstation, whatever. That was how they got to Google services. But it was clear that in the future there would be a mobile ecosystem, and they wanted to make sure that they were there as well.
Adam: We want to make sure people can get to Google search, everybody switching to phones.
Chet: Imagine a world where, let’s take Apple as an example. Let’s say that Apple came out with their phone and, for whatever market reasons, they end up being the only smartphone available in the world. So now imagine, well, they’ve got the phone. They have a dedicated audience that is accessing the internet through their phone. You don’t have to imagine too hard to understand, well, what if they had their own search engine and their own ads infrastructure and their own services built on top of that?
Then, all of a sudden, Apple is the place. That’s the portal through which you get all of these services, and I think that was a future that scared people.
Adam: What do you think they thought the odds were though? Maybe they just thought this is a long shot, but we throw a couple long shots in every year or something.
Chet: Yeah. I see it more like that because, let’s be clear, Google funded this, but they didn’t fund it like this is the next direction for Google. They funded it like it seems like we should try this thing. They took flyers on things. I think it’s really good that they did, and maybe it’s really good that they did this in a small way because I think everybody was just totally focused on trying to make this thing work as opposed to you throw too much money at a thing and you build the team up too big and you’re going to end up spending all of your times in meetings trying to figure out what to do.
It seems better to start small and become a thing first before you bulk up and try to do it in a more polished way.
Acquisition Time
Adam: So Google acquired this startup. They funded them enough that they could hire some people, but that was about it. Actually, that was probably ideal.
Chet: Most acquisitions, they acquire the company. They try to integrate them into the company. There’s a social culture mismatch. The founders leave because they got a big fat paycheck. The engineers fritter out to other projects. They spend all day in meetings trying to align with other product directions and priorities. And then nothing happens. I’ve seen so many acquisitions just die on the vine.
Somebody made a lot of money somewhere, but I’m not sure that the results were what the company wanted. This one absolutely did. So they got eight people in the door. And then three and a half years later, they shipped 1.0 with 100 people and all the opportunities that we’ve had since then. One of the aspects that helped that work was the company basically walled off itself from the rest of Google. It ran as a separate startup within Google.
Google cut the paychecks. Everybody got paid for the work that they were doing. But everybody outside the leadership essentially ignored Google because they just didn’t have time. Google was working on different kinds of software. They had different processes. They had different infrastructure. They had different priorities, and they were not interested in having those conversations at that time.
Convincing Manufactures
Adam: Besides money to hire people and office space, the other thing that Google brought to the table was legitimacy.
Chet: In the startup, you might call that chip manufacturer and say, “Hey, I’d really like to try your stuff out.” If they answer the phone, they may say, “Yep, that’s going to cost you X hundred thousand dollars for a development board.” Whereas if you’re at Google, that company may be approaching you and saying, “Hey, would you like to try my new development board?” Because they want to make sure that if Google is serious about this thing, that they are in the front door in conversations with this company so they can be a part of this.
They took Android a lot more seriously as a part of Google than they did when it was on its own, barely funded. I think that was an important dynamic.
Adam: The first phone the Android team was to deploy their OS to once it was built was the Sooner phone, a BlackBerry type device made by HTC. But first, they needed to build an operating system and, for that, they needed a team.
Hiring
Chet: They didn’t acquire a huge company with an existing product. They acquired a company that brought over eight employees, only three of whom were writing code. The only code that they had was 3000 lines of JavaScript.
Adam: Hiring was trickier than it sounded though because, at this point, Google hadn’t announced it was building a mobile OS. It was still somewhat of a secret.
Chet: They couldn’t advertise and say, “We’re building a new phone platform.” So they couldn’t necessarily get all of the candidates looking at those job postings that they would have wanted because they couldn’t say what they were doing.
Adam: Also, embedded operating system people are not easy to find. There’s no time to train a undifferentiated developer on the embedded space. There’s just too much to learn. But here, personal networks and especially geography helped. There were three companies in the Valley that Android could recruit from.
Chet: One of them was a company called Danger, Inc., which was started by Andy Rubin, co-founder of Android. So he basically had done a smartphone thing before five years earlier when it was even harder and the devices were even more constrained. Several of the people that eventually ended up on Android had worked at Andy’s company at that time. So they had already built essentially a smartphone platform using the Java programming language.
They had an app store. They had backend services. A lot of the elements in Android had been done before in a different and more constrained way on Danger. So there was a group of people that had been there building the skills and knowledge to do something like that. There was other people that were at a company called Be, which was an operating system operating through most of the ’90s, trying to get out there and sell to people and didn’t quite make it.
They made their own hardware called the BeBox. They figured out how to port it to different X86 and MAC architectures, so you could just buy the BeOS. They pivoted and tried to do an internet appliance. That didn’t quite work. They kept trying these things, none of which really made any market penetration or any real money for them, but it did attract cult classic people. A lot of people used BeOS, and a lot of the Android people ended up working at BeOS because it was an exciting new operating system.
It wasn’t Windows. It wasn’t Mac. It was a multi-threaded, multimedia-centric, super interesting platform. It just didn’t happen to sell to anybody. But in the meantime, the engineers working there, again, built up the skills knowing how to build something like this, kernel operating system, UI toolkit, all the stuff that you need, the application layer, the public APIs, again, all of the elements that they ended up using later on in Android.
So Be was acquired by Palm, who spun it out into Palm Source to theoretically build their next OS. That didn’t end up happening. They did build an OS, kept building the skills for another few years. And then that company got acquired by a company called Access who came in and said, “We don’t want to do this anymore. Why don’t we do this other thing?” All of the people that had come from Be and carried on into Palm said, “We’re done. We don’t need to do this anymore.”
Adam: So these people were available. And then another operating system team was looking for jobs, the WebTV team. They had built an operating system as well and a full UI, but then they were acquired by Microsoft. So the people to hire were out there, but first Google had to hire them. And then they hit another problem. They weren’t the type of engineers that Google liked to hire.
Google’s Hiring Hurdles
Chet: Google was very focused, especially at that time, on more of a pedigree education. Not that there’s anything against it, but they really were going and interviewing at top engineering schools. They loved the idea of a Stanford CS grad or MIT or whatever it is, some of these better known engineering schools where people were coming out with really good CS degrees because they could bank on that and because they could.
They were Google. People had heard of this company. They had name recognition, and so they could attract this kind of hiring talent. That was the people that the recruiters would go out and look for. And then people would come in to interview, and they would answer these math and coding and algorithm questions. They would do well because that’s exactly what they had trained for in their CS education.
Adam: But the Android team needed different skillsets.
Chet: They need embedded engineers. They need people that understand how to write operating systems. There’s an OS course in a CS degree maybe. Not a lot of people take it, and not a lot of people explore that outside of course. So it’s really uncommon to find those kind of people. Those people aren’t necessarily the same kind of people that are going to nail one of those coding interviews because they have been working on different kinds of coding all these years.
So many people, especially on that early team, came in after hacking assembly code on embedded systems for years. Well, that’s a different skillset, or writing performance graphics code or writing UI code. This was not necessarily skills that Google was hiring for at the time. So there was this mismatch in who Google traditionally would look for and would approve through the interviews and in the types of engineers that Android really needed to hire.
One of the greatest examples is how many of the people on that early team, not only did they not come from a top engineering school with a top CS degree, some of them didn’t even have college degrees at all. In fact, at least one of them didn’t graduate high school.
Adam: This was a problem. Google had a unified hiring process. So these embedded people would go through a generic phone screen, and they would fail it. Or they’d go in for an onsite whiteboard interview and fail that.
Chet: And yet Android knew that these were the right people with the right skills to bring in. So they ended up having to essentially create their own hiring process to make sure that they were bringing in and then approving the people that had the right skills for Android even if that wasn’t exactly the right skills for what Google considered as more of a plug-and-play software engineer. That wasn’t the goal. The goal was to hire people that could write an operating system for a mobile phone.
It was critical that you bring in, let’s say, Brian Swetland. He didn’t finish college because he got too busy on his hobby projects working on Linux kernel stuff. He was much more jazzed about his side projects, and so school kind of took a backseat. Well, that may not look good to a traditional recruiter, but Brian Swetland was absolutely the best person to bring in to start the Linux kernel effort for Android.
I think the people that started on that early team knew, well, we need something different here because otherwise we’re just not going to start with the right people on this small and growing team.
Adam: Thankfully, they overcame the hiring process barriers and they built the team from experienced people from Be, Palm, from Danger, from WebTV. This was an operating system dream team.
The OS Dream Team
Chet: They knew what they were doing. They knew the pieces that they needed to build, and they knew how to work with each other to reach the end goal, which is a huge advantage over, I think, most new software organizations where you’re hiring random people. They need to ramp up. They need to get used to how things happen on the team. So the combination of those people that had worked at one of these places, some of them worked at a couple of them.
Hiroshi, who now runs Android and Chrome OS and all kinds of platform stuff, had actually worked at all three. But the fact that they had, A) built those skills and knowledge knowing what they needed to do to build a platform or even, in some cases, an embedded or mobile platform, and B) they’d actually worked together before meant that they could sort of hit the ground running. A lot of these engineers, too, the people that work on operating systems on this kind of platform, this is what they love to do.
They love to bring things up from nothing. So giving them an opportunity to come in and be on the ground floor to see it appear from zero into software running on devices, I think, was really motivating for them. Honestly, there’s not that many of these things around. People aren’t creating new operating system platforms every day, and so this was exactly the kind of project that they wanted to work on.
Adam: This excitement and this experience was a big key to getting as much done as they did in those early years. There’s less design and discussion work to do when everybody’s already done this before, like the app store, for instance.
Chet: The person that they ended up getting to lead that team was Michael Morrissey. Well, he had a team at Danger that had built an app store. I mean it was quite different. They weren’t using the same code. They weren’t solving exactly the same problems. But he knew, okay, well, we’re going to need a persistent data connection between all of these phones and the backend server, and we’re going to have a single connection. And then it can forward those requests to Gmail or calendar or contacts or whatever.
He had solved similar problems in a different environment. So just being able to say, “Okay, these are the pieces that we will need to solve here,” I think was a huge advantage to getting started faster.
A Platform Needs Applications
Adam: Swetland and the team start working on getting Linux to work on these early devices, and others are working on other systems programming work, like device drivers and power management. It turns out Linux has a lot of support for doing standard server type things, maybe even desktop hardware things.
But running on a lower-powered device with lots of multimedia, like audio and cameras and Bluetooth, well, there was a lot of gaps to fill. And then the team needed to decide how applications were going to be written for this platform.
Chet: When they got acquired, the only code that they had was JavaScript because they just needed this demo running, and whoever was writing it was writing JavaScript. JavaScript was common. So when they came in, they said, “Okay, well, obviously JavaScript is going to be one of the languages, but why don’t we do multiple languages? Why don’t we also offer C and C++?” Because many of the people on the team were embedded programmers and C and C++ allow you to get much more low level details.
You can really more manually control all of the allocation of memory and what’s going on. It’s much lower to the hardware. And then there’s also this job of programming language thing. This seems like a good idea because many developers know this language in the real world. So why don’t we do all three? All three makes sense. And then they got to the end of that year, and they realized we’ve got 15 people on the team.
We’re trying to accomplish an infinite amount of things. How about we just do one language? Why don’t we just try not to solve every problem in the universe because we really should ship 1.0 someday?
Building A Java Runtime
Adam: And then word came down from on high somewhere that that single language was going to be Java.
Chet: I believe the main reason for that was a common theme at that time was lowering the barrier to entry. By the time the SDK launched in late 2007 and they said, “Hey, programmers, try out our new platform,” programmers are like, “You don’t have any users. Why would I spend a lot of my life trying to figure out how to program for this thing that, by definition, has zero customers for the software that I’m writing?” That was the case until 1.0 came out a year later.
There continued to not be a lot of devices for some amount of time after that. But if you can at least make it as easy as possible for those developers, at least they don’t have to reinvent their programming lives in order to figure out how to get something to run on your platform. So if you give them a language that they already know because they probably used it, A) in school, and B) in their enterprise jobs elsewhere, then you’ve made it easy for them to understand the language of writing Android apps as well.
Adam: I mean the downside being, as you said, it’s a memory-constrained environment, and Java’s not known to be super memory efficient, I guess.
Chet: Right. So at the time, they had to write a runtime for it. Java, the language, needs a runtime. It’s a translated language. It’s something like the original slogan coming out of Sun was, “The right ones run everywhere.” That’s because the binary files that it produces are not actually binaries that run on the device. Instead, you have an intermediate layer, which the runtime on desktops and servers. That’s the Java VM.
But it basically translates between this bit codes, the Java class file format and the binary code that’s actually going to run on the runtime device, be it, in this case, a phone or desktop or X86 map, whatever it is. So you need this runtime running that is, A) translating, and B) also doing memory management. This is major difference between Java programming and C or C++ programming because a C, C++ programmer needs to know that they allocated that thing so that they can free it up later.
Java, from the beginning, has always managed that for you through garbage collectors. The downside, as you say, is, well, that means there is another thing running and maybe we don’t have as direct control over the memory.
Adam: So one of the things they did instead of just using the Java class file format directly, they translated it into something they created just for Android called DEX code, which ran in a VM they created called Dalvik.
Chet: Which allowed them to tune some things and be more optimal in a more constrained environment than the class file formats of traditional Java language would have allowed. So they created a new class file format that they would compile into. And then they needed their own runtime to be able to run that Dalvik class file format and translate that.
Adam: The scope of it seems wild. They’re like, “Yeah, we’re building our operating system. But yeah, we’re also going to build runtime for Java and our own bytecode format.” It seems like why not just throw that on while we’re building a operating system? Let’s build a runtime and a new binary format for Java code.
Chet: First of all, I should point out that this is similar to something that they had done on Danger. It was a different file format, but they also were using the Java programming language and they were also compiling into their own file format and running their own runtime. The people that came in, including Brian Swetland, but also Dan Bornstein who was on that early team, had done something very much like this before.
So again, it was a case of we know what we need to do. We do know the steps that we need, and we know that it’s physically possible. So then they just got down to work. I’m with you. If I pose myself that problem, I wouldn’t know where to start because I’ve never worked on a runtime. I know what they are in theory, but I don’t actually know how to execute one. So it would take me a very long time. That’s a very daunting project.
But you get someone who has been working on runtimes for many years and they know this is what a garbage collector is. This is what the runtime job is, and this is what a file format needs. They have, in their mind, different file formats and different options. I think they can break the problem down a little more successfully so that they can have that. One to two people essentially was who wrote the early Dalvik.
Over the course of a year or two, they had Dalvik up and running with a very tiny team because it was the right people with the right skills to understand what the problem was and execute it.
Adam: Yeah, it’s so wild.
Chet: There was a lot of work that happened at that time.
The IPhone
Adam: So the team was heads down working towards Android 1.0 and shipping their first phone, the Sooner phone, the phone they thought would blow the BlackBerry and Nokia phones out of the water. But then Apple, to much fanfare, released the iPhone, and the whole market changed.
Chet: So the iPhone came out and, all of a sudden, that kind of defined due to marketing, due to uptake, due to a good job, it’s a solid device and company shipping consumer products. It defines a smartphone as having a touchscreen. They didn’t invent the touchscreen. They weren’t even the first phone with a touchscreen. But the way that they did this or the way that people absorbed that made it a checkbox item so that if anybody wanted to compete in the smartphone market after the iPhone came out, it clearly needed a touchscreen on it.
So it was clear, after the iPhone came out, that they couldn’t launch that Sooner device because there’s no sense competing with a BlackBerry after iPhone exists. So that one was killed and they pivoted and said, “Okay, 1.0 device is going to be the G1. It’s going to be a touchscreen device instead.”
Adam: The G1 was the T-Mobile G1 in the US and the HTC Dream elsewhere. It had a touchscreen and a keyboard and buttons on the bottom of the screen, one for back, one for menu, but also a couple more buttons and a full keyboard that slid out from behind the screen and a trackball. This is a lot of stuff. This is a lot of peripherals to get working. So the team had a lot of work to do before they could ship a phone. But Android now had something unexpected going for them. The iPhone had changed the market.
Chet: So in hindsight, it seems like, wow, when the iPhone launched, that must have been really stiff competition. Absolutely. Really stiff competition from day one, continuing today, will continue in the future. But they also created an opportunity for everybody that wasn’t named Apple and wasn’t one of the carriers that they picked in each market. So when the iPhone launched, they basically had one phone manufacturer that benefited from the iPhone, and that was named Apple.
They had one carrier in each market in the US. It was AT&T. So if you’re not named Apple and you’re not AT&T in the US, what are you going to do? It’s not like you can say, “Well, I guess they won.” No, no, no. They want a phone, but they did not have a platform that was going to have the capabilities that they needed to compete against that phone, and they weren’t going to get them soon because none of these companies were platform-building companies.
They would build these one-off OSes, which were clearly not going to be sufficient to compete in this growing smartphone market with all these capabilities that smartphones needed, including the touchscreen. That basically created the opportunity for Android to come in and say, “Well, it’s not ready yet, but we are working on this thing.” And then the conversations with those manufacturers could start.
The Memory Pressure
Adam: So now all these carriers are interested, but Android still hasn’t shipped a phone. Meanwhile, the team also has some ideas about how people will use phones in a different way than how we use desktop computers.
Chet: Well, one of the recognitions early on that’s impressed me was that the team realized people are going to be running apps in mobile phones in a different way than we do on desktops. In desktops, you have these monolithic apps where you spend most of your time, be it Chrome or Microsoft Word or whatever. I’m going to be in this app and I’m going to do a bunch of things. Whereas they realized, well, mobile phones are different.
We’re going to have little micro apps, and these micro apps are going to interact with other micro apps. I’m going to go into this social application. Then I want to take a picture which is going to launch a camera application. And then I’m going to want to filter that picture, which is going to launch a filtering application, and they go bah, bah, bah, bah, bah, bah, between all these applications.
Adam: This quick switch really only works if multiple programs can reside in memory at the same time.
Chet: If each of these applications is coming to the foreground and everything else behind it is dying at the same time, it makes it much more difficult and painful for the user to go between these micro experiences because they’re always taking a long time to launch.
Adam: Now the team is back to struggling with memory issues. Every team wants to use all the memory it can, but that’s just not going to work.
Chet: It’s a multitasking operating system where you can run as many things as you want. But if something launches into the foreground and doesn’t have enough memory, then the operating system will automatically start killing stuff in the background until it has enough memory for the thing that just launched. So you’re always in a state where your application may just get killed. And then the next time the user goes back to it, it may take a little bit longer to launch because it has to be restarted.
So if everybody takes maximum memory, then you’re really not going to be able to have these multiple things running in memory. The user’s always going to be getting a slow start to all the applications, and it’s much better if everybody can be much tighter so that you can have multiple things in memory all at the same time. I’m going to get the numbers wrong here. But let’s say I think there was 168 megs available for applications to use for applications and platform. That’s all they had, which meant that if your application was coming in too big, you better work on it to cut it down.
Finding Memory
Adam: The web browser group was working weekends to try to get the memory usage down. Then Brian Swetland came out of nowhere with a simple fix.
Chet: He made a little one line change in the kernel that freed up 32 megs. So he basically hid the memory. He made a little change to the kernel that basically only exposed 168 megs to the operating system, so that’s all they had to work with. Even though there physically was more memory available, he just wasn’t giving it to them until the end of their release. By then, it was too late. They couldn’t expand to fill the space. Instead, they could just be a part of a much larger sandbox that was opened up to them right before they shipped.
Now, all of a sudden, you had a couple hundred megs on that device that only had 168, and that’s a huge change. He gave back 30% or whatever it is. People were not happy with Brian.
Adam: Was that a key to that release’s success that they had that extra room or was it just a dick move by somebody?
Chet: I don’t know. I mean there were so many aspects of that early release that were critical. I don’t know if this was one of them, but it was … I wasn’t there at the time. Some teams were very unhappy about this because they’d put in a lot of effort trying to cram things down into a fictionally small space. But I have to believe, especially on the limited devices like the G1 that launched with 1.0, that the ability for things to run in a much smaller, more trimmed down state, I think, was better in general.
You don’t want launcher to be killed all the time because every time you launch every application, launcher’s just taking up too much memory.
Adam: So with that trick, the team managed to ship Android 1.0, and the G1 started to be sold. It wasn’t actually a great success though.
The G1 and the Droid
Chet: The uptake of the G1 wasn’t huge. It was more like, “Oh, that’s interesting.” And then people went about their lives. In the meantime, there were all these other players out there with interesting phones of their own, including Apple was on a nice solid trajectory going up. So over a year after 1.0 came out, then in the US the Droid launched and that was the first commercially successful phone or at least what the team thought of it.
You could actually see significant numbers of people buying this thing, and there were ads out there that were really successful. All of a sudden, Android became a thing partly due to this device or the timing of when this came out with the capability of the platform or whatever. I think the team realized, oh well, maybe we’ll actually be here for a little bit. But six months after that, when I got there in May of 2010, the platform was still a solid fourth or fifth place.
Above us was iPhone, Microsoft. Nokia was shipping phones with Symbian OS, which some other phones, I think Sony was shipping some Symbian stuff. I mentioned RIM, BlackBerry. All these really big deep pocket players with very successful platforms were doing very well, thank you very much. And then Android was kind of coasting down at the bottom of the heap. So I don’t think that looked like success. That looked like, oh, wouldn’t it be nice if we still get to be doing this in a couple or few years?
But we had no idea where it was going. It was in that single year that I joined where everything changed.
The Industry Embraces Google
Adam: The thing that changed was that manufacturers were forced by the iPhone to up their game, and Android was now shipping devices. They weren’t iPhone killers, but there was potential there.
Chet: So when the iPhone launched, maybe they weren’t ready to really bet on Android yet. But by the time 1.0 came out and could demonstrate that the G1, maybe it wasn’t a consumer success, but it was a commercial success in that you could look at it and go, “Yep, I get it. I understand what Android wants to be, and I understand that they can be a real smartphone platform. If I need a platform to ship my device, maybe I should go have a chat with them.”
If you think back to that time, everybody had their own phone. They were seen as much more one-off devices where you might get that phone and then, next time, you’ll get a totally different phone. And then you need to learn how the contacts and calendar work on that thing. And then, all of a sudden, we’ve got this new idea where, no, no, it’s going to be a single platform. It’s going to work the same everywhere, but with different form factors and different price points. Maybe that’s the vision of the future.
Adam: So the manufacturers are on side, but there’s still so much to do. Remember, the Android team is years into moving as fast as they can. They really want to build this mobile platform though, and so the pace of shipping just increases.
Increasing Pace
Chet: I mean it’s crazy. If you look back at the timeline for releases, they were shipping major releases every two to three months for the first couple of years. This is the same level of release or even larger than we now ship annually because this is when all of the fundamentals were coming into place in this very early platform.
Really major things like internationalization support and supporting multiple densities and sensor support, things that now we just take for granted, of course, they’re part of the platform, well, in the early days, they weren’t. So they needed to be done. And then they would just be working on multiple ones in parallel.
Chet Joins
Adam: It’s around this time that Chet joins the Android team.
Chet: It was pretty exciting, I mean especially after a career of working on products that were buried in the enterprise somewhere or maybe didn’t succeed or just weren’t available for friends and family to see. It was kind of exciting to work on something that if this happens to work, real people in the real world will actually use the stuff that I’m working on. That was pretty motivating, I think, for me and for other people, but also just being around other sharp and motivated people working together on a thing.
To me, there’s a big difference between putting in a lot of effort and hours and time and energy into something to see if you can push it through and get it there. The flip side of that is a death march. I don’t know if you’ve ever been on one of those. This didn’t feel like that. It was more like I’m going to jam on this thing because it’s really exciting right now, and I’m excited to see what is possible.
Adam: To be clear, nobody was forcing them to work these long hours.
Chet: It was more that everybody could see this window of opportunity that presented itself just given the timing of the situation when things were coming out, and they were really excited about being able to offer something that could actually be in this market. They just wanted to see things work. So it wasn’t the people they were working for saying, “You need to be here.” It was definitely a communal effort.
I want to work with these people and, together, we’re going to put in entirely too many hours and see whether we can actually get to the end of the tunnel here and launch this thing.
Adam: Did the stress of it ever get to you? Was there times when the releases built up or you felt worn down by it?
Chet: I mean yeah, your energy runs out. I didn’t reach the burnout point. Most of the real push was toward the end of the release. And again, if you have that limited window of push, I’m fine with that. I think it’s a reality, to some extent, of software that there will be a time when you have a hard deadline that you have to meet and more than enough stuff to do to get there. So maybe you have to push a little bit harder for a limited time, and then you can ease up and say, “Okay, let’s take a break. Let’s have a real working schedule” before, unfortunately, it may come up again.
Hitting a Date No Matter What
Adam: Most of my experience has been cloud type things where it’s easy to release something. When you have a specific release date, I guess, what is the impact of a bug?
Chet: Yeah, that’s a very good question. I think this is what drove the team to push a little bit harder because the constraints on that type of product were very different than the constraints on the server web backend stuff that the rest of the company was doing. Because if they have a bug in search, they could fix it that afternoon. They can push a release almost immediately and everybody, by definition, all of their users will get that fixed immediately.
But now you imagine a phone. Well, we ship a release and that user buys that phone. They go off the grid, and that’s it. They will not get a fix because the manufacturer didn’t feel like shipping an update. They didn’t feel like taking an update, whatever it is. So you could not count on them getting it, which meant it had to be that much more solid by the time we shipped. Also, with hardware manufacturers, like I said, the deadlines that they had to hit were very serious because it’s not like, “Hey, we’d like to get this feature out by December.” It’s like, “No, no, no. Our partner is committing an assembly line to building these things that need software on them, and that assembly line starts running in December.”
If we delay the release for two to 17 months, they may go out of business or find somebody else to work with. We are going to hit December no matter what. So then it’s all hands on deck to hit that, and that kind of deadline pressure also doesn’t really exist so much in other software environments where, well, you don’t hit December, that’s fine. Or you could hit it with something that you could then update in January.
That dynamic just didn’t exist in the device world.
Adam: So yeah, the pressures on Android are different. Android did build an over-the-air update system, which itself is pretty cool. But mainly, you’re trying to hit a release date. You want your code flashed onto these phones that are then sold in stores all around the world.
“We Are Really Going To Need That”
Chet: One of those things in the Honeycomb release specifically was hardware acceleration. The Honeycomb release, you probably don’t know it because it hit very few devices in the world because it was intentionally aimed at tablets, which were new at that time. So this was the first tablet release for Android. Other companies, other manufacturers had shipped tablets, but they were using something intended for phones, and so the experience really wasn’t what we thought that it could be.
So we wrote a lot of stuff in order to make the tablet experience much better. But one of the things that’s kind of obvious about tablets is the screens are bigger. Well, if you have more screens, that means more pixels to fill, which means you’re doing more operations to draw that. The CPU on those devices, particularly at that time, just couldn’t keep up with the amount of stuff that we were throwing at it. So it was clear that we needed hardware acceleration in order to get the performance that would actually give people a real smooth experience on these devices.
So at an earlier stage in the release, my friend and colleague that had brought me in also doing graphics and UI stuff and he had proposed doing this hardware acceleration thing. They said, “Well, no, it seems a little risky. So why don’t we not do it in this release. We’ll do it in the next one?” So he backed off and we all had other things to do. And then very near the end of that release, one of the executives came and said, “No, it really looks like we’re going to need that. So how about you do that thing that you proposed several months ago?”
It was two months or something to ship. This isn’t two months to finish our part and then it sits in QA. It was two months to ship the release, and we’re basically taking the patient, pulling their brain out, building a new brain and putting it back in, and then sending them out to the real world. So we were checking in major fundamentally platform-changing things within days of that software release actually landing on phones that real people would use. That, to me, was like, “Oh wow, it’s different here.”
Adam: Was it a success? Whenever I’ve rewritten a major thing and then shipped it, it turns out a lot of the old stuff was important.
Chet: I mean, to me, there are always bugs in software. That is the one true thing about software is it’s impossible to write software without writing bugs at that time. It did generally work. It did the job that it was supposed to. We did have hardware acceleration. We did hit the performance that we needed to.
Adam: This is an impressive achievement, shipping graphics acceleration in two months when you’re a new team member. But that was just the speed that this team moved at back in those days. One way they did that was just by removing all of the overhead.
Process-Free Development
Chet: We had individuals working on large swaths of technology, and they were the only person responsible for it. There wasn’t necessarily a lot of design happening in committees and code reviews with many people and iterations through the changes and tests being written and tests being … That was considered maybe we don’t have time for this stuff. We really need to ship the software. But the fact that one person could get that entire thing in their brain meant that they were more in control of all of the code and maybe didn’t need some of that overhead because they could do the right thing.
They would also put in the time. When it went in and it didn’t work quite the way it was supposed to, they could then just plow the time and effort in to make sure that it was robust enough by the time that we shipped. That’s something that is difficult to do as a software project grows and you have much larger teams and multiple people working on the code base. You can’t have people just lobbing in random code into the code base because that person next to you needs to understand how that integrates with their changes in the same code.
It was less of a factor when the team was really small and nimble and therefore the amount of process and overhead was unfortunately quite small. I would say we’ve gotten better over the years just because we needed to. You need to with a growing team.
Adam: If you currently or in the past have worked at Google, maybe on search or ads or one of the tens of internal data stores I hear about, then yeah, Chet’s work life might sound a lot different from how you experience Google.
Chet: So I think of Google, I think of it as a more mature engineering organization in terms of they’ve got processes. This is the way that we write code here. This is our style guide. This is how we do code reviews. This is how we check in. This is the thousands of tests that are going to run on every line of code that you have. At least this is my theory. I should point out I’ve been in the Android org since I joined, and I actually don’t know anything about how anything else works in the rest of Google.
But this is my personal interpretation of what happens over there. In Androids, first of all, for very good reasons, they used an entirely different infrastructure. It wasn’t just capriciousness like, “We don’t want to use theirs.” Instead, it was going to be an open source operating system and they wanted developers to be able to access that, partners for the OS, but also developers to access the public APIs and build the applications.
They wanted, again, to lower the barrier of entry as much as possible, which meant they couldn’t require people to buy expensive tools to edit or to code review or to merge the code or all the stuff. They weren’t going to require source code repository licenses. Instead, they needed to use tools that were going to be free and available to everybody. So they built their own infrastructure and, to some extent, ignored what was going on in the rest of Google.
And then at the same time, they had these killer deadlines that I talked about. They could see this window of opportunity. If we get this available, maybe people will care. They were running as fast as possible. That meant that although code reviews are a really good idea and I would encourage everybody to always have their code reviewed, maybe not all of the code was being reviewed all the time in early Android.
Maybe people were actually self +2-ing a lot of their code because it was more important to them to get that thing in now than it was to send it out to the team and wait for approval because they had 50 other bugs to fix that day. So that was one aspect of Wild West was just code was just coming in as quickly as possible. Really good engineers, really responsible and they would chase the problems and fix them as they found them, but maybe there was a shortcut on what would have been a more robust and traditional process there.
There were maybe some unit tests being written on the team. But for the most part, it was not a test-driven culture at all because, again, that was seen as overhead we don’t need and can’t have time for right now. Instead, I know how the code works. I can mentally test it, and it’s going in now. There really wasn’t a focus on what I would say was more traditional software culture of this is how we write software.
We design it with a group of people. We have a design doc. And then we all agree. And then we write the code. And then we reviewed it. And then we iterate. We have tests, obviously, to test all with the functionality to make sure … No, no, that was entirely too much overhead for the team at that time. So they ran a lot more quickly.
Adam: Do you think they succeeded despite or because of?
Chet: I think it’s a mixed bag. I do think that them running very fast was a key element in Android being relevant at all. If Android had come out a year later, I’m thinking it wouldn’t exist right now because maybe somebody else would have come out in the meantime. I know Palm was working on webOS at the time. Some stuff was really interesting there. Maybe this would have given Microsoft more of an opportunity for some of their stuff.
There were so many players in the market. Which one of those would have come out with something that would have attracted manufacturers at the time that they needed it while Android was being more robust? So I do believe that running really fast and maybe cutting a lot of those corners was responsible for some of the timing that actually made it succeed.
Adam: I’ve seen code and been like, “Why did they cut these corners?” And then you realize the context and realize that I wouldn’t be working on this if it wasn’t done this way. It literally wouldn’t exist. The ones where they spent all the time and made sure everything was very polished and whatever, they never got out the door. That’s why I’m not working on those ones. I’m working on this one that actually was released.
Chet: Yeah, exactly. I’ve always thought that about Android where, yep, some of those early APIs are a little rough and we should fix those and improve them and offer better things over time. On the other hand, the fact that they’re rough means that it was able to succeed, which means that now we get to be around trying to fix them as opposed to it just not mattering.
Everything Changed
Adam: So that pace leading up to 2011, it really made a difference.
Chet: It was in that single year that I joined where everything changed. So by the end of 2010, by the time we got to 2011, not only had we continued to grow as more and more diverse devices came out all across the world from all kinds of manufacturers, but we also passed all of those other manufacturers’ lines. If you look at the sales of devices at those times, Android just, it did what we call a hockey stick. It just kind of changed slope to be much steeper, and it just kept going up.
By the end of 2010, nobody really talked about numbers in that market anymore because Android was there. Clearly, iPhone was doing very well. But if you look at the numbers, it’s like iPhone, they captured the market that they got very solidly. But Android had everything else because it wasn’t just this device in this market. It was all devices in every market all over the world. And then the other manufacturers, over the next couple of years, they just went away. And then there was only two.
Adam: The whole industry, the industry where they were outsiders changed just like that. And suddenly, Android was at the center of it. For example, Steve Horowitz, who ran engineering for the team, he used to go to the Mobile World Conference every year. It was a big get-together of handset manufacturers.
Chet: In those early days pre-1.0, he remembered being in meeting rooms and some of the people would come in and they would get it. They wanted to be a part of this thing. Some of the companies would come in and just blow them off like, “Oh, well, you obviously don’t know what you’re doing because you’re not one of the traditional players here.” And then fast forward five years and Mobile World Congress, the entire show floor was basically Android.
So the change from who is going to take us seriously and listen to us and be a part of this, and then eventually everybody that’s not Apple ends up making Android devices because that’s the way the smartphone market works.
Still Running Hard
Adam: Chet is still on the Android team 12 years later, and they’ve changed some things. They do code reviews. They have bigger teams. In fact, there’s more people now working on the Play Store, on the app store, than built Android 1.0. But the Android team is still hungry. Even me mentioning that they’ve been successful seems to make Chet itch, like maybe I’m tempting fate by calling them a success.
Chet: I think of success as I think Android was able to deliver this thing and stick around, but we still run really fast and hard because I’ve been in tech for many, many years. You may have a good thing now and who knows what’s going to happen several months from now? I’ve seen so many companies and technologies and products and entire platforms go away in a relatively quick amount of time because something new comes out and some fad or some new capability or some new platform that we didn’t envision comes out of nowhere.
And then it’s a matter of how fast everybody around can pivot to take advantage of that or to become a player in that new market, and maybe you can’t. I’ve been in companies that couldn’t. Their main products went away and they couldn’t figure out what else to do, and then those companies went away. So you look at those numbers. You’re like, “Well, yeah, I mean I guess Android’s doing pretty well as a platform.”
But I want to be very careful. Yeah, we’re out there now and we keep working really hard because we’d like to stay there. But I think we’re also very aware that competition is stiff and it always will be.
Adam: So you guys are still running hard, I guess? You’re still running your life depends on it? Or I don’t know.
Chet: Probably a little less manically certainly than pre-1.0 or even when I joined. But it’s a serious amount of effort knowing what we need to do to provide the platform that other manufacturers and carriers want and that users are going to want on their personal devices.
Timing, Luck and Passion
Adam: So what’s the big lesson here? What do I learn from the Android story?
Chet: Certainly, one of the big takeaways is pursue your passion. It’s clear to me people didn’t join the team and work that hard for many years because someone told them to do that, because someone was paying their paycheck. These are people that were really passionate about working on that kind of software. When they had the opportunity to do so in an environment where, oh my gosh, we could provide an entirely new platform in the world, wouldn’t that be exciting, then they just doubled down and worked harder than they ever had before.
The end result was they actually got to see that dream reach reality. Now, a lot of people don’t. There’s a undeniable aspect of timing and luck here. But I think if they had not had that passion, they wouldn’t have had that opportunity at all. At the same time, even if it hadn’t succeeded, they still would have had a good time working on this thing. It was really interesting for them to do this. They weren’t just writing code because someone paid them to write the code. They were writing code that was interesting for them. They were solving problems that were fascinating.
I think that’s one of the hidden benefits of being a software engineer. I hope nobody ever tells our bosses how much we like doing this or they wouldn’t pay us this much. Sure, you got to get the basics. It’s nice that you can get a paycheck. But I don’t think that’s why people were there at the time. They didn’t join Android because someone said, “Here’s a bucket of money.” They joined Android because this was the problem they wanted to solve.
So to me, it’s all about enjoying what you do every day. Maybe if you’re really lucky, you get to see success out of that thing, and you can be with your friends and family and say, “Oh, look at that thing that I wrote.” But maybe not. But in both cases, if you enjoyed doing it, does it matter that much? I think it’s more important that you actually enjoyed the journey because who knows whether it’s going to lead there, but you do know whether you’re enjoying the process of getting there at the time. It comes from looking at the end goal saying: Do I want to be a part of that thing? Do I want to march there with all of my compatriots running with me – next to me – in the four-foot desk next to where I sit?
Outro
Adam: That was the show. Thank you so much to Chet. You should buy and read his book, Androids: The Team That Built the Android Operating System. It’s also available as an audiobook on Audible. There’s lots that isn’t covered here. He interviewed over 40 people on the early Android team, and you really get a peek inside of their lives.
If you like the show, if it entertains you, if it brings value to you, which I guess it must if you’ve made it this far, consider donating through Patreon. That’s what encourages me to put time into it, and it tells me that you want more of it.
Also, this month for Patreon supporters, there will be a bonus episode where Chet shares some of his process of actually trying to get a job at Google. Getting on the Android team was not straightforward. It didn’t initially go so well. So please find the Patreon link in the show notes or go to patreon.com/adamgordonbell.
Until next time, thank you so much for listening.
Sticher
Overcast
Apr 02, 2024
57
min