Adam: Hi, this is CoRecursive and I’m Adam Gordon Bell. Each episode is the story of a piece of software being built.
I remember a final project in university and undergrad that I worked on. It was a semester long, self-directed class, and it was one-on-one with my advisor, my professor. I remember being in his office and going over the code I was working on and showing him all I’d written and being so proud. This was in 2004, and my project was Enterprise Java Beans. It was written in Java, but really it was written in XML. There was XML config for ANT and there was XML for Tomcat deployment and for ORM configuration and XML described how the services communicated and there were schemas that validated the XML. There was XML logging for config. And honestly, I think he was confused about what it was all for.
And I tried to explain to him what I had learned online, that it was easier to change and you didn’t have to recompile things, and it was human readable, but machine verifiable and my project worked in everything. Originally, he had wanted me to do something related to search heuristics, but I wanted to do real world stuff. I wanted to do Enterprise Java Beans, and here he was and I’m in his office and he is trying to figure out the point of it all.
I ended up getting good marks on that project, but I’m not sure I ever convinced him about the merit of that approach besides him just saying, “I guess this is how they’re doing things in industry right now.”
But today’s show is about somebody who saw XML and said what my professor was too polite to say: “no, this can’t be it. This can’t be the solution. This is too complex.”
Adam: Microsoft was the big bad guy, both hated and feared.
Douglas: So they decided to have an alliance, and the first thing they agreed on was that Netscape would put Java into the Netscape browser, so they did, in the form of applets. So you could write applets in Java and they would run in the Netscape browser. Do you remember applet?
Adam: I guess I briefly remember those. Those were horrible.
Douglas: Yeah, they were absolutely horrible. The next thing Sun demanded was, you have to kill Mocha, which I think by that time had been renamed LiveScript, because you’re making this look bad. We’re telling everybody that Java is the last programming language you’ll ever need, and you have this stupid looking thing called LiveScript. Why are you doing that? This is just confusion.
Adam: Netscape had their reasons.
Douglas: They remembered something that Apple had made, called HyperCard. HyperCard was this very nice interactive scriptable system that Bill Atkinson had made and was distributed for free for a while, on every Macintosh. It was all event driven, all scriptable, with a built in graphic editor and program editor. It was this really nice thing, and a lot of people started programming in this thing and were writing in a style of programming that the professional programmers of the day thought was impossibly hard, which was doing stuff based on events.
At that time, all the professional programmers were blocking on input and they couldn’t understand how you could do anything else, but the beginners didn’t know that it was hard, and so they were doing amazing stuff. Some of that stuff eventually turned into best-selling products, like the Myst series, came out of HyperCard. So Netscape thought they could do a similar thing for their navigator browser that, if they could get people programming in the same way that they did on HyperCard, on the browser, but now they can have photographs and color and maybe sound effects, it could be a lot more interesting, and you can’t do that in Java.
Adam: But Sun was not happy about this. They said, “We thought we agreed that Java was going to be how you script the web.” And Netscape probably said, “Listen, we can’t rebuild everything to make it centered around the JVM. That’s too much work and this scripting thing, we have works and is beginner-friendly.”
Adam: Doug knows all this in retrospect, but at the time, all he knew was, here’s this new scripting version of Java.
Douglas: So I took their explanation of what the language was and I read the spec or what material was available at the time about it, and I thought, “This is appallingly stupid. I don’t know how they could do this.”
Douglas: We had built this amazing 3D world that you could interact in. You could have an avatar and you could customize the avatar and you could go walking around and teleport to places and encounter people there and talk to them like we’re talking and do things. You could exchange objects and play games or build things or buy things or whatever it is you want to do.
Adam: This was like Second Life or the Metaverse, but years before that, they had been working on it all throughout the ’90s, before the web was really much of anything.
Douglas: We were way, way, way too early to do this stuff. And the reason you’ve never heard of it is because we were way, way, way too early.
And we looked at the web, I looked at the web and I said, “The web is deficient in all of these ways, in obvious ways.” It wasn’t interactive. It didn’t give you enough control over the visual presentation. It was not secure, that’s a really big one there a lot of things that it couldn’t do that still doesn’t do very well. One of my big predictions that was completely wrong was that the web was not going to make it, because it was just so poorly designed.
Adam: For the online thing they were building, the monetization idea was to sell it to other companies.
Douglas: … Because there seemed to be interest in phone companies and computer companies and entertainment companies, all trying to figure out what can we do online.
Adam: Electric Communities was a Bay Area of BC funded social network.
Douglas: We had offices at the corner of, oh God, I don’t remember these names now, De Anza and something, we were catty-corner to Apple. Next door was Javasoft. There were a lot of tech companies in that area. Well, Stevens Creek Boulevard, that’s where we were. We were about 100 people. We grew and shrink and grew according to how we were doing with fundraising. But yeah, we were a fairly big company.
Adam: Their online metaverse thing ran on the JVM.
Douglas: And that was my decision, and I hope it’s the worst decision I will ever make in my life. I’m pretty confident that it is, but I still got some years left.
Java was problematic for lots of reasons. One was, it wasn’t secure enough. We had a set of requirements for security and Java did not need it, but we thought that was okay, because Java was open and so we could fix it, but then they reneged on the open thing and then we were trapped. So that was difficult, trying to make it secure within the restrictions that they put on us.
Adam: One of the big problems was startup time.
Douglas: So it took us three minutes to load the environment. Java has a class loader and it incrementally brings each class in and verifies it and puts it in memory, And we kept hoping that we’d get better and it didn’t get better.
So we decided to park it and we pivoted. We acquired another social media company called The Palace and started trying to figure out how to put ads on the palace in order to make money.
Adam: The Palace was a program you downloaded, and then you connected to a palace, which was like a server, but presented as a series of graphical rooms all around a certain topic. It was like a graphical chat room. This is around the year 2000 and everybody wanted to be online, but the web was static, and this was a attempt to make it more interactive and social. Fans of Korn or South Park or whatever was popular at that time, would join those chat rooms and show off their avatars and chat to each other.
Douglas: The difficulty we had was that advertisers were really skittish about putting their ads next to user generated content. What if we put up an ad and people say bad things about us? What are you going to do about that? And we said, “We won’t do anything about that.” And they said, “Well, we’re going to spend it somewhere else.”
Adam: Companies obviously, eventually came to terms with ads on user-generated content, but not soon enough for electric communities. And so, since the Palace wasn’t bringing in any money, they decided to try to do a consultive approach. They could build online experiences for other companies. So they took on a project from Turner Broadcasting.
Douglas: We’d had a long relationship with them, and they were going to do a children’s collectible tradable card game that would run on the web, and they wanted us to work on it. They knew about the Palace technology that we had, and we said, “Yeah, we could do that.” And so, the first question was, how do we put the Palace technology into the browser? One way would be to make it a Java applet, and in fact, we’d already done that twice, and they both turned out really poorly, was not happy with the results. So I didn’t want to do it a third time.
Adam: Turner Broadcasting is convinced, now Doug just has to tell his team.
Adam: This style of programming was not popular. OO was supposed to be the great hope for the future, but Doug started to think that maybe this was a game changer. Maybe Electric Communities were onto something, they could raise more money and start building a new interactive web, but as they started thinking about more fundraising, the .com bust hits and they have to start scrambling.
Dot Com Bust And JSLint
Douglas: It was a little bit brutal. I remember vultures coming into our offices and looking around, looking at the furniture and at the equipment and trying to scrape stuff off our carcass as cheap as they could. And we had an investor who offered us a bridge loan and was going to lead the next round of investment, but they reneged on that. So they ended up buying the company for the cost of the bridge loan. That was pretty painful. And that’s hard, because you invite all these people to come in and share the dream with you, and then, bye, we can’t pay you anymore. That that’s a really difficult thing to do.
Douglas: That gives you a lot of free time. So we’ve gone through most of the layoffs and just had to be around to sign things and let people in the building, stuff like that.
So if you don’t use those forms, then bugs are easier to detect and your programs are better. And so eventually, I came up with the principle that if there are two ways to do something, choose the one that’s less likely to cause errors. And that turned out to be a wildly radical idea, that there was a lot of resistance to that. Some people took it as an insult, like, “Well, obviously a stupid person could use that feature incorrectly, but to suggest that I can’t, that’s just mean.” A result of that was that JSLint taught me how to think about quality in coding, which was something I’d never really considered seriously before. So JSLint is much smarter than I am about coding.
Douglas: And you can use semicolons to disambiguate the grammar and make it clear as to what the intention is or to cause the bug to crash. Once I understood that, I started teaching, always use the semicolons. And it’s a lot of work. You have to go… At the end of every line, but it’s worth it. And over time, I’ve found others like, don’t use ++, because there are problem cases with that. Use +=1 instead, that seems to be much more reliable.
Adam: So Doug built up a list of these and adds them to his Linter, and eventually Electric Communities shuts down and Doug has to move on to whatever’s next. He knows exactly what he wants to do.
Douglas: So being in a startup, I think is the most fun I’ve ever had. It’s really great. You get to choose all the people you’re going to work with. You get to choose people who are really smart. Everybody is really engaged in the collective problem that you’re all trying to solve together. You don’t have any stupid suits getting in your way. It’s great when it’s working. It’s great.
Adam: So for Doug and Chip Morningstar, another Electric Communities alum, the next step is obvious.
Adam: They call their company State Software and they start building a proof of concept.
Adam: XML was the new hotness at this time.
Douglas: It was very popular. All the hipsters were excited about XML. There were lots of big companies that were promoting XML, including Microsoft and HP and IBM and Sun and everybody. And they all had these enormous tool stacks that were there to make XML less awful to work with. And we just didn’t want any part of that. That just looked like a waste of time. So we came up with this idea for moving data back and forth, and it worked.
Douglas: For me, the most difficult thing was raising money. You’re constantly going to Sandhill and calling on people who don’t understand what you’re doing, and are looking to take advantage of you if you can, and they’re going to do that, but you have to go on your knees anyway.
I found that stuff to be really hard, although some of them I really liked. And sometimes I’d be sitting in those meetings and I’d be thinking, “I wish I was rich enough to sit on the other side of the table, because what they’re doing right now looks like a lot more fun than what I’m doing right now.” And it was even more difficult raising money then, because at this point, the.com bubble had popped and all VCs had been hurt really badly by that. So they were only funding sure things at that time, in late 2001, early 2002.
Selling Single Page Web Apps
Adam: So State Software was only able to get one small investor, but Chip and Doug persisted in building their vision of the interactive web, and then they started trying to sell what they had built.
Adam: It’s like you had an electric car and they were like, “Well, where do we put the gas in?”
Douglas: It was very much like that, very much like that. There were some people who said, “Oh, we just committed to XML, sorry, we can’t do anything that isn’t XML.” And there are some people who were a little bit more open who said, “Well, we can only do things if it’s a standard.” And they said, “Well, it is a standard. It’s a proper subset of the ECMAScript standard.” They said, “That’s not a standard.”
Making JSON a Standard
Adam: That was it. That was the creation of JSON, which everyone is using today. But back then, everyone rejected it.
Douglas: He’s someone who should have known better. He had a website called scripting.com. His style of scripting came from a clever program that he had written for the Macintosh called Frontier, in which he had a scripting language and an outliner and a word processor and a database, all in one program. And the idea was that you could do virtually anything in Frontier with a little bit of scripting. And he was also one of the big promoters of SOAP, the Simple Object Annoying Protocol.
I don’t remember what the A was, but it might have been atrocious or abominable, I don’t know.
But SOAP was a big deal at the time. They were right in wanting simplicity. They didn’t accomplish it, but they put simplicity in the name as sort of an aspirational thing. And so, when I started showing how JSON works, he was really threatened by that. And on his website, which was well-read at the time, he complained that, “this isn’t even XML. We should find who did this in string them up now”, which was a really ugly thing to say.
Fortunately, nobody listens to Dave Winer, so I’m still here.
Message in a Bottle
Adam: And then, with what little funding they had and their troubles convincing customers, State Software had to shut down. The biggest thing they left behind in retrospect, was Doug site, json.org.
Douglas: It was like I put a message format in a bottle and threw it in the internet. I’d start working in television. So at that time, we were making the transition to digital television. Up until that point, we’d had analog television. And so, I was going to all the conferences on technology, on law and regulation and everything, and writing reports about my predictions about how the digital transition was going to go and when it was going to happen and what the standards were going to be.
Adam: So it seems like Doug has lost, or at the very least, XML and full page replacement has won. Doug’s often exile, writing reports for cable TV companies, and XML is just getting more and more popular. And then XML requests are added to the browser.
The AJAX Revolution
Douglas: Suddenly there are all these people writing for the browser who’d never been in the browser world before, and they were being told that you had to use XML to transmit the data. In fact, the X in AJAX was originally intended to be XML and said, “Well, this is too much work. What else can we do?” And they looked at JSON and said, “Oh yeah, that’s easy. We’ll do that.” And it started to take off.
Adam: This was a very pragmatic choice, but the XML gurus weren’t going to move on so easily.
Douglas: So there is an intellectual trap in XML, that it’s really complicated. It doesn’t look like it should be complicated. It’s just angle brackets, but the semantics of XML can be really complicated, and they assumed it was complicated for a reason. This problem of data interchange is so complex that there’s a conservation of complexity, that your format has to be complex, and the tools supporting that format have to be complex, so that the applications are manageable. And that’s a false assumption, there is no such thing as conservation of complexity.
That generally, if you push on simplicity anywhere, you can get it everywhere eventually. So simplicity is the thing you should be driving, not complexity, but they didn’t know that. And I think if you have bought into that, you have emotional connection now to this format, and you’ve been telling people that it’s complicated for a reason, and it’s good that it’s complicated. And then this little piece of syntactic fluff shows up, which is solving the same problems without any complexity at all. You look stupid. And also, you have invested a lot of time and energy into learning this thing, which turns out is becoming irrelevant. And that’s a tough thing. And if you’re a consultant, it’s even harder, because you’ve established a standing in the community that you have clients because this stuff and this stuff is no longer the thing. And so, a lot of them took this really, really hard.
Douglas: So they went to Yahoo while I was doing my consulting, and they called me up and said, “Hey, you should be working here.” Because at that time, it looked like Yahoo was going to become an interesting company.
Adam: So Doug started working at Yahoo.
Douglas: The Yahoo offices were in Sunnyvale, California, on the edge of the Lockheed Martin campus. I think they had four buildings there and several other buildings scattered around in nearby cities. It was a really nice place to work. The people were great, they were very friendly. I had the best job in the company. Basically, I didn’t have any real responsibilities.
Adam: Maybe he didn’t have official responsibilities, but he did have a goal.
Adam: It’s a complicated way to sell something.
Douglas: Yeah, well, sometimes that’s what you have to do. We didn’t have the authority in management at Yahoo to say, “We will do the right thing.” Instead, we had to trick everybody into doing the right thing.
Web 2.0 and The Conference Circuit
Adam: Talking about the same subject month after month, Doug found he had even more to say.
Douglas: I would talk about something and I would get a thought and I’d next time I gave that talk, I’d put it in, and eventually the talk would be so big that I’d take chunks out and make a new talk about the chunks I removed, and then I would keep growing again, and another talk would come from it. So it became the mother of a lot of stuff that I talked about. One of my favorite conferences was The AJAX Experience, in San Francisco. That was the first of the important web conferences, and it was several days, going very late into the night, and everybody was there, and it was so much fun and we were all influencing each other. We were all trying to figure out what this was about.
Online Booty Call
So one day I got an email from my hosting service saying I was using way too much bandwidth and that they’re going to put me in a more expensive bracket. And I thought, “Whoa, what? What’s going on? I don’t think I’ve suddenly become so popular. So what happened?” So I looked at my logs and I see I’m getting a whole lot of traffic from OnlineBootyCall, and they’re loading
json.js file and include it, which had a stringify function you could use.
So apparently I’d left a copy of
json.js On my server and they were loading it from there, which is a really bad practice, because if you load some third party script from another server, they get all of the privileges that you have. So from a security perspective, it’s a terrible thing to do, and they’re doing it and they’re costing me money now.
So I decide I have to stop them. So what should I do?
I wanted to do it in a way which wouldn’t hurt anybody, but would just make it clear that something needed to get fixed. So I decided to add a statement to the beginning of JSON.js, which was alert of the string important, remove this statement before deploying;. So I pushed that, and now suddenly everybody goes to OnlineBootyCall, they get this alert, and the people at OnlineBootyCall see that, and they go, “Oops.” And they fix it.
Doug’s friend saw the change in
json.org/json.js and asked Doug about it, and Doug told him the whole story, and his friend wrote it up and put it on his blog.
Douglas: And the guys from OnlineBootyCall posted a reply or a comment on Steve’s site and said, “Whoa, we screwed up. Sorry, Doug was right and we shouldn’t hotlink other stuff. We won’t do that again.”
Adam: So happy ending, except now Doug has to start keeping a closer eye on his server logs, because this website needs to stay up. JSON has never been more important, but also, things like Hotlinking can cost him a lot of money.
Douglas: And one day I noticed that JSON. Something else, maybe ActionScript or something, there was some other similar file that was still on my website. It was getting hammered by a site in Russia. I thought, “That’s weird, but I know what to do.” So I went to the file and I put this alert on it.
Adam: What should be happening is, all these visitors who are loading that file should be getting the alert, the same as onlineBootyCall and contacting whoever is doing the Hotlinking to fix it, but instead, nothing happens.
Douglas: I thought, “Whoa, that’s really weird.” So I made the file inert, nothing happened. Said, “Okay.” I deleted the file and now I have error responses and retries.
Adam: However this file’s being loaded, it clearly doesn’t differentiate error types. It’s getting a 404 and just retrying again and again, and that’s just making it worse, adding more server load.
Douglas: So then I thought: I know, I’ll navigate the page, I’ll change the page’s location and send them someplace else.
So I got these Russian guys, how can I teach them a lesson? And I thought, “I know, I’ll send them to fbi.gov and they’ll look into it and that’ll frighten them so much that they’ll stop doing this and leave me alone.”
So I did that. Next day, all of my websites are down, nothing’s working.
So I call up my hosting company and they said, “Oh yeah, you’ve had a security breach. Apparently someone got in and is using your site to do a denial of service attack against the federal government. And we’re going through the system, trying to figure out how they accomplished that.” And I said, “Oh, I did that. I’m sorry I didn’t intend to fight the federal government, sorry.”
The IETF and ECMA
Douglas: So they assigned a handler to me who definitely helped, gave me advice on which things were important and helped me with some editing. But overall, it was a really painful process. And in the end, I didn’t even get the mime type I wanted. I wanted Text/JSON. They gave me Application/JSON, which was weird because JSON is not an application, it’s a text format. It’s a way of representing data in text. And I think maybe that… Well, I don’t know why they’ve forced that on me. I’d like to think it’s because there were some XML fans who were resentful about what I’d been doing and decided that I didn’t deserve to have text that was reserved for XML. So they got text XML, but I got Application/JSON, which didn’t affect me personally at all, but for the people who use it, it’s not a big deal. It’s a little ugliness in the header.
Adam: How long did this process take?
Douglas: It seemed like forever, but I’m sure it wasn’t very long.
Adam: And then, I guess the whole process is drafting the document and then once they all agree on it, then they send out the actual [inaudible 00:41:19].
Douglas: They send it out and then it’s a permanent document or as permanent as bits can be. Then later, they came back to me and asked me to be the editor on a project to make an RFC, which would be the JSON Standard, which is a different document. And that was even more painful. So in the end, I left IETF and went back to Ecma, which publishes the EcmaScript standard. And I found that to be a much nicer group of people. And so, we published Ecma 404, which is the official JSON standard.
Adam: And from there, JSON became an ISO standard. And so XML fans can’t claim that JSON shouldn’t be used because it’s not a standard. But really by the time of standardization, and by the time Doug’s book started to get popularity, the tide had already turned.
XML Lost and Doug Won
Douglas: One indication of that is, you can go to Google Trends and you can do a query for JSON, XML. And you see two lines, as for the whole history, which Google keeps back to about 2004. And you see XML at the top, declining, going to zero. And you see this little JSON line going up, and they crossed several years ago, and JSON’s still slowly growing, and XML is still declining. The half-life of XML seems to be about three years.
Douglas: I’ve been all around the world, talking to wonderful people everywhere. Most everybody’s happy to see me. That’s always nice. So yeah, it’s been wonderful. I’ve had a really good time and a few of the things I’ve done have become influential, and that’s good too. I’m famous for a programmer. Programmers don’t generally become famous, but I’m fairly well known. Once I had experience, I was on vacation in Hawaii, met my brother there for lunch. He lives in Hawaii. And a couple of guys walk up and ask, “Are you Doug Crockford?” I go, “Yeah”, because they’d seen my videos. And my brother was really impressed by that, it’s never happened to him.
Embrace New Paradigms
Adam: One of the lessons Doug would like to pass on is, don’t be too tied down to the way you develop software. Embrace and explore new paradigms.
Douglas: I used to be a big advocate of Object-Oriented Programming. I used to sing in that church and I don’t anymore. I think Object-Oriented Programming was an important advance over procedural programming, which is what we had been doing, but we’re stuck. We’re supposed to change paradigms every generation, and we haven’t. So we’re way behind now. Object-Oriented Programming hit the mainstream in 1980. We’re in the 2020s and we’re still there. We should have moved on.
The Next Paradigm
Adam: So what’s next?
Douglas: I think it’s going to be actors.
Adam: Actors are all about message passing and failing fast. And they’ve had some success with Erlang, but Doug thinks this idea needs to be taken further. We need a new language.
Douglas: So for a while, I had ambition that I would be the author of the next language, but I have not seen any case where anyone who was successful with one standard, had a second hit. I’ve seen lots of attempts, like the guy who designed Pascal had a huge win with that, but he had intended it to be a teaching language, but it became a big application language, and it was not well suited to that. So he came back with another language called Modula, and then a sequel, Modula-2, which were much better languages, didn’t take off. So it seems to me that you only got one chance, and I got my chance and I won with JSON. And so, I am not going to be the guy who gets to decide what the next language is. The best I can hope to do is to influence the brilliant woman or man who designs the next language. And so, when I go out speaking now, that’s really who I’m talking to. I’m trying to inspire someone to make that language and to get it right.
Adam: And so, if you’re out there, next language creator, and want to know how to succeed, here’s some advice from a person who really did change how web programming is done.
Douglas: The success of JSON was totally serendipity. Getting the domain name definitely helped. There are some things that I didn’t do that definitely helped. I didn’t secure any intellectual property protection on it at all. I didn’t get a trademark for the name or for the logo. I didn’t get a copyright for the specification. I didn’t get a patent for the workings of the format. My idea was to make it all as completely free as possible. I don’t even require any kind of notice. No one has to say, “Thank you, Doug, for doing that.” It’s just free for everybody. And I think that definitely helped.
But what steps do you do?
I don’t know.
Adam: That was the show. Thank you to Doug Crockford for your time and for sharing your story with us.
There’s so much of Doug’s story I couldn’t cover here. His early days are fascinating. He worked at Lucas Arts, he worked at the Skywalker Ranch, he worked at Atari. And Chip, Morningstar, his State Software co-founder, before Electric Communities created Habitat, which was like this Metaverse or Second Life or something like that. But for the Commodore 64, it was so ahead of its time, it was practically from the future.
Anyway, I’m going to share some of the details of that either on Twitter or in the next supporter episode or in the newsletter.
And yeah, if you want to support the podcast, join as a supporter. Follow the link in the show notes.
And wait, there’s one more thing I wanted to find out from Doug.
Adam: What do you think is the XML of today?
They have gotten so big and so weird. People seem to love them. I don’t understand why.
And I think it could work for everybody.