Adam: Hello and welcome to CoRecursive. I’m Adam Gordon Bell.
Back in high school, during the dial-up era, my family had a computer, and so did most of my friends. We all connected through this chat site that was popular in our little circle. One of my friends had this paper with HTML-like handwriting on it in pencil.
If you typed it into the chat app correctly, which was hard because it wasn’t clear what all the symbols were, it would launch a bunch of pop-ups and crash your browser. It was like this secret hack where you could take down the chat site, and it was handed around from person to person.
Back then, nothing on the computer felt very real or important. It was just like a fun playground for teenage mischief, for pushing boundaries and messing around. There was once a fight that broke out at school over something that someone said in chat, but mostly it was just playing and a little bit of trolling, occasionally crashing people’s browsers.
But now the internet is just life, right? And when the internet became serious business, when it became part of reality, did we lose that fun?
Today’s show is with Nolen Royalty.
We’ll dive into his wild encounters with the internet’s chaos, how his creation sometimes got misused, and also his worry about being hacked and data breaches.
Nolen: This was happening on Monday evening, and my girlfriend had made dinner.
I thought that I had a data validation error that let people stuff strings into this bit set because I wasn’t, I hadn’t really even thought about what it was that my script did at this point.
And I had kind of like, gotten re-obsessed with like, what is going on here, right as dinner finished. And she was like, can we like, sit down and eat? You’ve been very busy recently. And I was panicking about like, there are these URLs. I have been hacked. I have been hacked.
Adam: But before we dive into the hack, let’s rewind to a moment in his past that reveals something crucial about him. A moment where he pushed the boundaries, expected punishment, but got something entirely different. Something that might just explain why he’s drawn to the chaos of the internet in the first place.
Nolen calls himself a deranged game builder. He dropped out of college, he worked at AWS, and then Jane Street, a trading firm. But today it’s not about that, right? It’s about the deranged game building part of Nolen’s career and his exploration of the bounds of computer systems and what you can do.
Boarding School
Adam: And it all started back when he was in boarding school, where they had something called the Student Wide Information System. It was like email, and it was chat.
Nolen: But point being, it was, you know, an important part of student life, and also it was the way that you turned in your homework. It had like a weirdly powerful mail rule system. So you could write rules in the same way that, like in Gmail, you can filter your mail or whatever, right? You could write rules that said, like, when I get a message from Richard, apply the Richard tag. The rules could take a lot of actions, and in particular, they could send emails.
And one other quirk—the reason that I kind of got into doing the thing that I ended up doing—was that when you got a new email, it made a pretty loud beep. And I had this friend, Brody, who always had headphones on during our study hall. We had study hall for two hours a night, um, Sunday through Thursday. And whenever he got an email, he would like jump because the noise was loud and he had headphones on. And I thought that was funny. So it’s kind of fun to like send him a message and see him jump.
Adam: Then Nolen started tinkering with the email’s rule system.
Nolen: So what I did is I wrote a rule that said, whenever I get a message for myself with an empty subject, one, send a message to my friend, Brody. Two, delete the email that I just got from myself. And three, send myself two more emails that also have empty subjects. And so then I would, I would enable this rule and I would send myself a message.
And then it would send Brody 2, and then 4, and then 8, and then 16, and, you know, pretty quickly a lot of messages. And he would get, like, ding a ding a ding a ding a ding, right, and, like, jump out of his chair, fall over, and it was very funny to me.
Adam: Brody fought back, setting up rules that would delete Nolen’s emails before they could cause a beep. And then, he made rules that would send things back. Nolen tweaked his emails to slip past Brody’s defenses. And so, the nerdiest battle began. Each of them writing rules to outsmart the other.
Nolen: And that was all totally fine and fun, and we did this for like, you know, a week, or something. And then one day, both of us, just because whatever, our defenses were good enough, forgot to disable our rules. So normally you would like enable this for like three minutes, and send a lot of emails, and then it was fine.
But instead, we just totally forgot, and we like went to bed, and we woke up the next day, and the school email server was down.
Adam: No one could send emails. They got the system back up briefly, but the rules were still active, causing it to crash repeatedly over a couple of days. Eventually, they traced the problem to the rules and informed the dorm supervisor that Nolen and Brody’s battle was behind the email chaos.
Nolen: And he came to me and was like, what’s going on here?
And I explained, and he was like, that’s hilarious; that’s so funny. Uh, and he made me a t-shirt.
But, I was kind of proud of finding this thing, and I think it certainly made me, I don’t know, there’s like some through line from that to plenty of the stuff that I do now, in terms of like, how can I misuse this thing in a way that’s kind of fun?
And so, being encouraged then I think was pretty nice.
Adam: Imagine expecting detention, expulsion even, and instead being met with laughter and encouragement. This supervisor didn’t just forgive Nolen’s digital mischief; he seemed to celebrate it. That moment was a hint of what was to come for Nolen: the New York Times interviews, the media buzz, the Washington Post, and the speaking gigs; they all trace back to his knack for bending things outside their normal use.
Deranged Game Maker
Adam: So fast forward, Nolen is an engineer, then a manager at Jane Street, the quant firm. He builds up a bankroll that’s large enough to quit and do something that he’s always wanted to do: make video games.
Nolen: And I had this big list. I had spent all of 2022; every time I had a game idea, I would put it in the list. And so I had this big list of ideas that I thought I would be working from for a while. The first thing I made was Tetris, just to kind of like get an understanding of how to put objects on the screen in Godot. A thing I decided pretty early was that the goal was to build lots and lots and lots of small games. I understood that one common failure mode for people who try to make games is that they’re like, Oh, I really love whatever. This is a silly example, probably, but like World of Warcraft. So I’m going to build a big MMO.
And don’t think about how that takes years for a team of, you know, hundreds, and it’s probably not a good starter project. You learn so much by finishing a game and seeing people play it or playtest it. And the thing you want to do is get a lot of reps in of that. So it’s like I’m gonna make games that take me like a week or two as fast as I can and blog about them.
And there’s a sign right above me that says, “The plan: 1. Make games as fast as you can; 2. Blog a lot,” that I put on my wall to kind of, I don’t know, give myself some goal.
Adam: That sign is still there. And so Nolen builds games, but it’s harder than he thought.
Nolen: Like, it’s actually really hard to just like go to your little, I don’t know, second bedroom and, and sit at the computer all day and write code and not talk to anybody about it.
And you can tell your friends about it, but like, I don’t know, your friends aren’t working on it with you, and you probably can’t talk to them as much as you would talk to your co-workers about what you’re working.
Recurse Center
Adam: So Nolen decided to apply to the Recurse Center, a place that’s kind of like a writer’s retreat, but for coding.
Nolen: It’s, you go there and you write code for 12 weeks or six weeks if you do a half batch. And when you join, there are a bunch of people who have been there for six weeks to kind of show you the ropes. Then they leave six weeks in, and now you’re showing people the ropes. The goal is just like, write as much code as you can every day, according to whatever interests you, right? So like when you apply, I think they do like a basic coding test thing, but mostly they’re looking for like, can you write code?
And do you have an idea of what you want to work on in a very self-directed way? So I applied, and I told them I was going to make 8 games in 12 weeks. And I showed up and I started making games.
SQL Pairing
Adam: The Recurse Center wasn’t just about solo coding marathons either. They had workshops. Nolen joined one on pair programming, where his task was to create a small game.
Nolen: They told we were solving, writing a solver for the game Mastermind.
Adam: Yeah, with the mines.
Nolen: No, not Minesweeper.
Adam: Mines. Oh yeah. Mastermind is like the colors.
Nolen: Yeah, the colors. So it’s actually very much Wordle. Like, Wordle is Mastermind, but there’s only one player, and you’re guessing words. And Mastermind is a two-player version of Wordle, where you’re guessing the colors of some pegs that somebody’s hidden from you. But it has kind of similar mechanics.
Um, but they told us we were going to write a solver for Mastermind or like a backend for Mastermind or something. I submitted that I wanted to work in SQL, and they matched me with someone else that could work in SQL. So we wrote a Mastermind thing that worked entirely as like a big SQL query.
And it was really fun, it was like a huge blast, and we showed it to a bunch of people, and everyone was like, yes, this is like, this is funny. We’re glad that you did this. And that was very exciting to me.
Wordle
Adam: Nolen got inspired to create a game using OpenSearch.
Nolen: That is the way that your computer knows how to search a specific site. So if you’ve ever gone to like, you’ve typed Wikipedia into your search bar and hit Tab, and then typed a query, and it’s like searched directly over Wikipedia, right? The way that works is that under the hood, when your computer visits Wikipedia, Wikipedia serves it a document that says, if the user wants to search my site, hit this endpoint.
The spec happens to include this crazy thing where, optionally, browsers can choose to support search suggestions. So just like when you type into the Google search bar, it guesses what you’re, what you want to search for. Your website can offer those suggestions. So you say, hey, whenever the user types a new character, hit this endpoint, and I’ll return some ideas of what they might be searching for, along with, like, URLs to directly visit that page.
So the whole bit was that you can return whatever you want from that. So why don’t you return Wordle answers based on the Wordle guesses that they’re typing into the URL bar, if that makes sense.
Hacker News
Adam: So Nolen took this idea and created a Wordle-inspired project. Then he wrote a blog post to break down how it all worked.
It’s nice to have a website where you can point to everything you’ve made, but I think it’s also just a really nice way to, like, share, codify what it was that you learned from a project and explain what you were thinking. And so for this, I made a blog.
Nolen demoed this at a Friday presentation at the Recurse Center where anyone could grab a slot and show what they were working on.
Nolen: And I think that was the first time that, like, I saw that people enjoyed something that I made as I like presented it, and the response was like pretty good.
Getting positive feedback on that was really fun.
Adam: Someone at the Recurse Center shared Nolen’s blog post on Hacker News, and then it gained traction. Encouraged by this response, Nolen kept going to the Recurse Center and he just kept making games.
Traditional Game Making
Nolen: I made Uh, several more kind of more traditional games, you know, games with like high scores and guys that you controlled and you like shot at enemies.
I remember taking a game of mine, a more normal game of mine, to a playtest.
And being there with some friends from Recurse who had brought games that I thought were much more interesting than what I had made.
And Being at this playtest event, and getting feedback on my game from some people, and just being like, embarrassed to be showing the game. Being like, oh I really, not like it was a terrible thing, but just that it so clearly wasn’t what I wanted to be doing. I wanted to be one of my friends showing these much weirder, more experimental things.
And I was getting feedback on like, How the numbers in my game could be adjusted and like oh, man I just really don’t want to think about like balancing numbers in games that’s not where what I’m what I’m what I’m interested in what I’m passionate about and so I think like the the big thing there for me was Learning that what I wanted to be doing was this kind of weird stuff And also that I was just better at it.
Like, I’m very, very into taking a weird bit of technology and pushing it as far as I can. I have zero patience for picking nice numbers that make a game feel balanced. I don’t want to work on games that need nice numbers to feel balanced. I want to work on, like, weird experimental stuff.
Flappy Dir
Adam: He wanted to do the stuff that bent the rules, that surprised people, that maybe even broke things, a little bit in a fun way. Just like that original boarding school email prank. But now, he was channeling his creation powers into this, and after 12 weeks, Nolen’s batch ended. But he kept showing up at the Recurse Center every day.
Nolen: I came up with this idea for making a game inside macOS Finder, by renaming files and doing a bunch of other hacky stuff, um, which turned into a version of Flappy Bird that you ran inside macOS Finder.
That, I think, was like the first time that I remember making something and like, realizing after, like, I demoed it at Recurse. I was no longer in batch, but you can still show up to demos whenever you want. And, like, people, like, lost their minds. Like, it was, you know, like, the reception was really, really, really good. A bunch of people wanted to talk to me about it after.
I had also just like loved making this thing. I thought it was so, so fun, and I loved talking about it, like getting to present it and like the story of how to make it faster was really fun. I got a bunch of advice from some friends on like the problems, and some people helped me solve that. But that was the first thing I think I made that I was like, Oh wow. Like this is, this is I think actually good.
Adam: Meanwhile, Nolen is wrestling with his direction.
Nolen: I like having a plan, and when I don’t feel like I have a plan or a goal, objective function, it’s hard to know that I’m making progress, and I do need to be making progress.
I think was really, really challenging at points because I would wake up sometimes and be like, what am I doing? Like, what in the world? What’s the goal? Like, what am I trying to do? Does anyone like me in any way? Like, what? And, um, I sometimes still feel that way and find that really stressful.
Adam: But then FlappyDir was released, and it went to the top of Hacker News. Then it was on Daring Fireball. GitHub has a YouTube channel, and they covered it on that.
From this experience, Nolen learns to refine his taste a bit. There is an appetite out there for these sorts of conceptual weird games, and so he keeps building.
Stranger Video
Nolen: And then , I made this game called Stranger Video, which I think, uh, is one of the things that I’m still probably the proudest of. , and Stranger Video was a website that showed you live video of a stranger’s face until one of you blinked.
And that’s all it did. It used MediaPipe, which can do, it’s a library from Google that has face detection, to crop video to only show faces and prevent you from showing anything else. And also to detect whether your eyes were opened or closed. Uh, that was it. Like, it was just a website with silent video of somebody else on the website.
Where all the restrictions were kind of coming from this place of, like, I’ve used chat roulette, I understand what happens when you just have strangers look at each other on the internet and what they choose to show. And I don’t want them to be able to show those things. I want just only faces.
Was also kind of thinking at the time about like some social anxieties I have around, like, when do I make eye contact with people that I’ve noticed far away and kind of wanted to channel that into a game.
Adam: So Nolen releases this game, and for some reason, it went viral in Japan, like, really viral, for a couple of days. And then even some of his friends were very touched by it. Martin, one of his friends, said he never had so many people look at him and smile in his life.
Nolen: And that was like, this profound like, oh wow, like that is, I’ve delivered, I’ve given Martin this like, I think genuinely cool novel experience, and kind of opened my eyes to like a new thing, type of thing that I could be building.
People playing with the restrictions was really interesting.
So, your eyes needed to be opened, but you could put pieces of paper over your eyes that had little eyes drawn on them, and that tricked the camera. And, like, if this was a competitive game, that’d be a problem. But for a weird social experiment, that’s actually just funny and fine, I think.
Like, I made this thing, and I tried to figure out why it worked, and why I liked it so much, and how to do more things like it.
And that was certainly a moment of validation and of like, okay, like I’m proud of this and I want to make more things like this. And people like it when I make these things and I think it has artistic value.
Talk Paper Scissors
Adam: So the next constrained collaboration game was Talk Paper Scissors. The phone version of Rock Paper Scissors, but with strangers.
Nolen: So you, like, call a phone number, it asks you to say, Rock, Paper, Scissors, use voice detection so that you can’t say anything but those words, place your audio to someone else, place their audio to you.
And that wasn’t as much of a hit as Stranger Video, but again, got, I don’t know, like 50,000 calls or something. Like, enough calls that I, it was kind of validated again, like, this is, people like to play with strangers.
There are not a lot of environments that are obviously restricted enough that you feel comfortable doing it. Both games thrive on constraints, right? They connect you to strangers, but they have limits on how you can interact.
And I think in particular, I had this theory of, like, the trick to this is constraints, and the cool things that come from this are how people subvert those constraints. That, like, you add constraints to constrain the assholes, but also that gives people material for, like, funny things to do that wouldn’t be possible with more of a sandbox.
But since kind of the thesis here is like simple interaction with constraints that prevent jerks, it didn’t really feel like something I could force. You need to come up with an interaction that you think is going to be compelling and fun and maybe silly. And you also need to come up with a way to constrain the obvious ways of abusing the action.
NYU ITP Summer Camp
Adam: Stranger Video and Talk, Paper, Scissors weren’t just games; they were experiments in constrained chaos. Nolen was almost designing for playful subversion, for people to find the edges and to push against them, just like he had in the early days.
Meanwhile, he craved community, that interaction that comes with building things together.
So, he went to NYU’s ITP summer camp.
Nolen: which stands for like, I actually don’t know, interactive telecommunications program or something. It’s pretty old, so they probably used the word telecommunications. But it’s a school that’s focused on like interesting artistic mixed media stuff. tech art things. It’s like a real, you know, you can get like a master’s there. I don’t want to get a master’s, that’s a lot of time and money. But they have a summer camp that’s like hang out there for three weeks with people that think it would be fun to hang out there for three weeks, which like.
I don’t know. I probably am going to get along with most of those people. So I did that this year because I didn’t want to do Recurse again.
Adam: At the summer camp, Nolen dove into face tracking tech. It was fun, but it was a lot of work, and he couldn’t release anything. And it left him feeling eager to just get something out the door.
Neal Fun
Nolen: I think I feel really good when I’m regularly releasing stuff, and when I’m not, I I get kind of down on myself and I got lunch with my friend, Neil, who makes the website Neil. fun, which is a really lovely collection of fun, different games.
Um, Infinite Craft is a very popular game of his from, from this year.
Adam: like you two should know each other.
Nolen: Yeah, we’re both in Brooklyn. I think like a surprising number of people that build funny stuff like this are from Brooklyn. But at the same time, that scene is not that feasible to know a lot of those people.
But I think we originally met via a friend of mine from Recurse, and then, like, I don’t know. We’ve got common interests and sometimes get together to bounce ideas off each other. I think explicitly the goal was we were bouncing ideas off of each other. We had originally planned to do kind of a Stranger Videos style thing together.
And then he made Infinite Craft, and that went crazy. Like, that went completely insane. So he was busy for months, and I was doing other stuff. Then that calmed down, and we were like, okay, we’ll come back and we’ll collaborate on this thing.
I pitched one million checkboxes just as kind of a joke.
One Million Checkboxes
Adam: The pitch to Neil for 1 million checkboxes is simple. It’s a website with a million checkboxes on them. If you check one, it’s checked for everybody else. That’s it.
Nolen: And we chatted about it for a bit, and we’re kind of like, Eh, this is funny, but I don’t think there’s that much there, and I don’t think either of us thought it was, like, a bad idea, but we were both kind of like, this is not the thing that we should collaborate on. But, I thought it was funny, and kind of interesting, and I knew I could build it really fast. Like I knew it was just not that much work. I know how web sockets work. I know how to put checkboxes on a web page, and I was super hungry for like making something quick, right? Like I said, I hadn’t shipped something in a while.
And so near the end of the conversation I was like, hey, do you mind if I just go build this? I think it’ll take me a few days and then we can like collaborate on one of these other ideas like when that’s done, and that was fine.
Adam: After that Friday lunch, Nolen spent the weekend figuring out how to build 1 million checkboxes in a way that would feel responsive in real time. It’s a lot of checkboxes—a million of them—and they need to stay in sync across all the users.
Nolen: How do I only render the checkboxes that you’re looking at? I’ve never built an application that uses web sockets of this scale. Like, can I sketch out an architecture that will handle the couple hundred people that I think will play and that I could scale up if it was a little more popular than that?
And so I kind of came out of that weekend with a plan. And then I built the site in two days. Um, you know, I actually wrote the code, um, that, the next Monday and Tuesday. So on Wednesday, he launched it, the million checkboxes, and that’s when things got interesting.
I was excited about it, I thought it was funny, I was proud of like, the constraints I had added in particular, In ways we can talk about later, made it difficult for people to draw because I understand what happens when you let people draw on the internet.
And so I was proud of all of that and, and thought, you know, it was evidence that I had kind of learned how to make websites, which wasn’t true at the start of the year, so I was proud of that too.
Adam: Nolen thought it was just a curiosity. Unlike Stranger Video, all you could do was check a box. There wasn’t a lot to do. But somehow, it took off.
Nolen: I think it was first Mastodon. My theory is like, it was Mastodon, and then it was some big names on Mastodon posted about it.
And that got several people to post it to Hacker News, and Hacker News collects votes if you repeatedly submit the same link in quick succession, and that got it onto the front page there. And then I think, like, once it was kind of popular, the fact that it was kind of popular drove the popularity. Like, you went to the site, and oh my god, like, all these boxes are changing all the time.
The site is so much more compelling because it’s weirdly popular, and that kept it more popular. I don’t know. I think like the first three hours were me noticing like, wow, like there’s some activity on the site and then, oh my God, like there’s really activity on the site and then, oh my God, my servers are going to like, this is going somewhere.
And if I don’t start working, the server is going to fall over, and I need to stop checking the load and start writing code, and that was, I don’t know, that was the morning. But it was, it was a very fun, if still slightly confusing thing to wake up and see.
Demo Day
Adam: That week marked Nolen’s third at summer camp, and it was time to show off what he had made, his new spin on Pac Man.
Nolen: And the camp show was that Wednesday night, like you’re supposed to be presenting something that you had worked on there. I was planning on presenting my Pac Man game. And instead, the game had gone like pretty viral by that point. And so, I was manning my booth, trying to show people my game, but mostly, like, spitting up more servers, trying to keep the website online.
Adam: How many people did you reach?
Nolen: So, the numbers here are a little fuzzy, for two different reasons. The first is that, um, I originally, so I logged everything. But when I wrote the logging code, I was like, there’s no way that more than a million boxes are checked in a day. So my logs will be truncated at a million lines a day because, like, that’s an insanely high number. Like, that’s stupid. And I didn’t think about that until like 5pm or something. So I lost all my logs of what boxes people checked for the first several hours, which is a bummer, so my estimate of players from the first few days is from looking at my Nginx logs.
Which, you know, I have like IPs, etc. And I think there’s something like 500,000 unique IPs there. It’s hard to know; you know, sometimes people are going to be like many users are behind the same static IP, uh, at a school or whatever. In the first few days, on average, we checked something like 50 million boxes a day, which is just totally, totally nuts, and well beyond expectations.
Adam: 50 million checkboxes, half a million people. These numbers are wild, right? Picture the largest stadium or music venue you’ve ever been to. It’s more people than that. The largest stadium in the US per Wikipedia is Michigan’s big house, and that’s a hundred thousand people. We’re talking 500 here. That’s, that’s five times that many people over the course of 48 hours.
And these half a million people fall into two camps, right? There’s the checkers who like to check things and watch the score go up, but then there’s also the uncheckers, unchecking the boxes and driving the numbers down.
Nolen: Like, I think you can look at like media coverage of the site and plenty of the time, it’ll say something like, it’s a website where the goal is to check all the boxes. And I’m like, I didn’t, I didn’t say that. Like I just said, here’s the check boxes, do whatever you want. And it’s fine that that’s your goal, but like, that’s your own goal.
But I think that that was part of the appeal here is that I think in general, people are much more attached to goals that they come up with themselves. And saying, here’s the thing, do whatever you want with it, and you deciding, my goal is to prevent all the boxes from getting checked, or my goal is to prevent box number two from being checked ever, or I want to check every box.
Adam: Pretty quickly, people start having their own stories about how they’re using the site.
Nolen: So, an example of that is I would often hear from someone who would find a place where someone was like checking a bunch of boxes, and they would follow that person and uncheck the boxes.
And they’d be like, “Yeah,” and like, they were getting so mad, right? Like I was trolling them, and they’re getting so mad. And again, it’s like, you don’t actually know that, right? Like you have no idea whether they’re noticing what you’re doing.
Like you have no idea what’s going on with that person. Maybe they find it funny. But because you have such limited information, it’s really easy for you to like play pretend in this way. And that’s just super fun. Like you get really into that. So I think it really let people tell those stories in fun ways.
Backend Problems
Adam: Okay, let’s get into the weeds for a minute. Imagine managing a million pieces of shared mutable state with tens of thousands of reads and writes per second.
It’s a scaling challenge. It needs to feel real time to engage users. Plus, it has some kind of aggregate data. You need to have a count of the number of things each user has flipped and then a total.
Seriously, if you’ve ever had to stop and scale a datastore or a database, stop and think about how you would handle this for a minute.
What’s the hard part? To me, the issue is the number of writes. If you’re going to use a traditional database, it’s a lot of concurrent writes.
And some of the rows, depending on how you organize the data, are going to be really hot. Those first hundred checkboxes, I’m sure, are where a lot of people are clicking.
So then I start thinking about something in memory, but how do you keep all this state in lockstep?
Nolen: Yeah, yeah, the architecture was like, Redis has a bit set, just roughly a million bits, with some nice operations for flipping individual bits. So Redis has that. Redis has a count of the number of boxes checked, stored separately, just so that we don’t have to iterate through a million things to determine how many are checked, and the website’s supposed to display that.
Side note, Redis also, you can write Lua and invoke it via Redis atomically, which is really sick. So like, you really don’t have to worry about races as long as you write your logic in Lua scripts.
So it was really easy to say like, Oh, we’re storing this count separately from the number of check boxes. There’s an obvious race there, kind of, you know, depending on the primitives you have, but like, let’s not even think about it at all.
Adam: Redis has a PubSub feature. You can flip a bit, bump a counter, and then broadcast the change all in one Lua script.
Nolen: And the rough architecture was, you had a bunch of workers whose job was to accept WebSocket connections from people on the website. They offered an API that let clients flip bits, along with some basic rate limiting. There’s also some client rate limiting, but that’s obviously trivial to avoid.
And then whenever a client flipped a bit, you would write that to the PubSub. And then every WebSocket worker process was also reading from that PubSub, and whenever something got written there, it would update all of the clients that were connected to that individual instance.
Adam: This is smart, right? State management is atomic and fully outsourced to Redis. But scaling the workers to handle the WebSockets—that’s still a challenge. And then there are other things that Nolen didn’t even think about.
Bandwidth Costs
Nolen: I think it was night one when I realized I hadn’t thought about bandwidth cost.
I was managing all my servers myself. Um, and so I had originally decided, okay, like, my costs are capped. Like, I know how much my servers cost. I’m spinning up a bunch of servers. Like, this is probably gonna cost me like a couple hundred bucks. I’m getting some donations. That’s probably fine.
And then, I remembered that, you know, you get charged for bandwidth. And I’m sending like a weirdly large amount of data, especially at this point I hadn’t thought about how to reduce the size of these updates that I was sending clients.
Um, and so I was pushing out, you know, hundreds of megabits a second or something. So Nolen started doing the math and estimating costs, and the math gave him an estimate of 10,000 for his bandwidth so far.
And like, the pit in my stomach, right, like it just felt terrible. That’s all. But I called a friend, uh, my friend Greg, who I worked on TalkPaperScissors with, and he talked me through it, and we agreed that my math was wrong.
But also, I started working on like, I can vastly reduce the amount of data I’m sending. That’s gonna help a lot. And two, I happened to be pretty comfortable using the Linux command line utility TC, which stands for traffic control, and is an obscure utility for limiting bandwidth on instances.
My job back at Amazon Web Services a decade ago involved using it a lot. And so I was like, oh, I can just like add a rule that drops all traffic, like all outbound traffic, above I think it was 250 megabits a second. That let me calculate an upper bound, right? And that was what really mattered to me, was like, the upper bound was kind of high, but DigitalOcean’s bandwidth pricing is actually quite sane.
So that was, that was the one time that I was really scared. Beyond that, I was delighted and baffled and excited and super motivated to like keep the site up and keep working on it and improve it in different ways. I had made a mistake that I often make when I make things like this, which was, um, I had very little attribution to myself or any self-promotion when I made it.
There, I don’t even know that there was a link to my website. There was certainly not a link to, like, my mailing list or a way to donate me money or anything like that.
The Media
Adam: Nolen added links to his website and his mailing list, and just like that, the media came calling.
Nolen: And I did the interview with the Washington Post, um, which was fun. They asked me how I was monetizing it. I was like, um, not really. But I think in the question, they were like, are you going to sell this to the New York Times?
Like Wordle. And I made a joke about like, I will sell it to the New York Times for 1 a checkbox. Um, and like 12 hours later, I got an email from the New York Times, which quickly came with a followup email saying, we’re so sorry, this is not an offer to buy the site, but it was just someone there who also wanted to talk to me.
And I did all that, and that was super fun. That was the first time I’d ever done any media like that. And, pretty validating. Like, it was very convenient for the family, in that they don’t really understand what Hacker News is or whatever, but hey, I’m in the New York Times, and that is meaningful to them.
The Rewrite
Adam: Next up, Nolen needs to boost the streaming server’s performance.
Nolen: So I had a friend from the startup that I used to work at back in Michigan, who these days is like a performance engineer at Facebook, who is very good at making things fast.
And we’ve been wanting to work on something together for years and hadn’t had the time. And I was like, one, you really want to work on this problem, and two, like, I actually really, really need help here. So I called him up and went over to his apartment at like 2 p.m. on a Sunday and we worked until 2 a.m. on a Sunday.
Adam: Part of the big rewrite was moving from an untyped Python Flask app to Go.
Nolen: During the rewrite, I noticed at several points we screwed up our rate limits in one way or another. There were a couple of pretty embarrassing bugs. It was, you know, like we’d just been working on this thing for a long time. It was getting really late.
And so with one of the first versions of the Go rewrite we launched, um, the site went totally crazy because we had broken our rate limits. And like, Redis briefly broke; we were sending Redis too many updates. Um, and that was crazy.
Adam: Wait, normally, you could only send a limited number of requests per IP. Otherwise, you could just open developer tools and write a little for loop, and start looping through and checking all the boxes. But were people actually hitting that limit? Were people writing bots and hitting the API?
Nolen: You know, sometimes you’d go somewhere and like, 30 boxes would clear all at the same time, and you’d be like, that’s not a human. But I think the scale of it wasn’t clear until we removed the rate limits for a little bit, and the bots went crazy.
And during the rewrite, I had noticed, briefly, some data in Redis that looked weird. And by looked weird, I mean it contained URLs.
The First Crash
Adam: Nolen had spotted some weird data before. In fact, he had had the site crash due to weird data on the first big night.
Nolen: And the way that it crashed is that I wasn’t validating indices. So I had an API that, um, the clients used to check or uncheck a box.
And it took an int. And I didn’t validate that the int was in the range 0 to 1, 000, 000. And someone was like, what if I manually call this and send 10, 000, 000 and check that box? And that totally worked. And Redis’s behavior around the data structure that I was using was like, Oh, I was told to check bit number 10, 000, 000.
I have a million bits. I guess I need to add 9 million bits of padding to check bit number 10 million. Right. So that happened, the site crashed, they did all sorts of weird things. And I woke up to this and quickly figured out what was going wrong, but made a couple of utilities to dump the database. And the goal, you know, my goal then was like, I will bring the site back up and I will just truncate the bit set to the first million bits.
So I had these scripts that dumped the, the data in the database and they happened to, um, just based on like what Reddit does when you ask it to display this data structure, convert that data to ASCII. So you have 125, 000 bytes, right? A million over eight.
And each of those bytes is a number, and we look it up and decide what ASCII character is. And like, that is what
Redis prints, unless you’re asking it to print a character, not in the ASCII character range, because ASCII’s only, what, like 127, otherwise you get like a hex digit, approximately.
Adam: While running the script to dump checkboxes, Nolen and his friend Elliot spot the URL in the data, but they have bigger priorities to fix.
Nolen: And I saw that while Elliot and I were working on it and we were in the middle of this rewrite from Python to Go. My Python had no types. It was, you know, wildly unsafe, in all of these ways. We noticed it briefly, and we’re like, “Oh, you have another bug around not validating user input,” but I thought that I had a data validation error that let people stuff strings into this bitset. Um, because I wasn’t really even thinking about what it was that my script did at this point.
And so we noticed this, and we’re like, well, we’re rewriting this in a much more type-safe way; this bug will just be fixed, so we don’t have that much to worry about, right?
Adam: With the new service up and running, they breathe a sigh of relief, and they call it a night.
Nolen:
and. then it got like hilariously faster with the Go rewrite.
I have these CPU graphs that were pegged at a hundred and then like we launched the Go version and they’re pegged at, you know, one. Like, there’s just nothing, you know, it’s, it’s, it’s ridiculous.
Monday
Nolen: And I woke up the next day, and I ran the script again, I guess just because like it was still kind of bugging me, like I had noticed these URLs. It seemed like maybe I had been not quite hacked necessarily, but like something bad was going on.
And I ran the script again. And I noticed that the URLs were still there.
And I was like messaging him being like, “they’re still there.” We’re going back and forth and we’re like, “this shouldn’t like we’re type safe. Like there’s no obvious, like you’re just injecting strings into my database, bug here.”
What is going on?
And then I remembered something, which is that I had gotten a bunch of DMs on Twitter about OMCB. Many of them were just kind of like, “hey, I like this thing.” But one had been from someone who was like, “Hey, can I help you rewrite the site to make it faster? I am friends with a bunch of people who are good at writing performant code.”
And I was like, “ah, maybe, I don’t know.” I like collaborating with online strangers and giving you access to my code base, it’s a little sketchy, but that is a very sweet offer.
And then he had sent me this other message that I had no idea what to make of at the time that was like, “look at the end of the site or something.” A friend of mine left you a message. And I like that message popped back into my head, and I had no, like, I think I had sent him like question mark, question mark, question mark at the time. Like, “what in the world are you talking about? That’s like such a weird thing to say.” But realized like, okay, maybe there is something there.
CatGirl.win
Adam: This all happened on Monday evening, and Nolen had started to connect the dots in his head, but also his girlfriend had made him dinner.
Nolen: And I had kind of like, gotten re-obsessed with like, what is going on here, right as dinner finished. And she was like, can we like, sit down and eat? You’ve been very busy recently. And I was panicking about like, there are these URLs. I have been hacked. I have been hacked. And had, like, decided, okay, like, I’ll stop thinking about this for a second.
And sat down and then had this thought and, like, you know, it all came together. And then, like, rushed back to my computer to, like, you know, confirm what was going on.
And I think like all the pieces kind of came into place. Like, started like looking at the actual site. Oh my God, there is a pattern here. Oh, holy shit. Like, they’re not, the data is in my database because what they are doing is checking and unchecking boxes.
To make the binary representation of this, when interpreted as ASCII, spell out this URL. Like, this isn’t hacked, this isn’t I have a bug, this is like, someone who is being very persistent about writing a weird message into my database.
Adam: The only way this could happen is with a bot, right? Any visitor can check or uncheck a checkbox, but this URL is always there. And encoding that URL takes a lot of checkboxes in a row, right? 192 for this specific URL. That implies a bot is enforcing the data in this very specific part of the checkbox grid.
Nolen: And the URL linked to a Discord. Um, it took me like five minutes to convince myself that I could click it since it was catgirls. win/omcb, and that seemed super sketchy, but I did everything I could to convince myself that it was a real website, that it was safe to visit, and then decided that I couldn’t convince myself of that, but that I would regret it forever if I didn’t click the link, so I just had to click the link anyway.
And I clicked the link, went to a Discord. The Discord is called CheckInBoxes, and it was full of people. Teens talking about botting the site, um, who had been kind of gathering because they were all looking at this data, right? Like they were all, they had all like figured out how my API worked. They were writing tools that hit the API and the command line, looking at either the base 64 or ASCII representation of the data for one reason or another, and had all noticed these URLs and had joined the Discord of, I think, one of the first people to start botting the site.
Adam: That’s And. How many people were in the Discord? I think there were about 20 when I joined, and then by the time I shut the site down, about 60.
Nolen: Which is just, like, mind-blowingly cool, you know? Not remotely what I expected. Like, it’s kind of impossible to convey the feeling of going from “I have been hacked. This data is still here,” right? Like, I think it felt worse because I thought that I understood; like, yeah, I was kind of owned; someone had stuffed these strings in. But then we fixed it, and then, oh my god, I haven’t fixed it. Like, maybe somebody has, you know, root on one of the boxes; like, something is really bad. To wait, there’s this whole community, and they’re doing something that is like insanely cool.
Drawing With Checkboxes
Adam: Nolen created 1 million check boxes without a fixed grid because he wanted to stop people from drawing offensive images.
The number of check boxes wide was device specific, so you could draw whatever you wanted, but most people wouldn’t see it. This tied into his idea of curbing bad behavior.
But if you mass check and uncheck boxes and build a CLI app and somehow write to the bit set as ASCII, you find a URL, and it leads you to a Discord, and just like that, a community forms.
Nolen: and their big goal, in addition to gathering more people, was drawing on the site, which they were doing by treating it as a thousand by thousand canvas. So they would have a bot that would take in an image and coordinates and a size. From that, determine the set of checkboxes that would need to be checked to form that image on this hypothetical thousand by thousand canvas that they were drawing onto. And then listen to events and check in on checkboxes to make sure that the image stayed drawn on the canvas, which was just insane and so, so cool.
Adam: So did it farm out the work to multiple of them, or are they each?
Nolen: I think within the Discord, people had a sense of who was drawing what where. So they solved the problem of like, obviously if you run two of these bots on the same territory, you just check and uncheck boxes forever and nothing ever gets drawn and my server crashes. But for the most part, almost everyone who was drawing stuff was in the server.
Um, there was one exception of some people that were trying to draw the logo for Osu! the game, who weren’t in the server and were fighting over territory with the people in the server, and that was pretty funny to see. And then the other big problem for them was I had rate limits, right?
Like I rate limited people based on IPs, and you could work around them by getting more IPs. So the people that were like the fastest and able to draw the biggest things were the people that could come up with the best ways to get a bunch of IPs to send their traffic from. And these are like kids mostly, they’re like mostly, you know, like 18, so they don’t necessarily just have a bunch of money to rent a bunch of servers.
So they were coming up with clever ways to route your traffic through residential IPs; they were using a bunch of stuff like that. So that was kind of the big constraint for them was having enough IPs and coordinating which check events you farm out to which IPs. You need whatever a central server that’s like listening for those events and then farming out the work to a bunch of different workers, which is like kind of an impressive amount of infrastructure to spin up for something like this.
Drawing Projects
Adam: So what were they drawing?
Nolen: When I joined, they were drawing really simple stuff. I think the first thing they wrote was “be gay do crime.” After that, when I joined, they had a QR code that linked to the Discord, which I thought was really funny.
But they started getting more and more complicated over time. One way that they got more complicated is they started coming up with big detailed images to draw. They did a Windows blue screen of death, which I thought was really funny. They tried to draw all sorts of things over the vast majority of the grid, which just took more technical work. I say “vast majority” there because they were very careful not to touch the top few or bottom few rows, as they figured that’s where most people were playing, and they knew they were kind of degrading the experience, which I thought was pretty sweet of them.
The big innovation later was, oh, we should start doing GIFs. What are animated images that we could put on? Obviously, we got some Rickrolls at some points. We got a lot of Jake Gyllenhaal GIFs. Do you know Bad Apple?
Adam: No. Yeah,
Nolen: quite understand this phenomenon, to be honest, but there is a video called Bad Apple.
It is a cool, interesting, like, anime video thing. But the point is, it’s a meme to make Bad Apple run inside stuff. kind of like how with computers, it’s always like, can I run doom on this thing?
It’s like, can I run bad Apple inside this thing? So there was a project, bad Apple that a lot of people were devoted to. And I actually hadn’t thought that they had succeeded. And then when I looked back through all the data after building some, my own tools to inspect it, uh, found that someone had successfully run bad Apple on it.
So that’s, that’s cool and exciting.
Communication
Adam: Nolen’s main concern is figuring out the right way to connect with this community.
Nolen: I didn’t want to, like, ruin it, right? Like, I think it was very fun that I was there, and they, they, like, were excited, they had a lot of questions for me, they had found bugs, whatever, they wanted to talk to me. But at the same time, I did think, like, this is your fun little hangout for you and your mostly, like, teen friends, and I am, like, a 33 year old software developer, and, like, we’re in very different life stages, and, like, you should have your own space.
So I did kind of try to, like, pop in, answer questions, I would pop in and make announcements like, Hey, uh, an interview with me is running in the Washington Post. There will be new users. Please, like, limit your botting for a little bit. Things like that. And they were, they were happy to listen to that, which was great.
But they had a bunch of questions, about, the site, about how I was solving things. And just kind of like about my deal. Like I think they’re just like interested and I was happy to talk about that
in a few cases, people have, gotten in touch with me somehow to ask whether any of the teens in the discord are looking for jobs. And so I have relayed some job offers to them.
a couple of things like that. Uh, but yeah, we’ve chatted a fair amount and they’re great. I don’t know. Um, I’m so proud that. They decided to point their talents at my website and I think they’re just crazy talented.
The Validation
Adam: For Nolen, just knowing this group existed changed things.
Nolen: This felt like it really validated this thesis that I had had, from back when I made like Stranger Video and other things.
I had had this belief for a while based on these smaller but still moderately successful stranger interaction things that the way to approach a problem like this, or a way that I think works well, is to have a small, fun, novel interaction with enough constraints that people can’t, you know, trivially ruin it, and that those constraints produce really fun, interesting behavior.
And this just felt like the purest possible validation of that thesis. That, like, I believed this, I had given some talks where I believed this, and I said it in the talk, and I think the talks were okay. But, like, I didn’t have enough proof. And this was, like, all of the proof I would ever need. This was so much of a reason to continue pursuing projects with this, flavor.
Yeah, it was, it was, it was validating to kind of like this core thing that I knew to be true, but needed to prove in a bigger way. And that was also, that was also really, really thrilling. And, I don’t know, we’ve talked a couple of times about clarity of what to pursue and, you know, what I know how to do and, and what the goal is.
And I think that was, that was another step in that direction.
Adam: It’s funny how this direction that Nolen has refined is still on. It all started off with that playful exploration and boundary-pushing around email rules and how they were handled back in high school. Back then, he was the one who exhibited trolling behavior that others saw potential in. It wasn’t just his dorm supervisor who made him a teacher; he also emailed his high school computer science teacher about the rules he was creating.
Nolen: Like, I don’t think I knew what Turing completeness was, and I’m not sure that the rules were actually Turing complete, but I was kind of like, this is way too powerful for a school email system.
But like, my, you know, memory of it, it’s basically him being like, “Don’t mess everything up.” Like, that’s pretty funny. I’m sure he was kind of proud. Yeah, I don’t know, I think back to that sometimes in terms of like, wow, I really, I, you could easily imagine a world where the response by the school was like, “You’re expelled. You broke a bunch of stuff.”
And the fact that instead, I was kind of mischievous, but it was kind of funny, and it was like, pretty, ultimately pretty low stakes, and people just kind of were encouraging was really nice.
There are a lot of versions of trolling that are really fun. If you go back to the version of the internet that we had in 2008, I think there are lots of places where people did some kind of large scale collective trolling that is remembered very fondly. Like 4chan rigging polls to spell out funny things.
The thing that was the most emotionally satisfying to me was certainly discovering this and then just being so proud and excited of them and of myself for making something that they thought was worth their time. Getting to be proud of them and not mad at them, you know, was so meaningful to me. I cried a lot about it, which is not something I normally do.
I think that when people think about trolls now, this is more like, I don’t know, people being really mean on Twitter. Like, providing people a very low stakes way to troll where the worst thing you can do is uncheck a checkbox is great, right? Like, it’s fun to do that. It’s fun to prank somebody in some small way, and it’s totally okay.
And I don’t know, I think people really enjoy it when you give them a chance to do that.
Outro
Adam: That’s it. That’s the show. Nolen’s had multiple projects spreading around the web since we last talked. He’s really good at this. Reject Bad Apple is that Bad Apple animation he previously mentioned, but playing inside Vim using regular expressions. EveryUUID.com is a website with every UUID on it, somehow.
So, thank you Nolen for pushing the boundaries, You’re clearly one of the world’s experts at twisting things beyond their intended use for our amusement. Go to his website eieio.games, and he can send you a newsletter whenever his new creations come out.
Thank you especially to the supporters who keep me at this. The main benefit of being a supporter is literally showing your support. And I do appreciate that, but there is also a Slack channel and some bonus episodes. Really, just thank you to everybody who joins the Patreon supporters’ group, just to show that they support what I’m doing here.
And yeah, we also have a pretty awesome Slack channel that you can find on the website. So I hope you liked this one. And until next time, thank you so much for listening.
Sticher
Overcast