The following is a rough transcript which has not been revised by The Jim Rutt Show or Samuel Arbesman. Please check with us before using any quotations from this transcript. Thank you.
Jim: Today’s guest is Samuel Arbesman. Sam is a scientist in residence at Lux Capital and a senior fellow of the Silicon Flatiron Center at the University of Colorado and a research fellow at the Long Now Foundation. He’s also the host of the Orthogonal Bet podcast from Lux Capital. Welcome, Sam.
Samuel: Thank you so much. Great to be chatting with you.
Jim: Yes. This should be fun. Sam’s written three books. The first is called “Overcomplicated.” The second one, “The Half-Life of Facts” – I’m gonna read that one, that just looks very interesting. And then the one we’re gonna talk about today, “The Magic of Code: How Digital Language Created and Connects Our World and Shapes Our Future.” Very interesting. So what caused you to write this book? What was it that motivates you? A very personal book.
Samuel: Well, thank you. Yeah, and it was. There’s a lot of stories about my childhood and my experiences with computers. And to be honest, when I look at the current conversation around technology and computing, it feels slightly broken. I feel like we’re either knee-jerk adversarial towards technology, or worried about various aspects of computing, or we’re just entirely ignorant of it. And the truth is, a lot of these responses, they’re not necessarily bad. They’re totally reasonable. But when I think about my own experience with computers growing up, it wasn’t just worry and concern. It was full of wonder and delight. And I didn’t feel that computation was just this branch of engineering – it was this weird humanistic liberal art that connected to language and philosophy and biology and art. And when I think about my childhood full of, I don’t know, the early Macintosh or the Commodore VIC-20 or things around like fractals and SimCity, there was all these weird and exciting aspects to computing, and essentially that’s kind of the point of the book. For me, it was to rekindle that sense of wonder and understanding and to try to provide a sort of healthier way to approach computers to kind of connect the machine to the human, and hopefully repair that somewhat broken relationship to technology.
Jim: Yeah. I really like that. You made the distinction between the wondering stance and the utilitarian stance. I thought that was good. And actually, I kind of did it in reverse. As an undergraduate at MIT, this was still in the days of punch cards and IBM mainframes and stuff. And I had a pretty negative reaction to all. I took one computer science course, kinda distinguished myself by not being a computer nerd at the time. But then in the late seventies, I started seeing these things. I was a college textbook peddler of all weird things and was seeing on these professors’ desks these early North Star Horizons and Mitts and what have you. What the hell are those things? Computers. Really? That’s kinda cool. And then, in 1980, I dumped about 90% of my net worth and bought a totally loaded Apple II, you know, with a big – not even 64, 48K of RAM. Oh, wow. Okay. And floppy disk drives. I was the most popular computer kid in Lexington, Kentucky because I could copy floppy disks with my two drives. There was a way to do it with one drive, but it was a real pain in the ass. You had to page it back and forth to memory and all this stuff. So, you know, it was in those days, and I just immediately fell in love. It was like, wow. This is what it’s about. Now I understand why people like computers, which I’d never liked before. It was the glass house and a bunch of rules and punch cards and all that sort of stuff. And I’ve always found delight and wonder in computing, and it is interesting and curious for it now to become just a job like any other.
Samuel: Yeah, I think there’s this sense that right now when we think about it, there’s this corporatized or enterprise software sort of approach. And underneath the hood, there’s all these fun and exciting and just weird things. And to be honest, that utilitarian sense and the wondering sense, they’ve always coexisted within computing. So alongside the punch cards, there were people doing early experiments with early computer games. And then with the advent of personal computers, not only was there the more playful stuff that you were introduced to, there were also all the office software and actually putting a personal computer into an office space, which probably felt far less exciting for some people. And the truth is, even nowadays, when we think about the web and we think about the walled digital gardens of social media and everything like that, it doesn’t feel that exciting. But underneath it, there are actually people who refer to this as the poetic web – they’re trying to find and build these just weird websites that are fun and different and just kind of there for the sake of being enjoyable. And I think we need to find paths to that aspect of computing, because for most people, either it’s been lost – maybe they had it when they were younger and they kind of forgot about it – or some people have just never even been exposed to it.
Jim: Yeah, and you make an interesting point that a lot of it’s about scale. Right? I had a very curious career. When I was 17, I had the title of programmer for like six months when I was, as part of my student labor to pay off my student loans, the programmer for the Earth and Planetary Science Department at MIT. I think my big project was writing a model of the atmosphere of Jupiter, which was the first planet other than Earth to have its atmosphere modeled, under the direction of a professor. But since then, I’ve never had the title programmer, but I’ve been manager of software development, senior VP of technology, CTO, all kinds of crazy stuff. But when I do personal writing, it’s always at the one-person scale. And that’s where the wonder still lives – where you can hold the whole thing in your head, and you can play with it and twist it and change it. And I got a pretty big capacious head both in physical circumference and I think in processing capacity, and so I can keep a pretty big program in my head. But that is then qualitatively different from the skills you need to manage software development. At one point, I had 200 techies on one project, and that was like, holy shit. It was really not much different than running a railroad or something. And I will say that particular one, everybody had fun. We were doing some interesting, hard, fresh things, but still, an awful lot of the reason it was successful and/or less successful than it could have been or more successful than you might have expected – and it was all three actually – it had much more to do with the execution of management than it did about the technical magic, which was kinda sad.
Samuel: Yeah, and I think perhaps one of the reasons I’ve been able to be a little bit more connected with that kind of wondering stance approach is I’ve never worked as a professional software developer. I’ve done a lot of programming in the course of scientific research and just using it as a tool for various projects. But as a result, it’s always been at the level of working on the order of one person. And so because of that, it always feels a lot more playful. I’m sure if I had moved into the realm of more professional software development, a lot of that kind of fun might have been leached out because sometimes you have to do that in order to push up to ship something and actually get something out the door.
Jim: Yep. And it also, of course, depends on where you are. Even today, there’s still two or three man teams building cool little apps, and that still has a lot of that magic aspect to it because the amount of stuff you can build today – even before AI – with two or three people using the unbelievable depth and scary stack of software people build stuff on top of these days, you can get out some pretty cool stuff with two, three, four person team, which you could not do in my day. In my day, you needed big crowds because you had to write everything basically.
Samuel: This is one of those things where people can do fun little things with small teams because we’re able to build on that foundation of all the hard work of the big teams. And there is this kind of nice feedback between those two different things. And certainly now with generative AI, it is a lot easier to build sophisticated software as just a single person.
Jim: Yeah, it is amazing, actually. And that was one of the points you made early in the book is that where I think you call it a hinge of history where AI is doing lots of things. I’ve gone on the record and say I believe that the AI revolution, not just including LLMs but the other AI things that are coming – I’m not a believer that LLMs will get us to AGI, contrary to some folks’ opinions. I think it’s an important piece and solved the puzzle we didn’t expect to be solved so early, which was language. But there’s still lots of other parts about reasoning and things that these things are not that good at actually. But it is amazing how it has opened up for those who have the right mindset, the ability to have people with no technical background at all become power users of this amazing technology, whether it’s creating graphics, creating music, or writing code. I have a close collaborator, a good friend of mine. He’s probably written a 10-line Python script or something, but nothing any more than that. He’s now creating games and all this stuff. He’s become a master of non-techy techy, using the new LLM-empowered tools. It’s really quite amazing.
Samuel: Oh yeah. And the truth is this is almost the reality of this dream that people have actually had for decades within computing, which is the idea of creating means for non-programmers to actually build software. This is the whole idea of democratization of code or end-user programming, or whatever we want to discuss or term it. This idea that building software should not be the domain of some sort of scribal elite – it should be available to everyone. And so if you have an idea, you should be able to build things. And it doesn’t necessarily mean you have to build something that is some slick piece of software that could be turned into some venture-backable startup. It should just be something that you might want to use for yourself or for you and your loved ones. Actually, the novelist Robin Sloan has this great phrase of “an app is a home-cooked meal.” And I think that’s really what we need more of. We don’t necessarily have to be cooking in industrial-strength kitchens. We can just be making our own little things. And now through coding enhanced by AI, people can really do that in ways that would never have been possible before.
Jim: My friend Peter Wang coined the expression in December 2022 that was “the Wright Brothers moment,” you know, when GPT-3.5 shipped. For the first time from the general public’s perspective, the thing got off the ground. It could actually do something useful for most people. I now like to use the analogy: it’s France and Belgium, 1916. Right? The airplanes are getting good faster, and they’re becoming diversified. We now have fighters versus bombers. We have spotters. We have reconnaissance planes. And then was the cool French invention where you could shoot the bullet through the propeller by synchronizing the-
Samuel: They- yeah, the timing of the propeller.
Jim: Yeah. The and the firing of the machine gun mechanism were linked together in an extraordinarily clever way. And so things were happening extremely rapidly, and that’s what’s happening right now. We’re seeing diversification, etcetera. But anyway, I’ve predicted that AI in all its glory will have more impact than the Internet and the smartphone. And then to go back to find a foundational technology as powerful, you’d have to go back to the PC. And of course, the smartphone’s only a little PC. Right? And the network, well, it’s kind of implicit with PCs, at least it was very soon there soon after they were launched. And so, we’re- you know, this is- and like the PC revolution, it opened up unlimited opportunities for people to find their niche. Right? And it’s so unexplored even to this day that, I goddamn it. You know, I wish I was 45 or 35 or 25, instead of some retired old dude. This would be so much fun to be out there and be a person with the ability to just invent with all these amazing new tools. No. I definitely think, yeah, there’s this great I-
Samuel: I think one of the features of AI that is most exciting is just kind of the open-endedness where people are just gonna be trying lots of different things and experimenting with how to use them, and we don’t really know where they’re gonna go. And yeah, whether or not, as you mentioned earlier, LLMs are going to be the be-all end-all or it’s gonna be some combination of different techniques. I’m pretty agnostic there. I think we need to try lots of different things, and we don’t necessarily need to say that a specific type of technique is gonna solve all of it. But yeah, it’s gonna change a lot. I’m hesitant to say how it’s gonna change things only because I’ve discovered I’m pretty bad at prediction. So I don’t wanna make any grand predictions, but I’m excited to see where it’s gonna go. And certainly when it comes to software development, it’s changing things already massively.
Jim: Everybody I talked to, she knows all about how to use. I had a guy on recently, professional high-end software architect for a big company. He estimates his personal productivity is up 30x.
Samuel: That’s amazing. Wow.
Jim: I figured my own personal productivity – I’ve just finished a little software project where I didn’t even look at the code. I decided to see if I can do it entirely with the AI, and I would say 7x, down. I’m not quite done. When I’m done, then I’m gonna look at the code. The code may horrify me, we’ll see. But anyway, let’s get back to the book and get down to some of the specifics. One of the things you talked about is the dual nature of code, that it’s this interesting combination of text, but that could result in action. Tell us a little bit about that idea.
Samuel: Yeah. So I mean, when people think about code and kind of the stuff of computation, there’s not really any good analogy to what it is. Because when you’re typing text out or generating it, it is text. But the difference is, unlike words in a book, it actually can be executed. It can actually do things. And so it’s somewhere in between the stuff of ideas and things that actually operate in the real world. So it’s certainly not concrete or physical or anything like that, but it’s also not just fun ideas because it has the ability to act in the world. And so one of the points I make in the book, and tying back to kind of the title of the magic of code, is that in many ways, it is the embodiment of these ideas that we’ve had for millennia of using text to coerce the world and using words to control the world around us, of like spells and incantations and sorcery. And we now have that to a certain degree in the real world, which is – we can do magic with code where it was this thing where it seemed like a cool idea for a long time and only since, I don’t know, seventy-five years or so whenever the first digital computers came around, now it’s a reality. And because it’s text that works, that operates in the world.
Jim: You used quite a few analogies to magic and to the Jewish mystical Kabbalah and also the Talmud, as examples of code that had impact in the world. Don’t eat pork. You know, there’s text on a page that has impact in the world or, you know, Book of Joshua. So I say, you wanna really understand Yahweh? You gotta read the Book of Joshua. You know? Kill them all. Right? And those words had impact. And you’re actually quite eloquent in comparing how way back yonder magic wasn’t just for the shamans. Everybody practiced magic at one level or another, in a way that today, at least in some sense, we’re all using these magic spells.
Samuel: Right. And yeah, I certainly give a lot of analogies when it comes to magic and code. Hopefully not so many that the analogy fully breaks. But you’re right – there are a lot of deep similarities. And clearly, it was not just the domain of the elite. Like, on the one hand, magic was not something that just worked. It wasn’t kind of this wishy-washy wave a wand and you’re done. There was a craft and work to it in the same way – whether you’re studying for seven years at Hogwarts School of Witchcraft and Wizardry or studying to understand Python better or whatever. Like, there are similarities.
But at the same time, there was this idea that magic could be democratized, that there were certain spells for, I don’t know, finding lost cattle or whatever it was that could be performed by everyone. And we’re now seeing, going back to the democratization of code, that’s the thing we should want. We want everyone to be able to write software and to engage with it. And I would say probably the first widespread example of that, which many people don’t think of as coding, is the spreadsheet. Because if you’re writing functions in spreadsheets, you are doing coding at least to a certain degree. And that’s one of the few democratizing code examples that I think people have been able to point to for several decades. Now with AI, maybe we’re gonna have some new ones. But I think there’s both obviously effort and craft, but it shouldn’t just be the elite. There should be ways of allowing everyone to have a part to play in this kind of magic.
Jim: Yeah. I still recall when I bought my first copy of VisiCalc for my Apple, and this was in the days when software was still sold in Ziploc bags with dot matrix printer labels on it. And it had a serial number on it – it was 254. I very much wished it was 255.
Samuel: Oh, that would have been great.
Jim: But it was 254, and I took it home, and I started building models like crazy. Oh my god. What the hell? Right? This was, you know, I was still a young business dude at the time. I’d been trying to conceptualize how to model a company and things of that ilk. And in minutes, I was doing more than I could do with writing BASIC programs on my Apple and stuff. And then I ran into the fact there was a bug in early VisiCalc. If you built a model that was too big, it needed a memory buffer to write the model out to disk. And about 4:00 in the morning, I went to save this crazed model I’d created and totally filled up my computer and it wouldn’t save it.
Samuel: That’s terrible.
Jim: That was so annoying, but yeah, it really is an example. There was you also called out another example, which I was never a heavy user. Used it a little bit, or I don’t think I used it directly, but I used the clone that ran on the PC with HyperCard. That was quite a thing in its day.
Samuel: Yeah. HyperCard. That was my first experience with programming. HyperCard was this piece of software that came bundled with the Macintosh, probably beginning in the late eighties through maybe the late nineties or so. It was this authoring tool for building what it referred to as stacks of cards. The cards were basically just pages, and you could connect them together into what was basically a website that lived on your own machine. You could enter text, images, draw on them, but you could also add little buttons that would make a beeping noise or bouncing noise. You could have a button that would take you to another card. Under the hood, there was actually an entire language called HyperTalk, and it was this relatively simple language to understand and read.
There’s a phrase attributed to the computer scientist Seymour Papert, one of the developers of Logo. He referred to “low floors and high ceilings.” HyperCard really embodied this idea – it had low floors, it was very easy to just build a simple stack of cards, but the ceilings were very high. You could build all these different, very sophisticated pieces of software. For example, I think the first version of the computer game Myst, which was the best-selling computer game in the 1990s, was built originally in HyperCard. People used it for running databases and different kinds of things. I remember building a simple password generator using HyperCard.
For me, that was really my first taste of programming, and it was possible because it had that low floor. It allowed me to just mess around and slowly but surely become comfortable with programming. For a certain generation of Macintosh users, it was eye-opening and life-changing. Now I think many people look to it as this kind of mythical golden age of end-user programming or software development for non-programmers. I played with it again in emulation on the Internet Archive’s website. It was a little janky – not quite as good as what I remembered in my memory, but it was still very powerful and unbelievable. I certainly think we need that kind of feeling to be in our software development.
Jim: Now, of course, the thing that killed it was the web. Right? And when I look back at the web as a person interested in computer architectures and protocols and stuff, I’m still horrified by HTML and HTTP. The fact that our world is built on these shitty, shitty protocols is just unbelievable.
Samuel: That we can do all these amazing things on top of something that clearly was not designed for all those things. The developer of HyperCard, Bill Atkinson, who I believe just died last week, he kind of regretted that he never thought about building a sort of networked version of HyperCard, or that he never quite thought of the web. Certainly, if we had had that kind of architecture, a more HyperCard-y architecture underlying the web, I think the democratization of web development would be that much better. That being said, the truth is the early days of web development in the mid-nineties, there was not that much of a difference between professional websites and amateur websites.
Jim: That was exactly my point – in the early days of the web, you could say, “Whoa, look at that! The brick store down the road’s got a website,” and you tell your secretary to go show her how to use “show source,” copy and paste.
Samuel: Right. You look at the source, and you learn from it. And now it’s completely impenetrable. You look at the source and you have no idea what’s going on, which on the one hand is really unfortunate in terms of allowing people to have that low floor and kind of that entry point. But I guess on the good side, it is still impressive that we are able to have come so far on such a shoddy foundation maybe. I continue to shake my head. It’s like, “Oh my god.”
Jim: I absolutely nothing I hate worse than primary web development. I’ve done very, very little of it. I try to find other tools that will do so. Currently, if I have to build something for the web, I use Flutter, which is a Google product that’s a real programming language and lets you do everything on the web and more with normal programming tools and debugging, all kinds of good things. No CSS. So if you want to try doing web without all the horrific sludge of the last thirty years, check out Google’s Flutter. Well, let’s get back to the main line of the book though, and this is then, you know, we’re kind of hinting at it. We’re dancing at the top level. But what has made all this tower of babel possible is layers and layers of abstraction. And you go all the way down back to the days when programming meant connecting wires. Why don’t you tell a little bit the story of how the tool chains arose?
Samuel: Well, certainly one of the other reasons I actually include a lot of history is I think it’s very important to understand the history of computing to get a better sense of where we’re going. I think the world of Silicon Valley, unfortunately, is deeply ignorant of technological history, sometimes proudly so, and that’s certainly a problem when it comes to not trying to just reinvent things and understanding path dependence. I also wanted to delve into the history because it’s interesting, but also because given that AI and everything else is changing so quickly, the history is the stuff that I know isn’t going to be changing.
When it comes to the history of computing, there is this idea of abstraction – this idea that you can abstract away the details. You can ignore details and build upon interfaces and higher level things without necessarily having to know the details that have come before you, which has allowed us to build really sophisticated things. We were talking earlier about how you can now build very powerful things with only one or two people because we have this foundation upon which to build.
But that foundation started a very long time ago with chips and binary and working with wires and cables and very early ones, and then working with machine language, which was really instructions in zeros and ones, and eventually assembly code. Then eventually, people realized we need better languages for interfacing with our computers, and they created modern computer languages, which were then compiled into these lower level languages. The truth is that’s how things have always been – even though we might be operating at some very high level using sophisticated web frameworks, at some point it all goes back down to binary and machine code. I think most people who are engaging with computers are aware of that, at least at some vague level.
Jim: If you took the core intro course, you’d probably know that somewhere at the bottom are the bits.
Samuel: But even programmers who work with these things on a daily basis – you can’t look at that for too long without being overwhelmed by it. You kind of have to say, okay, this is the fact of these things. When things go wrong, maybe I have to spelunk down into the depths, but by and large, I really don’t have to do that. I actually think the fact that you don’t is an unbelievable accomplishment. And so there is this computer scientist, Edsger Dijkstra – he was foundational for writing a lot of early algorithms and early ideas…
Jim: He did Algol, didn’t he?
Samuel: I actually don’t know. He might have been involved. And he wrote a lot of essays about computing. And he was kind of this interesting curmudgeonly figure. And one of the phrases he talks about is this idea of the radical novelty of programming. I think when people talk about programming, they try to use lots of analogies, which I do as well in the book. I’m like, oh, it’s like writing a recipe or following a recipe, or writing a set of instructions or doing these kinds of things. And it is – or assembling LEGO bricks or whatever it is. And it is sort of these things, but it’s also wildly different because of this vast edifice of abstraction that we build upon. And I think that’s unbelievable that we’re able to do these kinds of things, but it’s certainly worth revisiting and thinking about at least a little bit.
Jim: And as you say, that’s how three dudes can build a cool project. And it’s interesting how the stack – there’s a depth of the stack because the people understand certain things down below, but they don’t understand down like back in the day when my companies were doing PC-based fancy information systems. We always had to have on our staff someone who could debug the output of the compilers.
Samuel: Oh yeah. And I look back at how – I mean, I read about these legends of Steve Wozniak and the way he was programming things by hand and then running – and those kinds of accomplishments, they’re unbelievable, but I cannot even fathom doing anything like that because we are now so distant from those levels on a daily basis. And there are certainly some people who are involved in that. But by and large, I think most programmers kind of make a point of trying to avoid those details.
Jim: Yeah. So you have a floating stack. Maybe you go down a layer or two below where you’re working every day, but no lower than that.
Samuel: Completely.
Jim: And as the stack gets taller, the distance you are from the real machine – because even in our day when we thought we were cool because we could debug assembler that the compiler put out, we had no idea what was going on at the chip level, which is another level of abstraction. I later got involved with a couple of chip design software companies, and there’s physics underneath the noise and averaging things out to zero and one is not as obvious as it seems.
Samuel: Oh yeah. No, it’s unbelievable.
Jim: There’s a whole stack of things that give us stable zeros and ones. Physics does not give you zeros and ones.
Samuel: Right, right. There is a hard set of – yeah, there’s a great deal of effort put in to create the discretization of the zeros and ones of the binary. To avoid kind of all the messiness of physics, which is probably one of the reasons why these things are being constructed in clean rooms – to avoid kind of the messiness of the real world.
Jim: Yeah. And in fact, now at the four nanometer scale, you actually have to deal with the quantum effects. So the whole thing just gets gnarlier and gnarlier. There are your magicians, there are your sages and your shamans – the people that create 50 billion transistor chips the size of a couple of postage stamps. It’s mind-boggling. And I think about the stack of humanity’s technology, somehow that’s the top. But this whole thing is focused on creating those suckers. And of course, as we go into the AI world, it becomes even more the case where, you know, how many chip equivalents equals one person in cognitive ability? And that number will continue to improve at Moore’s Law probably for another five or eight years at least, which is kind of mind-boggling.
Samuel: Yeah, that’s wild. Yeah, that’s super interesting to think about.
Jim: It really is. Let’s return to the idea of code and magic. There’s one little section in the book that I found very charming. Maybe you stretch the analogy a little bit, but within the tolerable range for entertainment purposes, and that’s where you laid out four categories of names, combining and manipulating text, grimoires, and sacrifice. Would you be willing to go through all four of those? I just think that they’re kind of interesting and fun.
Samuel: Yeah, sure. So the first one, the idea of names. In stories of magic, the ability to name something or have or just elucidate the true name of some creature or someone confers upon you a certain amount of power, because you can control something with names. And so for me, the relevant analogy there in code is that of variables. So understanding the variable, kind of this name of some sort of bit of information within your code is the same kind of thing. And it turns out there are simple variables that just hold single numbers or a single word of text, but oftentimes, the variables themselves can actually have complicated structure under the hood. And so understanding the nature of that name, the nature of that variable can also actually give you a certain amount of power.
And then I also discuss how there is a certain type of variable called a global variable where these are variables that can be used anywhere within a computer program. They’re often considered to be far too powerful for your own good because someone else who uses the code or you yourself might use it accidentally somewhere else, and you can have all these unforeseen consequences. So there are certainly true names that have far more power than you might want, and so global variables are an example, and we should avoid them.
The second one is around combining and manipulating text. And so the example I give there is within the idea of the Kabbalah. This kind of Kabbalistic manipulation of letters and words that somehow if you can kind of figure out the right combination of letters, then that can give you power over a golem or some sort of other creature or whatever it is, or you can actually do magic. And at one level, that is essentially what code is. You can have…
Jim: That’s a direct analogy with spells.
Samuel: Right. Exactly. So it’s exactly with spells. You put the right strings of zeros and ones together. You can do that kind of thing. I give one example of, I think, I talk a lot about regular expressions as kind of these fun which are compact expressions for snippets of text where you’re kind of doing searching, replacing, or finding, and things like that that often have this feel of looking like kind of random strings. They are not random. They’re just kind of weird things you have to cobble together. But if you do them the right way, it can immediately and almost magically pull out the exact dates within a large file or whatever it is or names and things like that. And so that was kind of another example I gave.
Jim: By the way, that’s one of the things that LLMs boost your power on the most is regular expressions. Trying to write a useful regular expression hurts your head. I know no programmers, maybe I know one, who’s a true expert in regular expressions. But turns out even early ChatGPT was a wizard at it. You just describe what you want, and it generally gets it right on the first shot. If not, get it right on the second shot. And you give it a couple of examples of what it did wrong. So if you ever need to use regular expressions, folks, just fire up ChatGPT if you don’t have anything more powerful than that.
Samuel: Oh, that is good. It also just shows you the extent to which humans are limited in many different ways, but our minds just can’t hold that many things in our heads. And when it comes to regular expressions, you often have to hold a lot of things in your head, and it’s very hard to do that well.
Jim: We only got a working memory size of seven max. Exactly. And some of those strings are 30, you know, tokens or at least 15. And that’s like, ugh. I can’t do that.
Samuel: Right. And so, yeah, so the third example you gave was the grimoire. And so it’s these texts of magic, texts of spells and books of spells and sorcery and incantations and things like that. And, obviously, they exist in the magical world. But it turns out we have these in software, whether it’s computer manuals, we have like Numerical Recipes in C or whatever these things are. But, of course, one of the grandest grimoires is Stack Overflow. It’s a crowd-built website where people can ask questions and people can provide little snippets of code that can allow you to do all the different things. So if you need to flatten a list in a certain way in Python or some other language, you look online and invariably, there is someone who has also had that problem. And you can go and look into your grimoire and find those answers. Now, of course, with ChatGPT and some of these other tools, the usefulness of these is somewhat lessened. But yeah, there is a grand tradition as well of kind of combining and collecting little tricks and tools and techniques and small algorithms together into these kinds of books similar to the grimoires of magic.
Jim: I don’t know if you mentioned it or not, but the one for my generation, every real nerd had them on their bookshelf was Knuth’s Art of Computer Programming. Three volumes full of recipes all written in his own personal pseudo code.
Samuel: Right. He had his own pseudo code. He had his own, I think, chip. Like, it was like a custom chip that he…
Jim: A thought experiment chip.
Samuel: Yeah, a theory – a thought experiment. And I think I mentioned Knuth in various other things. I don’t know if I mentioned Art of Computer Programming. I mean, it’s also this multivolume thing, which is still under construction. Like, he is still currently working on this thing, which I don’t actually know what volume it’s up to. But you’re right, there’s some foundational algorithms in there as well, which is delightful.
Jim: For the longest time, back before all these libraries existed, where all these libraries people have now implemented these things like say in the Python world – it used to be just go up, look in Knuth and look it up and say, okay, now how do I convert this to my language? And in twenty minutes, you’d have something that would have taken you three days or more to do if you could do it at all. It was extraordinary like sorting algorithms – every sorting algorithm I’ve ever heard of.
Samuel: I think, yeah, one entire volume is just sorting, I believe.
Jim: If I remember correctly. The thing is nuts, but every nerd in my day had it on their bookshelf. I proudly had a copy.
Samuel: Yeah, I definitely think I have the first volume somewhere – I’m pointing down, yeah, down next to me. But yeah, that would definitely be another example of a grimoire.
Jim: And then your final category was a little bit of a stretch – sacrifice.
Samuel: So that one actually was inspired. I had this wonderful conversation, it was like this online salon where I was talking about magic of code, and someone mentioned this interesting idea around sacrifice. And it got me thinking more deeply. And I thought it worthwhile to include because in stories of magic, there’s often this idea of something must be sacrificed in order for the magic to operate. And for many people, when we think about how these powerful AI tools are operating, we realize that they’re not coming out of nowhere. They are enormously sophisticated algorithms, but they only operate because they have huge amounts of data being poured into them. And so, essentially, what we are doing, in somewhat of an analogy, we are sacrificing huge amounts of creative work into these systems in order to get them to become as powerful as possible. And you can see some of these hints when there are certain versions of image generation tools where you would generate an image and you would get sort of a fuzzy Getty watermark on there. And you realize, oh, they are actually using copyrighted images as the grist, as the raw material for these kinds of things. And so it just made me realize that, right, there is an element of sacrifice as well in allowing these powerful AI tools to operate.
Jim: Yeah. That is an – so, of course, this is a hugely litigious field. No one quite knows how it’s gonna land. Right? And of course, the other thing that’s important for us to think about is the whole idea of copyright is relatively new. I don’t think it’s any earlier than the seventeenth century. Used to be creators created, people copied. Right? And that was just the way it was in the world. You know, think of Homer, poor dude. As far as I know, he got no royalties.
Samuel: And ideally, like, the ideal world of copyright is that you have to find a way of balancing the idea of, okay, you want creators to be incentivized to create, but you also recognize that creation should eventually become part of the public domain. They should be public goods that can be the raw materials for further creation. And I think right now with the AI tools kind of just sucking in all this stuff, I think there hasn’t been a lot of deep thought, at least maybe from the side of the companies doing these kinds of things of how to think about this properly. That being said, I certainly think certain of our copyright laws have erred on the side of locking everything down. We need to have some sort of common source of foundation of creative work that can be drawn upon. And I think right now where it’s is it? Seventy years after the death of the creator? Seventy-five years?
Jim: Yeah. Whatever it is. The Disney rule.
Samuel: Exactly. That feels overly excessive. And so we should figure out what is the right balance. And I’m not sure we’re there yet. But I am glad that a lot of people are having this kind of conversation because there needs to be a balance. The answer is I don’t know where the answer lies. I am glad people are having these conversations. I do wonder if some of these lawsuits will squelch certain things in sometimes counterproductive ways. But I also recognize that they can’t just suck up and hoover up all the information that’s out there without realizing that it actually does have implications. So that’s a long way of saying I don’t know what the right answer is, but I do know that we haven’t figured it out yet.
Jim: Yeah. Our current laws obviously didn’t anticipate this. And so John Perry Barlow, I think very vividly said about copyright: “Copyright protects the bottle, not the wine.” So it’s only the exact sequence of words and not the ideas. And LLMs are great at sucking the ideas out without giving you the exact words. Occasionally, they do by accident, but they’re not supposed to. You can sort of coerce them into doing so to some degree, but in general, they don’t. The ideas, which under our system of law are not copyrightable, just as titles are not copyrightable, interestingly enough. This was just a case of first experience where the law doesn’t actually apply. My bet is if there’s an honest outcome in the courts, the courts will rule that calculating vector statistics from a body of work is fair use under any conventional read, but it doesn’t mean it’s correct. It means that that’s how the law should be literally interpreted. And unfortunately, with our broken politics, I’m not at all sure how we’re gonna get to have a discussion about how to do this balance, which you eloquently point out. The creators need incentives, but humanity needs to be able to benefit from the creators, and this is a classic social question. Right? There’s no right or wrong to this. It’s a balancing act. Oh, by the way, I do have an answer on the right answer for normal copyright. And it’s very simple heuristic: it should be ten years so that when you’re 25, you can dub the music in from when you were 15 into the movies that you make.
Samuel: Okay.
Jim: You know, so when you’re 25, for me, that would have been okay. I can start using Led Zeppelin in my home movies and in my YouTubes and things. And in truth, and you know this as a writer, very few people make any money after ten years. And so what you’re essentially doing is clipping off some lottery tickets, and there’s no real need for lottery. You didn’t write your book on the thought that this thing is gonna sell 17 million copies and people will be throwing red tomatoes at you because you offended some politically correct thing. You figured out, yeah, this is what this kind of book sells, and it’s worth my work to do it and for other reasons. And of course, winners of lottery tickets wanna preserve those lottery tickets for as long as possible, and that’s the real corrupt thing about our copyright. We just said ten years. Yes, a few little bit of injustice will be done at the margins. Very few things are created to have a ten-year useful life, maybe a new dictionary or something.
Samuel: And not only that, there are very few situations where something is not successful for the first ten years of its life and then suddenly becomes successful. And that’s the situation I think people are trying to protect against, but that is clearly very much an edge case.
Jim: Yeah. It’s like such an edge case relative with all the other benefits that would come from doing a ten year. We have 25, I wanna use my the music that I fell in love with when I was 15.
Samuel: That’s interesting.
Jim: It’s a very simple heuristic. Let’s kind of continue this because it’s related but different. And you do a little bit of a dive into open source. And open source isn’t quite as intuitive as the name sounds, right? There are different flavors of open source. So tell us what you know about open source.
Samuel: Yeah, and so open source, it’s in some ways kind of an extension of what we’re talking about – this idea of being able to develop things that become part of the common foundation that people can build upon. And so in this case, open source, and there are certainly different variations, but the idea behind it is that the source code, in this case the actual code that this piece of software is built upon, should not be locked up. So oftentimes when you have a piece of software, you have the kind of compiled version. So you can use it, but you can’t necessarily edit it or see how it was written. But when it comes to open source, that aspect of the software is available for people to either add to, to combine, to build upon, to copy, or to kind of build their own versions of it, often with various kinds of restrictions. It kind of depends on the various license and things like that.
But it turns out that kind of openness really is ultimately just an example of sort of the creative process more broadly, which is going back to what we were saying, that all creativity is this process of recombination. And when you combine recombination with this idea of abstraction within software, that is this idea that you can only build things if you have a foundation to build upon. So rather than having to reinvent everything from the ground up, you can incorporate various software packages that other people have built. You can build upon other people’s software. You can take something that someone has made and then make it your own.
And ultimately, that is kind of how all creativity really is done. And so open source is kind of one example of that, and I go into the ways in which software has been built over time. And actually, one of the examples I give is sort of even before open source was really open source, which is the idea of Unix and Linux. And so Unix was originally developed by Bell Labs. It was not designed to be open source software. It kind of escaped, and then there’s many different variations. And for me, the way I think about it is I actually think about it as a sort of textual oral tradition, kind of by analogy of ancient mythological tales.
When it comes to mythological tales, there’s no real canonical example, like a canonical version of a story. There might be many different variations about the stories of Hercules or Zeus or whatever it is. But over time, people add to the stories, people modify them, they change them. And the ones that are kind of the best and resonate with people, those are the ones that stand the test of time. And in many ways, that’s what happened with Unix. It was kind of modified and changed and honed, and there’s different variations and more truly of software in general, especially open source software. That is kind of how it evolves, that there is this kind of textual tradition. Ultimately though, it’s not enough to just have something as open source. You can’t just have the thing that is available to be downloaded. It has to also be part of a community. And because in the same way that ancient mythological tales were part of the community of…
Jim: The bards.
Samuel: The ancient Greeks, right? Like, they were the ones promulgating these things and people who listen to them and enjoy them. You need the same kind of… you need a community that is actually preserving and maintaining these pieces of software as well. It doesn’t work if you just say, “Oh, here it is. You can download it.” But if no one really cares, it kind of just vanishes without a trace. And so this combination of tradition and community is something that I think many people sort of outside the realm of open source software don’t necessarily realize that these things are really powerful and really important for allowing that kind of open-ended creativity.
Jim: Yeah, it is interesting. You know, your point about it is a live ecosystem. Unless it’s a very narrow thing that you’re trying to get your open source software to do, I always look at how many commits have been done recently. Right? There’s one I use that hasn’t had any commits for seven years. But well, they might…
Samuel: They might have just honed it to perfection, and they’re like, “We are done.”
Jim: We’re not gonna break it ever again. Exactly. Which is not a…
Samuel: Bad thing either.
Jim: But anything that’s somewhat modern, you’re definitely looking for it. And you do reference the Python ecosystem a fair bit. In Python, there’s layers and layers and layers of open source software like NumPy, which is this matrix math thing is used in stacks upon stacks and stacks of software. And then everyone relies upon it. And if there’s any changes between additions that has to be stone stable, has to be backwardly compatible, and this all requires considerable work, not just within the core team of building NumPy, but with the other people in the community so it could be tested to make sure there are no breaking changes, etcetera. It’s quite an elaborate ecosystem actually.
Samuel: Oh yeah. The Python ecosystem, it’s unbelievable, and I’ve relied on it in my own research and work. The sheer volume of libraries and packages is amazing, but it goes back to exactly what you’re saying – you need this active community of maintainers. And I think many people, even users of software and programmers, sometimes think, “Oh, because this thing is open source, there’s gonna be this huge community of people working on it.” And by and large, that seems to not be the case. Oftentimes, it is a very small number of individuals who are maintaining even very powerful and active and popular packages and pieces of software. And oftentimes, people don’t realize this until something breaks or you discover some weird bug in some foundational bit of the Internet or whatever it was. And you realize, oh, it’s just like one or two people who have been working on this kind of thing thanklessly, even though it’s really, really important. And so there’s this, I think attributed to Linus Torvalds from Linux, the idea of like, “with enough eyeballs, all bugs are shallow.” Like, if you have enough people looking at things, then eventually all the bugs will be kind of worked out. And I think in theory, that is true. But by and large, there are not nearly as many eyeballs as we would like, especially when it comes to maintenance. Because these things which should have huge numbers of people involved and actively maintaining and modifying them, they are really just kind of a small number of put-upon people, which is kind of unfortunate.
Jim: Many of the little ones I use, 1.5 seems to be the magic number. There’s one guy that does like 80 or 90% of the work. There’s two or three people that each do 5% or something.
Samuel: That would not surprise me.
Jim: Even, you know, one I use a lot, ZeroMQ, very tiny team, a very exotic piece of software that’s a messaging infrastructure for any language basically. It works really nice with Python. And, you know, if you ever need to do interprocess communication, really high speed, 200,000 messages a second, check out ZeroMQ. And real easy. You know, it’s really, really perfect. That is interesting. The incentives people must be getting to do this work is oftentimes thankless, particularly on a mature piece of software where you’re not adding anything too interesting. You’re making tiny incremental improvements, fixing weird little bugs and obscure corner cases and something. But we have to be damn glad that somehow either psychic income or reputational income or somehow it works.
Samuel: Yeah. No, I’m glad that overall, even though these systems are far from kind of the theoretical ideal, it does work much better than would be expected, which is wonderful to see.
Jim: Yeah. Well, let’s move on a little bit. You talked about some of the kind of interesting ideas of microcosms where small bits of things can produce amazing things, and you referenced a game, I think it’s called No Man’s Sky. Is that the one where a single number generates the whole world?
Samuel: Yeah. And so there obviously is a huge amount of code that is going into elaborating that single number. But you’re right. These emergent microcosms are really examples of the fundamental idea of computers, which is a computer is not necessarily doing any specific thing that is mind-bogglingly complex or sophisticated. It’s just doing a lot of very simple calculations a huge number of times, and those kinds of things can really add up. And so as a result, if you have a clever algorithm, it can actually elaborate a very sophisticated thing, very sophisticated procedurally generated world within the case of a game, or especially within kind of the world of mathematics.
There are these mathematical objects like fractals or cellular automata where you can describe them in a very sort of compact way, sometimes a handful of lines of code, but they can sometimes have infinite complexity where you can jump into the Mandelbrot set as deep as you like and see new things, sometimes things that no human being has ever seen. If you dive in deep enough or in the case of cellular automata, it’s just these simple rules of how different dots on a screen are interacting or points on a screen or it’s kind of this grid of black and white dots. And so like the classic one is John Horton Conway’s Game of Life. And from that, you get this emergence of various creature-like objects that move about the screen or do different things.
And for me, I just find that aspect of computing unbelievable. I certainly had my first kind of experience of a lot of these through early screen savers that use a lot of these simple bits of code to just make really cool images on the screen, ostensibly to prevent screen burn in on your cathode ray tube, which is now irrelevant. But in the process, kind of used a combination of math and randomness and just delightful animations to make some really interesting things. And I think that whole world, whether it’s creative coding now or back then screen savers or the demo scene or whatever it is, are able to really take seriously this idea of using a small snippet of code to unfurl this entire microcosm of complexity.
Jim: Yeah. It’s very cool. The regular listeners to the podcast know I have a game out in the world called Network Wars. You can check it out at networkwars.com, or get it on iPhone or Android. It also has a 32-bit world generator number, and everybody plays the same games. So you can rank yourself against everybody else because everyone’s gone through the same sequence of games, which is kinda cool. And it was an idea I had – I didn’t realize somebody else had had it. They have a single seed to generate the world, but it makes sense actually from a programming perspective. Now this section of the book did horrify me with something I said, I think I said in the pregame, I did learn a few things despite knowing a fair bit about the history of computing, and that was code golf. Oh my god. Exactly the wrong thing to do. But nonetheless, tell the audience what is code golf?
Samuel: So by analogy with golf where a lower score is a better score, code golf is the practice or the game or the hobby or pastime, whatever you wanna call it, or maybe sort of some misbegotten practice based on the way you reacted – to complete a task in as few lines of code or few characters of code as possible. So it might be to create the smallest program possible that will actually calculate the number pi or will, I don’t know, draw the Mandelbrot set or whatever it is. And people compete, and they’ll say, okay, let’s see who can make the smallest program in PHP or Python or whatever it is.
And it is alternately horrifying, maddening, confusing, and also humbling and impressive. Because it’s a means of taking a language and sort of bending it to your will in a way that you might never have expected and do something that you never really thought was possible. I found that unbelievably interesting. It was amazing to kind of see how people are able to compress something really sophisticated into this very small, well-nigh unreadable snippet of code. And it was sort of an extreme example of these kind of emergent microcosms of something people were working very hard to compress as much as possible.
That being said, I will caveat this with – and I believe I quote this in the book – there’s this classic textbook “Structure and Interpretation of Computer Programs.” And in it, they mention that within like the first pages, that programming languages are for people and only incidentally for machines to execute. And so ultimately, the idea is that when you write something in code and use programming languages, it is for yourself to understand. It is for other people to be able to understand. And then, of course, it’s also for a machine to actually run, but it should be somewhat readable. Now, of course, Code Golf kind of goes very much against this, but it shows the degree to which there is all different types of expressiveness and style within programming languages.
Jim: Yeah. And that’s why I was like, ah. Because that person who spent more of my life managing the development software than actually doing it. The one thing you don’t want your people to do is write these, you know, obscurely elegant one-line bits of code that generates six levels deep of recursion and do the whole thing. Because and actually, one of the first commercial things I ever wrote had all those bad habits in it. Right? And I actually had to hire two people to actually, like, chipping away at a block of marble to try to make sense of this horrific thing I did, and it was an object lesson. Never never do that. Right?
Samuel: And there are certain languages that try to lead you away from that kind of thing more easily than others. I mean, people have talked about and described Perl as a write-only language – very hard to understand. I definitely can remember at least one instance where I wrote something and then the next day didn’t actually understand my own code. And so there are some languages that make that a little bit more possible than others. Other ones through white space or certain kinds of things try to force you to make things a little bit more readable. But yes, you definitely can bend these systems far beyond the way they might have been designed.
Jim: Yeah. There was one in particular – for many, many years, I taught myself one computer language a year just past the hello world level. Sometimes I’d do this live or toss the knees or some other little simple project. But the one I refused to ever look at was APL.
Samuel: One I’ve never done.
Jim: It was just like a matrix language. It uses very few letters. It’s all weird symbols, Greek symbols, upside down letters. You know? Apparently, it’s hugely expressive. You can do a lot of stuff in one line of code, but I go…
Samuel: Oh yeah, I’m sure that one is perfect for a lot of this code golf, but it’s one that I am willing to live without.
Jim: Because in cooperative programming, you know, the first thing to keep in mind is somebody else has got to maintain the code. And then typically in a bigger project, in cubicle coding rather than individual coding, modules of the code have to talk to each other. So it’s very useful for people to be able to look at each other’s code and understand what they do. Code golf or, you know, the impenetrable Perl. Now I did take Perl at one point because I did a lot of text manipulation and it was the best thing – great for that time. But I rigorously wrote my Perl just like C.
Samuel: Okay.
Jim: For the purpose of making it readable, which you could do, basically.
Samuel: Right. Yeah. The one interesting thing about Perl is the way it was designed. My sense is, like, it kind of gave you lots of different ways of writing it. And so yeah, if you were comfortable writing it like C, you could do that. And so it was this very style-agnostic language, but I think sometimes as a result, it caused a lot of problems in how things…
Jim: Python as soon as I discovered Python, basically.
Samuel: Yeah. I did the same thing.
Jim: I used to sometimes say, you could write anything that could be written in 300 lines of Perl, but don’t ever do that.
Samuel: That sounds not unreasonable.
Jim: Let’s talk a little bit about programming languages and how they differed. And actually, was preparing my notes this morning. I did a Google – I said, as far as I know, all the major programming languages are based on English. The Google agreed with me. Actually, I asked Perplexity – I never use Google anymore. I use Perplexity. I should be hitting them up for a sponsorship actually, because I tout Perplexity all the time as far better than Google, which it is. But it gave a few weird examples. But as far as even Perplexity knew, almost all of our computer languages are based on English. Isn’t that interesting?
Samuel: Perhaps it’s not surprising given that a lot of early computing was done in the English language or in English-speaking countries. But you’re right. That is interesting that there are not many other programming languages, or maybe I mean, maybe APL…
Jim: Yeah. There’s a couple weird symbol ones.
Samuel: Yeah.
Jim: There was one, it’s called Brainfuck, I think it was, that…
Samuel: Oh, that one’s nuts. Yeah. That one’s crazy. I think it has like eight symbols, and it can do everything. It’s kind of this warped, like, Turing machine sort of thing. And yeah, it is Turing complete. It can do all these interesting things, and it’s completely unreadable and is very much true to its title. And there is this whole realm of esolang or esolangs, these kind of esoteric languages where it’s sort of like programming language as a twisted work of art. And there are a lot of those. They’re very interesting because they give you a sense of the true space of programming languages.
That being said, by and large, the vast majority of programming languages kind of fit into a number of different categories where most of them are sort of like the imperative kind of C-like languages. And then you have some that are more functional, like Lisp or something like that. But by and large, there are these straightforward sorts of categories. And as a result, oftentimes, if you learn one within those categories, it’s actually not that hard to learn all the other ones in those categories as well. But there is not non-English variety as far as I’m aware.
Jim: Turns out that there was a German translation and a Chinese translation of Python. Those seem to be great. They basically just translate the keywords, so that wasn’t really a new language. There were a couple other weird little languages, but nothing much. It was kind of interesting.
Samuel: I like seeing those, the esoteric languages. It really gives you a sense of, oh wow, programming can be a lot weirder than I might have expected.
Jim: Of course, the one line that is relatively significant, and I think it’s growing on the functional side, is the line between imperative programming and functional programming. Functional programming goes way back to one of the first computer languages, which was indeed LISP. And now we have things like Haskell, Erlang, and F Sharp, and all kinds of other things. Wanna take a shot at trying to describe the difference between imperative programming and functional programming?
Samuel: Yeah, I’ll do my best. So imperative, really, the way to describe that is it’s a series of instructions that are kind of executed one at a time and they operate on variables and information and there might be loops and things like that. While functional languages are based on taking functions that operate on other functions. You would have a list of numbers or code or objects or whatever or variables, and then you would have a function that would operate upon that list. And then you could make another function that could operate upon that list. And so it’s kind of these functions that then operate on other functions and lists. And so it’s this kind of very nested sort of way of thinking, while imperative is much more like one step at a time. I think that would be a rough, likely not entirely accurate way of describing the distinctions, but I imagine it would be one where one is nested and one is stepping through. Right? Does that seem like—
Jim: It’s not exactly wrong. No.
Samuel: Yeah. I would say the other interesting thing about functional languages – and as you mentioned before, some of the barriers between these kinds of languages are breaking down. So for example, you can do functional programming within Python, which might have more traditionally imperative features. Actually, I do a lot of that where you can use the word “lambda” to generate your own functions, and then you take those and operate on a list or array or do whatever you want. The one interesting thing about functional programming is it often involves recursion, this kind of self-reference in a way that non-functional programming does not. I remember the first time I learned a more functional language after having learned traditionally imperative languages – it was just a wildly different way of thinking and kind of required me to rewire my brain and rethink how I thought programming could operate. It was frustrating in the beginning because I had no idea what I was doing. And then once it clicked, it was enormously eye-opening. That’s another powerful feature of functional programming.
Jim: And then also typically, functional programming things are immutable. That’s the other key aspect. The interesting thing about recursion is I have long been saying that recursion is a good tool for smart people. But in fact, I made a life career decision back in the mid-eighties when there was this big LISP explosion, one of the earlier AI summers before the AI winter came along. Symbolics, the LISP Machines Inc, lots of my college buddies went to work there to try to get me to come work there. I came over and visited them over in Cambridge – my little company was down the road over in Allston. I went and talked to them and they showed me all this stuff and I caught myself a LISP. But I said, “Dude, you got a serious problem. You’re trying to sell technology to the IT departments of major corporations. To be able to understand recursion requires an IQ of 130. Number of IQ 130 and above in corporate IT departments, I can tell you from personal experience, is small.” So your whole enterprise here is based on a false premise. Thanks, but no thanks. And I still warn people of that. If you have PhD computer science type people, or maybe just a tad below that in capability, then yeah, encourage recursion. If you have normal IT people, absolutely forbid the use of recursion without permission of the VP of Engineering.
Samuel: Well, and it’s similar to what we were talking about with global variables before. Like, there are some very powerful techniques, but we traditionally shy away from them because they have sometimes a little too much power. And I feel like recursion in a very different way is another one of these where it can be very, very useful. It can also completely destroy your memory if done properly. Not your human memory – oh, it could do that too – but also the actual memory of…
Jim: Your machine. Can run away. I will allude to my Network Wars game, actually. The world builder system I wrote was this horrendous spaghetti code of special cases and all that. And I had a blinding insight that I could make it all go away with a very tight recursive function. And I had it all in my head for about the fifteen minutes it took me to write it down, and it worked. And the thing’s been in the marketplace for like six years, and no bugs reported. It was there, but it was like one of these things where I probably reduced 500 lines of code to 30 lines of code.
Samuel: That’s amazing. And those kind of experiences are… yeah, it’s a very special moment when you have these flashes of insight and realize you can combine it and do it this very different way. And then after those fifteen minutes or whatever it is, you realize “I don’t fully understand what I did anymore.” But it seems to have worked.
Jim: Yeah, it worked. Never had any… I tested extensively with a test framework I created that could test for valid worlds and never failed. I go, “I don’t know. I don’t know.” I mean, I’m gonna go back and look at that code. It scares me to even think about it. Let’s move on to the next topic, something that’s always been near and dear to many of our nerdly heads. I’ve never found many that were that good, and that is software for thinking. Software for thinking or tools for thought.
Samuel: This is something that people have wanted quite early on in computing. People have thought about computers not just as tools for simulation or calculation, but as a way and means for helping you analyze information. And actually, one of the most seminal thought experiments for this was around the time of the beginning of the first digital computers. It’s this essay in The Atlantic, I think, in 1945 by Vannevar Bush, who was a developer of eventually the National Science Foundation. It’s called “As We May Think.” In it, he describes this thought experiment, this machine called the Memex, which is this desk with projectors and microfilm. It’s this very interesting thing where he’s basically describing a rudimentary World Wide Web, where you can connect to lots of different information and step through it and trace the way in which you do your research and think about these kinds of things.
This kind of thing was sort of this North Star for many years. Through the years, people have tried to build software as a way of helping us think better. Because the truth is we are very limited beings. There are a lot of things we can do well, and thinking is one of them, but thinking also has an upper bound. There’s a small number of things we can keep in our heads at the same time. If I want to go through all the things that I know or things that I’m thinking about on a topic, I can’t just rifle through my mind the way I would like a file cabinet. I have to kind of just have free association, and that’s gonna be extremely imprecise, impractical, and nonsystematic.
If we could have tools to help us do the act of thought better, then we should build these kinds of things. The people who have thought about computers as basically tools for thought, as machines for helping us improve our thinking, have really been guiding this kind of thing. Certainly, spreadsheets have been one of these kinds of things. Spreadsheets that allow us to imagine the world as it might have been are certainly these kinds of tools for thought.
In many ways, a lot of the current excitement around AI and LLMs does this kind of thing. The way I view this is if you want to be able to stitch together ideas or help with brainstorming, you want to be able to have some sort of landscape of ideas or information. That is where the world of LLMs operates. It operates in this latent space of ideas that are kind of all embedded in this high-dimensional space. If you can navigate that, then that can actually ideally help with some of thought.
People have pointed to ways in which if you pour scientific articles into these things, maybe it draws connections between different papers. I think I point to one example of this paper where it was looking at research within material science, and there was some material that was discovered with certain specific properties. They showed that if you had poured in all the literature before that discovery and looked at how these papers were related based on key terms and things like that, you could have actually had an AI-based discovery showing or at least hypothesizing that it would have had this property several years earlier than would be expected.
I actually think with AI, you might be able to have some of these kind of AI tools for thought. That being said, a lot of the tools for thought space is kind of colonized by better note-taking apps and things like that, which are not bad. I’m always a fan of better note-taking apps, if only for the fact that the more people transcribe and capture when it comes to information, the more raw material they’ll have when they’re going back and searching or creating new things. Whether or not they need the full features of all these is an entirely separate thing. But yeah, I love thinking about these sorts of tools and at least the need for these sorts of tools.
Jim: Yeah. It is interesting that I’m now using AIs a lot for that purpose where you sort of half know something. Now what exactly is Hegelian dialectic? You know, I know enough that A, B equals C. But obviously, there’s a lot more to it than that, so I’m engaged in this mailing list with a bunch of Hegelian philosophers. So before I jump into that, I did deep research. ChatGPT-3: explain to a smart 15-year-old the Hegelian dialectic, and you know, you quote actual stuff from Hegel in reasonable to understand English. Right? And oh my god, so they have an AR-15 for arguing with Hegelian philosophers that I would never have done on my own. But on the other hand, you reference all these various personal organizing tools. I’ve tried so many of them over the years, and not a one has ever stuck, at least for me. Have you ever found one that worked for you?
Samuel: To be honest, I don’t use the most sophisticated ones. I have a fairly simple note-taking app, and you can stitch together different notes and say, okay, here, this one is connected to this other one, or have various citations to other notes. For me, I think just the fact of capturing the things I want to write about, that’s the most important thing. I’ve never really used anything that sophisticated. I’ve tried. In general, it’s also the kind of thing where the best tool is really the one that you use regularly. It really doesn’t matter the specific details.
I actually remember this – years ago when I first started writing my first book, I got the book deal, and then I said, “Oh great, I’m now gonna spend the next few weeks looking at all the best writing tools and figure this out.” And my editor told me, “Just use Scrivener.” He said a lot of people use it, it’s good enough, and then you can just move on with your life. And I’m still using Scrivener, and it’s a great tool. I’m sure there are many more sophisticated tools or specialized tools, but it’s good enough, and it’s the one that I feel comfortable with. I think that really is what it comes down to – you kind of just have to choose one and live with it, and that will work.
That being said, I think having a simple tool that maybe is able to engage with AI, or you can kind of pour your notes into AI and then look at them – I think that kind of thing can be very valuable. Having those kinds of conversations with AI around trying to understand things or figure out the specific jargon and terms that you might have on the tip of your tongue but can’t figure out exactly what it is – that kind of thing is really useful for. But when it comes to note-taking, to be honest, I don’t actually have that sophisticated taste myself.
Jim: That’s funny. I’ve always thought, like you, it’s worth spending some time finding the perfect tool. And then I try one and go, no. I go all the way back to Lotus Agenda, probably one nobody remembers. I was friends with Mitch Kapor, and that was actually his favorite piece of software. Lotus Agenda. He later spent millions trying to create an open source version called Chandler, which never cohered, and it was even more sophisticated than Agenda, which was quite sophisticated. But I’ve tried Evernote, I’ve tried Notion, I’ve tried Obsidian, I’ve tried four or five different flavors of mind maps, and none of them have ever really done it for me. It’s funny. I still use Scrivener. You know, for a writing project, Scrivener is at about the right level.
Samuel: It’s sophisticated. There’s a lot of things you can do in it, and it’s also like open-ended enough and kind of-
Jim: The downside, it’s like 1997 technology, something like it.
Samuel: Yeah, it’s not the most… yeah, it’s not the… does not feel the newest.
Jim: Trying to share files with other people is a pain in the ass.
Samuel: Oh yeah, you kind of act poorly.
Jim: The other one is just a funny old tech gazing tech story was Symantec, now a huge company. Right? Back in the day, it was three guys above a bank in Central Square in Cambridge. And their first product was something called Think Tank, which was an outlining tool written in assembler language for Windows. Fast as shit. For years, whenever I started a new or a business plan for a new business, I always say, think of 100 business ideas, I actually work on 10 and I do one. Right? So at the 10 stage, I build little outlines. And for years, even after you couldn’t get DOS machines anymore, I always kept the DOS machine of old enough vintage to run my copy of Think Tank because my fingers were chip.
Samuel: And I think George R.R. Martin, the novelist, I think he still has a DOS machine for WordStar.
Jim: Oh my god.
Samuel: Because he was most comfortable with that. Yeah, and so it’s really it’s whatever tool you’re most comfortable with. Like, that’s the one for you.
Jim: Alright, let’s now step up from our own little petty keeping track of stuff to this very area much closer to the work I do in the real world, which is numeric modeling and world simulation. Tell me about your adventures in learning about that.
Samuel: Yeah. So my own personal experience has probably been in the realm of complexity science and complex systems. I kind of got into it initially in things probably related to biology, but then also in social systems and network science. There’s the whole realm of agent-based modeling, using agents that all interact. This is basically the whole idea of large systems with huge numbers of interacting parts.
But more broadly, whether or not you’re using differential equations or agent-based modeling or whatever tools you want to use, computers are able to do this really interesting thing. Prior to the modern digital computer, the ways in which you’d build a model is you’d have a description of a system – like a mathematical formula that was kind of just inert, or some sort of physical model that maybe could be dynamic like an orrery of a subset of the solar system. You kind of crank it and the Earth moves around the Sun, the Moon moves around the Earth, and you can kind of see all the relationships there and maybe look and see how eclipses work.
But not until the digital computer did you really have this ability to create a description that actually was dynamic. And that was, at least in my mind, one of the most interesting things about computing – that you could build these descriptions that really elucidate entire dynamic worlds as simulations. And in truth, these are one of the first things that were done with modern computers. It wasn’t like we had to wait until the seventies, eighties, or nineties. This is like almost immediately, these were the first use cases.
Since then, through a combination of better data as raw material, better computing power, and better algorithms, these simulations have gotten more and more sophisticated. We can now model the weather in ways that we could not possibly have imagined decades ago. We can model how pedestrians move through stadiums and trying to understand the behavior of societies. There’s all these different applications we have.
But at the same time, we still have to remember, these are still models. They are necessary simplifications. So I think one of these questions is how do you balance the sophistication of the model with things that are still kind of understandable to the user? What is the goal of the model? Is the goal purely prediction? Is it understanding? Because if it’s understanding, then a simplified model is actually going to be much more useful. So for example, SimCity is not actual cities, but if you just want to gain some sort of insight into the ways in which complex systems have unanticipated consequences and systems bite back and things like that, SimCity is great for that. Is it useful for parking in a city? Not really. I don’t think there are actually any parking lots in the original SimCity. But that kind of consideration of how we balance simulation and modeling and just the fact that it can do these things, I think, is incredibly exciting.
Jim: Yeah. A lot of the work we do at the Santa Fe Institute involves agent-based modeling and large-scale simulations, things you write about in the book. A-Life used to be a hot thing at Santa Fe.
Samuel: Yeah. So I have a whole chapter about aLife. One, because I did just personally find it fascinating, but I also think it’s this in some ways, while it did have its heyday in the 1990s, maybe like early 2000s, there’s actually some interesting things still happening right now within aLife, and I’ve been following it carefully and have a number of friends who are kind of involved in this space. But also, it’s just an unbelievable example of one of the very fruitful ways of thinking about modeling and simulation, which is not necessarily trying to model the entire world in great detail or fidelity, but rather saying, okay. I’m going to strip down a certain field down to its essence, down to its bare bones, whether it’s the features of life, the nature of evolution, and then slowly but surely build it up within computation and see what are the features that matter, and then how do these pieces interact and really try to get an understanding of, in this case, not just life as it is, but as they say, life as it could be. And I find that entire thing absolutely fascinating.
Jim: Yeah. I was totally addicted to aLife when I first retired from business, and I started playing with evolutionary computing first, and then aLife and then agent-based modeling. And when I went out to Santa Fe Institute, I was so proud of my models with 47 parameters, and they’d slap my hands and say, boy, get those – you don’t know shit about a model who got 47. If you get it down to three, we’ll talk to you. If you get it to two, it’s better. If you can find one parameter, best of all. Yeah. Then they can take that to art to some degree. But on the other hand, to model real systems, you do need a fair number of parameters, and aLife is fine. Because as you point out, you can do the simplest things like Sugarscape, which are very, very simple early agent-based modeling. You know, landscape of sugar and mounds of sugar and things that go around and eat sugar and fight each other about sugar. And then you can have very elaborate aquariums in a computer that have very elaborate food webs and reproduction and seduction and everything else. Always like, who’s currently working in aLife? I’ve truthfully, I haven’t looked at aLife in probably fifteen years.
Samuel: So one person who is doing a lot of work and kind of managing some organizations related to this is Olaf Witkowski. He’s based in Japan. And actually, I think a lot of some of the cutting-edge work in aLife right now is actually happening in Japan as well.
Jim: Interesting. Yeah. We do on the podcast a lot of origin of life work, which is different but related. Right? There’s some bleed through, and it’s gonna kind of have a different purpose, but they also use essentially biochemical simulations to think about what might be possible and proto-cells and things of that sort.
Samuel: Oh, yeah. And there’s definitely some overlap there in terms of trying to kind of really understand where life comes from, how life works, how it does its thing. And it’s absolutely fascinating to see how people are thinking about these kinds of things.
Jim: Let’s go one last thing to exit, which is something that I say, don’t ever tell anybody about this when they’re tripping, and that is the simulation hypothesis.
Samuel: Yeah. So the simulation hypothesis is essentially this idea that asks whether or not everything – like us, humanity, the entire universe – is all in a computer program. And to be honest, as the question stands, it’s a fun thought experiment. There’s no real way of answering it as it is. For me, I find it very useful not as kind of the answer to everything, but more as a means of thinking about our relationship, like the relationship between physical reality and computation.
So whether or not you want to think about what would it mean to play at the edges of a computer program that you yourself are in – what would that mean? Questions around what does it mean to even ask the question about the laws of physics actually doing computation? These are all kind of fun, weird questions, and the simulation hypothesis allows you to ask those questions in kind of a fun, sideways way. And so for me, I’m basically agnostic as to the question. I think trying to answer it entirely feels like an exercise in futility. That being said, taking some aspects of it seriously is enormously fascinating, and it can lead you to think about buffer overflows and weird little exceptions and breaking out of computer games. So it was a fun question to examine, but only in the sense that it leads me to think about the deeply physical nature of computers or other aspects of games or what have you.
Jim: Yeah. I’m with you. It’s an interesting thought experiment. I remember we, some of us were first talking about it as sophomores in college, you know, going, “Oh, I wonder if our universe is a simulation in some kid’s computer.” Right?
Samuel: I feel like that is, like, the ideal time for talking about that question.
Jim: With a couple of beers along the way. Right? And I’ve also come to the conclusion that interesting thought experiment but probably way above the pay grade of humanity anytime soon to know the answer to. And one of the nice things about learning about complexity science is to realize there are lots of questions we may never know the answer to and just be happy to live in the air in this kind of meso scale that we live in, which is its own interesting stack of emergences. What happens below? What happens above? We may never know, and who cares? There are plenty of fun things to operate in the scale at which we can operate.
Samuel: That is a great philosophy and approach.
Jim: Yeah. And, unfortunately, not that common one, unfortunately. In your book, you mentioned in passing two billionaires that supposedly had funded research in looking for glitches in the matrix. Is that really true?
Samuel: That is mentioned in a vague way in one New Yorker article. I think it was in the context of profiling Sam Altman a number of years ago. And a number of people have kind of mentioned that as well. I do not know the details. I have no idea the veracity of that. I assume it is at least somewhat – there must have been some accuracy to it back when it was written due to the New Yorker’s fact checking. I do not have the names of those billionaires or any details. Yeah, that would be very interesting to learn more about that.
Jim: Yeah. And I suppose if I was a billionaire, I might put $5 million against looking for glitches in the matrix. Why not? Right? Better than buying yet another large yacht.
Samuel: My concern, though, is what happens if you do find those glitches and we get turned off? That could be a very negative outcome.
Jim: Or, you know, as people who hack things like Second Life and things like that, remember, you can find glitches in the matrix and make infinite amounts of money because you figured out how to create money out of thin air. That’s probably Peter Thiel’s motivation. Anyway, well, this has been a very fun conversation. I want to thank Sam Arbesman for coming on and talking about his really quite nice book.
Sam: The Magic of Code. Yes. Thank you very much. I had a wonderful time.
Jim: It was a great conversation. Really good.