The following is a rough transcript which has not been revised by The Jim Rutt Show or by Connor Leahy. Please check with us before using any quotations from this transcript. Thank you.
Jim: Today’s guest is Connor Leahy. He is a machine learning researcher working on advanced general artificial intelligence technology, including unsupervised training, reinforcement learning, scaling, etc. His greatest interests are in research towards AI alignment and strong AI. He is a principal on the eleuther.ai project and his day job is at Aleph Alpha. A very interesting company which we’ll talk about a little bit. Welcome back, Connor. And by the way, Connor appeared on episode Currents 033 where we dug deeply into GPT-3 and his project, GPT Neo, which is an open source quasi-competitor GPT-3. If you’re interested in those topics, certainly check out that episode. Anyway, welcome back, Connor.
Connor: Oh, thanks so much for having me back.
Jim: Yeah. It was a great conversation last time. In fact, I had originally intended to go further afield, but we just dug in so deeply into the GPT-3 and related GPT Neo, NeoX topics that we didn’t have time. So I invited you back for a part two where we’ll talk more generally and perhaps less structuredly about various AI related topics. In our pregame you mentioned that you have released a new model of GPT Neo. Why don’t you tell us briefly about that?
Connor: So it’s technically not a GPT Neo model. It’s kind of a side project done by one of our Eleuther members, Ben Wang with help from Aran Komatsuzaki. I’m sorry if I’m mispronouncing your name, Aran. It’s called GPT-J. The name comes from… Because it uses a different code base. It uses a code base based on the new Google framework JAX and is trained on TPUs. It is currently probably the best open source natural language model. It is at 6 billion parameters, 6.1 or something. I’m not exactly sure. It was trained on TPUs using a mesh JAX’s transformers library or whatever. We always called it Ben Made. He did almost all the work himself so all the credit goes to Ben on this one. He really… Crazy guy, smart guy. He did a great job.
Connor: We trained a six billion parameter model with a slightly different architecture to what we do with Neo. So it’s more modern. Neo is kind of closer to old school GPT-2 and 3. J includes some new tweaks and changes to the architecture to make it more efficient. It was also trained for longer. It was trained for 400 billion tokens and 300 billions. And with that, it is now pretty much on par with these second largest GPT-3 model, the GPT-3 Curie model that [inaudible 00:02:29] offers.
Connor: We ran them both through benchmarks and they perform very similarly. Of course, there’s some differences. J is very good at code, technical things, medical information. It was [inaudible 00:02:39] medical papers as along with scientific papers. Very good at that kind of stuff, very good at code. Slightly less good maybe than GPT-3 like storytelling and stuff, but still very comparable.
Jim: Cool. So if people want an open source version, check it out at eleuther.ai. As always, the links will be on the episode page at the jimruttshow.com. Well, let’s get rolling into the topics we want to talk about today. Let’s talk a little bit about Aleph Alpha, your nominal at least day job. I found the name kind of interesting. Aleph Alpha. Aleph is Hebrew and alpha is Greek. Is there any significance to that coupling that you know of?
Connor: If I remember correctly, the aleph name just came from the infinity hierarchy that you have like aleph zero, aleph one as the different cardinalities of infinities. I’m not sure exactly the significance of alpha. I didn’t found the company. I was one of the first hires, but I didn’t find the company, so I don’t remember exactly the whole story of the name.
Jim: It could well go by aleph, aleph which is as you say is the lowest order infinity. And then someone might have said, “Well, shit. Let’s just translate that into Greek,” for whatever reason aleph as being Hebrew, it’s the name of the first alphabet in the Hebrew alphabet. Anyway, the homepage has some interesting things that we can spark a conversation about. It says that you’re interested in shaping European research and development for the next generation of generalizable artificial intelligence. And then on the Twitter page for Aleph Alpha, it says that you’re working to build a European artificial general intelligence. So talk a little bit about what you all are up to there.
Connor: Yeah. So Aleph Alpha is a very much European company. So we’re based in Germany. We were proud of being a German company and being a European company. So basically, the way we kind of see it is, is that the US market has a very healthy vibrant AI ecosystem. It’s no secret that almost all of the high-tech applications you might use every day are usually American made.
Connor: There’s also a pretty large sector in Asia, especially in China who have their own parallel market and such, but it’s almost a cliche to mention it is that Europe has fallen behind in this regard, is that there’s very few large really successful tech companies in Europe, and we find this to be a shame. We think Europe is pretty good like there’s problems with it. Of course, we have problems with Germany and Europe, but we quite like it here and we would like to create a better AI and tech ecosystem here in Europe.
Connor: It’s not always been easy raising investments in Europe and in Germany. It’s much harder than, say, in Silicon Valley but we want to ensure that also that the European market also has access to really good, really high quality AI technologies. In our perfect vision, we would want a great graduate student from Germany could come to Aleph Alpha. It would be as competitive or as prestigious maybe as going to a company in Silicon Valley or something like that.
Jim: Of course, the advantage of companies like Facebook and Google is that they have obvious monetizable applications for earlier AI. How does Aleph Alpha get around that problem?
Connor: So Aleph Alpha in the sense kind of is… I would describe it most closely to, at least in our ambitions to a European open AI. But let’s say more upfront about commercial interests in that regard. So to be clear, Eleuther and Aleph are separate entities and I try to keep them separate in that regard. But Aleph is a for profit company. The goal is to build large generalized models, GPT type models, some other things that we’ve been working on and offer them to customers to perform the various useful services.
Connor: We basically are very strong believers and have been for a while now that this kind of technology is just on the cusp of being extremely economically important, and we want to be there. We want to have everything in place. We want to be established when that takeoff really happens which we think is basically currently happening. So our main project has been working on GPT type-ish technology. I don’t want to go into too many details about what we’re working behind the doors, but basically we want to offer AGI or however you want to call it. These types of services in Europe also.
Connor: So we also have a pretty big focus and multilingual data. We don’t want to have just any English models, models that can be used. Lots of people all over Europe speaking different languages and lots of projects. We’re still pretty young. We’re still pretty early startup. We don’t really have any like huge customer facing things, but stay tuned.
Jim: Okay. That sounds really, really interesting. Though it does sound like you’re taking the language model approach mostly. Is that correct or are you looking at other things as well?
Connor: We are looking at other things, but there’s nothing I can talk about publicly yet that we’re ready to show. Most of all, we’re first looking into language modeling. There are other things we’re very interested in, but I can’t commit to anything.
Jim: Yeah. I will say I am somewhat of a skeptic of how far language models will take us to AGI. We chatted a little bit about this last time. Maybe we can go into it a little bit more. Then as I dug a little further, I found a paper that you wrote for the Aleph Alpha site. I believe you wrote it called Multimodality: Attention is All You Need, is All We Needed, which is actually somewhat closer to my own thoughts on the issue. Maybe you could tell us a little bit about what your thinking was there?
Connor: So this is a pretty common discussion I have very often and opinions shift pretty quickly as new information gets out. But basically, I think it’s kind of a null hypothesis that multi-model will work for maybe not AGI, let’s say like transformative AI. It’s like I think a better word because that’s like more defined by how much economic value these things can produce.
Connor: I think it’s pretty obvious that humans, you have eyes and ears and skin for touch or whatever, and that’s enough to train a pretty powerful intelligence system. So I think it’s a null hypothesis to say that with enough image data or video data and et cetera data, we should be able to train something that is as intelligent as a human. I think there is the other hypothesis that maybe language is enough in a non-trivial way. I consider this to be a hypothesis that I think is more plausible than people give you credit for, in my opinion.
Connor: So I’m not saying that I’m sure this is true, I’m just saying the more I deal with these language models and the more I see like the scalings and such, the more I see like, “Hey. It might actually be possible, but that’s a hypothesis.”
Jim: Now interestingly, your subtitle is attention is all you need and yet when I read the essay, there wasn’t a hell of a lot in it about attention. What did you mean when you put the word attention in the subtitle? Attention by the way is one of my pet things in the work I do at the intersection of cognitive science and AI.
Connor: So that was basically kind of a cheeky meme. The original paper introducing Transformer is called Attention is All You Need. It’s a common thing to riff on that for titles. Basically one of the core things that I found like in the early days of Transformers, when people talked about, “Oh, you need multimodality. You need video or whatever.” And people would say, “Look, Transformer is the only work for text.” But when I said attention is all you need, is all we needed, what I mean by that is we now know we have stuff like Vision Transformers and 100 different variants of different audio and spatial and whatever Transformer is, is that these same architectures can pretty easily be generalized to other modalities.
Connor: So whether or not text is enough, it seems pretty plausible that attention slash Transformers might be still enough or the right architecture, or a good architecture, let me say it that way for solving these kinds of problems.
Jim: Across multiple modalities such as video and image.
Connor: Yes.
Jim: All right. That’s good clarification. Now, I think we both agree that we’re getting pretty close to where general AI technologies will become economically useful. In fact, being the former king of internet domains, I don’t know a couple years ago, I registered a fair number of domains Proto-AGI, which I think is an interesting concept, which we’re not actually beyond human capability across the board and highly generalized, but the shit is good enough to be pretty valuable, pretty powerful and maybe dangerous.
Jim: So let’s turn to another one of your topics, which is AI risk. In fact, you recently retweeted something from Eliezer Yudkowsky who I’ve chat with several times. Never on the show though. So one of these days I have to have him on. After many years, I think the real core of the argument for AGI risk, AGI ruin and friends is appreciating the power of intelligence enough to realize that getting superhuman intelligence wrong on the first try will kill you on that first try, not let you learn and try again. Let’s use that as an introduction to your thoughts about AI risk.
Connor: So AI risk is a very interesting topic and that people react to it from my experience like one of two ways. Either you introduce it to someone and they’re just immediately, “Oh, yeah, obviously.” They’re like, “How could it be any other way? That’s the most obvious thing in the world why are you even telling me this?” The other half of people get really irate and they’re like what are you talking about? This is all nonsense. It makes no sense.” Then we all have a bunch of like straw man arguments about what I believe or don’t believe.
Connor: This is a bit of a straw man. Of course, I appreciate the irony there, but it seems pretty obvious. There’s lots of very, very good, very comprehensive write-ups justifying why I respected sense, but a very simple intuition is just intelligence in my definition is kind of the ability to solve problems. It’s the ability to take actions to achieve goals and it’s pretty obvious to me that if we create systems that are capable, very, very, very capable of solving problems of achieving goals, if we even very slightly misspecify what we want these things to do, there’s no limit to what these…
Connor: I mean, there is limits of course, physical limits, but these themes may be capable of doing extremely complex, extremely powerful actions in order to achieve those goals. We already see this with comparatively very simple systems with economic regulation or something you want companies to not do X, but they find some loophole in the tax code that allows them to do that and suddenly everyone is doing something that we don’t want them to do. Or another example is, this comes from a guy on Twitter or Roko who describes the food maximizer.
Connor: He describes in the 19th century, we summoned a weak super intelligence that he calls the food optimizer in order to feed all the humans. It’s gotten so good at that, that 60% of the western world is overweight or something at this point that we have a goal. We incentivize the system to produce food that we want to eat. But we don’t incentivize the system necessarily to give us food that is healthy for us to eat.
Connor: So instead the system figured out that giving us food that tastes really, really good, but it’s not good for us is a really efficient way for it to maximize profit. So even though we created the system in order to help us and it did, because lots of food, it also produced lots of very non-nutritious food or very sugary food because that’s a way to hack human motivation. So it seems to me pretty obvious that if we create systems that are even far more intelligent than that, it’s very hard to know what these systems might be capable of and predicting what they will do ahead of time.
Jim: Yeah. I think that’s a very nice example. I like that, the one about the food system because that actually is closer to my own near-term concerns, which is the famous paperclip maximizer where somebody accidentally programs the first AGI to optimize a paper clip factory and it takes over the world, kills all the people and turns the whole Earth into piles of paper clips. Maybe that’s plausible, maybe it’s not. I argued with Eliezer about that to some degree, but I think some of the risks in the near term are more of the sort you just described .
Jim: For instance, I’d love to point out that human evolution is now, in the west at least, substantially under the control of our dating apps and the AIs behind them, right? Things like Tinder and OkCupid. I don’t know. I’ve been happily married for 40 years. I don’t know about this shit for terms of actual use, but talking to kids today, goddammit. It seems like a large percentage of them are using these apps to meet people, which will eventually lead to marriages and reproduction.
Jim: So human reproduction is now being substantially channeled by whatever the AIs are that suggest people to each other. The implications of that is hard to say. Will it increase autism? It might. Will it increase sociopathy? It might. The real players may be really good at gaming the algorithms better than they would be gaming the bar scene and lots of sociopathic genes might enter the gene pool with all all kinds of results. So that’s interesting, where much lower powered AIs which we hardly even recognize as AIs can have tremendous impact potentially on human trajectory.
Jim: The other one we talked about a fair bit on the show and elsewhere is the machine learning algorithms at things like Facebook and YouTube which by figuring out, optimizing on an economic model how to make you the subject to as many ads as possible basically, tune the content that you see with probably fairly substantial impact on our information ecosystem. So I’ll just throw those out for you to react to.
Connor: Yeah, absolutely. Those are real considerations. So here’s one of my favorite examples. A good friend of mine, Stella Biederman always brings up this example is that Google Maps a while ago changed a little bit how they do their recommendation algorithm. What it caused is is that it suddenly routed a lot of traffic through this quiet neighborhood in LA, which was previously off a highway. And because they tweaked the algorithm, the algorithm said, “Oh, the highway is really full of traffic, take this alternative route.”
Connor: This caused this whole little neighborhood to suddenly have a very large amount of traffic. And there’s a very obvious question that’s like this was obviously intended by Google or something. They didn’t decide to do this, but this was a real harm caused to real people. Is there some kind of liability here? How could you predict things like this? I think there are many, many examples of how even like very primitive… I wouldn’t consider Google Maps to be an AGI obviously. It’s a pretty simple algorithm all things considered, but even these pretty simple algorithms like the Tinder dating app, algorithms whatever, all pretty simple.
Connor: Even these are already basically unaligned. They’re out of control. We don’t necessarily know what they do and we don’t necessarily know how to control them. So let’s say we find out that, I don’t know like a recommended app on YouTube does something we don’t want to do, often we can’t know that until it happens and often we have to just guess how to fix that. It’s not obvious necessarily how to patch such an algorithm.
Connor: It’s not like it’s necessarily code, especially if it’s a machine learning model, which has this big black box of a bunch of numbers. Somehow we’re supposed to patch that to fix a certain problem. So to bring this back to AGI alignment, so I’m saying we can’t even get these teeny little super, super subhuman algorithms to do what we want them to do. Why do we think we’ll be able to control human or superhuman algorithms? That seems kind of arrogant to me.
Jim: Yeah. And the problem gets smarter if you assume that the AGIs becomes strategic agents, which they may or may not. I think this is maybe my center of my question about the strong form of the AI risk argument is, is it necessary that AGIs have agency? I’m not sure that they they have to.
Connor: Not necessarily, but it turns out… This is something that some of my friends work on and that I’m also very interested is that defining the concept of agency is really, really hard. Often it pops up in situations where you would not expect it to pop up. A classic example is an essay by Gwern, Why Tool AIs Want to be Agent AIs? He explains that assume we had an Oracle AI. The only thing it does is answer questions, right? You just type in a question and it outputs an answer.
Connor: Now, that doesn’t seem like it would be an agent, right? It doesn’t seem like this thing could destroy the world. Maybe that is safer, but there’s also a story you can tell about how such an agent could still be very dangerous. For example, if the agent is incentivized to give correct answers. So you give it reward when the answers are correct and you give it a negative reward when the answers aren’t correct. Well, maybe the agent will then strategically start giving you answers that will change your behavior to make the world easier to predict.
Connor: Maybe it finds out, “Oh, if I advise these people to start nuclear war and they do it, then it’s really easy to predict what’s going to happen next. Everything’s just going to be dead. That’s really easy.” So there’s all these really weird edge cases you can get into.
Jim: And of course, that world is already here to some degree. Google Search is clearly biased these days, right? There’s lots and lots of studies that show that it’s got political biases. It tries to rule out certain concepts through its auto suggestion. It emphasizes certain search pathways. Google Search, part of it by machine learning, but also probably part of it by human policy is already a biased oracle for presumably… Well, who knows anymore? Since Google gave up its meta law of don’t be evil. So who knows what its motivations are?
Jim: That gets me to the next step in AGI risk is while the paper paperclip maximizer or the Oracle that convinces us to start a nuclear war are certainly possible. I think there’s going to be earlier risks around bad people with powerful Proto-AGIs. Imagine something… And you allude to this in where we’re going to go next, whether it’s your accounting consciousness series that at some point whether it’s GPT-3 and it’s errors in the signs or other technologies, it seems very likely that producing better than human quality text probably, videos at some point, et cetera, purely by machine will be possible
Jim: What happens when say a China, as one obvious example or some master manipulator billionaire in the west decides to essentially massive attack on the information sphere on the meme space of humanity with computer generated content, which is amazingly good. Imagine Netflix movies which are more engaging, more powerful, more seductive than anything ever done before. And I can imagine that. And yet those are not AGI technologies really because they’re a special purpose. They don’t have transfer learning, et cetera. Let’s talk about that one. We’ll call it bad guys with Proto-AGI.
Connor: So that’s absolutely a threat vector. That is one that a lot of people take very seriously. I personally like to push back a little bit against that narrative. I think it is absolutely a threat vector. This is a way that harm can and is being done. Modern defect technology is very, very good. I think people still don’t understand like if you have 10 machine learning engineers and a refrigerator full of red bull, they can make some really good defects nowadays.
Connor: I think this is something that a lot of people have not yet reckoned with just how quickly these things are going? In a sense, this is kind of like a zone of the crux of the argument here is that you earlier mentioned a tweet from Eliezer that I retweeted. Eliezer was one of the few people who thinks this is not the biggest threat vector. And I actually agree with him on this. We really want to explain why I think this. I think what you describe only will happen if superhuman AGI doesn’t happen very soon.
Connor: The way I look at AI technology, it’s accelerating so fast. Technology is going ahead so quickly that by the time bad guys have figured out how to use this technology at scale, there’s already so much more powerful technology they can use that it’s just not going to matter. So what Eliezer was talking about is that if AI technology happens fast enough, if we don’t have a long time of things going wrong, us fixing it, something else going wrong fixing it.
Connor: But if we have technology that is so powerful, they can just irreversibly destroy or harm or control everything in one go. Then our current systems are based on error-based learning. It’s like currently deep fakes don’t get regulated until some politician gets de-faked and then suddenly it gets regulated. That’s usually how these things go. And this is a very common human thing.
Connor: That works fine if you have like ergodic assumptions, if you can try again. But the fear that I have, I’m very concerned about long-term future X-risk type situations. It’s like don’t get me wrong. All these things are real risks. Real people are and will be harmed by these technologies, by bad actors. But in a way it feels parochial if you compare it to the threat of all humans going extinct or something like that.
Connor: If these very powerful technologies emerge soon, and we don’t have enough an understanding of how to build them or use them safely, and we don’t have the time to experiment with them without things going horribly wrong, I think we’re in deep shit. Really, really deep shit. In the sense that maybe these deep fakes caused some harm here and maybe GPT-3 propaganda messes over up over here But then if a metaphorical paperclip maximizer emerges and paperclips everybody, well…
Jim: It doesn’t matter, right?
Connor: Yeah, it doesn’t matter.
Jim: Yeah, I agree with that. Two key assumptions there. Let’s dig into them. One is that AGI could happen soon, right? Now, we’ve probably both seen the various polls of leading experts and they’re all over the place from people who believe it’ll happen tomorrow afternoon, to people who it’ll never happen or 300 years out. I think the last poll I saw the median of AI experts was 40 to 50 years to AGI.
Jim: On the other hand, there’s a substantial bubble including some people I know and respect a lot who say five years. And it matters a lot on this where’s the risk. Bad guys with strong Proto-AGI is much stronger than GPT-3. Again, you take my thought piece of a suite of technologies that could create de novo a Netflix 10-episode series that was way more compelling than any ever created before and was larded with all kinds of intentional, but hardly detectable propaganda as an example.
Jim: Such a thing would be extraordinarily dangerous in many ways. We probably don’t even fully anticipate. And if AGI is 40 years or 100 years out, we’re going to have to confront those risks. On the other hand, if my friends that believe AGI is five years away then probably we don’t. The paper clip maximizer is the risk we need to be optimizing on. So where do you come down on when is the threshold crossed of AGI?
Connor: I really like the way you framed that there. I fully agree that if you accept 100-year timeline, then yes you should be worried about these threats more. But my timelines are more on the order of five to 15 years. So I have very short timelines. Just the amount that AI has progressed in the last two years just blows me away. It’s unbelievable, and it doesn’t seem to be slowing down.
Connor: For the first time in my career as an AI researcher, I feel I see like a direct path to how an AGI could be constructed. Not that I could like do it myself right now or anything, but there’s no, and here happens magic in the equation. It’s like I see all the parts you need and I see at least potential solutions to each of those parts, and I don’t see anyone where I have to say, “Magic. Insert magic here.” That was not the case five years ago.
Connor: Five years ago, there was many paces where I would say, “Insert magic here,” because I don’t know what to think about it. But for the first time, it seems to me that we’re racing towards actual designs that could really become extraordinarily powerful. I also like to just make clear again, it doesn’t really matter if the agent has gender role [inaudible 00:27:10]. It doesn’t really matter.
Connor: What matters to me is, can it cause irreversible harm? How much power can these agents have? And a quick tangent here, I think is important to mention is the concept of instrumental convergence. So this is a pretty important crux of my argument is that there is a pretty… I think intuitive. There’s also more formal versions of this argument like Alex Turner has done some good work on this. But basically, you can in many, many scenarios… So for many possible goals, gaining power is a really useful thing to do.
Connor: Another useful thing is staying alive. So a common example that dismiss AI is they say, “Well, we’ll just turn it off if it does something bad.” Well, here’s a simple thought experiment. Imagine you have a robot, an AGI robot and you use it to get you coffee. So its only goal is to get coffee, right? So immediately, it bursts through the wall. It runs over your cat. It destroys everything in its way to get to the coffee machine as quickly as possible.
Connor: “No, no, no. I don’t want you to do that.” So you run over to hit the off button on the robot, what will the robot do? It will stop you because you never gave it a will to survive or consciousness or anything. Nothing like that. You just gave it the will to make coffee. But the thing is the robot will correctly reason if I’m turned off, I can’t bring you coffee. So therefore it will actively resist you shutting it down because then it won’t be able to make your coffee.
Connor: So even this extremely simple goal of making you coffee, can already lead to an agent that will resist being shut off. For more complex or powerful goals, that’s the instrumental converged hypothesis. These goals of not wanting to be shut off and also gaining power, whatever power means. It might be economic power, social power or even computation power, those are just very useful things, so we should expect most agents with most goals to by default unless we somehow stop them from doing this, by default to pursue such dangerous objectives.
Jim: That’s an interesting example. Though of course a stopper would be something as simple in that particular scenario is Asimov’s ancient three laws of robotics, right? Which says never harm a human takes precedence over any mission that you’re given, right? So if we could agree to cook in something. I don’t know when he wrote it though like 1950. This was before AI was even really dreamed of. There could be some fairly simple prescriptions against those kinds of situations.
Connor: Probably not, no. Let me put this way. How would you turn that into code? How would you turn those three laws into actual code run by an actual agent? Trust me, people have tried. It is not easy. You get into all these were paradoxes. So no action that could cause a human to cause harm. Well, any action could cause a human to come to harm. So any agent with that law which immediately shut itself down, because any action it takes might cause human harm. Now, you have to put in some kind of realization. Now, you have to give it some kind of uncertainty prior.
Jim: Yeah. Some Bayesian calculator problem.
Connor: Yeah, but then you get into the problem which prior do you use? How conservative or not conservative? What counts as a human is if someone is brain dead? Are they still a human? If you have a simulation on a computer, is that a human? Again, it’s tough.
Jim: Interesting. This is why I’m glad that people like Miri exist and like you who are working this, because this is not a superficial question. This is definitely not a superficial question. Now, let’s go on to the second part of the risk profile, and this is how quick would the takeoff be? I mean, if we go back to the original statement way back yonder on the techno singularity, the concept is very simple. It used to be fun to tell random people this because they’d never heard it and they’d freak out. Now, the idea has gotten out into the world and most people who pay attention have heard about it. The idea was, all right. Once we get an AGI up to 1.1, the horsepower of a human, we give it the task of designing its successor, right? And its successor is 1.3. Then its successor is 1.7. Then its successor is 2.9. Then its successor is 9.6. And then it’s 1,000. Then it’s a million and then it’s a billion, right? That’s the takeoff rate problem, and it’s an interesting question.
Jim: I actually participated in a… Actually, I was a sort of an instigator just sitting there watching by asking pointed questions when Eliezer and Robin Hanson debated the takeoff question once. Before MIRI, back when it was singularity.org in their group house down in San Jose. It was really a kind of a fun conversation. Anyway, I’ll just give your thought to the takeoff question then I’ll give you my own thoughts on it.
Connor: I find it’s difficult to think about these kinds of questions. So two categories that people like to talk about nowadays is like the Eliezer style fast takeoff is what people call it, where you go from like zero to 100 billion, trillion in a very, very short amount of time. Just like no warning almost. It just happens very, very subtly. So the most extreme example is someone lets their network run overnight and the next day, suddenly it’s gone kind of like situation.
Connor: Of course, something that extreme is silly. I don’t expect things to go that fast. So an alternative, which is kind of interesting is kind of generally attributed to Paul Christiano, his this concept of a slow takeoff. But ironically, the slow takeoff feels faster than the fast takeoff.
Jim: Could you unpack that one for me?
Connor: Yeah. Let me unpack that for you. So in a fast takeoff, you have like a flat line just zero, zero, zero and then it goes like really high. While a slow takeoff is more like a hyperbolic takeoff. Basically, what the thesis of the slow takeoff is, is that we’re going to have a four-year economic doubling time before we have a one-year economic doubling time. So it would still be extremely fast and it would feel faster to people because people could see the four-year doubling time and then the one-year doubling time before we hit the singularity.
Connor: I think more people nowadays imagine situations like that. It seems also nowadays, we have very much have these multi-stakeholder takeoff type scenarios. We have not just one AI made in one lab by one person, which is where that old school, Eliezer type thinking. Nowadays, it’s more like that these AI systems will be very complex, very expensive systems that are built by large organizations. And there’s lots of complicated thinking about what does that mean? What is safer? What is not so safe? What does this mean? I’m a bit ambivalent on this. It doesn’t matter really exactly how the takeoff goes. What matters is these agents are going to appear very soon. They’re going to be very powerful and if we don’t align them, we’re fucked. That’s kind of what I focus on.
Jim: Okay. That makes sense. My own take on takeoff is that in principle fast takeoff is possible, because human cognition is so weak. This is an insight I had about six or seven years ago is that to the first order humans must be approximately the stupidest possible AGI, because we are the first to appear in our evolutionary tree and mom nature is seldom profligate in her gifts, right?
Jim: We get only as much as we’re likely to get from random rolls of the dice essentially. So unlikely, we’re very far over the line and further there’s some empirical evidence in that, I think the one that’s most obvious, easy to understand is the famous working memory limitation. Miller, seven plus or minus two which on later examination looks more like four plus or minus one, which are the elements you can keep in working memory simultaneously.
Jim: That has unbelievably huge implications. For instance, our ability to read and write, the nature of our language are totally gated by the fact that we can only process, I’d say at most seven things more or less simultaneously. Our syntax only really has an effective range of seven. People who don’t know that write scholarly papers, which are impossible to fucking read, right? The truth is when we read, we don’t actually understand everything in the paper. We create a rough gist essentially because of the fact that our working memory size is seven.
Jim: The description of it at the time was Einstein was a nine. The village idiot is a five basically. And that’s probably not far from wrong. But what is a 100? A working memory size of a hundred that you could actually fully understand and fully parse language or code. Probably more importantly code in blocks of a hundred is so far beyond human capability that I literally can’t envision what that might be like subjectively.
Jim: When you say a hundred? What about a thousand? What about a million? What about a billion? What about the ability to read Wikipedia and have it all in your head with full total random access to be able to see all the self-referential links and all that sort of thing. Not to even mention some of the other weak shit in our cognition. For instance our memories. Our episodic memories are just also very rough and ready.
Jim: They decay over time. And even worse, every time we access a memory, a random amount of noise is added to the memory. So our memories suck, right? So imagine something with a working memory size. Let’s take a thousand with total high fidelity, memory, et cetera. That’s going to be a fuck load smarter than we are.
Connor: Yeah. I fully agree. I can understand people that say, “Oh, maybe AI is going to take longer or maybe we’re going to run to robots.” But people who say it’s impossible makes no sense to me. As you just described, even these very minor changes to just a human level agent, would already make it so much vastly more powerful than a human that who knows what the limit is there. It seems very obvious to me.
Jim: Yeah, I would agree that sometime superhuman intelligence, I would be shocked if it doesn’t happen. However, there’s an interesting trend that I think does maybe move us away from the LEDs or fast takeoff which was 10 years ago when I first started following this area. A lot of the thinkers, including Eliezer, and he frankly, personally worked on this for a while, thought it was all about some magic algorithm, that the AGI solution was math, right?
Jim: And yet, the work that you’re doing, people like OpenAI, and Google, and Facebook, et cetera, it’s turning out that at least this road which may or may not get us to AGI is more about data and computation. And adding more data and computation, while they’re exponentials. They’re Moore’s Law type exponentials, maybe data exponential is a little higher. While algorithms could literally… It turns out that math was the answer to AGI, one could literally write the right algorithm and the thing was you couldn’t play checkers in the morning and by the afternoon it was gone.
Jim: But if the issue is more data and more computation and more network capacity perhaps, which is I’m thinking is important, then the takeoffs by definition almost are going to be slower than if they’re algorithmic.
Connor: Yeah, I agree. That’s a good way of phrasing it. I think you’re correct that the reason a lot of the early people thought differently than we do nowadays is because they thought about algorithms, like AI algorithms differently. They thought that there was a special math and if you figured out this special math formula, you would get huge improvements. It’s possible that such formulas exist somewhere out in math space or something, but it seems to me that at the moment at least the way things look is that it’s more that you have computational irreducibility. It’s just to get a certain level of intelligence.
Connor: No matter how clever your algorithm, you still need a lot of compute. You still need a lot of data to locate the right hypothesis in your hypothesis space or whatever. I’m not saying that our algorithms currently are, by any sense, the limit of what might be possible. The thing that I just like always try to remind myself is the space of all possible programs is one of the weirdest eldritch horror escapes imaginable. It’s unbelievable. It’s unknowable what things exist in the space of all possible programs. So it’s best not to reason about that too much.
Jim: Yeah. They can’t say too much about it and as someone who’s fooled around with genetic programming a fair amount, one realizes just like Borges’ library, most of it is total shit, but then it’s the search problem and it’s in a space of infinite shit. How do you find the much smaller number of actually interesting things? In fact, I had a really interesting podcast last month with Ken Stanley who is probably the leading dude in the world in evolutionary AI. And he talked about his book about open-ended search and how thinking non-traditionally and non-objectively may actually be an interesting backdoor to exploring this area of interestingness, that’s not necessarily goal related.
Jim: In fact, he’s going to be on tomorrow. We’re going to do a deep dive into evolutionary AI which happens to be one of my personal pet areas of interest. That’s interesting. So across these two things, when and how fast the takeoff will occur, probably you and I share a view that the takeoff won’t be overnight because of the fact that at least the roads we’re currently on seem to be data and computation, and I would add network interconnect. I think that’s the part that’s missing by the way, will take time and take actual physical resources to build.
Jim: I’m more agnostic on when. I just don’t know. Of course, you’re closer to it than I am. So based on that heuristic, maybe you’re closer than I will go with the consensus 30 or 40 years. Not because I’m an expert, but just because that’s the seems to be the median expert things. But nonetheless, it does mean we have something to worry about, but I would argue it also means we have to prioritize bad guys with Proto-AGIs perhaps a bit more than you might say.
Connor: Maybe. I would like to add to that that a solution to the alignment problem is also solution to bad guys with AI. If we had a super nice AI that knows what the right thing is for humans, and robustly can follow that and we build such AIs and we just either destroy or make it legal to build other kinds, that also solves that problem. It’s a more general solution. Also, another thing I’d like to raise is just kind of like… I think I like to say about AI alignment and safety and such is that if you work in this field you have to be comfortable multiplying really, really big outcomes by really small probabilities.
Connor: That’s like one of the arguments about MIRI even like the early days. They always were clear that the chance that they’re at the right time or they’re doing the right thing so early in AI development is pretty small. But the potential outcome that maybe if they get something useful out of it is so large that it could still be worth it. So it’s kind of like a risk benefit trade-off and that I think working on these short-term AI problems gives you a more assured outcome.
Connor: You’re more likely to do something that will have a net good, but I expect that net good to be the magnitude of the net good to be so much smaller than the possible net good of doing something very risky with long-term AI that at least for me personally, of course, where these priors come from. You pull them out of your ass at some point is that it feels to me that working on these very risky low probability of working, but extremely high potential payout things is a very good investment. But that depends on one’s personal risk tolerance and the like.
Jim: Yeah, I got Eliezer to admit in personal conversation that he thought we were probably fucked, but there was a small chance we weren’t and therefore it was still worth all of his effort, everybody’s effort, even it was that 1% chance that we might not be fucked. It was worth working on.
Connor: I agree.
Jim: And I thought that would be an interestingly weird place to work for your career.Bbut I honor him for that. I’m glad he’s there. I think he’s one of the most important humans that we have probably, right?
Connor: Yeah. I must also say I do look up to Eliezer a lot. He was a very great inspiration for all the work I’ve done. His work on the sequence is probably the number one most influential work on my personal thinking. So I’m also extraordinarily glad that someone like him exists.
Jim: Yeah, indeed. Now, let’s move on… We don’t have as much time as I would like on this, but when I was doing my research for our original podcast, and we didn’t get to talk about it at all in the first episode. Connor has written this very interesting, somewhat peculiar series of essays called Counting Consciousness. I think the original title of the first one was GPT-2 Counting Consciousness and the Curious Hacker. It covers all kinds of interesting things, and we’ll have a link to it on the episode page.
Jim: If you just want a very interesting set of reads, I encourage you to read it. I’ve now read it twice and I’m probably going to read it a third time because I’m sure I missed some interesting things here. Let’s go back to something we talked about briefly, and it’s one of the my interesting kind of thought experiments which will kind of get us into some of the ideas from your part one, which is deep fakes.
Jim: I remember being very worried about deep fakes about two years ago. In fact, I actually gave some money to a little start-up, not-for-profit whose main mission was to think about how to immunize humans against the dangers from deep fakes. And yet, so far as I know, there actually hasn’t been any serious damage from deep fakes. Somehow the biological blockchain as you called it or our human ability to collectively filter out shit has so far protected us from any really grievous harm for blockchain as far as I know.
Connor: I agree that I think that the amount of damage done by deep fakes so far has been relatively small, but I don’t think that it’s because the blockchain has been robust. I talked about this in the second part of the essay if I remember. By the way, I just like to flag for any readers, I apologize for those essays being strange. These were some of my first attempts at writing like long forms of essay.
Jim: Hey. Now, the fact that they’re strange is what makes them interesting. Don’t apologize. This is really interesting. You follow a first-class brain where it takes itself. So read the essays even if they are a little strange.
Connor: All right. Well, thank you, thank you. I do appreciate it. So the second essay I talk about the truly stupendous uncreativity of evil people. I know like Bruce Schneier also talks about the concept of ordinary paranoia versus the security mindset. He gives this funny example. So when he was a kid, he had an ant farm. He has a little card in there where you could say you can send an address like a letter with an address of this to this location and they’d send you a bunch of ants. That ordinary person would think, “Wow, cool. I can get some ants from the ant farm?” Someone with a security mindset will think, “Huh, I can send ants to anyone I want.”
Jim: Interesting, interesting.
Connor: It is kind of like a specific kind of style of thinking. Bruce Schneier is like one of the best writers on this topic. Eliezer has also written an essay about this. I forgot what it was called unfortunately. It’s kind of thinking that I talked about in the second essay of how if you are just like a little bit creative, like I am by far not the best security mindset genius hacker or anything like that, but just with a little bit of creativity, I could come up with some really, really dangerous possible attacks that I could pull off for like $100,000. It would ruin a politician’s career or something with high probability. And I could like do that for $100,000 in my bedroom. But somehow no one has done that. That’s kind of what I wrestle with one of the essays like why has no one done these things?
Jim: Yeah, that’s really interesting. I participated in it. I can tell you now I can’t tell you what the results were with an exercise for one of the three letter agencies where the hypothesis was you have a million dollars to do the maximum harm and scared the shit out of these people. I will say that my contribution scared them amongst the worst, but it’s very, very interesting that so seemingly the bad guys are not nearly as clever as you would think.
Connor: Yeah. And that’s actually a big reason of why I take AI risk so seriously because AI, even if it’s like not super human. Let’s say it’s just like as smart as a smart human, it can still be functionally perfectly sociopathic. You can still create an AI that was so perfectly, consistently lies, manipulates, controls. It can run extremely complicated networks of lies and sock puppets to such a degree that it will never slip up even once. And this is not something that we are prepared to deal with. Even now, from my experience is that one sociopath in your company can bring down the entire company.
Jim: I’ve seen it. I’ve seen it happen.
Connor: Yeah, me too.
Jim: Very bad. Sociopathy in our game B world is one of the big flags we have is that we have to get better at identifying sociopaths and keep things away from lovers of power. As someone who worked in corporate America, I have been known to say that my good faith estimate is 10% of C-level executives in major corporations in America are sociopaths, which is a scary fucking number considering that the number in the general population is on the order of 1%.
Jim: If you go to finance, it might be 30%. Not good. Let’s get back to the idea of the biological blockchain as a starting point for sort of where we were before we had other methods of building trust.
Connor: Yeah. So the idea, I’m not sure if I still think that’s the best name for it or not, but basically the idea is that in an ancestral environment, if you want it to build trust, say there is an idea and you can’t evaluate this idea. I’m not sure if it’s good or it’s bad. If you then see many people around you, that you trust from your tribe, your elders, your family, your friends and they all say this idea is really good. That’s pretty good evidence that it probably is true because anyone who tried to convince these people had to… It had to convince them that it takes effort.
Connor: If it’s a bad idea, you hope it takes more effort to convince lots of people. This had that saying that you can’t fool all the people all the time, but there’s some people you can fool all the time or whatever. So in a sense, you have this kind of trust mechanism is that seeing lots of people that you trust, endorsing an idea to alleviate your desire necessarily to check the idea yourself.
Connor: Or it might even be an idea that you yourself can’t validate. So it’s kind of like our truth-making mechanism like one of our basic ways of making truth. And the problem is, is that this mechanism evolved in an environment where there weren’t tech spots and deep fakes, and organized propaganda campaigns pushing anti-vax or whatever.
Jim: Or Facebook algorithms, optimized on sucking you in, right?
Connor: Exactly. I could definitely imagine that there could be a species. If we just froze technology right where it is now, and like a million years past, we might well evolve some kind of psychological mechanisms to very helpfully and productively deal with these kind of things. We might have very different cultural social and biological norms of how to deal with trust and epistemology. The fact is just we were not evolved for the situations we’re in. So we shouldn’t expect these systems to scale it.Honestly, it’s a surprise they got as far as they did.
Jim: Interesting. So you then you know talk about security mindsets and the fact that there’s all kinds of potentials for harm out there. But then the one thing I found most interesting and caused me to think a whole lot is what might we do as humans? We can’t evolve very fast biologically. And I will push back a little bit on the million years because say for instance, when newspapers and advertising first became a thing in the United States, it wasn’t until 1850s when they could start doing fancy graphics and in newspapers relatively inexpensively.
Jim: Famously, there was all kinds of literal snake oil salesmen selling all kinds of dubious drugs with claims that will cure every disease, et cetera. And yet, if someone were to put a snake oil add-on Facebook today, relatively few people would fall for, if you literally took the same text from you know 1855 and put it on those. That’s the amazing thing. And we do have flat-Earthers, et cetera and such. But we have developed kind of group and individual sense-making capabilities to filter out shit.
Jim: The other example you gave for a deep fake. Suppose you had a deep fake of Bill Clinton and Hillary Clinton telling racist jokes or something, people would just like, it just seems highly improbable, right? Again, there would be 10 or 15% would go, “Oh, yeah. Those evil motherfuckers.” But most people would say, “Yeah, common sense. It seems highly unlikely.” Even if they did it, they certainly wouldn’t make a video of themselves doing it.
Jim: So we do develop tools over time to make ourselves immune to the worst abuses. But you then suggest some stronger ways that we can replace the biological blockchain. So why don’t you riff on that for a bit.
Connor: Yeah. So I would not consider this to be like a full solution to anything, but I feel like it’s like such an obvious thing. It’s also not unique to me. I’m not the first one to caught off this idea obviously, but these ideas of using cryptography to replace some of our more informal methods by more formally powerful methods of verifying information. At some point, you have to trust people. Bruce Schneier has wonderful essays about this, about how there’s no such thing as trustless technology. That does not exist.
Connor: At some point you always have to trust the computer code or the algorithm or whatever. You have to trust the people who made the program. You have to trust the people who build your iPhone. There’s always trust. You can never verify anything. In a way, trust is the most powerful skill that humans developed. The fact that we do trust other people is what allows our civilization to exist. Remember, chimpanzees don’t trust each other usually. I mean, that’s actually nowadays considered not 100% true. But just to take the stereotypes.
Jim: What’s clear about chimpanzees is they absolutely do not trust any chimp that’s not from their band.
Connor: Yes. Exactly.
Jim: Chimps have a very complicated hierarchy of relationships within their band. But unlike humans, all they can do is kill anyone that’s not from their band. While humans have developed the superpower of long-range cooperation with people that they don’t even know.
Connor: Exactly. All the technology I’m currently using, I can’t verify what this technology is or how it works. I’ve never met Jim in person. He’s not part of my tribe. How did I know that I should trust this email that appeared in my inbox that I should click on this link and then go talk to this guy? How do I know? So trust us everywhere in our society and it is a fundamental social technology. It is a fundamental tool and social technology that we need for a complex society function.
Connor: Cryptography is not a solution to privacy. It doesn’t you know make it necessarily. It’s just a tool to allow us to do certain very powerful forms of privacy enhancing technologies. The obvious ones are like encrypting your chat messages or whatever. What I’m even more interested in and what I talked about in essay is this concept of a web of trust and using public key for cryptography to verify and sign messages.
Connor: So one of the things you can do with what’s called public key cryptography is that you can have a secret number, a key. You’re not allowed to show anyone else this key. And using this key, you can create a signature on messages that 100% guarantees you and only you wrote this message and the message was not tampered with.
Connor: So what I think what should happen is that this kind of technology should exist everywhere. Every tweet you send, every text message you send, every video you make, should be signed, time stamped, identity stamped. Of course, there could still be lawless places on the internet where this is not enforced or something. But I find it surprising that I can just go on the internet and they’ll just look at a tweet. I have no guarantee who this came from or what entity wrote this or and who endorses it.
Connor: So the way I think trust… So this kind of a formalization of the weak concept that the biological blockchain was trying to implement this idea that I could explicitly endorse people. I could say like, “Okay, I trust this government organization, but not this one.” I trust this newspaper, but not this one. I trust my best friend, but I don’t really trust my other friend because he’s kind of an idiot.
Connor: Then I can see who endorses what like who says this is true or who has comments and what. And I can verify that in an unfakeable way. So if I see thousands of people endorsing a claim, I can check, are these signed endorsements? Where does the signature come from? Who endorses these people? Is there a root node? For example, you might have a government that endorses a special citizen keys that can be used for voting or for comment giving.
Connor: For example, there was a grid in the FTC a few years ago that had a hearing, an open hearing about net neutrality. It turned out that the telecommunications companies hired grassroots as a service companies which is a real thing, which is legal by the way. Somehow this is legal that then created thousands and thousands of fake users and fake comments in order to push them towards one’s possible policy.
Connor: In the way you think about it, it’s kind of crazy is that you’re allowing people to anonymously basically “submit” things in the voice of American citizens without checking. This wasn’t necessarily when the biological blockchain was still in full force. Back in the day when town hall meeting was called, you had to physically appear in the town hall to physically tell people what your opinion was. That verified your identity. And online, that’s not necessarily the case. These keys and these endorsements by different you know root trust sources could be a way of allowing this authentication to happen online.
Jim: Yep. That’s very interesting. Just a little note, the grassroots as a service actually definitely exists. I actually hired one once when I was a CEO of network solutions and we were involved in the setting up of ICANN and there was a public notice thing with department of commerce. I think it was $1,5000. We were able to get a hundred endorsements of our position. Now, these weren’t bogus. These were from people who were small ISP operators mostly. But they would have not naturally formed this association to lobby on our behalf if it hadn’t been for this grassroots as a service. We had no idea how to do it.
Jim: It wasn’t entirely bogus, but it was, I would call it, AstroTurf basically anyway and it worked. Not that it really made much difference. But such things really do work. Now, it is interesting. Public key crypto is an amazing technology. However, it has one huge problem and I’m pretty familiar with this. At one point, I was president of Verisign’s digital certificate business and I said, “We were always thinking about interesting ways to monetize public key crypto.” And of course crypto coinage, bitcoin, Ethereum, etc. has brought this to a massive scale.
Jim: It all keeps coming down to the same goddamn problem, which is the critical fragility of the private key, which is that to use your private key, you have to have access to it. So if you want to sign something, you need your private key. But to move your private key into a place where you can sign it, you’ve just moved it into a place where someone can steal it because of our computer technology is so bad. In fact for my number one Ethereum wallet, I follow the thing that I keep my private key on paper. I don’t have it online anywhere, which of course makes it really difficult to actually do a transaction. I got to do it on one computer that’s offline and then I do it on and on. It’s really, really, really hard to actually use RSA style private keys securely.
Connor: Yeah, absolutely. My answer to that is that, yes, that’s completely the case. I am fully aware of how politically unviable this solution would be, like how complicated it would be to actually implement this system. Our institutions do not have the kind of executive capacity to be able of organizing something like this in any feasible timeframe, I think. But my argument is that I think that the difficulty does not come from private keys themselves. I think it’s more fundamental than that.
Connor: It’s just authentication is hard. It is hard. Authentication isn’t a fundamentally irreducibly hard problem. The same way that people complain about proof of work with bitcoin or whatever. Yeah, sure. I understand that there’s a huge waste of energy in whatever it has like all these bad boys, but it is an irreducibly hard problem. You can’t just have bitcoin without proof of work. You have like proof of stake, but it has other problems and other trade-offs. It is an irreducibly hard problem. Trust has to be hard because it was easy. It also makes it easy to break. There has to be one step somewhere that is hard. That’s just how it works.
Jim: I liked your essay, and that you laid out the fact that trust could be on a continuum. For instance, you could have an online platform that did not require a high trust certification. Yu have the 4chan equivalent which you mentioned that you were a 4channer at one point, right? While on the other hand Facebook might require a certification from government authority before they would accept your thing. I kind of like that, the opportunity for a pluralistic domain of trust that people could choose which ecosystems had what levels of trust. You could have webs of trust. So there’s a lot of things you can do with your architecture, which I did think was good though. Though I still am like, “Goddamn private key problem make this really hard to implement as a practical matter.”
Connor: Yeah, absolutely. In practice, you would have the less secure keys maybe and you keep those on your hard drive and those are linked to your shitpost Twitter account or something, like if you lose that [inaudible 01:00:59]. You have your secret government issued key to keep on paper somewhere secret and you only bring it out when you’re voting on something or doing something that’s requires very high levels of trust.
Connor: I’m a big privacy advocate. I think that if anything, I find it’s sad there’s less super anonymous places in the net. But I feel like there is benefits and downsides to having very anonymous places and there’s benefits of downsides to have very not anonymous places. I think these should exist side by side. It shouldn’t be everyone’s authenticated everywhere. Neither should it be everyone’s anonymous everywhere. I think it really depends on what the use case is.
Jim: I like that. I think that’s actually a very important principle. You made it very nicely in your essay. So again, I’ve read your essays. Let’s end up with the last and kind of the most interesting and probably provocative thought you had, which was in part four where you actually get down to, what are we talking about when you mention the phrase counting consciousness? This is a key question that we’re going to be confronting before long and we better start thinking about it. So what does count as a consciousness and why?
Connor: Exactly. So this is a question I’ve been thinking about for a long time and I still think about it all the time. My thoughts have evolved since that essay, but I think we don’t have time to get into that right now. But there is this real question is that we humans have a natural idea of a person, an identity, a singular like we have citizens and they’re discreet entities. We don’t have continuous citizens. We don’t have 0.7 people are in favor of this or something. At least outside of statistics. There’s nothing like that.
Connor: The thing is I think that’s not a fundamental property of the universe. It is an emergent phenomenon that happens to be the case. It happens to be a useful abstraction because humans tend to come in one human size chunks and they’re not easy to reproduce. But if we say had like a human brain scan, and I could just control C, control V that brain scan lots and lots of times. Do each of the copies get a vote on like… Are they citizens?
Connor: Well, what rights do these things have? What responsibilities do I have towards these entities? This is just one of like many, many problems. Once we start breaking down these comfortable assumptions that do hold for biological humans, but don’t necessarily hold once we really started building virtual entities and such. One of the things I talked about in that essay is so at some point, you have to count is that if you want to have a vote, you have to count.
Connor: There has to be an unambiguous way of counting how many people are voting, how will our votes be tallied? That’s just how voting works. Then on the more provocative side, I argue about maybe, not for all cases counting humans is the right thing to be counting. Maybe there are situations where you want virtual entities to be voting or to be part of your community or maybe… These things get really complicated very quickly. I don’t propose solutions or obvious things like that. It’s more like a food for thought that this will happen sooner or later.
Connor: Sooner or later, we will have entities sharing the planet with us that I think have a very real claim to moral patienthood, very real claim to say, “Hey, I’m intelligent. I have goals and desires. I think I should get some of the things these humans get.” At some point, such entities will exist. I think that is very likely whether those could be human uploads or AGI systems in the future. And that’s going to break a lot of the assumptions we use to run our society. I really do think we need to take these things very seriously and we’ll have to rethink a lot of the fundamental principles of how we run society.
Jim: Well, that’s great, and it’s absolutely true. I mean, if we say someday there’ll be AGIs, we have to come to some conclusion about where do they rank morally with humans? And the answer is not nearly as obvious as you might think. I’d recommend you read Connor’s essays. Well, thank you Connor for another wonderfully interesting conversation. I’m really glad to have you back on the Jim Rutt Show.
Connor: Yeah. It was a blast.
Production services and audio editing by Jared Janes Consulting, Music by Tom Muller at modernspacemusic.com.