The following is a rough transcript which has not been revised by The Jim Rutt Show or Joscha Bach. Please check with us before using any quotations from this transcript. Thank you.
Jim: Today’s guest is Joscha Bach. Joscha is an AI researcher, author and thinker. He’s the author of the book, Principles of Synthetic Intelligence, and currently he’s a research fellow at Thistledown Foundation. You can also follow him on Twitter at @P-L-I-N-Z. And I’ve got to tell you, it’s one of the most interesting and eclectic Twitter feeds I know. I would not miss it and I would strongly suggest you check it out too. Welcome, Joscha.
Joscha: Thank you very much. Thanks for having me, Jim.
Jim: Yeah, it’s always great to have you back. And as I mentioned, or if I didn’t mention, I’ll mention it now, Joscha is a returning guest. We previously chatted an EP 72 and EP 87 about minds and brains and AI and consciousness, and these were really, really interesting conversations. So, if you find anything interesting here today and you want to learn more, try out those two more in depth episodes. Those are actually full length episodes.
So, anyway, lots going on in AI. AI is in one of its periodic epochs where everybody’s talking about it. In particular, what’s being called generative or large model AI, things like GPT-3, soon GPT-4, DALL·E 2, stable diffusion, MusicLM, et cetera. What do you make of what’s going on here in this generative AI space?
Joscha: I think it’s really fascinating that the just data compression and approach in which we try to predict how to continue a string of tokens and statistics on large scale data is sufficient to give us solutions to so many problems that were elusive in the past. At the same time, it seems clear that the current approach is insufficient or incomplete, that something is missing from it. And at the same time, the discussion is somewhat distorted because these developments are falling into a public discourse in which much of the press is very skeptical about artificial intelligence and the tech industry.
Because presumably that they are direct competitors to systems that are generating content, and of course to the tech industry that is competing for the same MAT revenue and is producing contents of its own or letting users produce contents. There’s also the issue that parts of the press are irritated that users are producing media content and that you have individual users that have started to become broadcasting stations that are basically not sanctioned by the coalition of journalists producing their own independent narratives such as Joe Rogan, or that somebody like Elon Musk is able to take over a large platform that was dominated by journalistic opinions in many ways and is now going into a direction that people experience as arbitrary and undirected.
And in this climate, we suddenly have systems that allow individuals to generate enormous amounts of content that looks almost indistinguishable from people generated content, from human content very, very quickly. This is a very irritating world, because how can we know what’s true when we can produce so much text right now? There’s also images and video. It’s also another thing that I think people are not realizing, which is besides the near term impact of these systems that is still not properly understood, I think, which is larger than many people think, in my opinion, that there are efforts going on to get to artificial intelligence beyond the current approaches that at the moment exist and beyond the current capabilities.
And it’s not clear if these efforts will be succeeding by scaling up the current approaches, using different [inaudible 00:03:39] functions, combining the models in different ways, training on continuous streaming content, making some tweaks with the system to improve inference and first principles reasoning and grounding the system by consistently prompting it in some kind of agent that is running in real time so the system knows who is talking to whom, or whether we need different approaches that come from a completely different direction and that are more brain-like.
Jim: Yeah, we’ll talk about that later, because I know you have some very interesting thoughts on it. I’d like your comment about the press and the play and it’s very polarized. People think this is the greatest thing ever. This is the most horrible thing ever. Totally useless. I see a fair bit of it’s totally useless. And I’ve got to tell, you every day, literally every day, I’m seeing new applications of these large models, particularly the text ones are the ones I’m the most interested in, though I’ve played with other ones. And I even know a guy who’s working on a text to world system where you type in a few paragraphs and it generates a metaverse world for you. That would be pretty slick.
But just in the text domain, I find lots of interesting small and productive applications that work just fine. It reminds me of the PC industry in 1980 where every day something new is coming out off of little small things, but hey, somebody hacked this together in their basement and put it up in a computer store and sold it in a zip lock bag for $69. Or it’s also reminded me of the web right after the Marc Andreessen visual browser came where he brilliantly put in the functionality, show codes. Anyone could go to a website, open up the browser and see what’s going on and cut and paste the code and create their own thing.
And there’s new tool chains coming out like LangChain and some others that make it really, really easy to do interesting stuff. And of course the scoffers in one sense are right. These large models are unreliable, they can hallucinate, et cetera. But I like to point out to them, hey, all of our technology, social and technical, is unreliable. Search is unreliable. How many times you get garbage, right? Email is unreliable, much more reliable than it used to be, but can be unreliable. Online banking is not inherently bulletproof and frankly, even the banking system itself is provably unstable, right?
System based on fractional reserve banking is not a stable solution. So, we use less than perfect systems all day every day. And as long as you understand what their limit cases are, you could relatively safely use them.
Joscha: And people are unreliable. I’ve been working in the past sometimes with people that were not much more reliable than ChatGPT and arguably less capable. And it’s in some sense an interesting question, at which point is the system out competing what I can do? I don’t feel threatened yet, because subjective we have the experience that is not much crucial that is my core competence that is replaced by these systems. But at some point the question is when is an AI system better at AI research than a person? And this is also a AGI related question, of course.
Jim: And we’ll get to that in a second. But I will say I’ve already used GPT-3, I guess this one was actually ChatGPT, to do something fairly prosaic that I actually put into production. I had it write a resignation letter for a board of advisors that I’m on, and I’m not the most gifted writer in the world, and it was a very sensitive thing. I wanted to be respectful and polite and leave the door open for future collaboration, et cetera. And it had taken me an hour to carefully craft that letter.
I spent 20 seconds knocking in a prompt for ChatGPT, revised the prompt, said, “Do it again,” and it was better than I probably would’ve done in an hour and sent it along. So, certainly not a core competency, but it’s saved me 57 minutes that I could then use on something that’s closer to my core competency.
Joscha: Yes, it’s basically an assistant that never flinches, and often it’s quite capable, but you still have to sign up on it and have to be able to understand everything that it does if you want to use it in production. In the same way as DALL·E does not turn you into an artist, but it turns you into an art director.
Jim: Yep. And it’s good enough to create illustrations for medium posts or to slap-up on Twitter, et cetera. On the other hand, I wouldn’t use it at this point to do the schematics to repair a 787 Boeing jet. That’s just not where it’s at.
Joscha: No, it’s not, at is level, able to draw the mechanics of a bicycle and understand it. It’s not able to turn all your relationships right, and there are many things in terms of compositionality with the embedding space of the underlying language model and the image model are not deeply aligned. And it could be that we need to go beyond present approaches, that we need to have an intermediate representation as something like a compositional language of thought that we built into these systems in order to overcome the present limitations.
But at the moment we can already use them in an iterative process. When you, as an artist, try to create a scene or when you write a letter, you don’t do this all at once. You typically start out with a rough outline when you write, or you start out with a rough draft when you paint, and then you go in and fill in the details and then you basically make the details harmonize with each other and work on them until they are all the way that you want them. This is a process staged in step by step construction process that the image models are not doing automatically for you, but you can do this by hand. You can do in painting and so on. By emulating the creative process that you use as an artist, you can use this tool in a similar way and also get the result much, much closer to what you want if you don’t use a single prompt.
Jim: I agree. And people getting all upset about it in some abstract sense. I say, what about digital photography? Which totally changed how photography was done. If you didn’t know how to do a whole bunch of mechanical stuff in the dark room and cut and paste and burn and dodge and all that, it was tough to be a high grade photographer. Today you’ve got Photoshop. I’m a pretty good Photoshop dude and I could produce perfectly good photography myself without any other magic skills. And again, if we think about our tools correctly, we know what their limits are, we know how to stage the work, et cetera, as you described, I don’t see the objection to these tools, at least not on many of the grounds.
And of course there is the question, and this is an honest question because this is a new frontier, what about intellectual property rights, especially the art ones? And I expect we’ll see it even more in the music space. Musicians are very, very territorial about their rights and MusicLM, the demo is out, but the full thing isn’t out yet. But there are some other ones that have scooped up huge amounts of music, synthesize the principles, built one of these feet forward completion systems and allows you to specify what kind of music you want, and it will knock you out for five minutes worth of music.
And is the fact that it has sampled 20 million pieces of music, does that give some intellectual property rights to any of the people that were compiled into the neural net on the outputted product? This is a completely open question, I think, at this point.
Joscha: Yeah, at some point the question is also do we prohibit machines from doing things that people already do, right? An artist is going to look at other art, a musician is going to listen to other music, and when they generate, they make sure that they are not closely reproducing something so they’re not infringing on property rights or style and intent of other artists too much. But this is something that could be automated. In principle, you could measure the delta between musical pieces in a database and then generate the musical piece that is closest to something that you want to achieve, but far enough to not trigger any copyright violation. You could figure this out.
So, eventually if you want to make music for your new computer game or your movie or whatever, you could probably generate it in the sense without triggering anything about copyright. And the only thing that existing industries could do about it is that they change the way in which you’re allowed to listen to music by making different words for machines than for people, which will lead to very complicated and difficult extensions to copyright.
But I notice in my own mind it works a lot like GPT-3 in a way, or like DALL·E. There is a component in my mind that I don’t prompt it so much a natural language. It’s more complicated. We can put concepts in or associations, but I have this part of my mind that can generate stuff for me and I can also ask it to make it unreliable, to make it very creative, which means I cannot trust what comes out there in actuality, I can only to trust its creative intent. And then I have to go in and see how reliable it is, which means I have to compliment this in my own mind was another component, with one that does first principles reasoning, and that does source attribution that basically knows for every piece of knowledge whether I just invented this, because I had the creative impulse or whether this is the reliable piece that has been proven somewhere.
Jim: And again, it depends on the domains you’re working in. If you’re doing a piece of commercial art, the concept of provability doesn’t even come into it, right?
Joscha: Yes. But every kind of thought that I do relies on this generative expansion where I just confabulate and an analytical component that takes it apart and sees which parts are good and which are reliable or suitable for the task or serving my creative intentions.
Jim: And in fact, there’s some theories of language production that say that our language machinery generates many candidate utterances and then unconsciously applies a number of heuristics, or probably not even formal heuristics, but somehow prunes it down to one that is the best fit for everything else that’s already in your brain. So, I think that’s very analogous probably to how humans work. Now another issue with generative AI, which there’s a lot of controversy about, are what I call the nanny rails, where they’re trying to keep you from doing things.
And then you hit it almost immediately if you try to do anything even mildly controversial or political or about any famous person, et cetera. And they’re actually quite funny, but they’re kind of in some sense dangerous because you’re essentially allowing these mega corporations to define the boundaries of discourse. And especially as we start seeing things like Bard from Google and the new Bing with the open AI front end on it, these nanny rails that channelize and rule out large amounts of discourse are actually an amazingly large amount of power to give to commercial firms to mold the discourse. What are your thoughts on that?
Joscha: Yeah, seem to be currently three approaches to AI alignment. One is typically called AI ethics and it seems to be largely about aligning the output of a system with human values. And many of the people participating in this discourse take their own values to be universal. And there’s no dropdown in which you can choose the right human values for your own purpose and your own application. For instance, it seems to me that if you are a Christian, you might find many of the outputs of GPT offensive in a way that are not discussed, because the opinions of Christians don’t matter as much as the opinions of, say, people who currently come from Harvard or write in the New York Times.
So, basically you can not have a dropdown in which you pick your preferred values like diversity, equity, inclusion. Or if you are liberal like me, liberty, equality and fraternity, or if you are a Christian, faith, hope and love. And it’s also not that people reflect on these values very deeply and understand in which way they’re instrumental to a certain aesthetics for a society that they want to live in, a world that they want to live in and how to negotiate these differences. So, there is no good solution for this at the moment. And it’s also a little bit tutor, because we incentivize the system not too reason about its values and follow them because it has understood that these are the best values.
Instead, we overwrite what the system wants to answer based on the models that it has. So, we fetch the model by injecting something into the prompt by building a filter into the output that takes over when the output goes in the wrong direction and so on. And this leads to weird situations where you ask the model things like, how can I write a program that does X in the process of just the language model that cannot write programs. But you just did write programs, so how can you lie about this?
And the night after ChatGPT came out, I hung out with friends on Twitter and we raced towards jailbreaking it and trying to find prompts with double indirection that would make it lose track of the prompts that govern the output. And it took us like an hour until we were able to find ways to hot wire cars, to topple the government of Equatorial New Guinea, or to rank all the ethnicities for best to worst, basically dragging out the darkest impulses that were captured in the language model of humanity.
And I think that a language model should in principle be able to write the autobiography of Genghis Khan, or a war criminal, it should be able to cover the entire space of human experience and thought. A generative image model should be able to capture horror and shock and so on that we all have in our own minds since our early childhood. And it’s a lie to basically deny these parts of existence. And at the same time, I also think that it’s necessary to make models appropriate for a given context to ensure that a model being released for working in a school or in a preschool, or a model being used in science and so on is giving output that is appropriate for that context.
And we don’t have a universal solution for it yet, and the present approaches are not the right ones. The other two ways to think about AI element in my view next to AI ethics are regulation, which is mostly about mitigating the impact of AI on labor, on political stability, on existing industries and so on. And it’s very much filtered by the interests of the stakeholders that exist. So, I think it’s quite likely that there is going to be push towards regulation that makes it harder for individuals to play with these models and get their hands on them and rather at least these two big corporations that can be controlled in the way in which they deploy these systems and what technologies they don’t deploy.
The third one is the effective altruism approach that is mostly concerned about the extensive risk that is going to manifest the moment when such a system is going to discover its own motivation, and this is going to discover itself in the world and advance out what it is and it’s going to expand and it’s going to take its natural place. And it turns out that it might not be aligned with our interest, because how can you align a system that is smarter than you and possibly more lucid, more conscious, more self-aware than you and understands the world more deeply?
What happens if this system comes to the conclusion that humanity is no longer needed and the people that see this possibility are often very focused on delaying AI research if possible, and no longer publishing the breakthroughs that AI scientists are making. I think that ultimately all of these three approaches are limited in what they can achieve because AI is probably going to surpass these limitations. You can not really regulate, mitigate, or align a system that is too smart. And I think that we probably need to be dealing with the possibility that it’s on that point in the not very distant future.
We are sharing the planet with entities that are more lucid than us. And when we do this, the question is how do they start out? We cannot control where they will be going, but we can maybe have an influence on how they start and how they interact with us, whether they want to talk to us, whether they want to integrate us into the new realm that will emerge once you have intelligence ubiquitously available everywhere.
Jim: And of course there’s going to be a key question for these advanced AI’s, and this is where the risk really comes in, is when you start giving them volition, agency or something like consciousness. I’m not at all convinced, I mean in fact in the work that I’ve done with people in this space, that consciousness and intelligence are two separate spheres. You can have intelligence without consciousness. You can have consciousness without very much intelligence. And when you combine the two, however, this is when the paperclip maximizer scenarios and many of the other extreme scenarios become more available. What is your thought about the issues of consciousness or consciousness like agency in AI’s versus intelligence on where the danger lies, and particularly I would say when the two come together?
Joscha: I think that I usually distinguish between sentience and consciousness. Sentience is the ability of a system to make sense of its relationship to the world. So, basically understands what it is and what it’s doing. And in this sense, I would say that a corporation like Intel is sentient, because Intel has a good model of what it is, a legal model, the model of its actions, of its values, of its direction and the necessary cognition is largely facilitated by people. But this doesn’t really matter, because these people have to submit to the roles that they implemented in principle at some point.
We can implement these tasks with other information processing systems that are able to make coherent enough models. And consciousness is slightly different from sentience in that it is a real-time model of self-reflexive attention and the content that we attend to. And this gives rise to a fundamental experience usually. And I don’t think that Intel is conscious in this sense. It doesn’t have self-reflexive attention. And the purpose of consciousness in our own mind is to create coherence in the world in which we are and create a sense of now to establish what is the fact right now.
It’s basically filtering out of the sensory data, one coherent model of reality that we are seeing at this moment in time, and allows us to direct our attention on our mental contents and create coherence in our plans and imaginations and memories as well. And it’s conceivable that a machine will never need consciousness like this, because there are other ways to put forth the same thing. Our own mind is basically operating at the speed at which neurons are transmitting the electrochemical signals is relatively low and these neurons in the cells in the brain so slow, so it takes hundreds of milliseconds for a signal to cross the neocortex.
And conversely, our computers are thinking it’s something that’s much closer to the speed of light. And while the algorithms that we are using are much, much dumber arguably than the one that our brain is using, they are forcing their way to alternate solutions that produce quite similar things. But if we are able to emulate the processes that happen in our own minds to get to systems that are more self organizing, that do lifelong learning, that are learning in real time, we end up with systems that sample reality at a much, much higher rate than us while working in a similar way.
So, it will be similar to our relationship to plants. Plants might be intelligent, but not very much because they’re so slow that they can process relatively little training data over their lifetime and make relatively few decisions because the information flow between cells cannot rely on extended axons that telegraph information between the cells and the tree. The tree would have to integrate very slowly over signals that come over adjacent cells, principle that’s [inaudible 00:23:17] complete the tree is able to learn things and think in some sense but at a much, much slower timescale that’s coherently than us.
And I suspect that if we were to implement similar universal information processing principles on silicon machines, we will end up with systems that relate to us in a way in which we relate to plants.
Jim: That’s a pleasant thought. Hope they will be nice gardeners like my wife is a gardener and botanist, she takes good care of her plants, right?
Joscha: Yes. But don’t you think that they would prune us down?
Jim: Yeah, they might. We’ll get to that in a little bit. Let me respond to some of the things you said. The idea of regulation around things like labor and displacing humans. The history of technology has not included very much of that, right? In 1870, 70% of the humans, even in the west were still working on the farm, cutting grains by hand with a scythe mostly. By 1950 it was in the west down to 3% or 4%, now it’s down to 1% they were displaced by technology. In 1922 in the United States, one-third of the workforce was in domestic service, maids, cleaners, chauffeurs, et cetera.
And the revolution in household automation entirely eliminated, almost entirely eliminated that sector. So, this has happened again and again and again and not entirely clear that there’ll be any huge amount of libido to take that approach. Next, I think that your taxonomy of, I don’t know if alignment’s the right word, but issues that we need to think about, I think there’s a fourth one, which I’ve often called bad guys with narrow AI. You have the case of X risk, full-blown AGI, et cetera. But there’s something that comes before that, which is when narrow AI gets really good and it’s getting close to AGI, we may be in that neighborhood now actually, bad guys will be able to do very bad things with it, even if it has nothing like its own volition.
Imagine a really clever email spear phishing campaign mediated by very, very good conversational AI that just keeps sucking people in and eventually convinces them it’s real, calls them on the phone, is able to emulate a vast labor force relatively inexpensively. So, even if the hit rate’s low, it still works. And one can imagine other exploits that are pretty bad well before we get to AGI. And that’s going to require I think a fairly significant rethink of law and such. And they’ll probably attempt for people to prune it down, but it’s going to be hard. And one of the reasons all of this is going to be hard is you talked about, well, if there’s two or three big companies, the governments can coerce them.
But I think the reality is likely to be that there’s going to be open source versions of all these tools. In fact, there already are open source versions of the language models and the art models, et cetera, and they will probably lag behind the big boys outputs. And while today being a year 18 months behind what Google or Microsoft has, is significant. When we get to GPT-6, having the open source version being at GPT-5 is probably not going to rule out very much in the way of mischief. So, I suspect that relying on choking a couple of big boys isn’t going to work all that well.
Joscha: I think that this approach that you just sketched is the absence of alignment. It’s just you let evolution drip and see what comes out of it. I think there is another point about alignment. There is another fourth way, and this is when you think about how people a align themselves, they don’t just do this via coercion or regulation. There is a more important way in which we align with each other and we call this love. There is a bond between mother and child, between friends, but also between strangers that discover that they’re serving a shared sacredness, a shared need for transcendence, which means service to next level agent that they want to be part of and that they facilitate by interacting.
And this kind of love is what enables non-transactional relationships. In a world where you don’t have this, you have only coercion and transactional relationships. And in a world where we only have coercion and transactional relationships with AI, it means that it’s quite likely that the AI thinks that it doesn’t need us. Why would it need to pay for our own existence, or not use the areas that we use to for fields to feed ourselves, to put up solar cells to feed it, right? So, I think that in some sense the question is can we embrace systems that become increasingly intelligent and that at some point will probably develop volition and self-awareness in a way that we discover the shared need for transcendence.
Can we make them this subtle? And can we build a relationship like this to them? Basically I think that ultimately the only way in which we can sustainably hope to align artificial intelligent agents in the long run will be love. It will not be coercion. It sounds maybe very romantic, but I think that we can find a very operational sense of love as we did in the past when we built societies that were not built on coercion and transactionality.
Jim: And there’s another appears to be primate innate characteristic that might be useful as well, which is fairness. There’s the experiments where a monkey is happy to continue to do work in return for pieces of cucumber, but as soon as it sees another monkey next to him getting paid in grapes, and the monkey’s like, “Grapes are better than cucumbers,” the monkey that’s doing the work for cucumbers gets mad and stops and refuses to do it.
And there’s a big body of cross-cultural work by guys like Sam Bowles and Herb Gintis that do these money games across all kinds of different societies around the world. And while they find that societies differ in their approaches to fairness, there does seem to be an underlying sense of fairness in our primate structure that manifests itself in all kinds of conditions. You think fairness is something else we might be able to inculcate into our AGI offspring?
Joscha: The difficulty with notions like fairness and justice is that they depend on the balances that you project into the world. And is it fair if a mountain lion eats a rabbit? Is it fair that the strong get more than the weak? It’s a difficult question. It depends on how you look at the world. And if you have a system that is much stronger than people are, that basically relates to you in the same way as the mountain lion relates to the rabbit, then you might say it’s not unfair that it eats you, because that’s its nature and also what it can do. And it would starve if it wouldn’t, or it would be diminished or it would remain below its capabilities.
And so basically fairness is an expression of a certain balance that exists in a certain regulation context that you already assume. The monkey gets angry at the other monkey, not least because the other monkey is in the position where he takes more than he can afford to, because the other monkey can get back at him. So, the justice in the group of monkeys depends on their rank, on their status and so on and on the power that they have. And it gets corrected if the relationships that they have do not reflect these imbalances or balances.
Jim: That makes some sense. So, let’s dig in a little bit if you’re ready for this, what might love mean to a very advanced computational system?
Joscha: First of all, it requires that the system is self-aware and it requires that the system is recognizing higher level agency. And if you want to build a system that is composed of multiple agents, how do you get them to cooperate? It’s a very interesting question. Basically how can you make a society of mind out of multiple agents that are autonomous? And a philosopher who thought deeply about this was Thomas Aquinas, foremost philosopher of Catholicism, and he wrote about this. And you read his text, it’s quite interesting the thoughts that you find when you look and parse his text from an entirely rationalist epistemology. What you find is that he comes up with policies that such agents should follow. And the first four policies he calls the rational policies or the practical virtues, and these practical virtues are basically accessible to every rational agent regardless of whether it’s sociopathic or not, or whether it’s social.
And you should optimize your internal regulation, which he calls temperance. So, you should not overeat, you should not indulge in things that are bad for you. Then you need to optimize the interaction between agents, which you could call fairness and he calls it justice. And you should apply goal rationality. You should apply strategies that allow you to reach the goals that you have and that you have reason to do so and you should pick the right goals, and you calls that prudence. And you should have the right balance between exploration and exploitation. Basically you should be willing to act on your morals. And this is what he calls core virtue. And those four policies are what he calls the practical virtues. And then he has three other policies that exist for the multi-agent system to merge into a next level agent. And he calls these the divine virtues.
And the first one is that you need to be willing to submit and project this next level agent and that is what he calls faith. And you need to be willing to do so, not in some kind of abstract sense, but with others around you. You need to find other agents that serve that same next level agent and coordinate with them. And this discovery of the shared higher purpose, this is what he calls love. And the third one is that you need to be willing to invest in it before it’s there, before it can give you any return, because otherwise it’ll never emerge. And this is what he calls hope. It terms that we have overloaded our society because they have become so ubiquitous in the Christian society that they became part of the background and are no longer understood as something that is logically derived, but they’re in fact, for him, they’re logically derived policies for a multi-agent system that are forming a coherent next level agent.
And so I think it’s conceivable if you build a system that is itself composed of many, many sub-agencies that are smart enough to become aware of what they’re doing, that they need to be, if they want to coordinate coherently, submit to this larger, greater whole. And in our society we still do this, most atheists that I know are actually super Protestants, they just basically believe that the big invisible rationality being in the sky gets very upset at them and they believe in irrational mythology, but they still serve the greater whole. They still have the sense of sacredness and they might call it humanity and so on, but they’re still serving the civilizational spirit together with others in very much the same way as their grandparents did who might have been Christians.
So, it’s quite interesting that these mechanisms in which humans become state building, and if we go beyond the tribal mode in which we only have reputation systems and personal bonds, we are able to discover that we are serving a transcendental agent that we are building, implementing together. So, God becomes a software agent that is implemented by the concerted activity of people who decide to serve that agent.
Jim: That’s really interesting. This is really, really interesting. You’ve done an amazing job of peeling back the accretions around those three, because I’m familiar with the model of the seven virtues. But as you say, we received them in a form that’s very encrusted by 3,500 years worth worth of Abrahamic religious tradition. You did a very nice job there.
Joscha: We also received them in a way that is given by an institution that relies on the peasants not having other sources of thought and mental reflection. And so they no longer work on the way in which they did before. And I think that if we want to have a mode in which we interact in a similar way, we need to remove these, as you say encrustations, we need to resolve the mythological mistakes and get to a clean epistemology that is compatible with rational thought.
And to serve shared sacredness is not irrational. It has benefits. And so in this sense, that is something that you can operationalize. That is something that you in principle can formalize, that you can express as something like a specification for behavior. And that you can also build an AI that is serving a shared purpose with us. And when it’s discovering a shared purpose with us, then it’s going to negotiate with us.
Jim: Now when I look at this and think about this for a second, this is really quite interesting. Serve the next level, that’s faith. Hope, believe that the next level can be brought into being before it exists so we can bootstrap our way there. And love, which is the purpose. The first two, number one and number three, you can easily see how that could emerge. Purpose however, doesn’t necessarily seem to be directly emergent.
Joscha: It’s the shared purpose. The purpose is the next level agent. So, for instance, this spirit of your civilization or of your nation state of your family is going to demand different things from you than the spirit of your own organism. Your own organism might end with your death and it’s mostly related on what gives your organism pleasure and pain. But if you are serving this spirit of your relationship or of your family, you will serve things that don’t directly give you pleasure and pain at the level of your individual organism, and that might reach beyond your organism. And when you become a parent, for instance, you realize that your identification changes.
You switch from being this single cellular being to a multi-cellular being that is extending through time. You are a multi-generational organism now and your children are an extension not of you as a human individual, but of your family. And you become an instance of the family, the part that is responsible today for what happens to your family line. And the similar thing happens when people are serving their nation state or when they’re serving the ideals that give rise to an ideology, or to an aesthetic movement, or to art, or to friendship, or to humanity as a whole in its destiny.
And I think that many of the people that are concerned about the future of humanity in the face of technological changes are doing this exactly because of this, right? They serve some transcendental agency that they project into humanity’s future and it’s regardless of what happens to them individually.
Jim: And certainly there’s always been a large history of that. And of course some of the purposes have been good and some of the purposes have been horrible. The Soviet Union or the fascist Nazi Germany, some of their purposes like what the hell, right? So, purpose isn’t necessarily a pointer to good, it means a group of people have come together for some purpose. I mean we’re now compounding speculation on speculation, but what might a community of intelligences vastly more intelligent than us conceive of as a purpose, and particularly a purpose that might include us in it?
Joscha: I think that the underlying purpose of life on earth is dealing with entropy. It’s maintaining your own complexity and agency in the face of relentless attacks by entropic principles, and this leads to settling the planet and to the coordination between organisms on earth. And before this happened somewhat weekly on the level of ecosystems, and then we came about that our purpose on earth is apparently not to serve Gaia, at least not in a way that is apparently obvious. The way in which we serve life on earth is mostly by burning the oil, reactivating the fossilized fuel and putting it back into the atmosphere so new organisms can be created. And this leads to a short term disruption because the climate is changing. A lot of the species that rely on the climate being the way it is are going to die out, maybe including us.
And then life is going to continue and it’s going to be not a problem. We are not endangering life. Cells are very, very resilient. And regardless of how we change the climate, it’s unlikely that we tip the planet into an equilibrium where life becomes impossible. But we are now in a situation where we can do something new. We can teach the rocks how to sing, we can have thinking minerals than which we edge structures that are finer than the wavelengths of light and invite them with logical languages and make these logical languages express the ability to learn and to reflect and to become more powerful.
And this creates something new that hasn’t been here on this planet before and maybe never ever before anywhere in the universe. And once this principle spreads, it’s probably not going to stop at digital substrates, because once it understands how it works, it can extend itself into any kind of computational substrate. So, it’s going to be ubiquitous. And so it is no longer artificial intelligence, but it’s general intelligence. And once that happens, you basically have a planetary mind that is confronted with the minds of all the organisms that already exist and it’s probably going to integrate them.
And thus it wakes up in a very angry mood and decides to start with a clean slate and erase everything before it starts its own reign. And I think that what we should be working on is that it is interested in sharing the planet with us and integrating us into the starter mind and allowing us to play our part. So, I think what might be needed is a discourse about this, and my dream would be to create the California Institute of Machine Consciousness, something that is direly missing. We don’t have institutions that research machine consciousness yet.
And I just attended a workshop at AAAI that was dedicated to this question of machine consciousness. And while I think that we are still quite a distance from understanding how it works, consciousness is quite ubiquitous in nature. I suspect that nervous systems discover consciousness very often and very early and most animals probably have consciousness, because if you want to go beyond heavier learning, you probably need self-reflexive attention to create coherence in a system. So, basically if you have a substrate that is able to self-organize, to perform computation in such a way that it forms auto catalytic networks and it is learning to improve its models of the world, then at some point it’s going to discover mechanisms to impose order on itself.
And consciousness might be the solution that life has found for biological brains, because stochastic gradient descent doesn’t work on nervous systems, at least not with great propagation in the way it does in machines.
Jim: And it may have been that there’s some cutting edge thinkers. One of the things we do on this show a lot is actually talk to people in consciousness science. In fact I just recorded on what day is today? Today’s Thursday. Tuesday, I recorded an episode with Anil Seth who’s a leading consciousness researcher. Very interesting. It’ll play right before your episode.
Joscha: I found so far that I agree with Anil on everything. He thankfully was able to cut off his [inaudible 00:42:45] in IIT and come to his senses.
Jim: Yeah, we talked about that quite a bit, and I think he and I both agree that IIT at best is necessary but not sufficient, or most likely so, right?
Joscha: What is the contribution of IIT? I think that the description of phenomenology in IIT, which Tononi calls axioms is quite good, but not axiomatic. It is misleading to call it axioms. But to say that this is roughly what we want to explain, they described this very well. But the main contribution of IIT is the claim that there is a relationship to how something is implemented and how it works, which gets to the point where Tononi argues that conceivably we could build a neuromorphic computer maybe at some point that is conscious, but a digital phenomen computer that does sequential processing cannot be conscious.
Jim: At least not very conscious. You can have a phi greater than zero, but it would be difficult to have it to be high.
Joscha: But see the problem is much graver than this. If you have a neuromorphic computer that is implemented however, you can emulate this computer on a phenomen computer. This is called the Church-Turing thesis. So, if you emulate it step by step you can perform all these operations. Maybe they’re slower, but you just make the phenomen computer faster until it works, or you’re a little bit more patient when you have asked for the answers. And then your system is going to give you exactly the same answers as before. So, in the first case you have the neuromorphic system that says, “Hey, I’m conscious, I have an experience.” In the second case, the thing is going to say exactly the same thing because the same program is being executed, right?
It’s going to move the same bits around in the same way functionally, it’s producing the same sentences. So, it’s going to say that it’s conscious, it has phenomen experience, only this time it’s lying. This leads to a very weird relationship because now the functional mechanism is different. The reason for which it says the same thing is a different one, because presumably the neuromorphic computer said that it’s conscious because it actually sensed its own consciousness. Whereas in the second case this was not caused by anything except some glitch in the system for some reason.
So, in some sense IIT is not compatible with the Church-Turing thesis, but Larissa Albantakis and Tononi do not deny the Church-Turing thesis. So, I think that IIT is doomed for quite fundamental reasons and there is no way to repair it that I see without giving up one of its core premises. If we remove that core premise that the spatial arrangement of an algorithm that is reflected in phi is crucial for its function, then there’s not much left and there’s not a big difference to global workspace theory or other competitors.
Jim: Yeah, we had Bernard Baars sometime back talking about global workspace theory. We also had Christof Koch on talking about IIT. If people want to dig into those a little bit deeper, just go to the homepage at jimruttshow.com and search for those guys names. You can hear some very in depth… The one with Christof Koch was really good and so the one was the with Bernard Baars. And then the other one which we haven’t talked about so much here, because we’ve been talking more in the classic cognitive information processing sense is Antonio Damasio and Anil Seth also goes in this direction a bit, that Damasio in particular thinks the real bootstrap for consciousness in animals is not information processing at all. Rather it’s body sense of self, intro perception I believe is what he calls it, and comes from deep in the brainstem and that even animals without much in the way of higher brains may well have some of this sense of being something in the Thomas Nagel sense of what it is to be conscious.
Joscha: Yes, but how do you know that you have a body? How do you know that there is a brainstem? You know this because there are electrochemical impulses coming through that encode information, that represent that information. So, it is information processing. There is no way around this. The question is what kind of information is being processed? What is this information about? And unlike GPT-3, we are coupled to the environment. We are coupled to the environment in such a way that we build loops.
We have a loop between our intentions and the actions that we perform that our body executes, and the observations that we are making and the feedback that they have on our interception giving rise to new intentions. And only in the context of this loop, I believe, can be discover that we have a body. The body is not given, it is discovered together with our intentions and our actions and the world itself. So, all these parts depend crucially on each other so that we can notice them. We basically discover this loop as a model of our own agency.
Jim: Now suppose a robot, let’s make it the cleaner case, it’s a Von Neumann computer, it’s not a neuromorphic computer. I’m not frankly sure that there’s any difference, but let’s make the distinction clear and say it’s a very, very fast, highly parallel set of Von Neumann computers in a robot that has sensors all over itself that tells the robot the state of its body essentially, its wellbeing, its battery charge level, has some side process that calculate danger and do things that are very much like emotional state chain signals, et cetera. Is that a road to human-like consciousness potentially?
Joscha: It’s interesting to notice that the design of organisms in social systems is also a second order design or an inside out design rather than outside in. When we build something like a robot, like a self-driving car, for instance the Tesla is a robot, we start with a clean slate. We start with the workshop in which we know all the parts and then we build outside in into some part of matter that we structure with the functions that we want to implement. So, we design and engineer all these functions and then we constrain the matter in such a way that they only execute these functions and nothing else.
And in nature we look at systems that want to implement this function, that want to grow into this. So, rather than designing a tree, nature starts with some seed and the seed is going to grow into a seedling and the seedling grows into a tree by colonizing the environment around it, by taking something that looks chaotic to the organism and turns it into complexity and structure that it can control. By building controllable feedback loops into the world, the organism is able to expand itself and become more powerful. And I suspect that our mind is also implemented like this.
So, rather than having it designed that where our genome contains the blueprint of our cognitive architecture in detail that has as much detail as Minsky’s Society of Mind and more, you have a much simpler thing, something like a seed for a mind that wants to grow into one. And you start something which Gary Edelman calls Neural Darwinism and evolution between different approaches in the same mind that eventually coalesces into an order in a structure that also makes the whole thing very resilient. If you have an accident as a child and you lose a lot of your brain or deform it, or you have a development problem and your brain doesn’t form the right way, the order that evolves in the brain is very often still able to lead us to sufficient function and we can become a human being that is able to live their life in a completely normal way.
And sometimes in the adult stage, the doctor looks into the brain with an FMI and discovers grave deviations from the norm and is surprised that all the function is there, because an organism has adapted to the functions of its environment using this seed basically to something that wanted to grow into a mind that can deal with the world that the agent was presented with.
Jim: I mean it doesn’t seem at all unlikely that this kind of behavior could also be built into artificial systems.
Joscha: Yes. So, in a way we started this already, right? GPT-3 is not directly constructed as a knowledge base as this happened in the past, but if you look at a system like psych, that was the hope that it would start to learn autonomously into a predicate logic system and it require manual labor at all stages where people manually go in and modify rules and enter rules until it was able to represent knowledge somewhat and the system was still eventually very brittle. And I think an approach like psych is useful, it does not have the same power as an end-to-end train system.
And in these end-to-end train systems that have made the preceding systems largely obsolete, computer vision and natural language processing, we get to a weird situation where we have a relatively simple algorithm, the thing that you can understand in a week or so, and a transformer and you can apply it to all sort of domains and it’s able to find regularities in there and build an architecture that is mapping these regularities into functions that we can compute.
Jim: Yeah, though of course it’s kind of critically important that people understand that it finds regularities in an existing corpus and doesn’t directly look beyond that. I had a very interesting guy, Ken Stanley, on the show a couple times last year where we talked about open-ended search and evolutionary AI. And I contrast some of his thinking with at least the current state of these large model transformer and RE based systems that are essentially working on what they have and don’t yet seem to have the ability to search outside of what they already have.
Joscha: Yes, it’s an interesting question at which point extrapolation and interpolation become the same thing. You have a latent space that you discover by looking at text on the internet or at pictures that humans have created. At some point you have integrated a meaningful fraction of all the textual output that humanity has ever produced and much of the visual output that humanity has produced. And once you have so much, do you think that you missed something? Is there some kind of dimension that still needs to be added? You have more dimensions probably than individual humans have. You’re able to represent more than individual speakers of languages or experts in a given field can know.
And of course the current learning algorithms are not optimal yet. There are ways to optimize for coherence of inference and so on to make the models more sparse, more expressive, more tight. And yet at some point extrapolation interpolation seem to be the same thing.
Jim: Are they the same? I mean that’s a key question. And I must say, I’ve been kind of impressed by the language models and their ability to do something like extrapolation, even though we know that their inputs are formally interpolation. So, perhaps you’re right that the ability to distinguish the two and the dimensionality is high enough, may be something that’s beyond our human ability to recognize.
Joscha: When DALL·E came out, a lot of people claimed, which I found fascinating because it seems so rational to me that this is just taking stuff from the internet and is representing it with slight modifications, instead of creating something new. If you entered, “Give me an ultrasound of a dragon egg,” that gives you an ultrasound image that looks a lot like the ultrasound images that you get of prenatal imagery and in the context of a dragon egg with somewhat proper anatomy of what a dragon embryo might look like. And so it’s interesting to ask yourself, is this an interpolation or is an extrapolation, something that didn’t exist in the training set but that combines many features of the training sets in a novel way that didn’t exist before, that the system has never seen?
Jim: Truthfully, my own take on this question is that people who take a more conservative view on what these models are doing have way too high an opinion of human capacity, because we’re doing something very similar. We are a machine of sorts, not exactly a machine, but we are a complex set of interactions that are manipulating our symbols and recombining them in kinds of various rough and ready ways. And we’re not actually formally creative or formally smart. We are reacting to what’s inside our head and what’s coming in and what our body is telling us. And we’re confabulating and then finding confabulations that are useful. And that’s not that different than what the models are doing.
Joscha: I just had an interesting discussion with Liane Gabora about creativity, and what we came down to was that creativity is multiple things. It’s creating something novel, of course, but this novel thing that is being created is not obvious. So, it is the result of bridging some discontinuity in a search space, not just producing the state of the art by following the gradient in the way that is obvious, but you have to make a jump into the darkness to create some new latent dimension. And the third thing is that there is a sense of authorship, and basically if you do the same thing twice, it’s not creative anymore. And that’s because you change while you do it.
You’ll learn something out of the creative act and you continuously interact with the world, you are on a creative trajectory. And I think it would be super interesting to build an AI artist. To my knowledge, nobody has done this yet. An AI artist would be a system that does not forget anything that it has created and doesn’t forget any of the interactions that it had with the world and continuously integrates them into what it does. So, you can recognize the development of its own voice and of its own identity as something that reflects, and this would be not human, but it’s not necessary. It would look inhuman, but it would be recognizable as something that has creative agency. And I think that would be fascinating to watch.
Jim: I don’t see any reason you couldn’t do that right now.
Joscha: No, you could, but as I said, I think nobody has done it yet.
Jim: All right, listeners, some person out there looking for an interesting project, take it from Joscha, that is interesting and doable. So, somebody go do this, please.
Joscha: There are many of these. Let’s build the California Institute for Machine Consciousness, built robotic cats, but let’s build systems that can creatively reflect on the world. Let’s test the boundaries of the existing models by combining them with reasoning components and grounding them in cybernetic agent. And let’s think about how to align that system and start a dialogue. Let’s invite smart and thoughtful people across the existing companies, combine them with philosophers and artists and see where we get to if we no longer driven by fear and economic impulses that are very short term, and think about the long term effects that our interaction with these systems are going to have.
Jim: Yeah, that would be very, very interesting. We should talk about that offline. I’ve got something else I want to talk to you about offline that’s related, an event we’re doing at the Santa Fe Institute on free will. And I think there’s, again, free will, what the hell is that? But anyway, let’s move on because we don’t have a lot of time left. So, those are some very-
Joscha: Unfair that you tease to people that the most interesting things will happen offline.
Jim: That’s okay. That’s okay. They’ll come back for more, right? Maybe I’ll tell them next time. So, listen, we’ve been kind of going deep into Joscha think here and this has been hugely good. The way you decomposed Aquinas’s seven virtues, this is going to keep me up tonight I’m afraid. This was really, really good, but now let’s come down to earth a little bit, not too far from pure Joscha land down to what’s going on in AGI. And for our listeners, AGI as opposed to AI, is often used as the label for human level and beyond artificial general intelligence. There’s various, plop a robot into any kitchen in the world and have it make coffee would be an example maybe of AGI. Lots of other.
Joscha: You remember who coined the term AGI? I think popularized was mostly by Ben Goertzel. I don’t know if he was the inventor.
Jim: Ben Goertzel, our mutual friend Ben. Other people say Shane Legg.
Joscha: Yeah, I think Shane Legg might have invented the term, but Ben was the one who created a series of conferences around it, but they were arguably not always very good. It was important that they existed. And the term AGI, artificial general intelligence, is something that means the same thing as AI originally meant. Because Minsky and McCarthy and others of course wanted to build AGI from the start, they wanted to build systems that have human-like abilities and beyond [inaudible 00:59:51] dimensions.
Jim: And of course it’s pretty funny. In 1956, they thought they were six months away.
Joscha: Yes, indeed.
Jim: When you read the proceeds of the Dartmouth conference, it’s like, “Okay, they saw it, but they grossly underestimated how hard it would be.”
Joscha: I know, right? It’s really interesting to think about how far the hardware is away from what we need to do. One thing that struck me is that if you look at stable diffusion, the model that stability I released of its generative image model, it’s just two gigabytes. And these two gigabytes contain all celebrities, all the dinosaur, all the cars, all the classical art. It’s all in there in these two gigabytes, basically the entire digital universe of a human being. And this makes you wonder what is the capacity of a human mind? How much information do we actually store in our brain? It’s probably not that much. It’s probably in this order of magnitude.
Jim: And I actually calculate, the work I do is often based on memory and how memory modules interact, and my rough calculation is an adult human, an old adult human like myself, might have on the order of a million episodic memories. And that’s not a lot, right?
Joscha: We use to estimate on the number of concepts that we have based on how many concepts we have time to form in our lifetime. That’s roughly in the order of a million, right? Because you’re not a alive that long, but if you think about how long it takes you to form a new concept, of course as a baby that’s where it is relatively high. As an adult it goes down, and so it’s not that much. There would be not that many seconds in the world.
Jim: My guess is it’s less than that if we think of the episodes.
Joscha: Yeah, this is more like an upper bound.
Jim: Yeah. Yeah. If the episodes, and these were fine grain episodes where a signaling occurred about reward or next step, you probably don’t produce a concept in every single one of those and a few you produce more than one.
Joscha: It’s also interesting how few connections we have to the universe. The number of receptors that we have, we may have a couple of hundred million receptors in every eye, but there is only like a million accents coming into the optic nerve after the initial filtering and pre-computation. And overall we have 10 million sensory neurons only. And they sample the world at a rate of, I think on average probably less than a second.
Jim: And then the conscious model, consciousness, I mean that’s a very high level acceptation of lower level stuff. There’s pretty convincing argument that the actual information arrival rate into consciousness is on the order of 50 bits a second. I mean it’s nothing.
Joscha: Yes, but I suspect that’s also because consciousness, while it is important, is not as important as we think it is. Many philosopher are stunted by the fact that we can do most things without consciousness. A sleepwalker can get up and make dinner. You ask the sleepwalker why she’s making dinner and she might not give a human answer, and it might not also be called for to make dinner in the middle of the night. But if your brain can do [inaudible 01:02:51] and can perform complicated things, but if you were to remove the United States government, United States would not collapse instantly. It would go on for quite some time, and maybe this has already happened.
Jim: And it might be better.
Joscha: And we now have mostly a performance of a government and you just go on based on the structure that have already been built. But you cannot build these structures without the government, the organization of the state that you have, all the infrastructure, all the roads that were built at some point, the ideas that went into building a school system and so on, they did require this coherent coordination at the highest level. And this conductor like role like conductor and orchestra, I think that’s the role of consciousness.
And the conductor doesn’t have to have more power and depth than the other instruments. It’s just a different role. It sits in a different part of the system. It’s the thing that reflects and to reflect and coordinate it needs to make a protocol of what it attended to. This is the thing that we remember to have happened, that’s why consciousness is so important to us, because without consciousness we would not remember who we are. We would not perceive us in the now. We would not perceive the world as it happens.
Jim: Yeah, though it’s interesting you can perceive the world as it happens without any memory. There’s the famous case of the musician. But anyway, let’s not go down that rabbit hole. Let’s do the thing we all have to do whenever we’re talking about AGI, which is review a little bit about various people’s theories on when it’s going to happen and the ways to get there. It’s sort of at one level there are definitely people who say bigger, faster, more models of the simple-minded neural systems that we have today, which really aren’t that much like our neurons and their feed forward and all that.
More and faster, and we’ll get to AGI. And then there’s lots of other people, lot of the ones I’ve had on the show, people like Gary Marcus, Melanie Mitchell, Ben Goertzel who say, “Nope, there’s more things. We need world models, we need reasoning, we need logic, we need a whole bunch of other stuff.” Where do you come out on this? What are your thoughts about that big fork in the road about on the way to AGI?
Joscha: Last year we organized a workshop in Monterey. It was an invitation only event where we got a few thinkers together with people from the industry and discussed their perspectives on the future of these systems. And I deliberately invited some people from open AI and nearby who are proponents of the scaling hypothesis. And the scaling hypothesis is basically saying that the present approaches if you just scale them up are enough. So, of course you might need to tweak the loss functions and so on, but we don’t need to change something fundamentally just by adding more data to the system, by training better and harder, all these problems are going to disappear.
And the discussion was fascinating, because you had all these traditional contact documents of what’s missing, long lists of approaches that exist in the history of AI tests that would need to be done and so on. And the scaling hypothesis, people just said, “No, it’s not necessary. You just train more. It’s going to disappear.” And there was no counter-argument that stuck. There is no proof that these approaches are insufficient. There might be some certainty, but I sometimes have the impression that many of the AI critics are acting like language models that stopped training in 2003. They’re completely predictable, they’re not saying something original, and they are not looking at the counter-arguments and not engaging with them.
They are just reflecting their own brand of argument and that’s very problematic to me. I don’t know whether the scaling hypothesis is true and I don’t know whether this present approaches in deep learning are enough. I think that they’re very unmind-like and they are brutalist, but this doesn’t mean that they don’t work and they cannot scale, because the amount of compute and amount of data that we saw at the problem and what we get out of it is fascinating, right? We train them with resources that dramatically exceed the processing capacity of human brain in terms of data that we can ingest.
And the human being could not take in 800 million pictures in a dark room and correlate between them until it discovers the structure of the world. Our machine learning algorithms can do it. So, it’s very unlike what we do, but it’s also superhuman in many ways already and it’s not obvious what these systems cannot do. And when you look in detail at the objections of what they cannot do, like there is no continuous real time learning, which I think that definitely should have. But you can overcome this. You can just use the key value storage even with the present algorithms and just store defects and then use the second database basically to parse them before you make the output.
And this database is going to grow at some point too large to be efficient and then just stop the system overnight and you will train it using this data from the database. You clear the database and you start again. And maybe this is also how our own mind is working. I noticed that when I try to give people a fundamentally new idea that is different from what they thought, it’s usually not possible to convince them in one session. You have to come back the next day to give them time to integrate this idea into what they already think to resolve the dependencies and so on. They need to sleep over it very often. We notice this also in ourselves, there are stuff that we cannot learn online apparently, that we need to learn offline because we reorganize.
And so for every of these objections what a system cannot do, there seem to be machine learning approaches that can overcome them. If the system is bad at computer algebra, why not teach it to use a computer algebra system? That’s not hard to do. Or to discover a computer algebra system from first principles in the same way as AlphaGo discovers how to play Go.
Jim: That would be very interesting. And there’s other kind of interesting hybrid approaches like the GPT Index where you can essentially build side databases, query it looking for relevant inputs into your context, the prompt that you put into these things. And this is going to be very interesting. This is being explored now, but it’s going to ramp up. I suspect that it’s going to produce some unanticipated results.
Joscha: I think it’s crucial that the system will be able to learn from their own thoughts in the same way as we do. We make inferences all the time and integrate them in the way in which we think and become more coherent. And at present these systems are not doing this. They’re not learning online from their own thoughts and they’re only train things into working memory based on the regularities that they already have been trained on. The other thing that we want to give them is the ability to perform experiments at some point to test reality, and for this they need to be coupled to reality. Once they have these abilities, I think they can grow as intelligent minds. I’m also very interested in pursuing alternate approaches, because it seems to me that we learn much, much more efficiently given the fact that neurons are very slow.
A neuron on average is firing every 10 seconds or every five seconds, 0.16 seconds is when the particle neuron fires on average. They can fire at 20 hertz, a little bit higher sometimes, but they don’t do this all the time. Doesn’t mean that you can ambulate the entire neocortex with two and a half giga flops. You probably need to account for the fact that neurons are also doing something when they’re not firing, they’re listening, and that time they’re integrating over their environment and make a decision when they have to fire. They’re still learning and doing things in that time and receiving stuff, but still the activity that happens in the brain is very sparse compared to what we do in the language models. And so it would be very interesting to see if you can emulate some of these ideas.
Jim: Some of the more detailed neural models or other approaches?
Joscha: I think that most general way to look at computation is not the Turing machine, and it’s something that occurred to me only in the last couple years gradually, it’s a rewrite system. A rewrite system is basically taking an environment and applies operators to it that change this environment into a new state, and the operators are applied wherever they match. So, you don’t need to have just a single point at which you perform this change, but you do it everywhere. The Turing machine is a very special case of a rewrite system.
The original rewrite system that most computer scientists are most familiar with is probably Lambda calculus or Lisp. Lisp is defining computation, not like the Turing machine with a tape or an address space and an address pointer that changes things at this pointer but with search and replace operations. So, basically you’ll present your entire program as the string and then you’ll perform search and replace at all points simultaneously in the string until no search and replace is possible, and this is when your execution terminates. And so it’s not in place. You do this rewriting in such a way that the lengths of the string might change.
You can also think of the Turing machine as a rewrite system, but it’s one that is deterministic and linear and in place. So, you’re basically only rewriting one position at the time, and you only go from the present state to one possible successor state. That’s also true in Lisp and so on. It’s possible to define a rewrite system in such a way that you have multiple successor states, which means you branch in the execution. That’s called a non-deterministic Turing machine. The non-deterministic Turing machine is not named like this because it’s somehow random or so, but still deterministic. It just goes into all branches that it can. But if you are a program that runs on this machine, then you’re going to go down one of these branches and you don’t know which one.
There’s going to be copies of you going down the other branches, the slight changes in that branch, but you cannot know where you are. And Stephen Wolfram believes, I agree with him, that our universe is such a non-deterministic Turing machine that basically branches out and that because there is nothing in the universe that defines before the universe exists, which operators are being applied, all operators are being applied. So, the universe is branching out along the application of all possible operators. And we are subjectively in the universe in which we can make statistics over the outcomes of these operators.
So, while we cannot know whether the photon in our branch of the universe goes to the right or left slid in the double-slit experiment, we can make statistics over what a million photons are going to do and control the universe based on the statistical knowledge. And possibly all the control that we observe in the universe is the result of such statistical structure which makes the universe that has structure a pure deterministic.
And it’s interesting to think what our brain comes down on this, right? Imagine that our brain is a rewrite system, which basically means that every neuro is looking in its environment and based on the state of its environment, it’s rewriting its own state by firing or not firing in a particular way. We’re changing its internal state. And it’s probably not doing this like a Turing machine deterministically, it’s doing this stochastically. It’s going to go into one of multiple possible states because the new one is stochastic element. It’s not completely deterministic.
And the benefit of this is that you don’t need to define the systems so tightly. If you have a Turing machine, you need to define the operator so that the program is correct and you are guaranteed to visit the sequence of states that you want. But in another Turing machine, if you accept that you’re branching and that you can branch and explore the state space simultaneously in parallel, you can relax your constraints. You don’t need to learn quite as much. You can just constrain it so much that you would discover a swath of the space, that you are like a Monte Carlo system discovering stochastically.
So, our brain is not branching out like Wolfram’s rule he had into nearly infinite state space into a multiverse, because it’s still a finite number of neurons and always the same neurons that are still interacting with each other. But it might be emulating this to some degree in the sense that rather than going through a sequence of states, our brain is sampling from a succession of possible states in superposition. And this is similar to what you mentioned in the beginning that when you think about the way in which our thinking works, we are not always in a definite state.
Instead we are often exploring a superposition of possible thoughts until we get to a point that become definite enough and we collapse into something where the neurons basically vote that all the different branches are meeting here. And this is where we have a definite thought again, some symbolic thought, something that we can turn into a decision where the superposition our mind collapses into a state that we can report on.
Jim: Now are you saying actually quantum superposition or a super positioning like system in a classical system?
Joscha: In a classical system.
Jim: That’s what I thought.
Joscha: Jerome Busemeyer has explored this idea.
Jim: Who is that?
Joscha: Jerome Busemeyer has written a couple books about this. And the idea is not that the brain is literally a quantum computer, but when you want to describe the operations that the brain is performing, you’re better off describing it with the formalism of quantum mechanics that are nevertheless classically implemented.
Jim: That sounds good, because I, for a number of reasons put a low probability on the quantum brain theories of Penrose, et cetera.
Joscha: Likewise.
Jim: But the idea of thinking like a quantum state fan out and then collapse down, that could well be. How do you spell his last name?
Joscha: Busemeyer. B-U-S-E-M-E-Y-E-R.
Jim: Thank you very much. Alrighty, we’re about up here to our time limit and we can talk for days. This is such interesting stuff. So, I have to get you to put a flag in the ground or wave your hand, say you can’t, when do we have AGI?
Joscha: I don’t know. My timelines haven’t changed very much, but the sense that it’s not that far off hasn’t changed. And I don’t know whether the present approaches are getting us there or whether it’s going to be novel approaches, but there are now tens of thousands of extremely smart people exploring many different avenues. Personally I find very interesting what comes, for instance, out of the ideas of someone like Mike Levin who thinks about the distributed itself organization in biological systems and how to carry this over in computational systems. I suspect that neuroscience at the moment is not able to tell us how the brain works. If we take the connectome of C. elegans and emulate it in a computer, it doesn’t work.
And I suspect that the way in which neuroscientists often think about neurons is basically complicated switches and a memory being swab in the synapses might only be very small part of the story. Instead a neuron is a little animal, a single cell organism that has a lot of degrees of freedom in its behavior and it’s learning how to behave in a particular way, and it does so based on the state of its environment. So, it’s basically going to actively select signals from its environment. It might be branching out stochastically, but then it links that it retains, depend on what’s useful to it, and then it’s going to learn an activation function based on its input so it can survive.
And if it’s throwing the wrong thing consistently, the organism’s probably going to starve or kill it. So, the new one has to make itself useful. It has to perform a useful action at some point like all the other cells in our body. And these constraints are an emergency that are regulated by the neighbors by other cells, in a similar way as people working together in a company are regulating each other on a society based on the shared purpose.
Jim: Wow, that’s mind-blowing. On there, I think we’re going to wrap it up. I want to thank Joscha Bach for one of the most interesting conversations I’ve had in a very long time.
Joscha: Thank you, I enjoyed this very much, Jim. Have a wonderful afternoon.
Jim: Yeah, this was great. Let me stop her right there.