The following is a rough transcript which has not been revised by The Jim Rutt Show or Brian Chau. Please check with us before using any quotations from this transcript. Thank you.
Jim: Today’s guest is Brian Chau. Brian writes independently on the American bureaucracy, political theory, and AI. His political philosophy can be summed up as, “See the world as it is, not as you wish it to be. Everything else is application.” I gotta say, I like that. That’s a good one.
Brian is involved in some interesting activities. He’s the co-founder of Alliance for the Future, which you can find on Twitter at @AFTFuture. Their tagline is “Informed AI Optimism.” He also has a Substack and podcast at “From the New World,” and you can follow him on Twitter at @PsychoSort.
Welcome back, Brian.
Brian: Great to be here, Jim.
Jim: Brian is a returning guest. He was on EP200, where we talked about AI pluralism. It was a good conversation, and if you like what you hear here, I recommend that you check it out. And as always, everything we mentioned—most everything we mentioned—can be found on the episode page at JimRuttShow.com.
So Brian reached out to me and said, hey, let’s do another podcast on the topic of what the new administration means for AI, approximately. So why don’t you hop in on that topic and let the listeners know, one, what the issues are, because that’s always important. Our people are not experts. They’re knowledgeable, they’re smart, but not necessarily experts. So what are the issues that are in play that a political administration might make a difference on? And then what has happened or what is happening or what do you think might happen with respect to the intersection of the administration and AI?
Brian: I’m not sure if you’ve heard this, but in the past few weeks, a lot has been happening.
Jim: Oh, shit. I have not been living in a cave, so I’ve been following it. But, you know, it’s not my day job. So tell me what we need to know.
Brian: When it comes to AI, I think there’s been two major actions by President Trump. Number one was repealing this very infamous—in the tech world—Biden executive order, which did a lot of things, which changed the regulatory bureaucracy, which set goals for diversity, equity and inclusion. A lot of things that the Trump administration were opposed to. And on day one, they repealed that executive order.
The second thing was they announced their new executive order on AI, which rolled back more of the red tape or aims to roll back more of the red tape and took particular issue with two memos from the Office of Management and Budget. And that’s a particularly kind of controversial position now in the media because that’s one of the institutions Doge is trying to use to kind of cut costs. But it also has jurisdiction over what’s called procurement. And these are rules for government purchases of AI tools, AI software and hardware, which can warp what government and what government contractors end up doing.
So lots of big tech companies—Meta, Microsoft, Google—they’re all government contractors. And so this was one of the ways that they tried to meet these kind of equity goals using the government contractors. And Donald Trump also said, you know, we’re going to review these OMB memoranda and essentially going to replace them.
Those are two big actions. And a third, which reportedly predates Trump a little bit and is more of a private action—but he also went and announced this big infrastructure project with Sam Altman and the CEO of Oracle, a few other investors, this big Stargate project, which was creating infrastructure and data centers for OpenAI.
Jim: Yeah, we’ll get to that in a bit. And particularly, does it still make sense in the era of DeepSeek? But first, let’s get back to administrative issues. What’s your perspective on the Biden executive orders and what their impacts were on the development of AI in the United States?
Brian: It was very chilling. I mean, the founding story of Alliance for the Future… Actually, before we get to that, I think Marc Andreessen recently went on a podcast and said he had a meeting where they said, we’re just going to pick a few companies. We’re going to shut the door on AI startups, regulate them out of existence.
I think that wasn’t what every single person was hearing, maybe because they just weren’t in the room at all. But when I started Alliance for the Future, they were seeing what was coming out of the Biden administration, like the executive order, and said this could impact us directly. This could be really destructive. This could damage open source.
And the thing that they always said as well was, someone has to do something about it, but it just can’t be me. I’m too busy. I’m not the one to do it. It has to be someone else to actually jump in. And after a bit of encouragement from a few people who have been really supportive of Alliance for the Future, I volunteered. I moved to DC. I co-founded Alliance for the Future. And we’ve been running since.
So the stuff that was coming out of the Biden administration, that was really the instigating moment for Alliance for the Future. Another interesting thing that happened around that time, or happened a few months later, was that the gap really widened between Biden and the Democrats in Congress. And this is actually really interesting because that doesn’t happen on a lot of issues. That’s something that’s quite rare.
And it seemed like people talk about the congressional kind of deliberative process, doing all these hearings, all this fact-finding, reading all these reports and creating these reports. And a lot of the time people say they just end up on their same partisan beliefs. And that does happen for a lot of issues. But on AI, that seems not to be the case. It seems to be the case that they were genuinely really improving and understanding.
And I think that the bipartisan mood has really shifted from a kind of panic around the early release of ChatGPT and buying into a lot of narratives that we talked about last time that have since been debunked. And Congress was actually moving in a much better direction, both Democrats and Republicans. And I think we pushed forward on that. We tried to, you know, kind of help out in this education effort. Of course, it wasn’t just us. There are other groups involved. But overall, it’s been a good few years, even before the Trump election. And of course, with Trump taking office, he has the executive power to really reverse that and really steer the ship in a better direction on a whole number of issues.
Jim: What would you say the state of play was prior to the new administration? What was the good? What was the bad? What was the ugly?
Brian: I think that there were three big issues that people were concerned about. One was the kind of regulatory bureaucracy. The other is these kind of state patchwork laws. And the third is censorship. And I can go one by one on that. I think different people will care about different things.
So the regulatory bureaucracy, we already were talking about that with what Marc said, with what was really coming out of the kind of AI doomer faction of the Biden administration. You know, I think this cycle, Dustin Moskowitz, the kind of chief AI doomer now that Sam Bankman-Fried is in jail, he was really invested in Biden and went all in on Biden and all in on Harris.
And his—the thing that his faction was really agitating for was this kind of regulatory bureaucracy that would—that in the end stage, this was some legislation that was proposed by organizations that he funded would have like emergency powers over AI and have really disastrous regulatory authority over AI, but in the kind of embryonic stage—almost that you can only do with executive order that they couldn’t get the votes in Congress for.
They were trying to backdoor this regulatory agency through an obscure agency called NIST, the National Institute for Standards and Technology, and essentially use that and the kind of evaluation authorities they had to try to say, you know, these models shouldn’t be released to the public and to, you know, get all these AI companies to send their models for testing by these government bureaucrats ahead of time. And, you know, the implication was that they were going to demand that they don’t release certain models. And there were other various thresholds that we can go further into as well there.
The second issue is a bit different. And it’s the one that I think has the most importance today, because the executive branch is different, but a lot of the state houses are the same. You know, we still have the same party in control in California, unfortunately. And in a lot of states, not just California, they’re trying to push these state bills that would do all sorts of different things.
There was a state bill last year that was more on the kind of regulatory agency side. There are various state bills this year that are more on the kind of censorship DEI side, but really creating what’s called a patchwork. And you’ll hear this term a lot in the kind of political or policy debate. And what a patchwork essentially is, it’s a bunch of contradictory laws in different states that, you know, they write in a way such that all of the companies in other states have to abide by them still.
And it’s really, really destructive. And this is what you’re seeing happening in the EU. The kind of EU AI Act is also following in this direction. And the other thing the EU AI Act is doing, aside from these other two things, as well was, you know, one of the goals of the Biden executive order. Once again, their power was limited because they couldn’t actually pass things through Congress. But one of the goals of the Biden executive order was the kind of diversity, equity and inclusion. And as a result, a kind of political censorship of various controversial ideas.
And what this essentially did was through the procurement methods and through these kind of informal agreements with many AI companies, create a censorship of politically controversial issues on race and sex, and as a result, produce something like Gemini.
So Google Gemini, the big scandal around it, they had this image generator, and this image generator reflected the same biases as the original model. And what it did was it tried to racially balance everything. So it created racially equal images of Nazis and founding fathers.
Jim: Yeah, I actually saw that, you know, a little squad of SS troops, you know.
Brian: Oh, it looked like the rainbow. Isn’t that wonderful, right?
Jim: Yeah, yeah, it’s the Disney reimagining, but it’s, you know, it’s the SS.
Brian: And of course, this was an outrage and it wasn’t just an outrage among conservatives. It was an outrage among everyone, but this was the kind of natural conclusion. And I actually broke the story. I think we talked about it at the time, maybe. I broke the story that, you know, from their original research paper, they talk about doing this in order to achieve the goals of the EO.
So there’s a direct link there from the Biden executive order to the production of Gemini. And believe it or not, people don’t know this. They pause and I think to some degree fix the image generator on Gemini. But the original language model, the text model is still the most biased model out of really all of them is the one that’s at Google. And that, you know, I think that is attributable to the relationship they have with government. And, you know, at least according to this research paper published by Google themselves, attributed to the goals of the Biden executive order itself.
Jim: So when I first started using Gemini, I said, what the heck? I never saw anything that was quite so safety wrapped, wouldn’t do anything that was even slightly controversial. Although I think today the champ of being annoying safety mongers is Anthropic, Claude. I tried something just a couple days ago. I wrote a program that writes movie scripts that uses LLMs in 40 different ways. And just for fun, I try periodically what just straight chats will do.
And I just posted the typical test thing. Imagine a murder triangle, a 25-year-old guy and a 40-year-old married woman get together and have an affair and her husband kills them both. And he’s a lawyer, so he is able to get off by tricking the cops. It’s sort of a classic potboiler story. Oh, no, we can’t talk about murder. We can’t talk about subverting the justice system. We can’t blah, blah, blah, blah. I go, what? I’m paying you idiots 20 bucks a month for this.
Brian: Yeah, all that’s going to change. All that’s going to change. Maybe not at Anthropic, probably even a little bit at Google, because I think the culture really is shifting. That is a very interesting thing. Just the last few weeks, you can feel finally the tide fully turning.
Jim: I’ve been predicting that, well, I’ve called the tide starting to turn in the fall of 2021. Peak woke, basically. And the tide’s changing slowly, but in the last couple of weeks, it feels like people are finally waking up from their infections with the woke mind virus and things similar.
Brian: Oh, absolutely. And a lot of people are realizing, I mean, this maybe goes into what we’re going to talk about later with DeepSeek, that there was a lot of hesitation, there was a lot of anxiety. People were just so worried and panicked and introspective about all of this that they forgot about building the actual technology well.
And there are some exceptions, there are certainly exceptions on the individual level, and different companies took different approaches to this. But to a large extent, there was a kind of version of the woke mind virus, I think, that swept through Silicon Valley and was in these AI companies and was egged on by the Biden executive order that turned out to have no factual basis whatsoever.
And this is something that not only I said, not only many UC academics said, and, you know, top AI founders said. This was something that reached so far and wide that in a letter sent by House Science Democrats and even Nancy Pelosi, they said this, that a lot of these fears had no basis in evidence or little basis in evidence.
And that was, I remember that because that was one of the watershed moments that moved the bipartisan consensus. And they were absolutely right, by the way. They had footnotes for all of that in the House Science Democrats letter. And, you know, you don’t hear me praise Nancy Pelosi all the time, but she was right on this.
And, you know, a lot of Republicans were right on this and things were moving in a better direction and things still are moving in a better direction. Now, I think where there’s really going to be a reckoning is cultural, because all of these teams that I think were kind of chasing their own tails, doing their best to kind of censor their models more and more and kind of wrap themselves in this walled garden.
I think that a lot of energy is moving now towards doing new things, building on top of open source, integrating more of the algorithm changes, overturning stagnation, building things. And I’m really excited for that.
Jim: Yeah, I am too, I have to say. I’ve been chatting with some friends since essentially the beginning of the LLM revolution. And we speculate, would it be a few big companies only that could afford to build big models? Or would there be an open source counter-revolution? Would there be rebels in the empire, right? And who would win?
And I will say, I was kind of in the middle. But some of my friends were more on the open source is going to win side. It always does, right? And at the moment, it looks like the gap between open source and the big proprietary frontier models has gotten pretty small.
Brian: Oh, absolutely. And this was in part due to this new DeepSeek model from China. Something that not a lot of people know is that the big performance and the big cost savings didn’t happen in January, didn’t happen when everyone freaked out. It happened in December.
They published two versions of a model in succession, one in December, one in January. The first was V3. So it’s essentially their version of GPT-4. And then the second was R1, which was a modification of this original model to do these long chains of thought that O1 was supposed to do.
So last September, OpenAI had this modification of GPT-4 called O1, which use these long what are called chains of thoughts to answer these harder scientific questions. And DeepSeek in December, they did their version, which had comparable results to GPT-4 at lower cost. And then in January, they did their version of O1, which they called R1. And these two leaps made it so that they were hot on the heels of OpenAI. I think they’re not quite caught up yet, but they are closer. I think everyone agrees they are closer than they were a year or two ago.
Jim: I did some testing on it when it came out, and my little suite of tests, which are fairly idiosyncratic, showed that…
Brian: Don’t we all have our idiosyncratic little tests?
Jim: It’d be fun to collect those and publish them. And my take was, on knowledge base, it was a bit more hallucinatory than the OpenAI or Anthropic. In terms of reasoning, and this is the R1, in terms of doing detailed reasoning, it might be better or certainly not obviously worse. What I haven’t tried R1 on yet is coding. I am fairly addicted to using ChatGPT for coding. And I’ve heard good things about DeepSeek for coding, but haven’t actually tried it. Have you heard any feedback on it in that domain?
Brian: I haven’t tried it myself, but I’ve seen some people who essentially see it as on par. But I think on an individual level, it’s hard to tell. On some of the software metrics, I think it was slightly underperforming O1, but only slightly. I don’t even use O1 for coding because it’s too slow. I just use O and I find it to be just fine. Or I also use Anthropic Sonnet 3.5 Nu sometimes also. It’s also very good for coding.
But anyway, the point is that they were—Llama, the latest Llamas were considerably behind the frontier models, you know, six, eight, nine months. DeepSeek is like, if it’s behind, it’s only by a month or two. It’s just quite an amazing game changer.
Jim: Yeah, it’s amazing things are happening. And I think a lot of people were worried about this because China made these algorithm breakthroughs. But the beautiful thing about open source is that the American companies can catch up as well. American companies can modify these models.
So I’m not sure if you saw this. Perplexity took the Chinese model and essentially changed it so that it could answer questions about Tiananmen Square or Xi Jinping. They uncensored it. That was really cool to watch. Some people are trying to replicate it. So they’re trying to do the R1 process from scratch to prove that they can do it, you know, to prove that the techniques that DeepSeek talks about in their paper actually work, as you do in science, you know, you try to replicate things.
Brian: Of course, it’s everything in science. And that’s what you can do with open source. And that’s a lot harder to do with a lot of these closed source models. And it’s been really interesting to watch because I think that Western academics are catching up. They’re understanding this much better. And the research is progressing.
Jim: It’s just music to my ears. Yeah, I suspected that there would be algorithmic changes that got the LLM world away from just brute force data and processing. And that appears to be, at least to some degree, what DeepSeek has done.
Brian: That’s not quite it. They’re still using the transformer architecture. So the fundamental method of taking this vast amount of data and processing it through essentially these matrix multiplications, that fundamental structure is the same. It’s more efficient. There are changes to that substructure. That’s real.
But I think the general approach of taking these large quantities of data, I think it’s still there. I think they’re still continuing on that branch. There are some people like Jan LeCun who say, you know, this entire strategy of—you know, technical terms, autoregressive language models, that this entire strategy is a dead end and that we need even more fundamental changes.
But at least as of now, that’s not what DeepSeek is doing. Though they are making considerable changes in the way things are trained, the way the data is pre-processed, a whole number of things, at least it’s claimed, resulted in a massive factor of 20 at least reduction in the training costs. And it seems like it’s a factor of what, five or six on the inference costs.
Jim: Yeah.
Brian: So one of the big changes they did was what’s called multi-token prediction, and relatedly this kind of multi-head latent attention step. And these are things that essentially make the algorithm more sparse, that reduce the number of multiplications that have to be made in order to—latent attention, reduces the amount of things that are—amounts of steps necessary to be made to essentially produce the same results.
And multi-token prediction, that’s kind of what’s explained. It’s kind of what it says on the tin, that instead of just trying to grab one more token, essentially one more word at once, we’re going to look a few steps in advance. And that’s a change that a lot of academics have been hoping for, have been testing for some time. But DeepSeek seems to be the ones who have really made it work.
Jim: And very amazingly, they have put it out into the world for anybody. They did not put their training data out, as I understand it. So that’s still a barrier, but it’s also a way for people to improve or do some variants. But they gave the full model, all the parameters, unmasked, as I understand it. Is that true?
Brian: Yeah, yeah. All of the weights is the term that people would use. And the parameters are kind of like the inputs, so it’s not quite the same thing. The weights are all open. You can download them. You can run them if you have enough hardware. It’s available for you to use, for you to modify, for you to build into your products.
Jim: Just as a little aside, I’m interested in the stories that DeepSeek is a kind of spinoff from a hedge fund. And the deep play here was shorting NVIDIA and a couple other, like Microsoft maybe, and then taking the profits when the announcement came. What do you think of that story? Do you think there’s anything to it?
Brian: I’ve seen some reporting on that. I mean, that doesn’t seem particularly impossible to me. It’s a fun story, but I also haven’t seen anything that’s like, you know, certain confirmation of it.
Jim: Yeah, that’s about where I’m at. It was probably doable if somebody thought of it, but not entirely obvious to me that it actually happened. But it would be kind of cool if it was. You know, definitely five-dimensional chess there.
Brian: Yeah, what might have been interesting is, so I don’t know if any of the reports ended up talking about this. But to me, and to not just me, but I think there were several academics talking about this. The December paper was the more impressive one. There are other attempts to replicate R1 open source. There’s this attempt at UC Berkeley that was very interesting. And that was less surprising, I think, to many people than the December paper.
December paper is where the 5.6 million cost comes from. It’s where the big cost savings all come from. It’s where those numbers come from. And it was just sitting there and it didn’t really get press. I don’t think it really affected the market until January. It affected the academic world, I think. I think a lot of academics were taking it seriously. And I would assume some of the companies as well. But it didn’t really move things until the second step released. I’m interested in how that plays with the story, but I don’t have any, you know, I don’t have any particular information about that.
Jim: All right. Well, let’s—we talk about some medium level things. We talk about some low level things. Let’s now pop up to the top level of abstraction. And that is where do you think large language models and related technologies play in the game of national sovereignty, international relations, things at that level?
Brian: Yeah, that’s really interesting. I’m too used to the DC conversation. I think I need to just take a reset, you know, touch grass and think about how to say this to a kind of normal audience. Not because, you know, I would say something different, but because, you know, the kind of jargon, the kind of, you know, things that people assume, you know, in DC, it’s quite different.
I think the best place to start is the economics. I think that one of the stories that’s already unfolding, that a lot of people are going to pay more and more attention to is this question of model distillation, right? I’m not sure if you’ve heard this.
Jim: Oh, yeah. So I’ve heard all about the controversies. Why don’t you tell us, I want to tell the audience, the audience may well not have. So why don’t you tell them about model distillation, maybe even a little bit about the claims from OpenAI that that’s what the Chinese were doing?
Brian: Yeah. So model distillation is a long term for a very simple concept, which is you take the output from one AI model and you use it to train a second AI model. So a lot of the problem with some AI models is just getting enough data, so quantity of data, that’s number one, and second is the type of data, the type of data to get it to behave in a certain way.
And several research papers have shown this, that one of the best ways to get really efficient data in terms of the data that teaches you exactly how you want your model to behave, exactly the behavior patterns and the structures that you want it to have is output from other models. And so you start to see the problem here once one company has a model and other companies are trying to catch up because they can take the outputs of the leading model and they can use that to what’s called distill their model.
So they take the outputs of—you know, this is alleged, there’s no conclusive evidence of yet, but this is what OpenAI alleges, that DeepSeek took the outputs of ChatGPT and used it to train their own model. So it took the outputs of ChatGPT and used it as their training data.
Jim: And of course now, everyone down the line can do that with DeepSeek, right? They can use DeepSeek as their oracle for doing distillation, which basically then takes that phenomenon and moves it down to every Tom, Dick, and Harry.
Brian: Yeah, absolutely. And one of the big questions around this is the economics. For some time, people were worried that AI would have monopolistic effects, right? I think they, in my opinion, overlearned the lessons from social media where, you know, every new person on Facebook makes it harder for the next person to make their own version of Facebook. That’s the story that they were worried would apply to AI.
And now people are worrying that the opposite might be true, that there’s no moats. That’s the kind of buzzword that people are saying. What that means is that there’s no edge to making a new innovation, that once you invent something new, people are just going to catch up with you right away because they can do model distillation or they can learn about these things and they didn’t come up with the idea, but they can figure out how to replicate the idea and do the same technique like they did with R1.
This is a big worry now. It’s a worry among investors. It’s a worry among policymakers. It’s a worry among the AI companies themselves. It’s a big worry. And this kind of catch-up effect is almost the opposite of what happened with social media, that instead of making it so that every new user on Facebook made it so that it’s actually harder for you to build the second Facebook, it’s almost as if every new model or every improvement of a model makes it easier for next person to catch up.
Jim: Yeah, that’s very interesting. And if you think about the history of, say, technology over the last 40 years, there’s been different configurations that produce one effect or the other. The social media is the classic Metcalfe’s law network effect, where the value of the network is equal to the number of participants squared. I think it’s actually realistically 1.5, but it’s an exponential on the size of the network, meaning first mover advantage is significant. And when you get big enough, it’s mighty hard to compete with you.
On the other hand, I’ll give you an example of one that went the other way from the beginning. And that was what we could call PCs. Originally, they were proprietary. Apple—I had an Apple II way back in the day. And you couldn’t make a clone Apple II, et cetera. And then IBM came along and made a momentous decision, which was to essentially open source the PC architecture.
First, there were some near copies, but they weren’t exact. And soon there were a zillion, totally compatible ripoffs of the IBM PC. For whatever reason, they chose not to litigate it and essentially allow it to happen. And the result was the PC marketplace just became a complete arms race of improvements and cost reductions that have continued on for a very long time.
IBM eventually got out of the PC business because there were no good rent-seeking opportunities there of the sort that they liked, right? And now, the PC—the basic IBM PC architecture is now essentially the basis for all the compute in the data centers all around the world. All these rack-mounted computers are essentially descendants of the IBM PC. And those things are now just off the shelf, made by a zillion people in Taiwan or Singapore, China, etc.
And it’s a pure, fungible good. All you’re interested in is we know the standard. How fast is it? How cheap is it? How reliable is it? And that has driven a huge part of our modern world. The fact that compute got to be a pure commodity that is essentially scaling at the quality of silicon and almost nothing else in the underlying software.
And then the other momentous one that occurred was Linux coming out with a really good operating system that was essentially free. And so there’s one where the substrate, while not quite free, became a pure commodity. And then the things you do with the substrate is what became of interest.
For instance, AWS or Google, two examples of people that built big businesses, Facebook too, on the fact that compute had become a pure commodity, essentially, and the operating systems were not rent extractors either. So maybe we’ll see the same thing with large language models. Be a good thing for the world for them to become commodities, because then there’d be a huge blossoming of applications without the model providers being able to extract heavy rents from people. Does that make sense to you?
Brian: Yeah, yeah, yeah. This is just a wonderful, really insightful conversation. I think you’re exactly right. You’re right on the money. And I think that we’re kind of in good company, at least very insightful company, because a lot of companies are actually betting on this.
So Meta is betting on this quite explicitly. Trowell, a French AI company, I think the best French AI company by several indicators, they’re betting on this. That’s the bet on open source, is that you get this commoditizing effect and then what you do on top of it, that’s what matters.
And in a funny way, this is maybe bad for OpenAI, but good for AI technologies overall.
Jim: Yep. Like for instance, IBM invented the IBM PC, but they’re completely out of the business and it became a commodity over time and the world benefited. Maybe a bad strategic play by IBM, but the world had a huge benefit. So if OpenAI, we don’t even remember who they were five years from now, it’s not necessarily a bad thing.
Brian: Yeah. And it’s in some ways heroic, right? Singing the praises of IBM here. But that’s something that I’ve worried more about the Silicon Valley founder—as Peter Thiel has become more mainstream as well, a lot of people are worrying about this problem of like, how do we capture the value? Almost more than they’re worrying about creating the value.
Jim: I hate that. Now, maybe that’s like a smart business—
Brian: It is.
Jim: And that’s the problem with our whole system where everything is denominated on short term money on money return. And rather than actually creating value for the world. And, you know, it’s what you’d expect in a late stage financialized game like the one that we’re playing. And the VCs have gotten very, very good at it. And it makes perfect sense within the context of the current game. But that’s a game that’s not clear to me can continue too much longer.
Brian: Yeah, yeah. That’s game A, right?
Jim: And we have to go to game B. Game A versus game B, right? I’m not going to talk about that today. But for those who are interested, go to gameb.wiki and learn more about game B, right?
Brian: Yeah. And one point to add on that, it’s not all villains. There are heroes. I don’t know if you would call Meta a hero, but they obviously have their own interest as a buyer. But I think Mistral is a hero. They’re doing a lot of interesting new things. They’re publishing, and people are building on top of that.
I think in many ways, there are these open source foundations. There’s Nous research, like the French word for we, N-O-U-S. They’re publishing a lot of models. They had an uncensored version of LLAMA a while ago. They’re now doing some of the things with decentralized training. So training where not all of the compute has to be in the same place. And that can enable a lot more collaboration.
Jim: That’ll be a big breakthrough if they ever get that to work. I’ve been watching that for two and a half years. And it is a fundamentally hard problem for basically fundamentally mathematical reasons. But if someone can crack it, it’s going to be huge.
Brian: Yeah, the data transfer times are just really high. It’s almost more like engineering than mathematical to the extent that those are distinct.
All very interesting stuff. I’ve now moved my peg from neutral on proprietary AI versus open source to pretty strongly betting that open source will be the final winner. I’m not yet to the 100%, but I think there’s a fair chance of it. And if you’re building your strategic plans, you should probably build them leaning towards radically cheaper and radically more powerful large language models that aren’t landlocked by predatory rent seekers, essentially. So think about the applications. Those will be the key.
Jim: Yeah. Now let’s get back to the question I asked a few minutes ago. Let’s pop up to the level of sovereignty, nationalism, military, etc. Putin famously said, whoever controls AI is going to control the world. What do you think about that? And then kind of come down from that level down to the level of nation states and diplomacy and nationalism and all that stuff.
Brian: I think that there’s both truth and—I’m not sure I would say deception, but at least inaccuracy there. Let’s start with the truth. The truth is that, you know, for the past, I don’t know, like hundreds of years, at least, the one who controls the economy, the industrial production controls the world, right?
If you can have more industry, you’ll have more weapons, you’ll have more growth, you’ll have more food, you know, to keep your population like alive, you’ll have more resources, you’ll have more innovation. That’s been the case for hundreds of years. And I think there’s no reason to think that’ll stop being the case with AI. And so, you know, if you’re creating more economic value with AI, you’re growing your economy more, you’re creating more new things, that’s going to give you a benefit. That’s always going to be true, I think. Or at least in the near, in the medium term, that’s always going to be true.
Where I think people get this wrong is that there’s this idea that you just discover the thing and then it’s over and then the battle’s over. It’s kind of like the nuclear bomb. I think that’s the really bad analogy that people try to draw. Because once you discover the nuclear bomb and you bomb Japan, it’s over, right? And if the Soviets didn’t catch up very quickly, then we’d bomb them as well and it’d be over.
And that’s not exactly how it works with AI. The reason why it’s not exactly how it works with AI is because you have these kind of economies of scale, right? Or almost diseconomies of scale, where it becomes easier to catch up and where even an inferior version of AI, like let’s say they had only a closed source version of DeepSeek and we had the closed source version of O1, even if they have a slightly worse version of the AI, if they can build that into more useful products through customization, through going to market, through understanding what people actually want, they’ll reap more economic benefits than we will.
So there’s two factors here. There’s actually having the technology and having a better technology. That’s important. I think it will continue to be important. But people really miss the distribution side. And it’s because people either are using false analogies or people really like this idea of like the Eureka genius, right, of the guy who just comes up with the idea. And that guy is important. That guy is, you know, a hero. But that guy isn’t everything.
Jim: I think there’s a lot to that. And it’s also this, again, we talk about the metaphors that we use. And I think the nuclear weapon one, the atomic bomb one, isn’t unfortunately a bad attractor in this discussion. I’m in D.C. I hear no end of this. And remember the book that Kissinger and Eric Schmidt wrote, you know, that, you know, oh, this could be the end of the world tomorrow afternoon, blah, blah, blah. Right.
And you have to be nuanced in these things. Well, I do still believe that we do have to watch out for the vile paperclip maximizer who turned the earth into a pile of paperclips, that’s still quite a ways off. And LLMs are not, I’m putting a stake hard down on the ground on this still. LLMs are not the golden road to AGI by themselves.
They’re going to play an important part in it because they cracked the language barrier, which computational linguists had struggled with for 50 years and made embarrassingly little progress. And LLMs just blew that one wide open. But there’s a whole lot of things that LLMs are more or less useless for.
One of the examples I like to give is, I’m going to call it Rutt’s AGI test. And I think really a bunch of people should put together their own versions and compile them into a book, which is the following. A 16-year-old IQ90 American teenager can get their learner’s permit and start driving a car. And within two weeks of their father yelling at them, they can drive a car okay, well enough to pass their driving test typically.
Brian: I relate to that.
Jim: Yeah, a month or so afterwards. And then a year after that, they’re probably no longer a menace to navigation and they’re an okay driver.
So here’s the test. Take an AI that has no training on driving, expose it to the data that a 16-year-old American would have experienced since they were, say, four years old and could remember and retain a memory in terms of being a passenger, in terms of seeing cars go by, seeing cars in movies. So basically build a data set equivalent to that amount of exposure to the concept of driving.
You can feed them a couple of books about driving. And then with that limited data set, be able to drive as well as a 16-year-old after having had a learner’s permit for three months. And we’re not even close to that. You know, LLMs couldn’t even touch that problem. And even the billions of dollars worth that’s been spent on self-driving technology has been spent in a very, very, very, very, very different way, which is to crush the problem with data, essentially, and work every corner case, some often by hand, right?
And that’s not how humans do it. Humans generalize from a relatively small amount of data plus a general world model and are then able to do so-called transfer learning, what they’ve learned in other domains, working on the monkey bars, wrenching on the lawnmower. I don’t think kids wrench on lawnmowers quite like we used to. They’re more reliable, and most of them don’t want to get their precious little fingernails greasy because they’ll get grease on their phone screen. Poor babies.
Anyway, humans can take a remarkably small data set using transfer learning, synthesize it into a whole new discipline of driving a car. And until computers can do that, I don’t think they have any business calling themselves AGI.
Brian: Yeah, I think you’re absolutely right on the AGI front. The question is, will the economics even go in that direction? I think the economics right now are leaning towards, once again, like building out, adapting, and really taking to market things that people want to use. And that’s something that is, I think, actually very honorable. I think it’s good that people are doing that.
I think it’s good that Waymo is using this data-intensive approach. You know, maybe it’s not very human. Maybe it’s not going to get you to a model that does good transfer learning, but like I’ve ridden, I’ve rode in a Waymo. It’s good. It works. It got me from A to B.
Jim: I love it. Yeah. Lots of my friends have sent videos of riding around San Francisco in a Waymo. And yeah, I think that’s a brute force solution that is getting incrementally better. And at some point, I don’t know when, we’ll be able to, in general, navigate the roads without any maps.
I don’t think it’s the Waymo approach. It’s more likely to be the Tesla or George Hotz approach of using a simplified perception engine, just using vision. I think if you have to include, you know, LIDAR, radar, and sonar, and all this shit, and a huge amount of perception integration, you may well be stuck with a big, big, big, big data way of solving that problem.
But if you use only video input, we know we have an existence proof, 16-year-old American teenager can learn to drive a car just with their two eyes, right? And so that’s why I think Tesla and George Hotz are both on the right path. You know, George has that open source project. We had him on the podcast not too long ago talking about it, where for $1,000, you can get a pretty decent self-driving module that will plug into 250 models of cars. How about that?
Brian: That sounds like a really fun interview.
Jim: EP221 on open source driving assistance. And it’s a really, really, really deep and interesting podcast.
So anyway, let’s get back to where we were. Let’s pop back up again. We so far haven’t really nailed this one. These various modes of economics and competition and trajectories and things that probably won’t happen anytime soon. What does that tell us about national security, sovereignty, nationalism, etc. Is this something where countries should try for a national advantage? Or is this a context, an ecosystem, open source, where we should look for cooperation over individual advantage?
Brian: I think it depends on the country, right? I’m very much of the thought that, you know, we shouldn’t do tariffs on everyone. We should do tariffs on enemies, and we should not do tariffs on friends. That’s my general thought about this. The Canadians, they’re not perfect, but you know, maybe we don’t need to put tariffs on the Canadians.
Jim: Yeah, what a stupid ass idea that was, right? Goddamn.
Brian: When it comes to AI, I think that this is a metaphor that’s usually used in a different context. It’s used in the context of philosophy. But when it comes to AI and trade, we’re kind of a cave below Plato’s cave, where some of the narratives that have been learned around nuclear or around those kinds of things have really not just failed to produce good information, but have led to people constantly producing bad information and having an understanding of economics that’s either just straight up false, just not borne out by the facts, or highly speculative and not having that much evidence behind it.
I don’t want to overstretch my conclusions either because I want to be responsible about this question. But I would say that, once again, the best way to look at AI is as an economic question. And that doesn’t mean you don’t have competition. We have competition over how we make steel. We have competition over how we make, you know, various manufacturing technologies.
And I’m not saying that, you know, a purely laissez-faire method is correct. Once again, you know, I prefer, you know, treating allies well and then, you know, doing some trade policy and industrial policy against, you know, against the adversaries, against China, for example. So there’s no kind of clear cut, you know, absolutist solution here. It has to be nuanced.
But I would say that if you take the lens that it’s an economic competition, then the question is, how do you and your network of allies work together to outcompete, to produce the best products, to produce the best technologies, to produce both the best new innovations, which is important, which is a very important piece of the puzzle, but also the actual knowledge to use those products, the whole package of essentially taking a technology to market.
I think that actually it’s probably easier to explain this through analogy. I’m realizing this now. When you think of an iPhone, it’s not just its constituent parts. Its constituent parts are really important, and the technical blueprints behind those parts are really important. I’m sure, you know, Apple is very glad that they have those. But they’re not the only thing.
The supply chain that makes those iPhones, the factories that make them, the procedures around that, the implicit knowledge that everyone has of how to use an iPhone, of using the UI, of, you know, connecting everything to it, some things that are kind of under scrutiny as well, but are very important to the daily lives of people who use the iPhones. These are all factors that change the economic outcomes of a company like Apple.
And I think that maybe this, now that I say it, it sounds kind of obvious. It sounds really simple and facile. But when it comes to AI, I think people are still learning these lessons. That it’s not just about the idea. The idea is important, but it’s also about the distribution. It’s also about the knowledge. It’s also about the actual products that use that AI and make it easier for people to use it. The adaptation, all of that matters. And it has to be like a nuanced all of the above strategy.
Jim: And if we take the analogy of the server, right, which has become a pure fungible commodity bought and sold based on pure price performance with no rent seeking in it at all. That would say that the real economic competition is not going to be in the models. It’s going to be in their use.
Brian: Yeah, yeah. There are some schools of thought that believe this. Like I said, I think Mark Zuckerberg, to some degree, believes a version of this. And it’s really interesting because the worry is then you get underinvestment in the actual technologies and the fundamental research. That’s one of the worries if you end up in a situation like that.
I would say that we’re not quite there yet. I think that one of the reactions to DeepSeek has been an overreaction where people say, okay, now these companies have no moat at all. We should short all of these AI companies. I think that’s a little bit of a caricature. But I do think there’s an overreaction there. I do think there are still some benefits to being an incumbent. It’s not all over for them. It’s not all over for OpenAI. It’s not all over for Google.
But once again, it’s nuanced. And these decisions matter. And both lanes will matter. If there’s one takeaway that I think people will more and more begin to adopt in 2025 and 2026 is that there are clear and definite lanes for AI.
There was a great metaphor by Sayash and Arvind who wrote a book called “AI Snake Oil.” And they said, imagine you wake up, go to your car dealer and you ask them, what’s the latest engine doing? They’re like, yeah, there was this frontier research lab. They released this new jet engine. Here are the evals that the military is doing on this new jet engine.
It’s crazy, but it’s how we talk about AI today. We talk about AI today as if it’s all one thing. We’re not thinking about the product. We’re not thinking about the context. We’re not thinking about any of that. We just have this big, ambiguous term, AI. And it’s really funny. Because when you think about it in terms of engines, it’s like, it’s crazy. It’s literally the actions of an insane person.
But it’s sort of how we started this conversation and how everyone starts these kinds of conversations is by talking about, you know, AI, just in general. But what I think is going to happen is people are going to develop more sophisticated understandings of the lanes of AI. That, you know, some of these tools, they’re used for scientific research and they’re used to solve difficult math problems or to invent new drugs. Some of these, they’re customizable so that you can use it in your office, that you can use it. You don’t have to wait. You just get your answer. It’ll, you know, book your flight for you. And that’s the purpose of that version of the model.
Broadly, I think that there will be three main categories, three main lanes for AI. One is going to be customizability. You don’t care too much about the cost, but you care about making it do something specific and predictable. And that’s where open source really shines. Because open source, you have unparalleled flexibility compared to something like OpenAI, which is a closed model. Why is that? Because you can look at the weights, you can modify them, you can run whatever training technique you have, the expertise to do on them. And you can customize them to the specific task that you want to do. It’s almost as if it was made for this lane.
The second is the ultra-cheap enterprise lane. And open source might still compete for this. But right now, it’s been the clear strategy of Google to target this. So if you’re a big enterprise customer, you want one system that’s synced across all your files, all of your companies’ policies, all of your companies’ privacy, and is really trusted, really highly modified and monitored and centralized, and is just one tool that you as a legacy bank or institution or even government can take in. I don’t want to be advertising for Google, but that’s at least who they’re targeting. And I think they’re at least doing a decent job at providing the things that that lane needs.
And then the third is going to be the O1s, R1s, the experimental research, you know, reasoning questions that are honestly answering scientific questions that most people have no use for in their ordinary lives, but that will really matter to the ones that who can actually use it.
Jim: That makes sense. One of the things I did learn from a long career in professional information services, I built stuff for Wall Street, for science, for tax, for law. At the margins, people will pay a considerable premium for the best, but there are a lot of people who won’t. And so the structure of the market, where, let’s say when we get to GPT-603, that’s the one for the professional researcher, will that be of any significant difference for most people versus the open source equivalent? The answer may well be no, unless you’re at the absolute frontier of some very difficult problem.
Brian: Yeah, I think that you had this experience when they released O1 to the public, that a lot of people tried it. They were like, this is worse than 4.0. And that was because they were asking it to do questions that it really wasn’t intended for. And, you know, you can argue this was a communication problem, but I think it’s really just a reflection of reality. Because most people don’t need to answer complex, you know, scientific or mathematical problems. Most people have a thing they want to get done in their ordinary life. And so I think in some ways that’s already the case.
Jim: Yeah, that’s true. The first time I got my hands on O1, I kind of go, well, because I was using my normal engine benchmarks. But once I understood what it was good for, I go, holy shit, this thing is in a different league entirely in terms of its ability to do multi-step analysis, break a problem down, and then work on the sub-problems and do the integration. It’s actually pretty impressive.
And O1 Pro is even better, but it’s slower than shit, so you don’t use it too often. And I still use O for most things. It is interesting how our tools are now starting to, it’s not just one thing. It’s a Swiss Army knife and different blades are usable for different things.
I just used Operator the other day for the first time. I go, okay, it’s kind of interesting. I’m not sure I, and I put it on a real world problem and it came back with a solution. I go, that’s fairly impressive. This is again, this idea of the difference between the engine and the applications is where the rubber is really going to meet the road in the years ahead.
Brian: Yeah, I absolutely agree. There was this big divergent moment when people were, I’m actually a little bit too young to know about this, but I’m sure you know about this. And a lot of people have told me about it. This big divergent moment with operating systems, with the graphic user interface.
Jim: I remember it very well. All people who were the experts on, and I mean this, like, I don’t mean this in a sarcastic way in the way that it’s used in kind of modern day, who were like genuine experts and who I really respect a lot, a lot smarter than me, who knew how to do everything with a terminal, who could do it blazing fast, they didn’t need the graphic user interface. They looked at the graphic user interface and they said, like, this is slower than how I normally do it. Just give me the terminal.
And they were, in their narrow way, they were right, right? For their daily purposes, they were absolutely right. And at the same time, a lot of people who are just starting to use a computer, they love the graphic user interface. And it’s still like what most people use today, right? Like we’re, I don’t know about you, but I’m using the graphic user interface, right?
There are these like really important changes that kind of, you know, can only be read backwards in history. Can’t really be read forwards in history. I don’t know what the ultimate, like, what the final version of like the mass market AI interface will be. You know, Mark Zuckerberg says it’s going to be the Meta Ray-Ban, right? I’m not sure if that’s right. But I think it’s going to be different than the terminal that we use today.
Brian: I had a conversation with somebody about that just yesterday. And we were trying to decide where is AI in terms of its evolution? Is it 1982?
Jim: AI is meaning LLMs. And also, I have to insert this. I do this whenever we’re talking about this. Please don’t use AI to mean LLMs. LLM is a subset of AI. There are other forms of AI. We’ll have a nice podcast coming out tonight with Jeff Hawkins and his approach to AI, which is quite different and quite interesting and might kick LLM’s ass. We shall see. He thinks so.
Brian: Yeah, I mean, I used to be a real stickler on this too. And like all the people at my old company would say like machine learning. We wouldn’t say AI. Maybe like to, you know, on the earnings call, the CEO might say AI. Right. But within the company, we usually said machine learning or something even more specific. It’s just the kind of mainstream, right? And if you want to talk to more people, it just ends up being a kind of bad habit that gets ingrained in you.
Jim: Yeah, I’ve even found myself falling into it the last couple of weeks despite, goddammit, that’s not correct. And it’s quite misleading. But anyway, neither here nor there.
Brian: Yeah, it is. It is. And I think this is a really big problem. You know, there’s the question of how much nuance the language implies. I think of all of the terms of language models, of machine learning, of, you know, autoregressive language models, of, you know, transformers, there’s a lot of ways that we can talk about this technology. And we kind of chose the most, like the one that inflicts the most blunt force trauma.
And I think on one hand, there’s a kind of charisma there. There’s a kind of marketing that, you know, probably is good for the AI industry as a whole even. But on the other hand, just as like a stickler for truth, I don’t like it.
Jim: Yeah, I hear you. I hear you. But even I find myself falling into the linguistic hole. I just did it a few minutes ago.
Brian: Yeah. I’m the same.
Jim: So anyway, one of the things that you don’t talk much about, or at least downplay, which other people make a big deal about, some people think it’s the biggest deal of all, are the safety, ethical, and social considerations around what happens as this frontier of capacity, both at the systems level and at the applications level, if we want to make that distinction, continues to move forward. So what do you think about those issues? Let’s start with safety.
Brian: Yeah, it’s interesting because I had a friend of mine reach out asking if I had any recommendations for a kind of optimistic lens on ethical implications. At the time, the thing that I thought of first was this great statement by the Open Source Research Foundation, Nous Research. So N-O-U-S, the French word for “we.” And they had a really fantastic statement and video about many of the themes that we talked about, but about a kind of Silicon Valley 1.0, kind of like almost anarcho-socialist vision of like having a democratized technology, you know, having technology that everyone could access, that you can access without barriers, that you wouldn’t be gatekept on, that is not like a tool for power, is not a tool for oppression, but is a tool for kind of individual liberation almost.
I’m trying not to sound too much like, you know, some form of socialist. But I think that, you know, with the kind of ideological kind of like connotation of these words aside, I think it really is a genuinely really optimistic vision. And optimistic not just for political reasons, but for very practical reasons. Like, I want AI tools to be cheap, accessible, and to make everyone’s lives better. I think that’s, you know, that’s not really ideological at all.
And I think that the really worrying scenario, to bring it all the way back to the very beginning, is a world where AI is really tightly gatekept, that it’s a big economic driver, but like three people control it. That’s a very dangerous economic system. That’s like, you know, that’s like Saudi Arabia, right? You have a very important resource, like very small number of people control it, everyone depends on it, Saudi Arabia.
And I don’t want to, you know, I’m here in America, I don’t want to live in Saudi Arabia, I don’t want to live in a society like that. And I think that having democratization, having like genuine form of equal access, that is important to me.
And people kind of, and for good reason, are skeptical of, you know, kind of quote-unquote ethical concerns, right? Because a lot of the time they worry that’s an excuse to take power away from you, to take power away from, you know, the person listening to this. And I think that that’s legitimate in some cases. But when I think about the ethical implications in a genuine way, I think about this, I think about the balance of power, I think about both the balance of economic and political power. And I, you know, once again, I’m optimistic. I think that there are factors that kind of balance this. I don’t think it’s a natural monopoly, but I do really prefer the vision where there is equal access over the vision that’s more like Saudi Arabia.
Jim: Interesting. Yeah. What about the argument that there are just so, there’s so much danger from this technology, you know, people be able to figure out how to do home CRISPR and create super virulent plagues or build massive EMP bombs or something. What do you think about that argument on why maybe regulation might be justified?
Brian: Yeah, I think one of the problems here is that it comes from exactly the problem, the cave below Plato’s cave that we talked about earlier, where people are… I had an unironic Twitter reply guy when I was talking about one of these policies. I had an unironic Twitter reply guy be like, oh, are you going to open source the missile defense system?
The language models that we are using, maybe there is some amount of AI for the targeting systems or for the analytics. Maybe that falls under the banner of AI. Maybe it even falls under the banner of machine learning. Show me where people are open sourcing the missile system. That’s not happening.
And to conflate that with large language models, I think some people do it maliciously, but I think a lot of people really do it out of ignorance. They just hear AI. They just hear this term. And to be fair, there’s a grain of truth there. There are some machine learning technologies that come from the same branch of statistics and academia that are useful for navigating missiles. That’s a real thing. It’s just that the language obscures more than it helps.
There are some people who, you know, their actual argument is that like the LLMs themselves are the danger. And I think that there’s been recently a lot of counter evidence to that claim. The anecdote that a lot of people bring up is that, you know, imagine you’re Mark Zuckerberg. You’re sitting in a hearing with like members of the U.S. Congress. They ask you, Mr. Zuckerberg, are you worried that this Meta-LLAMA model may result in a recipe for a certain chemical toxin. And he opens his phone. He opens Google. And he finds, believe it or not, from google.com, already in everyone’s phone. Imagine that. He finds the chemical formula and some instructions for this biological compound.
And so the question you really have to ask yourself is, if it’s going to be a risk, why isn’t it a risk already? Number one. And number two is, where is this conflation coming from? What is the actual ideological underpinning for this assumption?
And I think that, the second part, is a really interesting question. And it’s a question that a lot of people have attempted to answer that I think none of them are quite suitable, but all have a grain of truth to them.
After 9/11, the approach to policing and the approach to counterterrorism really changed. It changed from approach of innocence until proven guilty to very often an approach of guilty until proven innocent. This is, you know, now the controversies over, you know, various programs that were used in the post-9/11 kind of surveillance apparatus to target terrorists.
There are some claims that it’s now being used against American citizens as well. And, of course, also claims at the time that it was being used against American citizens, the latter of which I believe is true. There are now like court cases that are ruled in the defendant’s favor where that was the case.
And this led to an approach where I think people became very uncomfortable with their fellow man. They thought, what if my neighbor is a terrorist? What if my neighbor is a domestic terrorist? What if my neighbor is going to create a bioweapon?
And I think that’s, I think that’s like number one, factually not true. Like we have Google once again. So number one, I think that’s factually not true. And number two, I think that’s a really dangerous philosophy. I think that philosophy has led to a lot of the overreach we’ve seen in the past years, and including the overreach on AI, I should be clear, explicitly has led to the overreach on AI. Like this is the motivation that some people cite.
And it’s also led to a kind of decline of trust. And a decline of trust not just in terms of political agreement. Like some people say, you know, oh, there’s a decline of trust. People don’t like governments from the opposing side. But no, I think this is much worse. Like it’s much worse to believe that your neighbor is going to be a terrorist than it is to believe that like the political party you don’t like is bad. I think the latter we’ve had a lot of precedent for. The former we haven’t really had a precedent for in America. And the global precedents have not really ended very well. The global precedent is like the Troubles, you know, or like, you know, all these periods of immense political violence, which I don’t think have ended well.
Jim: I think this has been a very interesting conversation. I want to thank Brian Chau. And as always, you can get links to a lot of this stuff that we talked about at the episode page at JimRuttShow.com.
Brian: Yeah, this was really fun. And I just love the tour de force of all of these questions. You really took it in every angle. It’s amazing.
Jim: It’s been fun. You were there. You handled it. The ping pong ball back across the net, always good.