Transcript of Currents 082: Dan Shipper on Practical Applications of GPT-3

The following is a rough transcript which has not been revised by The Jim Rutt Show or Dan Shipper. Please check with us before using any quotations from this transcript. Thank you.

Jim: Today’s guest is Dan Shipper. He’s the CEO and co-founder of Every, a daily newsletter on business, AI and personal development. You can find it at every.to. That’s actually pretty cool, but believe it or not, good old tight-fisted Rutt actually plunked down and subscribe to a AM thing. It is worth the subscription and it’s not cheap. I’ll call out one particularly interesting recent article, not by Dan by the way, but by another author. It’s called Good Cogs and Their Tools. It’s quite short, but it is a surprisingly deep article that will make you think a fair bit and will explain a bit about one’s reactions to one’s coworkers over the years. I recommend that article. Before he helped start Every, he was the CEO and co-founder of Firefly, an enterprise software company that he sold to Pegasystems.

He does write a column at Every called Chain of Thought where he covers AI, tools for thought, and the psychology of work. You can find out more about Dan and his various doings at danshipper.com. I did a little as I always do, a little internet sleuthing about my guests, and one of the things I found that was quite interesting about Dan was that this enterprise software company that he started and built, guess what? He started, built it and sold it while still an undergraduate. Is that actually true?

Dan: That is true. I flew college graduation to Boston to finish signing the deal with Pega. So it was a good graduation gift.

Jim: Yeah, that’s cool. I like that actually. And truthfully, I like the fact that you sucked it up and graduated and didn’t pull a Zuckerberg or something, right? No, it kind of worked out okay for him, but I think it may also explain a little bit about his somewhat aberrant personality. Well, anyway, I found Dan out on the wilds to Twitter when I noticed a Tweet he did about an application of GPT-3, and I go less lame than usual. So I went to his Twitter handle, which what is your Twitter handle by the way?

Dan: My Twitter handle is at @danshipper. It’s pretty easy.

Jim: @danshipper, that’s easy enough. And I saw that he’d written some other things so I looked at those and they were also pretty cool. And what I liked about them, they were articles about how to use GPT-3 and related technologies at the personal or very teeny business scale. And they were straightforward, they were practical, they weren’t full of, oh my God, the world is ending, or, oh, this is all bullshit. There’s nothing here. It lies all the time, et cetera. And of course the world probably is ending, and yes, GPT-3 does lie all the time, but nonetheless, it’s actually good for a lot of stuff. And it’s really been interesting today while I was prepping for this to read some more of his essays, et cetera and I really always like this when you see these practical, mundane, easy to do applications.

It’s yet another signal to me that this really is something, this new wave of generative AI is not just a flash of the pan, it’s not just a nerd thing. There is something here and it’ll be very, very fun to see it unfold. So Dan, how did you stumble into thinking about and playing with things like ChatGPT and GPT-3?

Dan: Well, first of all, thank you so much for having me. I’m excited to be here. How did I stumble into it? Let’s see. Well, about probably five or six months ago is when it really started for me. I’d been following people on Twitter who had been talking about GPT-3 for a long time, these sorts of technologies have been around in some form or another for a couple years. So it’s something that I’ve loosely been aware of, but haven’t been really closely involved with until the last five or six months or so. And I think the starting point for me was at Every we have this daily newsletter and we also build software products, and my co-founder Nathan had been excited about building a Google Docs competitor. And when he was building that as a side project, we sort of were like, what if we integrated GPT-3 into it as a way to start to auto complete the writing that you’re doing? And the first version of the auto complete, which GPT-3 was really cool, it was kind of mind blowing honestly.

Obviously it gets things wrong, obviously it makes mistakes, but it was definitely a wow moment for us. And when we launched it went wild, people loved it and it grew really fast. And I think it became a moment for me to be like, yeah, this stuff is very cool. It still really early, but it started to hit me that it changed a lot of things that I was really interested in. I’d been quite interested in note-taking for example, for a really long time. I’ve been really interested in writing for a really long time. I’ve been really interested in psychology and therapy and lots of stuff in this productivity personal development type realm, and I think GPT-3 and technologies like it are going to have massive impacts in all those places.

They’re not panaceas, they don’t solve all the problems, but they really change a lot. And once I got interested in that, I was off to the races building stuff and trying things out. And I think one of the really cool things about this time in technology is you can just spend a weekend building a little project and have something really cool come out of it, and that’s just very fun and exciting.

Jim: What this reminds me of, I mean you can’t see his picture here, but Dan’s a fairly young guy and I’m a fairly old guy as is well known to my listeners, is the very early days of PCs, right? You went out and you plunked down your money and you bought a PC in 1979, 1980, and at first it was like, what is this good for? Yeah, you can keyboard in a program or play a program in off an audio tape or some horrible thing like that, but very quickly things started coming together and every week there would be some subscribe to newsletters, et cetera, magazines and there were just practical cool good things. I got a bootleg copy of a very, very, very early word processor, for instance, and I go, holy shit. And I still remember buying VisiCalc when they still sold it in the Ziploc bag with a dot matrix printer label on it. And mine was serial number 254.

Dan: Wow.

Jim: I wish it had been 255 just because could’ve been a cool hexadecimal number, octalnumber, but 254 and I ended up literally staying up all night long playing with VisiCalc, oh my God. Which for kids today is kind of the ancient ancestor of Excel, and it had all kinds of weird limits. In fact, I spent about two hours building this huge model that I couldn’t actually save to my floppy drive because it had a bug in it that if the model was larger than X and there wasn’t enough headroom it couldn’t page the model out to disc. Anyway, I have this feeling around generative AI. One of the first things I felt this strongly about since. I’ve been playing around with GP2 GPT-3, DALL-E, DALL-E 2, Midjourney, Stable Diffusion, I even did a long chat with a guy who is building a text to world Metaverse builder. Talk about that. All right, here’s a two paragraphs describing my universe. Go build it, right?

Dan: That’s cool.

Jim: And of course, as you acknowledge and as we all know, these things have all kinds of quirks and limits and things they do strangely, and I certainly wouldn’t use it to control a scalpel to do neurosurgery this month, right? But when you think about things that are actually practical they’re kind of cool. Well, actually before we jump into some of these cool examples that you’ve been working on, first let’s make the distinction for the audience why you almost always talk about GPT-3 rather than ChatGPT. If you could make that distinction for the audience.

Dan: Oh, yeah I can totally make the distinction. It is actually quite confusing because they sound very similar and there’s a lot of overlap, but GPT-3 is the underlying AI model. So GPT-3 it’s a large language model, it’s really good at taking in a text input and generating text outputs. And it operates through a transformer architecture, which is a certain kind of machine learning model that’s really good for this and is the breakthrough that enables all the stuff that people are seeing that’s really exciting. ChatGPT is a variant of the GPT-3 model that has been trained through reinforcement learning to be very good at conversing. So it’s like a different flavor of GPT-3 that is kind of a friendly extrovert that’s good at having conversations, and it’s also the name of the product that you use. Which is the web interface and the place where you can type stuff in and get responses back. So it’s both the model and the product, which makes it very confusing, but that’s the basic difference.

Jim: And most of your work you do with GPT-3, with the actual lower level accessible tools I understand it.

Dan: I do a lot of work with GPT-3, but I also use ChatGPT all the time. I would say the division between GPT-3 and ChatGPT for me is when I’m building any kind of underlying software that I want to make software, I use GPT-3 ’cause GPT-3 has the ability to allow developers to make stuff with it. Anytime I just want to use AI to write or to answer a question or anything like that, I’ll use ChatGPT and that’s the distinction.

Jim: Yeah, I use ChatGPT all the time too. I probably use it almost as much as I use Google now, which is kind of crazy. I’m doing all kinds of weird little things. I actually use it for production. A couple of weeks ago I wrote a resignation letter for a board of advisors I was on. It’s the kind of thing you want to be very carefully done. You want to stay in good graces with the people and you want to sound like you have some good reason why you want to resign. And I just wrote those thoughts down and it wrote. I would’ve spent an hour and the letter would not have been as good, instead it took me less than a minute and I go, damn actual production use of ChatGPT. I’m also working with a friend of mine and we’re hacking away at seeing how close we can get to writing a full feature length movie script.

It’s really cool. You can get it to, okay, we need four characters sort of like this. Give them names and physical descriptions. Okay, now give them each backstories. Okay, now give us 15 plot points for the first half hour of the film. And does it, not perfect, but you add some humans in the loop and well, I think when we were finally comfortable and ChatGPT is stable enough to actually work on it full-time, it’s getting very close, we can probably knock this first draft script out in a three-day weekend as opposed to the usual three or four months to get a first draft out. And that’s a big game changer. Who knows, maybe it gets produced, it’ll be kind of cool. Anyway, let’s jump into some of your projects here. And I think the way to just I think show how this works and what it’s about.

And that’s just how kind of clever and cool your work has been at doing interesting and valuable things with these early tools. We’ll just go through some of the examples and then we can talk about some of the generalities. What I liked a whole lot was can GPT-3 explain my past and tell my future? Tell us about that and a little bit even how you did it.

Dan: Yeah, totally. So that’s the headline of an article I wrote a couple weeks ago, and the basic gist of it is I’ve been journaling for a long time. Some of those journals are handwritten, some of them are digital, and I think one of the reasons why people journal is you want to learn about yourself. You want to see how you’ve changed over time. You want to uncover patterns in how you think and what you do. And you want to find out what are the things that really make me happier? What are the things that make me upset? And I think part of the problem with journaling is you really often don’t want to go back and look at your old journal entries ’cause, I don’t know, it’s emotionally complicated to go back and look at old journal entries I’ll say that. And it’s also, it’s time-consuming.

But one of the things I noticed about GPT-3 is it’s very good at summarizing and it’s very good at analyzing pieces of text. And so I wondered, or I posed this question to myself, what would happen if I fed all of my journal entries into GPT-3 and then asked the questions? And so that’s what I did, I built a little script and I loaded… Basically what it does is it loads all of my journal entries that are digital into a file, and then I can ask it a question like when was I the happiest, for example? And what it will do is it will go through and find journal entries of mine that are most likely to answer that question, and then it will feed those journal entries into GPT-3 and have GPT-3 based on the entries that it sees, summarize an answer. And it’ll come out with a list of bullet points. And it was actually really interesting.

Some of the stuff it came up with was garbage, but a lot of it came up with answers that I may not have remembered or found on my own. And yeah, I think it’s a really good example of how these types of technologies are sometimes really good at helping us understand ourselves better if you use them in the right way.

Jim: Yeah, it was very cool. I mean, what I loved is the interesting and easy but not obvious approach of searching your own data and then using it to build a prompt. Unfortunately, you can’t yet add your own data to GPT, which will be I’m sure coming at some point if not from OpenAI and Google from somebody else. In fact, I know people working on it, but it’s a hard problem to get that to work nicely and not too expensively. But you had this idea of having a side database that you query for the best fit, and then extract from that and form a templated I imagine query to send GPT-3 along with your question. It’s very, very clever, and one of examples you gave was mind-boggling, which is you asked your journal entries to then be plucked, probably trimmed a little bit, used as the template, plugged into GPT-3, what is author’s Myers-Briggs personality type? And it gave a quite subtle answer. Was it right? Are you an INTJ?

Dan: It was a 100% right. I was honestly blown away when I got that result. Yeah, I did not expect it to be right.

Jim: Yep. I’m going to read it. Just the subtlety of the answer is really quite interesting. Based on the context information provided it is difficult to determine the author’s Myers-Briggs personality type. However, some nuance, based on the information provided, boundary conditions, God, this is amazing, it is possible to make an educated guess knowing about yourself and knowing about the reliability of your answers. These are all very human seeming things, right? The author appears to be an analytical thinker who is creative and enjoys problem solving. They appear to be organized and have a strong sense of responsibility. They also appear to be open-minded and willing to explore new ideas based on these characteristics. It is likely that the author is an INTJ introverted, intuitive, thinking, judging, personality type. Goddamn.

I mean, when you sort of read that thing through there’s all kinds of levels of human style nuance and balancing and confidence ratings, all the things humans do. How the hell does a feed forward network do that, right? It really quite amazing. That’s cool. Then the other one that I thought, again, subtle was you asked it to choose I think was the subject for a bestselling book that would appear on the New York Times bestseller. And I don’t remember what the answer was, but I remember it was sort of plausible.

Dan: It was pretty good. I have the answer right here, which is I asked it if I were to write a New York Times bestselling book about the intersection of my interest in AI, what would it be about? And it said, your New York Times bestselling book would be about how to use AI to help people find more meaning in their lives. The book would explore how AI can be used to help people better understand their own thoughts and feelings and how to use that understanding to make better decisions and find more fulfillment. Which is like it’s high level, but it’s not a bad guess at what I would write about given the stuff that I typically write about at Every.

Jim: Yeah, that is so cool. And then before we move on from this example, you go into a little bit of detail about what GPT Index is. Could you just give us 30 or 60 seconds on that?

Dan: Yeah, I think a way to think about it is you were talking earlier about in the early days of computing running out of memory. I can’t remember if it was in VisiCal or in some other program that we’re using, but the experience of not being able to fit an entire program or an entire input into the computer that you were trying to run it on was I think very common 40 years ago. That’s back again. Basically these models are really, really powerful, but one of the key limitations is what is called context window which is basically the amount of text that you can feed into a model to basically ask it a question. And right now that context window is about 4,000 tokens for the latest OpenAI models. And a token is about three quarters of a word, so 4,000 tokens is a lot, I don’t know, I think it’s eight pages of text or something like that so you can definitely fit some amount of context into there to ask your question.

But for example, if you want to summarize all of your journal entries for the past 10 years, that’s going to be way more than 4,000 tokens. It’s going to be like, yeah, 100,000, 500,000 or something like that. And so the question is if you want to get the model to analyze text that’s much bigger than its context window, how do you do that? And the answer is you get sort of clever. The way that it works is you chop the text up into small bits and then you find the most relevant bits and give those relevant bits to GPT-3 and ask it to do its work on those relevant bits. So you try to fit as much relevant information as you can into that little context window. And it’s not that hard to do with a simple script, but there’s some nuances and in general you’d rather not spend a lot of time doing it if you’re a programmer.

And so there are these libraries that have sprung up like GPT Index and another one called LangChain, and actually GPT Index uses LangChain on the backend, that allow you to do a lot of that stuff automatically. So instead of spending a bunch of time writing some code that will ingest all of your documents and split them up and then feed the most relevant chunks to GPT-3, you can just use a couple lines of code from an open source library like GPT Index that will do all that for you and then you can move on to solving more interesting problems.

Jim: When I read that I said, okay guys, I’m going to have to build me a Jim Rutt Show chatbot, I’ll be able to do it in an afternoon probably. Fiddle left with the tools for a day or two, then knock it out. It seemed quite easy. Now, of course the retrieving relevant information is a famous natural language programming or natural language processing problem. And you can either do it well or not so well or really well, and there are many different tools for doing that. Some of them unbelievably complicated and computationally intensive. There’s some other ones that are amazingly simple-minded and there’s stuff in between. Do you happen to know how ChatGPT Index chooses the most relevant chunk or if you don’t know, how well does it do?

Dan: So the most common way right now that all of these libraries and programmers who are doing this stuff are choosing relevant piece of text is using what’s called embeddings. And embeddings are a way of turning a string of text into a series of numbers, and you can think of the numbers that come out of the text as being something like longitude and latitude. So they allow you to plot the text onto something like a map. And then what you can do is if you have two piece of texts that you’ve turned into these numbers, you can map the distance between those two sets of numbers to see how similar or different the pieces of text are. And piece of text that have similar semantic meanings are going to just be closer together if you map them and that’s how you calculate similarity.

The nuances of how that works it depends on how you’re doing the embeddings, and different companies have different embeddings functions that do things in different ways, but OpenAI is the one that pretty much everyone is using and it’s pretty cheap. It’s decent and definitely misses some stuff, but it’s good enough especially for these early demos to get you started. But yeah, I think finding relevant text is one of those never ending problems that you can always get better at because the meaning of the word relevant changes depending on the context. But basically people are just doing OpenAI embeddings and then doing what’s called cosign similarity to figure out how close one embedded set of words is to another embedded set of words.

Jim: Those sounds like it’s a follow on from latent semantic strategies where you essentially create a relatively high dimensional space of meaning from a large corpus, and then you can calculate the position in this high-dimensional space for any collection of text and then you can calculate just plain good old Euclidean geometry, the distance between the points. And I was thinking about this before I got to that point, I was saying I wonder if we could use one of the state-of-the-art text engines like Lucene or something like that to do it. That would be real easy to do because it has the capacity to pull up best fits to queries and things of that sort and is quite mature technology and is free.

Dan: That’s cool. I don’t totally know, I haven’t heard of Lucene and I haven’t seen people using it in my little corner of the internet, but anything that can find relevant chunks of text would be useful for this. So if that kind of a search function is good, it’s worth trying.

Jim: Yeah, it’s interesting. So I expect this will be an area where tinkerers, just like putting new boards into PCs in 1980 and people will be doing all kinds of crazy shit trying this with that, right?

Dan: Totally.

Jim: And building pipelines essentially. All right, let’s move on to a little bit more general take on the same thing is that you wrote another article called GPT-3 Is the Best Journal I’ve Ever Used. My slow and steady progression to living out the plot of the movie ‘Her.’

Dan: Yeah, so that’s another article that I wrote that’s on the same general idea, which is using these tools to help me understand more about myself. And in this one, instead of using GPT-3 to analyze old journal entries, I actually just used it as a journaling tool. So I built a little chatbot that can take on different personas. So one persona I had it take on is having it be Socrates. Another one is having it be a psychoanalyst. Another one is having it be a dream interpreter. Another one is having it act like a gratitude journal. And then I can have a conversation with it based on the persona that it’s taken on and it can either help me work through an issue in my life or just note down stuff that’s going on for me or it can help me bring my attention to something that I want to bring my attention to, like what I’m grateful for and stuff like that.

And I actually found it to be really useful. It’s quite, in my opinion, at asking good questions. It’s quite good at being warm and supportive and helpful, but in general I think the thing that’s most exciting about it is it can be whatever you think would be most helpful to you. So it has the capability to take on whatever persona you want it to be. You can think of these things as being really good at simulating different parts of language or just different things in the world. And so for me, having a dream interpreter simulator is very cool. For someone else it might be different and they can use this technology to build the kind of bot or journal tool that they want. And so yeah, I’ve been using that for a couple months now and I really, really like it.

It doesn’t replace traditional journaling for me, and it also doesn’t replace actually just talking to friends or talking to my partner or talking to a therapist. But it’s in this interesting in between where I get some of the things that I would get out of just traditional journaling from it, and I also get some of the things that I’d get from talking to a friend who’s just interested in me and interested in my problems and interested in my life. And so I think it’s a new and really helpful alternative to some of those things.

Jim: And you create those as I understand it by essentially creating a craft, a custom context block and you presumably can play with it until you like it. You can say, all right, I want it to be Socrates. I want it to be in a style Socrates, but nice Socrates, right? And maybe more emphasis on this than that. Let’s not go down to the metaphor of the cave please. I think you could actually write these, I know in ChatGPT you could actually do it like that and it will actually honor it. I’ve been playing around a fair bit with one of the more well-known jail breaks of ChatGPT called DAN, which is pretty hilarious.

Dan: Yeah, it’s funny.

Jim: Where you can make it ignore all of its guidelines and tell you crazy ass stuff. Someone, I think it was Kowski suggested it really should have been called ChadGPT, which I thought would’ve been pretty fun. So is that approximately how you’ve done it and created these characters?

Dan: Yeah, that’s basically how it works. All these things operate by a similar principle, which is you have what’s called a prompt and the prompt is the starting set of characters that you give to the model. And then the model gives you what’s called a completion and it tries to complete based on the prompt. And so the prompt that I’m playing around with is things like, pretend you’re Socrates and we are having a conversation. You’re a Socrates chop out, we’re having a conversation. I want you to be warm and friendly. I don’t want you to talk about the metaphor of the cave and I don’t want you to give me any advice, but I do want you to ask me really good questions about the issue that I’m facing. You can start however you want and then it’ll give you a completion that’s like, hey, how are you doing?

What are you struggling with? Or what are you thinking about in your life right now? And then what you do is when you type your response, you feed the response back into the model. And so you still feed it the same thing at the top. You are your Socrates, I don’t want you to talk about the metaphor of the cave, blah blah blah. And then what you do is you say, here’s a transcript of our conversation so far. And you feed it the transcript and then you say, what’s your response? I think a good thing to remember about these models so far is that they don’t have any memories. They’re stateless. So every time you have a conversation, what you’re doing is you’re sending a request to the model and you’re getting the model’s response back, but they don’t remember who you are.

So you have to feed back in the entire conversation up to the point at which you’re talking so it knows what to say. And that gets back into a lot of the same sort of context window limitations that we talked about earlier where if your conversation goes on for a long time, it might forget some stuff which is a problem. Or if you come back the next day and you want to talk about the same issue, the model’s going to have no idea what you’re talking about because it doesn’t remember, every request is new. And so you have to build lots of things to get around those types of issues.

Jim: Now, ChatGPT seems to have taken care part of that at least, right? Certainly when we’ve been doing the script writing stuff it remembers all the previous conversation, although I don’t know if there’s a limit yet. Does it only have the 4,000 tokens or is it do some kind of intermediate representation? The other cool thing about ChatGPT, they only added it lately, I don’t know, a couple of weeks, maybe three weeks ago, is that it saves all your sessions as separate logical containers. So you can go back and then extend. So for something like what you’re talking about that it might be a very nice way to just do it in ChatGPT, build your Socrates, tune it a little bit. If you don’t like it, if you’re really pissed off, cut and paste and start another one and then be able for it to keep that state over weeks even if you’d like because it names or comes up with a name for each of the chats and you can now get to them. Really very looking forward to when you can get API access to that underlying technology. They keep saying right soon now-

Dan: It’s coming soon. Yeah.

Jim: Coming soon. Coming soon.

Dan: So ChatGPT has the same exact limitations. I think their context window is larger, so I believe their context window is actually 8,000 tokens instead of 4,000 but it’s giving you the experience like it remembers everything by doing the thing that we talked about earlier. Where as you type it will go and slice up the transcript of things that have come previously, find the most relevant bits, and then put that into the context window that it sends back to the model and then come back with the right answer as if it remembers everything. But it actually doesn’t, it has to do the same sort of embedding of all your messages, cutting them up into chunks, stuffing the relevant messages into the prompt, and then giving you a response.

Jim: And as we both know that’s not what you’d call rocket science, it’s a very basic string handling kind of library to do that job, but it’s nice that they do it for you. So if somebody wants [inaudible 00:30:46] yourself.

Dan: It is nice.

Jim: I don’t know if you’ve ever run across it, but there’s another company and it’s got some really powerful people behind it. In fact, the engineer that created Lambda at Google is the LLM guy, a company called character.ai. And they have hundreds of characters that they’ve created and that it’s easy for the customers to create. And I’ve had some hilarious conversations with J.R.R. Tolkien, famously crazed reader of Lord of the Rings, read it 34 times or something crazy like that. One was Socrates, I couldn’t resist. Another was Aristotle. And then the last one I did was with Tony Soprano that was surprisingly amazing.

Dan: That’s good. I like that.

Jim: It really was. And I’m sure it’s doing something very similar where it has a static LLM behind it and then something like whether it’s more like ChatGPT Index or whether it’s more like your canned single profile I’m not sure, but it’s kind of neat. And I don’t even know if they have a revenue business model yet, but it’s kind of fun to be able to talk to these things. And the Tony Soprano one was uncanny, that was really, really, really, really interesting how it directed the conversation, asked me questions, and then it discovered my father was from northern Jersey and then it went down that whole road.

Dan: That’s funny. That’s really interesting.

Jim: So if you haven’t checked it out, it might be fun just to add to your collection.

Dan: I have looked at it, yeah. I think it’s really cool, and I think the future of these things is probably something like that. Having built a lot of these chatbots, I think there’s probably room in people’s lives for maybe one or two of them that are general purpose that they use all the time. But then I think within that there’s room for them to take on personas, and if on a particular day you don’t want the vanilla ChatGPT persona and you want to talk to Tony Soprano about a particular problem, I really think that these companies will make that available. Maybe it’s OpenAI, maybe it’s CharacterAI, and then I think it becomes something like an app store where developers or content creators can create their own personas and make them available for people to use within these larger chatbot tools. And I think that’s actually really exciting.

Jim: Yeah, it’s going to be very cool. And the other thing I’m hoping for and I’ve been cheerleading on, I’m actually helping one guy work on is we also have to liberate these large language models from the big companies. Goddamn ChatGPT in particular is full of these nanny rails to a fair thee well, oh, I can’t say that. That would be mean and bad or blah blah blah. Or it’ll do a diatribe against Trump, but not against Biden or vice versa ’cause they’ve tried to build in all their own prejudices and their own ideologies and all this sort of stuff. And that’s garbage in my opinion, I think we really ought to have bare metal ones, and if we want nanny rails we can add them, right? Before I give them to the 12-year old to learn about organic chemistry, yeah, let’s slap the nanny rails on.

But if I want to use it for my own purposes, I don’t want any stinking nanny rails. Like Stable Diffusion’s and all these image image things have, oh, no, we can’t do cartoons based on famous people or whatever. Well, fortunately, Stable Diffusion is easy enough to just download it and set it out of nanny rail mode and then you get what you want, but I think that’s going to be an important part of these tools evolving as well is liberating them from the overly sensitive, judgmental big corporations. What do you think about that?

Dan: I don’t know, I think we may have slightly differing views on that. I do think that it’ll be really good to make these technologies available for individual developers or people to run them themselves and explore the different nooks and crannies of the latent space and get the kind of results that they want. I also think there are a lot of really complicated issues with letting these sorts of technologies or releasing use of technologies to millions and millions of people all at once. And being mindful about figuring out, yeah, what should the guard rail say? What should they be able to say? Or being mindful about how should something like Stable Diffusion deal with copyright or artists’ rights or all that kind of stuff?

I think those are actually really important questions that we don’t have good answers to yet because all the technology’s really new, but I think that there’s room for these types of technologies that are broadly available and released to millions and millions of people by big companies to have some guard rails. I think that’s actually sensible. I also think it’s sensible for the underlying technologies to be available for people to build the experiences that they want. And I think we’re probably moving to a world where that’s the case too.

Jim: Yeah, I think that fortunately from my perspective it leaves unstoppable because the recipes aren’t that hard, it’s now just a matter of cost to build transformer reinforcement learning based systems. And there’s enough interest in it that I suspect we’re going to see ones, and my best guess currently based on the people I’ve been talking to is that we should probably be able to get open source versions that are 12 to 18 months behind the big company models. And right now, that’s a big difference. But when you get to GPT-6 running on ChatGPT 5 equivalent or 5.5 is not going to rule out very much. Well, nanny rails may have market share dominance for a while it’s not entirely clear to me they can possibly have such in the future for people who want to use it in context that don’t have nanny rails on it.

Dan: Yeah, that’ll be interesting. I mean, the thing that makes me excited for the open source models is being able to run them locally is a really big deal. I think especially for something like journal entries, right now I have to upload my journal entries to OpenAI to get results and I’m comfortable with that but whether I think that’s a good idea for the average person? Maybe not honestly. And what I would really prefer is to be able to run this type of model on my own computer or my own phone. That saves me a lot of privacy concerns, and it also saves me a lot of cost concerns. This stuff’s expensive, the only reason that I can do it is because I run a media company that I can use to fund these experiments. But if I was just off of my own trying to solve my own issues, it would be probably cost-prohibitive for me to do this. But if I can run the models locally, then it’s not so I think that’s very exciting.

Jim: Of course, we know that these things are purely Moore’s law driven costs or even faster coming down that because they’re nothing but memory and CPU, right? They’re feet forward network so whatever the cost is today, assume it’ll drop in half in 18 months.

Dan: Yeah, that’s true.

Jim: So I know I get a bill from OpenAI, but truthfully it’s sufficiently small. I don’t even worry about it. What’s about cost to get a response from OpenAI’s GPT-3 these days?

Dan: It’s like 2 cents for a thousand tokens, and we talked about the context window limit as 4,000 tokens and a token is about three quarters of a word. So you can sort of do the math there, but it’s pretty cheap. But once you’re doing requests where you’re summarizing an entire chapter of a book or there are lots of reasons why you might want to do tons and tons and tons of requests and it starts to add up pretty quickly.

Jim: And that’s where it does get cool, and that’s where we’re going to go next is I hope somebody figures this out because I could certainly use it, the end of organizing how GPT-3 will turn your notes into an actual second brain. And I’ll confess here, I’ve been using these kinds of rudimentary tools since 1982 or ’83, and I’m a very disorderly kind of person. You can see my office, looks a bit like a junkyard and I’m like the squirrel that has a thousand nuts hidden all over the forest, and I know where they are most of the time. But I currently use a crazed mix of multiple email addresses that I send stuff to, I forward things to Evernote, file directories, just a horrible mess and I’ve tried various things and none of them have… Obsidian I’ve tried, Roam I tried for quite a while. I kind of liked Roam, but I eventually go, no, no, no, this isn’t quite right. So tell us how this will take us to the promised land of having all our shit organized and easily findable and summarizable, et cetera, using if not quite ChatGPT-3 soon.

Dan: Yeah, let me take you through it a little bit. I think the premise of that article is, yeah, I think large language models solve a lot of the organization problems especially for notes that we’ve been running into for many, many, many years. And I think in order to understand why, you have to understand why organizing notes is really hard. And the key thing in my opinion that makes organizing notes really hard is it’s about how you organize in general. So things that are easy to organize are things that you know how you’re going to use them and when you’re going to use them in the future. You can create an organizational system for that pretty easily. A really good example is people at companies don’t really argue about how to organize their customer list, they just use a CRM.

Everyone just has a CRM, maybe they have some slight customizations about how it works, but it’s a fairly standard piece of software that everyone uses to organize their customers. Other examples abound, but four things that fit into it like a really well-defined process and you know where you’re going to use them organizing is already pretty much solved. The problem with notes is that a thing becomes a note when you actually don’t really know when you’re going to use it in the future, or you might use it for many things. And that means that there’s infinitely many ways to organize it, and you might want to organize it one way for one project and you might want to organize it in a totally different way for another project. And you really want it to come up for you when it’s relevant, but you don’t know when that is.

And that becomes sort of this intractable problem and there’s thousands and thousands of systems that purport to help you with that that all work in various ways, but don’t totally solve it. And I think large language models really change the equation there because they don’t require you. They can A, create flexible organization systems on the fly that change based on your needs so you don’t actually have to do the organizing yourself. For example, they can create automatic tag hierarchies, they can create folder hierarchies, they can link notes together, all that kind of stuff. So you’re not doing the manual work to be like, where does this go or where do I put it? It can just put it in all the places that you might need it, and it can create new hierarchies all the time based on what your needs are. I think large language models can also really be good at synthesizing and enriching the notes that you’re taking.

So as you’re taking notes in a meeting, it can go in there and fill in things that you didn’t put or find other notes to put together into like a hey, I found a pattern in the same way that my journal entry software did that for patterns in my journal entries. So I think large language models can be sitting on top of your notes and finding patterns in the things that you’re writing about and surfacing them to you all the time, which I think is quite useful. And then I think the big thing that large language models do, the really big unlock is they can be a copilot for you as you’re writing to bring all of your notes to bear on your writing all at once every time you touch your keyboard. And I think that’s actually the key thing because when you’re writing stuff, for example, I often I’ll write down a quote or an idea, and the reason I want to write all that stuff down is because as I’m writing, I write essays or blog posts, I want to be able to use that in the things that I produce.

And one of the cool things about large language models is as I’m writing it might read what I’m writing and suggest a quote from a book that I read a long time ago and took notes on and just plunk that note right at the end of my cursor and tell me, hey, do you want to use this or not? And it’s pretty easy for me to be like, no, no, no, that doesn’t fit what I’m writing about, but it’s also a pretty magical moment if it does fit and I wouldn’t have thought of it before. And we already have this type of experience for programming, so GitHub Copilot, for example, is doing this already for code samples and it’s tremendously valuable. I wouldn’t want a program without it, and I think something similar is coming for all the personal notes and personal knowledge that we’re taking where as you’re writing it’s just finding exactly pieces of information that you’ve saved that are relevant, and then transforming it into a form that would work for the context that you’re putting it into.

I think that’s a really key thing is for a long time now we’ve had pieces of software that as you’re writing maybe it can suggest here four notes that are relevant to you on the sidebar or whatever. I don’t think that works very well, and the reason is because it’s very expensive cognitively and in terms of time as I’m writing for me to see those notes pop up and then go click them and be like, oh, is this relevant? And if so, why? It’s not something I want to do in the context of writing. It’s actually I think a distraction and it’s very expensive, but for these technologies what these technologies enable you to do is instead of being like, hey, here’s a note that you can go read, it’s like, no, no, here’s the next piece of text you want to write using the notes that you’ve already taken. And I think that is the big unlock that’s incredibly valuable.

Jim: Yeah, I can see so many ways this could pull, and what’s interesting is everybody has different styles of work. So one solution isn’t going to fit everybody. For instance when I was talking my own ash and trash method of organizing information, I forgot to mention the main one, which is Gmail, right? I’ve got 22 email accounts that get consolidated into my main Gmail account. I just checked it was 141,000 messages in my inbox. And truthfully, the most common way I find something is just search Gmail and it sort of works. But I’m just trying to find something this morning, I know it’s in there, I just couldn’t find it. Couldn’t find the right query. And unfortunately this is now the particular problem I was asking the question about was there’s so many near misses that it was just impossible. So as you were talking I was thinking out loud, what would be cool for me would be for it to have all my emails and all my notes, all my publications, all my social media things.

‘Cause one of the things I fairly often do is I will have gotten some clever thing I said on Twitter or something, and when I want to write something I want to go find it and steal that in the moment perfect sentence that I created on Twitter. And what I’d love would be to have two screens, one that I’m writing and a second screen that’s navigating in some kind of semantic space. And I don’t want to actually see the raw nodes, as you say that’s overload, but I would love it to sense that from what I’ve written so far what are some of the closest nodes in latent semantic space that are relevant? And with a description, say a two sentence description. And then you’d be able to click into it and then it would recursively break down into sub spaces with labels when you’re just kind of struck like that. Something like that for me would be… I mean, I would pay $100 month, no questions asked. I’d probably pay a $1,000 a month for that.

Dan: I think that would be incredibly cool.

Jim: And it strikes me we got to be within striking distance of that now.

Dan: It’s very close. You could spend a couple weekends and have something that is useful in that way and it wouldn’t be that hard to build.

Jim: Yeah. And I’ll give an example of something somebody did for me that uses some of the… Again, he did it in a few hours is amazing. A guy named Stephen Reid. It’s Stephen with PH, S-T-E-P-H-E-N-R-E-I-D.net/k/jim. He took all the transcripts from this podcast and he ran it through some scheme of finding topics and I think he came up with 125, something like that. He then went and created a knowledge graph connecting all the episodes through the nodes of topics, but this is where it gets to generative AI to make it much more valuable. For each of the one or two word topics that he discovered, he had ChatGPT write a description of that topic. And these are mini essays, these are like three or four paragraphs on each one.

Dan: That’s cool.

Jim: It’s really amazing. So you’re clicking through this thing, okay, well, what is stochastic annealing? Or what is complexity science? Or what is Game B even? And it was amazing. And he told me, I asked him how he did it, he said, yeah, he wrote a template and he did some just very simple fully automated things including a prime and got ChatGPT to write definitions or GPT-3, it was actually through the API. And he said that of the hundred and some essays that it generated, he only had to fix three of them. It was amazing, right?

Dan: That’s incredible.

Jim: And the guy’s not even that much of a techie. I mean, he’s a techie, but he’s not an AI guy. He’s a Ruby guy. He’s a good, very talented Ruby guy and so he was able to kind of glue wear this stuff together in just a remarkably short period of time. And it’s really been fun to play with. And so this is just again like PCs, more or less amateurs are able to produce these amazing results.

Dan: I wasn’t around for the first PC wave, but the first wave that I remember that I can compare it to is mobile. When I was in middle school and high school, I started programming mobile apps. Even before the iPhone came out I was programming apps for Blackberry, and that was super cool ’cause no one had really thought of doing software for phones. But until really Blackberry started to get popular, phones weren’t advanced enough to make software worth it, or at least from third party developers. But Blackberry started and there were a couple other ones, I think Nokia and a couple other ones had third party developer programs. And so you started to be able to create these cool little apps and there was so much low hanging fruit ’cause no one had ever done it before, and then the iPhone came out and it was like an explosion overnight.

I don’t know if you remember this, but just everyone was making an app, everyone had an app, everyone wanted to make one, and you could just spend a couple weekends and have a really cool app business that did something useful for people. And over time those ecosystems have gotten a lot more mature and it’s a little harder to do something interesting in a weekend with an app because all the low-hanging fruit has been picked, but I think generative AI is a sort of new frontier where, yeah, it’s possible again. Things are exciting again, you can tinker and build stuff and that’s just really fun.

Jim: The other Cambrian explosion I remembered was when the graphic browser first came out, particularly the one that was the Netscape browser particular. Marc Andreessen’s browser that he wrote as a graduate student at the University of Illinois. And before that the web existed, but it was kind of this text thing, it was links, right?

Dan: Yeah.

Jim: And then he wrote this quite nice graphic representation of HTML, and what people were able to figure out how to do with HTML relatively rapidly were totally amazing, way beyond what HTML was designed for. I still shake my head that the whole world’s built on HTTP and HTML, two of the worst protocols ever designed by human beings, but very, very quickly things took off and one of the things was the brilliance of Mark adding to the browser show source. So in those days, just even getting a static page up to do from scratch was pretty hard. You found one you liked? You show source, cut and paste, change the words, change the colors. Oh, I got my own page. Great. And it was kind of very, very viral, very, very quickly-

Dan: That’s funny.

Jim: As the combination of HTTP, HTML suddenly was transformed by the show source capacity in Andreessen’s browser.

Dan: I love that. I hadn’t thought of that as being a design decision, but it’s obviously a decision someone had to make and I think it’s brilliant. It makes the web what it is. There isn’t necessarily an equivalent here because I think a lot of people think of their prompts as being proprietary. A lot of people like to share prompts, so I think it’d be interesting to think about what the equivalent functionality would be in this ecosystem. But yeah, I think you’re right and that that’s such a cool anecdote.

Jim: It really is. Now, just talking about that one of the foundations of evolutionary theory that whenever there’s a top predator you’ve just opened a niche for someone to eat that top predator. And that niche often doesn’t get filled for quite a while, I mean, fill the niche to eat a grizzly bear? That may never get filled, but there’s a theoretical niche now for something to eat grizzly bears. And I was just thinking as you were saying that, that prompts people keep them proprietary, well, obviously there’s going to be a meta tool for creating prompts for you.

Dan: I think that there’s a lot to be done here with not just prompts, but what I would probably call just data pipelines in general. It’s you start with a bunch of texts, and then you want to go step by step to splitting the text up and transforming it with a prompt and then transforming it again with another prompt and finally having your output. Right now a lot of that’s being done in code, but I think a lot of these sorts of GPT-3 tasks are very similar. It’s like how do you take a bunch of texts and then get an answer or an output that you want from it? And I think there’s a lot of room for a more visual almost like notion or no code type way to look at that and see the outputs and explore the space of outputs that you can possibly have. And yeah, I think there are going to be tools for making that more accessible and then sharing your results better so I’m very excited for that.

Jim: Yeah. And you mentioned a tool in passing that I had just stumbled across on Twitter a couple weeks ago and looked at their GitHub page, but haven’t actually tried it out and that was LangChain. Do you have any hands-on experience with LangChain?

Dan: I do, and I have less experience than some of the other stuff, but I think LangChain is really interesting. And the general gist of it is current large language model APIs are sort of built for one-off calls, so you take some texts and you send it to OpenAI and then you get it back. But really often what you want to do is you want to do a bunch of different LLM or other type of model operations in succession. And so what LangChain does is it allows you to create what’s called a chain where you start with a bunch of texts, and then you can define a chain of prompts and other functions that will take your text, maybe it reformats it and sends it to OpenAI, takes the result and then sends it back to OpenAI with a new prompt and then takes the result and searches the web and then it send it back to OpenAI, all that kind of stuff. It chains things together so you can create much richer LLM applications with less coding.

Jim: That sounds like the bomb. I think I’m going to build the Rutt chatbot with that. And it’s a Python library, right?

Dan: Yeah, it is.

Jim: Good. Good. [inaudible 00:56:43] learn another goddamn language, right? My fingers know Python fairly decently now. It’s not my top language, but I can get shit done in Python and if I had to learn some goddamn new language, I’d go [inaudible 00:56:55] I’m not sure enough about that. All right. Well, this has been very cool. So now maybe this will be our last vignette before we wrap up here. You mentioned a VC, I’m not sure I’d know how to pronounce his name, Yohei Nakajima maybe, something like that?

Dan: Yohei, yeah.

Jim: Yohei Nakajima. And you laid out a whole list of things that he has used these kinds of tools to do, and that he’s apparently a guy famous for not wanting to waste his time on repetitive bullshit so he’s willing to invest in automating stuff. Maybe you could just sort of rattle off relatively quickly some of the things that he has done.

Dan: Yeah, he’s a cool guy. As you said, he’s a venture capitalist and sometimes for a chain of thought I do interviews with people that are doing cool stuff in AI and yeah, he’s just someone who hates repetitive tasks and he’s been trying to use large language models to eliminate as many repetitive tasks from his day as possible. So one of his first ones is he calls it Mini-Yohei, and it’s basically a chatbot that replicates him and what he knows so that founders in his portfolio can ask questions and then it will respond. And it sort of eliminates I think some of the more repetitive questions that VCs might get, but one of the interesting things is when it responds it loops him in and he can follow up, and then he can also mark which of its responses are better and which of its responses are worse so over time it gets better, which I think is pretty cool.

Another one that he has that I actually think is useful and I really want is he has a bot running on his email where it summarizes all the email interactions he has with people. So if he’s about to go into a meeting with someone, he can pull up a dossier that’s like, here’s every email conversation you’ve had summarized, and here is the latest thing that you talked about last time you were talking to this person. Which I think is incredibly useful, and then he has a bunch of other ones. He has a tool that summarizes tech name and sends him a digest every day so he doesn’t have to read tech name. He has the same thing for his kids’ school, so he has a bot that reads all of the emails from his kids’ schools, and then it creates a searchable interface for him to see, oh yeah, when was that? When’s that event? Or What do I have to do by this date?

And he can just search and it’ll just tell him the answer which I think is quite cool. So lots and lots of different applications for this technology to save little bits of time during the day.

Jim: Yeah, I mean, I can again see how that would be so good. And the other thing I do, I probably do 15 or 20 Zoom calls a week, and you can now hook them up to Otter and get transcripts made. Is Zoom actually doing transcripts? I don’t know. I would love to have all those transcripts available and then link to the emails with the people.

Dan: Coming soon.

Jim: I mean, goddamn come on guys. Come on entrepreneurs out there do this stuff. I’m kind of too set in my ways, too old, too rich, too lazy to go chase these things myself, but come on you young guys go out there and do this stuff. This is a wide open frontier that’s really just opened up here in the last year or so. And these examples that we’ve talked about, these are really useful things.

Dan: I agree. I agree. I think it’s super exciting. It’s a fun time for anyone that likes to tinker, to build stuff, to play, and it’s still really early so there’s tons and tons and tons of opportunity that just requires energy and excitement and a little bit of programming knowledge.

Jim: Yep. Well, I want to thank Dan Shipper for a really wonderful conversation here today, which I think brings this idea to life that this stuff is for real. There are actual real things you can do with it right now that aren’t even all that hard. And like these other Cambrian explosions, PCs, then PC software, then websites, mobile apps, now there’ll be another Cambrian explosion, and somebody out there listening to Jim Rutt Show is going to build the next Unicorn so get to work folks.

Dan: Amazing. Thank you so much for having me.

Jim: Yeah, this has been a wonderful conversation. Thanks very much for the work you do because I’m actually going to steal some of it as I try to figure out how to do the Jim Rutt chat. And just for your enthusiasm and your good common sense.

Dan: Amazing.

Jim: All right. Thanks.

Dan: Thank you.