The following is a rough transcript which has not been revised by The Jim Rutt Show or Jamie Joyce. Please check with us before using any quotations from this transcript. Thank you.
Jim: Today’s guest is Jamie Joyce from the Society Library. She’s got a quite interesting resume in nonprofit work, and you can get to it at jamiejoyce.com. One of the things I really liked about her resume, a little different than some, is she puts some, I don’t, you might call them themes at the very top, which I thought was quite interesting.
She talked about my activism, human rights advocate for freedom of thought. Now there’s a good idea, and something that’s in mighty short supply from all sides in our current social shit show. And then my archival work, making all arguments claims, evidence related to sociopolitical issues accessible. So we make more informed, inclusive, and unbiased decisions. What a great concept there, now to see if she can really do it here. And this is the one I thought was most interesting and kind of like how does this fit with the other two? My artistry, I manipulate the irradiation of 2.5 mega volt particle accelerator beams, yep, that’s right, to make lightning bolt emojis in acrylic, and I make wooden books. Okay. Just a few seconds about lightning bolts in emojis.
Jamie: Me?
Jim: Yeah.
Jamie: I mean, so people may have seen some of this kind of work on Instagram or TikTok, where essentially you can take a microwave transformer or a neon transformer and manipulate it so that you can discharge electricity on wood. It creates this wooden burnt fractal shape called Lichtenberg figures. And I was making this art pieces for gifts, essentially, for friends and family. And a photographer friend of mine said, “I really want to capture this.” So he took a bunch of photos of me making these Lichtenberg figures on wood, and he published them on the internet.
And then someone left a comment on one of these pieces, and they were making a joke about the machine that I used, because I laser engraved, do not touch this machine, it’ll kill you. And he made an OSHA remark about that. And I looked up who this person was. And it turns out he was a particle physicist. He works at Fermilab. And he and a group of his physicist buddies actually go to a non-Fermilab particle accelerator and make Lichtenberg figures with 2.5 million volts of electricity using an electron accelerator beam. And so they invited me to join their little crew. And I started making art an acrylic medium with them. And it’s one of my favorite forms of art. It’s quite expensive endeavor. So that’s a little bit about that.
Jim: That’s kind of interesting. But today we’re going to talk mostly about the Society Library and related projects. And Jamie was the founder of the Society Library, as I understand it. And it is today the executive director. Let’s start out with the names. It’s kind of an interesting and curious name. Society Library. I remember somebody telling me, “You ought to check out the Society Library.” That’s all they told me. I go, “What the hell is that?” And we did, and Jamie and I had a few conversations. In fact, one of these days we’re going to work on a project. I had a project I was interested in doing, and she was interested in doing it, but I couldn’t really do it at that time. And then later when I could do it, they were busy. So one of these days going to make something happen here. But anyway, tell us where’d that name come from, and what does it signify to you?
Jamie: Society Library is supposed to represent this concept of society having a library of its own ideas. We have the internet, but it’s pretty unstructured. The way in which we search through content is not really in a linked way. So if I search for a specific claim on Google, it may bring up a bunch of news articles or websites where this claim lives, but it’s not actually a linked database. So the Society Library’s mission, we do many different things, but in building our libraries, what we do is we collect arguments, claims, evidence, sentiments, opinions from a variety of different forms of media, and then we create linked knowledge databases.
And the ontological structure of the library itself that we’ve chosen is deliberation. So we choose how this data’s going to be linked. It’s not going to be just hyperlinked. So we want to actually link arguments and claims together so that they support larger propositions in attempting to answer wickedly complex questions. So basically what this resulted in is the mapping of debates about societal issues, but really Society Library is just a library of society’s ideas, ideologies, and points of view.
Jim: And I know you sort of keep them separate into libraries and debate mapping. Why don’t you talk about those two concepts and how they’re interrelated?
Jamie: I mean, they’re very similar because the debate maps is the ontological structure of the data itself. But the libraries are meant to evolve in a slightly different direction, even though we still rely on this deliberative ontological structure. What we’re also trying to do with the Society Library and what our infrastructure already supports is to be able to link the multimedia expressions of a concept into one node.
So there’s like, let’s say, one claim, nuclear energy is the safest form of energy production. Not only can you express that in language and through multiple linguistic registers. So you can maybe say this at a fourth grade reading level, or you could be much more precise and sophisticated in how you express that. But this type of concept is also expressed on podcasts, like I just did right now, but also in videos and in images and graphics and that sort of thing.
So the library itself is moving in a direction of how can we start creating a new interface, so that people can browse knowledge in whatever multimedia format they want at differing levels of linguistic register and sophistication that they want? While the purpose of all of our work with creating debate maps is really to create a suite of methodologies and processes, so that we can collectively inquire into truth as a society. Which is more along the lines of how do we break down and link and structure this data for a specific end purpose, which is to answer questions.
Jim: Interesting. So in the library, you create a series of nodes, presumably networked in some fashion, right?
Jamie: Yes.
Jim: And then from each node you associate various pieces of content in various styles.
Jamie: Yes, exactly. So we already have this in our visual database software. You can open a node and you can see the various ways in which you can express it. You can see where it came from. If it was derived from a quote, you can see that quote. You can see the backed up URLs in the internet archive, and then all the media artifacts that we associate with it.
Jim: Now, if you wanted to be comprehensive, that would be a monstrous job.
Jamie: It would be monstrous if we were comprehensive, but that would require a lot more automation than we currently have access to.
Jim: I mean, you’d have to be Google to do something like that, right?
Jamie: Totally. Totally. But as long as we start meeting a bare minimum threshold of just offering a diverse way in which people can interact with the content, then it serves our educational purpose. Because given, especially, people’s varying attention spans and varying levels of subject matter familiarity, a more simple way of expressing the content, or a more visual way of expressing the content is going to appeal to different users or readers of the library than maybe a more sophisticated audience who’s going to want to look at the debate maps themselves and see how arguments break down into claims and see how the claims are supported by evidence and that sort of thing.
Jim: Now, this kind of work obviously begs the question, are these people biased? And if you think about just the most recent disclosures on Twitter, as we were recording this a couple days ago, Matt Taibbi has been publishing various things from Twitter’s files that seem to be indicating that maybe Twitter is putting the thumbs on the scales in some of the decisions they made about who to kick off and what topics to allow, et cetera. And more and more we have those questions.
And as I was looking around on your website, I was very pleased to see that you had a quite serious section on virtues and values. And talk to me a little bit about, one, let’s start just with your theory on how you can remain a trusted third party, and we can assume you’re not putting your thumbs on the scales. Then we’ll go down the list of virtues and values, which I thought were actually quite interesting.
Jamie: So the virtues and values is just one way in which we work to try to overcome our own human limitations and biases. And that kind of relates to culture. So we’ve got our virtues and values that have to do with how we relate to the work that we’re doing, how we look at information and its relationship to people. But we also have 22 different methods to overcome our own biases in the actual data structuring and research work. So we even borrowed some techniques from A CIA Tradecraft Primer about doing devil’s advocacy work. So when we find a claim, we invert the claim to its exact opposite, and then we try to steel man that as much as possible. So we have a bunch of techniques and we’ve got all these search engines that we’ve built that pull from diverse sources that have been identified as likely politically leaning one way or the other, to diversify our input.
There’s a number of people who are looking at the same content. So we have lots of methods and processes to overcome our biases. But then there’s culture and beyond just the virtues and values, if you go to our website, you can also see there’s a number of knowledge policies. And these knowledge policies outline how we think about a lot of the wicked issues with knowledge management. And in addition to that, organizationally and infrastructurally, we try to be a trusted third party by being a platinum star transparent nonprofit organization. If I’m not mistaken, the last time I checked, only two percent of US nonprofits have that level of transparency recorded with GuideStar. So we’re doing everything we can.
In the future, I think, as we fundraise and grow, we can make it even more so embedded in our methodology that multiple people are working independently of each other, examining the same content, and then we can see if there’s a statistical consensus of many people saying, “Okay, yes, we’re labeling this data as this,” and if there’s a difference in how people are labeling something, if we fail to have intercoder reliability, that may indicate bias, or someone is seeing something that someone else isn’t seeing or someone is intentionally trying to corrupt it. So we aim to be more rigorous over time. But basically it’s in methods organizationally, and then that culture.
And people are worried about bias. We are too, our goal is not to convince anyone of anything or to drive them to believing a specific outcome. Because, again, we’re really trying to promote this human right, which I believe is article either 18 or 19. There’s two human rights article 18 and 19 that have to do with freedom of thought. One is more religious and one is a little bit more political. But human beings have the right to believe whatever it is that they want to believe, even if some may consider that wrong or not as helpful as their ideologies, people just inherently have that right. So we’re really standing in that.
And what’s wonderful about the Society Library being a library is that we are picking up and continuing on a tradition of librarianship in the United States, which is also very much rooted in anti-censorship, being radically inclusive of examining all beliefs and people having the freedom to interact and read and consume whatever information that they want. In the United States, and in this tradition has been brought to libraries all across the world, there’s this belief that enlightenment is achieved through access to information. And no one individual possesses, or even institution, possesses the wisdom to discern what information is going to be better for people to consume.
So it’s just kind of make all the information available, empower people to consume more, not less. And that’ll lead to even broader, more complex understandings of things as opposed to trying to control what people consume. So we have a lot of different things in our favor, a lot of structures that we’ve built up around us. And the Society Library does hire librarians. So we do hire, essentially, what I believe to be completely underrated data scientists and analysts, which are librarians. Because they’re trained to organize and consume information and label it and put it in a databases and all these things, they’re trained with that culture in mind as well. So that’s all very helpful to us trying to operate with the ultimate integrity.
Jim: That’s just lovely. And it’s got to be hard in the current environment where you have the wokes and the cancel cultures on one side and that don’t say gays on the other. And it seems like the grand enlightenment idea of the marketplace of ideas is under challenge from multiple directions right now.
Jamie: Absolutely. And that’s one of the reasons why we actually created our policies. So I have been dragged on Twitter. We used to say this thing at the Society Library, “Where we support,” what was it? It was “equal inquiry into all points of view.” And we had to change that to equal scrutiny of all points of view in terms of what we’re enabling to our readers, because a lot of people were getting concerned that we were just platforming a bunch of ideas that would be harmful. We’d be doing more harm than good. And so we have this policy about platforming and about people worried about just by putting two ideas on the same platform, you’re giving undue weight to one, which may not otherwise be given weight in different contexts.
But of course, the Society Library is all about contextualizing information deeply. We’re not just collecting all of these points from all different parties and putting them in one place and saying, “Okay, they’re all equal.” Everything is contextualized by pro and argumentation and whether or not something has evidence that we’ve been able to discover. And we have a whole series of labels where we indicate we were not able to find any evidence for this, or this has yet to be fact checked.
So we try to signal to readers, we put work into this and we didn’t find anything. Or we just do the work of connecting evidence and steel manning arguments, or we let them know, “This is too vague or large of a task. We don’t have the resources to fact-check this,” just to give people little clues about how much weight they should give this piece of content in consideration of it. But we also never identify something as true or not true. We certainly don’t go to that extent. We just indicate the work that we’ve put into making each and every idea in the database the best representation of itself as possible.
Jim: And you say some of the Twitter hound dogs didn’t like that?
Jamie: Yes. Yes. One of the reasons why we wrote the policy is just so you can’t argue about this stuff on Twitter with the number of characters, it’s very hard to get that across. So by having our policies, some of them are just very short essays, but many paragraphs explaining the nuances of how we’re navigating this wicked issue, and there’s a variety of wicked issues that we’re addressing, just so we can link to it. So when someone’s like, “You’re equally platforming all of these ideas, and that’s going to create false equivalence and yada yada.” And it’s like, “We have a policy about this, if you’d like to read about it, here you go.”
And all of our policies exist as Google Docs. So if anyone has any criticisms or they want to refine our thinking on it, anyone is more than welcome to leave a comment on it and we’ll see it. And we’ll think about it, because we care so much about enabling enlightenment by enabling access to all these ideas, for the purpose of people being able to scrutinize all these ideas and come to their own informed belief about something willfully.
Which may not otherwise be possible in the information environment that we are in, because people may not have the opportunity to see, hear are all the different points of view that I could examine on this issue, just because they may have to spend 10,000 hours doing that work. And a lot of people just aren’t up to doing that work.
Jim: And, also, most of our platforms and institutions are now captured by somebody or have fairly strong biases. Even academia has now been much more, not fully captured, there’s definitely some still solid islands out there, but fair bits, whole academic departments are captured by ideology these days, which is a scary thought. And having an island where you have rigorously built processes to avoid that is actually very hopeful to my mind.
Let’s briefly go over your virtues and values. I also just want to commend you for having the nerve to say virtues and values in our world today. Some people say, “Oh, that’s a reactionary concept.” I sometimes, I fairly often talk about virtues and values.
Jamie: Well, I certainly don’t have a better word for it. So I think virtues and values is dead on.
Jim: It’s a perfect name for it. Don’t let the post monitors get after you, though, they’re not going to like that. But my answer to that is, fuck them, basically. First one, interestingly, it’s the one that I always used as the core value in my companies, the core, the number one foundation, intellectual honesty. “The truth is the truth, regardless of who says it,” is the words that you used.
Jamie: Yep.
Jim: You want to put a little bit more gloss on that?
Jamie: Yeah, sure. I Mean, so at the Society Library, we somewhat see knowledge as both distinct and not distinct from the human beings that express them. Sometimes it matters who says something, if we have to rely on their reputation as the only evidence that exists that something may be true. But oftentimes, an idea exists unto itself. And so we are very careful not to really judge where a piece of content comes from. It could be a conspiratorial thought leader, it could be from a scholarly article. All things as we ingest them should be scrutinized and interrogated for their potential truth. And we shouldn’t just assume just because someone’s a popular thought leader or it’s like a small obscure blog or something like that, that it may not actually contain extremely valuable information.
And this we’ve actually seen again and again, and I put the language of a conspiratorial thought leader, I’m not sure my exact words, a conspiratorial website I think is what I say, because we’ve seen this happen. We’ve seen Alex Jones, for example, express something, and right off the bat, a lot of people may say, “oh, this is Alex Jones, not to be trusted. I don’t believe it. It’s nonsense.” And yet, if you just do a little bit of steel manning and a little bit of digging, maybe he, to a certain extent, didn’t get it all incredibly accurate. He misstated something, but that’s could be an indicator that there’s truth there.
So it’s really important not to discount knowledge on its face, because it may have memed and mutated and changed and reached us, and therefore has changed. But it may be an indicator that there’s some real value, there’s some real truthful knowledge out there that we need to go out and find. So we need to be really honest about accepting information as it is, and then doing the work of creating the best representation of it. And not just assuming that we would know from the onset that, oh, this is not accurate because of where it comes from.
Jim: I mean, that’s literally the classic argument ad hominem, right?
Jamie: Yes.
Jim: It doesn’t actually mean just saying bad things about people. It means saying that you can judge the argument by the person. And to your point, if Alex Jones says something, I’m going to be a bit skeptical just because of his track record as a proven liar, but it doesn’t mean he lies all the time, right?
Jamie: Right.
Jim: And the old folks saying, “Even a broken clock is right twice a day.”
Jamie: Twice a day. Yeah.
Jim: I remember a podcast I did with somebody who had this disease in a very extreme form. We were talking about a form of monetary economics, and I suggested that the Nazis had done a very good job with this, in their first three or four years in power. And he was like, “The Nazis can’t possibly ever do anything that’s good or useful, and you should never, ever look at Nazis for examples of anything.” And I quoted the book, I said, “Go read this book. Actually, they did very, very innovative things in the way they operated their monetary system from 1933 to 1937.”
And I told him who it was, and I gave them all the background. And then, “Well, you can’t say that.” Wait, what kind of disease of the mind does this person have? Nazis, very bad. Agreed, we killed them, but doesn’t mean everything they did was stupid, right?
Jamie: Yeah.
Jim: And there’s a classic example. Now, this was a very bright guy, but he just could not get it through his head. It was weird. It was talking to an alien from another planet.
Jamie: Well, and this actually ties into the second virtue and value, which is intellectual independence.
Jim: Let’s go there.
Jamie: And so intellectual independence means be not in possession of ideas that possess you. So essentially this ties into the culture of the Society Library where we as analysts are looking to serve the public. And therefore, in order to do so, we strive to not be captured by certain ideologies and opinions. In order to enable all of these other virtues and values to seep into our work, as much as we possibly can, we have to at least strive to be intellectually independent.
So the last line as we’re describing that virtue and value is detach yourself from opinions. If you have one, it has you. And sometimes when you’re talking to people, you can tell that they’re just, maybe they haven’t really thought about this opinion, maybe they have, maybe it’s coming from a trauma or an experience, but they’re just really motivated to just put this idea out into the world. And anything that may question it, they’re really not actually free to entertain that what they’re believing may be wrong. And so we strive culturally and in our work to be independent of that kind of force and capture of ideas themselves. Because if we’re trying to equally scrutinize all ideas, you need to have an independent thought process for that.
Jim: I think the common name for that is motivated reasoning, right?
Jamie: Mm-hmm.
Jim: You see it all the time on the internet, people argue, you see, it’s very clearly from their ideology rather than the facts or logic or reason or anything else. And it’s one of the logical fallacies to avoid. This next one’s another one of my favorites, intellectual inclusion.
Jamie: Let no intellectual stone be left unturned. So these are all closely tied together, but basically this informs our methods. So this virtue and value is related to our methodology, and it is promoting us to be radically inclusive of all the different points of view that are being expressed about a specific issue. So when we’re creating these library collections, or we’re mapping a specific debate about something, we really work hard to make sure that we are not missing anything.
And we have techniques for doing this. So I mentioned the devil’s advocacy work. Not only is that to keep us from biasing the content by focusing too much on a pro and not a con, but it also helps us unpack the space. So when we’re building these databases, one of the first steps we take is to do this, we call it a topic flyover. So we look at an issue, it could be nuclear energy, which I’m going to use that as an example a lot, because we just finished a collection about that, and we find all the highest level topics.
So people are concerned about waste and safety and the economics and how long it takes to build these reactors and all these things. And we go through this process of collecting materials, deconstructing them, finding subtopics, then creating a set of materials on all the subtopics, finding further subtopics, and just slowly unpacking the space. And so when it comes to intellectual inclusion, we’re trying to unpack and cover as much of the space as possible, while also not prescribing what should be included. So if we, again, encounter a piece of information that we find to be ridiculous, we are resource constrained. So there’s going to be some issues that there’s just more to say about it.
There’s been a lot of argumentation about nuclear waste, for example, than maybe something a little bit more obscure. But it’s so important to be radically inclusive, because if this is a library of society’s ideas, then it needs to represent the full scope of what people believe. Which to a certain extent could be arguably infinite, and we don’t have infinite resources to cover all of that. But this virtue is about making sure we are as radically comprehensive and inclusive as possible, because there may be some tiny obscure little piece of information that is actually going to make a huge difference.
When we worked on COVID-19, we got a very tiny grant. We were not able to finish the collection, because we got a one-time, very small grant to just seed the collection, but we were picking up on hydroxychloroquine and ivermectin back in March of 2020. So right as the pandemic was picking up, these were tiny little ideas at the time. Eventually they’d become extremely culturally and politically relevant in the United States, and we didn’t have the resources to keep tracking those things.
But because we were super inclusive, we picked up on these trends really early on, that eventually became major topics. So we work to be inclusive in that way, so we don’t miss anything, because we’ve even proven to ourselves that when you’re really inclusive, you may actually be picking up on something that’s going to be really important later.
Jim: Now you do point out that there’s literally an, well, not quite literally, but extremely large number of possible ideas and formulations. At some point you do have to be pragmatic. So for instance, suppose you were doing a database on geography, or let’s say the Earth as a topic. How do you handle something like flat Earthers, right?
Jamie: Yeah.
Jim: Well, which, scarily, there’s a lot of those running around loose at the moment. Do you just say, “Wait a minute, that’s just too absurd to even include?” Or do you include it and say, “Here’s the theory of flat Earth, here’s the steel man blah, blah. And here’s evidence on the other side.” How did you deal with flat Earthers in a database about the Earth?
Jamie: We would totally include flat earth deliberation, absolutely. Well, this actually relates to some of the other virtues, which we haven’t mentioned yet. But we would absolutely be inclusive of that. Because people have these ideas, and for something like flat Earth, that’s a popular enough idea. It’s certainly not a large number of people I don’t think who subscribe to that set of hypotheses, but it’s a fair number that it’s gotten media attention. So it may be worth the resources to collect their arguments, collect their claims, and then put that in the context of counter-arguments and counterclaim.
And then instead of this conversation having to happen over and over and over and over again, and between people who may not be prepared to engage with all of the hypotheses and ideas that someone who is coming from that point of view could propose, instead, it’s like it all exists in a library for everyone. So if anyone even just has a curiosity about what people think about flat Earth, or if someone from the flat Earth community really wants to see, well, what are the counter-arguments to the ideas that I hold? Or I just want to make sure all my ideas are represented in this database. It is a library reflective of society, and we would absolutely include something like that.
Jim: Okay. Well, that’s great. Let’s go on to your next virtue value, earnest service. I love the word earnest. Actually, one of my pet minor campaigns these days is against irony. It seems like irony is the stance that so many people take, and it’s so goddamn annoying and trying to figure out what they actually mean. In fact, I did a recent podcast with Nora Bateson. Her and I were both somehow had a little Twitter stream about it, and we both agree that we’re tired of irony. And we had a podcast episode called The Move to Earnestness or some such thing. Earnest service, what does that mean in your mind?
Jamie: Yeah, it means we do not exist to change what people think. We exist to change the context in which people think. Let’s use the flat Earth example, the goal of the Society Library is not to create these collections to make people who do not subscribe to flat Earth believe that flat Earth is a thing, nor vice versa. We just aim to serve people by doing the work of creating these really high integrity collections where they can view all these different points of view. So they don’t have to spend thousands of hours arguing in unstructured ways on the internet or doing a bunch of independent research.
The purpose of the library is to collect and organize information of public interest and make it available for public use. And so we take that very seriously. But we’re in a digital age, so we’re dealing with multimedia formats, and we’re not dealing with things necessarily confined book by book or artifact by artifact, but we really go down to the claim by claim level. And so we just really aim to do that very earnestly.
And one thing that we recognize is it’s almost to a certain extent, futile in terms of our grand vision. There’s so much work to be done. And because we don’t exist to enable a specific outcome in terms of minds changed, we do this almost out of just the importance of the work itself. Now, there are ways in which we can measure if the work that we’re doing is having a pro-social impact, does it decrease polarization and aggravated attitudes as people are viewing different points of view? As opposed to them maybe having a conversation or a Twitter argument with someone who is their ideological opponent.
Maybe viewing the ideological opposites on this platform is going to not be as aggravating as interacting with another person. So we can see if we have pro-social outcomes, which is something that we’re very interested in measuring, but we’re not trying to change anything. And so it’s just like we’re taking this humble approach of we’re here to serve and provide this information, and maybe it’ll make a difference in how people think about something. Maybe it won’t, but that’s not our job is to change how people think.
Jim: A little thought just hit my mind, how would you compare and contrast what you’re doing with let’s say Wikipedia?
Jamie: So first of all, let me just say I love Wikipedia. I edit on Wikipedia. I have their swag. I’ve been a donor for a really long time, but Wikipedia is the encyclopedia of the internet. So they have to make a bunch of editorial decisions about what’s going to fit on that single page. And they have a bunch of rules about what is worthy of weight. So they do not include ideas of undo weight, and they do care about potentially platforming ideas, which may end up being false. And the structure of how they organize content is very different than how we structure content. So we structure content in an argumentative format. So there’s positions, there’s arguments that support or refute whether that position may be true or not true. Those arguments break down to claims. Every single claim can have any number of pro and con arguments supporting whether that claim in an argument may be true or not true.
So it’s like we’re much more of a knowledge graph, and they’re much more of a digitized encyclopedia. They literally have a page per topic. It’s cooled that they’re hyperlinked. I think that they do a really good job on scientific subjects and historical subjects too. But when it comes to highly politicized and polarizing topics, wiki wars are definitely a thing. I almost got put into a Wikipedia timeout, because I was trying to, in my opinion, de-bias a page, but a topic was kind of bubbling up to be of extreme cultural relevance in a moment where people are trying to make sense of it. So they’re going to the Wikipedia page to see what Wikipedia thought about this. And I noticed that the page was overly biased.
And to me, it was just kind of obvious that Wikipedians, and I think they were being earnest, and I think it came from good faith, but I think the Society Library has a bit more epistemological depth than Wikipedia in terms of how confident we are that something may be true or not true. And so I was trying to just take out the biased words and the editor kept reverting my edits and I kept reverting their edits. And it went back and forth, back and forth, back and forth, and we were editing each other’s work within seconds of each other. And I got told I’ll be putting a timeout if I keep doing that.
So I love Wikipedia for what it is, but we need something more than just Wikipedia. Because I don’t think it should be up to a few editors who happen to have time to work on these pages, to be the ones who are the arbiters of what is definite or a fact.
Jim: I have noticed a drift towards a bit more bias in Wikipedia, and I now generally cross-link it. For whatever bad reason, I’ve kind of gone down to some philosophical rabbit holes over the last few months. And I find trying to just get a beginning orientation in a new thinker, and now typically will look at both Wikipedia and the Stanford Encyclopedia of Philosophy.
And now the Stanford Encyclopedia of Philosophy is famously biased, because it depends who writes the article. They’re typically written by one person and they just scream of the bias of the author, often, not always. Wikipedia is somewhat better, but the two in parallax are kind of quite interesting and help you get at, “All right, what does actually the structure here? What do people actually think about Heidegger? What do people actually think about Hegel, et cetera?”
And I will say I do, I have also seen the drift a bit more in Wikipedia, especially the last two or three years, four years, maybe, something like that. Just as the sort of the madness is reigned around our whole Infosphere of motivated thinking around almost everything, right?
Jamie: Yeah. And to me, that makes sense given the context. People have thrown it around the world, epistemic crisis. I mean, we are dealing with all this information overwhelm. We have AI generated content now. So I can understand that people who care about stewarding the epistemic commons may have these knee-jerk reactions to over-correcting or trying to be more definite and confident, because they believe that’s a public benefit. As opposed to being a little bit more uncertain or having some caveats in their language, they may think that that leaves, opens some room to interpretation where they believe there should be none.
But the Society Library just approaches things as almost everything is debatable. And even though there are more evidence to support some positions than others, instead of just telling people that, why don’t you show them? This is the kind of evidence, this is how much evidence, this is the methodology that was used to create this evidence. Has it been reproduced? Has it not been reproduced? What kind of consensus are we talking about? Versus these are all the other different points of view, and this is the evidence that they have available to them. And that’s just simply something that you can’t really fit on a Wikipedia page, unless you’re summarizing it and using vague language by saying things like, “X number of scientists believe this,” or, “”Evidence suggests this,” which is language that sometimes used in journal articles.
Instead of just saying that, which is very vague, it’s like a zip file packed with a lot of meaning, unzip that so that everyone has the opportunity to scrutinize what is the evidence? What were the methods that created it? And what institutions created that evidence? What are these observations? And if they then want to increase their confidence of belief because they were actually able to scrutinize all of this, then that’s fantastic as opposed to just taking someone’s word for it. And sometimes that’s a really good heuristic. Sometimes just like experts say, is a good heuristic, but in the information age, we no longer need to accommodate that vaguery.
Jim: I was just thinking that as you were saying that. One could imagine, let’s say a Wikipedia entry on the “Earth. They might say, “Blah, blah, blah. And oh, by the way, there’s also these flat Earthers.” And rather than writing a long article about, they might just point to your node. Could you imagine that kind of hybrid where the library becomes a nodal extension of editorial efforts like Wikipedia or Stanford Encyclopedia of Philosophy or whatever?
Jamie: Yes. So I’ve already approached Wikipedia about this, actually. So the Society Library, we have something called Society Library papers where you can essentially see our whole knowledge graph, but viewable as an interactive piece of paper. So you can click on a sentence and unpack the argumentation that supports or refutes that sentence, or you can open up a sort of mini Wikipedia page about every single claim. So you can see where the claim comes from, the quote from which it was derived, the resources that support it, the pro and con argumentation. This is where all the media artifacts live.
And so I’ve approached some folks at Wikipedia, not the big bosses yet, about, “Hey, wouldn’t it be lovely since the Society Library specializes in taking content and extracting claims and creating this structured claim set about topics, if we could potentially auto-generate within a different program, maybe like Wikipedia debates, I think Wiki-debates already exist, but something comparable where we essentially have a page for every claim. And then people are able to explore whatever small reference topic that may be in a big encyclopedia page by Wikipedia, there may be a lot more content that people could view, and we would be happy to provide that and link to Wikipedia.”
We’ve made no progress there. But I think this is going to be increasingly important, because recently, well, maybe not recently, but in recent years, Google started paying for enterprise access to Wikipedia, which means that if you search something on Google, and Wikipedia has data on this, one of the first search results that is going to surface could be either Wikipedia itself, one of those boxes that doesn’t link to anywhere, but just provides an answer, or a Google generated page that is auto-populated with Wikipedia content.
And something that I did is I just started fiddling with Wikipedia pages to see, do I actually get to change what is the number one answer on Google and actually make it wrong by just changing Wikipedia? And that is absolutely the case. And so that’s a little worrisome. So I think that so long as Google is relying on Wikipedia for its sense making, that maybe if there’s other entities like the Society Library or other groups that really want to be extremely thorough and rigorous, in vetting content and contextualizing it and qualifying it, that maybe that we should partner with them. Especially for commonly searched things, so that people have a little bit more detail when they’re getting these answers surfaced to them as opposed to whatever happens to be on Wikipedia by whichever editor happened to have created that content. Because-
Jim: And whenever they took the snapshot, also, right?
Jamie: Exactly.
Jim: Because as we know though, was it like Hillary Clinton page has been edited a million times or something, right?
Jamie: Sure. Yeah.
Jim: So it’s not just you and one editor. It’s hundreds of people battling over control of that turf. Well, let’s go on to your next second to last virtue and value, love of information and individuals. And you had a turned out to be a Latin little tag there, nec timeo nec sperno, which Google Translate translated, I hope, rightly, I neither fear nor hope.
Jamie: Ah, okay. Interesting. That’s hope. So I thought it was neither fear nor despise, but hope is probably correct. This is going to be a fun little thing to fact-check on ourselves.
Jim: Yeah. Who knows if Google was correct? I mean, I just typed in Google translate, blah, blah. So, again, I don’t know, maybe they blew it.
Jamie: Well, I wrote these a long time ago, so I don’t trust my former self as much as I trust my research skills today, so…
Jim: Gotcha. So check it out. Let me know. Let me know.
Jamie: Yeah, I will. But basically what this means, this really sets the context for how we think about our work. And for me, this has actually become extremely important as we have seen our works effect on our analysts. So in the fact-checking world, there is this effect that happens with fact-checkers where they experience disillusionment, because they’re doing this deep epistemological work and they’re running into claims and arguments and evidence that maybe contradict what they thought was true, and that can be disillusioning or that can cause them to reduce confidence.
And this is beyond all the issues of them viewing gore and awful things that may also contribute to mental health issues. But there seems to be this kind of depressing, destabilizing effect that can happen for people who do deep research. And at the Society Library, we do not experience that at all. We have a very healthy, positive, bright curious kind of thrill in doing our work.
And this is something that our students have told us. So we have developed this methodology for this really rigorous deconstruction of content into claims, and doing this steel manning work and devil’s advocacy work and all of that. And as we were hearing, “Oh, fact checkers are having a real problem like a epistemological crisis, and it’s affecting them negatively.” When we teach students this and we’ll intentionally ask students, “What do you believe about climate change on this particular subtopic?” They’ll state that and then we’ll have them go do a bunch of adversarial research to prove the opposite is true.
And because of our culture and because of this love of information and individuals, which we try to embody, it’s a thrilling thing. In fact, many, many students have told us unprompted that we have given them a new sight. They’re able to see the depth of meaning of words and sentences and claims as they’re deconstructing them and doing their research to try and prove or disprove it. And so that’s just very, very positive.
So that’s the effect of the virtue and value of love of information and individuals. But what love of information and individuals means is that essentially we just realize that as people are expressing themselves, that can mean a number of different things. They may be expressing a need and that may be under many layers of abstraction or distraction or deceptions. But we see it as an opportunity to our work to connect people with ways of expressing themselves that are higher fidelity and are more accurate not only to them, but potentially to the debate topic itself.
And so it’s just so important that we honor the way people express themselves, even if it’s maybe not as accurate or they’re using bias language or they’re really emotionally upset. It’s just part of our job to realize that they’ve seen something in the world, they have this worldview, they’re trying to communicate it, and that it’s our job to take care of that and connect it with stronger arguments and evidence and make it easier to be understood. Because being understood is such a fundamentally important human experience. And so we work to do that. And you really have to have a love of people, an appreciation of people in order to do that earnestly.
Jim: Yeah, it’s amazing that you can find people that can do that, right?
Jamie: Yeah.
Jim: I suspect you have to be really careful about the people you hire to do this kind of work.
Jamie: Absolutely. Absolutely. And we often hire librarians, because librarian culture is, I guess, arguably thousands of years old. But at least in the United States, they’ve had the Library Bill of Rights around for, gosh, maybe like 80 years now. And so there’s a long tradition of just being really humble and loving knowledge and loving what knowledge does for people. And when a lot of people think about public libraries, in particular, a lot of times public libraries are community centers. And they think a lot about how to serve stakeholders, including people who may not be homed, people who don’t have jobs, people who may not speak English.
So in general, it seems like the institution of librarianship is just very human centered, and very service centered, and also obviously just absolutely in love with knowledge itself. So they’re good ones to hire and train in our methodology, and they work very hard, we’ve found.
Jim: That’s good. We’re just introducing our two-year-old granddaughter to libraries. She loves books and can read a few books yourself, which is getting a little scary, but oh, man, did she love going to the library? We turned around both to the library where her parents live and the library where we live. And she has library cards at both. And watching her walk around looking at the shelves, it’s really quite amazing. We’ve always been, as a family, big supporters of our local libraries, and just think that they’re some of the most valuable institutions per dollar that exist in the world.
Jamie: Totally. Yeah, I would agree.
Jim: All right. Your last virtue, gosh, we talked about these a lot, but I think this has been a very good conversation. Because I think it kind of get to the essence of what the issues are in doing the kind of work you do that you have the Pro Truth Pledge.
Jamie: So this one’s not so much a virtue or value, but it’s just us signing up for the Pro Truth Pledge, which is just kind of aligning our virtues and values with this other organization. So the Pro Truth Pledge is this group of people that are trying to create cultural buy-in for the importance of truth. So they’ve kind of listed all of these behaviors and those behaviors are essentially asking people to commit to sharing truth, honoring truth, and encouraging truth.
And so that means speaking up when something is inaccurately referenced. If you said something and someone else misquotes you and you have the opportunity to correct that, have the courage to go ahead and do that. Don’t just like let opportunities slide by if you can improve the epistemic commons in any way, whether professionally or in your personal life. So it’s just about aligning with dignifying, being honest and being as truthful as you possibly can. So that’s a different organization, but we did sign the Pro Truth Pledge. And they’re just on a mission to get as many people to signal that they’re on board for that mission to make it-
Jim: And what’s the name of that organization?
Jamie: Pro Truth Pledge. So you can go to protruthpledge.org and see all the organizations and individuals who sign up.
Jim: Cool. I’m go have to get a couple of not for profits I belong to, to go sign up for that. It sounds like a good thing. No, you’ve mentioned in passing a couple of the domains where you have done your work today. You guys are a tiny little organization and you have obviously very limited resources compared to the possible mission of annotating all of human knowledge. And so you have to start somewhere. And two of the ones you’ve talked about are nuclear energy and climate change. You want to go deep on one of those or you want to talk a little bit about both of them? What’ll be your druthers?
Jamie: Well, first, I’ll just honor the Pro Truth Pledge and correct you quickly that the Society Library is not interested in annotating all of human knowledge. We identify ourselves as a library of society’s ideas, ideologies, and points of view. So it makes sense that that would be implied. But we actually have a limited scope. We deal with topics related to the definition, distribution, and defense of human rights, resource management, international policy. So that means topics like climate change and COVID and these sorts of things because we’re dealing with rights issues and resource issues.
But I’d love to talk about the nuclear energy collection. Our COVID collection, our climate change collection, we are bigger than we’ve ever been, but we started out incredibly small, no institutional backing, no funding. So even though we’ve done a lot of work on these subjects, none of them have been ready for public publication.
But our nuclear energy ones have. So we got a grant, modest grant, but it was big enough to map out the debate in California about the last remaining nuclear power plant, which is called Diablo Canyon. So if you go to our website and go to libraries and click on state under nuclear energy, you can find this collection in multiple formats. We have a search engine where you can search through it. We have our debate map of it, we have our Society Library papers.
But essentially, we were asked to answer one question, which is, what should happen to this last remaining nuclear power plant? Because it was set to be decommissioned the two reactors in 2024 and 2025. And so what we did is we ingested a whole bunch of content, news articles, scholarly articles, documentaries, books, TV clips, websites, statements from politicians, and we extracted all the claims and arguments from that. And we built a database, about 5,000 qualified arguments about economic issues, safety issues, political issues, environmental issues, ethical issues. And we built out this database in support of nine different positions you can take.
So basically the community, when it comes to whether or not this nuclear power plant should remain open or closed or whatever, they took nine different positions: which it should close down as scheduled; it should close down immediately; it should remain open; it should remain open only as a poly generation facility, or it should become a different type of reactor; it should remain open, but only if PG&E doesn’t run it. So there’s a variety of different differentiations between just being open or closed. And so we mapped out the arguments that support each one of those, which is really fun.
And now that we’ve completed something end to end and published it, because climate change and COVID are such a massive topics, absolutely massive. And we definitely found the scaffolding that encapsulates the debate in the United States, there’s still a lot of work left to do, because you really have to deconstruct down to the data. That’s extremely important. And now we’re interested in automation.
So can we start creating the tool to automatically ingest, deconstruct structure, indicate to our analysts the type of research work that has to occur in order to have real time libraries and answer these questions in real time. Because we kind of have this deceptive relationship with our information environment. If we search something or put something into GPT-3, we’re going to get an answer. We’re going to get a search result. So we’re like, “Oh, everything’s great. We’re getting answers to our questions.”
But a lot of the questions that we have are very broad, vague, and are definitely debatable. And instead of trying to rely on some AI doing the debate behind the scenes and just delivering the answer based on statistical computation or however the heck these things do what they do, instead we’re presenting the menu. Here’s the number of answers that people have to this question. Now you can vet of the evidence and argumentation available, which one you are likely to believe is true.
Which is more important with ought questions, like what should happen to the nuclear power plant, is not really a matter of fact. It is trying to project what would be the best outcome. But that’s also based on values and where people are situated in the conversation. So it’s more of an ought argument, not an is argument whether something is or is not true or something like that.
Jim: Glad you corrected me on the domain that you’re working on. Do you see yourself working more on ought arguments than is arguments?
Jamie: No, we definitely do both. So when we were mapping out content about COVID, one of the primary questions we were mapping is, what is this COVID-19 virus? And there was a number of different answers that people had, including there’s the lab leak hypothesis. It was an intentionally released bioweapon. It was a natural zoonotic virus. So we do map a lot of is, but this one just happened to be ought, because it was a political issue.
Jim: That somehow feels methodologically different or maybe you guys have a methodology that is able to encapsulate the most. Maybe talk about a little bit the similarities and differences of working with is and ought.
Jamie: So the method is kind of the same, but the reason why it’s the same is because we accommodate an emerging structure from the data itself. So in this instance with Diablo Canyon, we were kind of given the prompt, map what should happen to this. But whenever we’re finding what the precise wording of questions should be, that comes after we initially scope the space itself.
So it’s kind of tricky to explain, but if you just ask question, the question itself is, if you’re trying to answer the question seriously, going to put guardrails on what is relevant to answering that question. By asking a question, you’re immediately omitting information that’s not relevant to the question. So the question that you’re asking is very important, if you want to be radically inclusive and sincerely inquiring to all points of view about that question. So what we do is we have a topic and we just ingest a whole bunch of content, deconstruct the claims down to their individual components, cluster them by topic, and then we start to get this emergent process of understanding what are people actually arguing about?
So whether our question is or odd is kind of depending upon the data that people are generating about the thing. Because we’re interested in answering society’s questions, which means we need to know what questions society has. So the climate change content, it’s easy to explain with climate change. So we did a bunch of work with climate change and we found about 278 subtopics of debate that Americans engaged in about climate change. But every single one of those neatly fit under only six questions. Most of those questions is questions, but there is one ought question.
So the six fundamental questions of climate change, which every single claim of the hundreds of thousands that we have fit underneath, and only underneath one is what is climate change? Is it happening? What causes it? What is the impact of it? And then there’s the what could or should we do about it? And that is more the ought. And then why has this debate lasted for so long? So of the six, five are is and one is ought. The ought sometimes is very important to people, because a lot of people make assumptions and are already clear on what the is ones are.
So a lot of people have their beliefs about climate change. They have their beliefs about what causes it, what’s the impact of it. These are matters of fact. They’re already decided, not everyone’s on the same page, which is why we would be radically inclusive to map out the different arguments about what do we mean when we use the term climate change? Are we talking about catastrophic climate change or matter of fact climate change or a hoax or whatever? But those questions naturally emerged just by looking at what people were saying and it kind of cohered into these is or ought frameworks.
Jim: Interesting. I was just going to hop back a little bit to something you just mentioned in passing, and that is the use of machine learning, artificial intelligence to bootstrap this work. This is really kind of top of mind for me because I’ve been playing with these new toys as they came out. And I got managed to grab ChatGPT the other day, and I’ve been playing with it.
And I will say it’s like, whoa, GPT-2 was kind of okay. GPT-3 was better. DALL-E, of course, lots of fun. We’ve had a lot of fun with DALL-E and Midjourney and Stable Diffusion, the other art generators. But the ChatGPT just seems to take this all to a new level. Not that it’s language model seems that much more sophisticated, but the way it outputs it as an actually reasoned answer to a question is just staggering. And it’s just like, whoa. So where are you guys at learning about how your mission and your desired deliverables map up with today’s machine learning and artificial intelligence?
Jamie: I had a volunteer recently that was tinkering with teaching it, giving it prompt, essentially, to execute and automate different parts of our methodology and it does it extremely well. But here’s the thing, we either need to use this artificial intelligence to create smaller models that ingest the kind of inputs and are given the correct prompts to create the outputs to slowly build the structures that we build, because it’s really good at you give it a prompt, it gives you an answer. And so to be as ridiculously comprehensive and to have all this chain of providence that’s still kind of like you have to put together a bunch of components in order to get that output, which is something that we’re interested in doing.
But at the end of the day, ChatGPT, GPT-4, unless they’ve got a bunch of data lying around that I wasn’t aware of, which I suspect they don’t yet have, because I’ve been asked to provide some of it, not to OpenAI directly, but other large language model groups, there’s still a lot of work that will need to be done, because if you’re really serious about deconstructing down to the datum, that means you need to have the datum.
And for a lot of the work that we do, a lot of the knowledge that we’re deriving claims from is not accessible in a repository that may have been used to train these things. So let me make this make sense. I was recently brought onto the internet archive, because one of the problems we were experiencing at the Society Library is an immense amount of inefficiency in getting access to government documents, data, research, et cetera. All these different federal and state and municipal agencies may have their own websites and their own archival repositories, which means if you’re trying to fact-check and find information, you may have to go to each one of these individually in order to root through their archival systems in order to find this report and find this piece of data.
So the internet archive has decided they’re going to collect all of this content and put it into one searchable machine reader or repository. And my understanding is that this is of interest to these AI groups to have access to that data, because it’s very high quality language, it’s very substantial research that’s being performed. But if they want it, then I’m guessing they don’t have it. And if they don’t have it, that means no matter how much I’m querying these things, it’s going to give me an answer, but it may not actually be pulling it from the types of reports where the accurate information actually is.
So we definitely can use it, it seems as though it’s viable in automating a lot of our methods. I want to raise more so that we can do more of that. There most likely will always be humans in the loop and there’s still more data to collect to actually help it give us accurate answers. But the chain of thought reasoning function is incredible. And that’s something that actually human beings kind of struggle with is stringing together premises and being really, what’s the word I’m looking for? Really fine pointed in understanding how premises have all these causal relationships to indicate a conclusion or suggest a conclusion. That can be heavy lift for humans, but seems to be no problem for AI. And that is fantastic news.
Jim: Especially now. I mean, yeah, GPT-3, sometimes the causal logic broke down terribly, and it was just gibberish. But whatever they’ve done with GPT-3 and a half plus, whatever the secret sauce on top of it is in ChatGPT, it’s really very good in that domain. I mean, is it perfect? No. I mean would it get at a philosophy course at Princeton? Probably not, but could you fool your high school teacher with it? Definitely. My wife and I were talking about that today. Compare and contrast Conrad’s part of darkness and Hardy’s Tess of the d’Urbervilles. I’m going to try that later this afternoon. I suspect it would get by my dear beloved English teacher Mrs. Carr in 12th grade honors English. And that’s kind of scary, but also kind of cool.
Jamie: Yeah. And I mean that’s the thing we have to remember too when we’re assessing these artificial intelligence systems and we’re concerned about them hallucinating. It’s like, well, people hallucinate too. People bullshit as well. But it’s a matter of if we want to really rely on these systems to be accurate, then them being accurate really matters.
Jim: Yeah. And what was I going to say about… Oh, yeah, I read something, I wish I could remember who it was, somebody from OpenAI was saying that one of the GPT ends that weren’t in their goal, it’s probably not going to be in four, is to give a slash command or something and have it footnote-
Jamie: Yes.
Jim: … its output. Now, I think it would require entirely different kind of model behind the scenes. I mean, I know a little bit about how large language models work and they’re just not designed to do that today. But I can also easily envision how you could build one that is designed that way. And this is where some of your expertise could be very interesting. How do you make sure it’s inclusive? How do you make sure it’s balanced? How do you make-
Jamie: Yes.
Jim: And then their secret sauce, which goes beyond what you do, is, “All right, we have diverse, we have balanced, now which one do we actually want to choose as the reference?” I mean, because that’s the art and science of a academic writer. I’ve written a few papers and collecting the evidence is kind of interesting. But you end up with nine papers that say sort of about the same thing with different nuances and different levels of quality and at different times. And the decision on which one to cite is actually a difficult problem.
Jamie: But here’s the thing is you don’t necessarily even need to choose. Maybe you choose which one is going to be the first on the list. But at the Society Library, in our database, you can actually see, every time we’ve run into a claim in a piece of content, we just include the quotes from all of them. So when you go to cite library papers and you click on a claim and you open that Wikipedia page, there are going to be a whole list of quotes from references in which this claim lives. And so a user could browse to any number one of those sources and they have a snapshot, because they’re given the context of that claim in a quote, in a paragraph. And it’s just listed there. And that kind of gets to this whole concept of having a linked knowledge database.
So there’s a claim ID. That claim is expressed through various different phrases. It’s expressed through different visualizations and it is connected to all of these other sources through which it’s been expressed. And readers may want to go in a number of different directions, because those resources may reference that claim but also accommodate a whole different perspective or more knowledge or new knowledge, which is what hyperlinking is supposed to do on Wikipedia. You go to the page and you see more about that thing. So maybe the AI doesn’t necessarily have to choose, but people can filter and sort how many of these references they want to see or something like that.
Jim: That’s an interesting user experience question, because to go and figure it all out yourself, all right, here’s seven papers, which one is actually relevant? To what degree? And are the nuances important or are they not, et cetera? Is cognitively expensive, it’s a hell of a lot of work to go read seven papers and decide what they mean relevant to a one sentence claim in an academic paper. One could argue it’s the job of the author of the paper, or it’s the job of the AI to choose the one that’s most relevant. They’re two different things. And truthfully, it’d probably easier for the AI to list all of its sources, rather than it would be to rank and choose the right one.
But in some sense, the work of an academic writer is to choose the right one. So it’s like two different things. And truthfully, there’s so many footnotes in your typical good quality academic paper to have to go fiddle fuck with all of them and try to figure out which of the six or three or nine are the best would be impossible, just wouldn’t be practical. So there’s probably room for both, actually. But in terms of a replacement for the classic academic paper, I think I still vote for making the right or choose one. Different than your mission though.
Jamie: Different than my mission. And it also gets back to user experience. So you have to open up the page where you see all these quotes and references. It’s not at the bottom of the page. There’s not a bunch of bracketed numbers that are going to interrupt your reading. It’s just like you can open up this whole little dashboard or page if you want to explore that. But you can otherwise just keep reading through these claims and just move on. So it’s up to them and their own research interests.
Jim: Cool. Well, thank you for a really interesting exploration of the idea of the Society Library, your methods, your ethos, et cetera. It’s really been extraordinarily interesting. And people want to learn more about it, can check it out at societylibrary.org.
Jamie: Well thank you so much, Jim. It’s rare that people want to talk about culture and virtues and values, so thank you so much, because it’s actually extremely important. And with these AI companies and everyone who’s working in knowledge management, I hope that they all have their ethos and their culture and their north stars and virtues and values as well, because the relationship that people have to information is just so fundamental and so important, not only to us as individuals, but collectively.
Jim: Well, thank you for the great work you’re doing. And I look forward to working with you in the future.
Jamie: Sounds good. Bye, Jim.
Jim: We’ll wrap her up right here.