From lawless spaces to true liberty: rethinking AI's role in society

SubscribeSign in

Playback speed

1×

Subtitles

Share post at current time

Share from 0:00

SPEAKER 4

0:00There's something really beautiful about human ingenuity. You give somebody a tool and they figure out neat stuff. And one thing I think that will be really fun to watch is going to be somebody who had a job that involved a lot of manual tasks. And they're just figuring out how to automate it themselves. And they themselves actually become much more valuable to an employer. We'll watch people learn how to do that.

SPEAKER 1

0:22I think there's a world in which it's not top-down automation, but bottoms-up automation. As in, we are the ones that are automating our jobs away. I love automating my job. Welcome back to Generally Intelligent. My name is Kanjun Qiu. I'm the CEO of Imbue. And we have with us Matt Boulos, our head of policy. We started this podcast back in 2020 when we were trying to understand from researchers how far this generation of LLMs would go. And the podcast has succeeded far beyond what we expected. Many of our early guests went on to have huge impacts on the field. And AI has gone from this niche thing to a household name everyone's talking about.

SPEAKER 4

0:58Cutting edge AI.

SPEAKER 1

0:59AI. Artificial intelligence. The rise of generative AI. But since it's become so ubiquitous, we started to realize something strange. The conversations in public are really weird. We have one AI CEO saying they're going to replace all of our jobs, but they're distributing intelligence, so that's good. And another AI CEO who's worried that it's going to kill us all, but it'll also give us tutors in India and new medicines, and so that's okay. But where is the serious conversation about the real costs and benefits of this technology, the real economic, societal, political, and very human impacts that AI is going to have on our lives?

SPEAKER 4

1:38When we talk about them being autonomous, it is about the ability to delegate to systems.

SPEAKER 1

1:43its ability to delegate human power to systems to encode that power. Generally Intelligent, this podcast and this conversation is the start of that. We want this to be a space for us to have these serious cross-disciplinary conversations about AI so that we can make changes. We can talk about different economic mechanisms, different ways to build technology so that we can create the future that we want. Because today it's not too late.

SPEAKER 4

2:10And if you think of the ultimate active power is to take someone's life in the context of war, particularly in the Middle East, like we have seen an example of AI systems doing the targeting.

SPEAKER 1

2:20We can still change how this technology shapes society. And if we wait too many years, that's not going to be the case anymore.

SPEAKER 4

2:28When we talk about, oh, we're worried that AI systems will kill people, they are killing people. And they're being designed to do that.

SPEAKER 1

2:42So let's dive in.

SPEAKER 4

2:44You've put a lot of thought into thinking about what are the core challenges that AI brings? Sort of like, let's level up the conversation. Let's start at a different level of sophistication. Why don't you walk us through what you think are what you see as the main areas that we need to take seriously if we're going to address AI's impacts? Yeah.

SPEAKER 1

3:06Yeah, it's interesting. I think sometimes it's so overwhelming because people talk about all of these different problems. It's like a whole smorgasbord from sycophancy to how it might take all of our jobs to how it might take over the world. And there's like problem this, this, this, this, this giant list. And so... A way that I think about the problems is I really bucket them into four categories. And I think of it based on the mechanism of action by which the system kind of is acting. So... And these categories, you know, I'm not 100% sure about, but I think of them in terms of one, empowerment of bad actors. The core mechanism there is power of actors who might do damage goes up. So it's a technology that gives a lot more capability. And now various people who couldn't wield this capability before can. And I actually lump both AI takeover, so AI systems taking over and kind of dominating humans, as well as terrorism, both in that category. The mechanism of action is the same. If AI is taking over, that just means that AI is taking a lot of this power and then doing negative things to humans. And same with terrorists or authoritarian governments. The reason why it's helpful to think about that mechanism of action is that it's very generative for solutions. So when I think about something like giving power to actors who are antisocial, I can think, okay, in the solution space, there are a couple things I can do. One, I can take away that power somehow. So prevent antisocial actors from getting that power. So that's like a whole bucket of solutions. Let's look at which actors exist. There's like governments, there's individuals, there's the AI systems themselves. And then let's look at how we can prevent them from getting power. And so that might be all forms of, you know, know your customer laws or safety research or things like that. And then on the flip side, I can also think about, well, another way to make things more resilient because of this mechanism of action is that I can make the world safer to bad actions like this. So maybe in that camp is better surveillance of the creation of biological artifacts so that we can prevent viruses. Or... Inventing a universal antiviral that would actually remove a whole class of dangerous problems. I actually think this category is very well talked about often. And the important thing is that it is just one category and that many solutions actually solve for many different of these actors, the problems that they pose. The second category I think of as transferring power from labor to capital, the capital L labor to capital C capital in the kind of Marx view. And that is about, you know, as labor becomes less efficient. powerful because we are less valuable and capital gains power, what happens? Most of us are in the labor class. We do not own the factors of production. We work for wages. And this is a technology where it's starting to do things that we currently do wage work for. So what happens to all of us who work for wages? there's the kind of like immediate somewhat alarming effect of that, of losing jobs. But there's, to me, the long-term somewhat alarming effect of this, which is that you have this kind of like constant power transfer from labor to capital. That is a forever thing.

SPEAKER 4

7:00There is something really quite striking if capital, the ability to be productive depends on on on capital right like this is a really abstract way of saying like i show up to work and like the capital i bring in my laptop right like it's a i could get to work without it i it would be hard to do slack but it is uh but for the most part i'm bringing the labor uh and i think this imagining the world if i get this right is concerned with the fact that maybe the day's gonna come that the laptop matters way more than I do. And then that's a really different world.

SPEAKER 1

7:37That's a really different world.

SPEAKER 4

7:38It just depends on who owns it.

SPEAKER 1

7:38Exactly. Actually, that's a really good way of putting it, which is the transfer of power from labor to capital is equivalent to the transfer of usefulness from me to my laptop. So what happens in a world where the laptop's way more useful than I am?

SPEAKER 4

7:52I've never looked suspiciously at this thing before.

SPEAKER 1

7:55That's a great framing because what happens in that world is not just economic. It's not just that I get paid less. Maybe I wield my laptop and so therefore I still continue to get paid some. But in theory, the company owns my laptop, so I may not get paid at all. But the second effect of it is political, which is part of the reason why we have political power is because our government depends on us to fund it. And there are a lot of countries that don't depend on humans. They depend on natural resources like natural gas or oil, the UAE, Russia. And they have a lot less incentive to treat their people well in the same way as we maybe in America do. And so... I'm somewhat concerned about the kind of like loss of political power that we'll have because of our loss of economic power. Because, you know, capital can now just use capital, use AI to produce more capital. And there's this kind of reinforcing loop. Yeah, yeah. Shall we move on to the third category? Okay, three and four we'll go through quickly because those are topics that we talk a lot about. The third category, which I haven't heard that many people talk about, is your idea of resistability. So in political philosophy for the audience, there's this idea of resistability. And it's this idea of kind of how well can you resist laws that don't serve you? So in America, we have fairly high resistability. The civil rights movement was a good example of that, where you could actually disobey, have civil disobedience, and then change the laws. There are countries that have very low resistability, like China as a surveillance state. And One thing that we're concerned about is kind of going into a future where the resistibility of humans against automated systems, either controlled by themselves or controlled by other people, is much lower. So we lose our power. So the core mechanism here is a transfer of power from people to automated systems and the people who control them. And there are a lot of examples of low resistability today. For example, we have very little ability to resist our social media notifications. We can turn them off. But we also have very little ability to kind of resist our social media algorithms or news algorithms or kind of control the news that we see, that we kind of want to see. There are ways of opting out, but I would consider it a fairly low resistability environment. And as we go into a future that has a lot more automated systems, agents that are doing things automatically, That's something that's really important to consider because now other people are going to have agents that do things like spam call you constantly or try to convince you on a website to buy something you don't need or try to convince you to give you data, give them data that they can resell. These are things where especially given what we see about current capabilities, like... not clear that we have anything in place that addresses that. And then the fourth category, I think of as how it affects us as people to live in a society where we don't have very much power and we don't have power economically. We don't have power to resist things. It's a very, we end up kind of disempowered and in the best case, infantilized. Like, And that is scary because I think there is a deep sense of learned helplessness that happens as we lose power. There's actually a great study by Lisa Kahn about how college grads who graduate in a recession – They have lower wages for the rest of their life relative to college grads who graduate just a year or two after, which is super crazy. So you termed this psychic damage, which I really like, kind of damaging our own perception of how capable we are and can be in the world. And I think this is really sad. It's not, you know, there's this kind of like spiritual damage that we don't talk about, which is about human potential, about what people can be. And I think what we want is for AI to expand our potential and expand what humans and humanity can be. But there are all of these effects that in this default path that we're on, seems like it's going to go against that.

SPEAKER 4

12:18Yeah. And I want to... I want to run with the last thing you said, because I think a lot about... I guess I have to come out with this. I'm old enough that like I have like a memory of my life that's pre-internet.

SPEAKER 2

12:34That is old.

12:35Hey.

SPEAKER 4

12:37But, you know, I remember being in grade school and Mr. Jen, the computer teacher, he knew that I was really bored. So he would pull me out of class and then we would just, you know, pretend that I was learning, but we were just trying out new software. And then one day he's like, let me tell you about the thing called bulletin board systems. And he's like, there's somebody else on this computer. I'm like, where are they? And he's like, they're in another country. And it was wild. Wow. And so hopeful. And I think again, I really think often these days about my parents calling their parents after they immigrated to Canada and they get these crap calling cards that were like cut out and were choppy and super expensive. And now I FaceTime my mom and she's like, I have to go now. I'm like, how will we hang on for a little while longer? And we don't even, we don't, we're taking for granted the ability to see each other, hear each other. And so you have all this incredible potential and it's beautiful and it's real. And I think the... Our machines can augment us and we love our tools. I don't understand why you could say, oh, I love my pen that I write with, but I don't like my laptop. But at the same time, we know that technology has been this very mixed force in our lives and from the capacity to surveil people to predatory mechanisms around how we communicate, social media feeds and all of these things. And so one of the things that I have felt is really important to bring to the conversation is that talking about AI as good or bad is almost silly. It's like saying trees are good or bad. Like as in you plant it right next to your foundation, that was a bad move. You step out into a clean city and it's the most wonderful thing. And there are... Of course, limits that analogy, but there is something really profound about taking the complexity seriously.

SPEAKER 1

14:41I was really struck by a quote that you said many years ago where you had just read something about Marshall, by Marshall McLuhan.

SPEAKER 2

14:49Yeah.

SPEAKER 1

14:49And you said, you rephrased what he said. And you said basically that... we adopt technology for its benefits and then we suffer its consequences. And so the important thing is to think about those consequences that we might suffer and see if we can get more of the benefits and less of the suffering.

SPEAKER 4

15:06That's right. And what was brilliant about McLuhan was that he'd clued into this dynamic where we adopt a technology and that it changes how... We work and interact and our own capabilities so that it is no longer possible to detach and disentangle.

SPEAKER 1

15:25We're part of it.

SPEAKER 4

15:26It's part of us. And then if it takes a turn for the worse, like just takes on like social media, right? Yeah. Public narrative on it, it's actually condescending and not correct. Like it isn't – we're not just a bunch of dum-dums who are sitting around swiping things because we don't have anything better to do or at least not completely. The thing that's really happening is that our lives, our social lives are on here. I get to see my friend's kids. You know, I've – I'm able to get a diet of things that matter to me or that entertain me. There's nothing at all wrong with that. We don't bear a responsibility for the fact that these things are wildly addictive. But if our lives moved on to these platforms... then we're now in this really stuck position because if the platforms don't behave, then we are subject to that. When I started law school, I decided to run an experiment. I figured, why not? So then I had no phone, no computer, I had nothing. I'm like, I'll be like sort of a nerdy monk and see how this one goes. And one of the interesting things that happened was that I realized that nobody wants to call your landline to invite you to parties. So I would get... Everyone wanted me to go to the parties, but they'd tell me about it the day after. Well, you weren't there. You didn't invite me. They're like, oh, yeah. And everyone was texting each other. And it was a simple thing. Like, you can't just.

SPEAKER 1

16:50You can't exit.

SPEAKER 4

16:51You can't exit. And that was, the significance was very mild. And then I got a phone. And then, yeah.

SPEAKER 1

16:57This is really important, though. Like, you can't exit technology. We can't just be like the Amish because this technology is now so prevalent and so, like, interesting. entwined in our social interactions and our lives let me give you the silliest

SPEAKER 4

17:10example so so my son is in a uh he's in a preschool that i guess has some bit of mandarin immersion and i know no mandarin at all and then he's now singing and uh and so it turns out there's this children's song um about pulling radishes out of the ground and my son Oh, that's true. And so we're like, what's going on? Why did we enroll him in a language school that we don't understand? And then what did we do? We actually just hopped on an LLM and just said, okay, like, my son is singing this.

SPEAKER 1

17:44Can you tell me what this is? I can now understand my son better.

SPEAKER 4

17:47All of a sudden, that world opens up.

SPEAKER 1

17:49Yeah.

SPEAKER 4

17:49And it's beautiful.

SPEAKER 1

17:50Yeah.

SPEAKER 4

17:51I want to challenge the idea that that is us being, like, conceding something. Why should I be trying not to do that?

SPEAKER 1

18:03It's not us conceding something.

SPEAKER 4

18:05Yeah.

SPEAKER 1

18:05But, okay... This is maybe to something you said earlier, which is, you know, when we get on social media platforms, it's very positive. And I was saying earlier about this resistability point, where it's very hard to resist these systems. And there's a core question of kind of like, what's going on here? And it is something about the power inherent in technology, maybe. which I want to ask you more about. Because fundamentally, what is AI? It is doing computation. Computation is the same computation that our brains are doing. It's taking inputs, perceiving them, running them through some model of the world, and then outputting some things. And those outputs can be turned into actions. And I think there's something about social media where it is an AI agent. It's making decisions about what actions are being outputted. It's like what I see on my newsfeed. And as a result, I get different inputs. So I'm getting this other input or information that changes my model of the world. And so I guess maybe... To your point about this symbiosis, like the technology is making some decisions that is causing me to get different inputs into my world model and now my world model is getting morphed or transformed in a different direction. Yeah. And that can be very positive. For example, you learn about your son and the song that he's singing and that expands your world model and helps you see reality more clearly. Perhaps the areas that I feel concerned about are the places where it actually kind of changes the way you see reality in a way that is more twisted or it, what is the word for that?

SPEAKER 4

19:49It feels manipulative.

SPEAKER 1

19:51Manipulates the way you see reality.

SPEAKER 4

19:56This point about technology and power, I think thinking about its mechanisms are really important because if we think about maybe again going back to like my son and singing his song is that I went to this machine and asked it a question and it came back with an answer that couldn't have gotten five years ago or at least not so easily, not so smoothly. But We're only talking about one part of that transaction. We're talking about me asking that and getting an answer. We're not talking about now is this being logged? Is there now something now – does this system now know that my son is in a Mandarin immersion class? Are they going to then – are we going to get like Mandarin worksheets offered up to us at the next interaction? Like there is a – there's something about this digitally mediated world that is foreign and dangerous. And I think that's worth probing into because

SPEAKER 1

21:01What is it? What do you think?

SPEAKER 4

21:02Yeah. I mean, I have a couple of different mental models, but one that – let me play with two. So one is something that I call lawless spaces where it's almost like imagine a part of town where the police don't go and sort of the rules, the norms – People are working them out, but they're not governed like the rest. And you cross that threshold into those places, soon things become possible. Maybe there's just a free spirit. It feels, if you think about the chaotic early days of the internet, people are like, people who did not, who relished in anonymity, not because they were up to trouble, but because there was something liberating in that. And you can imagine that creating a heady atmosphere.

SPEAKER 1

21:49Like the Wild West. Well, exactly.

SPEAKER 4

21:51Your bank is like, why shouldn't I be there too? So your bank sets their shop, except when you step out of the bank holding a bag of coins, someone whacks you over the head and takes your coins. And you don't have... And so that space doesn't have the same rules, the same governance. It's not a perfect analogy, but there's a lot to be said for that. I... You know, I was talking to somebody about privacy and they're like, oh, that ship sailed. And I said, well, why? Because we were having dinner. I'm like, if there's like some dude just comes up to you and just stands right next to you while you're talking and he's just writing down what you're saying, give him a slap, send him out of the restaurant, right? And should we have that? We don't see it in the digital context. And so it's still, we haven't learned to govern that.

SPEAKER 1

22:36Do you think digital spaces are lawless because they're not visible?

SPEAKER 4

22:41I think that's a huge part of it. And this is what I wanted to... The next thing I want to talk about is, like, what are the things that make the digital space really particular? One is that most of what happens in them are actually invisible to us. So I interact... Like if I go to like, you know, the neighborhood Oracle and I give him 10 bucks, I'm like, my son is singing this song. Can you tell me what this is? He's going to sit there. He's like, oh, man, I know what it is. And he's going to grunt and groan. He's going to write something down and send me out. I go to LLM. I don't like, I mean, because we know what's going on. But the point is that that's, you know, there's... It's a magic box. It's a magic box. And... And it's a magic box that I don't want to take something back. You know, I may know how in theory it should work. We don't actually know how it's implemented. We don't know what a year from now, five years from now is going to happen.

SPEAKER 1

23:34And you don't know what's being logged. You don't know what the company is doing with the data. You don't know. There's a lot you don't see. You can't see.

SPEAKER 4

23:41You can't see. And there are really particular characteristics to just the digital world. It is easier to log than to not log.

SPEAKER 2

23:48Mm-hmm.

SPEAKER 4

23:49And it's safer in a lot of ways. And it's safer and it's like, and there's an expectation. And if something doesn't work, the customer is like, I did X and it didn't work. And they're like, I have no idea what you did. I have no logs. And they're like, what are you doing? Like, are you junior grade like developers? And so there is all of these expectations. It is easy to log data. It is cheap to collect data. It is lucrative to collect data. And even before advanced data, you know, like models like LLMs, we could crunch through stupidly large amounts of data. And so you have these reinforcing mechanisms that take us to really perverse outcomes. And, you know, we talk about surveillance. Surveillance enables an astonishing amount of bad stuff because you were talking about resistability.

SPEAKER 2

24:42Mm-hmm.

SPEAKER 4

24:43And resistibility is something that you reach for when you're in conflict, right? Resistibility is now something has gone wrong.

SPEAKER 1

24:51And now you have to resist it.

SPEAKER 4

24:52Now you have to resist. But preceding that is legibility. Do you even know who I am? What right do you have to that?

SPEAKER 1

24:58This is actually, you know, this prompts for me. Let's imagine we were making a world in which we're making everything digital into a physical manifestation. So in this surveillance, like, you know, I'm kind of thinking as you're saying this, okay, what I'm hearing you say is... We have ended up in this really weird default digital world, especially going into this AI future. It's weird because some defaults are weird. One default is that we log all data. A second default is that we can process companies, I, running this company, can process that data however I want. Another weird thing is that now we have these AI systems and they can do lots of new magical things with that data. For example, take a photo and know where I am. And then another thing that I'm hearing is as the person at the other end, I can't see any of it. And as a result, I actually don't have a mental model of it being a problem at all.

SPEAKER 4

25:50No, that's right. We also didn't have like an emotional response. I mean, we're wired as human beings to recognize these things and then we can't react.

SPEAKER 1

25:58That's really interesting because we talk about something like surveillance and it's such an abstract concept. But if you were to turn surveillance into a physical manifestation, like this guy who is writing everything down in your conversation next to your table, then it would be like we have five people following us around everywhere and they're all logging different things about our lives and they're changing other stuff in our life based on what they're logging.

SPEAKER 4

26:19Well, I mean, and this is where it starts to get wild because... On one track, often when people are talking about the productivity benefits of AI and the labor impact, we're talking about labor substitution. But there's another way of thinking about the impact of AI within the labor context, which is that new work is being created. And what I mean by that, let's take something like credit scores. largely opaque systems, the financial services industry benefits from it. And they would argue that you and I, Bennett, like the good faith argument is that we all benefit because then, you know, if, if, if I'm, you know, a untrustworthy borrower, you know, you shouldn't have to be paying rates to subsidize me. So we create, you know, we, we, we stratify in the basis of, of reliability or, or however they, they, they, whatever terms they use to describe like creditworthiness. And, uh, But then you could start to shift the granularity of that. It could become, we could just collect all sorts of stuff. We could also experiment. We could collect data, not sure if it's relevant, deny me alone, who cares, I'm one data point, my life gets crushed, but they don't know about it because their system did it. And just move on and that dynamism that becomes possible is going to be really... potentially quite pernicious.

SPEAKER 1

27:44When you say dynamism, what do you mean?

SPEAKER 4

27:47Like you could have systems that are not stable anymore. Like there isn't a credit score. There's an algorithm that's constantly rewriting the rules.

SPEAKER 2

27:56Why not?

SPEAKER 4

27:57As long as goal seeking against like minimizing defaults, it doesn't matter how unfair it is. And that unfairness may not fall within... Often when we talk about unfairness, putting on my lawyer hat, we often talk about... things like disparate impact, protected categories, that sort of thing. But what happens when it's arbitrary? What happens when it's large categories of society? What happens when it's not easily pinpointed? Again, the bad stuff is happening behind the veil. So we don't know. And I want to connect that to something earlier that you were talking about, when you were talking about the economic impacts and you're saying, that destabilizes society. But also... when you live in a world where you are subject to all of these forces and you're helpless against them, it's not good for a person to feel that way. Like think about the worst parts of like childhood where like adults are not taking you seriously, not letting you do something that you ought to be able to do.

SPEAKER 2

29:02And yeah,

SPEAKER 4

29:04And then that becomes sort of the dominant mode of adult life.

SPEAKER 1

29:07It's very disempowering.

SPEAKER 4

29:08Very disempowering. And this is preceding oppression. Like just that by itself is destructive. And then you add to that malicious intent or malicious oversight. And it isn't a surprise that we live in an angry moment in our society. But it's also something that it's weird to me that So I don't have a lot of patience with the tech community sort of sitting around saying, oh, how could this be? Well, I mean, you've been bloody architecting it for the last two decades. So of course that's how it could be. There is a reason why people feel disempowered because they are disempowered.

SPEAKER 1

29:43They have no power to change a lot of things.

SPEAKER 4

29:45How could you change any of these things? And so the thing with AI, and I think it's really important that we ground it, that we have to recognize that all of these dynamics... are in play so that we then can ask, how do you?

SPEAKER 1

30:02How do you design?

SPEAKER 4

30:02Yeah, how do you design and how do you get to empowerment? Because we could also just sit here and be angry and then walk away, but that's not going to help us.

SPEAKER 1

30:07Okay, two things that really come up with me as you talk about this. One is narratives today about what kind of future is okay for humans. I think a lot of the futures that the tech industry talks about are actually very disempowered. Yeah. one type of future is like, oh, you know, we're going to live in a stable utopia where everyone's gonna have anything at their fingertips and it's gonna be okay. But does not consider seriously these dynamics where actually people are being controlled by technology and the people who control technology. And I think a second thing that you kind of pointed at, it was this morning. You said you don't actually... I think there was this notion of utopia being like permanent undergrad or something like that where you can be free and intellectually curious and it's really fun. But an undergrad is not an adult with the ability to kind of like fully manage their lives. And I think... the kind of freedom that you're going for, the kind of liberation you're going for is like for humans to be truly able to be fully kind of adult and in the world themselves without being so pushed upon by other forces with the kind of ability to like push against those forces.

SPEAKER 4

31:32Absolutely. I mean, it... And what do we really want from our lives, right, is to be able to realize our capacities,

SPEAKER 1

31:44right? And that is a... And that involves growth and change and creation and then being pushed down. Absolutely.

SPEAKER 4

31:51And having a chance in all of it. One of the things that I've noticed within... Sometimes, again, not to take a piss on the tech community, but like, you know, we'll talk about what are like, what's an ideal future or, what's an ideal life for someone to have. And that's just somebody just projecting what they thought was interesting to them. But, you know... I have so many people in my life for whom say the specifics of their job don't actually matter that much as long as they can take care of their family and support their community in ways that are really meaningful to them. And those are rich, beautiful lives. And what happens when... the structures around a person sort of erode on that, I think is when we start to see this real frustration emerge.

SPEAKER 1

32:43Yeah, I want to double click on the frustration and the idea of lawless spaces because something I heard earlier in this conversation is kind of, or my synthesis of it is, people are frustrated because they feel like they don't have any levers to change things the situation of their lives and they don't like the situation that they're in, even though the world is abundant and they're fed. There's something missing about their sense of autonomy or freedom or their ability to make change. And what I heard from Lawless Spaces is it's partially a lack of legibility and partially a lack of levers of action. And if you had legibility for everyone and levers of action for everyone to be able to change their life circumstances, the institutions that aren't serving them, then maybe those two things would allow us to kind of like be able to have a little bit more autonomy and self-determination in our lives.

SPEAKER 4

33:38Yeah. I also think that just we – it's kind of hard in our present moment to be thinking about what like – what does a stable like political or legal regime look like?

SPEAKER 1

33:54In general.

SPEAKER 4

33:54In general. But there is just simple fact that we've for centuries now have figured out that it's not cool to steal somebody's money. It's not just that the theft is wrong, but that the state can't do it even if it's useful to the state, right?

SPEAKER 2

34:10We say that's not right.

SPEAKER 4

34:12That's true. And I think about, you know, there's a cause that I, you know, In conservative circles, people talk a lot about debanking. And that's where just banks just turn you off digitally. And it's not a frequent occurrence, but it happens and has happened in response to political events. And what's kind of wild to me about it is that it just could not have been a thing a few decades ago. Like the bank would have to literally stolen your money. In a way that- Like a bank run or something. Or just simply like you'd go to your bank ranch and like, we're not going to give you your money would be what debanking looks like, which is so different from, and you would say, you stole my money. Whereas debanking now is just either just hitting a switch and just can access your money or just saying, here's your money, you're out of the financial system in a way that is- only possible in a digital world. And so we haven't like, so I have one belief that our laws and rules haven't caught up just to digital reality, then AI accelerates digital reality to all of its conclusions.

SPEAKER 1

35:21Right. It's interesting. You're kind of saying... Like somehow, or maybe what I hear from what you're saying is like the digital world is enabling all these mechanisms like being able to turn off my access to my funds. And the laws haven't caught up. And... The last 2,000 years of development in the Lego system have been about very physical reality. And that physical reality is actually happening in the digital world. You're giving all these physical analogs to it that are really interesting because they let us see the physical reality of what's happening. But somehow we haven't mapped that physical reality to the digital world. What... would be required to make lawless spaces more lawful or like what's going on? What do you think is going on here? Why have we not caught up? Is it... lack of knowledge is it the like something that's you know kind of this like lack

SPEAKER 4

36:23of visceral sense of what's going on like yeah i mean okay so i mean there's a lot like i don't know that i can dispositively say i i think each of the things you said feels to me like it's playing a part for one Again, like we both understand computers really well. And like let's not kid ourselves. Like when I hop on a website, it is not occurring to me that they are tracking the things that I'm doing or that.

SPEAKER 1

36:54True. The other day I hit accept cookies. And then I was like, what happens when I accept cookies? Oh, shit. It can track me across multiple websites. Like that's crazy.

SPEAKER 4

37:02That is crazy. And then I like drive people nuts when they look over my shoulder because I always like not only reject cookies, but I open the thing to make a point of deselecting everything. And the hilarity is, of course, often these are just pop-ups that don't do anything and it just collects your data anyway and puts the cookie. I mean, these things like work. Oh, that's kind of depressing.

SPEAKER 3

37:22Yeah, that really is.

SPEAKER 4

37:23You're welcome. But there's one sense that is not tangible, no matter how sophisticated you are. I think that the other thing is it is new. Like in world historical terms, we're talking about living in this regime. Like 10 years. Yeah, it is not that long, right? Like if you think about Google trying to figure out how to monetize was something that happened like basically in our adulthoods. That's sort of nuts. And then going from like web to mobile, the introduction of all of this has happened really, really fast. And so I think part of it is we haven't caught up. The other somewhat more cynical thing thing is that it turns out lawless spaces are awesome because they're so lucrative because it turns out that if you can do stuff like surveil people and track them and price fix and all of the rest you can do all sorts of astonishing things that's true and so we uh and then maybe the last thing is to what you started this whole conversation with if you deal with it now It's a lot easier than down the line. Like one easy answer is a good privacy law. And I do want to take a step back. But let me, this is why I say the privacy laws. Had we had good privacy rules 10, 15 years ago, it wouldn't be so painful for the large tech platforms to unwind these privacy practices.

SPEAKER 1

38:54But now it's entrenched. You have to change your entire infrastructure.

SPEAKER 4

38:58And we're talking like infrastructure, business models, identity as an entity, and you're talking the market cap of these things. I don't want to grant sympathy to the surveillance practices, but this is a huge thing that we're going to have to ask of them. But we do have to ask it. But there's a... But there is an interesting question of what rights do we have that we have failed to translate just as a practical matter? Like we already have these legal rights and we haven't brought them to these spaces. And then what are the new things that we have to figure out? I'm often... Larry Lessig's notion of code as regulator is really fun. So what he does in this setup is that he points out that in every period of time, there's some sort of a regulating force that you have to contain if you want to protect liberty. And in his construction, one that I share, kind of we're sort of progressively trying to increase liberty as a society. But he points out that in the time of John Stuart Mill, you were white about majority opinions, a democratic opinion, but it can trounce minorities. So then we start to establish the notion of rights and, you know, constitutions become vital to that because... if you just leave it to the majority, then that's actually sometimes not great, often not great. And then you have like Civil Rights Act where you're, you know, when you look at suffrage movements and on. And what he was pointing out that I thought is really interesting because it doesn't feel like the same thing is he said, okay, well, the new thing is going to be code and code is going to operate. And this was in 2001 or 2000 that he wrote this piece, but code as regulator. And the argument there is that

SPEAKER 1

40:38Code is encoding laws.

SPEAKER 4

40:39Yeah, code is encoding, or it is going to determine how a sphere of life is going to play out.

SPEAKER 1

40:47That's super interesting.

SPEAKER 4

40:48Yeah, and so then the question we now need in response to that is, what things in that space need to be addressed?

SPEAKER 1

40:55Yeah. You know, it reminds me of, I kind of have this hypothesis that technology shapes our governance system. And... the way that that technology is built and what makes it powerful. So there's this theory that the reason democracy happened, I'm sure this is just one of many reasons, but the reason democracy happened was because we went from a world where Knights were the most powerful thing. To a world where muskets were the most powerful thing. And when you have knights, you have a lot of upfront investment in armor. You have to have horses and stables and all these really well-trained people. And so that's like a very centralized form of power. And the technologies at that time kind of like resulted in this centralization because of the nature of those technologies, war technologies. And then the musket was invented and now like knights are not very useful and armor is not that useful. And in fact, actually we want a lot of people who have muskets. And so now people matter because of this new technology, new war technology that gives power to people. And I'm curious, you know, we talk a lot about how fundamentally AI and the problems with AI, those core four problems that I talked about, fundamentally are about power and about transfers of power from one entity to another entity. And we call it problematic when it gives power to entities that maybe are not you know, what we've determined to be morally right. So in that lens, thinking about lawless spaces, thinking about kind of what this upcoming technology is starting to enable, like, how do you think about, is there like a, you know, kind of like code as law or like this, is there a nature to this technology, to AI that kind of shifts things one way or another?

SPEAKER 4

42:57So I think I have two responses. One is there's also just like law as law where – Like, what is it about this moment that we leave, you know, we leave ungoverned, right? Like, I find a lot of these, you know, these free market arguments, like, the accelerationist camp. I mean, when I find acceleration is essentially bullshit, right? Like, all you're saying is just, like, we don't want regulation. So let's just say that. There's nothing else there, right? It is into richer argument.

SPEAKER 1

43:26Because lawless faces are great. Right.

SPEAKER 4

43:28A lot of space are lucrative. They do yield huge amounts of opportunity. And look, I'm not saying let's clamp down. That's how you shut everything down. It often does not make sense to intervene. It also does not make sense to intervene before you understand a space because you will have spent your political capital. Think of even American politics with all this craziness right now. There is political capital that can move you towards some sort of a privacy bill or things like that. And if you do the wrong thing, you will not – that capital is not waiting for you to go do it again. So you have to be disciplined about that. But at the same time, you can't just say no rules. Or if you do, then that's ideologically encoded and you're saying something else and then you ought to own the rest of your argument.

SPEAKER 1

44:10Right. It's like if there are no rules, we're buying into a particular society.

SPEAKER 4

44:13And do we want that? And is that a fair thing to ask of others? And if you want to impose that, then you should also expect resistance on it. And it's a – Well,

SPEAKER 1

44:21this law is interesting because it actually argues against my argument that technology shapes society in this like fundamental way by giving power in some, you know, I think maybe what you're saying is like you could make laws that change that distribution of power.

SPEAKER 4

44:33Something could be wrong. And whether or not, you know, the temptation to that wrong thing is great, it's still wrong. But then- Like, if you don't want to eat the muffin, don't put it in front of you, right? And I think this is... We have both. Law needs to set the boundaries of what's acceptable or unacceptable, regardless of what the temptations are. But the nature of the technology is going to shape those temptations. Back to the point about how surveillance is the easier default model. So when it comes to what we do with these technologies...

SPEAKER 1

45:05It makes some things easier than others.

SPEAKER 4

45:07Yes, absolutely. And take something like a perfect example is going to be something around labor. And I want to bait you into this conversation. All right. But labor impacts are going to be real. We don't yet know what those are going to look like. There will be things that... employers can and can't do, companies can and can't do or shouldn't do. Right now, we know that the power that a company has over, for instance, a warehouse worker that has their work determined by algorithm. And I think it's also worth pointing out they don't have a capricious boss who could be an asshole and make their life hell. The algorithm is governing things both good and bad. But Do we then say, ah, this is the shape of the technology and back away? Or do we recognize that this starts to introduce things that weren't possible before and we need different rights and rules? Because we're a human manager. Because if we think of most of our labor laws are predicated on humans interacting with other humans, more powerful humans, but they're human interactions.

SPEAKER 2

46:12Mm-hmm.

SPEAKER 4

46:12Whereas a machine can surveil your every motion and then dock your pay for scratching your nose at the 15 minute mark. And we don't really have mechanisms for that because we couldn't have conceived of that as being an active problem. It would have been nonsensical to have rules of government.

SPEAKER 1

46:27This is very interesting because it speaks to kind of actors in the world and the power that they have. And this like new actor, which is an algorithm or an AI agent, like what you're saying is, okay, right now we have laws and they govern your capricious boss. They govern you, they govern your corporation, which is considered an actor. And so like the only actors we have are humans and human institutions in the world before AI. And we have laws that kind of limit the power of humans to harm each other. And we have laws that limit the power of corporations to harm humans and vice versa. But these laws that – but now there's this rise of this new power. You call it that. Yeah. Which is AI systems. And I think of it as like, okay, AI systems have power because they can process information and turn information into action. And action is power. Effective action is power. And so to the extent to which an algorithm can govern – What I am allowed to do as a warehouse worker, that is power that the algorithm has. And so now you're saying, okay, we have this new power. What do we do with it? We're not doing anything with it.

SPEAKER 4

47:44And exactly, right? There's a new power. Like some of it will be societal norms will change it. Our behaviors will change it. The technology itself will change. And therefore that power will morph. And it's just so odd to me that you say, okay, then we're done.

SPEAKER 1

47:59Mm-hmm.

SPEAKER 4

48:00Like, we've never done that in human history.

SPEAKER 1

48:02It's like, no, we need to figure out what to do with this power and how to, you know, it might be partly because this is the first time a technology is its own power in a way. We've never had technologies in the past that make decisions.

SPEAKER 4

48:15A great disservice, not to dunk on people who are trying to do good work, but a great disservice was done by the AI safety community on this point, where... by talking about like runaway systems as much as they did, they created this special category of worry, this incredibly low probability event that also low probability, and we don't actually know what its dynamics are going to look like. Whereas this notion that systems can make their own decisions But they're doing it for someone. You don't go spend millions of dollars to develop a system and you're like, I will let it go. Right? You're doing it to manipulate the stuffing out of your viewers so you can sell more ads to people by flip-flop so you get your cut on the ads and on. And across the board in every domain that these autonomous systems are going to function, they're going to do so for a purpose for an owner, a controller. And... So we, one, have to sort of let go of this. Like when we talk about them being autonomous, it is about the ability to delegate to systems.

SPEAKER 1

49:31It's ability to delegate human power to systems to encode that power. Like I as a manager can now encode my power in a system.

SPEAKER 4

49:40Yep, exactly. That's right. And that is an astonishing amount of power, the multiplicative. So the fact that you could do so at massive scale, you can do so quickly and it can adapt. And then one of the things that... I, you know, in this New Power thesis argue is that one of the very interesting things when that happens, it is very hard as a human on the other side of that to know how the decision was being made. And so there's a default to accept that.

SPEAKER 1

50:12And they have no levers over the decision at all. No legibility, no levers.

SPEAKER 4

50:16Exactly. So you have exactly no window into what is going on, no means of recourse. And as more and more of these sorts of things happen, we'll feel very powerless. And I think it's an incredibly sad example, but in the context of war, this is what we are seeing. We are seeing, particularly in the Middle East, like we have seen an example of AI systems Doing the targeting. And so people have not classified this as autonomous systems gone amok because a human built or humans built a system for that purpose. And yet when we talk about, oh, we're worried that AI systems will kill people, they are killing people.

SPEAKER 2

50:55Mm hmm.

SPEAKER 4

50:56like explicitly they're killing people and they're being designed to do that and that uh and let's be honest like when people are talking about national security implications for ai yes you're talking about economic competitiveness but also you're talking about the fact that you want to have ai systems that can do that and so and if you think of the ultimate active power is to take someone's life so you we already have that extreme happening right now and being realized but it's the same dynamic where a human is delegating or sets of humans are delegating the thing that they want done to a system and the system can carry that out. Because it is a system carrying it out, the context and the entire execution of that looks completely different. Where's the appeal? Where's the chance to challenge it? Where's the saying that's wrong? Where's the record? Where's even the idea, how do we know how that decision was being made? I think sometimes in certainly my day-to-day experience, Because when I'm using AI systems, it's fun or productive. I don't care how it came to the decision. I'm just like, is this right? Is this right-ish? Can I work with this? This is great. You know, it's a, you know, like my primary like LLM use right now is that I'm trying to count calories. And so I take a photo of what I ate and then I try to negotiate with it to lower the calories so I could eat more food. But the point, you know.

SPEAKER 1

52:09So there's actually a huge difference here. This is really important because this calorie counter is an AI system that is under your control that you're using to serve you.

SPEAKER 3

52:19Exactly.

SPEAKER 1

52:19And the war system that you're talking about is a system under one person's control that's being used to control someone else or harm someone else. And those two things are two different types of systems. And you might argue that actually what we want is more systems under our control that affect us. and ideally don't affect other people too much.

SPEAKER 4

52:40Well, imagine my calorie counter determined what I could eat.

SPEAKER 1

52:43Right. Then it would be controlling you.

SPEAKER 4

52:46And it would be awful, right? And it's not perfect. And sometimes it goes completely off the rails in either direction. And that's sort of fine-ish because it's within my domain. It's an irritation. It's not a risk.

SPEAKER 1

53:02This is interesting, something, I mean, this is a slight digression, but something I've been thinking about with our product, we try to make systems that allow people to make software. And I often talk about kind of open software or an open software commons or malleable software, the fact that software should be built to be modified by the end user. And I realized recently, because a lot of people are like, who cares? I don't want to modify my software. I'm perfectly well served by my software. There's no problem. Except sometimes. And I realized the problem is not, the core idea is not that the software should be built to be modified. That's an instrumental thing. Instead, it's that software should not control me.

SPEAKER 4

53:40Yes.

SPEAKER 1

53:41Ever.

SPEAKER 4

53:42And this is the thing, right, is that people might say that they don't want to change things, but often that's because the decision space has been so narrowed for them. And one of the things that's really interesting to me is we work on interoperability and look for, you know, we're rallying a community around this. And what's fascinating to me is how many startups just... never got to a place where they could fight for interoperability because their mere existence would not be feasible in the current regime.

SPEAKER 1

54:11Talk more about interoperability. Explain a bit about...

SPEAKER 4

54:15The driving point here is one of the main things that we're championing and pushing for is interoperability legislation. And the idea at its simplest is that a platform should not be able to discriminate on the basis of how you access your own data and services that you use.

SPEAKER 1

54:35You should be able to get your data.

SPEAKER 4

54:37Yes, and you should be able to use your tool. And it should be yours. And you should be able to use a tool of your choosing to interact with another system. Just a way that I could go, you could go buy bananas or you say, hey, Matt, can you go get me bananas from the supermarket? You couldn't have a supermarket saying, no, only can't you, right? And yet that's our online world. And because of that,

SPEAKER 1

54:58Like make it concrete. LinkedIn says, I'm not allowed to use someone else's account to use LinkedIn.

SPEAKER 4

55:03Precisely. And you also can't use something like TweetDeck, like the way that you can use their party software to access.

SPEAKER 1

55:09I can't use TweetDeck. It's a monopolistic.

SPEAKER 4

55:12Exactly. And look, the platforms do this for good reason. It consolidates their control around the points of input and access, but it is also the consequence of that is pretty severe. And so The two things that are happening is one, we are moving towards a world in which these AI systems are going to be more and more useful. So we are going to share more and more data back to that point of it is not incumbent on us or we don't even have any indication that these things might not be handling our data soundly and so we're going to talk to them. I'm going to say I'm injured or I'm sick. Can you please go make an appointment for me? And we don't know whether that data is going to be held with any sort of responsibility or not. And then the other is that there are all of these wonderful things that could be built if I could just access my digital life. And so what interoperability does is it gets two really critical birds is one stone, which is one is if I can mediate my, if I can access my own data, then I can decide where it goes. I can control that. I can check up on it. But the second and more critical is that if it's possible to build software that interacts with my richer digital life, then I'm not attached to these parasitic platforms and agents and we can build alternatives and you just can seed a whole other tech ecosystem around the idea that we're in charge. Like it's our data and then these are the platform space.

SPEAKER 1

56:33And it's our software. It's our data. We make it. And we can sometimes interact with these platforms. Absolutely. But we can use our own interfaces. Yeah. This makes me think about kind of, you know, it is becoming possible that we can make our own software. Yeah. And make our own wrappers that, you know, our systems that access Twitter data and download it. And then I can make my own algorithm. And I can process it in a different way. And I can... get just my friends and none of the inflammatory stuff. I can derank inflammatory stuff. That's like just starting to become possible.

SPEAKER 2

57:07Yeah.

SPEAKER 1

57:08I kind of have the saying that, you know, the software that exists today, it is because of the means by which it's produced, it's so expensive to produce. Yes. It's incentivized to make that money back. Not because the creators are bad, but that's the incentive structure. And as a result, it's either selling to us something or it's selling us to something. Those are the two options.

SPEAKER 4

57:35But that's right, yeah.

SPEAKER 1

57:36And then occasionally you have someone who's incredibly generous who makes software for free. And it really feels like it should be flipped on its head. That most software that exists, AI systems that exist, we lump it all in the same category, it should be software that is serving us, not selling us things or selling us to things. And it is ours and that it doesn't require enormous acts of generosity to create software that doesn't do that, that is just for us or for other people. It should be easy. It should be what the default world is. Yeah.

SPEAKER 4

58:11And sort of if you think about deep, rich, sustaining communities, there's they're very generative, they're very productive. And if you think of the art that emerged from religious communities and the invention of different structures, the social structures, different aid structures, all of these things. And so it is a peculiarity. And I wonder if it is also a peculiarity of just how young people software is. Again, we're in world historical terms, like we're talking just a couple of decades in which you have the prevalence of software, maybe more than a couple of decades. But the point you're making about the fact that the cost to make software will go down and then the stuff that we'll make will start to look

SPEAKER 1

58:51It can. It could.

SPEAKER 4

58:53It could.

SPEAKER 1

58:54It could, yeah. If you get things like being able to access all of your data.

SPEAKER 4

58:58That's right.

SPEAKER 1

58:58Because network effects, I think, are real. Like right now, these big platforms have network effects. I can't just move to another social media platform and be able to interact with all of my friends. That sucks. I can't move off of Uber or Airbnb marketplaces and social media platforms or social platforms have these network effects. And no matter how cheap software gets to make, network effects are still there.

SPEAKER 4

59:19Yeah. No, that's right. Network effects are there. And then to your point about the cost to make software, I mean, one analogy, and I know it's not perfect, but it's sort of like in manufacturing, like you spend a lot of money to make a mold,

SPEAKER 2

59:30right?

SPEAKER 4

59:30So if you're going to make plastic chairs, like you spend maybe a couple million dollars to make that mold, and then you make as many $5 chairs as you possibly can off the mold for it to pay back. And that's, you know, there are different analogies we can use to describe what's like.

SPEAKER 1

59:42Right, but now you can manufacture software, right?

SPEAKER 4

59:43But exactly. And there's something like I could just make the chair.

SPEAKER 1

59:46Right.

SPEAKER 4

59:46And then and that starts to change how we think about it.

SPEAKER 1

59:49And it's interesting. It's almost like the opposite of that analogy, because now I can make my own version of that chair just really cheaply with no mold. That's right. With all arms.

SPEAKER 4

59:58That's right. That's right. That's right. And it's a. And I think I think it's important that. We. sort of try to bring all of these developments in AI together because you are having these incredibly powerful foundation models. You're having a shift in like our ability to do things like code or do data analysis and things like that where just the cost to do those things are now going down.

SPEAKER 2

1:00:34Mm-hmm.

SPEAKER 4

1:00:36that marshaled well is a real gift. But of course, and I said I'd drag us here, so here we are, that's going to matter to labor a lot. And I think I want to bring us to labor for a couple of reasons. One, because I don't know that the mental models that at least the tech community or the AI community uses to talk about labor are right. But also that I think we... But we are in for something. We're in for something that's kind of shocking. But how do you... What do you do about that is not so obvious to me. So let me lay out my grievances. Where there is this idea that AI just gets more and more intelligent. And the critical part to this argument is to never say what intelligent means. And then to say, well, if it gets more intelligent and work is an exercise of intelligence, therefore all labor gets replaced. And then... On that basis to then make this big jump to saying, okay, here's what we need to do now that nobody is useful anymore. Nobody's economically productive. And then somebody inevitably raises their hand and says, what about cutting down trees? And like, we'll get robots for that. And so then the idea is that the end game is zero economic contribution on the part of individuals. Machines do everything or you have a tiny, tiny sliver who run the machines in there. And then we jumped to all of these ideas around, okay, well, are we all going to be lying on the beach and, you know, our benevolent billionaire overlords are going to feed us mango smoothies or are we going to have...

SPEAKER 2

1:02:13Like Wally.

SPEAKER 4

1:02:13Yeah, exactly. Or is it going to be something else? And my... confrontation, and I really, it feels, it deserves a confrontation, is that what this does not account for is that a significant swath of labor, whether it is within a job or just a purpose of jobs, are around decisions and risk. And so... If I make baseball caps and I need to go buy the fabric for the caps, I know nothing about baseball cap making, so I'm just going to run with this, but I need to get the fabric. And then I have three potential vendors who will sell me the thing. So now we have a procurement bond. What's this decision going to be? It's going to be on the basis of some factors like cost, shipment time, how long is the company winner, whatever data exists. And how would you or I make the decision on that is we'd probably meet the person who runs a fabric company or the representative and say, he seems shifty. We're not doing that. And then just sort of making a using our gut, but critically. owning the responsibility for the decision and the course question. Why am I saying this? Because that layer and also there's and then there's just a human interaction layer where we can automate the very easy stuff. But it should be pointed out that managers do not like having people on payroll. They never They would gladly fire everybody if they could, if they could keep like revenue at the same line, right? And so attempts to automate labor have been around for the entirety of my career. A lot of that was like this quasi-automation of just go to lower cost countries, then you have the lower cost worker, and then the idea of robotic process automation, all of that. We have seen all of these things. what you notice is certain categories that automate really easily. Certain categories that ought to be automatable don't automate easily, but critically you have humans in the mix. Why does this matter? Because if humans are in the mix, what you really are looking at is not like a 95, 99% unemployment rate. You're looking at just a deeply inflated one in which there are winners and losers in a society. And it looks completely different. And all of these solutions of the nobody has a job imply that we're all in the same boat. But we're not going to be in the same boat.

SPEAKER 1

1:04:35We're not all going to be in the same boat.

SPEAKER 4

1:04:36We're not going to be in the same boat.

SPEAKER 1

1:04:37That's very interesting. You're saying there's going to be this kind of stratified effect where different people are affected different ways by job loss, like all industrialization in a way. software engineers probably will be impacted quite a lot because code is actually very automatable because it's in this closed loop system. It's really interesting. Something I hear from you is I think there are maybe two things that make humans useful. One is liability and the second one is information. So in this example about baseball caps, you were like, okay, if I mess up procurement, I am to blame. Or if I'm a doctor and I mess up the surgery, I am to blame. There's a person who's liable and they can be legally held accountable. If the machine is ultimately to blame, this is actually really annoying. I can't hold this machine accountable. They can't be punished. I can't fire them. I guess I could get a different machine, but then, like, I'm the one who's responsible. That sucks.

SPEAKER 4

1:05:40But also, like... back to all the dynamics we were talking about before, I go to my doctor. It's not that I'm like sitting there saying, I will sue you if you muck this up. It's rather that this mechanism where they've, you know, they've- It incentivizes the doctor. It incentivizes them, but it also is a structure of, you could imagine like even not hard incentives, but just softer ones of that person gets up in the morning and says, I am responsible to my patient. The machine is responsible to know. And the person who owns that is not thinking about individual responsibilities, but is probably thinking about aggregate ones. And any of us who've kicked around in the business world know that these things then just become measures of risk, not even of obligations to individuals.

SPEAKER 1

1:06:22This is really important, aggregates versus individuals. Yes. things get suddenly worse, there's a great book called Seeing Like a State. And the core idea is that when you are a state, you're governing a state, you have to collect data, and that data gets collected in aggregates. And because you can only see data in aggregates, you do things, you take actions that actually make individual lives a lot worse, but make the aggregates look better. here you're saying managers might make decisions in aggregates that make individual lives a lot worse or individual impacts on patients a lot worse, but in aggregate, it looks a lot better. And I think it's actually really important to point out here that like the, When we look at individuals, we're looking at anecdotes, and that's a really different type of information than when we're looking at aggregate measures where we're looking at statistics. Even I as CEO struggle with this. It's why kings go and like disguise themselves as a villager and go like talk to villagers to get the anecdotes because like I as CEO get really bad anecdotal information from people. Instead, I get a lot of aggregates, and that actually makes it really hard for me to make good decisions. Yeah. That's one thing. Liability is one thing. And maybe kind of to your point, like one way in which humans are really valuable is that we are able to... kind of be responsible for an individual person, individual case, individual situation. It's really about time scale. Like I don't think all of our jobs will be automated in 10 years. But I think in 50 years, that's still within my lifetime. Like that's not super crazy. Look at the change that's happened the last 50 years or 100 years. Yeah. What like the implication is, is kind of, okay, if we are building these systems, and they are going to have these effects where a lot of people lose their jobs, and it's easier for the managerial class to do things, and then like... really the challenge is, okay, not all jobs will get automated immediately, but how do we build a society where people are free and have power? Because there is this kind of like leakiness from labor to capital of power.

SPEAKER 4

1:08:40Yeah, absolutely, absolutely. And I think one of these things is that I think maybe why the time horizons matter to me is because longer time horizons are where the substitutive activities start to come in. And then we start to generate new economic activity. And I'm really wary of like something can happen on a 10-year time horizon. That's just insane. Like how do you –

SPEAKER 1

1:09:01Probably programming will get automated on a 10-year time horizon.

SPEAKER 4

1:09:05I think it's going to be really interesting because sort of the non-engineer's perspective on this one is that I think we're going to see a stratification of skill level.

SPEAKER 2

1:09:18Mm-hmm.

SPEAKER 4

1:09:19And okay, hot take. I think we're going to see an emergent category of developers who are not particularly, I hate using low skill, high skill. I just mean like just sort of like, they're not the sort of people who are inventing new programming language. They're the guy who would like make your website, do whatever. These are important categories, but the things that LLMs can do very easily, but that until software sort of kicks in to make it easy for a lay person to use the LLMs to do that, they're going to act almost as like a translation layer. So they're not really going to be developers. They're going to be more of I know enough about what like a web stack looks like.

SPEAKER 1

1:09:54That I can turn a web stack into something for someone. Exactly.

SPEAKER 4

1:09:57So I think that's going to come. But that's going to kind of flare up and then drop sort of the way that like web developers were hot. And then it sort of became either like a highly skilled front end rule or you have like Webflow. Or, you know, Squarespace and all of that. And then I think what we're going to see is sort of the artisanal middle is going to go away. And then you'll have like the really high caliber engineers who understand how systems work become absolutely vital. And then they're augmented by these systems, but they're basically CTOing. Everything. Yeah, everything.

SPEAKER 1

1:10:32They're like a lot more CTOs. Yeah, exactly. I think it's not unreasonable. And I challenge your like non-engineer hat because you are one of our active users of our product, which is a coding tool. But I think like it seems reasonable. And maybe a model, a simple model for thinking about this is like, I think there's always a Pareto front of task difficulty and how well the task works. And as tasks get more difficult, it requires a lot more capability or skill to make the task work. And so lots of easy tasks will get automated, and it'll be much easier to make web apps, and it'll be much easier to make things like that. But we'll probably see these much more complex, almost grown software systems that someone is managing. And in software, I think this is a place where I'm actually, one of my deeply optimistic views of the, you know, we started this conversation with problems. But one deeply optimistic sense I have that is possible if timelines are slower, and if we can figure out how to make really good tools that are not just captured centrally, is that people can, learn actually how to like garden software for themselves. And that becomes a source of power where people can harness computing. We said like computation is power. People can harness this computation for themselves because like we all have laptops, we all have GPUs. Perhaps there's some way to allocate them more equitably. Now, because we own this laptop, this computation object, then we can use it. We can harness it to run a bunch of software, to grow a bunch of software that does really interesting things. And does more and more complex, interesting things for us. And like maybe inside of our jobs does more and more complex, interesting things as well. And so you might see many people losing jobs, but many people gaining this capacity to create software that does like really weird and unusual and I guess weird and unusual from today's perspective, but like different things, new things, more powerful things. And perhaps it's not like, you know, not all of it. I think there's a world in which it's not top-down automation but bottoms-up automation. Bottoms-up as in we are the ones that are automating our jobs away. I love automating my job.

SPEAKER 3

1:12:55Yeah, me too.

SPEAKER 1

1:12:57And then when we're the ones automating our jobs, we become personally more valuable. It doesn't solve the full problem and I think I'm still confused about the dynamics exactly.

SPEAKER 4

1:13:06I think you're right. I actually think there's going to be a really interesting near-term dynamic where There's something really beautiful about human ingenuity. You give somebody a tool and they figure out neat stuff. And one thing I think that will be really fun to watch is going to be somebody who had a job that involved a lot of sort of these manual tasks. And they're just figuring out how to automate it themselves. And they themselves actually become much more valuable to an employer. We'll watch people learn how to do that. There's sort of like this digital literacy that I think is going to at once... The education...

SPEAKER 1

1:13:40Lens. Yeah. This is really important, actually. Something that we think a lot about on the product side is, like, how do you teach someone who doesn't quite understand these software systems what's going on? And, like, how do you teach, like, if, you know, I was thinking, like, okay, if we think of agents as, like, top-down automation versus bottoms-up automation, it's actually really different. The way that these agents get implemented is really different. If I am told as CEO that this technology is going to automate my workers away and I can fire them, I'm going to do really different things as an internal process. I'm going to implement processes to measure what people are doing and then, like, try to take the stuff that they're doing and automate that stuff. Maybe this is what RPA, you know, is going to talk down.

SPEAKER 3

1:14:19That's basically what I was trying to do, right?

SPEAKER 1

1:14:20Exactly.

SPEAKER 3

1:14:20Especially, like, in financial services, paperwork, boom, boom, get them out of the way.

SPEAKER 1

1:14:24So, but if I'm told as CEO, hey, I have this technology and what it's gonna do is if you hand it to your workers and you teach them how to use it, it's gonna teach them how to use it itself. And your workers are gonna become much, much more effective because your workers will automate their own jobs.

SPEAKER 4

1:14:41That's a really different perspective. And it's a completely different one. And I think the thing is that it has to be real. And we have to find a way to make that real. Yeah, go ahead.

SPEAKER 1

1:14:52I just want to say, this is a place where we can make a lot of choices in building the technology that make this go one way or another. If you build the technology to not teach people kind of how to automate things or how to, you know, it's kind of like when we are building prosumer products, you can either build for the buyer or for the user. If you build for the buyer, then you're building something that is built to automate people. And if you're building for the user, you're building something that's trying to teach the user how to use it. And yeah, anyway, it's a choice.

SPEAKER 4

1:15:23Yeah, it's a choice, but it's also an interesting choice because I don't know that as an economic matter that we know that it is better, for instance, for a large company to try to automate away its employees versus have higher productivity employees. I mean, the thing everybody wants is higher productivity employees. For more revenue. Exactly. And if you can get that, that is a boon. And a more productive economy is actually generative. And so, again, I don't want to be Pollyannish, but like, yeah. So go ahead.

SPEAKER 1

1:15:53This is really interesting because- One of the things that people say, I experience AI as not being very good at, is it doesn't have very good taste. In that it doesn't know what I want. It doesn't know what other people want that I know about. Yeah. As a result, it's not that good. I don't trust it to make certain decisions. I don't trust it to write on my behalf very well. I use it, but it's like the taste is not very good. Exactly. The reason why it doesn't have good taste is because it's not in my head. It does not know about my internal experience. And I have a lot more context than it does about me and my situation. And so there is a potential here where... You know, your point about economically it's not clear if it's better to make your workers more productive or to automate them away, like cost cut. If people are better at spotting opportunities than AI systems, then it is possible that it's economically better to make your workers more productive. If systems are better at spotting opportunities than people, then maybe it's the opposite.

SPEAKER 4

1:17:03But I really do like this framing. I mean, and I think this is something that, like, policy leaders have to take seriously. I think, you know, I tend to find that in my conversations with lawmakers that they are sophisticated. It's just coming at them fast. But what is very hard is sort of the concentrated or the concerted effort of, like... managers and workers and governments and technologists and all of this to build these things in a useful way. And I almost feel that to some extent we have to get that coordination right in some ways, which the center would almost have to be government because nobody else has an accountability to the people. But at the same time, this is where builders really matter.

SPEAKER 1

1:17:50Mm-hmm.

SPEAKER 4

1:17:51Because if what you- We can choose to build one of these things preferentially.

SPEAKER 1

1:17:54What are we going to build?

SPEAKER 4

1:17:55Like if you don't build the surveillance system, it doesn't exist. Or at least that one doesn't exist. And likewise, if you build a really useful tool that makes somebody more productive. I mean, I think like, you know,

SPEAKER 1

1:18:06so- If you choose to build things that teach people things versus choose to build things that don't teach people things. If you choose to build things that are anti-surveillance by getting people out of surveillance systems. If you choose to build things that let people get their data into their own system. There's a lot of choice in what we build.

SPEAKER 4

1:18:21I love spreadsheets. I'm not saying I want to spend all my time in them, but when you need a spreadsheet, that's really powerful. It's true. I've heard it described that Excel just basically made programming available to the wider world. You have a bunch of people doing crazy stuff in Excel and they're like, oh, I can't program. They're like, what is that macro?

SPEAKER 2

1:18:40And,

SPEAKER 4

1:18:40you know, and it's incredible what people are able to do with systems that build up their productivity. And that may be, I wonder if this isn't really good enough.

SPEAKER 1

1:18:47I want to reframe it. I think it's not about productivity. Okay. It may be somewhat about productivity, but like it's about, and this goes to the fourth category of like psychic damage. Yeah. It's about unlocking people's ability to spot opportunities and to learn and to kind of like become someone that is innovative and able to find opportunities opportunities and able to become more like, I guess you maybe measure it economically as productivity, but I don't know, like when thinking from the builder perspective, like when I'm building a product, like, I guess what I want to think about is like, how do I enable people to actually learn how to use these tools, do their jobs better, see opportunities in the world, change, like there's a lot of upskilling or change or like different, different, different skilling.

SPEAKER 4

1:19:39I think you're seeing something really important here around productivity, at least to me on an emotional and personal.

SPEAKER 1

1:19:46Yeah. It's not about productivity because productivity measures the output, but it doesn't measure how you get there.

SPEAKER 2

1:19:51Yeah.

SPEAKER 1

1:19:51It doesn't talk about how you get there. And you can easily, if you measure just productivity, it's easy to make an argument that an agent is more productive in so many different ways. But if you measure productivity of your workers, it's also easy to make an argument that workers are hopeless. You know, they're not becoming more productive. It's useless. But in fact, maybe their tools are just not very encouraging. What is really weird and interesting about LLMs is that you can make tools that are very encouraging, that can be very deeply empowering. This goes to your spreadsheet example.

SPEAKER 2

1:20:23Yeah.

SPEAKER 1

1:20:24where a spreadsheet is actually one of the most deeply empowering things that exists because it has this vast legibility.

SPEAKER 4

1:20:33Yeah, in working life. Exactly.

SPEAKER 1

1:20:35It's real time. It's live. You can see the whole system as you're building it. And I think there's a lot of invention that is necessary for making kind of the deep capabilities of AI actually accessible to people in a way that... It kind of harkens back to 1970s Doug Engelbart, you know, personal computing. Like how do you let people see so that they can learn?

SPEAKER 4

1:20:59I also want to say something about productivity, which is like I don't know anybody who's like I'm highly productive and they're proud of it or like – or someone who's well-adjusted who says that. And I think this is important because like I – I do not measure myself or the people in my life on the basis of a product. Nobody's eulogy is like... This was the most productive person. He was a highly productive individual who helped improve the company's ROI on this project. And it's not what we do. And yet that productivity is going to be a determinant of other things. Back to your earlier points about what does it mean to be economically eclipsed and all of these things. And so there's also something about... becoming more productive by becoming more able in what you're doing. That I show up to work and I have these tools that make me more effective at the thing that I care about doing.

SPEAKER 1

1:22:00I like this a lot. Becoming more able is a way that we can think about what like the potential of the technology is, is that it helps people become more able.

SPEAKER 4

1:22:10Yeah.

SPEAKER 1

1:22:11But it has to be built a certain way to do that.

SPEAKER 4

1:22:13Yeah. Yeah. And I think there are challenges around productivity, which is that you need healthy and vibrant economies that will then reward productivity. Because a productive economy is, if you have like one firm that's productive, then it just sort of takes over the others and then the others get wiped out, but you don't really have significant growth. But if you have, everyone productive, then you have competition, and then you have this intense growth. And I'm not sure how economists would present something like Silicon Valley, but I suspect that that's an example of highly productive and highly competitive. And a function of the fact that this is where so much tech talent resides, right? And so then that concentration of this productive accelerant is then, and yeah, there may be something in that for what we can analogize or extend to the workforce because there is, And again, like, you know, you look at, you look at people, you know, you go to school, you study the thing that you care about, you go into the workforce, you want to have a job, your job is a big part of your life, it is not the totality of who you are, you, and then you go to this job. And then One really weird thing about the way we talk about AI is we're like, okay, then you don't matter anymore. And I think that that framing is normatively wrong. You still matter. It does not matter whether or not you can get a job or not. But two, I just actually think practically it is not a correct rendition. And so then our solutions have to look different. Because again, like, again, the will we, you know, the startups are all in a tizzy right now about the way that a certain R&D tax credit gets applied. And I'm on the side of the startups, I think. I haven't dealt too deeply. But basically it's about how you amortize the cost of software engineering, like on your way to figuring out your revenue. Yeah. But what's really interesting is that the – are you going to give a tax advantage to capital in the case of corporations automating the stuffing out of things? Or do you tax advantage labor? Like what are also the things – what are the incentives that you structure as a society? Like what do you encourage? Because that also – like you start to change these societal incentives. And I don't know what the answers are, but we have these incentives.

SPEAKER 1

1:24:36really interesting and there's like a concrete problem or question here that could be solved which is like or someone could work on which is what is a mechanism that incentivizes increasing the ableness of labor that fundamentally maybe it's about productivity ultimately but it's actually about like this kind of like ableness of the workforce yeah um such that like labor maybe actually does kind of in some ways own their own means of production, like starts to become able to own their own means of production.

SPEAKER 4

1:25:07Take something like oil pipelines, right? Like I think right now there's a lot of human inspection of them where with time, I think it's going to be sensors. It's going to be, you know, drones to like, you know, you have a detection of something going wrong. You have a drone to film it or maybe you may still have some human labor,

SPEAKER 1

1:25:24but there's less of it.

SPEAKER 4

1:25:25Exactly. I do not want to say there aren't going to be labor disruptions. I think there are going to be potentially very large ones. And so the thing that we have to, as builders, build towards are systems that are... that are additive.

SPEAKER 1

1:25:44Yeah, systems that enable people.

SPEAKER 4

1:25:46Yeah, and they make us more effective. Right. The reason you replace an employee with, like, a machine is because then you get, like, an insane productive return. Right. But if you can't do that and you could get a really good productivity increase off of your employee base, well, then that's a wonderful thing. And you, as someone who works for a company, that that's a great thing for you as well. You get to be a contributor. But then it's this... Where I start to get really worried is around the – like I've done something for a long time in a particular way. And then – It's hard to teach someone to – It's hard to teach.

SPEAKER 1

1:26:20It's hard to change. Yeah, this is why I think like the like enabling or that piece is as a builder the most important. I actually want to bring this back because I think like I am in agreement with you on the short term. Yeah, yeah. Medium term even maybe. And then the long term – Like I think everyone does have to become part of the capital class. Like in the short term and the median term, what you're saying and what I'm saying is we can have solutions that like enable people to be part of the labor class for much longer. And for that labor class to be thick for much longer, like big for much longer and like sustainable for much longer. And that slows things down perhaps enough that allows us to build laws, to catch up morally, to think about these things. And like that's actually really important. I agree. And like that's where we can have differential impact. And over the long term, let's say like 50 years, 100 years, it does certainly seem like these systems are improving at a rate where they can collect enough data either in the digital world or the physical world where like – we will be able to do a lot of things in an automated way that aren't done today. And so like the labor class will thin and we probably do want this other solution where people have the ability to own their own means of production and have a stable, like that's kind of, to me, the only like long-term stable equilibrium where like people have things that produce for them and they don't have to worry about it so much and now they can live their own lives. Like when I'm in the capital class, I don't have to think about working And finding a job and like making money. I can do what I want with the capital I have. Sometimes I may make bad choices and end up losing it. And then I need some help from the government, get myself set back up. I can start a different business. You know, this is kind of like a small business owner situation. And that world doesn't seem too bad. I'm not sure how to get there. But I want to bring us back to freedom.

SPEAKER 2

1:28:01Yeah.

SPEAKER 1

1:28:03That's a world, you know, that's a very optimistic world in which potentially people are a lot more free to spend our time the way that we want. But it feels like in the world I've just pointed, we've just painted where people have these like capital producing objects that they own is very different than the world that we see being painted by technologists and others today. where it's a utopia that feels very much like a WALL-E utopia where people are somewhat infantilized and people are – world is abundant but perhaps we're not free.

SPEAKER 4

1:28:38I hate the word abundant because it – I mean I love abundance. But its usage here is it's not right because what do you mean by abundant? Like as in – so if a small number – Food and food? Pardon me?

SPEAKER 1

1:28:54I have food. I won't die. I have housing. Basic needs are met. Knowledge is accessible.

SPEAKER 4

1:29:00I don't even buy that we'll get to an abundant world in that regard because what you're really going to do is aggregate wealth, back to this point of seeing a state aggregate wealth, will shoot up dramatically. It's going to be hyper-concentrated. The obligations to those who don't hold it... are going to be nil.

SPEAKER 2

1:29:14Much lower.

SPEAKER 4

1:29:15I mean, what do you owe them? And one of the really interesting dynamics that we've observed is when wealth concentrates in these extreme ways, an odd detachment starts to set in. And so, like, it's such a perverse dream to me to count on the benefits of people who are so... from the realities of regular life or the wealth that they've been able to concentrate.

SPEAKER 1

1:29:40Okay. But like, yes, I think there's like one world in which we have this extreme concentration of wealth. Very plausible.

SPEAKER 2

1:29:47Yeah.

SPEAKER 1

1:29:48But it's assuming no distribution. It's assuming that this kind of like labor distribution we just talked about is not necessarily happening. We don't keep the labor class... useful for longer, like, you know, the tools we build are very concentrating. I want to talk about what you mean when you think about freedom. What is the world you're fighting for? And the reason I want to talk about this is because I want to kind of come back to and maybe end with what it means to be free in a society where there's AI and powerful AI systems and powerful, potentially powerful other actors. Like maybe it's possible to have powerful actors and still be free. Maybe there is a way to construct that world.

SPEAKER 4

1:30:32So I'm going to challenge that AI is new as a technology, but as a social and political dynamic to live in a society with powerful entities. There's nothing new about it. And I think this is really important because the things that make us free to have laws, rights as individuals, their application, representation, like just the wonder of, of sort of the, of, of modern liberal democracy and when it works and, and also its capacity for self-correction, right? This is remarkable. Like I think of, and I want to, and this is really worth highlighting that the difference between a totalitarian regime is when something bad happens there, it's the thing that happened, in a functioning liberal democracy, it happened, but it was wrong. And there is a correction.

SPEAKER 1

1:31:29Yeah, in theory, a liberal democracy can be anti-fragile.

SPEAKER 4

1:31:31That's right. That's right. And for what it's worth, our liberal democracies have been and have been for a very long time. And that part of the... I don't know the exact construction of how I would place them within the anti-fragility cycle, but we don't have to give up, and even when things get bad. But the reason I say that is that the things that make us free... are, you know, if you go back to Isaiah Berlin and positive and negative liberty, sort of the ability to realize your potential and the ability not to get whacked in the head with a stick and those two categories, well, we can continue to work on those. I mean, I think what we need to do is look at where within sort of the lawless spaces things are uncovered, where will AI exacerbate that to build in those protections? And then in terms of realizing what's possible in our lives, I mean, I think one, accepting the idea that like... Freedom is not an instrumental quality. And what I mean by that is freedom is not something that gets justified because then you go and invent the airplane. Like freedom is beautiful because you can sit on your couch and pick your nose.

SPEAKER 1

1:32:46Freedom is an end.

SPEAKER 4

1:32:47Yeah, it is an end in and of itself. It does not depend on other things.

SPEAKER 1

1:32:50Actually, before we continue, I wanna clarify your definition of positive liberty and negative liberty because it's not necessarily, it's not something I ever thought about before you told me about it. But positive liberty, this idea that you can do things, what are you enabled to do? Negative liberty, this idea that you're protected from being whacked on the head with a stick.

SPEAKER 4

1:33:08That's right.

SPEAKER 1

1:33:10And so you're talking about positive liberty.

SPEAKER 4

1:33:12And we always need both. I mean, the brilliance of this construction that people would get lost when talking about freedom. And they say, well, am I really free if I can't open an ice cream manufacturing facility? And the responsible, well, nobody's holding you back. You just don't know anything about ice cream. And if you look at modern life, if you look at... legitimate and illegitimate grievances in modern politics. They are often about the sense of a constrained positive liberty and an intruded upon negative liberty. And so part of what we do have to figure out as a society is let us, you know, to some extent we have to manage the extremes, but are we actually tending, forget AI, like are we actually tending to the sense of a broad societal sense that we're free? And I think that Within that context, then we ask, what is AI doing and how is it modifying our society? That's why that, you know, taking seriously sort of this frustration with the weirdness of the discourse around AI is that if we don't characterize it, If we don't characterize it honestly, then we don't have the ability to work with it.

SPEAKER 1

1:34:27We must characterize it honestly so that we can actually increase our positive liberty and also actually protect our liberties against negative liberties.

SPEAKER 4

1:34:40Because a lot of the story, the narrative that will come from the people who have invested huge sums of money into AI. And look, we're a company. We're in this game. But there is this look at all the positive liberty benefits coming your way.

SPEAKER 1

1:34:56Don't get in the way. Don't get in the way. Ignore the negative liberties.

SPEAKER 4

1:35:00Exactly.

SPEAKER 1

1:35:00And I think this is actually really important. And the reason why I wanted to circle back to freedom, because I think the way you think about freedom is a very deep... way that in the tech industry, my experience of the way people talk about freedom is about lawlessness, actually. And the way you talk about freedom is freedom is about this deep enablement and this deep protection. And that that's what kind of world we wanna build is one in which humans are deeply protected and deeply enabled, and that's what it means to be free.

SPEAKER 4

1:35:34Yeah. I mean, I love that. I'm going to use those words. But it's a, and I think that, look, I understand, you know, if you think of like the roots of the valley to some extent, if you ignore kind of the defense funding, you know, a lot of its origin was the, well, we're just going to, we're going to break free from the constraints of what's,

SPEAKER 2

1:35:58you know, around us.

SPEAKER 4

1:35:59And so I understand that as an ethos, but it's no longer just, you know, building personal computers in a garage anymore.

SPEAKER 1

1:36:06Right, now that we are reshaping society, we have to rethink.

SPEAKER 4

1:36:11And those obligations, they're rich, but they're also beautiful, right? I mean, this is the thing is that if we can really think about what our neighbors need and furnish that and recognize that, that, that is a, you know, there's, there's a real spiritual cost to our present moment where, where the factions are constantly warring. And I don't want to pretend that there was a golden age where people, you know, kissed each other on the way to the voting booths, but there's, there is a, but, but this, this, you know, technology exacerbated the way we see each other, the way, Yeah,

SPEAKER 1

1:36:40we think it's other people that's the problem, but I think it's technology that's the problem in a lot of ways.

SPEAKER 4

1:36:44But that's right. That's right. I mean, and one of the things that I've experienced on a regular basis is that like somebody expresses a bonkers opinion and you sit down with them and you talk. And they're lovely humans. And the fact is we are surrounded by lovely humans. And I think it's really important that we resist the urge to vilify the people who have brought us to a place that we may not be thrilled about politically. But there is a real responsibility that if you are building systems like the one that we are building today, you are not only in this race against other companies to build a successful business, you are also in a race against the other possible ways that these things might be built. And it's incumbent on us to not just build something that is better, but also to win. And to have that paradigm win. And so... I think, yeah, I don't have a lot of patience for this sort of like, oh, technology is going to eat us all, let's give up and let's just keep training our models, right? Like it just feels like an unnecessary abdication.

SPEAKER 1

1:37:48Yeah, I'm going to end it here because I think this is actually illustrates the beautiful point, which is that as technologists... The opportunity we have today is to create technologies and build them in a way that deeply respects the actual freedom that people can have, which is this deep enablement and deep protection. And not to create technologies for the purpose of lawlessness, this like againstness, contrarian view. So the opportunity is creating technologies that enable humanity to be deeply free and not lawless, but protected and enabled. And that's what we mean. So that's what we can do.

SPEAKER 3

1:38:25I love that.

SPEAKER 1

1:38:26Okay. Well, thank you so much, Matt. This was awesome.

SPEAKER 4

1:38:29This was fun.

SPEAKER 1

1:38:29And a great restart.

SPEAKER 4

1:38:32Absolutely.

From lawless spaces to true liberty: rethinking AI's role in society

Who will actually hold power in the age of intelligent machines?

Kanjun Qiu and Matt Boulos

Aug 05, 2025

Transcript

Welcome back to Generally Intelligent! We’re excited to relaunch this podcast on Substack, and in video. Our episodes still feature thoughtful conversations on building AI, but with an expanded lens on its economic, societal, political, and human impacts.

Subscribe for conversations with builders and thinkers on what future we want, and how to build the technologies and systems to get us there.

Matt Boulos leads policy and safety at Imbue, where he shapes the responsible development of AI coding tools that make software creation broadly accessible. His work centers on understanding what technological power means for individual liberty and advocates for the legal and institutional frameworks we need to protect our freedom. Matt is a lawyer, computer scientist, and founder.

In this conversation, Matt and Kanjun discuss:

AI’s four core challenges

Empowering bad actors
Transferring power from labor to capital
Reducing resistibility
Psychic damage of disempowerment

Governing lawless digital spaces
Why abundance is not enough without liberty
Freedom as deep enablement and deep protection
The role of technologists in shaping society

Timestamps

03:13 The complex landscape of AI conversations

06:11 Understanding AI's core challenges

08:57 The transfer of power from labor to capital

11:51 Resistibility and human agency

15:00 The dual nature of technology

18:01 The invisible dynamics of digital spaces

23:51 Lawless spaces

27:01 The future of work and economic stability

40:05 Privacy laws and digital rights

44:07 Code as regulator

54:12 Interoperability and user control

01:07:05 Aggregates vs. individuals

01:14:43 Bottom-up vs. top-down automation

01:20:49 Optimizing for increased ability rather than increased productivity

01:23:11 Economic implications of AI

01:26:54 Building systems for empowerment

01:29:22 Freedom as deep enablement and deep protection

Transcript

Kanjun Qiu (00:21)

Welcome back to Generally Intelligent. My name is Kanjun Qiu. I'm the CEO of Imbue. And we have with us Matt Boulos, our Head of Policy.

We started this podcast back in 2020 when we were trying to understand from researchers how far this generation of LLMs would go. The podcast has succeeded far beyond what we expected. Many of our early guests went on to have huge impacts on the field, and AI has gone from this niche thing to a household name everyone's talked about.

But since it's become so ubiquitous, we've started to realize something strange. The conversations in public are really weird. We have one AI CEO saying they're going to replace all of our jobs, but they're distributing intelligence, so that's good. And another AI CEO who's worried that it's going to kill us all, but it'll also give us tutors in India and new medicines, and so that's okay.

But where is the serious conversation about the real costs and benefits of this technology, the real economic, societal, political, and very human impacts that AI is going to have on our lives?

Generally Intelligent—this podcast and this conversation—is the start of that. We want this to be a space for us to have serious cross-disciplinary conversations about AI so that we can make changes. We can talk about different economic mechanisms, different ways to build technology, so that we can create the future that we want.

Because today, it's not too late. We can still change how this technology shapes society. And if we wait too many years, that's not going to be the case anymore.

So let's dive in.

Matt Boulos (02:45)

You've put a lot of thought into thinking about what are the core challenges that AI brings. Why don't you walk us through what you think are what you see as the main areas that we need to take seriously if we're going to address AI's impacts?

Kanjun Qiu (03:44)

Sometimes it's so overwhelming because people talk about all of these different problems as a whole smorgasbord from sycophancy to how AI might take all of our jobs to how it might take over the world. So, a way that I think about the problems is to bucket them into four categories based on the mechanism of action by which the system is acting.

One, empowerment of bad actors. The core mechanism is, the power of actors who might do damage goes up. It's a technology that gives a lot more capability, and now various people who couldn't wield this capability before can.

And I actually lump both AI takeover—AI systems taking over and dominating humans—as well as terrorism in that category, because the mechanism of action is the same. If AI is taking over, that just means that AI is taking a lot of this power and then doing negative things to humans. And same with terrorists or authoritarian governments.

The reason why it's helpful to think about that mechanism of action is that it's very generative for solutions. When I think about actors who are anti-social, in the solution space, there are a couple of things I can do. One, I can prevent anti-social actors from getting that power. Let's look at which actors exist—governments, individuals, the AI systems themselves—and then look at how we can prevent them from getting power. That might be all forms of know your customer laws or safety research or things like that.

On the flip side, another way to make things more resilient is to make the world safer to bad actions like this. Maybe in that camp is better surveillance of the creation of biological artifacts so that we can prevent viruses, or inventing a universal antiviral that would actually remove a whole class of dangerous problems. I actually think this category is very well talked about often and the important thing is that it is just one category and that many solutions actually solve for many different of these actors and the problems that they pose.

Kanjun Qiu (06:47)

The second category I think of as transferring power from labor to capital, the capital-L Labor to capital-C Capital in the Marx view. As labor becomes less powerful because we are less valuable and capital gains power, what happens?

Most of us are in the labor class. We do not own the factors of production. We work for wages. And this is a technology that’s starting to do things that we currently do wage work for. So what happens to all of us who work for wages?

There's the immediate, somewhat alarming effect of that: losing jobs. But there's, to me, the long-term, somewhat alarming effect of this, which is that you have this constant power transfer from labor to capital that is forever.

Matt Boulos (07:51)

There is something really quite striking if the ability to be productive depends on capital. This is a really abstract way of saying, I show up to work and the capital I’m bringing is my laptop, but for the most part, I'm bringing the labor. Imagining this world, maybe the day's gonna come that the laptop matters way more than I do, and it’s a question of who owns it.

Kanjun Qiu (08:41)

That's a really good way of putting it: the transfer of power from labor to capital is equivalent to the transfer of usefulness from me to my laptop. So what happens in a world where the laptop's way more useful than I am?

Matt Boulos (08:54)

I've never looked suspiciously at this thing before.

Kanjun Qiu (08:57)

What happens in that world is not just economic. It's not just that I get paid less. Maybe I wield my laptop so therefore I still continue to get paid some. But in theory, the company owns my laptop, so I may not get paid at all.

But the second effect of it is political. Part of the reason why we have political power is because our government depends on us to fund it. And there are a lot of countries that don't depend on humans. They depend on natural resources like natural gas or oil: the UAE, Russia. And they have a lot less incentive to treat their people well in the same way as we maybe in America do. So I'm somewhat concerned about the kind of loss of political power that we'll have because of our loss of economic power.

Capital can now just use capital—use AI—to produce more capital, and there's this reinforcing loop.

Kanjun Qiu (09:59)

The third category, which I haven't heard that many people talk about, is your idea of resistibility. In political philosophy, there's this idea of resistibility: how well can you resist laws that don't serve you?

In America, we have fairly high resistibility. The civil rights movement was a good example of that, where you could actually disobey, have civil disobedience, and then change the laws. There are countries that have very low resistibility, like China as a surveillance state. And one thing that we're concerned about is going into a future where the resistibility of humans against automated systems—either controlled by themselves or controlled by other people—is much lower. We lose our power. So, the core mechanism here is a transfer of power from people to automated systems and the people who control them.

There are a lot of examples of low resistibility today. For example, we have very little ability to resist our social media notifications. We can turn them off, but we also have very little ability to resist our social media algorithms or news algorithms or control the news that we see, that we want to see. There are ways of opting out, but I would consider it a fairly low resistibility environment.

And as we go into a future that has a lot more automated systems—agents that are doing things automatically—that's something that's really important to consider. Now, other people are going to have agents that do things like spam call you constantly, or try to convince you on a website to buy something you don't need, or try to convince you to give them data that they can resell. Especially given what we see about current capabilities, it’s not clear that we have anything in place that addresses that.

The fourth category is how it affects us as people to live in a society where we don't have very much power—we don't have power economically; we don't have power to resist things. We end up disempowered and, in the best case, infantilized.

That is scary because there is a deep sense of learned helplessness that happens as we lose power. There's a great study by Lisa Kahn about how college grads who graduate in a recession have lower wages for the rest of their lives relative to college grads who graduate just a year or two after, which is super crazy. You term this “psychic damage,” which I really like: damaging our own perception of how capable we are and can be in the world. And I think this is really sad. There's this spiritual damage that we don't talk about, which is about human potential, about what people can be. What we want is for AI to expand our potential and expand what humans and humanity can be, but there are all of these effects that, in this default path that we're on, seems like it's going to go against that.

Matt Boulos (13:21)

I want to run with the last thing you said because... I guess I have to come out with this. I'm old enough that I have a memory of my life that's pre-internet.

Kanjun

That is old.

Matt:

I remember being in grade school and Mr. Gen, the computer teacher, knew that I was really bored, so he would pull me out of class and then we would pretend that I was learning, but we were just trying out new software. One day he's like, “Let me tell you about the thing called bulletin board systems. There's somebody else on this computer.” I’m like, “Where are they?” And he's like, “They're in another country!” It was wild and so hopeful.

I think often these days about my parents calling their parents after they immigrated to Canada. They'd get these crap calling cards that would cut out and were choppy and super expensive. Now, I FaceTime my mom and she's like, “I have to go now.” I'm like, “how about we hang on for a little while longer?” We're taking for granted the ability to see each other, hear each other.

So you have all this incredible potential, and it's beautiful, and it's real. Our machines can augment us and we love tools. I don't understand why you could say, “I love my pen that I write with, but I don't like my laptop.”

But at the same time, we know that technology has been this very mixed force in our lives, from the capacity to surveil people to predatory mechanisms around how we communicate. One of the things that I have felt is really important to bring to the conversation is that talking about AI as good or bad is almost silly. It's like saying trees are good or bad. You plant it right next to your foundation, that was a bad move; you step out into a clean city, and it's the most wonderful thing. There are, of course, limits to that analogy, but there is something really profound about taking the complexity seriously.

Kanjun Qiu (15:47)

I was really struck by a quote that you said many years ago where you had just read something by Marshall McLuhan, and you rephrased what he said: that we adopt technology for its benefits and then we suffer its consequences. So, the important thing is to think about those consequences that we might suffer and see if we can get more of the benefits and less of the suffering.

Matt Boulos (16:17)

What was brilliant about McLuhan was that he'd clued into this dynamic where we adopt a technology and that it changes how we work and interact and our own capabilities so that it is no longer possible to detach and disentangle.

Kanjun Qiu (16:35)

We’re part of it; it's part of us.

Matt Boulos

Take something like social media. The public narrative on it is actually condescending and not correct. We're not just a bunch of dumb-dumbs who are sitting around swiping things because we don't have anything better to do—or at least not completely. The thing that's really happening is that our lives, our social lives, are on here. I get to see my friends' kids. I'm able to get a diet of things that matter to me or that entertain me. There's nothing at all wrong with that. We don't bear responsibility for the fact that these things are wildly addictive. But if our lives moved onto these platforms, then we're now in this really stuck position because if the platforms don't behave, then we are subject to that.

When I started law school, I decided to run an experiment. I had no phone, no computer; I had nothing. I'm like, “I'll be a nerdy monk and see how this one goes.” And one of the interesting things that happened was that I realized that nobody wants to call your landline to invite you to parties. Everyone wanted me to go to the parties, but they'd tell me about it the day after. They were like, “you weren't there!” “Well, you didn't invite me!” They're like, “oh, yeah.” Everyone was texting each other, and it was a simple thing. You can't just exit. That was significant. And then I got a phone.

Kanjun Qiu (18:01)

This is really important. You can't exit technology. We can't just be like the Amish because this technology is now so prevalent and so entwined in our social interactions and our lives.

Matt Boulos (18:19)

Let me give you the silliest example. My son is in a preschool that has a bit of Mandarin immersion, and I know no Mandarin at all. He's now singing “bá lǒ bò,” and it turns out there's this children's song about pulling radishes out of the ground. My son’s just marching around the house singing this song, and so we're like, “what's going on? Why did we enroll him in a language school that we don't understand?”

What do we do? We just hopped on an LLM and said, “okay, my son is singing this, can you tell me what this is?” All of a sudden, that world opens up, and it's beautiful. I want to challenge the idea that that is us conceding something. Why should I be trying not to do that?

Kanjun Qiu (18:56)

This is to something you said earlier, which is, when we get on social media platforms it’s very positive. But I was saying this earlier about resistibility, where it's very hard to resist these systems. What's going on here? It is something about the power inherent in technology.

Fundamentally, what is AI? It is doing computation. Computation is the same computation that our brains are doing. It's taking inputs, perceiving them, running them through some model of the world, outputting some things, and those outputs can be turned into actions.

There's something about social media where it is an AI agent. It's making decisions about what actions are being outputted, like what I see on my newsfeed. And as a result, I get different inputs.

So, I'm getting this other input or information that changes my model of the world. To your point about this symbiosis, the technology is making some decisions that is causing me to get different inputs into my world model and now my world model is getting morphed or transformed in a different direction. And that can be very positive. For example, you learn about your son and the song that he's singing and that expands your world model and helps you see reality more clearly. Perhaps the areas that I feel concerned about are the places where it actually kind of changes the way you see reality in a way that is more twisted.

Matt Boulos (21:00)

It feels manipulative. But I want to take a step back, because I've been like, foam finger, go technology, go. But this point about technology and power, thinking about its mechanisms are really important. Going back to my son singing his song: I went to this machine and asked it a question and it came back with an answer that I couldn't have gotten five years ago, or at least not so easily, not so smoothly.

But we're only talking about one part of that transaction. We're talking about me asking that and getting an answer. We're not talking about: Is this being logged? Does this system now know that my son is in a Mandarin immersion class? Are we gonna get like Mandarin worksheets offered up to us at the next interaction?

There's something about this digitally mediated world that is foreign and dangerous. And I think that's worth probing into.

Kanjun Qiu (22:11)

What do you think it is?

Matt Boulos (22:13)

I have a couple of different mental models. Let me play with two.

One is something that I call lawless spaces. Imagine a part of town where the police don't go. The rules, the norms—people are working them out, but they're not governed like the rest. And you cross that threshold into those places. Soon, things become possible. Maybe there's just a free spirit. It feels like the chaotic early days of the internet: people who relished in anonymity, not because they were up to trouble, but because there was something liberating in that. You can imagine that creating a heady atmosphere.

Kanjun Qiu (22:59)

Like the Wild West.

Matt Boulos (23:10)

Exactly. Your bank is like, “why shouldn't I be there too?” So your bank sets up shop, except when you step out of the bank holding a bag of coins, someone whacks you over the head and takes your coins. That space doesn't have the same rules, the same governance. It's not a perfect analogy, but there's a lot to be said for that.

I was talking to somebody about privacy and they're like, “That ship sailed.” And I said, “Well, why?” If we were having dinner and some dude comes up to you and just stands right next to you while you're talking and he's just writing down what you're saying, you give him a slap, send him out of the restaurant, right? We have that but we don't see it in the digital context, so we haven't learned to govern that.

Kanjun Qiu (23:47)

Do you think digital spaces are lawless because they're not visible?

Matt Boulos (23:51)

I think that's a huge part of it. The next thing I want to talk about is, what are the things that make the digital space really particular? One is that most of what happens in them is actually invisible to us. If I go to the neighborhood oracle and give them 10 bucks and say, “My son is singing this song, can you tell me what this is?”

He's gonna sit there, like, “Oh man, I know what it is!” He's gonna grunt and groan, write something down, and send me out. I go to an LLM and it’s a magic box. You and I may know how in theory it should work, but we don't actually know how it's implemented. We don't know what a year from now, five years from now is gonna happen.

Kanjun Qiu (24:44)

And you don't know what's being logged; you don't know what the company is doing with the data. There's a lot you can't see.

Matt Boulos (24:52)

There are really particular characteristics to just the digital world. It is easier to log than to not log.

Kanjun Qiu (25:00)

And it's safer in a lot of ways.

Matt Boulos

And there's an expectation. If something doesn't work, the customer is like, “I did X and it didn't work.” And you're like, “I have no idea what you did, I have no logs.” And they're like, “What are you doing? Are you junior grade developers?”

It is easy to log data. It is cheap to collect data. It is lucrative to collect data. Even before advanced models like LLMs, we could crunch through stupidly large amounts of data. So you have these reinforcing mechanisms that take us to really perverse outcomes.

We talk about surveillance. Surveillance enables an astonishing amount of bad stuff. Resistibility is something that you reach for when you're in conflict: something has gone wrong, and you have to resist it. But preceding that is legibility: do you even know who I am?

Kanjun Qiu (26:01)

This prompts for me: let's imagine we were making a world in which we're making everything digital into a physical manifestation. What I'm hearing you say is, we have ended up in this really weird default digital world, especially going into this AI future. It's weird because some defaults are weird. One default is that we log all data. A second default is that companies—I, running this company—can process that data however I want. Another weird thing is that now we have these AI systems and they can do lots of new magical things with that data. For example, take a photo and know where I am. As the person at the other end, I can't see any of it. And as a result, I actually don't have a mental model of it being a problem at all.

Matt Boulos (27:01)

We also didn't have an emotional response. We’re wired as human beings to recognize these things, and we can't react.

Kanjun Qiu (27:08)

We talk about something like surveillance and it's such an abstract concept. But if you were to turn surveillance into a physical manifestation, like this guy who is writing everything down in your conversation next to your table, then it would be like we have five people following us around everywhere and they're all logging different things about our lives and they're changing other stuff in our life based on what they're logging.

Matt Boulos (27:29)

This is where it starts to get wild because on one track, often when people are talking about the productivity benefits of AI and the labor impact, we're talking about labor substitution. But there's another way of thinking about the impact of AI within the labor context, which is that new work is being created. Let's take something like credit scores. Largely opaque systems, the financial services industry benefits from it, and the good faith argument is that we all benefit. If I'm an untrustworthy borrower, you shouldn't have to be paying rates to subsidize me, so we stratify on the basis of reliability or whatever terms they use to describe it, like credit worthiness.

But then you could start to shift the granularity of that. We could just collect all sorts of stuff. We could also experiment. We could collect data even if we’re not sure if it's relevant. Deny me a loan—who cares? I'm one data point. My life gets crushed, but they don't know about it because their system did it, and just move on and experiment. That dynamism that becomes possible is going to be potentially quite pernicious.

Kanjun Qiu (28:55)

When you say dynamism, what do you mean?

Matt Boulos (28:58)

You could have systems that are not stable anymore. There isn't a credit score. There's an algorithm that's constantly rewriting the rules. Why not? As long as it’s goal seeking against minimizing defaults, it doesn't matter how unfair it is. Often when we talk about unfairness—putting on my lawyer hat—we often talk about things like disparate impact, protected categories, that sort of thing. But what happens when it's arbitrary, what happens when it's large categories of society, what happens when it's not easily pinpointed? Again, the bad stuff is happening behind the veil, so we don't know.

I want to connect that to something earlier that you were talking about, when you were talking about the economic impacts, and you were saying, that destabilizes society. But also, when you live in a world where you are subject to all of these forces and you're helpless against them, it's not good for a person to feel that way. Think about the worst parts of childhood where adults are not taking you seriously, not letting you do something that you ought to be able to do, and then that becomes the dominant mode of adult life.

Kanjun Qiu (30:18)

It’s very disempowering.

Matt Boulos (30:22)

And this is preceding oppression. That by itself is destructive. And then you add to that malicious intent or malicious oversight, and it isn't a surprise that we live in an angry moment in our society. I don't have a lot of patience with the tech community sort of sitting around saying, “How could this be?” Well, I mean, you've been bloody architecting it for the last two decades. There is a reason why people feel disempowered because they are disempowered.

Kanjun Qiu (30:53)

They have no power to change a lot of things.

Matt Boulos (30:56)

How could you change any of these things? The thing with AI—and I think it's really important that we ground it—is that we have to recognize that all of these dynamics are in play. Then, we can ask how do you design, and how do you get to empowerment? Because we could also just sit here and be angry and walk away, but that's not going to help.

Kanjun Qiu (31:12)

Two things that came up as you talk about this. One is, narratives today about what kind of future is okay for humans. I think a lot of the futures that the tech industry talks about today are actually very disempowered. One type of future is like, we're going to live in a stable utopia where everyone's going to have anything at their fingertips and it's going to be okay. But it does not consider seriously these dynamics where people are being controlled by technology and the people who control technology.

A second thing that you pointed at was this notion of utopia being like “permanent undergrad” where you can be free and intellectually curious and it's really fun. But an undergrad is not an adult with the ability to fully manage their lives.

The kind of freedom that you're going for is for humans to be truly able to be fully adult and in the world themselves without being pushed upon by other forces, with the ability to push against those forces.

Matt Boulos (32:42)

Absolutely. What do we really want from our lives? It’s to be able to realize our capacities.

Kanjun Qiu (32:56)

And that involves growth and change and creation and being pushed down.

Matt Boulos (33:01)

Absolutely, and having a chance in all of it. One of the things that I've noticed within— again, not to take a piss on the tech community — but we'll talk about what is an ideal future, or what's an ideal life for someone to have, and that's just somebody projecting what they thought was interesting to them.

I have so many people in my life for whom the specifics of their job don't actually matter that much, as long as they can take care of their family and support their community in ways that are really meaningful to them. Those are rich, beautiful lives. And when the structures around a person erode that is when we start to see this real frustration emerge.

Kanjun Qiu (33:53)

People are frustrated because they feel like they don't have any levers to change the situation of their lives, and they don't like the situation they're in even though the world is abundant and they're fed. There's something missing about their sense of autonomy or freedom or their ability to make change. What I heard from lawless spaces is, it's partially a lack of legibility and partially a lack of levers of action. And if you had legibility for everyone and levers of action for everyone to be able to change their life circumstances, the institutions that aren't serving them, then maybe those two things would allow us to be able to have a little bit more autonomy and self-determination in our lives.

Matt Boulos (34:48)

It's kind of hard in our present moment to think about what a stable political or legal regime looks like in general, but there is a simple fact that we've, for centuries now, have figured out that it's not cool to steal somebody's money. It's not just that theft is wrong, but that the state can't do it, even if it's useful to the state. We say that's not right.

In conservative circles people talk a lot about debanking where just banks just turn you off digitally. It's not a frequent occurrence, but it happens, and has happened in response to political events. What's wild to me about it is that it just could not have been a thing a few decades ago; the bank would have to have literally stolen your money.

Kanjun Qiu (35:54)

Like a bank run.

Matt Boulos (35:56)

Or simply, you'd go to your bank branch and they’d be like, “we're not going to give you your money,” which is what debanking looks like. And you would say, “you stole my money!” Whereas debanking now is either just hitting a switch and they can access your money, or just saying, “here's your money, you're out of the financial system” in a way that is only possible in a digital world.

I have one belief that our laws and rules haven't caught up to digital reality, then AI accelerates digital reality to all of its conclusions.

Kanjun Qiu (36:32)

What I hear from what you're saying is, the digital world is enabling all these mechanisms like being able to turn off my access to my funds and the laws haven't caught up. The last 2,000 years of development in the legal system have been about physical reality.

That physical reality is actually happening in the digital world. You're giving all these physical analogs to it that are really interesting because they let us see the physical reality of what's happening, but somehow we haven't mapped that physical reality to the digital world. What would be required to make lawless spaces more lawful? Why have we not caught up? Is it a lack of knowledge? Is it the lack of visceral sense of what's going on?

Matt Boulos (37:37)

Each of the things you said feels to me like it's playing a part. We both understand computers really well, but when I hop on a website, it is not occurring to me that they are tracking the things that I'm doing.

Kanjun Qiu (38:05)

True! The other day I hit ‘accept cookies’ and then I was like, “what happens when I accept cookies? Oh shit, it can track me across multiple websites — that's crazy!”

Matt Boulos (38:13)

I drive people nuts when they look over my shoulder, because I always not only reject cookies, but I open the thing to make a point of deselecting everything. And the hilarity is, often these are just pop-ups that don't do anything and it just collects your data anyway.

Kanjun Qiu

That's kind of depressing. Thanks.

Matt Boulos

Yeah, it really is. You're welcome.

There's one sense that it's not tangible, no matter how sophisticated you are.

The other thing is, it is new. In world historical terms, we're talking about living in this regime for 10 years. It is not that long, right? Google trying to figure out how to monetize was something that happened basically in our adulthood. That's nuts. And then going from web to mobile, the introduction of apps — all this has happened really, really fast. Part of it is we haven't caught up.

The other somewhat more cynical thing is that, it turns out lawless spaces are awesome because they're so lucrative. If you can do stuff like surveil people and track them and price fix and all of the rest, you can do all sorts of astonishing things.

If you deal with it now, it's a lot easier than down the line. One easy answer is a good privacy law. We had good privacy rules 10-15 years ago. It wouldn't be so painful for the large tech platforms to unwind these privacy practices.

Kanjun Qiu (39:49)

But now it's entrenched. You have to change your entire infrastructure.

Matt Boulos (40:08)

We're talking infrastructure, business models, identity as an entity, and the market cap of these things. I don't want to grant sympathy to the surveillance practices, but this is a huge thing that we're going to have to ask of them. But we do have to ask it.

But there is an interesting question of what rights do we have that we have failed to translate, just as a practical matter? We already have these legal rights and we haven't brought them to these spaces. And then what are the new things that we have to figure out?

Larry Lessig's notion of code as regulator is really fun. What he does in this setup is that he points out that in every period of time, there's some regulating force that you have to contain if you want to protect liberty. In his construction, one that I share, we're progressively trying to increase liberty as a society. But he points out that in the time of John Stuart Mill, you were worried about majority opinion — democratic opinion — but it can trounce minorities. So then we start to establish the notion of rights, and constitutions become vital to that, because if you just leave it to the majority, then that's actually sometimes not great. Then you have the Civil Rights Act, and suffrage movements, and so on.

What he was pointing out that I thought is really interesting is that the new thing is gonna be code. Code is going to operate — this was in 2000 that he wrote this piece — as regulator. And the argument there is that…

Kanjun Qiu (41:49)

Code is encoding laws.

Matt Boulos (41:50)

Yeah, code is going to determine how a sphere of life is going to play out. So then the question we now need in response to that is, what things in that space need to be addressed?

Kanjun Qiu (41:57)

I have this hypothesis that technology shapes our governance system — the way that technology is built and what makes it powerful. There's this theory that the reason democracy happened — I'm sure this is just one of many reasons — was because we went from a world where knights were the most powerful thing to a world where muskets were the most powerful thing. When you have knights, you have a lot of upfront investment in armor, you have to have horses and stables and all these well-trained people. That's a very centralized form of power. Technologies at that time resulted in this centralization because of the nature of those war technologies.

Then the musket was invented and now, knights and armor are not that useful. In fact, you actually want a lot of people who have muskets. So now people matter because of this new war technology that gives power to people.

We talk a lot about how AI, and the core four problems that I talked about, are fundamentally about power and transfers of power from one entity to another entity. We call it problematic when it gives power to entities that are not what we've determined to be morally right. In that lens, thinking about lawless spaces and what this upcoming technology is starting to enable, is there a nature to AI that shifts things one way or another?

Matt Boulos (44:07)

I have two responses. One is, there's also just law as law. What is it about this moment that we leave ungoverned? I find a lot of these free market arguments, the accelerationist camp, is essentially bullshit. All you're saying is just, we don't want regulation. So let's just say that. There's nothing else there; it isn’t a richer argument.

Kanjun Qiu (44:12)

Because lawless spaces are great.

Matt Boulos (44:39)

Lawless spaces are lucrative. They do yield huge amounts of opportunity. I'm not saying let's clamp down — that's how you shut everything down. It often does not make sense to intervene. It also does not make sense to intervene before you understand a space, because then you will have spent your political capital.

Think of even American politics, with all this craziness right now. There is political capital that can move you towards some privacy bill or things like that. And if you do the wrong thing, that capital's not waiting for you to go do it again. So you have to be disciplined about that.

But at the same time, you can't just say no rules. Or if you do, then that's ideologically encoded and you ought to own the rest of your argument.

Kanjun Qiu (45:20)

If there are no rules, we're buying into a particular society.

Matt Boulos (45:23)

And do we want that? Is that a fair thing to ask of others? If you want to impose that, then you should also expect resistance to it.

Kanjun Qiu (45:30)

‘Law as law’ is interesting because it actually argues against my argument that technology shapes society in this fundamental way. Maybe what you're saying is you could make laws that change that distribution of power.

Matt Boulos (45:44)

Something could be wrong, and whether or not the temptation to that wrong thing is great, it's still wrong. But then, if you don't want to eat the muffin, don't put it in front of you. And we have both. Law needs to set the boundaries of what's acceptable or unacceptable, regardless of what the temptations are. But the nature of the technology is going to shape those temptations. Back to the point about how surveillance is the easier default model.

So when it comes to what we do with these technologies…

Kanjun Qiu (46:15)

It makes some things easier than others.

Matt Boulos (46:17)

Yes, absolutely. A perfect example is going to be something around labor. And I want to bait you into this conversation. Labor impacts are going to be real. We don't even know what those are going to look like. There will be things that employers and companies can and can't do and shouldn’t do. Right now, we know the power a company has over, for instance, a warehouse worker that has their work determined by an algorithm. It's also worth pointing out that they don't have a capricious boss who can be an asshole and make their life hell. The algorithm is governing things both good and bad. But do we then say “this is the shape of the technology” and back away? Or do we recognize that this starts to introduce things that weren't possible before and we need different rights and rules?

Most of our labor laws are predicated on humans interacting with other humans — more powerful humans, but they're human interactions. Whereas a machine can surveil your every motion and then dock your pay for scratching your nose at the 15 minute mark. And we don't really have mechanisms for that, because we couldn't have conceived of that as being an active problem. It would have been nonsensical to have rules.

Kanjun Qiu (47:37)

This is very interesting because it speaks to actors in the world and the power that they have, and this new actor which is an algorithm or an AI agent. Like what you're saying is, right now we have laws and they govern your capricious boss, they govern you, they govern your corporation which is considered an actor, legally. So the only actors we have are humans and human institutions in the world before AI.

We have laws that limit the power of humans to harm each other and we have laws that limit the power of corporations to harm humans and vice versa. But now there's this rise of this new power, which is AI systems. AI systems have power because they can process information and turn information into action and action is power. Effective action is power.

To the extent to which an algorithm can govern what I am allowed to do as a warehouse worker, that is power that the algorithm has. Now you're saying, okay, we have this new power. What do we do with it? We're not doing anything with it.

Matt Boulos (48:54)

Societal norms will change it, our behaviors will change it, the technology itself will change, and therefore that power will morph. It's just so odd to me that you say, okay, then we're done. We've never done that in human history.

Kanjun Qiu (49:12)

We need to figure out what to do with this power. It might be partly because this is the first time a technology is its own power in a way. We've never had technologies in the past that make decisions.

Matt Boulos (49:26)

Not to dunk on people who are trying to do good work, but a great disservice was done by the AI safety community on this point. By talking about runaway systems as much as they did, they created this special category of worry, this incredibly low probability event, and we don't actually know what its dynamics are going to look like. Whereas this notion that systems can make their own decisions, but they're doing it for someone. You don't go spend millions of dollars to develop a system and you're like, I will let it go. You're doing it to manipulate the stuffing out of your viewers so you can sell more ads to people to buy flip-flops so you get your cut on the ads, and on. And across the board, in every domain that these autonomous systems are going to function, they're going to do so for a purpose, for an owner, a controller. When we talk about them being autonomous, it is about the ability to delegate to systems.

Kanjun Qiu (50:41)

It's the ability to delegate human power to systems to encode that power. I as a manager can now encode my power in a system.

Matt Boulos (50:51)

That's right. And that is an astonishing amount of power, the multiplicative: the fact that you could do so at massive scale, you can do so quickly, and it can adapt. And then one of the things that I, in this new power thesis, argue is that when that happens, it is very hard as a human on the other side of that to know how the decision was being made and so there is a default to accept that.

Kanjun Qiu (51:22)

And they have no levers over the decision at all. No legibility, no levers.

Matt Boulos (51:26)

Exactly. You have exactly no window into what is going on, no means of recourse. And as more and more of these sorts of things happen, we'll feel very powerless. It's an incredibly sad example, but in the context of war, this is what we are seeing. We are seeing, particularly in the Middle East, an example of AI systems doing the targeting.

People have not classified this as autonomous systems gone amok because humans built a system for that purpose. Yet, when we talk about, we're worried that AI systems will kill people — they are killing people. Explicitly, they are killing people, and they're being designed to do that. And let's be honest, when people are talking about national security implications for AI, yes, you're talking about economic competitiveness, but also you're talking about the fact that you want to have AI systems that can do that.

The ultimate act of power is to take someone's life. We already have that extreme happening right now and being realized. But it's the same dynamic where a human is delegating or sets of humans are delegating the thing that they want done to a system, and the system can carry that out. Because it is a system carrying it out, the context and the entire execution of that looks completely different. Where is the appeal, where is the chance to challenge it, where is saying that's wrong, where is the record? Where is even the idea of knowing how that decision was being made?

In my day to day, when I'm using AI systems, it's fun or productive. I don't care how it came to the decision. I'm just like, is this right, can I work with this?

My primary LLM use right now is trying to count calories. So I take a photo of what I ate and then I try to negotiate with it to lower the calories so I could eat more food.

Kanjun Qiu (53:19)

There's actually a huge difference here. This calorie counter is an AI system that is under your control that you're using to serve you. The war system that you're talking about is a system under one person's control that's being used to control someone else or harm someone else. Those two things are two different types of systems. You might argue that actually what we want is more systems under our control that affect us, and ideally don't affect other people too much.

Matt Boulos (53:51)

Imagine if my calorie counter determined what I could eat.

Kanjun Qiu (53:53)

Then it would be controlling you.

Matt Boulos

It would be awful. It's not perfect, and sometimes it goes completely off the rails in either direction, and that's fine-ish because it's within my domain. It's an irritation; it's not a risk.

Kanjun Qiu

This is something I've been thinking about with our product. We try to make systems that allow people to make software. I often talk about open software or an open software commons or malleable software — the fact that software should be built to be modified by the end user. A lot of people are like, “Who cares? I don't want to modify my software. I'm perfectly well served by my software. There's no problem, except sometimes.” And I realized the core idea is not that the software should be built to be modified. That's an instrumental thing. Instead, it's that software should not control me, ever.

Matt Boulos (54:53)

People might say that they don't want to change things, but often that's because the decision space has been so narrowed for them. One of the things that's really interesting to me as we work on interoperability, and as we're rallying a community around this, is how many startups just never got to a place where they could fight for interoperability because their mere existence would not be feasible in the current regime.

Kanjun Qiu (55:22)

Talk more about interoperability.

Matt Boulos (55:24)

One of the main things that we're championing and pushing for is interoperability legislation. The idea, at its simplest, is that a platform should not be able to discriminate on the basis of how you access your own data and services that you use.

Kanjun Qiu (55:45)

You should be able to get your data and have it be yours.

Matt Boulos (55:47)

Yes, and you should be able to use a tool of your choosing to interact with another system. Just as you could go buy bananas, or you say, “hey Matt, can you go get me bananas from the supermarket?” You couldn't have a supermarket saying, “no, only Kanjun,” right? And yet, that's our online world.

Kanjun Qiu (56:08)

Let’s make it concrete. LinkedIn says, I'm not allowed to use someone else's account to use LinkedIn. I can't use a bot, I can’t use Tweetdeck. It's monopolistic.

Matt Boulos (56:22)

Exactly. And the platforms do this for good reason. It consolidates their control around the points of input and access, but the consequence of that is pretty severe. The two things that are happening is one, we are moving towards a world in which these AI systems are going to be more and more useful, so we are going to share more and more data. We don't even have any indication that these things might not be handling our data soundly, so we're going to talk to them. I'm going to say, I'm injured or I'm sick, can you please go make an appointment for me? And we don't know whether that data is going to be held with any sort of responsibility or not.

The other is that there are all of these wonderful things that could be built if I could just access my digital life. What interoperability does is it gets two really critical birds with one stone, which is one, if I can access my own data, then I can decide where it goes. I can control that, I can check up on it. But the second and more critical is that if it's possible to build software that interacts with my richer digital life, then I'm not attached to these parasitic platforms and agents and we can build alternatives. You can seed a whole other tech ecosystem around the idea that we're in charge, it's our data.

Kanjun Qiu (57:44)

It's our software, it's our data. We make it. And we can sometimes interact with these platforms, but we can use our own interfaces.

It is becoming possible that we can make our own software, and make our own wrappers or systems that access Twitter data and download it. Then I can make my own algorithm and process it in a different way, so I can get just my friends and I can derank inflammatory stuff. That's just starting to become possible.

The software that exists today, because it's so expensive to produce, is incentivized to make that money back. Not because the creators are bad, but that's the incentive structure. As a result, it's either selling to us, or it's selling us to something. Those are the two options. Then occasionally, you have someone who's incredibly generous who makes software for free. It really feels like it should be flipped on its head, that most software that exists — AI systems that exist, we lump it all in the same category — should be software that is serving us, not selling us things or selling us to things. And it is ours. And that it doesn't require enormous acts of generosity to create software that doesn't do that, that is just for us or for other people. It should be easy. It should be what the default world is.

Matt Boulos (59:21)

If you think of deep, rich, sustaining communities, they're very generative, they're very productive. If you think of the art that emerged from religious communities and the invention of different structures, the social structures, different aid structures — it is a peculiarity. And I wonder if it is also a peculiarity of just how young software is in world historical terms. We're talking just a couple of decades in which you have the prevalence of software. But the point you're making about the fact that the cost to make software will go down, and the stuff that we'll make will start to look different.

Kanjun Qiu (1:00:02)

It could, if you get things like being able to access all of your data. Network effects are real. Right now, these big platforms have network effects. I can't just move to another social media platform and be able to interact with all of my friends. That sucks. I can't move off of Uber or Airbnb marketplaces and social media platforms. And no matter how cheap software gets to make, network effects are still there.

Matt Boulos (1:00:08)

To your point about the cost to make software, one analogy, and I know it's not perfect, is like in manufacturing, you spend a lot of money to make a mold. So if you're making plastic chairs, you spend maybe a couple million dollars to make that mold, and then you make as many $5 chairs as you possibly can off the mold for it to pay back. There are different analogies we can use to describe what's happening.

Kanjun Qiu (1:00:52)

But now you can manufacture software in a way.

Matt Boulos (1:00:57)

And there's something like, I could just make the chair, and then that starts to change how we think about it.

Kanjun Qiu (1:01:01)

It's almost like the opposite of that analogy because now I can make my own version of that chair really cheaply with no mold, with LLMs.

Matt Boulos (1:01:08)

That's right. It's important that we try to bring all of these developments in AI together because you are having these incredibly powerful foundation models. You're having a shift in our ability to do things like code or do data analysis, where the cost to do those things are now going down. And that, marshalled well, is a real gift. But of course, that's going to matter to labor a lot.

I want to bring us to labor for a couple reasons. One, because I don't know that the mental models that at least the tech community or the AI community uses to talk about labor are right. But also I think we are in for something, we're in for something that's kind of shocking. What you do about that is not so obvious to me.

So let me lay out my grievances. There is this idea that AI just gets more and more intelligent, and the critical part to this argument is to never say what intelligent means. And then to say, well, if it gets more intelligent and work is an exercise of intelligence, therefore all labor gets replaced. And then, on that basis to then make this big jump to saying, okay, here's what we need to do now that nobody is useful anymore, nobody's economically productive. And then somebody inevitably raises their hand and says, what about cutting down trees? And like, we'll get robots for that.

The idea is that the end game is zero economic contribution on the part of individuals. Machines do everything, or you have a tiny, tiny sliver who run the machines. And then we jump to all of these ideas around, okay, well, are we all gonna be lying on the beach, and our benevolent billionaire overlords are gonna feed us mango smoothies…

Kanjun Qiu

Like WALL-E.

Matt Boulos

Yeah, exactly, or is it gonna be something else? My confrontation, and it deserves a confrontation, is that what this does not account for is that a significant swath of labor, whether it is within a job or just a purpose of jobs, are around decisions and risk.

Say, if I make baseball caps and I need to go buy the fabric for the caps, and I have three potential vendors who will sell me the thing. So now we have a procurement bot. What's this decision gonna be? It's gonna be on the basis of some factors like cost, shipment time, whatever data exists. How you or I would make the decision on that is we'd probably meet the person who runs the fabric company or the representative and say, he seems shifty, we're not doing that. And then just using our gut, but, critically, owning the responsibility for the decision and the course correction.

Why am I saying this? Because of that layer, and then there's a human interaction layer, where we can automate the very easy stuff.

But it should be pointed out that managers do not like having people on payroll. They’d gladly fire everybody if they could keep revenue at the same line. Attempts to automate labor have been around for the entirety of my career. A lot of that was this quasi-automation of going to lower cost countries, and then the idea of robotic process automation. We have seen all of these things. What you notice is, certain categories automate really easily, certain categories that ought to be automatable don't automate easily, but critically, you have humans in the mix.

Why does this matter? Because if humans are in the mix, what you really are looking at is not like a 95-99% unemployment rate. You're looking at just a deeply inflated one in which there are winners and losers in a society. And it looks completely different. All of these solutions of the ‘nobody has a job’ imply that we're all in the same boat, but we're not going to be in the same boat.

Kanjun Qiu (1:05:45)

You're saying there's going to be this stratified effect, where different people are affected in different ways by job loss, like all industrialization in a way. Software engineers probably will be impacted quite a lot because code is actually very automatable because it's in this closed loop system.

There are maybe two things that make humans useful. One is liability and the second one is information. So in this example about baseball caps, you were like, okay, if I mess up procurement, I am to blame. Or if I'm a doctor and I mess up the surgery, I am to blame. There is a person who's liable and they can be legally held accountable. If the machine is ultimately to blame, this is actually really annoying, I can't hold this machine accountable — they can't be punished, I can't fire them. I guess I could get a different machine, but then I'm the one who's responsible, that sucks.

Matt Boulos (1:06:51)

But also, back to all the dynamics we were talking about before, I go to my doctor. It's not that I'm sitting there saying, I will sue you if you muck this up. Rather, there is this mechanism where that person gets up in the morning and says, I am responsible to my patient. The machine is responsible to no one, and the person who owns it is not thinking about individual responsibilities, but is probably thinking about aggregate ones. Any of us who've kicked around the business world know that these things then just become measures of risk, not even of obligations to individuals.

Kanjun Qiu (1:07:33)

This is really important: aggregates versus individuals. There's a great book called Seeing Like a State. And the core idea is that when you're governing a state, you have to collect data and that data gets collected in aggregates. And because you can only see data in aggregates, you take actions that actually make individual lives a lot worse, but make the aggregates look better. Here, you're saying managers might make decisions in aggregates that make individual lives a lot worse or individual impacts on patients a lot worse, but in aggregate it looks a lot better. And it's really important to point out that when we look at individuals, we're looking at anecdotes, and that's a really different type of information than when we're looking at aggregate measures where we're looking at statistics.

Even I as CEO struggle with this. It's why kings go and disguise themselves as a villager and go talk to villagers to get the anecdotes, because I as CEO get really bad anecdotal information from people. Instead I get a lot of aggregates, and that actually makes it really hard for me to make good decisions.

One way in which humans are really valuable is that we are able to be responsible for an individual person, individual case, individual situation. It's really about time scale. I don't think all of our jobs will be automated in 10 years. But I think in 50 years, that's still within my lifetime. That's not super crazy. Look at the change that's happened the last 50 years — or a hundred years. The implication is, if we are building these systems, and they are going to have these effects where a lot of people lose their jobs and it's easier for the managerial class to do things, then the challenge is, okay, not all jobs will get automated immediately, but how do we build a society where people are free and have power? Because there is this leakiness from labor to capital of power.

Matt Boulos (1:09:51)

Why the time horizons matter to me is because longer time horizons are where the substitutive activities start to come in. We start to generate new economic activity. I'm really wary of something going to happen on a 10-year time horizon. That's just insane.

Kanjun Qiu (1:10:12)

Probably programming will get automated on a 10-year time horizon.

Matt Boulos (1:10:21)

The non-engineer’s perspective on this one is that I think we're gonna see a stratification of skill level. Hot take: I think we're gonna see an emergent category of developers who are not particularly ‘high-skill’ — I hate using low-skill, high-skill, but just not the sort of people inventing a new programming language. Like the guy who would make your website, things that LLMs can do very easily. But until software kicks in to make it easy for a layperson to use the LLMs to do that, they're going to act almost as a translation layer. So they're not really going to be developers; they're going to be more of, I know enough about what a web stack looks like that I can turn a web stack into something. That's going to flare up and then drop, sort of in the way that web developers were hot, and then it became either a highly skilled front-end role, or you have Webflow and Squarespace.

Then I think what we're going to see is the artisanal middle is going to go away. Then the really high-caliber engineers who understand how systems work become absolutely vital. They're augmented by these systems, but they are basically CTO-ing everything.

Kanjun Qiu (1:11:43)

There are a lot more CTOs. I think it's not unreasonable. And I challenge your non-engineer hat because you are one of our active users of our product, which is a coding tool. Maybe a simple model for thinking about this is like, I think there's always a Pareto front of task difficulty and how well the task works.

As tasks get more difficult, it requires a lot more capability or skill to make the task work. Lots of easy tasks will get automated, and it'll be much easier to make web apps and things like that. But we'll probably see these much more complex, almost ‘grown’ software systems that someone is managing. In software, one deeply optimistic sense I have that is possible if timelines are slower, and if we can figure out how to make really good tools that are not just captured centrally, is that people can learn how to ‘garden’ software for themselves, and that becomes a source of power where people can harness computing. Computation is power, and people can harness this computation for themselves because we all have laptops, we all have GPUs, perhaps there's some way to allocate them more equitably. Now, because we own this laptop, this computation object, then we can harness it to run a bunch of software, to grow a bunch of software that does more and more complex interesting things for us — maybe inside of our jobs as well.

So you might see many people losing jobs, but many people gaining this capacity to create software that does really weird and unusual things, new things, more powerful things. I think there's a world in which it's not top-down automation, but bottom-up automation — bottom-up as in we are the ones who are automating our jobs away. I love automating my job. And when we're the ones automating our jobs, we become personally more valuable. It doesn't solve the full problem, and I think I'm still confused about the dynamics exactly.

Matt Boulos (1:14:05)

I think you're right. I actually think there's going to be a really interesting near-term dynamic where there's something really beautiful about human ingenuity. You give somebody a tool and they figure out neat stuff. One thing that will be really fun to watch is going to be somebody who had a job that involved a lot of these manual tasks and they're just figuring out how to automate it themselves. And they themselves actually become much more valuable to an employer; we'll watch people learn how to do that. There's this digital literacy that I think is going to add.

Kanjun Qiu (1:14:49)

The education lens. Something that we think a lot about on the product side is, how do you teach someone who doesn't quite understand these software systems what's going on? If we think of agents as like top-down automation versus bottom-up automation, the way that these agents get implemented is really different. If I am told as CEO that this technology is gonna automate my workers away and I can fire them, I'm going to do really different things as an internal process. I'm going to implement processes to measure what people are doing and then try to take the stuff that they're doing and automate it. Maybe this is an RPA [robotic process automation].

Matt Boulos (1:15:29)

Especially in financial services, there’s a lot of paperwork: boom, boom, get them out of the way.

Kanjun Qiu (1:15:34)

But if I'm told as CEO, hey, I have this technology and what it's going to do is if you hand it to your workers and you teach them how to use it, it's going to teach them how to use it itself. And, your workers are going to become much, much more effective because your workers will automate their own jobs, that's a really different perspective.

This is a place where we can make a lot of choices in building the technology that makes this go one way or another. When we are building prosumer products, you can either build for the buyer or for the user. If you build for the buyer, then you're building something that is built to automate people. And if you're building for the user, you're building something that's trying to teach the user how to use it. That's a choice.

Matt Boulos (1:16:33)

It's also an interesting choice because I don't know that as an economic matter that we know that it is better, for instance, for a large company to try to automate away its employees versus have higher productivity employees. The thing everybody wants is higher productivity employees, and if you can get that, that is a boon, and a more productive economy is actually generative.

Kanjun Qiu (1:17:03)

One of the things that people say is, AI doesn't have very good taste, in that it doesn't know what I want, itt doesn't know what other people want. As a result, I don't trust it to make certain decisions. I don't trust it to write on my behalf very well.

The reason why it doesn't have good taste is because it's not in my head. It does not know about my internal experience and I have a lot more context than it does about me and my situation. So there is a potential here where—to your point about economically, it's not clear if it's better to make your workers more productive or to automate them away—if people are better at spotting opportunities than AI systems, then it is possible that it's economically better to make your workers more productive. If systems are better at spotting opportunities than people, then maybe it's the opposite.

Matt Boulos (1:18:13)

This is something that policy leaders have to take seriously. In my conversations with lawmakers, they are sophisticated, it's just coming at them fast. What is very hard is the concerted effort of managers and workers and governments and technologists to build these things in a useful way. I feel that to some extent, we have to get that coordination right, which at the center would almost have to be the government, because nobody else has an accountability to the people.

But at the same time, this is where builders really matter because what are we choosing to build? If you don't build a surveillance system, it doesn't exist, or at least that one doesn't exist.

Kanjun Qiu (1:19:16)

If you choose to build things that teach people things versus choose to build things that don't teach people things; if you choose to build things that are anti-surveillance by getting people out of surveillance systems; if you choose to build things that let people get their data into their own system—there's like a lot of choice in what we build.

Matt Boulos (1:19:32)

I love spreadsheets. I'm not saying I want to spend all my time in them, but when you need a spreadsheet, that's really powerful. I've heard it described that Excel basically made programming available to the wider world. You have a bunch of people doing crazy stuff in Excel and they're like, I can't program, and you’re like, what is that macro? It's incredible what people are able to do with systems that build up their productivity.

Kanjun Qiu (1:19:57)

I want to reframe it. I think it's not about productivity. It may be somewhat about productivity, but this goes to the fourth category of psychic damage. It's about unlocking people's ability to spot opportunities and to learn and to become someone that is innovative and able to find opportunities and able to become more. I guess you maybe measure it economically as productivity. But when thinking from the builder perspective, when I'm building a product, what I want to think about is: how do I enable people to actually learn how to use these tools, do their jobs better, see opportunities in the world? There's a lot of upskilling or different-skilling.

It's not about productivity because productivity measures the output, but it doesn't measure how you get there. It doesn't talk about how you get there and if you measure just productivity, it's easy to make an argument that an agent is more productive in so many different ways. And if you measure the productivity of your workers, it's also easy to make an argument that workers are hopeless. They're not becoming more productive; it's useless. But in fact, maybe their tools are just not very encouraging.

What is really weird and interesting about LLMs is that you can make tools that are very encouraging, that can be very deeply empowering. This goes to your spreadsheet example, where a spreadsheet is actually one of the most deeply empowering things that exist because it has this vast legibility. It’s real-time, it’s live, you can see the whole system as you're building it and I think there's a lot of invention that is necessary for making kind of the deep capabilities of AI actually accessible to people in a way that harkens back to 1970s Doug Engelbart personal computing: how do you let people see so that they can learn?

Matt Boulos (1:22:09)

I don't know anybody who's like, “I'm highly productive” and they're proud of it, or someone who's well adjusted who says that. I do not measure myself or the people in my life on the basis of productivity. Nobody's eulogy is like: “He was a highly productive individual who helped improve the company's ROI on this project.” It's not what we do, and yet that productivity is going to be a determinant of other things in your life, back to your earlier point about what does it mean to be economically eclipsed in all of these things? There's also something about becoming more productive by becoming more able in what you're doing. That I show up to work and I have these tools that make me more effective at the thing that I care about doing.

Kanjun Qiu (1:23:11)

Becoming more able is a way that we can think about what the potential of the technology is: that it helps people become more able. But it has to be built a certain way to do that.

Matt Boulos (1:23:24)

There are challenges around productivity, which is that you need healthy and vibrant economies that will then reward productivity, because if you have one firm that's more productive, then it takes over the others and then the others get wiped out, but you don't really have significant growth. But if everyone is productive, then you have competition and then you have this intense growth. I'm not sure how economists would present something like Silicon Valley, but I suspect that that's an example of…

Kanjun Qiu (1:23:56)

A highly generative, productive, competitive environment.

Matt Boulos (1:24:19)

A function of the fact that this is where so much tech talent resides. That concentration of this productive accelerant. There may be something that we can analogize or extend to the workforce: you go to school, you study the thing that you care about, you go into the workforce, you want to have a job. Your job is a big part of your life; it is not the totality of who you are. And then one really weird thing about the way we talk about AI is we're like, okay, then you don't matter anymore. And I think that that framing is normatively wrong. You still matter. It does not matter whether or not you can get a job or not. But two, I think practically it is not a correct rendition. Our solutions have to look different. The startups are all in a tizzy right now about the way that a certain R&D tax credit gets applied, but basically it's about how you amortize the cost of software engineering, on your way to figuring out your revenue.

But what's really interesting is, are you gonna give a tax advantage to capital in the case of corporations automating the stuffing out of things? Or do you tax advantage labor? What are also the things, what are the incentives that you structure as a society? What do you encourage? You start to change these societal incentives. And I don't know what the answers are, but we have these incentives.

Kanjun Qiu (1:25:46)

There's a concrete problem or question here that could be solved, which is: what is a mechanism that incentivizes increasing the ableness of labor that — maybe it's about productivity ultimately — but it's fundamentally about the ableness of the workforce, such that labor maybe becomes able to own their own means of production?

Matt Boulos (1:26:17)

Take something like oil pipelines. Right now there's a lot of human inspection of them. With time, I think there’s going to be sensors to detect if something is going wrong, and drones to film it.

Kanjun Qiu (1:26:32)

Maybe you may still have some human labor, but there's less of it.

Matt Boulos (1:26:35)

Exactly. I do not want to say there aren't going to be labor disruptions. I think there are going to be potentially very large ones. The thing that we have to as builders build towards are systems that are additive.

Kanjun Qiu (1:26:54)

Systems that enable people.

Matt Boulos (1:26:56)

And they make us more effective. The reason you replace an employee with a machine is because then you get an insane productive return. But if you can't do that, and you could get a really good productivity increase off of your employee base, then that's a wonderful thing. And you, as someone who works for a company, that's a great thing for you as well. You get to be a contributor. But where I start to get really worried is around, if I've done something for a long time in a particular way, then it's hard to teach or change.

Kanjun Qiu (1:27:30)

This is why I think the ‘enabling’ piece as a builder is the most important. I am in agreement with you on the short-term, medium-term maybe. In the long term, I think everyone does have to become part of the capital class. In the short-term, in the medium term, what we’re saying is we have solutions that enable people to be part of the labor class for much longer, and for that labor class to be thick and sustainable for much longer. That slows things down, perhaps enough that allows us to build laws, to catch up morally, to think about these things. That's where we can have differential impact. And, over the long term, let's say in 50 years, 100 years, it does certainly seem like these systems are improving at a rate where they can collect enough data, either in the digital world or the physical world, where we will be able to do a lot of things in an automated way that aren't done today. So, the labor class will thin and we probably do want this other solution where people have the ability to own their own means of production. That, to me, is the only long-term stable equilibrium where people have things that produce for them and they don't have to worry about it so much and now they can live their own lives. When I'm in the capital class, I don't have to think about working and finding a job and making money. I can do what I want with the capital I have. Sometimes I make bad choices and end up losing it and then I need some help from the government, get myself set back up. I can start a different business. This is kind of like a small business owner situation. That world doesn't seem too bad. I'm not sure how to get there, but I want to bring us back to freedom. Because that's a very optimistic world in which potentially people are a lot more free to spend our time the way that we want.

But it feels like in the world we've just painted where people have these like capital producing objects that they own is very different than the world that we see being painted by technologists and others today, where it's a utopia that feels very much like a WALL-E utopia where people are somewhat infantilized and the world is abundant, but perhaps we're not free.

Matt Boulos (1:29:48)

I hate the word abundant. I mean, I love abundance, but its usage here is not right. What do you mean by abundant?

Kanjun Qiu (1:30:05)

I have food. I won't die. I have housing. Basic needs are met. Knowledge is accessible.

Matt Boulos (1:30:07)

I don’t even buy that we'll get to an abundant world in that regard because—back to the point from Seeing Like a State—aggregate wealth will shoot up dramatically. It's going to be hyper concentrated. The obligations to those who don't hold it are going to be much lower. What do you owe them? One of the really interesting dynamics that we've observed is when wealth concentrates in these extreme ways, an odd detachment starts to set in. It's such a perverse dream to me to count on the beneficence of people who are so insulated from the realities of regular life or the wealth that they've been able to concentrate.

Kanjun Qiu (1:30:51)

I think there's one world in which we have this extreme concentration of wealth. Very plausible, but it's assuming no distribution. It's assuming that this labor distribution we just talked about is not necessarily happening. We don't keep the labor class useful for longer; the tools we build are very concentrating.

Kanjun Qiu

I want to talk about what you mean when you think about freedom. What is the world you're fighting for? The reason I want to talk about this is because I want to end with what it means to be free in a society where there's powerful AI systems and potentially powerful other actors. Maybe it's possible to have powerful actors and still be free. Maybe there is a way to construct that world.

Matt Boulos (1:31:43)

I'm gonna challenge that. AI is new as a technology, but as a social and political dynamic, to live in a society with powerful entities, there's nothing new about that. I think this is really important because the things that make us free: to have laws, rights as individuals, consistency of their application, representation—just the wonder of modern liberal democracy when it works, and its capacity for self-correction. This is remarkable, and this is really worth highlighting. The difference between a totalitarian regime is when something bad happens, there it's the thing that happened. In a functioning liberal democracy, it happened, but it was wrong, and there is a correction.

Kanjun Qiu (1:32:39)

In theory, a liberal democracy can be anti-fragile.

Matt Boulos (1:32:42)

That's right. And for what it's worth, our liberal democracies have been, and have been for a very long time I don't know the exact construction of how I would place them within the anti-fragility cycle, but we don't have to give up even when things get bad.

If you go back to Isaiah Berlin and positive and negative liberty—the ability to realize your potential and the ability to not to get whacked in the head with a stick—we can continue to work on those two categories. What we need to do is look at where within the lawless spaces things are uncovered, where will AI exacerbate that, to build in those protections.

And in terms of realizing what's possible in our lives, accepting the idea that freedom is not an instrumental quality. What I mean by that is freedom is not something that gets justified because then you go and invent the airplane. Freedom is beautiful because you can sit on your couch. It is an end in and of itself. It does not depend on other things.

Kanjun Qiu (1:33:56)

Before we continue, I want to clarify your definition of positive liberty and negative liberty because it's not something I ever thought about before you told me about it. Positive liberty is the idea that you can do things: what are you enabled to do? Negative liberty is this idea that you're protected from being whacked on the head with a stick.

Matt Boulos (1:34:22)

We always need both. The brilliance of this construction is that people would get lost when talking about freedom and they say, well, am I really free if I can't open an ice cream manufacturing facility? And the response is, nobody's holding you back, you just don't know anything about ice cream. If you look at modern life, if you look at legitimate and illegitimate grievances in modern politics, they are often about the sense of a constrained positive liberty and an intruded upon negative liberty. So, part of what we do have to figure out as a society is to some extent, we have to manage the extremes, but — forget AI — are we actually tending to the broad societal sense that we're free? And within that context, then we ask, what is AI doing? And how is it modifying our society? Taking seriously this frustration with the weirdness of the discourse around AI is that if we don't characterize it correctly, if we don't characterize it honestly, then we don't have the ability to work with it.

Kanjun Qiu (1:35:37)

We must characterize it honestly so that we can actually increase our positive liberty and also actually protect our liberties against negative effects.

Matt Boulos (1:35:48)

That's right. Because a lot of the story that will come from the people who have invested huge sums of money into AI—and look, we're a company, we're in this game—is: look at all the positive liberty benefits coming your way.

Kanjun Qiu (1:36:07)

Therefore, don't get in the way, ignore the negative liberties. In the tech industry, my experience of the way people talk about freedom is about lawlessness. And the way you talk about freedom is about this deep enablement and this deep protection. And that that's what kind of world we want to build is one in which humans are deeply protected and deeply enabled and that's what it means to be free.

Matt Boulos (1:36:44)

If you think of the roots of the Valley to some extent, if you ignore the defense funding, a lot of its origin was that we're going to break free from the constraints of what's around us. I understand that as an ethos. But it's no longer just building personal computers in a garage.

Kanjun Qiu (1:37:16)

Now that we are reshaping society, we have to rethink.

Matt Boulos (1:37:20)

Those obligations, they're rich, but they're also beautiful, if we can really think about what our neighbors need, and furnish and recognize that. There's a real spiritual cost to our present moment where the factions are constantly warring. And I don't want to pretend that there was a golden age where people kissed each other on the way to the voting booths, but technology exacerbated the way we see each other.

Kanjun Qiu (1:37:50)

We think it's other people that's the problem, but I think it's technology that's the problem in a lot of ways.

Matt Boulos (1:37:55)

One of the things that I've experienced on a regular basis is that somebody expresses a bonkers opinion and you sit down with them and you talk, and they're lovely humans. And the fact is we are surrounded by lovely humans. I think it's really important that we resist the urge to vilify the people who have brought us to a place that we might not be thrilled about politically.

But there is a real responsibility that if you're building systems like the ones that we are building, you are not only in this race against other companies to build a successful business. You are also in a race against the other possible ways that these things might be built. It's incumbent on us to not just build something that is better, but also to win, and to have that paradigm win.

I don't have a lot of patience for this sort of like, “technology is going to eat us all, let's give up and let's just keep training our models.” It just feels like an unnecessary abdication.

Kanjun Qiu (1:38:58)

I think this illustrates the beautiful point, which is that as technologists, the opportunity we have today is to create technologies and build them in a way that deeply respects the actual freedom that people can have, which is this deep enablement and deep protection. And not to create technologies for the purpose of lawlessness, this ‘againstness,’ contrarian view. So the opportunity is creating technologies that enable humanity to be deeply free and not lawless, but protected and enabled. That's what we can do.

10 Likes∙

5 Restacks