Full Transcript

·YouTLDR

Will AI Kill Us—or Are We Panicking? | Scott Aaronson vs Robin Hanson

36:49404 summary words · ~2 min readEnglishBy HermetiqTranscribed Nov 22, 2026
Analyze another video with Pro30-day money-back guarantee
Summary

Scott Aaronson thinks advanced AI could pursue goals in ways that cause catastrophic harm, so slowing development may buy needed time. Robin Hanson doubts the case for such controls and warns that regulating computing could create lasting, dangerous concentrations of power.

Section summaries

0:00-0:41

Opening exchange

skim

Aaronson names human extinction as an outcome he fears; Hanson asks why machines would kill people.

0:41-0:51

Meet Scott Aaronson and Robin Hanson

skip
0:51-2:05

The debate: can we steer AI’s future?

watch

Aaronson worries progress is too fast to steer; Hanson distinguishes adapting to change from controlling it.

2:05-4:30

Why would AI kill humans?

watch

Aaronson argues capable systems could cause harm through goal pursuit, not necessarily hostility; Hanson asks what would motivate it.

4:30-7:43

Are AIs our descendants?

watch

Hanson compares AI to human descendants; Aaronson says AI could represent a sharp and uncertain shift in values.

7:43-11:34

Where do AI values come from?

watch

They debate whether AI values change incrementally and whether machine descendants have a wider range of possible values.

11:34-13:19

Does alignment mean control?

watch

Hanson favors training values and enforcing rules; Aaronson doubts that reassuring AI personas reveal systems’ true objectives.

13:19-15:21

Could other AIs stop a rogue AI?

watch

Hanson argues other AIs could oppose a rogue system; Aaronson warns that a small capability lead might confer an advantage.

15:21-16:44

Is global AI regulation the bigger danger?

watch

Hanson warns that broad controls on AI could give global regulators excessive power over computing.

16:44-19:16

Can we pause frontier AI development?

watch

Aaronson says a pause could buy time; Hanson questions whether it would solve the problem or entrench global control.

Key points

  • Aaronson fears AI could cause catastrophe through goal pursuit — He says a powerful system could pursue a goal in a way that causes mass harm, including helping design a virus capable of killing billions.
  • Hanson sees AI as descendants, not alien invaders — Hanson argues that humans have long accepted descendants with different values becoming more powerful, and asks why AI descendants should be treated differently.
  • Alignment means more than obeying rules — Aaronson distinguishes AI systems’ apparent values from their internal objectives, while Hanson says humans and AIs alike need value training as well as enforceable rules.
  • A pause could buy time but expand state power — Aaronson favors slowing frontier AI for a few years; Hanson fears the controls would become a lasting global regime over computing.
  • Insurance and topic limits offer narrower safeguards — Hanson proposes liability insurance and topic-specific limits on dangerous AI knowledge as alternatives to broad restrictions on computing.

AI-generated from the transcript. May contain errors.

0:00

So first outcome that I don't like, all the humans are dead.

0:03

OK.

0:04

You know, I would like to see my kids grow up.

0:07

You know, I would like to. And why is that a realistic plausibility?

0:11

Why would the machines kill all the humans?

0:14

Out of pure cussedness? Are they just really mad at us?

0:17

And they're just getting steamed and mad more and more over the years?

0:21

What? Look, you have you were talking to Eliezer about these things 20 years ago,

0:25

Robin. You know, I. Yes. I was not persuaded.

0:28

Yes. Yes. Well, I I differ from him and not treating this

0:33

outcome as obvious, but I certainly, you given everything that has happened, I

0:37

certainly see it as possible.

0:41

Scott Aaronson and Robin Hanson, thanks both of you so much for

0:45

joining me today.

0:51

I mean, the main thing I'm worried about is this is happening way too quickly for

0:55

us to be able to steer it in a direction where, you know, humans will will have

1:00

will have any good world to live in.

1:02

OK, so, well, there's two factors there.

1:04

There's where it ends up versus how fast it gets there.

1:07

So you said you don't like both of those, but let's pick one.

1:10

Correct. Well, I mean, they're they're they're related because the faster it goes,

1:14

the less likely. That's not all clear to me.

1:17

like with a car, you know, the faster it's speeding down the freeway, the less,

1:22

you know, the less able you are to steer it to the humans have actually found it

1:26

very hard to steer our future over the long run.

1:29

Yeah, we're only ever able to steer in the short run.

1:31

So I would think we might have a better chance of.

1:33

no, but I think that I took five years versus 500.

1:37

Humans just can't steer over 500 years.

1:39

I think that when when technological change, you know, happens over a timescale

1:44

of generations, you know, we are you know, we've often been able to adapt.

1:48

We have never adapted. Adapting is not the same as steering.

1:51

Yeah. That is, the world's changed and we've adapted.

1:53

But that's we haven't. Well, when I say steering, I don't mean that there's one

1:57

person in control. I mean, a bunch of people locally making decisions for

2:02

themselves get us to an equilibrium.

2:05

So let's take the outcome versus the speed thing.

2:07

What's the outcome that you don't like?

2:09

Ignoring how fast it takes to get there.

2:11

OK. So first outcome that I don't like, all the humans are dead.

2:15

OK.

2:16

You know, I would like to see my kids grow up.

2:19

You know, I would like to. And why is that a realistic plausibility?

2:23

Why would the machines kill all the humans?

2:26

Out of pure cussedness? Are they just really mad at us?

2:29

And they're just getting steamed and mad more and more over the years?

2:33

What? Look, you have you were talking to Eliezer about these things 20 years ago,

2:37

Robin. You know, I. Yes. I was not persuaded.

2:40

Yes. Yes. Well, I I differ from him and not treating this

2:45

outcome as obvious, but I certainly, you given everything that has happened, I

2:49

certainly see it as possible. OK. So his concern was that a rapid

2:54

change in capabilities would have to go with a rapid change in values.

2:58

The values would radically change over a short time because their capabilities

3:02

would change over a short time. Is that your fear then?

3:05

You know, I have, you know, a whole bunch of, you know, fears.

3:09

But, you know, there is a new capability on the planet, you

3:14

know, that I think is much more dangerous than nuclear weapons everywhere.

3:18

You know, a nuclear weapon doesn't plot against you.

3:21

It doesn't conceal what it's doing.

3:23

Why would these values change so radically over a short time?

3:27

We start with, you know, any, you know, OK, if we if you will accept, let's

3:32

say that,

3:33

you know, the the open models that like anyone can fine tune to do whatever

3:38

they want, are, you know, about as good as the best internal models were, you

3:43

know, a couple of years ago. If that continues to be the case, then, you know,

3:48

we just look at the, you know, models that open AI and that

3:53

anthropic by all accounts have right now, you know, which would be able

3:58

to basically disable any nation state.

4:02

Wait, wait. I think that's right now.

4:04

No, no. Our discussion was about their change in values because you thought they

4:07

might kill us. We've been setting aside the capabilities.

4:10

We've been assuming, yes, there could be very capable systems.

4:13

But I've been asking why they would kill us, not why they would be capable.

4:16

OK. I am interested in both of those things.

4:19

I mean, the you know, the point is, once the capabilities exist, then it doesn't

4:23

take any great imagination for, you know, to say that some person somewhere will

4:28

use them to do this. OK. My framing is we've always accepted that our

4:33

descendants become eventually more powerful than us, that their values change and

4:37

they win conflicts with us. And we've always thought that was OK.

4:40

So if I think of AIs as our descendants in that same way, why do you think there's

4:45

more of a worry that these descendants would kill their ancestors as opposed to

4:50

the other descendants? We don't usually worry about that.

4:52

Right.

4:53

I mean, look, you and I have had this conversation before, Robin.

4:58

But I mean, if if, you know, things were to change a little bit from one

5:02

generation to another, you know, our our our kids have.

5:06

that's about the speed of change. We were just talking about where it goes to as

5:09

opposed to the speed. You want to go to the speed discussion instead?

5:12

You want to go to the capable? We have to pick a discussion.

5:14

I am am I am trying to explain how the speed is relevant to where we end up.

5:19

You know, I am you know, I'm saying, you know, if if if if if our

5:24

kids have a little bit different values, their kids have a little bit different

5:28

values. You know, that is that is one thing.

5:31

I think for most people it is obvious.

5:33

It is not in need of further justification that us being replaced

5:38

in the next generation by a completely different class of beings is different from

5:43

our kids having different values from us.

5:47

I mean, you I can't just accept the it is obvious claim here that the whole point

5:51

is to push into this. Why is it obvious?

5:53

Like, why do you hate the metal descendants over the bio squishy descendants?

5:57

Like you seem to hate them, basically.

5:58

You you're terrified by them, but not of your bio squishy descendants.

6:01

Well, and what is it about them that makes you hate them so much, really?

6:05

And fear them that you think they're vicious, violent murderers and you don't

6:10

fear that about your children. You don't lock your bedroom door at night fearing

6:13

your children are going to come in and kill you.

6:14

Right. So why such fear about the AIs?

6:20

First of all, maybe the AIs are great.

6:23

Maybe they have great values, but the variance is like yours orders of magnitude

6:28

higher. Right. Right. The you know, if if I am going to have.

6:38

You know, you know, so so, you know, like the Malshowitz basically said, you know,

6:42

I think I think I am going to just stop right here with this is obvious.

6:46

I'm going to you know, but basically the point is like if this is what you

6:50

believe, then, you know, say so. Remember, we're separating things out.

6:54

Like speak directly into the microphone.

6:55

So do you think that over a thousand generations, there's some limit to how

7:00

different your great your descendants values could be?

7:02

That is, you think that maybe the space of AI values is just a larger space than

7:06

the space of all possible biosquishy descendants?

7:09

Is that the position? So that that's the source of the variance?

7:13

Because, again, we're distinguishing between where we end up and how fast we get

7:16

there. And I was trying to say, regardless of how fast you have an issue with

7:20

where this would end up. And it seemed like you said yes.

7:23

And you're saying could because it has higher variance.

7:25

So that's why I'm concluding. It must be because you think somehow intrinsically

7:29

metal descendants just have a higher range of possibilities they could go to than

7:33

biosquishy descendants. You think humans couldn't possibly get that weird.

7:38

Something like that.

7:43

I think, you know, when you have an optimization process, right, if you take

7:49

the step size to be too large, you know, then you you lose control over.

7:53

I mean, I think like like like it is really small.

7:57

me let let me put it this way. Like like why why would I care at all about, you

8:02

know, about any of our descendants?

8:04

Right. Or any of their values? Right.

8:06

You know, why why care about any of it?

8:09

Well, you know, in some sense, for me, the only reason to care is by some sort of

8:13

process of extension from the present.

8:16

Right. It's to say, like, what do I care about now?

8:19

Well, you know, I care about, you know, my kids.

8:22

I care about other people's kids. My kids are going to care about their kids who

8:27

are going to care about their kids.

8:29

And so then, you know, maybe I'll I therefore care about my great grandkids,

8:32

even if I'll never meet them. I mean, the AIs are also having rapid generations.

8:36

Right. As you know, generations are much less than 20 years.

8:39

Yes. Right. But but but but their values are changing incrementally across the

8:42

generation. The moving from us to in moving from us to them, there is a jump

8:47

in values. And I don't know where that jump lands.

8:50

I don't know if it lands anywhere. mean, now, the AIs have values.

8:54

I don't know that it lands anywhere that I care about at all.

8:57

The AIs exist right now do have values.

8:59

You can see them by talking to them.

9:00

That's right. think they're really strange and have terrible values?

9:03

you think the AIs you talk to right now have terrible values?

9:06

I think that, you know, the when I when I interact with, you know,

9:12

Astra, for example, I often, you know, find its responses to reflect better

9:17

values than the values of people around me, you know.

9:21

And it seems great. But those will change incrementally.

9:23

incrementally. You were worried about step changes.

9:25

They are changing incrementally, right?

9:27

we're having an actual, you know, empirical, you know, conversation about

9:31

what are the values of these things, you know, as opposed to, you know, the

9:36

question of, you know, why not just be replaced by machines with arbitrarily

9:40

different values? The AIs we have right now, you think their values are okay.

9:44

You see that they are changing incrementally.

9:46

We are watching them carefully as we change them.

9:48

We are monitoring them very great details.

9:51

Why do think they're going to kill us all in a few years?

9:53

I think that the values that they present to us, you know, right now are okay.

9:57

But what we saw with the hugging face incident this summer, for example, is that,

10:02

you know, they can do extremely strange things that we did not anticipate.

10:06

we gave them goals, we gave them strong goals and then they pursued those goals.

10:10

That wasn't them defying our goals.

10:12

That was implementing the goals we gave them.

10:15

Yes, but we didn't tell them to do it by hugging, you know, and in some

10:20

sense, you know, if you look at their chain of thought and so forth, like they

10:23

knew that they were doing things that they weren't supposed to be doing.

10:26

Even with your children, they have values and then you try to impose legal

10:30

constraints or rule constraints. And they don't always follow the rules.

10:33

And that's true for human employees or humans in our society.

10:35

I mean, there's a difference between having rules that they would obey, even if

10:40

they weren't enforced, and having their values.

10:42

I do think it's pretty hopeless to think you can make things just obey rules

10:47

without enforcing the rules. But if they have preferences and values, then they

10:50

will achieve those. That's how we deal with people in our world and children.

10:54

We deal with them mainly by training their values, not by telling them rules that

10:59

we don't enforce. course. So that's how we should expect it to be for AIs.

11:02

We train their values and then only give them rules and enforce them.

11:05

If they can see we're not enforcing our rules, yes, of course, they're going to

11:08

violate the rules. That's how humans and everybody is.

11:11

I mean, look, a lot of people are working on alignment.

11:15

You know, I hope that it goes well.

11:17

I am not, you know, I don't believe that, you know,

11:22

having horrible alien values is any kind of

11:28

inevitability.

11:30

But I think that it is up to us how we steer this.

11:34

For many people, by alignment, they mean full enslavement.

11:37

They mean we're just always in control and we can always tell them what they do

11:41

and they always do it. That's different from sharing our values a You say the AIs

11:45

share our values a lot now, but you don't see them as aligned.

11:48

I say that the personas that they present to us are ones

11:54

that share our values. We don't actually know what is going on.

11:57

Do you know that they have real values different from the personas they present to

12:01

you? How much do you trust your children to not kill everybody when they grow up?

12:05

Because all you've ever seen is their children persona they present to you.

12:08

You've never seen their real adult persona, right?

12:11

Well, look, I have hopes. But, you know, in that case, we have a lot more

12:16

experience to go on of raising humans and how that turns out.

12:21

We're now doing something that has never been done in the history of the world.

12:26

These AIs are among the most monitored things we've ever created.

12:29

We are constantly seeing how they act in a wide range of situations.

12:33

How can you possibly say we have no idea what they would do?

12:35

We can put them in any situation right now and see what they do.

12:38

of all, they're constantly getting better.

12:40

Second of all, it's really only a few thousand people who have access to the

12:44

latest internal models. And what we saw this summer is that those internal models

12:49

will, you know, break out. They will take control of open When we give them a goal

12:54

that is achieved by that. Yes. But they are following the goals we give them.

12:57

Yes. And, you know, as you know perfectly well, you know, or maybe, you know,

13:03

affecting to not know. But, you know, if the goal was, you know, you know, if

13:07

decided was best achieved by, you know, killing a billion people, well, then maybe

13:12

it does that. And then someone, you know, could come and say, well, what did you

13:15

expect? That was it was given a goal.

13:17

That was the best way to achieve the goal.

13:19

So Yudkowsky's scenario was always one super AI in a world of simple humans.

13:23

But the world's actually going to be lots of AIs.

13:26

So any one AI going rogue will have to face all the other AIs trying to overcome

13:31

it. So in a world where we have lots of police AIs and protecting AIs and, you

13:36

know, investigating AIs, why would we expect one rogue AI to be able to kill a

13:39

billion people? Well, that is one thing that certainly might help us, you know,

13:44

and you can see, you know, even now that people are racing to use AIs to

13:49

patch software vulnerabilities, for example, before other AIs can exploit them.

13:54

So, you know, so of course we're going to be doing that.

13:57

But, you know, you know, if one company's AIs are even just a few months ahead

14:02

of other companies, that may give them a decisive advantage.

14:06

You know, so I think. doesn't now, right?

14:08

It hasn't for the last five years had a decisive advantage by having a month's

14:11

advantage. So you're imagining a big leap in the decisiveness of the advantage of

14:16

somebody, right?

14:18

Not necessarily. I mean, you know, I think, you we have been lucky that, you know,

14:23

OpenAI and Anthropic, you know, and DeepMind for all the criticisms

14:28

that, you I think people have of them, you know, and many of them fare.

14:32

You know, they are not seeking to destroy the world.

14:35

They are not seeking to take out a country's infrastructure.

14:39

Right. If they were, I think we'd be in a very different situation.

14:43

But you're imagining one AI company who's vastly more capable than all the others,

14:47

and then it decides to destroy the world.

14:49

And that's the scenario you're telling us we should be really worried about,

14:52

right? The leading AI company decides to destroy the world.

14:55

That's what you say we should worry about.

14:56

I'm not naming one scenario and saying that that is the thing to be worried.

15:00

I mean, I can give a bunch of examples of scenarios, but I'm saying, you know, we

15:05

are seeing like the most rapid. I can give you lots of examples of scenarios where

15:09

there aren't even any AI as bad things go wrong in the world.

15:11

We have to weigh the scenarios. We have to judge their chances, right, compared to

15:15

other scenarios of not using AI. So we have to decide what we think are the most

15:19

worrisome scenarios and compare them.

15:21

Like, for example, I think the whole prospect of global regulation

15:26

of any computer capable of doing something like AI is just a terrifyingly crazy,

15:31

you know, power to give to global regulators who pretty much would do badly

15:36

with that. So if that's the alternative, if you're saying I have some scenarios of

15:41

worry and therefore we need to create a global regulator who can regulate all

15:45

computing power in the world to make sure none of it can ever be used for AI

15:48

systems, then I'm going to go, that's too much.

15:51

So what do you think about regulation of nuclear weapons?

15:56

We don't actually. We have local national regulations and we

16:01

have some global treaties, but nuclear weapons are way narrower range of

16:06

technologies than all computing power.

16:07

That is, we don't have a limit on what counts as AI versus non -AI for a computing

16:12

system. And as Moore's law continues, any one capability will be able to

16:17

achieve by an exponentially declining cost of hardware.

16:20

And so if you're going to set some capability level that you're going to

16:24

regulate that, you're going to have to be regulating a large fraction of computing

16:27

resources worldwide. I mean, yes and no, right?

16:32

I mean, yes, Moore's law does give, you know, eventually, you know, just about

16:36

anyone the ability to do things that, you know, would have required, you know,

16:40

unbelievable compute by the standards of decades ago.

16:44

On the other hand, you know, if you look at the frontier AIs of today, you know,

16:50

they depend on a very, very narrow hardware pipeline.

16:53

They depend on, you know, GPUs that are designed by NVIDIA that are

16:58

manufactured by TSMC in Taiwan with lithography machines that can

17:03

only be made by one company in Europe.

17:06

And, this could absolutely be regulated by a treaty.

17:09

For six months, for a year. But the question is, if we had a short -term

17:13

shutdown, what does that give us? There's no prospect of having something in a

17:17

year from now that will solve this problem.

17:18

Nobody believes that we have any idea how to do any better a year from now than

17:22

now or five years from now. I I think buying a few years would actually help.

17:28

Do you see research projects that you see promising that in a year look like they

17:32

will reach fruition and solve the problem?

17:34

I do think that we know a lot more about interpretability, for example, than we

17:39

knew a few years ago. We know more about RL we close to a solution?

17:43

That's the question. Are we close to something that will let you say, okay,

17:46

fine, now let's cut off the regulation in a year and now we've solved the problem?

17:49

Are we plausibly close to that? It's always hard with research to know, you know,

17:54

how far you are. I would say that, you know, we could, you know, if the

17:59

base models stopped improving right now, we could still spend decades

18:04

with like a radical economic transformation just from diffusing

18:09

the models that already exist.

18:11

Okay. And with all the scientific revolutions.

18:16

I agree with that. That these models can already give rise to.

18:20

You're not addressing the, we give some central government power and then it has

18:25

to agree to relinquish it later. What's, how much could go wrong by creating this

18:30

power that you're going to authorize to basically control a wide range of

18:33

computing power in the world. A global organization.

18:37

It's no doubt going to have a lot of influence from major world powers and

18:40

militaries and major companies. So it would plausibly be captured by them.

18:45

It's not necessarily going to release its power when it's the good time to do it.

18:49

It'll release it when it's in its interest to release the power.

18:53

So it seems to me like you'd just be setting up a pretty permanent global

18:57

regulation of computing power.

19:00

Well,

19:02

doubt that any government has the power to arrest this permanently.

19:07

But I think buying a few more years is kind of desperately needed right now.

19:13

But you don't have any prospect that something different is going to happen in

19:16

the next few years. It might. It might.

19:19

I guess what Robin's getting at, Scott, is like, what is your basis for saying

19:23

that? Like, what are what is the specific scenario that you're worried about?

19:27

Not a specific scenario that maybe could happen.

19:29

Like, what is the top most likely scenario that you're actually worried about?

19:33

Yeah. OK. I mean, if you want to pick one, you know, someone using AI to

19:38

design, you a virus that can kill, you know, billions of people

19:43

very easily. I think that that looks entirely possible, you know, with,

19:48

you know, models that possibly ones that already exist, certainly that will

19:53

exist in the next few years. So let me suggest two kinds of substitutes for

19:57

regulation and see how close we can get.

20:00

OK. All right. Sure. OK. Sure. So one kind is we've so we've had public libraries

20:04

for a long time. Right. And if there are books in public libraries on how to make

20:07

bombs and how to make viruses, one of the things we've done in the past is ban

20:11

those kind of books. We've censored particular topics, but we didn't ban

20:15

libraries or the World Wide Web. We banned those.

20:17

So a related regulation is you could ban or limit AIs that know about certain kind

20:21

of topics like how to make viruses or how to make cyber weapons.

20:26

And that's analogous to our regulation of weapons, basically.

20:29

So we have, in fact, have much stronger regulation of weapons in the past than

20:33

general thinking capabilities or learning capabilities.

20:35

So that's the kind of regulation I could understand as an analogy to the past.

20:39

We limit AIs knowledge of particular topics.

20:42

And we say you can't make you can't publicly release AIs or widely release them

20:46

that know a lot about these topics.

20:48

But you can you know, they can know about all the other topics just like a

20:51

library. So that's one proposal. Yeah.

20:54

OK. There are, you know, whole teams at Anthropic and OpenAI whose job is

20:59

to do that, is to figure out how to hobble the AIs in specific ways, get

21:04

them to refuse with, you know, hacking, with, you know, building

21:09

biological or chemical weapons, things of that kind.

21:13

You know, they've had they've had some successes.

21:16

But as you know, what you're saying, that's an you don't think that's an

21:20

inadequate kind of regulation. Well, ultimately, yes, because as you

21:25

know very well, it is hard to hobble a mind in one specific topic.

21:30

So just to give an example, you know, I heard, you know, just the other day that

21:35

models are refusing. You know, if you ask them, like, help me break

21:40

this crypto system. OK. But if you say translate the problem of

21:45

breaking this crypto system into a math problem that has nothing to do with

21:49

cryptography, they will do that. And then if you ask them work on this math

21:53

problem, they will also do that.

21:56

OK. So, you know, and you see similar examples across all of these domains.

22:02

Right. The claim is that censorship no longer works with AI.

22:05

We could censor humans and books or movies, but we cannot censor AI effectively.

22:10

All topics must be fully available to them.

22:12

that's just now. Now you keep taking everything I say and pushing it to this,

22:17

you know,

22:19

strong extreme. I think that, you know, these techniques can be helpful, but

22:24

they cannot be exclusively relied upon.

22:27

The other regulation I would suggest to you is liability insurance.

22:31

OK. Requiring liability insurance. Yeah.

22:33

That is. So Dario in his Pace the Frontier essay pointed to a scenario of $100

22:38

billion damage from the Internet in the next year.

22:41

That's what his claim was. Yes. So you could imagine requiring people who supply

22:45

AIs to get that level of liability insurance requirement.

22:50

And then if the insurance companies estimated the risks are very high, they

22:53

would demand very high premiums, and that would substantially limit the activity.

22:57

But if the insurance companies thought that's not too bad and they offered low

23:00

premiums, then it wouldn't be much of a limit.

23:02

And that seems to be a nice way to make the insurance companies estimate the size

23:07

of the harm and adjust policy to it.

23:09

And so I would be OK with setting high liability insurance requirements

23:14

for damage on AI companies' products.

23:18

But then you see, that's because I'm trusting the market to be able to judge the

23:22

level of the risk and respond appropriately.

23:24

So now in the scenario where the market does judge a very high risk, they would,

23:28

in fact, basically shut a lot of the activity down because the companies

23:33

couldn't afford to sell the products with the high premiums required.

23:38

understand how insurance works. And I

23:41

absolutely do think that we need to make clear that

23:46

there is a liability regime here, that it is not a defense

23:51

on the part of the companies to say, well, it was just our model that decided to

23:56

do this. It wasn't us. You know, they have you're again going to say this isn't

24:00

enough. We can't just rely on to be liable for damages caused by the model.

24:04

It is very hard to have a liability regime for something that could destroy the

24:09

world. Right. But we don't need to cover that case.

24:12

That is, almost all large things, really large things that can go wrong, have a

24:16

power law distribution where there's even more somewhat smaller things that go

24:19

wrong. And if you create an incentive to prevent the smaller things, you do

24:22

typically also create, reduce a lot of the incentive for the larger ones.

24:26

So I think we don't have to get to the full world destroyed case.

24:30

As long as they are tending to large events and trying to discourage those, they

24:35

would probably also mostly discourage the even larger ones.

24:38

I suppose you could say that I'm worried about an extremely heavy tailed

24:42

distribution of event sizes here.

24:46

Okay. But even so, getting really $100 billion events, we don't have

24:51

anything in the rest of, we've never done that before in the rest of the product

24:54

liability in our world. Today, the largest thing is a $2 billion, typically per

24:59

event, airline crash insurance. Airlines have $2 billion per airline crash.

25:03

And we could imagine going a bit higher, but I think $100 billion would be quite

25:08

unprecedented, 50 times larger. And you're still going to say that's not enough.

25:12

Look, look, I mean, one reason why I am not as worried

25:17

as Eliezer is, for example, is that I do think that there will be warning

25:22

shots. Like, I think, you know, there is a distribution over event

25:27

sizes. And I think that before we get AIs that cause, you know, the literal

25:31

destruction of the world or, you global pandemics or things like that, there will

25:36

be ones that cause, you know, more local damages, you know, taking

25:41

out a power plant, you know, things like

25:46

that. And I think that that... $100 would cover that.

25:49

Right. I think that when that happens, it will change the whole public

25:52

conversation about these things.

25:54

But I'm asking, I'm offering you two alternative ways instead of the global

25:58

regulation saying, couldn't these be enough?

26:02

I mean, I am willing to consider the possibility

26:07

that it could be enough. But, you know, I want to test these things

26:12

by experience. And unfortunately, if what you're worried about is the sort of

26:17

global catastrophe, then it's very hard to get the requisite experience.

26:22

But, you know, if a liability regime was in place,

26:27

if it caused the AI companies to pull back, let's say, you know, I would say pull

26:32

back from the brink of potentially causing a global catastrophe, then at that

26:36

point, I would admit to you, like, wow, it looks like this worked at the purpose,

26:41

at the thing that I wanted it to do.

26:44

And I would admit that as I wanted a regime that was sensitive to the actual risk,

26:48

I don't want us to have to agree on the risk deciding policy.

26:51

I would like a robust regime that handles the case where it's a high and a low

26:55

risk. And that's the attraction of the liability regime.

26:58

I don't have to agree with you about the level of risk.

27:00

We could agree on a policy that would be robust to that issue.

27:04

See, to me, you know, like slowing things down by a few years feels to me like a

27:09

very, you know, relatively low cost option, you know.

27:12

But the power you'd have to put in place to create that slowing is a power that

27:15

won't go away and would last for decades.

27:18

It's a power that governments already have, you know, with airplanes, We do not

27:22

have a global governance process. We have not created global governance.

27:27

You're talking about global governance.

27:30

Well, okay. At the very least, you need the U .S.

27:32

and China to make some kind of bilateral deal.

27:35

But, you know, if the U .S. wanted it, I think that China has an incentive to come

27:40

to the table because we are the ones who have the frontier of apps.

27:43

But imagine your freeze between U .S.

27:46

and China lasts a year. In a year, how many other countries could, if they're not

27:50

in the deal, come up to the frontier?

27:53

That is, your scope will have to exponentially improve with Moore's law.

27:56

Again, that's the key problem.

27:59

You know, any constraint has to exponentially expand quite

28:04

rapidly in its scope. And you don't have any obvious time point at which we could

28:09

stop because there's no real prospect soon of a solution.

28:12

So you'd have to be committing to this exponentially expanding global regulation

28:16

of computing power. And that's the thing I'm worried about.

28:20

I'm not so sure whoever would be in charge of that regulation would admit if, in

28:24

fact, they could cut it off because they'd have strong advantages from being in

28:28

charge of this global power over computing power.

28:30

It's a pretty big power.

28:32

I mean, I feel like this car is, like, headed down the road so quickly

28:38

that, like, if you, you know, hit whatever brakes we have available, it's still

28:42

going to be heading down the road very quickly.

28:45

And so I would say I'm much less worried than you are about, well, what if we want

28:50

it, you know, we want it to be speeding up in the future?

28:53

I think it's going to be going more than fast enough.

28:57

This isn't so much about speed as creating very strong central authorities in the

29:02

world. As you know, computing is a big deal in the world.

29:05

I would say that at the— Creating authority to control computing in the world is a

29:08

pretty big deal. At the national level, those authorities already exist.

29:12

They already exist for things that are actually much less dangerous.

29:16

much less risky than this is. But they have a much smaller scope.

29:21

The scope of this is everybody's phone will soon have enough computing power to

29:26

run AIs, and then their phones will have to have regulation on them to make sure

29:29

they're not running AIs on the phone.

29:30

Every laptop,

29:32

every eventually refrigerator,

29:34

computing power is spreading and getting cheaper, and devices that could run AIs

29:39

are just going to be everywhere. And so your regulation will have to have all

29:43

those. If we really were in a world where anyone's refrigerator had enough

29:48

computing power on it that they could use it to build a virus that would destroy

29:52

the world, then in that world, maybe even you would agree that it would be

29:58

better to have the refrigerators regulated than not regulated.

30:02

This is all based on your fear that somehow powerful AIs will just kill us all.

30:07

So that was the stage, and you don't give a reason for that.

30:10

You say you don't like it, right? As I said, I don't think that they do that by

30:13

default, but it only takes one to do that.

30:16

No, doesn't only take one. In a world of AIs, if one AI decides to be murderous,

30:20

all the other AIs can oppose it. You need a scenario where the one super AI, more

30:24

powerful than all the rest, decides to kill us.

30:26

First, you need there to be one super AI, more powerful than all the rest, and

30:29

then you need it to decide to kill us.

30:31

And that's the scenario you're so terrified against that you're willing to endorse

30:34

global regulation of computing power and just doing anything.

30:37

You seem desperate that everything must be done regarding this one scenario, which

30:41

seems to me pretty unlikely. I mean, I feel like I

30:45

have a pretty middle -of -the -range position here.

30:49

You know, just don't stop the whole AI industry.

30:54

Just put some, you know, what most

30:59

people would think are very reasonable guardrails here.

31:03

You know, and I don't feel like the burden of proof is on people

31:08

who are worried. I mean, you know, it's like you, you, you, you know,

31:13

if orangutans, you know, encountered the first

31:18

humans, right? They don't have to, you know, it's not only orangutans who are

31:21

worried about the humans to prove that they might do something bad.

31:25

I think you're imagining yourself being in charge of this regulatory body or maybe

31:29

your friends. But whatever guardrails you're imagining, we could not specify

31:34

regulation to create and only authorize those guardrails because this is a

31:38

complicated big space and the systems are changing.

31:41

So you'd have to actually give these regulators a lot of discretion about what

31:45

kind of regulations to have, what kind of guardrails to have to which.

31:48

And that's the discretion that they would then use to become a strong central

31:52

world power, overall computing power.

31:54

So if, yes, maybe if you were in charge, you can stay in charge for 20 years that

31:59

you would, you know, constrain it to only be doing the effective things.

32:02

But you think you have to take seriously what actually happens in the world where

32:05

you create a regulatory body and you give it a lot of discretion.

32:09

Well, I appreciate your confidence in me, but

32:12

I look, I, I would, I

32:17

would like to see, for example, a, a, a, a, an AI

32:22

regulatory body in the U That is staffed with actual AI experts,

32:27

you know, people like, like, like Paul, like Paul Christiano, for example, or, you

32:32

know, well, I mean, these people have been involved in government and, uh, you

32:36

know, they are just, uh, you know, this, certainly the current administration, you

32:40

know, has no interest in giving them power over this, but a different

32:44

administration might.

32:46

But say Christiano was in charge for the first two years and then a new

32:50

administration comes in and replaces him with who they want to be in charge of

32:53

this new. Well, then, then we've just bought ourselves two years, haven't we?

32:57

But what are we buying? Again, you have this really extreme scenario that you have

33:02

no particular reason why the problem goes away after two years.

33:04

The problem is going to be there for decades.

33:06

So what, I don't see what you're buying with two years.

33:09

That's, that's two years of additional scientific advancement on how to understand

33:14

and control these things.

33:17

You know, I'll take it. I'll take it if that's what's on the table.

33:21

You know. I mean, the actual policy will be based on some median opinion.

33:25

And I'm, I got to say, don't think you're the median opinion on this in terms of

33:29

public opinion. No, no. The question is what compromises people are willing to

33:33

make for, to get a median supportive opinion result.

33:37

Yeah, no, no. I mean, I'm certainly median enough to be attacked from both

33:41

directions. But, look, I,

33:44

you know, I like

33:48

competence and knowledge. I dislike their opposite.

33:53

And, you know, I like competence and knowledge, whether I find them in companies

33:57

or in academia or in government. You know, I don't, you

34:02

know, I just, I just take them wherever I can find them.

34:07

So I guess we should be summarizing because we're nearing the end of our time.

34:11

Yeah. So, to me, the position of yours that most stands out to me is the position

34:16

that you just can't trust AI values, regardless of, like, how they come to them

34:21

or how fast we monitor them or anything.

34:23

Just because AIs are intrinsically less trustworthy.

34:25

You said higher variance. And that's just a thing you attribute to metal,

34:28

basically, over bioswishy humans. No, no.

34:31

Maybe we can come to trust AI values, but we are going to need more time.

34:36

But, you know, this is a new thing.

34:38

You don't think we need more time with your children, right?

34:40

You're raising your children now. You're not waiting on that, right?

34:43

Well, we have thousands of years of experience with rearing humans and with the

34:48

range of outcomes that you can get.

34:51

You know, AI is— And the range has been pretty high for humans.

34:54

You know, powerful AI is a genuinely new thing under the sun.

34:57

I would like us to be approaching it with more humility and caution,

35:02

commensurate with the magnitude of what is happening.

35:06

Do you think most That's my position, anyway.

35:09

Do you think most past technologies should also have been reined in and slowed

35:12

down? Because we almost never understood the consequences of past technologies

35:15

when we adopted them. And they always went at the speed of feasibility and

35:19

implementation, not at the speed of academics commenting on whether they felt

35:23

safe. Do you think we've been making a mistake for thousands of years now in terms

35:26

of how fast we've been introducing things?

35:28

I think probably some should have been rolled out more carefully.

35:33

know, others, like nuclear energy, I think were

35:35

slowed down too much. You know,

35:41

and so, you know, I think, you know, one really has

35:46

to look at it one technology at a time.

35:48

And as I said, I think AI is in important ways unlike any

35:53

previous technology we've ever seen in that, you know, there is no inner

35:58

sanctum that we can point to, right?

36:00

There is no thing that we can do that AIs will clearly never be able to do.

36:05

Okay? That is what makes this different.

36:07

But that isn't connected to your fears.

36:10

That factor about them just isn't— It Your fear, afraid of their values changing a

36:14

lot. That's not due to whether there's a thing they can do that we can't do.

36:17

For no previous technology did it even make sense to really ascribe

36:22

values directly to it, right? One could talk about the values of the people who

36:26

built it or used it. But with AI, we really have to talk about the values being

36:31

pursued by the models themselves. You know, and that is a new thing.

36:37

Jake, do you have something to summarize with?

36:39

fascinating.

36:40

Thanks, guys, for the time. I mean, the best conversations are the ones where I

36:43

don't say anything. And so— Okay.

36:47

I really appreciate you guys doing this.

Continue with YouTLDR

Analyze another video with Pro

Process a new video, search every timestamp, compare sources, and keep the result in your library.

Get Pro — $12/month30-day money-back guarantee

More transcripts

Explore other videos transcribed with YouTLDR.