He Risked Everything To Warn You: No One Is Ready For What's Coming, And The AI Companies Know It!

The scary open secret in the AI industry

right now is that it's possible that

we'll end up essentially creating a new

species that ends up ruling the world

with a 70% chance that this goes

horribly wrong like human extinction.

That's one possibility. There's many

more.

>> It's quite chilling what you're saying.

>> Yeah, it's uh

gets me down sometimes.

I basically told my wife like let's not

have any more kids. It's too uncertain.

I don't think they'll ever join the

workforce.

Everybody should be afraid that their

jobs are going to be lost. And I know

this because I went to OpenAI in 2022.

What I did there was forecasting what

the what the next couple years might

look like. And unfortunately, most of

the world is kind of asleep at the wheel

and doesn't really realize what's going

on with AI. So, I resigned.

>> I read it somewhere that you lost $2

million for not signing an

anti-disparagement clause, meaning you

couldn't criticize the company.

>> Yes, for reasons I'm happy to get into.

But, the main thing I've learned is when

I go talk to people at Anthropic and

OpenAI about forecasting, they're like,

"It's not going to take that long. You

need to shorten them again. Get them

back to 2027 or 2028." Because these

powerful CEOs, Dario or Sam or Elon, are

racing each other to be in control of

the most powerful AIs. And are literally

afraid that if the other guy gets there

first, he might become dictator. I mean,

Anthropic is on track to be the entire

economy by 2030. But, none of these

people should be trusted with that much

power. So, this is the most important

thing happening in our lifetimes,

probably in all of history, in fact. And

it's very important that it go well. So,

I think that there's a lot we can do to

like steer things in a better direction.

There's loads of benefits that we could

get from AI if we do it right. And if we

do solve the problems, then things could

be absolutely amazing for everyone.

>> Well, this report here in 2021, it was

remarkably [music] accurate. And then

just published this one.

>> Yeah. So, this is our new scenarios.

>> So, let's go through these slowly and

one at a time.

>> I would be incredibly happy if all my

predictions turn out to be wrong.

>> This is super interesting to me. My team

gave me this report to show me how many

of you that watch this show subscribe.

And some of you have told us, according

to this, that you are unsubscribed from

the channel randomly. So, favor to ask

all of you, please could you check right

now if you've hit the subscribe button.

If you are regular viewer of this show

and you like what we we here. We're

approaching quite a significant landmark

on this show in terms of the subscriber

number. So, if there was one simple free

thing that you could do to help us, my

team, everyone here, to keep this show

free, to keep it improving year over

year and week over week, it is just to

hit that subscribe button and to

double-check if you've hit it. Only

thing I'll ever ask of you.

Do we have a deal?

If you do it, I'll tell you what I'll

do. I'll make sure

every single week, every single month,

we fight harder and harder and harder

and harder to bring you the guests and

conversations that you want to hear. I

stay true to that promise since the very

beginning of the Diary of a CEO, and I

will not let you down. Please help us.

Really appreciate it. Let's get on with

the show.

>> [music]

>> Daniel Kokotajlo.

At the very heart of what you do,

um what is your mission? And why?

>> So, what would you do if you thought

that superintelligence was coming in a

few years?

>> I guess it depends

what the consequences were.

>> Well, let's talk about it. So,

superintelligence, AIs that are better

than the best humans at everything,

while also being faster and cheaper,

also able to

operate robots that can do everything in

the physical world that humans can do,

but better, faster, and cheaper. If that

really is coming in a few years,

then we need to prepare, and we need to

think about how to make it go well

instead of poorly. So, that's sort of my

answer is like, I'm doing that to the

best of my ability.

>> So, you believe it's coming in a few

years?

>> Yes.

>> How could you be so sure?

>> I spend a lot of time trying to forecast

this sort of thing. My sort of median

estimate, a 50% chance, is currently in

2029. Maybe it'll slip to 2028. It's

possible that it'll take significantly

longer, like maybe 10 years or something

like that. But, uh you know, for reasons

I'm happy to get into, seems to me like

it's probably happening by the end of

the decade. Which less important is the

the sense of how close we are. What's

more important is the pace of the

trends.

Anthropic

this time last year was making something

like a billion dollars a year.

And they're making something like 60

billion dollars a year.

So that's

60x growth in 1 year,

which is extremely impressive even for

very small startups, but for a company

of their size, it might be the fastest

growth in history.

Um we expect that rate of growth to slow

down,

but even if it slows down quite a lot,

they're still on track to be,

you know, the entire economy by 2030 or

so.

>> Why should the average person care?

>> The high-level thing is absolutely

everything is going to change for the

whole world, and including therefore for

them and their families. Um could change

for the better, could change for the

worse, depending on the details of how

it's done. So for example,

everyone could die,

you know? Um this is the classic loss of

control scenario, or one version of it.

If we do build these super

intelligences, and we

use them to automate all the jobs, and

we put them in the military, and we, you

know, have them giving advice to

politicians, and so forth, they will

eventually have accumulated enough

real-world power

that they don't need humans anymore. And

they're smarter than us, they're more

strategic, etc. At that point, we sort

of have to hope that they are virtuous,

that they have, you know, the goals that

we wanted them to have, the values that

we wanted them to have, etc.

And the sort of

scary open secret in the AI industry

right now is that right now that is kind

of just a hope. It's not something that

we can

be at all confident in, and in fact,

there's lots of evidence and arguments

that

it we're not on track to achieve that.

So there's lots of reason Like current

AIs, for example, will often lie uh to

people, or they will like you tell them

to do something and they go do something

else, and then pretend that they did it,

right? So

it's an inherently difficult problem to

make something that's super intelligent

and also

has the values and virtues that you want

it to have, and it doesn't seem like

we're on track to solve that problem.

Also, it seems like the sort of problem

that you could think you solved when you

haven't actually solved it, right? Uh

that's a big reason why this is scary.

So, for all those reasons, it's possible

that we'll end up essentially creating a

new species that ends up ruling the

world instead of us. And then maybe we

go the way of other extinct species in

the past that were outcompeted by

humans. That's one possibility. There's

many more. Even if you're not worried

about that and you think that the AIs

will be totally controlled,

there's the question of who controls the

AIs,

right?

When there's a couple corporations that

have made these superintelligences and

are using them to automate all the jobs,

well, that's a lot of power, you know?

That's a lot of money. It's a lot of

political power. They'll have the best

strategists, the best advisers, you

know, they'll think faster. Militarily,

uh, the countries that has these AIs

will be able to absolutely wipe the

floor with all the other countries. The

AIs themselves, it's it's kind of a

single point of failure like central

uh, control system where,

you know, the CEO of Anthropic, Dario,

he coined this phrase, "The country of

geniuses in the giant data center." That

was his

phrase to describe what they're trying

to build, you know?

I think that's a little bit misleading.

I think it would be more accurate to

describe it as army of geniuses in the

data center because

it's not like it's a bunch of diverse

different AIs,

you know, living in their different

parts of the data center. They're all

copies

of the same big model and they're owned

by the company. And so,

they all follow the orders given by the

company, right? People should be asking

questions of like, who controls this

army or these armies and what are they

going to be doing with them?

I think that we could very easily end up

in a sort of

uh, a situation where

some tiny group of people are

essentially oligarchs or dictators. And

ironically,

both of these risks, the loss of control

and the constitution of power,

are things that people in the industry

have been thinking about for decades.

Um, even before the AI industry existed,

you know, people thinking about AI were

talking and writing about these things.

And then part of the founding narrative,

the founding myth of DeepMind and OpenAI

and Anthropic is these problems are

real.

So, we need to get there first so that

we can handle it responsibly. Those are

I think the big two reasons, but then I

can go on. There's lots more reasons as

well. So, one thing is

you know, World War III, geopolitical

conflict. Um if AI does in fact get

incredibly powerful, that's going to

change the balance of power between

nations. That's going to disrupt a lot

of things.

That puts us at increased risk of crisis

more generally, right? Another one, what

about those jobs?

You you're going to lose your taxi job,

but not just the taxi driver, everybody

pretty much.

Um there might be a few exceptions like

people whose jobs for legal reasons are

only allowed to be done by humans, but

for the most part, everybody should be

afraid that their jobs are going to be

lost even if we manage to avoid all the

other problems, right?

>> This narrative has started to emerge and

I've had several interviews on the show

where I've interviewed people who are

very very scared and anxious about AI.

And these are people that have worked in

the industry for sometimes decades.

>> Yeah.

>> Um the counter narrative coming over the

hill is that this is doomerism.

That these people are for whatever

reason just trying to scare people and

that they don't really understand what

they're talking about. How do you

respond to that sort of counter

narrative? And you must have seen this

emerging yourself, especially from

people who stand to benefit, dare I say?

>> Yeah, exactly. This counter narrative is

fairly recent and it's been pushed by

the people who stand to benefit

um from it and it's not true. Like these

these concerns have been around for

decades since before the AI industry

existed.

They're actually pretty reasonable

concerns. Like if you take the companies

at their word and imagine that they are

in fact going to build

superintelligence,

well, it raises a lot of questions. Like

who's going to control it? Will anybody

control it? What about the jobs? You

know, like th- these are just kind of

obvious

implications to be thinking about and

worrying about.

>> Who are you and what's your story?

>> My name is Daniel Kokotajlo.

Um

I currently run the AI Futures Project,

which is a small nonprofit that

mostly focuses on forecasting the future

of AI.

Before that, I worked at OpenAI.

>> AI forecasting?

>> Yeah, so

think about how like

you know, industry analysts who work for

hedge funds and stuff will make these

forecasts of like

here is, you know, how many cars Tesla

will be selling 5 years from now or like

here's what the price of electricity

will be in 2 years, right? That's

forecasting. I was doing that but

specifically focused on AI.

The reason I was doing it is because

it's incredibly important to to see

where this is all headed.

>> Why did you go to OpenAI? What did you

do there? What did you observe while you

were there and how did it change your

perspective on the future of

AI but also I guess OpenAI as a company

and for anybody that doesn't know OpenAI

are the company that produced ChatGPT.

>> Yeah, so I went to OpenAI in 2022.

Uh a large part of what I did there was

more forecasting. AI 2027 is a scenario

that you may have heard of. I did like

smaller

you know, lower effort versions of them

internally for just internal circulation

of like here's some guesses as to what

the next couple years might look like. I

also worked on evaluations for dangerous

capabilities. So

you know, trying to measure the AI's

cyber abilities or persuasion abilities

or situational awareness and I also

briefly was on a

uh a capabilities team doing

reinforcement learning to create agents.

AI is in fact getting

uh a lot better and I can say more about

why, you know, scaling laws, um deep

neural nets bigger, trained on more

data, become more efficient, more

competent at those things.

I also

became a bit more disillusioned with the

AI industry. So

OpenAI, Anthropic, and DeepMind all had

these sort of founding narratives of

like yes, these risks are real but

we've thought about them and we're going

to try to handle them responsibly and

that's why it's important for us to

keep doing what we're doing and I

increasingly came to think that these

were rationalizations

to justify what they were rather than

sort of like deeply guiding their actual

behavior and that when push comes to

shove they'll follow their incentives

rather than

do what's actually good.

>> So you're inside OpenAI at the time and

you start to believe that they're

following commercial incentives versus

the I guess social or societal

incentives that they founded themselves

on.

>> Sort of. I mean what I wouldn't actually

describe it as commercial incentives. I

think I would describe it as

um

power-seeking incentives. So

like [clears throat]

it's true that the companies care a lot

about making a lot of money

but especially at the very top of these

companies like the leaders

they understand that this is about more

than just money. You know?

There are these emails that came up in

you know the the lawsuit between Musk

and and um OpenAI.

A bunch of emails were surfaced in that

lawsuit which you can go read and in

some of them

the founders of OpenAI were talking back

in like 2017 about how the reason why we

made OpenAI

was because we were worried that

Demis Hassabis at Google was going to

become dictator with AGI. Even back then

they were this obviously about more than

just money. Like these these powerful

CEOs are literally afraid that

if the other guy gets there first he

might become dictator and they don't

trust each other and so that's why

they are racing as hard as they can so

that they're the ones who get there

first so to speak.

>> Have you met Sam Altman?

>> Yeah.

>> And did did that shape your opinion of

his incentives or what why he's doing

what he's doing? Cuz there's a lot you

know speculated about what his

incentives are.

I mean his most recent narrative says

for the good of humanity. I think that's

what

>> Yeah, I mean I think the main thing I've

learned is don't pay attention to the

narratives. You know like uh what they

say to one person is just different from

what they can say to some other person

at the same time and what they say in

public is a third thing entirely. I

think you should judge people by their

actions not by their words.

>> And why are you no longer at OpenAI?

>> Largely the reason that I mentioned. So,

I became gradually disillusioned with

how the company was going to behave.

For example,

when I first joined in 2022, at least

the people I talked to, my colleagues at

the company, there was this general

sense of like, of course we wouldn't

actually just build super intelligence

as soon as possible. Once we started

getting really close, like once we

started getting to AIs that could

maybe automate the AI research process,

we would pause and figure out how to

make it safe.

That's cuz we're the good guys and

that's obviously the safe thing you

should do rather than just going full

speed ahead. But, we're worried about

other people who might not pause, you

know, our competitors, Google, for

example. And so, that's why we need to

be in the lead so that we have that room

to do the safe stuff, right? That was

sort of like a thing that seemed like

maybe like the median position or

something among the colleagues I talked

to when I was there when I started,

including people like Sam, you know,

including the leadership. And then by

the time I left, I was like, "Oh man,

they're really not going to do that, are

they?" Like

>> [laughter]

>> Like they they've sort of

you know, partly because this has become

more politicized and they've become

bigger and been under more scrutiny,

people have started asking like, "Why

are you doing this in the first place if

it's so risky?" And so, they've pivoted

their narrative to being more like,

"Actually, it's not that risky, you

know?"

Um

and so, yeah, I mean, it seems like

they're just going to keep going

roughly as fast as they can and hope

that they can figure it out on the way.

>> How did your time at OpenAI come to an

end?

>> Uh I resigned in 2024. I had a nice

goodbye party.

>> What were the reasons you gave for

quitting OpenAI?

>> I thought that we were rationalizing too

much and that we needed to think more

about what would actually be good for

the world. Um I wanted more freedom to

publish.

So, at OpenAI, as it became a bigger

company,

it became more of a normal tech company

with incentives and, you know, a PR

department and things like that. And so,

it started becoming more difficult to um

to publish the sort of research that I

was doing. For example, those scenarios

that I mentioned, couldn't uh couldn't

publish those, right? They're just for

internal use.

I thought that that was a shame because

right now most of the world is kind of

asleep at the wheel and doesn't really

realize what's going on with AI and

doesn't really realize what's coming in

the pipeline a couple years from now.

And the companies aren't really

incentivized to tell people that much

about it. I mean,

they say some vague stuff in a sort of

hypey way, but

um

you know, well, they didn't want me to

publish the scenario, for example,

laying out like here's

how things might actually look.

>> I'm just kind of super curious as to

what it's like being in a company like

that when they you know, chat GPT-3 is

released. You were there at that time,

right?

>> Mhm.

>> Um which was a moment where I think the

whole world stood up and realized that

this technology was

powerful.

>> Yeah.

>> Um and the conversation really began

from a society level.

Um company starts growing super quickly.

>> Yeah.

>> Quicker than I think anybody could ever

have imagined.

And what what was it like inside there?

What did you see change um over over

that period of time?

>> I remember one all-hands meeting where

Ilya said something like

>> Ilya being

>> Ilya Sutskever, who was um head of

research at that time. He said something

like, "Okay, now the world is starting

to pay attention. Each of you is going

to be the most popular person at every

party

uh for the next year.

Don't let it get to your head. Focus on

the mission. Got to build AGI."

>> [laughter]

>> The company grew a lot. It already

wasn't really feeling like a nonprofit

when I joined, but it definitely didn't

feel like a nonprofit by the time I

left. Um lots of new people came in.

Ironically, the like

amount of conversation about

superintelligence and the implications

of superintelligence arguably you sort

of went down over time

due to this growth, right? So, because

the company would like double and then

double again and then double again, all

these new people were coming in from

other parts of the tech industry who

hadn't really been thinking about these

things and were attracted by the high

salaries.

>> You lost $2 million

for not signing an anti-disparagement

clause,

which would mean you could speak you

couldn't criticize the company.

>> Ah, yes. Well, so um I got to keep the

money.

>> Oh, you got to keep the money?

>> what happened was after I had left, said

my goodbyes, etc.

Um I got the the exit paperwork and it

included this clause that said you

basically have to agree not to criticize

the company again.

Um and also a clause saying you can't

tell anyone about this.

And so

I thought that was kind of

rich coming from a nonprofit that's

supposed to be,

you know, for the benefit of all

humanity. So, I didn't sign it. And if

you don't sign, you don't get to keep

your equity. So, your compensation, you

know, what what they pay you is a bunch

of money and then also a bunch of

stock, basically. But then they had this

stuff in the contract that

they get to yank back your your stock if

you don't sign this thing.

Um

and my wife and I, you know, were

uh upset about this. We talked about it

for like a month or two, consulted some

lawyers, um and then ultimately decided

to just refuse to sign.

>> Which would mean you lost you would have

lost $2 million.

>> That's right. Which was like 80% of our

net worth at the time.

Um fortunately, uh

it didn't go the way we expected. It

blew up basically on the internet. Like

when people heard that that we had done

this and that we had said no, it became

like this huge scandal. Employees at the

company started like asking questions in

Slack and like asking leadership like,

wait, what? Like why are you going to

take away our equity? What is this? You

know, cuz a lot of people hadn't really

noticed this before. It had been

whispered about, but it hadn't been sort

of like

a thing that most employees knew about.

Um and so they backtracked and they

said, "Never mind, never mind. We'll

change the paperwork. You can keep the

equity.

It's fine."

>> And so management came out and said he

was embarrassed that he didn't realize

this was going

>> Yeah, he had no idea, apparently.

>> You don't believe him?

>> No.

I think he probably knew. And if he

didn't know, then people close to him

probably did, such as his head lawyer.

>> Why did you decide not to take the $2

million?

I mean,

most people would have, I think.

>> It's true, most people would have, and

most people did.

And you know, money is nice, but like

it's not the only thing, you know?

Sometimes it's good to take a stand on

principle.

I I keep mentioning superintelligence.

Perhaps I should say more about like

the

the sequence of events that the

companies are planning to do.

So,

right now, they're focusing on

automating coding. They're taking their

AIs, they're making them bigger, they're

training them for longer, and they're

especially focusing the training on

getting them to be good at autonomously

writing and editing code. Because

uh that will help the companies go

faster, right? If they can automate the

code, then they can do their own work

better and faster, and accelerate

progress.

The next step, which they've already

begun, is to

look at the rest of the research process

as well. Coming up with ideas,

um analyzing experiments, communicating

those results.

All the other parts of of the research

process, they're trying to figure out

how to train AIs to be good at those as

well.

So that they can have AIs do the entire

thing autonomously.

>> When you say do the entire thing, what

you mean [clears throat]

do the entire thing?

>> So like Anthropic and OpenAI in

particular are trying to automate

themselves. Like they're trying to make

it the case that

um they don't really need human

employees anymore. Uh they just have a

giant army of AIs that's

churning away,

doing all this autonomous research to

make better AIs, to train the new AIs,

put them in charge, so they can make

even better AIs and so forth. And of

course, not just not all just happening

internally, but also like interfacing

with the world, right? Like going out

and talking to people, collecting the

data, setting up the training

environments,

doing the business deals, and so forth.

Like they're they're trying to automate

all of that. The reason why they're

doing this is because they're trying to

get to a position where they have

AIs that are superhuman

at everything, superintelligence, and

they're trying to get there before their

competitors do.

Needless to say, this is incredibly

dangerous, I would say, you know. And in

addition to being dangerous,

it's a power grab, right? Like if they

actually succeed at this, then they'll

be sitting on top of this army of

superhuman AIs that will give them

immense leverage over all sorts of other

actors in the economy in so far as they

can work out something with the

presidents and, you know, integrate it

into the military or whatever, then that

would give the US immense hard power

over all of the countries, right?

Obviously, nobody knows exactly when

this is happening.

But a very disquieting thing has

happened over the last year to me,

which is that when we published AI 2027,

people were generally of the opinion

that my timelines were too short.

And that like probably it would take

more than 2027 until we got to

the sort of events that I was just

mentioning, you know, uh recursive

self-improvement, AIs automating the

whole research process,

superintelligence.

These These types of milestones

um they happen in 2027 in AI 2027,

>> which is this research paper you

published.

>> That's right. It's It's a scenario

forecast that sort of lays out like

month by month a possible future

trajectory. There was sort of like At

the time that we started writing, it was

my best guess as to what would actually

happen. Obviously, there's lots of

uncertainty, but, you know, I thought

it's valuable to make a concrete guess

just to sort of see what it might look

like.

And at the time we were writing this, a

lot of my friends in the AI industry and

in nonprofits and so forth that work on

AI, a lot of people were saying like,

"Yeah, that stuff's going to happen, but

like it'll probably take a couple years

longer than you think."

And now

it's more 50/50, especially when I go

talk to people at Anthropic and OpenAI.

They're often like,

"Yeah, no, 2027, that's basically what's

going to happen.

Just like you wrote. Why did you Why did

you become Why did you update your

timelines? Oh, yeah, context for this is

after after writing AI 2027,

I shifted my timelines to be a little

bit more conservative. So, at the time

that we published, my 50% mark was in

2028, not in 2027.

And then after we published, progress

just seemed like it was going a bit

slower, and so I updated to 2030.

Which is, you know, still could happen

sooner, could happen later. 2030.

Um but now, when I talk to people in in

the company, they're like, "It's not

going to take that long."

They're like, "Oh, you need to shorten

them again. Like, get them back to 2027

or 2028, you know."

Um so, that's a bit disquieting. Um

again, don't know how long it's going to

take, but this is the stated plans of

the uh companies is to do this

incredibly dangerous thing, and they

think that they're just a few years

away.

>> So, you wrote this um report here, What

2026 Looks Like, and you wrote this in

2021,

and it was remarkably accurate. Helped

make a name for yourself amongst um

amongst uh everybody in AI. And I Which

one was it that J.D. Vance, the vice

president, read? I think it was this

one, wasn't it? Yeah, this one. Um

and then so, then you published this

one, AI 2027, and this was published, I

believe, in 2025.

>> Uh yes, that's right. April.

>> Yeah.

>> What were you forecasting in here? What

are What are the key things that you

said in here for people that haven't

read it?

>> The high-level version of it is

they automate the coding, then they

automate the rest of the research

process, then the pace of progress

accelerates dramatically. They get to

superintelligence. They're working with

the government, specifically the

president, the executive branch

naturally wants to control this

technology, in other words, wants to use

it to beat China and integrate it into

the military and so forth. By this

[snorts] point, it's sort of

doing basically all the work itself. I

mean, it's it's superintelligence, so

it's coming up with all these great

ideas for how to integrate itself into

everything and all these new

technologies it's invented and so forth.

And uh because of the race dynamics and

because of the profit motive, they end

up deploying it everywhere. And it

builds robot factories that build more

robots that build more robot factories,

etc. Transforms the world entirely.

And then at some point it has enough

power it, meaning the AIs, have enough

power that they don't have to pretend to

to be aligned anymore.

Right? Um then they

stop listening to orders.

That's the race ending

of the 2027.

We also wrote a sort of different

branch, which is the slow down ending,

which is intended to sort of illustrate

the concentration of power issues um

that I mentioned previously. So,

what if hypothetically

the alignment issues get sorted out

sufficiently quickly? Like what if it

turns out that like

it's not too hard. With 2 months of slow

down, we can figure out how to make the

AIs robustly do what we want um and have

the values that we want them to have.

So, that's one possible branch. And in

that branch, uh it looks pretty similar,

you know, they take the jobs, beat

China, etc. Um

but instead of the AIs ultimately

killing everyone, they create this sort

of amazing utopia. But the amazing

utopia is

whatever the people who control the AIs

want it to be, right? And so that would

be a very small group of people, like

the presidents, some CEOs, etc.

>> There should be a button just down below

here. And if it says subscribe, you're

already subscribed. If it says subscribe

buh, that means you're not yet. And if

you're not subscribed, please could you

do us a favor and hit that button. It

helps to show more than you know. And

according to the algorithm, you're

someone that watches our show, but you

haven't yet hit that button. Thank you

so much. Is there any possibility, do

you think, that we never get to this

thing called AGI? And and how do we

distinguish AGI from this term super

intelligence? What's the difference?

>> Yeah, so the difference is that AGI is a

more vague uh and weak term.

>> Okay.

>> So, super intelligence is a bit more

precisely defined. It's better than the

best humans at everything, faster and

cheaper. Um AGI is more like it stands

for artificial general intelligence,

which means AIs that can do things in

general rather than like some specific

task. Yeah. And so arguably we've

already achieved AGI, right? If you use

cloud code or something like that, it's

like it can do a lot of stuff. It's it's

almost kind of like a little employee

that you can like have go do stuff. So

it's it is quite general.

It's not maximally general though. Can't

do everything. Whereas super

intelligence by definition

can do all the things that a human can

do but better.

>> And how does this sort of overlap with

robotics? Because obviously that we're

seeing this huge robotics boom at the

moment. There are some real world things

that humans can still do because these

AIs are still stuck in my computer.

>> The way that people talk about this is

that they

basically just say we've achieved super

intelligence for cognitive tasks. Then

you can talk about like

full super intelligence that can do the

physical stuff.

>> And are we going to get there? Are we

going to get there with both?

>> I think so. I mean again, this is not

something that we can be certain about.

Um, you asked like is it possible we'll

never get there? Yes, it's possible

we'll never get there.

I don't think it's likely though.

I think that

there's nothing sort of like magical

about the human brain. It's

you know, um, it's just a bunch of

neurons. It is possible for a digital

system to

do similar functions in the same way

that like,

you know, a plane can fly

just like a bird. Not in the same way as

a bird necessarily. Like it doesn't have

it's not flying in the same way that a

bird flies, but it flies, you know?

Um, so so it does seem like yeah, like

seems possible.

>> You've written all these, you know,

these research reports. You're working

on another one that'll be released um,

likely on the 9th of July.

You have worked inside OpenAI. You then

quit OpenAI because you were concerned

about what was going on there and about

the future of the industry. You know

more than I do.

Are you optimistic about the future or

pessimistic? Are we heading to a bad

place if things don't change um, based

on everything that you know?

>> I think we are headed to a bad place if

things don't change. Um, I'm not

confident in that. I would say something

like 70%. It's very very hard to

predict, of course, but yeah, it seems

like the current default path is heading

towards a very, very scary place.

>> How do you contend with that personally

and emotionally?

>> Um

it's rough. I mean, I think it It's the

sort of thing that like

gets me down

on a regular basis, but also I've been

dealing with this for so many years now

that

I've sort of gotten used to it, if that

makes sense. Um

yeah. Yeah, I I'll put it this way. I

would be incredibly happy if all my

predictions turn out to be wrong and

uh and AI hits the wall, for example.

>> It gets you down on a regular basis.

>> I used to be known as a pretty chipper

and optimistic person, but

um in 2020

my AI timelines predictions started

collapsing due to GPT-3 and the scaling

laws papers and um the bio anchor

report, which I I can talk about if

you're interested, but basically some

events happened in 2020 that convinced

me that actually this stuff was like

quite plausibly coming by the end of the

decade.

And

humanity is very obviously not ready for

this, you know, in a whole bunch of

different ways. And so that's obviously

very scary.

>> And that's a extremely scary world

because of all the things you've said,

but but again, because of this recursive

self-improvement where AIs can train

themselves. And at such point we're

starting to lose hold of what's going on

here.

>> I mean, the AIs are already training

themselves, to be clear. It's more like

closing the entire research loop, right?

So

>> everything.

>> Yeah, like right now a lot of the

training data is generated by AIs. A lot

of the reinforcement, like the grading

that happens, doling out of positive and

negative reinforcement, is itself done

by AIs.

>> Can you explain that in layman's terms

for

>> Yeah, so an important thing for

everybody to understand is that modern

AI systems are not software in the

normal sense. I mean, they are

technically software, but

they're not lines of code, you know?

It's not like some engineers at

Anthropic went and wrote lines of code

that basically says like, you know, when

the user asks for this type of thing,

then go do this type of thing for this

many steps or whatever. There's nothing

like that. Instead, it's a neural net,

you know?

>> What's that?

>> Well,

think about how the brain is a bunch of

neurons connected to each other

>> Yeah.

>> that are firing

um signals back and forth. The brain

learns over time

the types of patterns of firing that

caused success, that caused a dopamine

rush, or various other types of feedback

get reinforced and fire more often. And

the types of patterns that caused

failure, like touching a hot stove, get

anti-reinforced, they get, you know,

um destroyed, so that they fire less

often. And as a result of all of that,

you over the course of years learn to

act in the world, and you learn all

sorts of skills, and you learn world

models, you learn like beliefs about the

world, and you can sort of like mentally

simulate how it's going and stuff like

that. So, artificial neural nets are

like that, except artificial. So, it's

it starts off as a giant

tangled spaghetti mess of randomly

generated uh

artificial

connections called parameters.

These days, they might be something like

10 trillion parameters

uh it in the biggest AIs.

So, it starts off randomly generated.

So, it's of course completely useless.

Like, if you

give it some input, it'll just produce

gibberish as an output. But then they

train it, and they

start with pre-training, which is where

you give it a bunch of internet text,

and you show it the first piece of text,

and you put that in as the input, and

then it gives a gibberish output,

and then you positively or negatively

reinforced it based on how accurate that

output was at predicting the next piece

of text. Um so, it's basically playing

this game of like predict the next word.

>> Isn't that how it happens with babies? I

had a I think I had a neuroscientist

tell me that babies have more neural

connections

um than adults. And yeah, it says yeah,

toddlers have twice as many neural

connections as adults. And they, I guess

they whittle down through reinforcement.

Yep. We have more pathways when we're

younger. And just like the process of

training an AI, we're trained down to

like remove the ones that aren't useful

and build up on the ones that are.

>> Yeah, it's both pruning and

strengthening. And it seems like in

humans it's actually more pruning than

strengthening, but it's both. Uh, and in

AI it's the same thing, it's both. So,

the first portion of training is where

they train the AI to predict text, which

is kind of like training it to read. Um,

and it it's a similar thing does happen

in humans. So, basically,

the the random tangle gradually takes

shape and gradually sort of coalesces

into more useful circuitry that has

stored lots of facts about the world and

has stored lots of skills for how to,

you know, process information and

transform it and then produce

predictions.

That's just the first step. After they

do the pre-training, then they

try to teach it more useful skills

besides just predicting text. And so,

you know, by the end of the process,

they've thrown lots of coding problems

at it. And they've said like, here's a

coding problem, go. Here's a coding

problem, here's an environment, you have

access to this virtual computer, here's

like the code base you're working with.

You can write code, you can edit the

code, you can run the code, you can read

it, you can use the internet.

Go, go, go. And it does that for a while

and then based on how successful it is,

reinforcement happens and they have

thousands, maybe millions of examples of

coding problems like that that they

trained it on. And that's why they're so

good at coding now.

>> So, what does superintelligence look

like in this regard? Is it just more of

these connections? And how would they

get more connections? Can you explain

that to me like I'm

>> So, there's different AI models, right?

So, there's like,

you know, GPT-3 and GPT-4 and GPT-4.5

and GPT-5 and GPT-5.5 and 5.6, right?

Sometimes they're just the same previous

model but with extra training. Sometimes

they're are new model that's been

trained from scratch, including starting

the whole pre-training process again.

Over the last couple years, they've done

several new rounds of starting over from

scratch. And typically when they start

over from scratch, they make the whole

thing bigger, the the artificial brain

much bigger. Right now they're at

something like 10 trillion parameters.

Back in 2020, um

it was more like 175 billion.

So, we've grown like two orders of

magnitude

uh in 6 years.

>> Two orders of magnitude.

>> Yeah, like two 10 x's. So, 100 x, right?

So, that process is continuing. Um

they're also improving the algorithms

themselves. So, they're not literally

just the same type of AI but bigger.

They've also come up with all sorts of

ideas for how to change the structure of

the of the connections in the neurons

and so forth and change the like

reinforcement

algorithms that they're using and to

change the training data that they're

training on.

All sorts of tweaks that have made this

whole thing more efficient.

>> We're literally building a brain.

>> Basically, yeah. As they make more

brains, they're getting better at making

They're making them bigger and making

them more efficient and so forth.

>> And it's literally modeled on the brain,

like the way it works, right?

>> It's It's certainly heavily inspired by

the brain, but I I shouldn't overstate

the the analogy. Like there's lots of

differences, too. So, for example, the

transformer architecture um

>> Which is

>> Which is the architecture that they use

for for these LLMs

uh is not really recurrent. So, the

information sort of flows one way rather

than allowing all these sort of little

loops on the inside. Also, the the

backpropagation algorithm is different

from the sort of um learning that

naturally happens in human brains. So,

there are some differences, but yes,

like broadly speaking, uh we are sort of

making artificial brains. It's kind of

like for brains what like a plane is for

a bird.

>> Mhm. Yeah, that's a [clears throat]

really good analogy.

>> Yeah.

>> That that analogy helped me think

through a bunch of questions people

often ask about AI when they said, "Can

it be creative?"

But actually that analogy kind of helps

me understand that actually that maybe

that's not the question.

It's can it produce something that you

would consider to be creative because

[clears throat] creativity is people

think of it as like a process, but

actually it's it's judged based on the

output, isn't it?

>> I mean you you can get philosophical

about like is it truly creativity that

they have, but you can also be like

well, I mean just look at all the stuff

they're accomplishing,

>> [laughter]

>> you know, and it seems like they're

going to be accomplishing a lot more in

the near future.

>> Yeah, I do I I asked the question about

how this weighs on you personally

because I can I can sense that you're

actually personally bothered.

>> I mean that I think the situation is

crazy. Like

first of all, it's very exciting. Like

AI is really fascinating and interesting

stuff. I've been following the field for

more than a decade now.

I've been part of it

for some years and um

it's really cool, really interesting and

it's really fun to think about what's

going on inside these artificial brains

and why they are the way that they are

and it's really cool to see all the

applications of this technology out in

the world.

But it really seems like we're on a

pretty scary path and the more you think

about it, the more worried you get and

you know, in stories

it always ends well, but this is real

life.

And I I think we have to sort of

stare reality in the face and tell it

and realize that like it might not

actually end well, you know.

>> Were there any recent

dare I say I was going to say eureka

moments, but paradigm shifting moments

where even your own sort of mental model

of what's going on here and how this is

going to look were changed for better or

for worse?

>> For better or for worse and probably for

worse, things are kind of on track for

AI 2027. There are a few things that

have been different not exactly like

paradigm shift differences, but like

there have been some differences from

what we expected at the time we wrote

this. So

the government has actually got involved

faster than we expected and has been

more aggressive than we expected. So the

export controls on mythos being the

biggest example and also threatening

Anthropic with

being destroyed by the defense

production production act.

Um

>> [clears throat]

>> Another thing that's been surprising to

us is that Anthropic in particular has

gone from second place to first place in

the sort of in the race basically.

>> Why do you think that happened? Because

it seemed like ChatGPT were out front

and clear as it relates relates to

OpenAI were out front and clear but

suddenly Anthropic have uh

lapped them.

>> Yeah, I mean I guess they have um

probably higher talent density

um and better strategy

but not by a lot but enough to make the

difference.

>> Why do you think they have more talent?

>> Well

they don't have more compute. Like what

are the inputs, right? Like they're in

the lead now, they used to be behind.

What are the possible explanations for

this? Well, it could have been that they

had more resources like more compute

more money but that's not true. They

have less resources less money, right?

So then I guess talent's what is is the

next best alternative. You could maybe

say strategy.

Some combination of those things, yeah.

Something that wasn't just like the

amount of resources they had.

>> Just like Jon Jones where marginal

improvements in your cognitive

performance can have a massive impact.

Sometimes I podcast for 10 hours a day.

Over the last couple of weeks I've been

in filming for a TV show and then I have

like one or two days off to get all of

my work done which means there's lots of

cognitive load. And so I turn to ketones

because I find myself more articulate,

able to think more clearly, able to work

out better when I'm fueled by ketones.

And so the reason I became a convert of

this company and the reason why they now

are sponsoring this podcast is because I

remember one of my team members called

Christiana, she tried it once and came

up to my desk and she goes this is the

best product ever made. And I think in

part that's because she really cares

about those cognitive benefits as I do

as Jon Jones does and as I think most of

my listeners probably will. So if you

haven't tried these yet, all you have to

do is go to ketone.com/steven

and you'll also get 30% off your first

subscription order. You'll get exclusive

ketone IQ merch and of course cognitive

benefits that might just change your

life.

Much of the reason most people haven't

posted content or built their personal

brand is because it's hard and it's time

consuming and we're all very very busy

and if you've never posted something

before,

there's so many factors in your

psychology that stop you wanting to

post. What people will think of you. Am

I doing this right? Is the thing I'm

saying absolutely stupid? All of these

result in paralysis which means you

don't post and your feed goes bad.

I'm an investor in a company called

Stand Store which you've probably heard

me talk about and what they've been

building is this new tool called Stanley

that uses AI, looks at your feed, looks

at your tone of voice, looks at your

history, looks at your best performing

posts and tells you what you should

post, makes those posts for you. You can

also you just use it for inspiration and

sometimes what we need when we're

thinking about doing a post for our

social media channels is inspiration.

Building an audience has fundamentally

changed my life and I think it could

change yours too. So, I'm inviting you

to give this new tool a shot and let me

know what you think. All you have to do

is search coach.stand.store

now to get started.

>> Uh a friend of mine who knows some of

these people sat me down once upon a

time in London. He's actually said this

a few times to me but I remember one

particular conversation where he says

that

some of these AI CEOs predict the

probability of extinction at being I

think he said 7%. I don't know why I

have that number in my head but I

remember it being less than 10% and the

point he was making to me was that even

if it was 1%. Like if there was 100

buttons on this table now

>> Yeah.

>> and one of them would end the world.

Would I dare

>> I wouldn't press any of them

>> you know. [laughter]

Um

>> No.

>> I wouldn't press any of them but he made

the case to me that these AI CEOs are

very smart and they understand super

intelligence and that they think

actually if there was 100 buttons on

this table right now, maybe 10 of them

could end the world. I've heard you say,

I think it was on the the the Daily

Show, the interview you did, you said

that you think there's a 70% chance of

human extinction due to AI.

>> I wouldn't say human extinction exactly.

I'd say something like 70% chance that

this goes horribly wrong like human

extinction but that's just one of

several possibilities. But yeah,

basically

Like for example, possibly the AIs take

over and then don't actually kill

everyone.

You know, maybe they do something else.

Like just just cuz they've taken over

doesn't mean they're

definitely going to kill us, right? They

might, but they could do something else.

So that's what that's that's why I don't

usually say like

70% chance of like actual human

extinction, but 70% chance of like

something like AIs taking over, some

some sort of very big catastrophe like

that that could lead to human

extinction.

>> I see what you mean. So two points

there, which is you've been around these

CEOs. I mean you've worked for Sam

Altman at OpenAI before you quit.

Do you think that they think there's a

chance of human extinction?

>> Yes.

But

I think that the important thing to

understand is that

like people sort of believe what they

need to believe in order to think that

they're great people and that they need

to keep doing what they're doing. This

is what rationalization is. And so I

think that the tech CEOs have like

genuinely convinced themselves that like

probably things are going to be fine and

that the way to make things fine is for

them to keep doing what they're doing.

And like they need to like make sure

that like, you know, Sam needs to make

Sam's probably thinking like can't let

Dario or Elon

get there first, you know, I know

Dario's thinking Sam can't get there

first. Elon's thinking that like, you

know, they they they've all probably

convinced themselves that like, oh yeah,

like maybe it'll go horribly wrong, but

like

probably it's going to be fine and

probably

you know,

I should be the one in charge.

>> It appears to me that Anthropic are the

only ones that are all talking about the

potential chance of extinction or

catastrophic event or

um the down the real downside still.

They seem to be the only ones that are

still publishing on it and now they're

actually becoming the enemy in many

respects of the

>> Yeah.

>> the tech industry in San Francisco. I'm

watching a lot of interviews and it's

everyone's attacking Dario because he's

saying, "Listen, things could go bad."

They're calling him a doomer uh and

questioning his incentives. Even with

Mythos, which is a an a Claude model

that they started to warn the world

about, again, he is attacked immediately

for saying that.

>> Yeah.

>> My question is, do you see him as being

slightly different from Sam in this

regard?

>> Yeah, I mean, it seems like Anthropic

and Stereo have been more willing to

say and do things that are costly to the

bottom line.

Uh and at least in the last year or so.

That's an example of it. Um like I don't

think that really wins them favors in

the administration or among their

investors to say that type of thing. And

you know, a better example is just the

whole fight between the Department of

War and Anthropic was an example of them

doing something that like cost them a

lot of money and even more importantly

cost them a lot of power

for

something that like like they could have

just signed the contract, you know.

That said, I really don't want to be in

a situation where we're like, which CEO

is the least bad CEO? Let's support that

one. You know, like none of these people

should be trusted

uh with that much power, basically.

>> Nobody should.

>> Nobody should.

>> Regardless.

>> Regardless, yeah.

>> Mhm. So, uh on this point of the

buttons, you you you do believe that

they think there's a credible chance of

extinction.

>> Yeah, but they've [clears throat]

convinced themselves that like it's

probably fine and also it'll be even

worse if I'm not doing it, you know.

Like that's that's what they'll say

inside the companies, too. Like the two

people will be like, okay, well, if we

stop,

what about the other guys? Like they're

not going to stop, you know?

>> Yeah, this is this is always been why

I've had this outstanding question,

which is how does this not go bad when

human incentives seem to rule the day

when you look at history and all of the

human incentives are saying, well, if

you you're damned if you do,

I you're damned if you carry on

developing these bigger and bigger

bigger AI brains, but you're also then

damned if you don't from an a

geographical perspective cuz the United

States will lose to that country or this

company will lose to that company. So,

when you just look at human incentives

and goes, how does how does if just you

purely incentives and disincentives, how

does this end? Well, it carries on

going.

>> Seems like it. I mean, there there is a

caveat to that, which is a hopeful

caveat, which is that

first of all, if the world wakes up to

all of this,

then there can be a more serious

conversation about regulation and

international treaties and things like

that. And that can change the

incentives, right? So, the government

could come in and say like actually

here's some rules that you all have to

follow. And because they're rules that

you all have to follow, then you're not

incentivized to like

break them anymore because you get

punished if you break them and

everyone else is also following them,

too. And so, you know, it's fine. So, so

there is that sort of like ray of hope

that like we can change the incentives

if the government and especially the US

government but then later other

countries act to to change the

incentives. But that's not going to

happen until people sort of wake up to

all of this.

The second thing is that even

individually

at some point

you know, Dario or Sam or Elon might

realize that like actually it's like not

even in their own interest

to to keep racing unilaterally.

And it it on the problem with that is

it's only if it gets extremely obvious

and extremely dire. So, like

in in AI 2027, in that scenario, there's

this choice point that I mentioned. And

in one case the AIs are misaligned and

the other case the AIs are aligned.

At that choice point, we have like one

branch that depicts the the misalignment

ending and one branch that depicts like

they they slow down a bit and solve the

alignment issues.

>> Mhm.

>> The instigator for that choice point is

they see some evidence that their AI

might be misaligned and plotting against

them.

Right? So, if you actually see that

evidence

then it's like

oh gosh, uh

maybe we shouldn't put it in charge of

everything and let it rip, you know?

Because that evidence is staring us

right in the face that it's this

untrustworthy, you know? But if they

don't see that sort of very clear

evidence, then

I think they're going to convince

themselves that they need to keep going,

you know? But maybe they will see very

clear evidence like that. In which case,

even if we don't have regulation, they

might just sort of voluntarily stop.

Um so, that's the second ray of hope.

Like overall, I don't think that we're

like definitely doomed, you know? Like

[snorts] I said 70% but like

I could see it working out pretty well

as well.

>> Hm.

What about uh jobs?

>> Yeah.

So, I think I think I'm excited to at

some point get into the new thing which

is the more optimistic

>> [clears throat]

>> positive vision.

Uh and that will have a lot to say about

this.

Because in the in the in the prediction,

you know, in the year 2027, by the time

everyone loses their jobs, there are

worse things happening. Or like it's

it's kind of like too late by that

point. Um but yes, like once if I mean

just just think about it. If the

companies do manage to build

superintelligence, then by definition,

they're going to be able to take almost

all the jobs or all the jobs, right? Cuz

it's better, faster, and cheaper than

the best humans at everything.

>> And that's I mean, the timeline is by

the end of sort of 2030, you reckon you

think superintelligence might arrive.

I'm trying to think about when we could

start to see job displacement in the

economy.

>> We're already starting to see a little

bit of it now, but not very much.

>> Why?

>> Um cuz the AIs aren't good enough yet.

Like they're they're they're they're

impressive, but they're not like

they're not just a a drop-in replacement

for a human worker in almost any field.

>> And do you think that will be sudden?

>> I think it'll be sudden because of the

intelligence explosion dynamics or

recursive self-improvement dynamics. So,

you could imagine a different world

where

it's gradual.

>> Mhm.

>> And and this [clears throat] is this is

maybe how it is in a lot of science

fiction is,

you know, the AIs gradually get better

at a bunch of things and

you know, they gradually automate like

this one industry like pharma, then they

automate like

you know, steering drones, then they

automate like driving cars or something

like that. Um

but what's different about the real

world is that the companies have

converged on this strategy of automating

themselves first.

You know, automating the AI research

process.

And so,

if they are allowed to continue with the

strategy,

we're not going to see like,

you know, the robot taxis and like the

plumber robots and

you know, the lawyer AIs. We're not

going to see that sort of like broad

diffusion of AI into the economy

happening first because that's not what

they're focusing on first. They're

focusing on automating themselves,

automating their own research so that

they can do everything that they're

doing faster.

And they want that to sort of get going

and get to

you know,

very high levels of intelligence, very

high levels of general intelligence um

and then deploy more out to the economy.

economy. Right? So,

by the time it's actually coming for

like all these different jobs,

they will have had fully autonomous AI

research happening for months, maybe

years, you know?

And that means that like the AIs will be

vastly superhuman at AI research and

probably also vastly superhuman at lots

of other things just as a side effect,

you know?

If you're wondering what this looks

like, well,

we wrote about what it looks like. It's

sort of like this this wave smashing

through the economy after they do the

intelligence explosion internally.

>> What I'm hearing there is that because

the AI will be able to improve itself

and train itself, it'll be getting

better at everything at once and then

it'll be released at kind of once.

Is that accurate?

>> it's it's not it's not even exactly that

because even if it's mostly just getting

better at the things that it's doing

like research,

that'll have some spillover effects

to other skills as well.

And then when it turns to the focusing

on the those other skills, it'll be able

to do them very fast.

>> What jobs remain in such a scenario, do

you think?

>> I think that's actually a political

question, not a technical question.

>> Because

>> Because on a technical level, all the

jobs can be done by the AIs

if they've reached that level.

And so, it's a question of what jobs are

allowed

for them to do.

>> And what kind of jobs wouldn't be

allowed, do you think?

>> That depends on who's in charge. So,

there'd be some sort of political

conversation about like what we're going

to allow and disallow.

>> I mean, in this scenario, the humans are

still controlling them, the AIs.

>> Depends on what you mean by control,

right? So, there's like

there's do the AIs actually have the

goals and values that you want them to

have, and are they going to robustly

do that and behave as intended into the

future? And then there's like are they

obeying your orders for now?

>> Are they obeying the orders is really

what I'm saying.

>> Yeah. So, like even in AI 24/7 in the

scenario where the AIs take over and

kill everyone, there's a period of like

several years where they're still

obeying orders,

and they're, you know,

taking some jobs but not other jobs, and

they're helping to make better weapons

that the US government can use to like

do its arms race with China and so

forth. And that's why they're able to

get so much power so quickly is because

the governments and the corporations and

so forth trust them and is deliberately

deploying them into all of these

positions because it thinks that things

are fine.

But because these things are neural

nets,

you can't just like look inside and see

what it's really thinking. You can't

really tell.

>> I think this is a really important point

because unlike software where we can

look at the code and see what's going

on, theoretically, with AI you're saying

that we don't know what why it's making

the decisions that it's making cuz we

can't get inside.

>> One note of optimism is that it doesn't

necessarily have to be that way. Like

there's a a subfield of machine learning

called mechanistic interpretability, and

a a broader subfield called

interpretability more generally that's

trying to solve that problem and trying

to take these these trained artificial

neural nets and piece [snorts] them

apart and understand

like how the information is flowing and

how the decisions are being made, so to

speak. Um the problem is just it's a

very inherently hard problem. If you

have 10 trillion connections to look at,

you can look at any particular group of

them and be like, "Okay, so this is how

like this particular connection works."

But like how do you get a sense of the

whole, you know? How do you get a sense

of like

what's happening at a high level? And

the answer is, "Well, it might be

impossible." But people are working on

it and they are making progress, and

if they can make enough progress, then

we're in a very different and much

brighter world. I think that it would be

much less likely for us to get into

those loss of control scenarios if we

could just actually see what our AIs

were thinking and why and how at any

given time.

Right?

>> Yeah.

>> So, we would still have the other

problems to worry about, but at least we

could mostly solve that one.

>> It is pretty crazy to think that we're

building a technology, a brain that we

don't understand.

>> Yeah, it's pretty crazy. I mean, it's

one of those things where like

>> In a movie, like a sci-fi movie, a bunch

of scientists sit around this big brain

and they're all just like they're

they're making it more they're feeding

it.

>> Yeah.

>> And they don't really know what the

it is.

>> Yeah, I mean, it's it's kind of just

like obviously a dangerous thing to be

doing.

>> Yeah.

>> Um but we're doing it anyway because of

this history of how the field has

developed in the last 10 years where

you know, people were like, "Oh wow,

yeah, that's obviously dangerous. Oh no,

what if someone else did it and did a

bad job of it? Therefore, we should do

it and do a good job of it and now

they're in this race where

where they're racing each other and

they're also under all sorts of

political pressure to like pretend that

it's not as bad as it seems because

they don't want to like

anger their investors, they don't want

to anger the White House.

>> One of the the key questions we had from

our audience was which and I kind of

asked you this in part, but which jobs

are genuinely likely to survive AI and

what skills should people {slash}

students focus on over the next 10

years?

>> That's kind of like

like imagine if you were someone living

in Mexico

in like 1500 and then you hear that like

the conquistadors are coming.

You could be asking yourself like,

"Okay, well, what sort of job should I

be switching to to like survive this

transition?"

But like, you have a lot more to worry

about besides that. But yes, I think I

would say that like if we managed to

avoid the loss of control problem

and we end up with humans still

in charge of the AIs and humans can like

say what the AIs goals and values are

supposed to be even as they become much

smarter than humans and even as they run

the whole economy

then probably there will be regulation

that protects some areas

and you can try to guess at what those

areas might be. Maybe stuff that's more

like

like like judges potentially.

>> What about podcasters?

Be honest.

>> Probably not podcasters, I think. Um

stuff like

you know, being a nanny

maybe, right? Like I think that even if

there's a robot nanny that's like really

really good, I think a bunch of people

might prefer to have an actual human

because they might be creeped out by the

idea of a really good robot nanny. So,

you can sort of you can sort of reason

like that. There's also like

stuff that might be legally protected.

Like maybe judges, for example, like are

going to be legally required to be

humans and not robots.

>> Some people say though there's going to

be so many jobs created that we can't

foresee right now like there was in the

industrial revolution or the internet

boom or whatever.

>> The problem with that is that

um past technological advancements have

been more narrow. They've like automated

some things but not everything.

But we are talking about a hypothetical

future situation in which everything

gets automated. So, there isn't any new

job that you could do that AI couldn't

also do.

Except if it's like protected by

regulation or something. That's that's

that's also a thing. But so like for

example, right now there's this sort of

like cycle where

you know

the AI's learn to do a certain thing

like write copy or like draft code or

like debug something.

And then humans who used to do that

thing switch to managing AIs or switch

to doing the other stuff that the AIs

can't do.

And that's why there's been this dynamic

historically of

you know, new jobs opening up and people

flooding to them. But

if it gets to the point where the AIs

can do everything that humans can do and

better and faster and cheaper, then

whatever that new job is that you might

have switched to, that the AIs can

switch to that too and they'll already

be be better at it than you.

>> Because we haven't seen widespread

unemployment yet in the economy, do you

think people are getting a little bit

complacent because what I'm seeing on my

timeline is a lot of people saying I

told you so, I told you everything would

be fine. And when you look at the the US

unemployment rate, currently the it's

flat to slightly down. If you look at

the UK, it is up. The trend is up

compared to last year. We're at about 5%

unemployment. The US is at 4.2%

unemployment.

>> Yeah. Basically, nobody has said that

there would be mass unemployment by now.

Or at least we didn't say that. You

know, and we were historically one of

the more bullish people on AI progress.

In AI 2027, because of the dynamics that

we just described, the mass unemployment

doesn't happen until 2028 or 2029 after

they already have superintelligence.

Because, again, the companies aren't

trying to cause mass unemployment as

step one. That's like step three after

you know, it's like step one, automate

themselves.

Step two,

have this recursive self-improvement to

get to superintelligence. Step three,

expand out into the economy and automate

everything. And so,

this is really unfortunate from

humanity's perspective, because one

might have hoped that

if there was this broad wave of

automation going through the economy,

people would sit up and pay attention

and think about where all this is headed

and demand good regulations from the

government.

But,

that's not actually what the strategy of

the companies are taking. You know,

they're going to be getting the

superintelligence first and then doing

the broad wave of automation, which

means that by the time they're actually

doing all of that,

uh well, it's already going to be moving

very fast and the AIs will already be

very powerful.

>> In your 2027 report, so you wrote that

in 2025, but it is called AI 2027, you

said that in mid-2025 we'd have the

autonomous employee, which is sort of

like AI agents taking instructions over

Slack or Teams.

That happened. I've actually got an AI

agent in my WhatsApp I can talk to. Of

course, you've got Claude by exploded,

obviously, around the world. And and

now, um you know, Claude have talked

about uh their new Slack integration.

But, lots of people are using agents

now. And that happened, I'd say for us

at the We really sort of caught onto it

at the the start of 2026.

You also said by 2026 companies begin

replacing entire corporate departments

with AI agent subscriptions. 2027, the

final job. AI automates the job of the

human AI researchers themselves and

begins the machine learning research to

upgrade and build the next generation of

AIs.

>> Yeah, yeah. So, again, timelines.

We are uncertain about how long it will

take to achieve these milestones. In

this scenario, they happen at those

times, but

by the time we had actually published

this scenario, our timelines had shifted

back a little bit. Specifically, mine

had. So, like

my 50% mark was 2028.

>> Mhm.

>> For that for the full automation of AI

research milestone, not 2027.

Uh

and then other people on my team had

more like 2030, 2031, things like that.

So, I I I kind of want to like

maybe try to illustrate this with the

you know we have like this probability

distribution. It's like a

smeared out probability mass. And like

the 50% mark is this particular year,

but there's like a lot of possibility

that it happens

>> Later.

>> years earlier or years later, right?

>> Got you. What is this AI 2040?

>> So, AI 2027 was our best guess

prediction as to how things would

actually go.

>> Yeah.

>> AI 2040 plan A is our recommendation for

how things should go. So, we called it

AI 2040 because in this scenario, uh

they build superintelligence in 2040

instead of much sooner because they

delay things.

>> Why do they delay things?

>> To manage the risks and make sure that

power is distributed equitably.

They basically like

regulate AI development so that it still

continues, but at a slower, more

reasonable pace uh in a more transparent

and safe way

and spread out over more countries and

companies. And as a result, they get to

superintelligence in 2040 instead of in

say 2030.

And then we call it plan A because

well, it's our recommendation. Like

we've we've come up with a plan for

what government should do. And uh

the scenario is an illustration of what

it might look like to implement that

plan. In a similar way to how AI 2027 is

kind of an an illustration of what it

would might look like

to do with the companies are currently

planning to do. If that makes sense.

>> And is this wishful thinking or is this

what you think is going to happen?

>> No, it's definitely not what we think is

going to happen.

>> It's not what you think is going to

happen?

>> No, no, what we think is going to happen

is still

something more like this, right? We we

don't expect the world to listen to us,

right? This is our recommendation, but

we we we hope that that people do

something like this and we think it's

possible, but it's not our like

prediction for what's going to happen by

default, you know.

>> So, I do want to run through the plans,

the potential plans, and also plan A,

but um just to close off on how things

might look after the year cuz I think I

wanted to touch on robotics, too, and

I've got this graph here which talks

about share of labor output.

>> Yes.

>> Yeah.

>> Um which I found to be quite striking.

I've been sat here wondering as an

employer who employs hundreds and

hundreds of people

when when all this stuff is going to

happen. And you know, we're still hiring

more people as things stand. There are

some roles where our consideration is

changing, shifting considerably.

And I'd have to say that, you know,

we're probably in the phase where our

teams are AI-powered and they're using

agents to do some of their work now.

But I'm wondering as an employer like

when is it

when does this happen?

>> Yeah, great question. So, if we could

maybe zoom in on this a little bit.

>> it on the screen.

>> So, this is in the AI 2040 plan A

scenario. And notably in that scenario,

there's significant regulation

introduced in 2029 that slows down the

pace of AI development.

In the scenario, they do that sort of at

the last moment. So, in the scenario, if

they hadn't done that, then it was about

to take off similar to how it does in

the AI 2027.

Um but as you can see like in the

scenario, there's still

a bunch of jobs

at the point that they implement it. And

this gets back to what I was saying

earlier is that if you wait until most

people have lost their jobs

to regulate the AI companies, that's

already too late because

they will probably already have super

intelligent AI by then because their

strategy is to first get super

intelligent AI and then do all that

stuff.

>> think you say that it would collapse the

economy and cause even more harm to

suddenly regulate something that all of

us and all of our lives were then at

that point relying on.

>> Oh, but it's a risk well worth taking. I

mean, we It's true that right now a lot

of people use AI for a lot of things,

but like if we could somehow slow or

halt AI development now to set up a

better way to do it, that would be well

worth it. Um even though there would be

significant costs.

>> But you can't over here, right? Can you?

At this point where AI and robotics are

doing most of the labor output.

>> That's right. But in but in but in in

this scenario, in the AI 2040 Plan A

scenario, they put in the regulations in

2029.

And then they slowly and carefully

develop AI

in a way that avoids all the problems,

which we can get into in a little bit.

And so eventually, yes, eventually the

AIs take the jobs. Eventually

basically the whole economy is run by

AIs and robots, but it it happens

gradually over the course of

the 2030s instead of happening in this

sort of crazy shock,

you know, a year later.

Right? Because in this scenario, they

don't let the companies

recursively self-improve and get to

super intelligence as fast as possible.

Instead, they regulate AI development so

that the core capabilities of the AIs

are improving at a more reasonable pace

and also in a more transparent way so

that the scientific community can see

what's going on and help make it safe.

>> But it's

I guess I noticed here that in both your

scenarios, eventually AI and robotics do

pretty much all the jobs.

>> Yes.

>> So you kind of side there with Elon when

Elon says that working will be a choice.

>> Uh

>> Because I mean we're going to have to

>> I mean, if [laughter] it by definition

if it can do all the things, then

it can do all the things. I think that

there's a question of like should we

allow there to be AIs that can do all

the things, right? Some people think

that the answer is no and we should just

shut it all down and prevent these types

of AIs from being created in the first

place. And we're actually kind of

sympathetic to that. We we have our

Should we bring out the plans diagram?

>> Yeah.

>> Thanks. Yeah. So,

our scenario is called AI 2040 plan A.

It's a scenario in which they slow down

AI development to make a super

intelligence happen in 2040 instead of

earlier. And plan A is our

recommendation. So, this is sort of

illustrating our recommendation. But,

for comparison, we made like mini

scenarios illustrating different

alternative plans, which we call plan S,

plan B, plan C, and plan D.

Plan D is basically

the same thing that happens in AI 2027.

Like, the race continues. There's very

little regulation.

Um you can read about that in AI 2027.

Plan C also very similar to what happens

in the slow down ending of AI 2027 where

they solve the alignment problems. So,

in that ending,

they like slow down a little bit,

pivot more resources to AI alignment and

AI safety research,

get lucky and succeed, and now they have

aligned AIs,

and then they speed up again and take

all the jobs and beat China and all

those things.

Plan B is

it's kind of like plan C in that

well,

basically in plan B, you're

uh being more aggressive towards China

and you're like

taking actions to sabotage or cyber

attack them to like keep them behind so

that you have more breathing room to to

solve the alignment problems yourself.

Plan A is our recommendation. It's uh

domestic regulation and then an

international deal

to continue building AI, but in a much

better way.

Plan S is shut it all down.

If you want to have a future where

there aren't AIs running around that can

do everything better and faster than

humans, you kind of want something like

plan S. What What do you want?

Plan A is our recommendation.

I think that I'm sympathetic to plan S,

but for reasons we explained, we

recommend plan A instead.

>> And And do you think is most probable?

If you're being honest?

>> Plan D.

>> Which is that they just

>> yeah, 24/7 type of thing where they keep

racing. They don't really slow down

significantly.

Um

and uh

things happen extremely fast.

The diagram sort of explains like

roughly the reasoning behind this, too.

So, like there's this high-level thing

of like

do you want to keep racing

as fast as possible to make the AI

smarter and smarter, to put them in

charge of more things so that we can

beat China?

You know,

if you're happy with that, then

you get down and it says variation of

happens here.

If you are worried about that, well

you get to something like this.

There's more different options besides

these, but this is kind of like the ones

that we could compress onto a screen.

>> Do you have children?

>> Yeah, I have two children.

It's kind of sad.

Like

I think that one way or another this

will probably all be over by the time

they're old enough to

join the workforce.

So, I don't think they'll ever join the

workforce.

>> When you say this will be all over by

the time they join the What do you mean

by this will be all over?

>> So, these milestones that I described,

like AIs automating the AI research, AIs

getting super intelligent. Um

AIs then exploding onto the economy,

taking the jobs, building robot

factories to build more robots to build

more factories,

etc. GDP starting to

go vertical.

That sort of thing is what I mean. Like

all of those events transpiring.

Maybe there's like you know, 10, 20%

chance or something that

hits the wall

and and none of this comes to pass even

if you don't do anything.

>> How old is your oldest?

>> Six.

>> Six.

Boy or girl?

>> Girl.

>> Girl. So, your daughter comes to you and

says, "Dad, what should I um what should

I study in school?"

>> I mean, again, like if these radical

transformations happen, then

the world will just look completely

different and

what sort of jobs you set yourself up

for basically, won't matter that much,

probably. I would say um that the thing

to do is

well, A, try to make it actually go

well. Like, if you can exert any

influence at all on history and how this

all develops, you should be trying very

hard to steer the future in better

directions.

And then separately from that, on a

personal level, you should focus on

well,

being a good person and doing things

that are sort of good in their for their

own sake, rather than good because

they'll set you up for later employment

because that later employment is going

to be very uncertain um basically.

>> Elon talks about this age of abundance

we're heading towards.

Age of abundance

>> There'll definitely be abundance.

The question is who controls the

abundance?

And what do they do with it?

Right? Are the AIs controlled by anyone?

Or are they doing their own thing?

And then if they are controlled by

people, who controls them? And what do

they do? And what's the sort of like

political structure governing how they

make those decisions?

>> I think it was Geoffrey Hinton that said

to me, he said there's no example in

nature where a more intelligent species

is has less control than a less

intelligent species. Thus saying that

we're quite arrogant to think that in a

world where there's this artificial

brain that's a gazillion times the size

of mine, that I'm going to give it

orders.

>> Yeah. I mean, that that's the thing is I

I think it's like

that should be our default assumption.

Is that like, well, there's these

brains, we can't see exactly what

they're thinking. We're going to make

them smarter than us and put them in

charge of everything.

>> And then we're going to give them

bodies.

>> Yeah. And then they're going to be

autonomously building new factories and

so forth. And like, how is this supposed

to end well again? Like, isn't this just

exactly like us picking a new species

that's then going to outcompete us when

it doesn't need us anymore? Like, I

think that is just the default

trajectory. Now, there's a whole

argument we can get into about like ways

that we could get off of that default

trajectory. So, for example, there's

research into interpretability that I

described previously. And if that

research bears fruit, then you will be

able to actually see what they're

thinking. And then that would be an

excellent tool for shaping them and

controlling them and making sure that

they do what we want, right? There's

other sorts of um

AI alignment research agendas that are

making progress. And if enough of those

agendas succeed sufficiently, we can

avoid this problem. Of course, also

there's the regulatory side, too, where

like part of what makes this difficult

is that we're building these AIs in race

conditions, you know? Like the the

companies are secretive about their

recipes for making these AIs because

it's secrets that they want to protect

so that other people can't copy them.

And so a lot of this is happening, you

know, behind closed doors. Only a few

people can really see

the recipes that they're using to train

these AIs and and so forth. And then

oftentimes when the AIs

behave in unexpected ways or even just

like blatantly misaligned ways,

sometimes that information doesn't

really flow out to the public because

the companies are not really

incentivized to tell everyone about how

they messed up and how their AI is evil.

It's just not very conducive to

scientific progress on these issues. If

the regulatory system was different,

then perhaps we could be in a better

situation, make faster progress. Also,

of course, we wouldn't be planning to

put these AIs in charge of everything as

fast as possible. And we wouldn't be

planning to like let them self-improve,

you know? Like the these are choices

that we could not make, you know?

>> I don't speak Vietnamese, but this show

can because of AI video technology from

our sponsor, HeyGen. I get messages

every single week from those of you

listening to the Current CEO all around

the world, and you express how much

impact it's had on you and your life.

And if that's true, then those

conversations shouldn't only reach

people in English. HeyGen can take one

recording of me and deliver in any

language while keeping my voice, timing,

and expressions intact. But you don't

need a studio like this to make it work

for you. Record 15 seconds of yourself

and get an AI avatar that delivers

studio-quality video in over 175

languages. We're up to 20 languages now,

and we're not the only ones using it.

HeyGen is already used by 30 million

people, including 85% of the Fortune

100. Whether you're building an audience

on social media, launching an online

course, or rolling out training across

your team, check out HeyGen now. Your

first three videos are totally free at

heygen.com/doac.

That's h e y g e n dot com slash d o a

c. See you there.

>> Ilya was As you said, he was one of the

leaders at OpenAI, and he left and he

started his own company now, Safe

Superintelligence.

Very curious name of a company, Safe

Superintelligence, after leaving OpenAI.

Did you ever get to work with him?

>> Uh I wasn't directly working with him. I

had a couple chats with him.

>> Do you think he's he's genuinely

concerned as well?

>> I think he is, but I think it's I think

he's similar to these other CEOs, where

I mean, just think about the sort of

incentives that they're under, right?

Like

they can sort of see the problem,

and then they can

be like, okay, but like if I don't if I

stop, if I quit my job, and or do

something else,

that's not going to solve the problem,

cuz the other CEOs are going to keep

going.

And even if all of us didn't go, then

maybe China would keep going. So, like,

man, it seems like this is just going to

happen one way or another, whether I do

anything about it or not.

I guess I should be involved, you know,

and like maybe I can make it go well,

and at any rate, like I don't want to be

out in the cold while these other people

I don't trust are in charge of

everything. So, they all sort of like

reason through all of this and then

convince themselves that like the thing

to do is for them

>> to build their AI.

>> build it and to do it better. And I

think Ilya's just the latest example of

this. Elon's another example. Dario's

another example.

You know, arguably OpenAI at the

beginning, Sam was an example, although

like Elon and Dario were at OpenAI early

on, so

>> What do you think they should all do

then?

>> So, I think what should happen is some

sort of international regulation, or at

least domestic regulation, similar to

what we described in plan A.

>> Okay, so talk me through plan A.

>> Yeah.

So, in this scenario

AI takes longer to the to get to

recursive self-improvement and full

automation of AI research than it does

in 2027. We figured that we should try

to illustrate like a range of different

possibilities because we do have those

sort of uncertainty intervals. So, we

chose 2030 as the

moment when full automation would

finally be achieved and things would

really kick off.

And then working backwards from that

when's the last moment you could really

have good regulation? 2029. So, in this

scenario

AI progress slows down a little bit

naturally and the AI companies keep keep

racing, but they don't quite succeed in

automating uh themselves in 2027 or in

2028 or in 2029, but they're getting

really close and they're going to do it

in 2030.

And then in 2029, the government steps

in and regulates them. What regulations

do they do? Well, they basically just

shut it down temporarily.

>> Can I ask um

how does the elections overlay with your

time frames here? Because there's going

to be a big election, isn't there in

2028?

And it seems now that sentiment has

really really turned against AI in in

sort of in the general public and that

it will be one of the big ticket items

on the on the ballot.

>> We think that it'll be maybe the most

important issue in the presidential

election in 2028. Um I think a lot of

people most people will be quite

concerned about where things are headed

and that's part of why we we chose

to depict things the way they were doing

in this scenario because that helps

explain why they might do this sort of

regulation in 2029 is that the voters

have been demanding it and the

presidential candidates have been

promising it.

>> And in this scenario and then 2027,

would the general public have felt the

consequences of AI much more severely

than they have now by then?

>> Yes.

Although still even in 2029 in this

scenario, they still mostly have the

jobs as as depicted here, right? So, in

in 2029 in this scenario, lots of jobs

now involve managing AI agents.

You you mentioned you have an AI agent,

right? Well, in 2029 in this scenario,

the AI agents will be much better. Still

though, not enough to just completely do

everything. You know, that was the sort

of thing that would come in 2030

in this in this timeline. Again, we're

uncertain about timelines.

Things could go faster than depicted in

this scenario, and in fact, I think

things probably will go a bit faster

than depicted in this scenario, but

we're uncertain. We already did the very

fast timeline scenario, so now we're

doing the slower timeline scenario. But,

maybe we should talk about the

high-level goals. So,

they want to have AI continue, but in a

slower pace so that they can make it

safe.

>> The politicians, you know, the president

and the people who voted for the

president and, you know, the heads of

other governments and so forth. So, goal

one, slow things down.