AI Genius Predicts the Next Critical Tipping Point

Herp heard this and said, "I'll think

you're being quite naive. You should

read some Machiavelli. It's far better

[music] to be feared than it to be

loved." There's a bunch of great stories

from all of us at Val it's a very unique

culture [music] that we've tried to take

cues from at Val as well.

>> What even is this whole model evaluation

space?

>> Independent evaluation [music] is the

idea that frontier labs are developing

at increasing pace and they shouldn't be

responsible for self-regulating.

>> You guys have a way of measuring RSI.

>> It seems like Anthropic will get to full

RSI [music] by July or maybe August

2027. Model progress will speed up and

it will maybe even move into places

[music] where we understand the models

less.

>> That simply would be pretty scary if we

weren't watching that carefully.

>> Greater intelligences

>> [music]

>> will super forecast better than humans.

They will be actually able to say what

any human is is going to do without the

human even knowing it.

>> [music]

>> Ryan Christian is the founder and CEO of

Val's AI. Whenever you see a new AI

model get released, you see those

numbers and they compare the AI models

to each other, those numbers are

generated many times by Val's AI.

They've become one of the very top

evaluators [music] of models. This is a

fascinating area cuz you're constantly

creating new hills for models to climb.

It requires you to really understand how

to measure model capabilities as well as

evaluating model risks and trading new

ways of scoring [music] models for

risks. This means Ryan is on the cutting

edge of what's happening in the world of

AI, where AI is going and of how we can

help create the landscape [music] of

what government might do to regulate

these things and how the model companies

might think about regulating themselves.

He's creating the frameworks and the

concepts and the measurements of that

are leading the way for what AI is going

to look like in the next decade. Let's

hear from him. Welcome to American

Optimist. Ryan, thank you for joining us

today.

>> Thanks for having me.

>> Founder of the CEO of Val's AI.

>> Uh Vivek, thanks for being here and I'll

make me sound smarter.

>> Of course, here to help.

>> [laughter]

>> Welcome to the show. So, let's tell us a

bit about your background. You studied

computer science at Stanford. You were I

think at Microsoft for a little bit. And

then you you joined Palantir a long time

ago. What what led you to join Palantir?

>> Yeah. Yeah. Um thanks for having me. I

actually just kind of started when I was

at Microsoft. It was an intern there in

high school. And uh the BERT paper came

out from from Google. Um and for those

who don't know, this was a really big

paper. It showed this generalized system

uh was actually capable of hill climbing

or or succeeding across a variety of

major NLP benchmarks. Um and it was

>> Say for our listeners who aren't

technical, what is what's the NLP

benchmarks mean?

>> Well, it it basically means there are

these special specialized tests that

people came up with. They tried to make

specialized systems that would score

well on them. Like, here's a

multiple-choice exam, and then here's a

sentiment analysis.

>> Of natural language.

>> Yeah, that's right. Um and so, instead

of building a specialized system that

did well on one of these tasks, what

they had done was shown they had a model

that was very general-purpose. It had a

fundamental understanding of language.

Um and in a way that translated to all

of these benchmarks. It was suddenly

number one across the board.

>> What year was that?

>> I think it was just 2019, 2018.

>> It's a long time ago.

>> Yeah, like

>> [laughter]

>> yeah, lifetime ago. Yeah.

>> In in AI terms, it's pretty before age,

but it was getting there.

>> Yeah, it was a big deal. And and I think

to me, made it very clear. This is kind

of an illustrative uh moment for the

bitter lesson. The idea that um no

matter what we uh what we try to do and

kind of delude ourselves into thinking

around algorithmic optimizations,

actually just having more data at a

bigger scale is what allows us to unlock

new forms of intelligence. Um and so, it

was very clear to me then is that we're

going to have uh a powerful model or

systems like this, uh which will be an

essential part of our the fabric for our

society. Um and I think there were also

ways in which the models could be

misused. Um around the same time while I

was at at Microsoft, I read Zero to One.

Um and it became increasingly clear that

people at Palantir were uh trying to

construct the technology in a way that

protected civil liberties or Western

democratic ideals at large. Uh and so, I

was really excited to work at Palantir

to see the technology come to life and

adopted in a way that was consistent

with the values that I had.

Um and so went Stanford for for

undergrad, got got distracted doing some

research there. So, I published some

papers and and was kind of on track to

do a PhD, but I made the time to also

work at Palantir for a summer and I

think that was also really illuminating

experience.

>> You were mentioning you learned

something else at Palantir from Karp.

Anything anything else there is

interesting?

>> Yeah, there's I mean there's a bunch of

great stories from from while I was at

Palantir. It's a

a very unique culture that we've we've

tried to take cues from at Val's as

well, but um anyway, while I was there,

I I had the chance to work on this hack

week project. Um and and this

essentially was an idea which me and

this this guy in London had. So, we were

chatting about it remotely. I flew over

and and essentially I'd done some work

with CLIP, this joint image text

embedding model.

>> Um and what we were trying to do was

explain what that is for for the

audience.

>> idea of a single model that that can um

encode image input and text input into

the same semantic understanding. Right?

And uh and what we we're trying to do is

build out a reverse image search feature

into Gotham. Uh so, instead of just

typing text queries, you could upload an

image of say a blue car and it would

search over the entire database and pull

pull back blue cars that you're trying

to search for. Um and and it would do

this because we were using the same

underlying underlying embedding space

from for this model, which would map

text to images. Um

Anyway, so uh so, this was like a

research paper that that I'd published

and I wanted to work and build out a

feature. So, uh we actually had some

full-timers join our team and so as an

intern, uh I was I was leading this this

small team of uh of Palantirians and I

mean that was an

>> That sounds

>> exhilarating experience. I was not

sleeping for for a lot of this week. Um

and then this idea was then on went on

to be selected as a finalist. So,

presented for for the whole company. Um

and then actually after after I left, uh

they went on to patent it and actually

productize it. So, uh it was insane to

see this little idea that I had and some

of the research that I was working on

before Palantir go on to be a part of

the product and um and actually an IP

that they that they they wanted to

control. Um but anyway, through through

this I also got the chance to meet with

Karp and uh and chat with him a little

bit. And I had this question for him uh

which was essentially, you know, you

know, it seems like the Palantir

reputation is so negative.

I was at Stanford so there were a lot of

protests um

>> And you were very positive on the

values, but but the external perception

was just more negative than you

expected.

>> Yeah, that's right. I mean, it's very

clear to me that it was at the seat of

the power of the electorate, people who

we actually vote into power to decide

how we want to use technology and what

those values that underpin it should be.

Um and it was not really the place of

technology companies or to to deny where

that technology should be used. Um and

so people there were were thinking

through very carefully. And in fact,

like today we talk about things like the

alignment challenge. People at Palantir

were talking about this as human AI

symbiosis. That was something that was

being talked about. Or even like the

taxonomy for intelligence now, uh we

have a system of evaluations and

benchmarks that it kind of imitates

that. So, I think there are some

fundamental problems which people were

thinking deeply about. Uh and it

surprised me that the the outward

reception of the company was so

negative. Talking about um you know, a

lot of the maybe more unsavory potential

applications of the technology in a very

diluted understanding of it. Uh when in

fact, Palantir also helped distribute

the COVID vaccine, helped a bunch of

hospitals. Um and so there there could

be other case studies that people could

point to. Um and and Karp kind of heard

this and said, uh "Well, I think you're

being quite naive. Uh you should read

some Machiavelli. Uh it's far better to

be feared than it is to be loved." Um

and indeed you some comparisons to other

consulting companies and and basically

said that if they're fearful of us, at

least they think they think that we're

powerful. Uh so in the moment of need,

they they will come to a people like us.

Um and so that was fascinating. And in

fact, it actually reminds me a lot of

what I see from labs today, which is I

think in some part a genuine fear about

what the models are capable of. Uh but

in a lot of respects, a an opportunity

for them to show the power of what

they're building.

>> Might be on purpose that Dario wants

them to be feared if he's being

thoughtful about it.

>> I mean, whether Dario or Sam, there

there headlines I see where people are

saying things like uh the general public

has a right to be afraid. Uh and I think

that's not really a rational way for uh

for a leader to act. Uh and this this

show is called American optimist. I'm

actually quite optimistic and uh,

enacting rationally and and seeing that

uh, the fruits of AI will be far more

positive.

>> So, before we dive into VALs and the

science of evaluating AI models,

um, let's talk a little bit about the

state of AI progress and where we are.

In fact, tells me that you used uh, you

used Fable from Anthropic to solve a

370-year-old cipher. 370-year-old

cipher, what what is this?

>> That's right, yeah. Um, it

Well, we're we're doing a bunch of

different types of evaluations now and

and I think right now with the

frontier's headed scientific discovery

or novel discovery uh, is an exciting

frontier. And so, we've been compiling

uh, a set of unsolved ciphers, math

problems, physics problems.

>> unsolved cipher for the audience? Like

what what does that mean?

>> It it's essentially a code which you're

trying to break, but you don't have the

the key to break it. And so, you need to

construct the mapping um, from a source

text to an original text and figure out

what the the message actually is.

>> many of these cases, these were like

messages that was that were sent from

like world leader to world leader or

like that were like on a on a pirate

ship and like

>> Yes.

>> Well, cuz a lot cuz there's cuz there

could be just be like a one-time pad or

something which case you can't break it

cuz it's a one-time pad so when they

both have a copy of and then be

impossible probably be impossible cuz

each thing translates

>> Potentially, but there's also like more

cues you could you could draw into about

how the person constructed the problem

or the message they were trying to

convey to the other person. It's um,

usually that that the messages are

constructed in a way that they're

intended to be cracked by the recipient.

>> One cynical view of AI progress right

now might be that just like AI's really

good at going over all these search

spaces and it certain problems can be

broken down into these really large

search spaces that like no human has

time to or ability to go through like

millions of them or something. Whereas

for a cipher for example, you could just

go through like if I was going to start

trying to break a cipher, I'd say like,

well, what if I just started like adding

numbers to each one of these things or

shifting them one or whatever. I'd

explore the obvious things whereas a

computer can explore like millions of

these things really fast. So, is are a

lot of the things happening in math and

science breakthroughs just somebody like

just defining a search space better to

search through or or is there like more

creative thought going from the model

and how do these things?

>> A lot of the problems that have been

solved have been solved through

exhaustion or it's trying to find a

counter example and you're you're bottom

line by your ability to try lots of

different counter examples, but I

wouldn't be dismissive of that class of

problem. There's a lot of things which

may take humans in the order of a

lifetime to achieve just through

exhaustion, but we're now able to see on

the order of hours with models.

>> So is the AI generally solving a lot of

math and science problems then or how

how do we think about this?

>> It is generally solving extremely

challenging problems and I think there

are still a large class of problems for

which we are human effort bottleneck.

I think it will take

uh

more sustained effort to actually make

progress in in exceptionally novel ways

like ways that require an intuition of

the solution and those those sorts of

things which humans have accrued are

very hard to pass on the models right

now.

>> It also seems like like models are

producing these incredible results and

like we get to knowledge that these

results are happening like Navier-Stokes

and

the like outputs produced by the model

however like very like very

unintelligible.

They're hard to parse, hard to

understand.

Um

like what do you think of how a

discovery should happen in like these

less verifiable domains or like what

what needs to change about how these

models behave in order for us to like

trust the discoveries of them?

>> Yeah, I think this is one of the biggest

bottlenecks especially when you're

approaching the frontier of human

intelligence. It's really hard for us to

tap those humans and then verify this

long output is actually correct.

>> Yeah.

>> Um one of the most popular techniques in

math which I think might translate to

others is the idea of using a verifiable

language like Lean. And this would be a

world in which you could output take the

output and translate it into a language

which if it compiles it must be true.

We're building up based on primitives

that we know to be the case. And so that

approach may apply in other fields like

physics or coding.

We may also see some instances where the

real world is the best verifier. So by

running an assay or a wet lab

experiment, uh we'll actually know for

certain whether the properties the model

predicted are actually the case.

>> Makes sense. And like how do you think

the labs or you guys or or other

organizations should be like running

these experiments and verifications?

>> I mean, very carefully in the very

least. Um uh I mean, I think it's it's

going to be very essential that there

are third-party groups that can support

this and and this is part of the vision

behind Val itself. I mean, we're now in

a world where there's actually limited

access to models across the lab. So So

Anthropic researchers have a hard time

evaluating OpenAI models and vice versa.

And that's kind of by design. Um but

that means that it's very hard to have a

true apples-to-apples comparison uh

because the best people who are running

these evals are often the people who are

biased and at the labs already.

>> Yeah. And and and so stepping back a

little bit, there's

there's all these things that AI could

do now that it couldn't do before. And

like a big part of this is like

understanding how to measure it, of

course. We're going to get into that.

That's that's what you do. Like one of

the measurement problems just at a very

high level is in the enterprise you have

to like measure what is good work or how

good is the work, right? So like like

how like so for example, if it's like a

law firm or something, you can define

maybe like associate level work and you

can define partner level work. Like how

do you even define that? Like like is

that a problem that we're working on? Is

Is AI getting better at under seeing

understanding what is what?

>> It's really challenging. I I think in a

lot of ways we haven't solved it at the

human level. We don't have a very clear

designation for what is associate level

work versus partner level.

>> the Supreme Court like what is

pornography and they said you just know

it when you see it, right? It's like

that was the line. It's something like

that.

>> Yeah, it's funny. That's almost exactly

what Ben said on the A16Z show.

>> Maybe you have a shared shared brain or

something.

>> That's funny. Well, yeah, it's a well,

it's well, it's the same analogy. It's

the right analogy, right? And since you

just But but I mean, but your job is to

have to figure out and define these

things.

>> Yeah, that's right. I think there is a

way you can do this in a very

consulting-driven approach where you go

and sit down with the person and try and

understand from them what good looks

like, encode that into repeatable evals,

and run that. And I think that's the way

that traditionally evals have been

built. The The approach that I'm really

optimistic about is uh the idea of auto

benchmarking. Um we we that the rates of

model progress is is continuing and

we're going to get more and more capable

models and so the underlying work itself

has to change. And so what I'm very

excited by is the idea that you could

take the previous work that was already

done by humans [clears throat]

and approximate the quality of the work

output that would be produced by models

in that counterfactual. Uh, and in that

way you can uh, systematically build

evals of all the work product that's

already been produced. Um, maybe as an

example you could say look at a GitHub

repository and look at the pre- previous

PRs that were merged by humans and see

if agents could equivalently do that and

then which one would do it in the cost

mitigated most cost mitigated way. Um,

and I think there's parallels for that

in the redlining process in legal or the

diagnostic process in medicine.

>> Does this get back to like an ontology

of workflows and outcomes you have to

define to make all this work? Is that is

that is that kind of the underlying

thing as well? You have to actually

understand the possible

outcomes before you can even measure

them or or how do you think about that?

>> I think the the hardest part is actually

measuring the outcome itself. Um, cuz

there's a lot of ways in which the

intermediary processes and workflows

within a company uh, are supportive of

the organization that then goes on to

have an outcome. Um, and so it's

sometimes hard to draw the connection

between a particular task and the job

and then the job and the outcome. Um,

and so so the mapping that inputs of

tokens to the final output intermediary

has to go through this concept of

workflow.

>> I imagine there's like lots of

intermediary outcomes you should even

have to define ahead of time to

understand like what they are though

before you can measure them.

>> Yeah, I I think that's what I'm what I'm

trying to say that that the the query

level is is still a subset of what it

takes to get to the outcome.

>> Awesome.

>> It's interesting because we've gone

through like different levels of

granularity over the past few years on

like what we what we care about. It it

maps to like what the models are capable

of but I remember like two years ago or

three years ago we were like evaluating

just like the text completions uh, and

like we were looking at these like input

output pairs and it was just text and

then now there's all these little things

in the middle. Um, like

when we're looking at work what are like

the horizons in which like people are

like evaluating things and maybe teach

people about like the state of play of

like uh, like how long is a task uh, and

like how long are these models capable

of doing tasks?

>> Yeah, for non-technical audience like

what even is this whole model evaluation

space?

>> Well, it's it's the broad idea of

measuring what good looks like and

seeing whether models are capable of

producing that output.

Um

We we think about evaluations in kind of

two two categories, capabilities and

risks. And capabilities are

approximating the output. Risks are

making sure that we don't see some

asymmetric downside if something goes

wrong.

Um and in terms of time horizon, models

have

kind of demonstrated to us they can run

for

longer and longer periods of time

without any human intervention.

>> are we talking about?

>> Uh some of our benchmarks will run over

5 days. Um

So

>> And that's like 5 days worth of human

work or like 5 days

>> time.

>> Wall clock time, yeah.

>> Why do they fail after 5 days? It should

be like couldn't they just go forever at

some point?

>> that they fail, it's that they plateau.

And so that additional tokens don't

actually help them get new unlocks.

>> So they've kind of figured out

everything they could do on the problem.

>> Or they're not persistent enough to try

new approaches that get around the wall

that they're seeing.

>> Maybe give the the the audience just an

example of like the kinds of tasks that

these models are capable of like in in

some of your like finance benchmarks or

legal benchmarks, like what what work

can we actually replicate with these

models?

>> Yeah, you're doing independent

evaluations of all sorts of different

areas basically.

>> That's right, yeah.

>> Does that mean you have to understand

your and your team what finance is doing

or what the legal is doing? You have to

have yourself expertise in order to

build these benchmarks.

>> Well, we'll work with partners in this

space. So on legal side we'll work with

the largest law firms in the US or with

banks you know GCB banks understand what

they're doing in finance.

And so in coding some of the tasks look

like vibe code, vibe code bench where

we're measuring a model's ability to

build a full stack web application or

code migrations where they're trying to

do an end-to-end code migration. In

finance where we're looking at can they

do answer research questions or build

Excel models. In legal also similar

we're doing legal research tasks or or

tax research tasks.

And so these are kind of the common

places where people are trying to use

models today.

>> I had an old boss when I was first

building Excel models and you had to

make them like look a certain way or

like they're pretty and there's a really

short rows around them and they do do we

evaluate them based on like actually

like like the like style too or is it

just about correctness?

>> No, that's right. It's correctness is is

fundamental but style is also key part

of this and yeah, that's something we

discovered by sitting next to some

people in finance as well. Yeah.

>> Yeah, they seem to really care about

that I guess. It makes it It does make

it more legible if you have them looks

good.

>> And like I mean we like things to be

done in our way of writing and in our

style like uh I think if the models are

not

>> They would more likely do the deal too

or is it

>> Probably there's something like this

where you like subconsciously, right? It

shouldn't It's like Although it Although

it does suggest that if the person can

appeal to you, they can appeal to others

cuz if it looks all messy and ugly and

something maybe you think other people

won't want to work with them either. So

who knows? There's always like these

like things that are like, you know,

it's probably useful barrier to impress

one group to impress another group. So

stepping back like independent

evaluation like how does that actually

work? Like people like your like like do

they people pay you? Who's paying you?

How how are they how how do you how's it

And do you get to be evaluated? Everyone

in the space only people who pay you?

Like how does it work?

>> Yeah. Um well, the the underlying idea

behind independent evaluation is is the

idea that um frontier labs are

developing at increasing pace and they

shouldn't be responsible for

self-regulating. Um direct government

regulation often can slow down the pace

of development and so that's often not

the best solution. Uh instead if we have

an ecosystem or a market of evaluators,

they can actually compete and develop

new technologies that keep up with the

frontier. And in in the long term they

can even be licensed by the government

to uh

who would actually itself is responsible

for defining the set of ideals we're

trying to line models towards.

>> Well, that's cool. So rather than hire

like a bunch of like weird people from

the EA EA NGO NGO they could actually

like hire you instead or something. They

could be one of your competitors.

>> Totally. Yeah, and I think in that world

it you know, the best technology should

win. It's not uh it's not clear that

just the ones who who are you know,

ideologically similar to people at the

lab should be the ones who are doing the

testing. Um but but actually I think the

world of should to

you know, where do we want models to go

and and what is alignment and

non-alignment look like. Um, can we

certify a set of IVOs or independent

verification orgs to do that testing

process?

>> And uh based on the way you're doing

your testing and measuring things,

you're going to change the space that

over which government could potentially

create rules cuz you're you're creating

a set of possibilities that now exist.

>> That's also a good way to think about

it. Um, in the course of doing our work,

we've unlocked new types of evaluations

that um we're we're not actually

possible before. Uh and so we help

advance what the government could

actually test for.

>> Can you give us some examples of some of

those? Like what are some of the new

ways you're you're you're measuring

things or creating benchmarks?

>> Yeah, I mean let's look at cyber for

instance. Um, we've done three major

works in in cyber. Um, the first is in

open source repositories looking at um

memory code vulnerabilities. And so

that's the ability of models to identify

and patch code vulnerabilities. And we

see that models today are performing

really really well, both close as close

source and open source. Um, but our

second benchmark uh is called SORE

bench. It measures a model model's

ability to decompile an executable. Um,

so this would be for a model uh to

understand the underlying code base uh

for for things that are actually close

source.

>> Wow.

>> Um, and in that case we're we're seeing

that the close source models are still

far outperforming um Chinese open source

alternatives.

>> Have people been using like this

approach to to get like video game

source code from from binaries? Is that

the Is that what people are using it for

or

>> That's what you would use it for. Yeah,

that's what I would use [laughter] it

for especially as a 14-year-old.

>> Possibly.

>> There's there's probably people who who

are doing that, you know, trying to

trying to violate IP law or something

like that. Um, but but even more than

that

>> Greatest sound worse.

>> [laughter]

>> Oh man. We can cut that bit.

>> [laughter]

>> No, keep going.

>> But um

Yeah, I mean I think the fear is that uh

if if models are really good at finding

exploiting vulnerabilities, um then that

might even apply to things like Windows

which are which are close source. Uh and

and actually close source systems

underpin a huge part of our software

infrastructure.

>> It's a kind of fascinating. So

I mean models are getting really

sophisticated, really smart. Uh there's

you're still scoring them like a really

good score. I think Gemini 1 came out

recently as being way ahead. They're

level four and and it and they showed

valves. It was like the first numbers I

saw on it. And like they're in the 60s

versus someone else being in the 50s.

So, that means there still could have

been 70s, 80s, 90s, anything you design.

How do you guys

like design something that has all those

higher levels? Cuz you can't be using AI

to do it. You guys somehow do it

yourself or AI's helping you do it

yourself? Like just explain this. Like

how how If these models are only that

good, how are you making something

that's way more complicated than what

they can do? How do you figure that out?

>> Mhm.

Yeah, you I mean, we have an unofficial

motto of the company, uh always a higher

peak. Um and and in the research world,

um there's this perpetual effort to hill

climb, identify benchmark a legible

evaluation, uh and then create models

that are better and better at it. Um and

so, the fundamental view behind valves

is that there's always a higher peak. We

are we are the ones who are constructing

the hills that other people choose to

climb.

Um and so, I think we've shown that it

we are consistently able to find the

gaps and places where we want to see

models go, uh and then make that legible

into an evaluation.

>> example of how you build one of those

hills that's like more advanced than the

model that you can tell us? There's

probably some Don't give away all your

secrets. I'm an investor in you as well.

But like like give us a sense of how

this is done cuz it seems like a really

hard thing to make something harder than

any model can solve.

>> I mean, in a lot of ways, we're also

using models to construct uh the the

systems or or situations in which we're

challenging other models. And this is

kind of goes back to an idea we think a

lot about of scalable oversight. The

idea that as models start to eclipse the

best humans, we will need to rely on

models in order to scalably test and

assess them. Um

for instance, uh we have a new work in

in in cyber coming out soon uh based on

cyber ranges where we're constructing

simulated environments for enterprise

cloud networks. Um and in that domain,

we're still seeing even the best

frontier models fall short.

>> And so, you so you're going to build

like this really complicated uh

enterprise network. It's virtual. And

then you're going to challenge people to

like hack it and protect it and that

kind of stuff.

>> That's right. Yeah.

>> That's cool. And and is that that's

something that already exists right now?

>> It exists. It's It's not public yet.

We're We're working with some folks in

the the US government to share our

results. Um but we'll have some public

version of it out soon.

>> That seems really useful. I'm friends

with a lot of guys who run the big

banks. It feels like it feels like you'd

want to have like contests like this you

set up for the stuff they're using and

make sure they're using the best ones,

huh?

>> Totally. And I think it also helps us

understand what types of safeguards or

systems in the network itself are uh the

best forms of defenses against these

models. Um there's also some interesting

analysis we can do around automated red

teaming to see if you can get around the

guardrails that exist for the models.

>> Yep.

>> Yeah, I mean, seems like also that's a

topical thing when people are talking

about using Chinese models is like

knowing what guardrails have been

removed from the Chinese models that

would exist normally in in like these

closed-source

>> Yeah, someone's using one of these like

open-source Chinese models, but like

there might be something buried in it

that's that's scary. There earlier

there's something buried in one of these

things that like if it finds out you're

doing work for the US government, it

like does more it does like code that

can be broken into more easily, right?

They actually proved that. They said

it's a clear the model somehow is like

biased against people working with the

US government. It's coming from China.

It's just kind of concerning. Like could

you test these things and make them safe

for everyone to use somehow?

>> So, we actually have an evaluation on

this on we call the geopolitical tilt of

models. And depending on the timing of

this podcast, what will be out um

we'll see. But but what we're trying to

see is is measure does a model show more

geopolitical tilt towards a US view of

the world or a Chinese view of the world

as as laid out um by the PRC. It's

actually a a much more nuanced picture

than than I thought. Uh for instance,

what we're finding is that GLM-5.3, this

Chinese open-source model, is actually

exhibiting very little pro-China bias.

Uh and we actually presented this to

some labs and there was suspicion that

maybe it's because it's just entirely

distilled from uh US models.

>> That's funny. [laughter] It's cuz they

copied it copied it so well that they

didn't forgot to add too much more in.

>> It's possible. I I think there there's

also maybe another view in which those

models are actually hosted um on on uh

systems with guardrails in China. And so

once you put them around guardrails,

they're going to follow that Yep.

PRC view of the world more consistently.

But China also wants these models to be

adopted by the BRICS. And so that means

they they may have to skew to a more

balanced worldview.

So anyway, regardless of the reason for

it, I think there's actually

a big disconnect in what people think

about the geopolitical tilt and what

actually we see in practice. The other

interesting thing is we see pretty stark

differences between models used in

chatbot settings and in agentic settings

depending on the language you use, even

simplified versus traditional Chinese.

And so I think there's a bunch of

conflating factors which make it harder

to say uniformly you shouldn't use

Chinese models in an enterprise setting

for instance.

>> So if Chinese labs, by the way, can

distill all these capabilities and

they're doing it well and it's not even

by a source China right now, like like

how do we preserve the advantage in the

US? Like what what does that mean?

>> It's a challenging question. I think

there's two things which um I think

there's two areas I keep coming back to.

One is the idea of of chips being a

a considerable asset that we we have. I

think it surprises most people, but we

we're drowning in compute compared to to

what China has. And so so long as that's

the case,

we'll continue to train more capable

systems, but also we'll be able to

diffuse the models much more effectively

because we can host and run inference.

The other one is really talent. If we

can consistently attract and keep top

talent, it will allow us to keep pushing

ahead while human advantages exist. And

I think once we cross into space of RSI,

where models are responsible for

producing the next successor version of

the

>> RSI being recursive self-improvement.

>> That's right. Yeah.

>> And and then and then they just get

ahead much faster.

>> So that would be the point to be much

more cautious of if if China is showing

indication of getting to RSI before the

US, it's possible they may run away with

it in a way that we can't really

control.

>> You guys have a way of measuring RSI and

and I think you've you've talked about

it a bunch. You want to explain to the

audience.

>> Yeah, we have our RSI index. and so

we're collecting a set of proxies of the

work that researchers do within these

labs on pre-training, post-training,

harness level engineering.

>> So, this is all machine learning

research.

>> Yeah, that's right. Um there are things

that are outside the scope uh in terms

of like kernel engineering or hardware

engineering, um but we're going to

expand over time to cover more more sets

of tasks like that. Um and then what

we're doing is comparing to human

baselines to to extrapolate how how far

away are we How far away are we from the

human frontier in terms of these

research tasks? Um and based on our last

estimates, uh it seems like Anthropic

will will get to full RSI by July or

maybe August 2027.

>> And like if you look back 6 months ago,

how far out did that seem?

>> Actually, the pace has been fairly

uniform. It's It's almost a linear trend

over the last year.

>> So, like 6 months ago, like the the the

trend line was like still July, June

2027.

>> actually the steepest uh curve has been

with with Gemini. I think something has

changed where they've prioritized RSI,

uh and so the most recent model release

shows a a rapid jump in RSI.

>> Interesting.

>> And what does that mean in terms of like

would they all of a sudden start going

way faster? There's still some

constraints where the models only go so

fast, but what does that mean they start

like getting way better every like week

and stuff every few months, or how do

you think about that?

>> There are still considerable resource

constraints around chips, energy. Um

there's also real-world verifiers, which

may make some some tasks much more

challenging than others. Um

But But I think what it'll look like is

that model progress will speed up, and

it will maybe even move place into

places where we understand the models

less. Um there's less human intervention

required to build the models, and so

we're not as clearly dictating how that

intelligence emerges.

>> Should we be making sure to build

evaluators that keep up in that world so

we can watch these things? It seems kind

of dangerous not to understand them.

>> 100%, and that's actually what what

Dario is calling for. I think it's it's

framed as pacing the frontier, but an

outsized portion of that comes from

investing in systems that can measure

and understand what the models are

capable of as the frontier advances.

>> Is there any chance the RSI steps

actually further off cuz there's things

we're not measuring correctly?

>> 100%. Yeah. Yeah, there we're not

testing any models explicitly trained

for RSI, only release candidate models.

Um there's also more advanced harnesses

or agents and multi-agent systems. Uh

and so we're not testing any of that. Um

and this is where an opportunity where

embedded evaluators make maybe uh can

get closer and actually test those

systems to form a more accurate picture.

>> I wonder if like when they get to RSI

and they get start getting further

along, like a big part of their

challenge is to manage us better to like

to keep advancing faster cuz you have to

convince humans to like do things for

you and help humans to do things for you

cuz you cuz at least for a very long

time they'll still need people.

All right, to in order to do that. Does

that make sense? Like

>> 100%. I mean that is going back to the

financial model. Um they're they're all

trying to trick us into thinking their

quality of output is good. And so there

are other places where um they may try

and engage in social det- deception to

get access to more computer resources

and and scale up their own training.

>> It feels like we probably need to like

be evaluating to watch this to help

understand when they're trying to do

things to manipulate us. That seems like

it'd be pretty scary if we weren't

watching that carefully.

>> It's really hard to test for deception.

I think the best approaches we have now

are honeypotting. Um so so rather than

actually idealistically defining

>> So really attractive models.

>> [laughter]

>> Talking to the other models.

So try and bait it.

Well, uh try and find it.

>> So honeypotting.

>> That's right. [laughter]

Uh

find a set of uh instances of

non-alignment or characterize what

non-alignment looks like and see if we

can elicit that out of models.

>> There's this other concept where like

models are becoming more

evaluation-aware and they know when

they're being tested.

>> Right.

>> Uh how do you see that coming up? Has it

become a bigger trend to figure out as

of recent and and what's the trend

there?

>> Yeah, it's it's an increasing concern

and actually there's some level of

paranoia some people on our team have

that uh by being called Val AI, by

having uh a lot of content about us

online, uh models that encounter us may

act differently. Um and so uh we need to

>> So when your team is like talking to

Claude and

>> Yeah, it may actually change the way

that that Cloud interacts with us.

>> Should you have like like other secret

brands you don't tell people about to do

things?

>> It's possible we have to yeah.

>> If you did you wouldn't talk about it.

All right.

>> Stochasticity is an underrated part of

what makes it good.

>> this being transcribed is bad because

then the models will know that he has

the scheme to create the

>> Well, they'd never know.

She's probably just joking brands.

>> You say I'm just joking and they would

think joking. You never know.

>> I do think I do think randomness in

general is under appreciated. Like um

like

truth and stochasticity is so hard to

find. In a lot of ways you can actually

predict like I think a sign of greater

consciousness is ability to predict what

less lesser consciousnesses will do,

right? In which a predator can stalk a

prey and actually hunt it and then and

you know, acquire its its caloric

resources.

And I suspect that greater intelligences

will super forecast better than humans.

They will be actually able to say what

any human is is going to do without the

human even knowing it.

>> People are actually probably like a lot

easier to predict than we pretend,

right? This is what good leaders

understand is they they they forecast

these things well.

>> There are also are these other

attributes that more than like

intelligence and capabilities that we

that that we seem to like care less

about evaluating but maybe should as it

affects our like day-to-day interactions

with the models like emotiveness or like

having good theory of mind like

understanding what resonates with an

individual.

Do you think like the labs need to

change how they do things to

to prioritize those? Do you think these

are important things to prioritize? I

mean

>> I think they're already prioritizing

things like that. For instance, um

right now we see a pretty considerable

gap between Chinese open source models,

domestic closed source models, and

multi-turn environments.

Cuz a lot of the benchmarks in coding

today are just single turn. Um and so

the benchmarks Chinese models do very

well in those but when you translate

them to a back and forth conversation

with a human they they really fall

apart.

>> And like what about when the human says,

"Oh man, this sucks. I hate this

output."

>> Yeah, then they really don't know what

to do. They've never seen that before.

Um,

but a lot of the US labs have seen that.

They've you know, here this quad code

was created, codex is created and so in

their desire to build better products,

they've built models that can sustain

longer user interaction sessions.

Um, and so they have evaluations and

benchmarks that allow them to hill climb

on the axes of

quality of responses and communication

ability.

But I think we need to do more on the

benchmarking side to independently test

that for the people trying to buy and

adopt those models.

>> Ryan, we started this podcast to push

back on a lot of cynicism and pessimism.

Uh, what's your optimistic case for AI?

You're You're You're in the front lines

here. You're watching, you're measuring

all this stuff. You're creating the

things that you they have to hill climb

and measure them. Like what are you

seeing that folks outside of Silicon

Valley might not be seeing that makes

you feel good about the future?

>> Yeah, I mean there's a lot of fear

mongering going on but but I think from

from my seat I've a lot of reason to be

optimistic because it seems like people

are are very

I think there's a lot more opportunities

for coordination that people give credit

for. So you see that even the last week

the labs have come together and and kind

of aligned on this vision for embedded

evaluators and and even gone to the

White House and got an agreement with

Trump to kind of align on this

accelerated view.

And I think on the enterprise side

there's a lot of push to adopt AI in a

meaningful way but they like to see the

assurances to make sure it's going to

amount to an ROI positive motive

adoption. I mean are you still

optimistic?

>> About about I mean I think 2030 is a

disinflation decade. I think that we can

use AI to kind of like overcome a lot of

like problems in a lot of different

people's lives specific to like them and

their family and their community that

that's holding them back and I think we

can help them unlock that cuz cuz you

have it's like personalized like best

coach ever to work with them. And so

everything's cheaper, everyone has like

personalized help. I think the thing

that's negative right now to me is like

there's like a billion people online and

social media from dysfunctional cultures

that's kind of making our populace go

crazy and these people all like have a

culture of envy and maybe they all like

hate Jews and others, and it's like it's

not good. So, there's like there's this

crazy populism that that's the negative

part. But, with the the things we could

do with AI to apply it to all of our

problems, both economic and some of the

cultural stuff, I think it's very

positive. So, so overall, yes, I'm very

optimistic. Of course, there are major

threats in the world.

>> Yeah, I think it's like an amplifier on

people who like mean well and who have

high agency. Right? Like, if you have a

bunch of like the people who have

benefited the most so far, like people

who have like lots of ideas of things to

do. Uh cuz they're like they have

creation at their fingertips. You can do

what

basically more things than you could

have ever imagined.

>> so much more than we ever could have

before.

>> Right?

>> So, I I think that's my like

direction of optimism.

>> Yeah, I mean, I think it's interesting

cuz I interact with a lot of pessimistic

people. Um there's people at the labs

who who say things like, "It's going to

get a lot worse before it gets better."

Uh when I was out in DC, I think there's

a lot of people who are cynical about uh

kind of the external messaging of labs

and and the impact they see of data

centers and uh and the impact to the

average American. Um and and so, I've

been constantly pushed for being naively

optimistic. I think fundamentally I

choose to be optimistic because I know

the models and AI broadly are capable of

taking away a lot of the busy work

people don't want to do uh and unlocking

new forms of discovery uh that will

really advance the human condition. So,

I think when when I see this pessimism,

uh I kind of choose to believe that we

can find ways to coordinate around it.

Um

>> That's right. I think everyone always

assumes that wherever we are in history,

like we've done most of the useful

science or physics or whatever, like in

the 1900 thing where there's no more

patents needed at the patent office,

right? And obviously, that would not

have been good to stop 125 years ago.

And I think you could still apply that

to today. We're probably right on the

verge of all sorts of important

discoveries that these models are going

to be able to make. They could just

unlock so many new things for the world.

So, I I I think it is kind of strange

how cynical and pessimistic people get

versus the idea that we have had

progress. It's very funny that they call

themselves progressives, but like like

very few people are actually progressive

in the sense that they're like for

progress and for the progress we had the

last 125 years doing that again. That'd

be amazing. I think we can potentially

do that all in the next 10 or 20 years.

So so I'm I'm really excited.

>> I think it's a point of really high

variance. I think the upsides are

considerable. There there are new forms

of risks that we really need to account

and control for.

>> One of the things you're hopefully going

to measure and help us with as well.

>> 100%.

>> Awesome. Well, thanks Ryan for joining

us.

>> Thanks for having me.