Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper
The economics of AI inference will collapse by orders of magnitude as workloads shift from low-latency interactive chatbots to asynchronous, long-horizon background agents that prioritize throughput over speed.
Decoupling inference from real-time human attention allows operators to utilize cheap, non-NVIDIA silicon and intermittent power grids across decentralized 1MW data centers, transforming intelligence into a universally accessible commodity.
Section summaries
Neil introduces Sal Research as an ultra-low-cost token factory and agent sandbox platform designed for long-running virtual machines. He explains that while the existing market optimized for low-latency coding assistants (e.g., Cursor), the real frontier is long-horizon tasks running for hours or days where latency is irrelevant. By decoupling user interaction from token generation, token consumption becomes an autonomous background operation.
- Reducing token costs by 10x unlocks entirely new product categories.
- Agentic inference will transition from low-latency interactive queries to long-horizon background workloads.
- Open-source model weights provide sovereign ownership and flexibility to run customized inference pipelines.
Establishes the core macro thesis of asynchronous background intelligence versus synchronous interactive chat.
The discussion covers test-time compute scaling, tracing capabilities from Claude 3.5 Opus to contemporary autonomous models. Neil highlights how background intelligence transforms deep research and cybersecurity into an automated 'proof of work' model, where spending compute to fuzz software finds jagged vulnerabilities. He argues that verifiable domains (formal math, code, scientific simulations) can be solved economically if intelligence is made abundant enough to waste tokens without guaranteed returns.
- Background agent token volume will scale to 90% of all inference workloads.
- Cybersecurity is evolving into automated 'proof of work' via heterogeneous model fuzzing.
- Verifiable tasks (software, math, scientific modeling) scale directly with available token budgets.
Provides concrete operational examples of high-token background agent use cases in software and research.
Drawing on his early engineering background at NVIDIA, Neil details the evolution of Tensor Cores since 2016 and the company's cultural obsession with 'speed of light' hardware utilization. He explains the fundamental computer science trade-off between latency and throughput: interactive systems require small batches with idle headroom, whereas background workloads permit massive batching that saturates GPU compute cores. NVLink enables low latency via tensor parallelism, but alternative schemes (pipeline and expert parallelism) offer superior FLOPs-per-dollar economics on non-NVIDIA chips.
- NVIDIA's engineering ethos revolves around reaching 100% 'speed of light' hardware limits.
- Latency optimization forces sublinear scaling and underutilizes GPU silicon.
- High-throughput batching on GPUs functions like a bus rather than a private taxi, maximizing compute density per dollar.
Explains the foundational hardware mechanics and mathematical trade-offs between batch throughput and interactive latency.
Neil breaks down the memory hierarchy differences between on-chip SRAM (used by Cerebras and Groq for extreme memory bandwidth) and stacked DRAM/HBM (used by NVIDIA). He identifies the 'original sin' of transformer architectures: pairing compute-bound multi-layer perceptron (MLP) layers directly with memory-bound attention layers. While wafer-scale SRAM chips excel at MLP matrix multiplication, dynamic and unpredictable KV cache expansion necessitates large-capacity DRAM, pointing toward hybrid chip architectures.
- SRAM offers petabytes per second of bandwidth but suffers from low physical die storage density.
- DRAM/HBM provides high gigabyte capacity required for dynamic context windows (KV cache) at lower relative bandwidth.
- Future inference clusters will likely hybridize wafer-scale SRAM engines with traditional high-capacity GPU memory.
Delivers an exceptional technical breakdown of memory bandwidth constraints, KV cache dynamics, and transformer hardware limits.
Addressing data scaling, Neil states that the public internet's 30–300 trillion tokens of high-quality text have been exhausted. The next paradigm is recursive self-improvement inside automated reinforcement learning (RL) gyms on verifiable tasks. He returns to infrastructure scaling, explaining how modern GPU programming is moving from individual accelerator kernels toward full rack-scale orchestration (such as NVIDIA's NVL72 liquid-cooled Grace Blackwell systems).
- Unconditioned human internet feedback has declining utility for frontier model post-training.
- Verifiable RL environments with automated self-grading provide the engine for continuous synthetic data scaling.
- Hardware orchestration has shifted from single-GPU kernel optimization to whole-rack (NVL72) cluster programming.
Connects post-training data dynamics directly to physical rack-level computing shifts.
The discussion covers the market dynamics of top-tier GPUs versus secondary accelerators from AMD, Tenstorrent, or startup ASICs. Neil describes an arbitrage model: while competitors chase scarce Blackwell allocations, huge value exists in programming alternative chips that vendors under-optimize. Comparing today's CapEx boom to the 2000 telecom fiber bubble, he argues current token spend is non-speculative because inference tokens are consumed immediately rather than hoarded.
- There are no bad chips, only bad pricing; unoptimized silicon provides software alpha.
- Inference spend is structurally non-speculative because tokens are consumed in real time.
- NVIDIA strategically allocates high-end chips to prevent single-customer power concentration.
Analyzes the semiconductor market, supply allocation dynamics, and CapEx sustainability.
Neil introduces the physical infrastructure strategy of running distributed 1MW micro-data centers instead of fighting over scarce 100MW+ or gigawatt-scale sites. Liquid cooling allows a megawatt of compute to fit in eight refrigerator-sized racks. By removing diesel backup generators, redundant fiber links, and strict SLAs, Sal can buy cheap intermittent solar/wind power, tolerating 80–95% uptime by shifting asynchronous agent workloads dynamically across facilities when outages occur.
- A 1MW data center can be compressed into approximately eight high-density liquid-cooled racks.
- Eliminating power and network redundancy slashes facility CapEx and OpEx.
- Asynchronous workloads easily tolerate 95% data center uptime via distributed software failover.
Presents a novel blueprint for reducing data center energy and construction costs.
Neil reviews global compute inefficiencies, highlighting that attention KV caches remain vastly uncompressed. On chip manufacturing, he argues fab yield margins and process corners can be relaxed for inference workloads. Regarding frontier labs, he views open-source distillation as inevitable through public artifact generation (e.g., GitHub repos created with AI tools), predicting that closed-source proprietary leads will remain short-lived 3-to-6 month windows.
- KV cache entropy is uncompressed by one to two orders of magnitude in current architectures.
- Process corner variances at fabs could be relaxed to utilize discounted, sub-tier silicon.
- Model distillation is impossible to halt due to widespread public diffusion of AI-generated artifacts.
Covers high-level fab optimization, memory entropy, and the structural dynamics between open and closed models.
Neil envisions a world where users consume a trillion tokens per day for complex proactive automation. He outlines the critical bottlenecks hardware entrepreneurs must navigate: TSMC wafer supply, HBM packaging capacity, advanced packaging, and energy access. Finally, he explains why NVIDIA focuses on building an ecosystem of profitable infrastructure partners (like CoreWeave) rather than selling direct end-user tokens, concluding with reflections on mastering the entire engineering stack.
- Long-term agent adoption requires dropping trillion-token job costs to tens of thousands of dollars.
- The primary hard bottleneck in AI silicon scaling is High Bandwidth Memory (HBM) packaging capacity.
- NVIDIA avoids competing with its infrastructure buyers to preserve a broad, competing ecosystem of customers.
Synthesizes long-term cost targets, semiconductor supply choke points, and ecosystem strategy.
Key points
- The Latency vs. Throughput Dichotomy in Agentic Inference — Chatbots enforce low latency at the expense of GPU compute efficiency, but autonomous background agents running for hours only need 10 tokens per second. Running wide and slow allows full hardware batching and vastly better token economics without needing low-latency interconnects like NVLink.
- The Architectural Split: On-Chip SRAM vs. Off-Chip DRAM — Transformers contain compute-bound feed-forward networks (MLP layers) and memory-bound attention layers. Ultra-fast wafer-scale SRAM architectures (e.g., Cerebras) excel at raw compute bandwidth for weights but struggle with dynamically expanding KV cache capacities, necessitating heterogeneous hybrid clusters with traditional high-capacity DRAM/HBM systems.
- The Scavenger Strategy for Compute and Energy Infrastructure — Instead of competing for scarce 100MW+ tier-4 data centers with triple redundancy, inference workloads can be distributed across decentralized 1MW micro-sites that tolerate 80–95% uptime and run on cheap, intermittent renewable power via resilient control planes.
- Shift from Internet Data Ingestion to RL Verification Gyms — Having exhausted the ~30–300 trillion tokens of public internet text, frontier model advancement relies on recursive self-improvement inside verifiable reinforcement learning environments (e.g., formal code execution and mathematics) where agents generate and grade their own synthetic task data.
“My job is to make the tokens as cheap as humanly possible. I will achieve that and I will do it through every layer in the stack available to me.” — Neil
“We think that whenever you make something 10 times cheaper, it's a new product category and uh we aspire to do that for tokens.” — Neil
AI-generated from the transcript. May contain errors.
My job is to make the tokens as cheap as
humanly possible. I will achieve that
and I will do it through every layer in
the stack available to me. I love the
supply side levers. I will use every
chip. I'll use every source of power and
I will use every piece of land in the
United States that's, you know, suitable
for this. We still treat the agent as a
person that is expensive to consult and
you should ask them when you have a hard
question. That's not the way to think
about intelligence. It's incredible that
the machine can think and we should
[music] try to get that into as many
hands as as many people as possible.
>> [music]
[music]
>> I think it's important early in these
conversations to just say the thing like
literally what you're building and what
it does today. So maybe just orient us
there with a with a brief description
like literally what the system is that
you're building and why it should exist.
>> Sal research is a token factory. We have
an API where anyone can send us requests
where they can use large language
models, open source large language
models for any task they want. Um, we
will serve those tokens to them at a
price that is unbeatable in the market.
We also support their ability to build
agents on top of this. We host what we
call sandboxes, which are longunning
agent virtual machines hosted in the
cloud that are designed for agents that
run for hours, days, or weeks. And so
you should think about you as a peer
company to others that serve different
kinds of inference. You're serving one
specific kind of inference and your goal
is to be the absolute cheapest provider
and enabler of a certain kind of use of
intelligence.
>> Exactly. The theme of our company is
abundance. We want to deliver this new
commodity of intelligence to as many
people as possible at a at a cost that
is sustainable for almost every
industry. We think that whenever you
make something 10 times cheaper, it's a
new product category and uh we aspire to
do that for tokens. We think it's so
profound that the machine can think and
now our job is to make as many machines
as possible in the world work towards
thinking.
>> So if you think about uh the theme of
the day being token costs is token cost
the right way to think about this like
is there some other way you'd put it
>> to start with? Absolutely. Token cost
today my north star is I want to have
the lowest cost per token in the
industry and do that by a mile. I don't
think tokens are the final unit of uh of
work or intelligence but they are what
we use today and so it's very
straightforward. I think after tokens
you start to move more towards um more
outcomes which is like a vague
direction. Uh you can imagine for
example today when you consume tokens
through an agent you don't actually
control how many tokens the agent
reasons for. It can reason for a certain
amount of time or it can call a certain
number of tools and increasingly I think
we will have agents do some unit of work
take as many shots on goal as they can
and however many tokens they use to get
there is going to be kind of a dependent
variable depending on the task.
So you think about like agents that
selfadminister a token budget as opposed
to a company setting a budget for how
many tokens engineers can spend per
month.
>> Why is there an opportunity that you can
tackle it? It seems like the entire
world is oriented around more better
faster cheaper tokens. Right now it
seems like the world is trying to solve
this problem very aggressively.
>> What was the unique opening that that
you saw that's maybe the market's not
being efficient in its attempt to tackle
this? So I think there's two things that
are tailwinds for our company. One is
got to be the rise of open source. I had
to talk about that first. I think we are
starting to see an increasing number of
our customers and the broader market
care about owning intelligence. They
they want to have control sovereignty
over the thing that they depend on. Uh
and so that created a much more robust
market for customized models or even
just like these vanilla open source
models that no one can ever take away
from you. You always have the weights.
You always have the right to deploy them
however you like. uh in that world
there's been a reasonably robust market
for the past couple years uh serving
these models at large scale. The
challenge is all those companies, you
could take your pick, base 10, fireworks
together, they all focus on low latency
inference, and they were pulled in that
direction by one very important
customer, uh, cursor. And
[clears throat]
I think that that was the right choice
about a year ago, and as of six months
ago, it started to look like maybe low
latency wasn't the only thing you wanted
from an agent. You wanted more
persistence, more long horizon tasks.
And now, it's to me very obvious that
the future of agentic inference is long
horizon tasks. You're going to run the
machine for hours or days at a time. It
doesn't matter if it spits out tokens at
100 tokens per second. Maybe 10 is just
fine. That comes with corresponding
advantages and efficiency.
>> Why are you so confident in that? It's
it like to me it seems like I want
everything as fast as possible.
>> When you're waiting on it, you
absolutely deserve the fastest answer
possible. My trick is I don't want you
to be waiting on it. I want it to be
proactive. I want it to be in the
background. One way to say it is like
the best latency is no latency at all.
When you wake up in the morning, the
work's already been done overnight. you
didn't even have to ask for it. Uh
that's the dream. We're not quite there
yet. But more importantly, I think the
more you're in the loop as you prompt
agents and wait for a response, in fact,
you're the bottleneck uh in helping the
in having the agent do more or less
work. What we'd like is the agent to
operate on more human time scales. You
don't manage your colleagues every 5
minutes. You ask them to do a high level
task and you come back and check in
maybe every day, but more likely once a
week. And that to me is the future of
human agent collaboration, more like
human time skills. say more about the
early indications that this is happening
and therefore you should be building
this company.
>> Well, so the first and most important
thing is the idea of test time compute
scaling. Uh the idea that you can give
an agent more time and it will give you
a better answer. So that was theorized
about 2 years ago now and uh but it
wasn't really something that we could
actually bet on until I would say late
last year with Opus 45. Opus45 was the
first agent that was at all suitable for
longer horizon tasks and you know it was
pretty mediocre and when it first came
out but you look at the more recent
models and what we've done on open
source as well and you see that agents
are capable of running for an hour at a
time. I wouldn't say it's days but
definitely an hour is quite suitable
today. And so just seeing that like
average turn or task length get longer
and longer uh it doesn't take many
points to have you kind of draw out the
exponential and see that agents are
worth running for longer periods. What
what do you think will be the market
share of longrunning agents in 3 years
or something like this?
>> You know, I love this market because
it's unbounded. There's no human in the
loop. So, you can consume as many tokens
as you like in the background. Uh versus
human attention span. If you tell me to
consume 10x as many tokens at codeex or
at cloud code, I'm actually not sure if
I can anymore. I'm already in a loop and
locked in coding for most of the the day
that I'm at the laptop. What is undowned
is how many tokens can be consumed in
the background or proactively. So
longterm, I think, you know, we're going
to end this year at maybe 50/50
background and uh and real-time
workloads, but I see this going to 9010
in favor of background.
>> What are the sorts of things like what
are your favorite examples of something
that gets accomplished much better as a
background task than as a human in the
lip task?
>> Most deep research, most questions where
you want to have a definitive answer
over not 100 sources, not a thousand
sources, but 10,000 sources or more. If
you want to build an authoritative index
of information like for example one of
our customers parallel web systems seeks
to do. They want to build an index over
the whole internet and they want to
monitor the internet in real time for
changes. That is the kind of crazy
exabyte scale task that you need a very
different kind of intelligence or scale
of intelligence to achieve. Deep
research is a top category for us and
then increasingly we see cyber security
following this direction. If you think
about, yes, there's so much code you can
generate, but there's uh exponentially
more ways to break that same code than
it is to generate that code. And there
are some great customers out there who
are working very hard to uh find agents
that can break any piece of software and
proactively patch them. So when Fable
first came out, for example, or Mythos
first came out, basically there was this
push in the cyber security community to
run Fable against every line of code
we've ever written and look for bugs in
20 different ways. uh meaning you're
looking for both memory errors, you're
looking for business logic errors and
looking for like network
vulnerabilities, all these things. And
these are all actually things that you
would write specialized agents for. You
wouldn't just have Fable look at the
source code once, you'd have it actually
set up environments where you can pen
pentest these applications. And at some
point, people started to make this joke
that security has become proof of work.
When you want secure software, it's
really a question of how many dollars
did you spend on anthropics APIs trying
to break into your software. uh that is
the best indication for how secure it is
because that's the best tool in the
world. And increasingly we found that
the frontier of intelligence here is
quite jagged. It's not the case that
Fable finds a supererset of all bugs in
software. You would find some bugs with
a very small model that you don't find
with the large model. Uh you'd find some
bugs with Haiku that you would find with
Fable and vice versa. So it encouraged
this very diverse approach to sampling
and trying to build cyber security
agents that break software autonomously
such that you can patch them. If you
were to get sort of like speculative and
imaginative about the sorts of things
that longunning very cheap very
longunning agents can enable. We talked
about some very practical examples deep
research um cyber security etc. But if
you if you get a little bit dreamier
about the use cases new product category
that this sort of inference will unlock
and I guess the question is just like so
what like what if you're maximally
successful dream a little bit about what
that might enable.
>> Yeah absolutely. So I think for
individual users what I'm excited about
most is this idea of proactive
intelligent agents. Um you can imagine
Siri that is running in the background
all the time to understand what's all
the emails you received in a day, all
the text messages you receive in a day
and and has a much more encyclopedic
view of your life and how to be helpful
in that life. Right now there's still
point solutions and so you have to you
end up doing a lot of prompting. Siri is
not very proactive. That's something we
can fix with abundant abundant
inference. If you trust uh the machine
enough that it's reliable and also
trustworthy as in private um you might
even imagine the machine can understand
how you interact with it and proactively
surface your next action whenever you
open your phone. Can we build a good
model of what you're going to do next?
>> My estimation is yes, we totally can.
>> And the key to that is incredibly cheap
intelligence.
>> You have to be willing to spend tokens
without any promise of return. That's
the unlock. The long lens view to take
on this is that we have we have a form
of intelligence that can tackle any
verifiable problem. Any verifiable
problem means most software. It means a
lot of formal like math proofs and
similar. And it could also mean
scientific discovery. These are all
relatively verifiable problems. And all
those things currently have a dollar
cost attached to them essentially.
That's a hidden one. It's like how many
tokens could you possibly harness to
make this work? And we have actually
started to bring it within view a dollar
cost for these long horizon tasks that
is reasonable. It's not millions, it's
thousands and maybe it could be hundreds
or even tens of dollars in the near
future to have a definitive answer to
any scientific question to any research
problem. RAMP is the only platform built
to make your finance team leaner,
faster, and better, saving businesses 5%
annually on average, so you can stay
focused on growth. RAM customers grow
revenue 3.2 two times faster than the
average American business. Visa,
Verscell, Kerser, Stripe, Notion, 11
Lab, Shopify, and 70,000 other
businesses all now run on RAMP. Mine
does too, and so should yours. Learn
more at ramp.com/invest.
OpenAI, Cursor, Anthropic, Perplexity,
and Verscell all have something in
common. They all use work OS. To achieve
enterprise adoption at scale, you have
to deliver on core capabilities like
SSO, skim, arbback, and audit logs.
Instead of spending months building
these missionritical capabilities
yourself, you can just use Work OS APIs
to gain all of them on day zero. That's
why so many of the top AI teams you hear
about already run on Work OS. Work OS is
the fastest way to become enterprise
ready and stay focused on what matters
most, your product. Visit works.com to
get started. Felix by Rogo is a personal
finance agent that turns a single prompt
into [music] finished client ready work
using your firm's own templates,
context, and standards. Send Felix an
email like, "Take [music] these comments
and turn them for me." Or, "Udate my
tracker with the context of these
emails." And Felix sends back finished
[music] PowerPoint decks, Excel models,
and sourced research. Felix works the
way your team already does, delivering
[music] work quickly and accurately
around the clock. Learn more at
robo.ai/felix.
[music]
>> And so, if we dream about that future,
we're we then become limited just by the
questions that people can ask.
Basically,
>> pretty much the questions we can ask. uh
the models are on the cusp of basically
taking even a high level question and
chasing it down every possible follow-up
you can have the model essentially take
that on its own
>> and the question is what is your token
budget
>> and we will solve the token budget
problem
>> what about non-verifiable
tasks
>> I put basically the entire category of
human taste into that category we have
not solved human taste yet and I don't
know that it fundamentally can be I'm
excited to be surprised here but um we
are focused on very quantitative uh
problems we leave the quality of writing
we the uh the beauty of art to to
people.
>> All right. Now, let's talk about the uh
the very clever stack of solutions that
you hope to build
>> ultimately to have this giant token
factory, extremely lowcost intelligence
supplier of extremely lowcost
intelligence.
>> I think you think about this in terms of
level software, hardware, uh and power.
>> Talk through what your master plan is to
approach this challenge that's so
different from what others are thinking
about doing.
>> You know, we always have to start with
software. you know where is the
opportunity on today's chips with
today's data centers to improve
efficiency and the first thing we did
was we tried to build the entire LM
software stack around peak GPU
efficiency meaning we're using Nvidia
GPUs we wanted to squeeze out more
tokens from the same chip than anyone
else in the world and that starts with
the lowest level of programming kernels
it's actually my background I spent my
whole life actually uh my whole
professional life working on GPUs and
kernels in Nvidia was my first job while
I was in college and uh I got to see how
the tensor cores got to earn their right
to be on the chip. This is back in 2016.
>> Just describe what that means for for
the lay person.
>> So, okay, tensor core is a specialized
unit on the GPU that accelerates matrix
multiplication.
>> Simple as that. There's been a long
history of how we evolved at tensor core
over time that we'll get into.
>> And why is matrix multiplication so
important?
>> That's a great question. I actually I
cannot say that there is a divine truth
of the inverse that explains why matrix
multiplies seem to be the atomic unit of
computation. But uh one way I've heard
it described to me is well it's a really
succinct way to mix two blocks of
numbers together and have them interact
in some interesting way. That's as much
as I can say about it. It is really
convenient that linear algebra turns out
to be a very compact representation of
arbitrary relationships in data. So
Nvidia great graphics company obviously
has had market share dominance in GPUs
and and gaming graphics for quite some
time. And then starting in like the
mid2010s they started to actually start
these like skunk works projects to make
the graphics processor more suitable for
machine learning tasks that they were
tracking. I remember actually reading
some of the like lab notebooks of some
of my managers when I was at Nvidia.
they would visit these small ML
conferences like ICML or NURPS at the
time and they would just take note of
these papers like oh this deep learning
thing seems to be catching on and what's
really interesting is that these grad
students are using gaming Nvidia GPUs in
order to train their large models we
should double click on this and figure
out what's going on here and by 2015
2016 at least Jensen had the conviction
to to kind of double down on hey this
usage of our models is only of our chips
is only going to grow let's start
allocating more and more precious
silicon die area to this capability that
seems to be emerging. Let's put the
first version of tensor cores on the
chip. So, we're talking about, you know,
taking this gaming chip which is
designed for painting pixels on a screen
and adapting it to do metric multiplies
and it was early and you would be
competing against the graphics teams
essentially when you ask for more
silicon area and any chip company.
There's always competition for that. It
is something that that the designers
guard so carefully. you don't ever want
to invest in the wrong technology
because that's opportunity cost that you
could have allocated to some other
functionality. And so we we kind of like
fought and tooth and nail and got just a
tiny bit of diary maybe like 5 10%
something like that for the first
generation of these chips uh to get some
some amount of acceleration for basic
convolutions which were the fundamental
operation for computer vision models in
the day. Uh, and then we had a software
team that was trying to squeeze all the
performance we could out of the chip.
And I think on that software team, which
is where I work, that's what actually
taught me the most about um, just the
ethos that Nvidia has around they have
this term called speed of light. They
always chase the speed of light for any
piece of hardware that they make. It is
so ingrained in every engineer's mind
that if the machine can do it, we're
going to push the machine to the
frontier until it does what we think is.
>> And the speed of light is the edge of
what's possible.
>> The speed of light is the edge of what's
possible. Exactly. uh if we think the
chip can run at this frequency and
produce this many multipliers per cycle,
we're going to get there. We're going to
break every bottleneck and get to that
peak level of performance. And so to
this day, I tell all my engineers like
we're chasing 100% speed of light. I
don't care about relative numbers versus
the competition. I only care about
absolute numbers. Uh what are we able to
do on the chip and how do we achieve
that?
>> Before we leave that chapter of your
time at NVIDIA, anything else beyond
that cultural touch point that really
like changed the way you think about
things or that stood out the most about
how the business ran back then or its
culture? I have a ton of stories about
Nvidia. We can I can tell you a few of
them. Um, one of my favorites is that on
the tenure side, a lot of people I
worked with in Nvidia in 2015, 2016 are
still there today. That company has
incredible retention and these are the
best engineers uh, frankly on the
silicon side at least I've worked with
in my whole career. They're extremely
extremely motivated and passionate.
They've believed in parallel computing
as a concept through its various
incarnations and have loved seeing the
chip evolve. This is their life's work
and they're extremely extremely
competent in that direction. They're
also a very frugal company. Nvidia and
all, I guess all the Silicon Valley
companies after 2008, they had some
cutbacks and like perks. So, no free
lunch. Uh, for example, Nvidia took it
one step further. There was no free milk
in the fridge. So, if you wanted to
drink coffee at Nvidia and you wanted
some milk, you actually had to chip in a
dollar every month to the milk club and
the milk club would stock Costco milk in
the fridge. And I remember that
distinctly. We don't do that at sale,
but uh
>> it's a it's a frugality that permeates
the company. And so coming out of this
time there, you get this experience of
what it's like to develop more efficient
usage of the underlying hardware through
software.
>> Yes.
>> And so so link that to, you know,
today's environment.
>> Yeah, absolutely. So, so I think um the
GPU is fundamentally a throughput
machine. The GPU is happiest when you
give it a lot of work to do and let it
chew through that work at peak
utilization of its compute units. But
that's actually not the way that we've
taken AI in the last couple years. We've
really pushed AI to be an interactive
chatbot tool is the most common form of
AI usage today. And in that world, you
care a lot about actually spitting
answers out to the to the person at the
keyboard as quickly as possible. To your
point about don't make the user wait, I
want things as fast as possible. And so
that's actually quite interesting for
the GPU. It's very difficult to put the
GPU in its happy path of being fully
compute utilized when you're trying to
spit out tokens quickly. There's a
fundamental trade-off on the GPU between
being uh throughput oriented or latency
optimized and everyone has chosen
latency optimization because the shape
of usage was chatbot oriented. I believe
that's the most profound change we're
going to see in the next year. We're
going to move away from chatbots to more
proactive or background agents. And in
that world, it makes a lot more sense to
build a stack around throughput.
>> Can you explain technically why the
trade-off between throughput and latency
is unbreakable? Why can't we have both
from the same hardware? It's quite
foundational in almost every system that
you could ever possibly look at. There's
always a trade-off between getting a
small amount of data through the system
as quickly as possible and leaving a lot
of buffer uh room for that or trying to
run wide and slow like narrow and fast
or wide and slow is like a classic
trade-off in all computer science. But
for GPU specifically, I think there's
one thing to focus on which is there's
this concept of like batching on the
GPU. We want to group many users work
together into a batch that we can uh run
all at once on the GPU. That's the
parallel processing of the GPU. We'd
like to have a lot of parallel work to
do. The thing is though, you're doing
net more work when you run a large batch
of compute together. And so you might be
filling all the units, but every step
along the way as you carry a a batch of
work through the GPU, there's more work
to be done. And so any individual token
or any individual user's request in that
batch, it's going to spend a longer time
on the GPU being carried with other
people's traffic. Maybe the way to say
it is um you know, if you want to get
downtown and SF, you can take the bus or
you can take a private transit. And the
private transit is going to have its own
direct path as the crow flies or you
know, using exactly the roads that you
want from point A to point B. a bus,
it's going to have to serve many more
people and it it has to fundamentally uh
do something that works for everyone
>> and so it takes a slower path and it
stops and and waits for other people to
get on and off. I think the bus versus
car analogy is pretty accurate
>> and it's a great analogy and so step one
for what you're trying to do is like
create the best possible bus on top of
Nvidia GPUs. Like that's step one of
your optimization.
>> That's exactly right. It means we
explore things like different
parallelism schemes. Maybe that's
another example I can give you is um
with Nvidia GPUs, one of the things that
they've really innovated on and done a
great job with is the NVLink uh
interconnect between GPUs. And in fact,
that NVLink system is so good that you
can if you have a large matrix multiply
that you want to perform faster. You can
actually cut that matrix multiply in
half and shard it [clears throat] across
two or more up to eight, let's say,
Nvidia GPUs and have them all work on
pieces of that larger matrix multiply
>> and have them connect their results
together at the end. Reduce their
results back together at the end. And
this is a great great way to cut the
minimum latency of of an operation.
You're each GPU is now doing 1/8 as much
work, let's say, uh, and therefore it
can finish faster but not eight times
faster. It's sublinear scaling. You'll
use eight times more hardware, but you
won't get eight times the speed. You
might get like four to fivex the speed.
You're not going to get strong scaling.
And this is because of communication
overhead. It's because every GPU is
going to be a little bit less efficient
working on a smaller tile of work than a
larger tile of work. And so, it's the
only way to speed up if you want the
minimum latency possible. You can do
that, but it is not the choice I would
make. For example, I would prefer to use
a different parallelism scheme like
expert parallelism or pipeline
parallelism. And we may do interesting
things to overlap and hide the
communication latency in a way that you
would have less ability to do that for a
low latency server.
>> So is the right way to think about
NVLink as a technology which improves
latency performance? Yes.
>> And only latency performance
>> which will segue into the next segment
of what we you know do differently as a
company. But yes, NVLink is mandatory I
would say for low latency inference.
>> So Nvidia is excellent at low latency
inference. And I'm telling you that we
don't really care that much about low
latency inference. So where does that
leave us? Well, I think I'm not holding
my breath for other companies broadly to
figure out NVLink quickly. It's a
challenging technology to figure out.
It's hard to scale. It's hard to
productionize. And so, if I do have some
other vendors chip and it is good at the
foundational compute components, it can
still do metric multiplies really well.
It just can't communicate those results
across its peers quickly. Well, maybe
there's a room for that other chip in my
stack as a really really good compute
per dollar option. And that's what I
actually optimize for in most cases is
how many flops does this chip have and
how much is it going to cost me per hour
to operate to own and operate. Uh and so
there are other chips that definitely
rank higher than Nvidia on flops per
dollar, but they may not have as much
interconnect. And so it's my job to
figure out what parallelism scheme am I
going to use that's going to make this
chip suitable for inference. It's not
going to be tensor parallelism. Nvidia
is basically mandatory for that. But
other techniques may work well for me.
So before we leave the latency part of
the story, can you comment on companies
like Cerebras or others that can perform
incredibly fast operations? I'm curious
like what you think about those
approaches, those companies, what might
happen in the future. What is your
prediction for the future of very low
latency focused hardware? Cerebrus Grock
uh and a couple others that are coming
out of stealth now I think have made a
very interesting bet on not just
building another GPU but actually
building a different kind of accelerator
that focuses on a different memory
hierarchy. Uh they want to maximize the
amount of SRAMM on the chip and use that
as very very fast memory for for weights
and KV cache. So SRAM versus DRAM
there's two ways to make memory for a
chip. One is to integrate the memory on
the logic die itself. Like meaning you
tell TSMC, I want this many megabytes of
of storage on my chip. Uh and there's a
way to build that. TSMC has a standard
cell library you can use and you can
just print out a bunch of cells of SRAM.
The problem with SRAMM is it takes a lot
of area on the silicon die. Um, so if
you want to build a large die like let's
say the Nvidia Blackwell at 800 mm
square. If you made that whole DS RAM,
it would be in the maybe like
singledigit gigabytes, it's not a crazy
amount of of data storage. Compare that
to if you're willing to take a different
process technology entirely. So not TSMC
anymore, but now Micron SKH Highix
Samsung. They build DRAM, which is a
whole different way to build memory
that's more focused on capacitors than
transistor cells. So, SRAM, the standard
way to build SRAMM is what's called the
6T transistor cell. It's a stable
transistor arrangement that allows you
to write a bit to it and then it holds
that state in that bit regardless of
whether you keep applying. Well, you had
to apply some power, but uh it it's
holding that bit without any sort of
like active management. It's static.
Now, dynamic RAM, DRAM, it's dynamic
because what you do to write some data
is you write a charge onto a capacitor
and as soon as you write that charge
into that capacitor, the charge is
dissipating. it's been leaking. And so
the dynamic part of DRAM is that you
must every 50 milliseconds or so refresh
every bit you've written. So you're
constantly juggling billions of balls in
the air essentially billions of bits
have to be managed by a memory
controller which is reading and
refreshing every bit on the DRM. Now the
benefit of that is you can get much much
higher density and it's a whole
different process technology. There's a
ton of different trade-offs. Hence why
we split the DM manufacturing into an
entirely different company like Micron
SKX and Samsung. These are the best
companies in the world to do this. They
build DM. And if you take DM from those
companies and you stack it uh into many
layers and you kind of print them or or
solder them around the main logic die
that you get from Nvidia, you can now
get hundreds of gigabytes uh like
Blackwell has 288 GB of HPM capacity
around the logic die. And the logic die
itself maybe only has like 500 megabytes
of of SRAM. So it's possibly multiple
orders of magnitude, three orders of
magnitude difference in density for DRAM
versus SRAMM. Okay, so let's go back to
Cerebras. What are they doing? Well,
they see this problem, there's not
really an obvious way to increase SRAM
density on the chip. But thing with
SRAMM is because it's so physically
close to the logic gates that actually
do the computation, the arithmetic logic
units are right next to the SRAMM that
they're going to pull from, the compute
units that are doing the matrix
multiplies can pull data from SRAMM at
just mind-boggling speeds. You know,
Serbis quits pabytes per second, 21
pabytes per second further away for
scale engine 3. And so compare that to
HBM on an Nvidia black wall is u you
know 10 terabytes per second or so in
that range. So once again, many orders
of magnitude difference, more capacity,
but proportionally less bandwidth
essentially.
>> And so what Cerebrus does is they say
that we're going to take as many of
these dies as we can. We're not going to
limit ourselves to the 800 millimeter u
reticle limit, the TSMC 800 square
millimeter limit that TSMC imposes on
us. We're going to take the entire wafer
and have actually every die connect to
every other die over scribe lines. And
we're just going to try to get as much
SRAM as we can on the whole wafer. and
we can get to like let's say 50
gigabytes of SRAM per wafer and then
we're going to stack many wafers
together in a pipeline or similar and
now we can have you know up to a
terabyte of memory very very fast memory
and you do all that work just to get to
the ability to read data from SRAMM at
yeah 21 pabytes per second per wafer
therefore you can now serve these
language models at extremely high tokens
per second because you can move the
entire parameter count of a large model
like Kimmy uh you can move all that data
in and off the chip or sorry in and off
the logic cores in about a millisecond
or something like that.
>> So there you go you have a path to a
thousand tokens per second
>> and so what is your prediction for like
that segment of the market? Okay. So I
think what happens to them is some
hybrid sort of outcome like we we had to
pair the Cerebras chip where it's very
strong. It's very very good at fast
access to memory with something that has
more capacity for memory because it's
true that you can take a one trillion
parameter model like Kimmy and fit it on
a large number of cerebrus wafers. But
you can't do something about the KB
cache very easily. The KV cache is
something that grows as people use the
model more and that is always dynamic.
You don't even know how much KV cache
you're going to need. It depends on what
your users how many users you have and
how many users you want to serve.
>> Can you explain KV cache just like in
>> basic? Yeah. So KB cache whenever you
use a language model every token you
send through the language model actually
uh stays in the context window of the
language model for as long as you're
having a conversation. So if I we talk
for 100,000 tokens, the 100,000th and
oneth token is still in the conversation
uh behind us and the model is
referencing all the past conversation
history in order to make better
predictions about what the next thing
we're going to say is. And so that KV
cache is a bunch of memory. Um you have
to store a representation for every
token that you send through the language
model. And it frequently gets to be
larger than the weights of the model
themselves. You have this like
crystallized knowledge in the model
weights and you have the dynamic
knowledge of the exact conversation
we're having in the KB cache is the way
I like to think about it.
>> Yep. And and this is why sometimes
people would observe like deep in a
conversation things start to degrade
because there's some sort of like
technical problem.
>> Yeah. So the KB cache is quite
interesting in that regard. The KB cache
is an exact representation of everything
that came before. We we store all the
information that we've seen in the
conversation. However, during training,
the model did not get trained primarily
on very long context conversations. It
got trained primarily on, let's say,
8,000 token conversations or 16,000
token conversations. So, if you take the
model to 200,000 tokens, there was some
training that happened at that context
length, but it's not the model's like
core strength. And so, there's there's
always been a challenge for the Frontier
Labs to figure out how do we make the
model exactly as intelligent at 10,000
tokens as we expect them to be at
200,000 tokens. And it's going to be a
perennial battle for us. We've had 1
million context windows as a concept for
for years now. Enthropic was I think the
first to hit the 1 million context
window length. I still, you know, use
/compact in my cloud code uh well before
1 million context length. I don't think
it's actually great to hit the full
length.
>> And so these extremely fast, extremely
low latency approaches ultimately are
limited by by this factor.
>> Yes, you can do whatever you want for
the weights. It's very possible to have
just unbeatable performance on weight
storage. However, the KB cache is going
to be a big thorn on your side. And so
three years from now, five years from
now, what role do you think these kinds
of chips play? Like what sort of market
share do they have in the heterogeneous
chip market?
>> Crisis and Grock and maybe a couple
others, you should think of them as
accelerators. What they are really good
at is being used in conjunction with an
more traditional GPU like device that
critically has this offchip memory built
in. You want offchip memory for capacity
and onchip memory for speed. We want to
hybridize these two things. So if you
take uh transformers in the limit, you
take a transformer to a million context
length. What ends up happening is you
have this you know computebound stage
which is the actual matrix multiplies
for the uh what we call the MLP which is
where most of the model's knowledge
world knowledge is encoded and then you
have the attention layer which is where
we're kind of dynamically adapting to
the current conversation. Attention in
the limit is usually memory bound and
the MLP in the limit is computebound at
large enough batch size. And I would say
the original sin of transformers is that
you've taken this extremely
fundamentally memory bound layer and
juxtaposed it right next to a
computebound layer. It is very difficult
to have a single chip that is good at
both compute operations and memory
operations. The GPU is quite balanced in
this regard, but you have to choose one
or the other. Cerebrus has a very fast
memory access for something like a
matrix multiply and it's really good to
host the the MLP the the weights
essentially on the Cerebrus chip but the
GPU has the capacity to scale to really
long context lengths and so you would
like to put the uh attention possibly on
the GPU and the MLP on the cerebrus chip
>> and I believe this is what's happening
with Nvidia and Grock. Can you riff for
a minute just on transformers and uh
>> yeah, you've been so good at explaining
some of the core concepts just for
people that again aren't aren't deeply
familiar with what this innovation was
in 2017
>> like what its strengths and weaknesses
are and whether or not you think it will
remain the dominant architecture or a
dominant architecture for the future of
AI. What it did was it it allowed us to
learn an unsupervised data really
effectively because transformers what
they're all about at the end of the day
is taking any sequence any arbitrary
sequence of data and trying to find
patterns in that data and they
critically the attention operation which
is the headline uh component of
transformers. It allows the model to
dynamically adapt to what it thinks is
the most relevant component of the
sequence. every step you take through a
transformer, you are essentially like
reweing the input that you looked at
before and figuring out which is most
relevant for your next prediction. And
so it it's extremely amenable to
>> uh learning arbitrary sequence data. And
the most interesting sequences of data
that we produce on a regular basis is
language
>> and that's how we got to dominance in
the language regime. But uh to zoom out
even further, I think what transformers
really did well is that they scaled.
Transformers make no such human prior.
Transformers just say, "Well, there's
going to be a pattern in the sequence of
data, and if there is a pattern, I'm
going to find it. I'm going to throw
more and more parameters at this problem
until it works."
>> Uh, and transformers benefit from a lot
of the computer vision work, too. For
example, one of [clears throat] the
challenges in computer vision was we had
a hard time going from hundreds of
thousands of parameters, which you get
for like linear models like support
vector machines or other legacy machine
learning models. Those had, you know, on
the thousands of parameters. Then we got
to deep learning and got to tens of
millions of parameters with computer
vision. The biggest models were you know
around like 150 million parameters was a
huge model for computer vision. And now
we routinely talk about trillions of
parameters and transformers are the link
to go from millions to trillions of
parameters.
>> And so if I think about the important
units of scaling being data and compute
>> does it stand a reason then that you
think transformers will just stick
around because that's the thing that
we're good at getting more of those two
things. Well, data is an open question,
but comput. Yeah, transformers are so
they're just such great sponges, you
know, like you you you increase the
compute available to a transformer by
10x and you'll get you'll get some log
improvement somewhere. Uh and and so far
the scaling laws really work. They're
really quite beautiful. And to
[clears throat] the point about I guess
what do transformers do really well?
They extend to almost any data set you
can throw at them. They're extremely
powerful general learners. And I think
what's especially useful about
transformers over other techniques that
we've tried to replace attention is
transformers represent any pair wise
relationship that you want. Any token in
the sequence can attend to any other
token in the sequence. So if there's any
relationship that's in the sequence at
all, you're going to find it with
transformer. Now it may be the case that
you don't need all toall modeling. You
don't need every token to look at every
other token. But if you need to,
transformers give you that option. And
until we know a better way to kind of
prune that space down uh a better way to
kind of have information modeling be
more selective attention is a very very
good operation. This is another kind of
trick that we learned in the computer
vision days. Like one of the old
Karpathy sayings is that you know if you
have a new data set that you want to
train a model for. Your first goal
should be to overparameterize the the
model and try to overfitit the data that
you have to prove that there is a
relationship that you can model or
memorize that your learning algorithm
works uh that you can instill knowledge
into the model. Once you can overfit
then you can compress and the
compression is how you get
generalization. You don't want to
actually memorize the data that you have
in front of you. you want to generalize
and therefore once you overfit the data
set then you can kind of work backwards
and try to find the general patterns
that fit into the smallest parameter
count possible.
>> What's your prediction for the future of
data and riff on the importance of data
in this whole story? I like the phrase
that internet was a onetime subsidy on
data. We got it for free. Uh it's
extremely high quality about 30 trillion
tokens of high quality text. Uh 300
trillion tokens if you take a wider view
on what qualifies as good text and we've
basically looked at it all already.
Models have seen the entire internet
many times over at this point. And there
is not a whole lot more to be done on
human data from the internet. The next
phase of data in my mind is model
self-improvement through RL environment
gyms. Basically, in fact, we don't even
benefit from getting more like random
user interactions with AI. It used to be
that, you know, the the new type of data
that we cared about a lot was the
interaction data from people using
chatbt and giving TetBT signals on what
they liked and didn't like. I like the
argument now that the median model that
we serve is so much more advanced than
the kind of un
than like a random human uh giving
feedback that the signal you get from
random human preference or I guess
unconditioned human preference is not
actually worth anything anymore. You
want expert human preference at this
point. The model has outgrown everyday
>> generic Yeah. Everyday Joe. Exactly. So
the feature of data to me is giving the
model a hard verifiable task and letting
it run in this gym where it's kind of
isolated and it just has a a problem
that it can make progress on and get
measurement of whether it made progress
on that problem or not. You can imagine
coding problems are in this category.
math problems are also in this category
and um increasingly more and more we
have we can just give the agent a
computer essentially and have it act
like it's a human worker and just give
it feedback on whether it's making
progress towards the target outcome that
environment becomes the data. I think
this is not a super differentiated take
but uh it's been really really
productive from what I've seen so far.
And you think that just goes on for a
really long period of time or is that
another like if I think about the
internet as this one big block like this
is another big block that will have its
you know day in the sun and we'll kind
of get it all and and then we'll have to
move on to something else.
>> I think it's actually more profound than
that. Basically the idea is that if you
want artificial general intelligence the
best way to get there is to just keep
stacking specialized intelligences until
you have no more gaps to fill. And the
test here, the only thing you need to
make sure you do to make this work is
you must make sure that your task is
verifiable. You need to give the model a
self-grading system. If you have that,
you have the recipe for self-improvement
on any task you like. And I think you've
seen this held up by the way frontier
labs spend. They used to spend much that
much on data. Now they spend a lot more
on RL environments. And these
environments absolutely capture that
relationship of recursive
self-improvement on a verifiable task.
>> Okay. Okay. Now, so I like that we've
veered off in different little side cars
here, but coming back to your initial
task of making existing hardware more
efficient.
>> Yes.
>> By being more in control of what's going
on at the hardware level through
software.
>> Um so, so yeah, just keep going on what
you've done so far and what you want to
do and then we're going to jump to
hardware and then jump to energy
finally.
>> Sounds good. So, yeah, I mentioned
kernels. It's surprising people think
kernels are done. There are great people
like Triau who write excellent kernels
and they're they form the bedrock of all
of our um modern deep learning is built
on flash attention. Modern transformers
are built on flash attention. But if you
deviate from the happy path at all, if
there's a new model that comes out that
has a slightly different way to embed
positional information like the change
of the rope system. Suddenly the kernel
that we had is not suitable for this new
model and we may have to make a a patch
to this kernel. I wouldn't say we're in
the phase where we had to invent new
kernels from scratch, but having the
ability to quickly modify existing GPU
kernel, sorry, a kernel, by the way, is
a it's a general term for any program
you run on the GPU. And so,
historically, kernels tend to be put
into a library where every kernel has a
very very scoped purpose. Typically, you
have a kernel for a matrix multiply. You
have another kernel for even something
as simple as addition. You want to add
two tensors together, that's another
kernel.
>> [clears throat]
>> And then increasingly we've started to
fuse those kernels together. So if I do
a matrix multiply and then I want to add
it to another matrix that I've also
multiplied maybe those two become one
kernel and I just fuse the operations
where instead of writing the data out to
DRAM and then reading it back in just to
do the addition maybe I can just do this
uh easily.
>> Why are humans still doing this? Like it
seems like the sort of thing that AIs
would be exceptionally good at
engineering more efficient kernels.
Maybe that's where we're going and we're
just not quite there yet. But if if we
aren't there yet, is that where we're
going? If we're not there yet, why why
humans still doing this? Why why is Tree
out so wellknown? You know, it's a name
I know.
>> I don't want to speak for Tree, but what
he taught me was uh you shouldn't write
kernels by hand anymore necessarily. I
like to say we write kernels in the
whiteboard. We go to the whiteboard, we
describe what we think the machine
should be doing, then we succinctly
describe that in in natural language to
the a model. And then the model is able
to do the execution of okay, here is my
input and output. here is the strategy
of how we want to dispatch this work
onto the GPU. I'm gonna go implement
this.
>> So, we're doing the conceptual design.
>> Exactly. And that I think I'm not sure
exactly why models are not superb at
doing this. I don't think this is like
our remote or anything like that. I'm
sure in 6 months time we'll have much
better models uh on kernel engineering
and I'm sure the labs would tell you
that they already do a lot of their
kernel engineering uh in a fully
automated way. And so software as an
edge,
>> yeah,
>> if I think about software as maximally
near speed of light, efficient usage of
of an underlying piece of hardware
>> is is going to trend towards not being
an advantage for a company like yours
over time.
>> That's right. The rising tide of
something like Mythos or GBT 5.6 Soul
that lifts all boats. It really does. Um
I actually don't think there's a point
in specializing to say we work on making
the model better for just kernel
engineering. I think that's actually not
not the most meaningful subset of of
like coding in general,
>> uh, kernel engineering in particular.
Maybe there's some like privilege
information you inject into the prompt
that's like a useful way to steer the
model to be better at writing kernels,
but broadly speaking, yes, we're all
we're all downstream of the frontier in
terms [clears throat] of this
capability. I I always love this uh this
from the history of energy there there's
always this like pendulum between the
raw source let's say coal
>> and then if there's a certain amount of
energy available inside of a chunk hunk
of coal like what percent of it we can
harness and use
>> and a big part of the history of energy
was getting that number from 10% to 95%
or whatever
>> right
>> where are we in that's like if I just
think about it at Blackwell or something
and Blackwell is the piece of coal
>> like what percent do you think we're at
like How how efficiently can we use an
existing piece today?
>> There's a lot of different ways to
analyze that. I think in some level we
are really efficient at optimizing the
performance when the GPU is doing the
thing that it's most happy doing which
is a large dimension matrix will apply
that operation runs at you know 70 80%
of peak utilization and it's limited not
by software but by power. The way Nvidia
quotes peak flops is a little
optimistic. You never hit that because
of power throttling but um
>> because of heat.
>> Yeah, exactly. thermals let's say 70 80%
it's saturated it's pretty good
>> but in practice you don't spend the
majority of your time in a transformer
in that happy path where you're doing a
large batch m matrix multiply
>> and so
>> uh our job is to basically build the
engine around the chip such that we are
feeding the GPU these large batches of
work at all times
>> and one of the most profound transitions
we've had in the GPU world in the last
year has been this moving of you know
you don't program one GPU at a time
anymore you should think about the whole
rack and maybe you should think about
the whole cluster, the entire data
center at a time. And with Nvidia again,
they've started shipping not just a
single GPU or a single motherboard, but
actually the the whole rack system is
something that they prescribe. They call
it NVL 72. Uh their latest chip, the
Grace Blackwell 300, um that ships as a
rack of 72 units. And it is it's an open
race to figure out who can program the
whole rack scale computer as efficiently
as possible. And my belief is that that
shape of compute is the future of both
efficiency and speed. In fact, Nvidia
does a great job of if you want the
lowest possible latency, you should be
using that chip. And if you want the
highest possible throughput, you should
probably also be using that chip as of
right now.
>> And it's all comes down to like this is
a very new paradigm of programming.
>> One of the things you hear is that the
market for the best chips, blackwells,
let's say,
>> is like a drug market or something right
now. Like there's all sorts of
fascinating things happening to get as
many of them as possible because
everyone's so short. Y
>> I'd love you to react to that analogy
like is that what it feels like
>> but then also to talk about what the
market is like for like not the bleeding
edge chips like if I if I
>> am willing to accept a slightly or or
moderately inferior chip
>> what's that market like let us into that
world
>> yeah okay a couple things number one uh
yes basically has a long-term view on uh
on all their chips they they see this
immense demand for the black hole chips
and they they can do what other
suppliers have done in the past which is
like just crank prices and made the
market you know supply and demand curves
will correct they'll intersect at some
point and everyone will be technically
happier but Nvidia sees the if they just
let the most deep pockets buy all the
chips that maybe hurts them in the long
term if that customer ends up acrewing a
lot of more power they understand that
compute is power today and so uh they're
quite strategic about how they allocate
compute that's the first thought the
second thought is that relationships
matter a lot nobody wants to have a huge
order of of a chip rental come in from
this new startup that says, "Oh yeah,
I'm going to rent 10,000 black wells for
for three years or 5 years." The startup
has only been operating for months
typically. Who knows they're good for
the money. The the way you convince
someone to give you access to compute is
is quite challenging uh these days and
requires some pretty either great
relationships or uh just incredible
financial backing to make this happen
>> on the Nvidia side. And it's all because
the scarcity is so high and demand is
just off the charts. Now, for other
chips, I wouldn't even call them
inferior. I I like to say there's no bad
chips. There's really bad pricing. And
uh I will make any chip work at the
right price. That's like kind of one of
the ethoses of the company. And let's
talk about AMD. AMD, I think great chips
overall. The challenge is that people
don't um understand how to program them
very well. So, you know, I've been
talking to you about how we have such a
great kernel team. We're so serious
about squeezing the performance out of
the hardware. Nvidia is pretty good at
doing that for their own chips. Frankly,
there's some alpha that we can squeeze
out, but actually there's a lot more to
be done on other chips because the
vendor does a little bit less work than
Nvidia does to make the best kernels out
of the box or or even better for me
there is alpha and just like other
people have this perception that AMD is
not as as good as Nvidia. That's music
to my ears. I'm very happy for them to
sleep on this chip and for me to buy as
much as I can.
>> Now, I think that that's not actually
super true anymore. I think AMD is
actually uh somewhat popular amongst the
some large buyers. Um you know I think
publicly Meta and OpenAI have bought a
ton of AMD chips and so we're
increasingly seeing that uh all the AMD
supply is also being allocated but
there's a long tale of other companies
that are popping up yet net new
companies are great uh such as etched or
or senova or dmatrix all these companies
are popping up and I think the main
challenge for them is scale can they
actually get enough wafer allocation
from TSMC to pump out chips to make it
into the market but certainly if there's
a new chip on the
I'd like to know about it as quickly as
possible and evaluate whether we can buy
a good fraction of that supply.
>> And so it's fundamentally an arbitrage
for you. Like if you can be much better
at eking out performance from chips that
have received less attention, you can
then resell that at a margin and it
could be a great business.
>> Exactly. Exactly. And I think that it's
not the case that everyone else is just,
you know, has a skill issue that they
can't uh, you know, make these chips
work as well. I think we're quite
competent in this. I think we're
probably one of the best teams in the
world to use multiple silicon
architectures and and be quite
aggressive in chasing down performance
in unlikely places. But um yeah, I think
it's the speed at which we we're willing
to kind of build our stack around a new
chip. We don't have a huge amount of
incumbency around well our data center
providers are only stuck with with this
class of chip and it's going to be a
huge pain for us to to deploy uh these
net new chips. We have some very
creative data center partners who are
willing to move very quickly and there's
a new class of those that we can talk
about and most importantly we don't shy
away from the challenge. Uh that's
frankly a big part of this is just
saying yes we love TPUs we're going to
make TPUs work. Yes we love tranium
we're going to make tranium work and if
it doesn't work um easily we're going to
find a way to fit it in with the
heterogeneous serving system it will
have a place every chip has a
comparative advantage we have to find
that advantage and then squeeze it in
that direction. Ju just as an interlude
before we get to hardware, data centers,
energy, etc. which will be really fun
part of the conversation. I'd love you
to talk about your perception of the
investor classes worry. Yeah.
>> Like you look at memory stocks.
>> Um or my current favorite is you look at
the chart that plots the percent of the
S&P 500 that's semiconductors.
>> Historically it was like 2 3 4%. Now
it's 19 20 21%. And it just sort of
looks like if you're a student of market
history, you get all these things
through time that are sort of reached
some crazy near-term peak and then and
then collapsed back to long-term norms.
>> Um, and I'm curious how that has all
investors worried.
>> Yeah. Uh so you know a lot of people
made a lot of money in Micron and Skhinx
and companies like this but everyone
feels like ah these you know on the long
term like compute's a commodity and uh
it will not represent a quarter or fifth
of the entire market capitalization of
the world and and so they're scared and
that's the setup. Um everyone
acknowledges that like there's a huge
shortage but everyone sort of feels like
ah we'll figure it out and these things
will revert back down to their their
normal place in capital markets. I'm
curious what you think about about that
narrative.
>> One thing, I'm less of a student of
history as more of a member of history.
I was I was born in 1997 and uh my mom
worked at Intel in the 2000 in the
run-up to the year 2000 and the the do
boom and crash and you know I remember
the time where Cisco was the most
valuable company in the world and and
Intel was close behind. I mean I mostly
draw parallels to that period of history
from 25 years ago to today. And I think
the main difference is that a lot of the
investment in networking equipment
historically was speculative. We
anticipated this future demand for users
that never came. And I think what's
interesting about token consumption or
AI consumption broadly is that it's no
longer speculative. People buy tokens
because they're immediately valuable to
them. You don't hoard tokens, you use
them immediately. This is also even
different from what we had 2 years ago
where there was a supply crunch for
hopper generation chips in 2023 2024. Uh
in that period it was all training
oriented spend and training is
inherently speculative. Now it's
everyone is instituting caps on how much
you can spend on cloud code. It's a very
very different world to be talking about
inference spend and predicting inference
spend to go up. I do think inference
spend monotonically increases. Uh
there's no speculation on inference
spend. Vanta automates security and
compliance for over 16,000 fast-moving
companies like Ramp, Cursor, and Harvey,
keeping them audit ready around the
clock. It's the number one Agentic Trust
platform, and it now helps companies
like yours watch for the risks that show
up between audits across your vendors,
your AI tools, and your whole [music]
environment. Every new tool your team
signs up for, every vendor that turns on
AI features, [music] is an opportunity
for something to go wrong. And most
security programs weren't built for AI's
pace of growth. The Vant agent works
like a 24/7 GRC engineer in the
background, finding issues, drafting
fixes for you, and cutting vendor
assessment time by up to 50%. Whether
you're a fast growing startup or a
global enterprise, [music] Vanta helps
you earn and prove trust. Invest like
the best listeners. Get a special offer
of $1,000 off Vanta at vanta.com/invest.
Ridgeline is the first endto-end system
of record with embedded AI for
investment [music] management firms
running portfolio accounting,
reconciliation, reporting, trading, and
compliance, all on one unified platform.
Firms are moving off legacy technology
and onto Ridgeline because of how far
ahead Ridgeline's AI features are
compared to anything else in investment
[music] management software. I've been
hearing from a lot of investment
managers about AI, and they fall roughly
into two camps, with some unsure where
to even start, [music] and others
convinced they can build their own order
management system over just a weekend.
The reality is that running an
investment firm will always require
governance, [music] controls, and a
single source of truth for your data.
And no amount of AI enthusiasm changes
that requirement. If you're serious
about your firm's AI strategy, Ridgeline
[music] should be part of that
conversation. And you can request a demo
at ridgeline.ai.
>> Coming back now to your take on
hardware. And so the unit level is
interesting to me like talked about
chips, talked about racks, talked about,
you know, clusters. I'd love to talk
about data centers
>> and you said you've had some interesting
partners doing some cool things. Talk us
about the the present and future of data
centers as you see it.
>> Y
>> because this seems like you know
obviously a critical thing for being
able to serve all this inference is like
lots of innovation in in this part of
the world and obviously you're focused
on it.
>> I think one of the themes in our
conversation has come back to what is
training versus inference like what is
the difference between these two?
categories and you know what was
different about two years ago being
training oriented and today being
inferenceoriented and I think the most
conservative players in the entire AI
stack have got to be the infra players
whether that's data centers or even more
conservative is TSMC the chip infra
people uh and so data centers
historically were built like AI data
centers they were built for training uh
and training is the superset workload
over inference you can make any training
cluster work for inference but maybe not
vice versa and what the difference there
is networking uh how much do you invest
in bandwidth between chips and how large
of a cluster do you need? There's
actually a diseconomy of scale to to
data centers in some way. Like it's way
more expensive and difficult to build a
um you know 100,000 GPUs in one data
center than it is to build 10,000 than
it is to build 1,000. And and we now we
just talk about you know how many
megawatts or gigawatts do you have? And
basically there's no way to build a
gigawatt data center in the United
States easily anymore. Even 100
megawatts is is increasingly hard. It's
basically impossible unless you're a
very special set of customers. Uh 10
megawatts is probably on the edge of
what's possible today and 1 megawatt I
would argue is plentiful. So there's
this incredible lore on the market where
you can find lots of aggregate power but
it will not be concentrated and that was
not interesting to anyone who's building
training uh data centers because you
just assume all be in one spot for no
one wants to deal with cross data center
training.
>> So the market has some lag in it. I
think that the market still assumes that
we have to go shake down those 100
megawatt and 10 megawatt data centers
wherever we can find them is still the
attitude I hear from a lot of data
center developers
>> but increasingly we're seeing a few new
thinkers realize that inference is going
to be suitable for these distributed 1
megawatt data centers and uh we're we're
quite in agreement with that and we are
very happy to buy small pools of compute
across the United States and use that as
our inference fleet. give us a sense of
literal physical size of uh 1 megawatt
versus 10.
>> Yeah. Well, so this got really wonky
with the advent of liquid cooling. Now
you can pack insane levels of power
density into a single physical rack.
Like a megawatt of compute, you you'd
imagine this like massive data hall,
like a huge warehouse basically. And now
you can actually pack that into Yeah.
around like around like eight racks
worth of compute. Each rack is about the
size of a refrigerator. You can just
imagine eight of them lined up. Um,
yeah, that's a megawatt. And so your
view would be that the future that you
want to help build is a whole bunch of
different chips that can be used
together. Yes.
>> That you can buy, you know, you're a
buyer to ek out the most per chip.
>> And that those chips can then be coupled
in very small
>> data centers
>> to just do inference. And that those two
steps of a whole bunch of random
compute, some of which is cheaper than
it should be,
>> your ability to eat more out of it, and
then small units of expression in a data
center
>> equals way cheaper intelligence. I
certainly think so. Yes, there's a lot
of ways to access cheaper flops if
you're able to be creative with what you
take. And so, one of the ways that I
describe what we do is we will buy any
chip anywhere in the world for any
duration of time. That is a level of
flexibility and liquidity that I think
no one else has right now. Uh we're very
aggressive about putting our money where
our mouth is and we will we will really
take any capacity uh and find a way to
make it work in our fleet. And that is a
big part of our advantage today and long
term we had to create more of that
advantage by investing in these data
centers that other people are going to
be skeptical of because you know what's
going to happen when you set up these
like this army of a thousand small data
centers versus the one the one big
gigawatt data center. Well, few things.
You're not going to have power
redundancy frank quite often. You're not
going to have backup diesel generators
on site. Those are all very expensive.
We cut all that overhead. We're not even
going to have redundant networking in a
lot of cases. We're going to put these
in facilities where we have good access
to power, a single source of power and
we're going to trench one line of fiber
to these data centers, but we're not
going to have like three lines of fiber
with redundancy and failover and SLAs's.
It's just going to go down sometimes. In
fact, I won't be surprised if some of
them get down to like 95% uptime,
>> which is bad.
>> Very bad. That's fatal, atrocious for
anyone else. survive in a in a big a big
gig
>> you'd have basically zero buyers for a
data center that has 95% up time
>> I'm that first buyer I will buy 95%
>> up time
>> and the reason for that is because of
this background engine thing that if
there's things running in the background
you don't care
>> partially uh it's actually two things
one is that we have a really robust
control plane that is going to be fine
handling any single failure in any
single data center as long as it's not
correlated with other data centers and I
can just move the workload somewhere
else I'm cool with that um the failures
happen at some rate and I am basically
linearly happy with a data center that's
95% uptime versus 98% versus 99%. It's
just linearly good or bad for me.
>> Mhm.
>> Now, you do need that async piece that I
mentioned of, you know, we serve these
long horizon agents because what happens
when a request fails is that I'm going
to have to go find a new GPU to put that
request on. And that means that for that
single turn of the agent's work, you
know, it's working for an hour, but then
it hits a roadblock because it's GPU got
pulled away. At that moment in time,
that agent is going to experience maybe
like an extra minute or two or three,
maybe even 10 of latency. But my
argument is that my customers don't care
because their agent was running for
hours.
>> They're sleeping.
>> Doesn't matter. [laughter] It doesn't
matter if like a single turn
occasionally becomes uh a little bit
longer. Yeah. So we tell our customers,
look, our average throughput is going to
be very competitive, but our P99, our
99th percentile latency, it's not going
to be controlled. It cannot be. And in
return, I'll give you unbeatable
economics.
>> And I think that's the right fit for
background agents.
>> Talk about power as a category. What are
you seeing that's interesting,
innovative, where do you think this
goes?
>> Okay, so I said I want 95% uptime on my
on my data centers. Could I even take
80% up time at the right price?
Probably. Um, and what does that mean?
Well, I'm a son of California. I love
solar and wind. I think solar and wind
power is way undertapped in the United
States. And the challenge has always
been this intermittency. You would even
consider solar and wind unsuitable for
data centers because you have a
persistent base load and an intermittent
power source. What are you going to do?
Well, I think we're actually not that
far from solving that problem. I am
totally capable of tolerating a outage
for my data center that's measured in in
even days or weeks which is like the
worst case nightmare scenario for a data
center is that we're going to have a
long-term outage because the wind is in
blow and the clouds are in the sky fog
is hanging over the valley for some
time. That's the worst case scenario.
It's in fact highly predictable and I
can just call in capacity in some other
place of the world whenever that
happens. uh I'll just model the weather
and figure out when my data center is
going to be offline, move my data my
workload somewhere else and it's fine.
The trick is that it's going to give me
better access to power that no one else
is going to touch because it is so
annoying to deal with that kind of
outage.
>> And if my chips are cheap enough,
they're probably not going to be Nvidia
racks. And if my chips are cheap enough,
I don't mind the capital cost of having
idle chips.
>> Yeah. So I've heard you describe this
entire system as like a scavenger
strategy.
>> That's right.
>> Is is that Yeah. Unpack that analogy a
little Well, first we scavenge chips and
then we scavenge power for those chips.
The idea is in both cases I do not want
to be in bidding against Anthropic or
Open AI for compute capacity. I'm not
going to win against them and I don't
want to. I want to be more creative and
use the supply that they don't find
legible today. And over time I amass
enough aggregate supply. I'm never going
to get concentrated supply. I will only
get aggregate supply. And over time I
build my aggregate factory that is
unbeatable in economics.
>> We are building a factory. We're trying
to build the best steel factory in the
world. Uh but it will come through mini
mills not through large
monolithic steel plants.
>> And and if I imagine the different
versions of this like how vertically
integrated you can be.
>> Yeah.
>> One version would be the extreme would
be you own everything. So that it's a
very capital inensive business. You own
the power you know source. You build the
data centers. You design your own chips.
You control the software that eats the
most out of those chips. And you sell
the end finish token
>> to your user like your user is me
>> and you just own the whole stack.
>> But you can imagine many other
permutations of the business
>> where you know you could whatever you
draw the line anywhere you could be
incredibly capital light own nothing and
just be like the coordination plane
>> across all this stuff the virtual
scavenger
>> right
>> how do you think about that question of
like which which type of these
businesses to be
>> you know there's actually two parts of
me to receive that question. One is the
CEO of a company that needs to work
every day and and grow as sustainable
and and as quickly as it possibly can.
The other is the founder and the founder
is much more imaginative and and just
loves this stuff. The founder in me
wants to do everything. This is my
entire life. I spent my entire life
thinking about chips, power, energy.
Like all I care about is this stuff. So
of course I want to be maximally
ambitious. I don't want to stop ever. I
will never stop until I have built the
most efficient system from soup to nuts.
>> You're doing real life factorial
basically.
>> Very much so.
very much so. So that's like the
emotional from the hard answer. On the
CEO side, I think
>> we have to be more pragmatic. I think
that the capital we're we're looking at
for owning everything is like you said,
it's insane. Yeah, software has high
leverage, so we have to start with
software, but ultimately, you know, do
we own power generation or can we get
great power purchase agreements with uh
utilities? I'm more inclined to pursue
like letting other people specialize in
the things that they're historically
good at and then see if we can get to
the scale. I think of it as like I want
to get to the scale where I earn the
right to take this under our wing. I
absolutely think that there's
efficiencies to be gained everywhere in
the stack. If you can break the
assumption that people I would be buying
from, they made assumptions about who
their customers would be. And I maybe
break those assumptions. It's a pretty
optimistic view. Uh I think it's only
possible because we're actually trying
to underwrite the largest market for
compute in the history of computing.
we're actually going to build so many
billions, trillions of dollars of
investment into inference. Uh, and
because of that focus, it makes sense to
build a lot of things that are custom
for inference. And it's my job to seek
all the places where that's possible.
And then as as they become obvious to me
and my and my partners, I will get my
partners to build custom things for me.
And if they can't do it for me, I will
do it myself.
>> If you had to just zoom out on this
entire system, software, hardware,
energy, etc., and stackrank the places
that you think that we are the most
inefficient today at producing useful
intelligent tokens.
>> Yeah.
>> What does that list look like?
>> I think compute scaling is actually like
very efficient. Uh as in like you give
me more flops and I will use more flops.
And I would say we're actually fairly
judicious already with our use of flops.
Uh if you look at a modern model, there
are very few models that are more than
10% dense, meaning 10% of the possible
number of experts you can activate are
activated. And I think the frontier
models are closer to like 1%. So fairly
sparse already. I don't think that we're
wasting too much on the MOE side. People
have been working with for quite some
time. They're pretty good at squeezing.
Where we are not good is attention and
its use of memory. Specifically, the KV
cache is quite uncompressed right now. I
think if you look at the entropy in a KV
cache, it's nowhere near it's not
earning its keep. Like we're storing
many kilobytes of data in the KV cache
per token. Um, and that's probably off
by an order of magnitude or two. And I I
don't know what the Frontier Labs do,
but Deep Seek certainly publishes really
interesting work to compress that
further and further. And they're making
good progress. And I think the fact that
they're able to make order magnitude
progress here every year or so signals
that there's a lot more room to go. I
guess this all on the micro scale. If
you zoom out further, I think that we
actually don't marshall our compute
effectively at all. Like we have all
this compute in the world. Nvidia is
pumping out 5 million Blackwell chips
this year. Where are they all going? Are
they all being used at all all the time?
I certainly doubt it. I think that at
some level we just need better
orchestration of compute across the
world. Uh this is very difficult to do
because a lot of the comput disappears
into private pools of compute that will
never see the light of day and those
GPUs sit very sadly idle. Uh it's
actually it pains me physically to see
that those GPUs are just you know
silicon and power went into that and
it's just sitting idle and I want to fix
that. how we or organize and orchestrate
the world's compute as a shared resource
and and pack it more efficiently. I
would estimate that, you know, we all
make fun of XAI for having, you know,
some challenges with total flop
utilization on its clusters, but um the
reality for the rest of the world is
it's far far worse. A ton of GPUs just
sit in warehouses or sit in private
pools allocated to a specific customer
um just don't get utilized.
>> You're attacking the efficiency of that
very directly.
>> That's way more effective. Yeah. Um,
what about fabs? Like what do you think
is the future of fabs themselves? Like I
think everyone is wondering
>> will the memory companies will TSMC will
Intel and others
>> be able to how will they expand capacity
basically?
>> Yeah.
>> Will we do it here in the US?
>> Um, yeah. Riffon fabrication of chips
themselves. Like if we could just snap
our fingers and have 100 times the
chips, you know, in the stock today, uh,
we'd probably have way cheaper way
cheaper tokens. So yeah, that that seems
like an important part of the universe
to hear your view on.
>> Yeah. Well, it's interesting. Everything
grows in balance with each other, right?
If we snap our fingers and double all
those things, you might fix a TSMC
bottleneck that there you're just going
to run into another bottleneck. You make
20% more chips, then you have another
bottleneck immediately. I will say
though, it is interesting what they
consider to be a mustd deliver. uh like
what they consider to be like an
invariant that their customers me are
always going to want versus what I think
of as like a more fluid relationship. I
think that if a fab exposes more of
their trade-offs to me, I'm able to make
more intelligent decisions about what I
think I can I can do.
>> One of the most interesting examples
here is that any fab has a lot of spread
in their like worst chip that comes out
of the production line and the best chip
that comes out of the production line.
There's a lot of variance in how chips
are made. Uh and then the question is
like you know if you have a company like
TSMC they work very very hard to tighten
what we call these process corners. We
want to keep the worst chip as close in
characterization to the best chip and
they get a great lens to make that
possible. But that means that they are
adding a lot of controls in the process
that maybe I don't need. Maybe I'm
actually willing to find a place for
that worst chip. You don't need to
tighten the process control as much
which takes more time and cost. Uh maybe
I'm willing to take a lot more rejects.
And I think for us it's like a more
holistic optimization around there's you
know cost of the dies, supply of the
dies and then the cost of power and
places we can put them. And my whole
goal is to actually so dramatically
expand the supply of of power uh across
the United States that I have a home for
a lot of chips that otherwise would not
have earned earned their place in a data
center.
>> Can we talk about how you designed the
system of your own business?
>> What lessons have you learned? You
talked about some of interesting Nvidia
lessons. Yeah.
>> But like bring me into the culture and
how you structure a team and a business
where this is the northstar.
>> I think there's a lot of um you know in
the limit thinking we don't worry about
the immediate nature of like when we
start working on a model the efficiency
is not going to be very good. Uh but we
we we think about like where we could
end up in in like a month or or six
months or a year's time. We don't accept
the state of the of the machines we work
on as fixed like even something like the
blackwell chip if we think that there's
some bottleneck that is holding us back
from achieving this performance. I mean
it's very important to me that we we
understand and characterize that very
well and write it down so we can both a
tell Nvidia about it or friends and also
to basically keep this in mind for
future chips that we buy. We want to
learn things that are what we think are
um essentially like invariant for us or
the company uh long term and and kind of
fold that into future decisions that we
make. We're very collaborative. I think
one of the most important traits that we
look for are people who either who are
both good students and great teachers.
Um a lot of our people on the team were
TAs in college and and loved the
experience of of sharing knowledge in
this way. uh we we do whiteboard
sessions all the time and I think the
collegial environment where everyone has
something to teach and something to
learn is is extremely important for us.
What are the attributes of people that
you would want to hire that you think
will be resilient to you know the work
environment 3 years from now when more
stuff is handled by machines?
>> Curiosity. It's 100% curiosity. You know
the one thing I cannot teach is love for
performance, love for uh digging into
every microscond that the machine is
working and understanding what's
happening on the machine at that time.
That to me is the most important trait
for a performance engineer and it's what
I look for. I don't look for lots of AI
experience. I don't look for, you know,
CUDA experience at all. That's actually
a huge red herring. I mean, CUDA as a
concept or GP as a concept have evolved
so much in the last 5 years. There's no
point asking for 10 years of experience.
I want to teach that, but I cannot teach
the love for performance engineering.
That is what I seek.
>> Can you give your assessment of the
major labs
one by one, but also then the
relationship of like closed source as a
category to open source and like what
you think is happening and will happen
>> in a line. I would say the labs pay an
immense premium to be 3 to 6 months
ahead of of everything else. Uh and I
think that's probably still worth it. I
think it makes perfect sense for open
and anthropic to do what they do. You
know there's a sensitive topic around
distillation which I think is part a
very core piece of the relationship
between closed and open frontier. And
you know I'd like to offer an
alternative view on that which is there
is the sense that distillation is theft
that you are taking something from the
frontier models when you distill on
their outputs. And in fact, even if
that's not your intent, even if you
don't ever try to, you know, scrape data
from anthropic, one thing I'll offer is
that an increasingly large percentage of
the artifacts we put out on the internet
are AI generated. Even if you just look
at GitHub alone, you know, what
percentage of repos created in the last
year do we think were created by cloud
code? Um, do we consider that to be
distillation? Because that's probably
all we need. I would not be surprised if
you could train a fable glass model only
on the outputs of code you consider good
on GitHub that's open source. And
certainly if we take the position that
users own the outputs of their
interaction with AI and they choose to
put that up on GitHub, which a lot of
them do, we're going to have latent
distillation for a long time. It seems
fundamentally impossible for me. Like I
I don't think it's fundamentally
possible to prevent the diffusion of of
information or model capabilities. It
will happen. The question is just how
fast. And so then the question becomes,
do scaling and improvement laws hold
forever or for a really long period of
time? And if they do, then there's value
to being three and six months ahead and
that will just last as long as it lasts
and they can charge a huge premium for
those tokens relative to a very cheap
open source token. Is that the right way
to think about it?
>> I think it's possible. I don't know that
the premium for being 3 to six months
ahead is going to last that long. I
mean, if you look at like enterprise
deployments, uh, they don't move at 3 to
six month speed. A lot of enterprises
are probably still on like 46, Opus 46
or Opus 47. They don't they don't adopt
the bleeding edge rapidly. There's a lot
of questions that people have around
rolling out any change at all. And I
think we're just so early in scratching
the surface that um I don't think
there's any way to call a winner in this
race and certainly I don't even think
this is a race that can be decided ever.
There's always it's a continual process
and fundamentally I don't think open
source ever goes away. If there's a
vacuum because one leader steps out, a
new leader will step in. There's too
much incentive and too much there's a
lot of tailwinds too. It's just it gets
easier every day to treat to train a
frontier class model.
>> And so your hope of what the future
looks like is what like what balance
between closed and open, you know, what
balance between model companies doing
everything because they have the
advantage of owning the stack or
whatever. You know, Enthropic can do
that, you know, is like the new Google
could Google just do that or something.
What do you hope the future looks like?
>> I want abundant tokens and diverse
harnesses. I want everyone to build
their own harness and and
>> every company
>> every company every user even make the
agent your your own. Uh I think we're
we're very not that far away from that
level of customization and capability. I
want people to own their intelligence
and I want that intelligence to be
customized probably not through weight
fine-tuning but probably through more in
context learning. That's a more
technical detail. But the underlying
input to this abundance future is about
is basically cheap tokens. My job is to
make the tokens as cheap as humanly
possible. I will achieve that and I will
do it through every layer in the stack
available to me. I love the supply side
levers. I will use every chip. I'll use
every source of power and I will use
every piece of land in the United States
that's you know suitable for this. And
in return, people will have the
incentive to explore what it's like to
have abundant intelligence. We still
treat the agent as a person that is
expensive to consult and you should ask
them when you have a hard question.
That's not the way to think about
intelligence. It's incredible that the
machine can think and we should try to
get that into as many hands as as many
people as possible.
>> You sit in such a unique seat and you
have such a unique perspective on like
what you're trying to do to make this
feature a reality. What do you think are
your most like divergent views of the
world versus your friends who are really
well informed and interested in this
stuff? Like what what make your what
ideas of yours make your friends look at
you like you have three heads?
>> Most of the ideas on chips, I would say.
You know, when I talk about building
custom chips and they ask me, "Oh, so
what's different?" Basically, it's it's
about sidestepping the HPM shortage and
focusing on more extreme offload to
other forms of memory such as flash. Um,
I'm quite passionate about that idea.
Everyone on my team knows that I keep
banging the drum around like what would
we have to change about the model
architecture to make offloading KB cache
to flash work at a much greater level.
And um I'm whiteboarding that all the
time. That's like in the community of
like inference people. you know we have
some divergent views on what you can do
if you design a system around serving at
you know one to 10 tokens per second
which is our whole north star more
broadly I think there is this like
larger sense around you know what do you
do how do people consume a trillion
tokens per day like that's the world we
want to create the capability for them
to do that
>> what's a trillion tokens like ground us
in how much that is
>> a trillion tokens well okay at openi
pricing that's at least $5 million at
the very least for 5.5 or 5.6 six. Yeah,
I think the dollars was probably the
most good metric. Yeah.
>> Yeah. It's millions of dollars.
>> Yeah. So, what's the world in which we
consume what currently costs $5 million
per person per day?
>> Yeah. I mean, we were asking for at
least at least um you know, three to six
orders of magnitude improvement in cost
per token. Uh get that into 5,000. You
probably have some customers. And in
fact, I would argue that we're for some
size of model, we are approaching a
trillion tokens being measured in, you
know, tens of thousands of dollars. And
that's something that you can imagine
running for a single job.
>> Are you at all worried that just like
the average person just can't and won't
do that like doesn't do that now with
their own brain? Like there actually
isn't that much demand for intelligence
in the world.
>> I never will believe in that. There is
always demand for intelligence in the
world. I think that the way in the
on-ramps to that intelligence are our
challenge as a product u you know
community. I'm not a product person so I
cannot say I had the best vision.
>> You want to enable those people.
>> I want to enable those people. I want
them to never be held back by the sense
that, oh, I my free tier users cannot
use or I can't afford to give them this
many tokens. And I hear that from my
customers all the time. Um, we want to
fix that.
>> What about the inverse question? Not
what you think is craziest, but like
what consensus thing you think is wrong?
>> One of the things I keep coming back to
is this question of Nvidia. I am bullish
on Nvidia in the short term. And you
know, Nvidia, you should never bet
against them. They're always going to
reinvent themselves. But like
fundamentally I think one thing that
surprises people is when I tell them
that hey if you look at you know Hopper
to Blackwell to Reuben and you compare
like for like like what is the
performance per watt of Bloat 16
multiply it hasn't improved all that
much or or even you take that one step
further go to TSMC if you look at TSMC 5
nanometer versus four versus three
versus two the performance per watt on
these chips doesn't change like a
dramatic amount
>> so the consequence of this is people
lose their minds over geopolitics like
what what happen if we lost access to
DMC for any reason. And um my contrarian
take is that it wouldn't be that bad.
Supply would take a shock for sure, but
the best processes that we have in the
west uh like Intel not that far behind
at worst like maybe 2x uh worse
performance per watt and the gap is just
far smaller than than you would make it
out to be if you talk if you follow like
the chipboard dialogue. What else is
happening in the AI world that is not in
your path? Meaning it's not like a
component of this whole system that you
would end up doing something in that
interests you most.
>> Well, we're fully downstream of models,
right? So the model people get to decide
how to design their their architectures
and I have only like very light I mean I
don't have any input to open AAI or
anthropic but um I can only pray that
they go in the direction that is a
minimal to me and the ch like or I have
to like do my best to predict where I
think they're going to go and build my
serving architecture accordingly. Both
software and hardware choices they have.
I think the most interesting game in
some ways to play like once again this
is going back to like the profoundity of
the machine thinking and how
consequential it is to decide to use
something like sparse attention versus
dense attention or um how consequential
it is to like use a different data type
like we were training in B16 but now we
can train in FP8 or FP4 lower precision
data types that is just an arbitrary
choice it feels like but it has profound
implications for what chips I can use
and and you know how I should build my
hardware think about the future of
compute
>> if you had a 100 entrepreneurs in a
room, all of whom wanted to create some
new compute startup.
>> Y
>> um and let's say they were specifically
wanted to make hardware chips or systems
or racks or whatever.
>> What advice would you give them on like
how to orient their companies or like
the type of company, not the specific
choice they're making on a tech tech bed
or something like this
>> because it seems like we're going to try
everything and that will be great for
the world. You know, some stuff will
work. But if you had to give them advice
on how to orient their business to be
successful in this coming world, what
advice would you give them? It's all
about the bottlenecks on supply chain.
So, you need to first convince me or
convince an investor that you understand
the like three to five bottlenecks that
dictate modern chip supply. There's TSMC
wafer capacity, there's HPM capacity,
and there's um like advanced packaging,
and maybe a fourth one would be power.
Like, where will you get the power? How
will you build these racks? Uh and I I
want to hear like you should have a
great answer to each of those four
bottlenecks and how you're going to work
around them because it's all arbitrage
at the end of the day. You're building a
chip because you think that Nvidia has
made some choices that are difficult for
them to change, which is true. Nvidia
makes a lot of choices that are
difficult for them to change. They're
not perfect. They're just really well
balanced. And so, you want to be spiky.
You want to pick something and say, I
think they've underpriced the impact of
how short we're going to be on HBM.
We're going to push really hard in this
other direction instead. Which, you
know, as a as an aside, I do think is
probably the thing to attack most.
>> Why? There's no easy way to bring on a
lot more fabs of memory and those guys
have been
>> so it's going to be a while until we
>> Yeah. Yeah. The boys in Boise don't uh
don't love huge capex for for cyclical.
>> They they've been burned on that many
times.
>> But conceivably like because of that
shortage, the world is just going to
route around it by making everything
else in the system more efficient.
>> I think they're gonna make everything
else more expensive. Think that iPhones
will cut their memory. iPhones are going
to go up in price and um we're just
going to deal with it. Why doesn't
Nvidia go all the way to the end and
sell tokens? Do you think
>> Nvidia is really smart about this? They
don't compete with their customers.
Nvidia takes the long view on
everything. Um, why don't they even
start with the Neocloud? Why don't they
just sell computer out the back door?
Well, Nvidia is really good. Jensen is
really good at making his friends
billionaires. He's made Cororeweave a
billion dollar company, many billion
dollar company. And there's no need for
him to kind of uh destroy that goodwill.
like he wants to create a diverse
community of NeoClouds and inference
providers who are all jockeying to
create demand for Nvidia such that if
any one of them decides to I don't know
vertically integrate or go with AMD or
any other option he's got three more
people ready to hungry to fill that
position.
>> It's great to have competition amongst
his buyers.
>> My favorite closing question for
everyone is what is the kindest thing
that anyone's ever done for you?
>> The kindest thing I mean I my immediate
first thought is like all the mentors
that I've had over the years. It's a
rare person who takes a lot of time out
of their their schedule and um and you
know makes it like their personal
interest essentially to to make sure
that you understand something that uh or
teach you something or or like ingrain
some value in you that they think that
you're on the cusp of understanding but
just push you over the line for
understanding. uh a lot of the people in
Nvidia that I mentioned earlier who
instilled that like love of performance
engineering in me but also my professors
in college who I remember like my
adviser in like sophomore year I was
very impatient student so I show up at
his office hours and say like I I want
to build AI chips I know what I want to
do why am I wasting time taking all
these like other basic classes and
networking and you know operating
systems and he just looked at me and
said like you know he laid out basically
like the whole stack and showed me the
depth of or the beauty of like
understanding every piece in the puzzle
like he he took my entire path of like
trying to focus on one piece of the the
system and said that you know it's so
rare that someone can actually
understand the entire stack from the
gate level silicon all the way to
building a great internet scale service
and you know you should aspire to be
someone who over the course of your
lifetime achieves that level of
understanding.
>> It is such a rare rare trait and um you
know it that level of expertise is so
noble to chase and I think and that
stays with me quite a bit. Not a common
but an advisory. Neil, amazing
conversation. Thanks so [music] much for
your time.
>> Thank you so much for having me.
>> You know how small advantages compound
over time? That's true in investing and
just as true in how you run your
company. [music] Your spending system is
your capital allocation strategy. Ramp
makes it smarter by default. Better
data, better decisions, better economics
over time. See how at ramp.com/invest.
As your business grows, Vanta scales
with you, automating compliance and
giving you a single source of truth for
security and risk. Learn more at
vanta.com/invest. [music]
The best AI and software companies from
OpenAI to cursor to perplexity. Use work
OS to become enterprise ready overnight,
not in months. Visit works.com [music]
to skip the unglamorous infrastructure
work and focus on your product.
Ridgeline is redefining asset management
technology as a true partner, not just a
software vendor. They've helped firms 5x
and scale, enabling faster growth,
smarter operations, and [music] a
competitive edge. Visit ridgeland.ai to
see what they can unlock for you.
Continue with YouTLDR
Analyze another video with Pro
Process a new video, search every timestamp, compare sources, and keep the result in your library.
More transcripts
Explore other videos transcribed with YouTLDR.
DuckDB-Quack announcement at AI Council
DuckDB · English
What Are Agent Plugins, Are We Ready for AI Employees & OpenClaw Hacks a Gym | This Week In AI
Mastra · English
Git Wasn't Built for Agents, So Mesa Rebuilt the File System | Agents Hour
Mastra · English
2º Leilão Nelore Sales & Sales
erural · Portuguese (Portugal, Brazil)
Leilão Virtual Cachoeira 2C e Fazenda & Haras Retiro Velho
LANCE RURAL OFICIAL · Portuguese (Portugal, Brazil)
Leilão Virtual Qualidade Corona
LANCE RURAL OFICIAL · Portuguese (Portugal, Brazil)
Simon & Garfunkel - The Sound of Silence (from The Concert in Central Park)
SimonGarfunkelVEVO · English
Why Everyone Wants You To Believe AI is a Bubble
GEN · English
Outsourcing the Muse? AI Creativity vs. Human Creativity | Boris Eldagsen | TEDxESMTBerlin
TEDx Talks · English
ثورة الذكاء الاصطناعي ومستقبل التفكير البشري
الموسوعة الفلسفية · Arabic
استخدام الذكاء الاصطناعي
الموسوعة الفلسفية · Arabic
6° Leilão Virtual Terra Prometida
LANCE RURAL OFICIAL · Portuguese (Portugal, Brazil)