How I Interview AI Engineers (And the Projects That'd Impress Me)

All right, let me set the context of the

video. This video we're going to go

through six seven projects in applied AI

specifically related to agents that you

should probably build if you're trying

to target companies like, you know, I

don't know, like an emergent any India

based or US based Genie AI startup. For

the last one month we've done a bunch of

offline classes on applied AI on topics

like context engineering, writing

emails, building an agent from scratch.

And it's becoming very obvious

there are more companies right now in

this specific niche and almost all of

them are going to ask the standard set

of questions. So, in this video I'm

going to take you through how I am

interviewing someone here at Super 30

for a recent company that's raised $15

million. What are the kind of topics

that people are learning over here and

around seven eight projects that I feel

if you build from scratch and really

well, then intuitively it'll be easier

for you to crack these interviews.

Without any further ado, let's get right

into it. Here are five high-level topics

that we've covered in the last one month

at Super 30. Whichever one gets the most

up votes in the comments is the one that

we'll release on the channel as well.

Specifically, these include memory as in

how does your agent remember who you

are, the things that it has done in the

past and how does it learn from it.

Firecracker, which is a virtualization

technology that Lambda uses as well. If

you've ever used AWS Lambda for spinning

up sandboxes, if you've heard of E2B,

E2B also uses Firecracker under the

hood. And this class covers how you can

create your own Firecracker clusters

from scratch. Context engineering, this

one's taken by me. It specifically

covers what techniques do you use as

your context grows

to make sure you don't get context rot,

to make sure you're not spending a lot

of money doing a specific task using an

agent and to make sure you're able to

manage the context of the agent. If the

model that you're using is something

like Opus 4.8 that has a context window

of 1 million, what do you do when your

long running tasks reaches around 700

800k tokens? How do you summarize

context? How do you compact context?

Evals and RL environments, how do you

write tests or evals for your agents?

How do you make sure they're actually

improving over time? How do you know if

model one is better than model two? How

do you build something like sweep ends

from scratch? And what do you do if you

join a data lab where your job is to

create RL environments or data that is

used to train state-of-the-art models.

And lastly, a video on cloud agents. How

does something like Devin work or how

does remote control in cloud go to work?

Slightly more of a full stack question.

This is a set of classes on the last one

month at Super 30. Let us know in the

comments which one you'd like on

YouTube. With that, let's move to the

interview process. Um, recently a

company's hiring through us. They've

raised $15 million recently. They're

building some sort of coding agent. And

before we refer someone there, I wanted

to interview them personally. So, this

is the set of questions that I have

prepared for them. I have to take this

interview today. Um, and the question is

pretty simple. If you're building

something like Devin where you have a

screen like this, a user comes, they

they're asked to, you know, fix a

specific issue on their GitHub

repository, and the agent starts over

here. You're able to see the timeline of

everything that happened, including the

agent starting a terminal, the agent

editing a file, the agent building some

process, the agent reading files,

writing files, etc.

Considering you have to build a system

like this, here are a few interview

questions that I thought of. Number one,

um, what all services do you need, which

is like the simplest question here. What

would the architecture look like? How

many backends do you need? How many

frontends do you need? What does the

infra layer look like? How does the

infra layer scale up if you have 100

people at the same time, you know,

talking to the agent? Um, probably need

100 sandboxes running, um, where the

agent is running, where the task is

being fixed, where the final commit to

GitHub happens from. How do you spin

them up? How do you spin them down?

That's the only infra-heavy challenge

over here. Other than that, you're

probably going to depend on an OpenAI

for the LLM calls. You don't have to

scale that up. Frontend and backend is

going to be fairly simple. So, the real

crux of what we're trying to get at is

does the person understand, uh, for a

project like this, you need sandboxes,

and how do you scale them up and scale

them down? Number two, what happens if

your server that's running your agent

crashes? This is an interesting one

because,

um,

not usually in full-stack apps do you

see long-running tasks, but

agents, and specifically, you know,

coding agents that are doing a very

heavy task can run for 30 minutes to

hours, a really long time. Um, so, what

happens in case your infrastructure does

go down? How do you resume a

conversation, um, or resume something

that the agent was doing from a specific

checkpoint?

There are many answers here, you know,

you basically need durability across

sessions. If a sandbox where the agent

is running, or if the backend where the

agent is running crashes, um,

whenever it restarts, or, you know, some

other part of your infrastructure needs

to start to pick up from there uh,

and continue the conversation with the

agent. Good thing about agents is

there's a lot of back and forth that

happens between

your agent and the LLM and hence if at

some point the agent does crash, if

you've backed up the message history

between the agent and the LLM,

you can always resume the conversation

from a different part of your

infrastructure. Um

Good read at this point is Temporal. If

you've heard of what Temporal does and

you know how they provide durability for

use cases like these.

And if you talk about that during the

interview, it'll probably be a good

sign. Number three, how do you do

context management? There are 100

strategies over here. Um

I think most newer companies would not

really reach a scale where they need

context management or context

engineering at all. But nonetheless,

every new company like Devin that is

coming or you know if you're building a

coding agent yourself,

the real thing to optimize for is what

happens when the conversation becomes

really long. For a shorter conversation,

most models perform fairly well. It is

only when you're giving it a very big

feature or a very big task or an issue

that's very gnarly is when the

conversation is going to be really long.

It's going to be a long-running task and

hence compacting the conversation from

time to time or summarizing it. If

compaction isn't working, um are like

standard techniques that you need to

know of. Here is a video by the folks at

Manifold where they talk through what

they do and what Claude does for context

engineering. Number four, how do you

evaluate your agents? This is probably

something you've never thought of in a

side project, right? If you're building

something like a Lovable, um

you never really think of is your model

getting better? Is your agent getting

better over time? How do you know if

your agent is actually improving? Before

you do a new release of an agent or if

you introduce a new model, how do you

objectively, not subjectively,

objectively know that the output has

gotten better? There are a bunch of ways

to do it. Um slightly harder for a use

case like Lovable, but for you know

other use cases something like a Devin,

it's not too hard. There are standard

ways that companies use right now to

write evals. If you've seen SweBench or

the newer benchmarks that have come now,

they have a deterministic way of telling

if a model or a model harness is

performing good or bad, you know.

And is it improving over time? I think

every company in some form or fashion

eventually is going to write their own

evals on their data to see if, you know,

as new model releases are coming, are

their engineers becoming more efficient?

Whichever parts their app AI is using,

um is it getting better over time or

not? There are again a bunch of

techniques over here. I urge you to go

through the paper of SweepBench or, you

know, look at the TerminalBench and what

they do, how do they work? I think this

is like a very niche skill right now. Um

but eventually this is like what a lot

of us will be doing, um writing a lot of

human tests RL environments to make sure

our agents are getting better over time

and if not, improving them. And lastly,

what metrics do you observe as your

application grows? So, the thing is we

already had observability before AI,

right? where we had platforms like New

Relic, Data Dog, um Prometheus, Grafana,

um various stacks to make sure your

application is running. Um LLMs are

slightly different that way. There There

are different set of challenges if

you're building agents. How do you

observe them? Um Specifically, things

like is the agent failing at some point?

If yes, what went wrong? Um Are your

users happy or not happy with the final

output? Be able to arrive at the final

output? Seeing the actual reasoning

traces of what led to the model

actually, you know, giving a specific

output. And now there are like a bunch

of companies that solve specifically for

LLM observability and all the older

companies like Data Dog are also coming

into this market. One specific company

that solves observability for agents or

LLMs is

And I feel companies that have hit scale

need to worry about this, you know, a

little too much. Just like a company, a

web two company that hit scale, um at

some point needs to worry about

observability and monitoring and making

sure alarm bells ring if, uh

you know, the back end is down.

Something similar over here. If your

agent is suddenly not performing if a

specific, uh infra provider is down for

whatever reason, you know, lack of GPU

right now. You should auto switch to a

different one. You should know what went

wrong. you should have visibility as to,

you know, what caused the agent to fail.

That's like a high level list of

interview questions that I had for

someone who's specifically interviewing

for a company like Devin. Irrespective

though, I think a lot of these questions

can be applied to other companies as

well. Any company that's using AI or,

you know, building some sort of an

agent.

These are like good set of interview

questions for users because I think

today everyone understands what an agent

is, what a model is, how do they work

together. The real question is, you

know, you understand how can you improve

the model, how can you improve the

agent, how can you observe the agent? If

at all, you need every agent to have its

own sandbox, how do you scale that? With

that, I'm going to move to a bunch of

projects. Some of these projects we're

building in Super Trendy as well. Number

one, simplest one, terminal agents. Um

and wait, building a terminal agent is

very easy. If you've seen the code base

of Pi, uh

which is this thing right here, it's

very readable. If you understand

TypeScript, you can read this code base.

It's probably less than, you know, 2,000

lines of code all in all. Um

So, building one is easy. Making sure

your terminal agent is performing at par

with Claude code uh using the same model

is the harder bit. So, what I'd really

urge you to do, and you know, I urge

everyone here to do as well, is not just

build this because that's the easy part.

Actually, benchmark it against other

agents and see if it's performing better

or worse. And if it is performing worse,

uh which it probably will, what

techniques can you use to rank up over

that? You know, there could be a bunch

of extra hacks that you could try for

now to just reach up the leaderboard of

Terminal Bench. You know, hardcode some

use cases, for example, you know, if

you're ever giving a Codeforces contest,

you a lot of times

hardcode a few use cases just so you

can, you know, pass more tests. You

know, it's not the final solution, but

you know, you you'll be able to pass

more tests if you add a bunch of edge

cases. Something similar. Just to see

what might be the right direction to

take to improve a model. A good example

of this is Pi by default does not have

sub-agent orchestration. Claude code

does. Uh

Codex does. Yet, Pi performs, you know,

almost equally well uh on the Terminal

Bench. That is one of many techniques

that might nudge the agent in the right

direction and make the agent arrive at

the final answer. And when you do that

is when you have a good terminal agent.

Project number two, Hermes Claude bot.

Both of them are again agents. Uh they

require a little more than building a

simple terminal agent. Uh they require a

lot of memory. They require a lot of

integrations to things like WhatsApp,

Telegram, Slack. Um

At some point, I would consider this to

be a little bit of a full-stack

challenge. Um it's actually just

connecting with a bunch of services and

letting the AI go free and take

autonomous decisions over time. The good

thing about most projects here is that

they're all open source. Um So, if you

do have AI, you can actually understand

their architecture fairly well. That's

how I've understood, you know, a bunch

of these architectures. It's harder to

dive into the code base of these now

because a lot of these are like by coded

and very verbose. But what's easier is

to make an LM summarize what the project

does and go from there. By is probably

the only open source agent that I was

actually able to read the code in one go

and understand.

Number three, Slack plus AI. This is a

new set of companies that are coming.

I don't know if you guys know Jack

Dorsey who's the, you know, ex-founder

of Twitter and now the founder of Block,

the famous company that laid off 40% of

its folks.

They recently released this project

called Buzz, again open source and then

you might have heard of Prompt QL. A

batchmate of mine works there now which

is why I realized, you know, they exist

as a company and it's a little bit of a

speculative use case but one that might

become very big. It's basically a

workspace, a more AI native workspace

similar to Slack. In Slack you only

have, you know, humans as members. Here

you can have agents as members.

You can tag agents and then they'll do

go their own thing. You can add, you

know, different set of agents. One could

be a front-end expertise agent, one

could be a essay writing agent so on and

so forth. You can tag them, they'll do

their thing. You can have a more fluid

conversation on an interface like Slack

compared to, you know, orchestrating

agents through a terminal. The code base

of Buzz is in Rust. So if you're

interested in Rust at all,

this is like simple Rust that you can

read. If you do want to contribute Rust

yet,

contribute to, you know, an AI project,

might be one of the good projects to

look at. Super set or T3 Code. I don't

know if you've seen these. T3 Code is by

Theo, another YouTuber.

It's basically a desktop app or a mobile

app where you can spawn agents, um,

coding agents. But the good thing about

this is you can actually select

different coding harnesses

for specific tasks. For example, in a

single UI I can ask Claude Code to solve

issue number one, Codex to solve issue

number two, Devin to solve issue number

three.

This is, I mean, primarily useful for if

you have subscriptions to all three and

you need a single interface to talk to

all of three of them. The interesting

thing there is how the architecture of

Super set is very different from the

architecture of T3 Code. If you use

Super set, it actually spawns the Devin

CLI, the Claude Code CLI, you know, in

various tabs and you sort of use them

versus T3 code

actually starts the underlying agent um

and hacks into the agent and you know,

the messages that you see are all T3 UI

coded compared to using the actual

terminal output from Claude code. There

are pros and cons to both of these

approaches. Basically, they have very

different architectures. The reason I

know this is because the next video is

going to be building something like this

from scratch. And the T3 code

architecture is slightly better than the

other one for many reasons, you know,

I'll get to that in the next video.

Both of them are open source. Super set

is a YC backed company. T3.code is doing

really well. I think it's growing really

well in terms of number of users.

Primarily again, built on top of coding

agents. T3.code will still teach you a

few things on, you know,

agents and what are the kind of messages

that Claude code provides you or the

Claude code agent, you know, what is the

kind of back and forth that happens

between the Claude code agent and the

LLM.

That is something you can learn if

you're building it the T3 code way. If

you're building it the Super set way,

it's mostly a full stack challenge, not

really a, you know, an agent or any AI

challenge at all. This is a project I

saw yesterday. I don't think this will

ever pick up steam. I think this is like

not the right way to learn, at least for

me. Um what it does is, if you want to

learn anything, for example, you know,

full stack, you can ask it to create a

learning track for you.

And then the nice thing that I mean, the

thing that I like generally is

generative UIs, it basically creates a

course like this. For example, I asked

it yesterday, you know, I not need to

learn attention is all you need or, you

know,

the attention or the transformer

architecture. This generated this course

for me, which, you know, looks very

close to if you in in 2020 wanted to

create a course like this, you would

have to go through a lot of effort to

gather all this data, write all of these

things down versus it was able to create

these slides in one go. So, it created a

structure like this. If I click on, you

know, one of these, I can actually look

at a bunch of slides. I can talk to the

agent over here and you know, it'll

guide me through it, which is and maybe

at some point we sort of work like this

or, you know, we learn like this. I

think right now it's a little I would

love to wait have a personalized

learning path for me. Problem is in I

personally feel for me, I need like a

lot of accountability, a human sort of

guiding me, which is why I tried this

yesterday and you know, did not really

go through with it.

Nonetheless, generative UI is generally

a cool you know, how do you make the

user build their own UI? Compared to you

know, static set of courses that exist

that you can buy. So, unsure if it's

like a great product or will it ever get

PMF?

Great technical thing to build.

Generally generative UI you know, you

can also build something like a AI based

slides plus mentimeter. I think that's

also like a nice project. Come over

there with a project an admin can come

to the platform, select a topic for

example, transformers and attention

followed by a 10 question or 20 question

quiz

that they can distribute to a bunch of

users and then you know, it's basically

a bunch of slides for a real class

followed by a bunch of MCQ, something

like that. Cool. Model router. This one

is interesting, new, slightly nuanced

and you know, probably there's no right

or right wrong approach for this right

now.

What is happening right now is in a lot

of companies, you know, the cloud costs

are really higher. I don't know if you

guys saw the Uber announcement, you

know, Uber turned down AI because people

were just burning through the credits.

And this is like a necessary evil, this

will happen when you

have a very big company and you give

engineers access to infinite credits.

They're probably not going to use it

very efficiently. Hence, it probably

will make sense at some point that all

companies give their developers access

to something like cloud code, but the

underlying model is decided by the

company based on the complexity of the

task. So, you there will be some sort of

a model router and when I say model

router, I don't mean something like open

router. I literally mean a router that

is smart enough to know whether the

incoming request requires fable or will

it work with GPT 5.5 and or will it work

with GPT 4 or will it work better with

open source 0.8.

A layer that is aware around which model

the request should be routed to based on

the complexity of the task, based on how

much credits does this user have

and maybe even block requests that are

you know, personal requests for the

user. If there's a developer that a

company has given cloud code access to,

they should not be using it for their

personal projects. So, you know, things

like these. A model router that is aware

of the incoming questions or requests

and is either able to block or route the

request to the right model. As I said,

speculative up in the air sort of a

thing.

No easy way to solve this right now. So,

whoever does solve this is going to save

companies a lot of money and make a lot

of money in the process. How do you

solve it is still a big question mark.

There are a few techniques that a few

companies are trying. Start thinking

from first principles. There are a few

approaches that I have in my mind as

well. Not fully a full-stack challenge,

not fully an AI challenge, something in

the middle. And lastly, benchmarking for

a specific use case. I think there's a

lot you can learn if you're building

benchmarks like three events.

It's not easy writing these e-vals.

Yesterday, we had a cohort class where

it was supposed to be a 2-hour class but

it extended to 3 hours because we were

trying to build e-vals for dub.sh. And

in those 3 hours, we were barely able to

even start the tests locally. Um

that tells you something. If you ever

join one of these data labeling

companies, you will be given access to a

private code base that you have to write

a lot of issues, tests, and e-vals for.

Um

Setting up the code base itself was such

a challenge yesterday that I realized,

you know, writing a thousand e-vals for

it is going to be really hard. Making

sure they're actually representing the

ability of a model to perform well on

that code base is going to be hard. I

was telling this in the code yesterday

that, you know, up until last cohort,

you know, the cool thing was to set up

dub.sh locally. And now the cool thing

is not setting up locally because, you

know, for a fact, most of the issues in

dub.sh are going to be solved by an

agent or, you know, some sort of an AI.

Hence, the cool thing now is can you

build a set of e-vals that, you know,

tell which agent, which model is better

at solving issues in dub.sh rather than

solving them yourself. And then as meta

or dystopian as it sounds, it does

actually seem like the right direction.

If you write a lot of e-vals and data on

how issues are solved in dub.sh, for

example, then it is probably going to

solve issues better compared to a human.

I feel at some point companies and

engineers in companies are going to

spend time not just writing code

but actually writing a lot of these

e-vals that not just represent the

ability of a model

to solve issues in that code base but

also probably train the model better on

that specific repository. So, it

performs better on that repository in

solving issues in the long run. So,

try this out for one week. Like, pick a

random open-source project. And And

trust me, it's not going to be easy.

There's like a lot of nuance in writing

these emails

and writing the environments that

actually test a model using them. You

have one nice project to pick. As I

said, we have five videos. Let us know

which one would you like to see on the

channel next. I have seven projects. I'm

already building Superset, but if

there's a different one you want me

building on the channel, let me know.

With that, let's end it. I'll see you

guys in the next one. Bye-bye.