Full Walkthrough: Workflow for AI Coding — Matt Pocock

[music]

>> Yeah, we're good.

Okay, folks.

We're at capacity.

Let's kick off. I don't want you waiting

here for 25 more minutes before we some

arbitrary deadline.

So,

welcome.

My name's Matt,

I'm a teacher, and I suppose now I teach

AI.

Um

We have a link up here, if you've not

already been to this, which is has the

exercises for the um stuff we're going

to do today.

This is going to be around 2 hours, so

we might just sort of kick off 2 hours

from now. Is that all right, Mike?

Yeah, perfect.

Um and

the theory behind this talk, or at least

the thesis under which I've been

operating for the last kind of 6 months

or so, is that

we all think that AI is a new paradigm,

right? AI is obviously changing a lot of

things. You guys are obviously

interested in this, and that's why

you've come to this talk.

And

I feel that

when we talk about AI being a new

paradigm, we forget that actually

software engineering fundamentals, the

stuff that's really crucial to working

with humans, also works super well with

AI.

And this is what my keynote is on

tomorrow, really. I'm going to sort of

be fleshing that out a lot more.

And in this workshop, I'm hopefully

going to be able to direct your

attention to those things, and

uh hopefully show you

that I'm right. But we'll see.

Um can I get a quick heads-up first? How

many of you guys um are coding have ever

coded with AI? Raise your hand if you've

ever coded with AI. Perfect. Okay. Uh

keep your hand raised.

Uh

let's all uh share those armpits with

the world. Um

how many of you code every day with AI?

Cool. Okay. Uh right, keep your hand

raised if you've ever been frustrated

with AI.

Okay, very good.

You can put your hands down.

Thank you for that show of obedience. I

really appreciate that. And we are also

being live-streamed to the Gilgood room

as well. I've not

uh

Did we send someone up to the Gilgood

room to just check they're okay?

Don't know.

But I see you,

and there is a way that you can

participate, which is we have the um a

Q&A. We're going to be doing kind of

have a sort of hatred of Q&As cuz

they're not very democratic. They're

mostly the sort of

um most talkative people get to um

get to participate and share. And so,

we're going to be going through this um

Q&A here. So, why do we have to wait

till 3:45? The room is packed, the doors

are closed. 100% agree.

And so, if you want to uh ask a

question, we're going to be I would like

you to pile into this async, and then we

can vote on each other's questions, and

hopefully get the best questions

surfaced so the for the entire room to

enjoy.

So, I want to talk about first the kind

of weird constraints that LLMs have.

And

those weird constraints are sort of what

we have to base a lot of our work

around.

Now,

there's a guy called Dex Hardy who runs

a company called Human Layer, and he

came up with this idea, which is that

when you're working with LLMs, they have

a smart zone

and a dumb zone.

When you're first kind of like

working with an LLM, and it's like

you've just started a new conversation,

you start from nothing, that's when the

LLM is going to do its best work.

Because in that situation, the attention

relationships are the least strained.

Every time you add a token to an LLM,

it's kind of like you're adding a team

to a football league. You think of the

number of matches that get added every

time you add a team to a football

league, it just goes

it scales quadratically. And that's

because you have attention relationships

going from essentially each token to the

other that are positional and the sort

of meaning of the individual token.

And so, this means that by around sort

of 40% or around I would say around 100K

is kind of my new marker for this. Cuz

it doesn't matter whether you're using 1

million

uh context window or 200K,

it's always going to be about this.

It starts to just get dumber.

So, as you continually keep adding stuff

to the same context window, it just gets

dumber and dumber until it's making kind

of stupid decisions. Raise your hand if

that feels familiar to you.

Yeah, cool.

So, this means that we kind of want to

size our tasks in a way that sticks

within the smart zone.

Right? We don't want the AI to bite off

more than it can chew. This goes back to

old advice like Martin Fowler in

refactoring. Uh like uh the pragmatic

programmer talks about this. Don't bite

off more than you can chew. Keep your

tasks small so that you as a developer,

a human developer, don't freak out and

don't start acting and going into the

dumb zone.

But

how do you tackle big tasks? How do you

take a large task like I don't know,

cloning a company or something, or just

doing something crazy,

and how do you break it into small tasks

so they all fit into the dumb zone?

One way, of course, you could do is I

mean, kind of what the AI companies

maybe want you to do, or the natural way

of doing it is just keep going and going

and going, you end up in the dumb zone,

charging you tons of tokens per request.

You then compact back down.

We'll talk about compacting properly in

a minute. And you keep going, keep

going, keep going, compact back down,

keep going, keep going, keep going.

And I think that's doesn't really work

very well because the more sediment I

we'll talk about that in a minute.

So, the theory here is then, and this is

what I was doing for a while,

is I would use these kind of

um multi-phase plans.

Where I would say, "Okay, we have this

sort of number four thing here, this

large large task. Let's break it down

into small sections so that we can then

kind of chunk it up and do each little

bit of work in the smart zone." Raise

your hand if you've ever used a

multi-phase plan before.

Yeah, really common practice, right?

This is kind of how we've been doing it.

Certainly, this is how I was doing it up

until December last year, really.

And any developer worth their salt will

look at this and go, "This is a loop."

Right? This is a loop. We've just got

phase one, phase two, phase three, phase

four. Why don't we just have phase N?

Right?

Phase N. Where we essentially just say,

"Okay,

we have, let's say, a plan operating in

the background, and then we just loop

over the top of it, and we go through

until it's complete."

And this is where um

Raise your hand if you've heard of Ralph

Wiggum as a software practice.

Okay, cool. Raise your hand if you've

not heard of Ralph Wiggum as a software

practice, actually. That's more like it.

Okay. So, there's this idea called Ralph

Wiggum, uh which is kind of um

sort of based on this,

which is essentially

all you need to do is sort of specify

the end of the journey,

where you just say, "Okay, we create a

PRD, a product requirements document, to

say, 'Whoa, okay, let's describe where

we're going.'" And then we just say to

the AI, "Just make a small change. Make

a small change that gets us closer and

closer to that."

And

Ralph works okay, but I prefer a little

bit more structure.

So, that's kind of where we got to in

terms of thinking about the smart zone,

and that's

kind of where I want you to first start

thinking about here.

Another weird constraint of LLMs is LLMs

are kind of like the guy from Memento,

right? They just continually forget.

They could just keep resetting back to

the base state.

Let me pull up this diagram.

I sort of I

I I really should use slides, but I just

prefer just like randomly scrolling

around a

uh infinite uh TL draw canvas. Thank

you, Steve.

Um

So, let's say another concept I want you

to have is that every session with an

LLM kind of goes through the same

stages.

You have, first of all, the system

prompt here. This gray box here is

essentially the stuff that's always in

your context. You want this to be as

small as possible. Cuz if you have a ton

of stuff in here, if you have 250K

tokens, like I have seen people put in

there, then that you're just going to go

straight into the dumb zone without even

being able to do anything.

So, you want this to be tiny.

>> [snorts]

>> You then go into a kind of exploratory

phase. This blue sort of where the

coding agent is going out and exploring

the code base.

Then you go into implementation.

And then you go into testing.

And sort of making sure that it works,

running your feedback loops and things

like this.

Raise your hand if that feels familiar

based on what you've done. Yeah. Sort of

the like the the main cornerstones of

any session.

And when you clear the context, you go

right back to the system prompt.

Oof, you go right back there. So, you

delete everything that's come before.

And

raise your hand if you've heard of

compacting, as well.

Yeah, okay. There are some people who've

not heard of compacting. So, let's just

quickly show what that means.

For instance,

I've just been having a little chat with

my LLM.

Uh

I want to make sure we sort of, you

know, just cover the basics so we're all

sort of on the same wavelength here.

I've just been having a chat with my

LLM.

I've been talking about a thing that I

want to build. How's the font size?

Should I bump it up?

Folks in the back?

Bump. Bump.

Bump. Bump. Bump. Oh.

I'm using Claude Code for this session,

but you don't need to use Claude Code.

Uh

in fact, it's often nice not to use

Claude Code.

Um

so, I've been having a chat with the

LLM, just sort of planning out what I'm

going to do next. It's asking me a bunch

of questions, and I can

I highly recommend you do this.

There's this tiny little status line

here that tells me how many tokens I'm

using, the exact number of tokens I'm

using. Um I have a article on my website

AI Hero if you want to copy this. This

is

Oh, wow, that is that shakes, doesn't

it? Um

this is essential information on every

coding session cuz you need to know

exactly how many tokens you're using so

that you know how close you are to the

dumb zone.

Absolutely essential.

And so let's watch it.

So I've got two options. I can either

clear

wrong and go back to nothing or I can

compact.

And when I compact then it's going to

squeeze all of that conversation, which

admittedly isn't very much, into a much

smaller space.

And this in diagram terms kind of looks

like this.

Where you take all of the information

from the session and you essentially

create a history out of it, a written

record of what happened.

And devs love compacting for some

reason, but I hate it.

I much prefer my AI to behave like

uh the guy from Memento because this

state

is always the same. Always the same

every time you do it. You clear and you

go back to the beginning. And so if

you're able to do that and you're able

to optimize for that then you're in a

great spot.

So that's kind of the two things I want

you to think about with LLMs, the two

constraints that we're working with.

They have a smart zone and a dumb zone

and they're like the guy from Memento.

So let's take a look at the first

exercise.

And I'm while I'm doing this, the way I

want this to work is I'm going to sort

of show you how um I'm going to be sort

of walking through it up here and I want

you folks to be kind of like tapping

away and doing things as well. So that

was just a little lecture bit. Let's now

actually get and do some coding.

For anyone who arrived late or anyone in

the Gilgud room uh go to this link

this link up here

to see the exercises and clone the repo.

You absolutely do not have to, you can

just watch me do it if you fancy it.

But let's go there myself and let's see

what exercises await us.

So essentially I've built a um this is

from my course.

This is a uh a course management

platform essentially, a kind of CMS for

instructors, for students, and this is

what we're going to be building a

feature in. So I'm going to take you

from essentially the idea for the

feature all the way up to building a PRD

for the feature, all the way up to

implementing the feature.

And hopefully you can take inspiration

from this process and use it in your own

work.

So

uh let's kick off. So

we're going to start by using a a skill

which is very close to my heart.

It's the grill me skill.

And this grill me skill is wonderfully

small wonderfully tiny and it helps

prevent one of I think the main issues

when you're working with an AI, which is

misalignments.

The uh

the sort of silent idea that I'm talking

against here, that I'm arguing against,

is the specs to code movement. Has

anyone heard of the specs to code

movement? Raise your hand. It's not

really a movement I suppose, it's just

sort of people saying specs to code.

Um

what it is is people say, "Okay, you can

write a program or you want to build an

app the best way to build that app is to

take some specifications

so to write some sort of like document

and then turn that document into code."

So they just turn it into code. How do

you do that? You pass it to AI. If

there's something wrong with the

resulting code, you don't look at the

code, you look back at the specs. You

change the specs and you sort of just

keep going like this. This is kind of

like vibe coding by another name where

you're essentially ignoring the code.

You don't need to worry about the code.

You just sort of keep editing the specs

and eventually you just keep going. And

I tried this. I really tried it. And it

sucks. It doesn't work.

Because you need to keep a handle on the

code. You need to understand what's in

it. You need to shape it because the

code is your battleground. And so

this is again is where we're going.

Let's let's get some exercises.

So

what I'd like you to do is go to this

page, the the grill me skill.

And inside the repo here

we have a slack message

from our pal. Uh where is it? It's in

the root of the repo and it's under

bur bur bur bur

Oh, where is it?

Mhm mhm client brief.md.

It's a slack message from Sarah Chen.

For some reason the Claude always

chooses Sarah Chen as the name. I don't

know why.

Um it's saying that in cadence, our um

course platform, our retention numbers

are not great. Students sign up to a few

lessons then they drop off. I'd love to

add some gamification to the platform.

And so when you're presented with an

idea like this, you need to find some

way of turning it into reality. Let's

say Sarah Chen is your client, you're on

a tight budget, you need to get this

done fast. How do you go and do it?

Um

raise your hand if you would um

enter plan mode when you're doing this.

Anyone a big user of plan mode? Yep.

Um let's actually shout out quickly any

other ideas about what you would do with

this or any Raise your hand if you

what what would be your first port of

call?

Yep. Ask for more info.

Sorry? Ask for more

info to verify what is the purpose and

where our current standing is. Yes,

exactly. Let's imagine that Sarah Chen's

gone on holiday, you have no idea,

right? Uh she's just posted this thing,

you need to action it before you go.

Well, my first port of call is I go for

this particular skill. I'm going to

clear my context.

I'm going to

uh get rid of

you, you don't need to be there.

And I'm going to say

um I'm going to invoke a skill

which is the grill me skill. Let's

quickly check.

Raise your hands if you don't know what

this is.

Cool.

Oh, sorry sorry. Let me be more

specific. Raise your hands if you don't

know what I'm doing here when I

uh do a forward slash and then type

something.

Anyone Everyone kind of understand what

that is?

I'm invoking a skill. I'm invoking the

grill me skill.

And what I'm going to do is I'm going to

say grill me and I'm going to pass in

the client brief.

So now

the LLM really has only a couple of

things here. It just has the skill and

it has the description of what I want to

do.

And this is virtually how I start every

piece of work with AI.

And while it's exploring the code base

I'm just going to show you what the

grill me skill does.

So this is inside the repo so you can

check it out.

It's extremely short.

"Interview me relentlessly about every

aspect of this plan until we reach a

shared understanding. Walk down each

branch of the decision tree resolving

dependencies one by one. For each

question provide your recommended

answer.

Ask the questions one at a time uh blah

blah blah."

What this does and what I noticed when I

was working with AI, especially in plan

mode actually

is it would

really eagerly try to produce a plan for

me.

It would say, "Okay, I think I've got

enough. I'm just going to poof plan

plan."

And what I found was that

I was really trying to find the words

for this, for for what I wanted instead

of that.

And Frederick P. Brooks in The Design of

Design, he has a great quote uh talking

about the design concept.

When you're working on something new

with someone

when you're uh all trying to build

something together

then there's this shared idea that's

shared between all participants and that

is the design concept. And that's what I

realized I needed with Claude. I needed

I needed to reach a shared

understanding. need an asset, I didn't

need a plan, I needed to be on the same

wavelength as the AI, as my agent. And

this is an extremely effective way of

doing it. So hopefully

Here we go. Nice. It has done its

exploration first of all.

It's invoked a sub agent which spent

97 93.7k tokens

on Opus.

Um

and it's asked me the first question.

Cool.

We can see that even though the sub

agent burned a a ton of tokens I haven't

actually um

uh increased my token usage that much.

Raise your hand if you don't know what

sub agents are. It's important question.

Everyone kind of clear what sub agents

are? Okay, I'll give a brief definition.

Which is that this this sub agents thing

here, this explore sub agent it has

essentially gone and called another LLM

which has an isolated context window.

And then that LLM has reported a summary

back. So a sub agent is kind of like a

delegation. You're delegating a task to

a sub agent. It goes eagerly does all

the thing, explores a ton of stuff and

then just drip feeds the important stuff

back up to the orchestrator agent.

To the parent agent. So okay. So

hopefully you guys have seen the same

thing. It's done an explore.

And we now have our first question.

Points economy. What actions earn points

and how much? Ooh, okay.

At this point you can ask it by the way

questions to um deepen your

understanding of the repo. I obviously

know this repo really well cuz I wrote

it, but you might not um

know what's going on.

So let's say my recommendation, keep it

simple, two point sources to start.

What's so nice about this is that not

only does it give us a question that

kind of aligns us here, we get a

recommendation too. And often what I'll

find is the AI's recommendations are

really good.

And so I'll just say

skip video watch events, they're noisy

and gameable. I agree.

Sarah's asked we'll keep the lessons in

the bread and butter.

Yeah.

Looks good, pal.

>> [snorts]

>> Now what I usually do is I usually

dictate to the AI. I'm usually actually

chatting to the AI instead of uh typing

here, but uh this is a relatively new

laptop and I couldn't get my dictation

software working on it um because

Windows is crap. Um

So, should points be retroactive? There

are existing lesson progress records

with completion at timestamps. This is a

really nasty question, right? Should we

actually go back and backfill all of the

lesson progress events? This is a kind

of question that you need to be aligned

on if you're going to fulfill the

feature properly. This is not something

I considered and Sarah Chen certainly

didn't consider.

Do I want it to be retroactive? Hmm.

Let's actually do a vote inside here.

Should we go back and backfill all the

records? Raise your hand if you think we

should backfill all the records.

Raise your hand if you think we

shouldn't backfill all the records.

There are a lot of fence-sitters in the

room. I'm going to say

you know, this is the kind of discussion

you're sort of having with the AI.

You're getting further aligned. Yes, I'm

just going to go with his recommendation

cuz I'm lazy.

Notice too how I'm able to keep in the

loop here with AI. I'm not you know,

it's it's pinging me these questions

pretty quickly.

I'm not having to go off and check

Twitter or something.

Levels. What's the progression curve?

Yeah, that looks about right. For

instance, yes, okay.

So hopefully you should be able to go

and um

kind of work through this with the AI.

>> [clears throat]

>> And essentially

try to reach an alignment. And this

grill me skill, this can last a long

time. This can I've had it ask me 40

questions. I've had it ask me 80

questions. I've had some people that

asks 100 questions too. Literally you're

sat there for an hour chatting to the

AI.

And what you end up with is essentially

this conversation history

that works really nicely and works

really nicely as an asset of the design

concept that you're creating.

This can also function like this. You

can

have a meeting with someone who's a

maybe a domain expert. Maybe I have a

meeting with Sarah. I feed that meeting

transcript into

I don't know, Gemini meetings or

whatever you guys are using. You take

that, you feed it into a grilling

session and you grill through the

assumptions that you didn't have.

So this ends up being a really nice kind

of

um

a really nice way of just taking inputs

from the world and then just turning and

validating them.

So okay.

Let's see. I really want to get to the

end of this, but I also don't want to

just like be sat here talking to the AI

in front of you for uh

a thousand days. So I'm just going to

say yes.

Let's see what happens.

So I'll tell you what, um while you guys

sort of have a little fiddle with this

locally, let's start a little Q&A

session now.

And

let's see. How's this going to work?

Can we keep the door closed or turn up

the microphone? It's quite noisy.

Uh

let's see. Mike, can we uh

door closed. Oh it has been closed. Mark

has answered. Beautiful.

So what I'd like you to do

is there any air con? Yeah, there is

some air con, I think.

There is some air con.

You guys aren't being lit here. I'm

being fro I'm being fried alive here.

Uh so what I'd like you to do is go on

to the Slido, which you can join here.

Have a if if you're not taking the

exercise, go on to the Slido, have a

little fiddle and vote on some good

questions. I'm just going to chat to the

AI for a second

uh until we reach a stopping point. So

do streaks earn points?

Um

streaks are standalone.

Let's see what else it comes up with.

Where does gamification UI live?

Let's have it in the dashboard.

I'm just going to scan these and blast

through them basically.

So how are we doing with our Slido?

Okay.

Have I tried Spec Kit, Open Spec or

Taskmaster instead of the Grill Me

skill? Do I find them more verbose or a

structured alternative? This is a great

question. So there are a ton of

different frameworks out there that

allow you to um sort of build up this

planning process for you. I personally

believe you at at this stage, when

there's no clear winner, when there's no

kind of like one true way and when

things are changing all the time, you

need to own as much of your planning

stack as you possibly can.

What I've noticed and a lot of my

students

is

they tend to overuse a certain stack.

They get into trouble

and they because they don't own the

stack and they don't have observability

over the whole thing, they just go

this isn't working. This sucks. Whereas

if

um

if you have control over the whole

thing, then at least you know how to fix

it or potentially know how to fix it.

So I'm even though I'm sort of giving

you uh a stack basically, I believe in

inversion of control and you should be

in control of the stack.

So bur bur bur.

Can I press zero, please?

Sorry?

Sorry, that was a lot of sort of

mumbling. Can I

Thank you.

I'm so sorry.

>> [laughter]

>> What you didn't want to give Claude good

feedback? What is what is wrong with

you?

Uh okay, cool.

Uh many of the questions asked by the

Grill Me skill are not necessarily

appropriate for a developer, rather a

PO. In larger teams, who should use it?

Yeah.

Um

Raise your hand if um

you've ever done pair programming.

Anyone ever done pair programming?

Right. I keep Put your hands down and

raise your hand again if you've ever

done a pair programming session with an

AI.

Right.

How did it go? Was it good? You enjoy

it? I think pair programming sessions

with AI is a great idea because you've

got a third person in the room who will

relentlessly quiz you and ask you

questions. It should If you don't know

the answer, it should be you, the domain

expert and the AI in the same room. If

you're have a question about

implementation, it should be you, a

fellow developer and the AI in the same

room, you know. You can be sort of

working through these questions in your

team. And I think actually

we're going to look at implementation in

a bit and we're going to see how you can

make implementation so much faster.

And but I think the really crucial

decisions, the ones you need humans for

you actually need a lot of humans and it

doesn't really matter how many humans

are in there. You can actually throw a

bunch like a kind of like mob

programming with AI essentially.

Uh what's my favorite meta prompting

tool? I think I kind of answered that.

Uh there's no air con. Let's just live

with it. Uh

how do I use the conversation as an

asset after the Grill Me session? Well,

we're going to get there.

Um okay, so I really want to

I want to speed this up sort of

artificially.

Just what

I This is the thing. So someone just

said okay, Ralph loop this. But this is

crucial because I can't loop over this,

right? I can't um

I think of there is being two types of

tasks in the AI age.

Where you have human in the loop tasks,

where a human needs to sit there and do

it.

Which is this.

We are the human in the loop, with

multiple humans in the loop. And there

are AFK tasks. There are tasks where the

human can be away from the keyboard and

it doesn't matter. Implementation, as

we'll see, can be turned into an AFK

task. But planning, this alignment

phase, has to be human in the loop. Has

to be.

So I've got to do it, unfortunately.

Um

I don't know.

Uh

give me a long list of all your

recommendations.

I'm running a workshop right now.

So I artificially

need you to

pull more weight.

So let's see what it does.

Uh let's answer a couple more questions

while it's doing its thing.

What is my opinion on PMs or other

non-dev roles vibe coding task?

Hmm.

Um I'm going to return to this later, I

think. I'm going to leave this

unanswered.

A bit of mystery.

I notice I'm not using the ask user

questions UI for Grill Me. Why? Um

there's a specific uh

UI that you can bring up in Claude Code.

I'll answer this just quickly.

Uh ask me a question using the ask user

question tool.

>> [snorts]

>> And this UI um is just sort of broken in

Claude and I really hate it.

You notice I'm using Claude, but I don't

like Claude very much. Like you you

really are free with this method to

choose any um system you like. And this

is what the UI looks like.

It's very pleasing when you first

encounter it, but then you realize it is

actually broken in a ton of different

ways.

All right, what did it come back with?

Oh blimey.

Oh no.

So

while this is doing its thing, let me do

some teaching in the meantime.

The plan here is that we take our Grill

Me skill

and we need to essentially find some way

of turning it into

a destination.

We need to go down to the

uh

We essentially need to

we're figuring out the shape of this.

That's what we're doing. We're figuring

out the shape of the tasks during the

grilling session.

And in order to

turn it into a bunch of actionable

actions for the AI

we essentially need to figure out the

destination. We need to know where we're

going. We need to know the shape of this

entire thing.

So I think of there is being two

essential documents that we need.

We need a document that

documents the destination.

Oh no.

It's so not bright enough. There we go.

Still not brighter. There we go.

We need something to document the

destination.

And we need something to document the

journey.

In other words, we need something a

document that's going to

figure out what this even looks like in

all of its user stories and figure out a

definition of done

and then we need to figure out what the

split looks like.

So, that's where we're going to go to

next.

So, once we finish with the grilling

session,

yeah, it looks great. Fantastic. I love

it. It answered

it answered 22 of its own questions.

There you go. That's quite

representative of what a grilling

session looks like.

So, at this point now,

I have used 25k tokens and all of that

or loads of that stuff is gold. I want

to keep that around. I've I've got 25k

great tokens there.

And what I want to do is kind of

summarize it in some kind of destination

documents.

So, this is um the next exercise

where we're going to

uh we're going to write a product

requirements document.

And the the product requirements

documents or the PRD

is essentially

that's its function. It's the

destination documents. And it's sort of

doesn't matter what shape it is. I've

got a shape that I prefer and I quite

like.

But, you can just choose your own shape

or whatever your company uses.

And all we're really doing is I'm not

too worried about that.

All we're really doing is summarizing

the design concept that we have so far.

And

the So, let let's try this.

So, I'm going to initiate this. I'm

going to say

zoom all the way to the bottom.

All I'm going to do is just say write a

PRD.

And we can take a look at that skill

now.

Write a PRD.

So, this skill

it does a few things.

It first asks the user for a long

detailed description of the problem. You

can use write a PRD without grilling

first, but I just like to grill first

and then write the PRD afterwards.

Then you can um get it to install the

repo which we've kind of already done.

Then we get it to

interview the user relentlessly so we

have a kind of grilling session again

and then we start um putting together a

PRD template. So, this is available in

the repo if you want to check it out.

And essentially this is what it looks

like. We've got some problem statements,

the problem the user is facing, the

solution to the problem and a set of

user stories. And these user stories

sort of define what this is. You know,

as

you you guys have probably seen things

like this if you've been a developer at

all. Um you know, there are cucumber is

a language you can use to write these in

or we just sort of

um

uh write them ourselves essentially.

Then we have a list of implementation

decisions that were made and list of

crucially testing decisions, too.

So,

I'm going to run this. Okay. And so,

it's finished its thing.

Ah!

Windows, let me close the thing. Thank

you.

I don't know why I bought a Windows

laptop. I think I just

I like the challenge. Um

>> [clears throat]

>> So, the first thing that it's going to

give me

are a set of proposed modules it wants

to modify.

Now, there's a deep reason why I'm

thinking about this. So, this is

at this stage

we have an idea, we have sort of specked

out the idea, we've reached a sort of

understanding of what we're trying to do

and then we need to start thinking about

the code

because at this point we need to

this is not specs to code. This is not

where we're ignoring the code. We

actually keep the code in mind

throughout the whole process.

And

the way I like to do this is I like to

just sort of think about a set of

proposed modules to modify. We're going

to return to this this idea of

continually designing your system and

keeping your system in mind.

So, it's it's saying recommend tests for

the gamification service is the only

deep module with meaningful logic. These

modules look right. Yeah.

Looks good.

And it's going to hang out a PRD.

Now, for ease of setup

I've got it so that it creates a set of

issues locally.

So, it's just going to create

essentially a PRD inside this issues

directory.

But, the way I usually do it

and you can check this out yourself is

you can go to my um essentially what I

consider my work repo

which is GitHub um dot com forward slash

Matt Pocock forward slash course video

manager up here.

And in here, this is essentially a app

that I create um that I use all the time

to record my videos and things like

this. I think I've recorded like

I pulled out the stats. I think I've

recorded like a thousand videos in here

or something nuts.

Um and you can see here that it's got

744 closed issues.

And this is essentially all of the uh

PRDs and all of the implementation

issues that I've put into here. So, this

is how I usually like to do it.

>> [clears throat]

>> So, that's what I'm doing with the There

we go. Yeah, I'm just going to say yes

and uh

and get that issue out.

Let's see. It is inside here.

So, we've got the problem statements.

People signing up for courses.

Uh the solution, the user stories, uh 18

user stories looks nice, some

implementation decisions, level

thresholds, etc. This is enough

information. We've kind of clarified

where we're going and what we're doing.

So, that's what we do. We essentially

have a grilling session and we've

created an asset out of it. Now, raise

your hand.

Should I be reviewing this document?

Raise your hand if you think I should be

reviewing the documents.

Yeah, I don't I don't look at these.

I don't look at these.

The reason I don't look at these is

because what am I testing at this point?

What am I Like when I read it,

what am I testing? What am I What are

the failure modes I'm trying to test

for?

I know that LLMs are great at

summarization

cuz they are. They're really good at

summarization.

I have reached the same wavelength as

the LLM, right? Using the grill me

skill, we have a shared design concept.

So, if I have a shared design concept,

all I'm doing

is I'm just essentially checking the

LLM's ability to summarize.

So, I don't tend to read these.

Let's have Let's have a Q&A cuz I can

feel you guys are itching for it. And I

think we might have like

I don't know, just a 5-minute comfort

break just to uh rest my voice and so

you can catch up with the exercises for

a minute if that's all right. So, let's

have a little Q&A sesh.

Uh

If I don't like Claude Code, which one

do I actually like? Um

uh

Have you ever heard the phrase um

uh democracy is the worst way to run a

country apart from all the other ways?

That's how I feel about Claude Code.

Uh we've answered that one.

Uh

What's your thoughts on developers

needing to very deeply understand

TypeScript now that fix the TS make no

mistakes exist? I don't understand the

phrasing of this,

but I think I understand meaning,

which is that

I believe that code is very important

and this is kind of going to feed

through the whole session and that bad

code bases make bad agents. If you have

a garbage code base, you're going to get

garbage out of the agent that's working

in that code base. We'll talk more about

that in a bit.

And so, I think understanding these

tools very deeply, understanding code

deeply is going to make you a much much

better developer and get more out of AI.

Uh and that answers that question, too.

Sweet.

Uh

Get out of there. There you are.

Now that we have 1 million tokens

available, do we ever actually want to

take advantage of that?

I've noticed that the dumb zone has

become less dumb lately. Okay, great

question. This goes back to our kind of

initial idea on the dumb zone.

Uh

I am I recorded my Claude Code course

using a 200k context window and on the

day that I launched the course they

announced the 1 million context window.

My take on this is that what Claude Code

did is they essentially just did this.

Wee!

They shipped a lot more dumb zone to you

essentially. Now, this is good for tasks

where you want to retrieve things from a

large context window. If you want to

pass five copies of War and Peace or

something to it and you want to find out

all the things that uh

uh I can't remember a character from War

and Peace. Uh

Why did I start with that?

It's good for retrieval.

It's less good for coding.

So, I consider that it is about 100k at

the moment is the smart zone. The smart

zone will get bigger and that will be a

really nice improvement.

So, folks, we're going to take it like a

5-minute comfort break if that's all

right just for my voice and to maybe you

can have a little move around or

something or grab a drink. I can just

notice some sleepy eyes and I want to

make sure that we're awake for the next

bit if that's all right. So, we'll take

5 minutes and I will see you back here

then. All right?

So, we have

our PRD

which I'm not going to read, our kind of

destination document. Let's quickly scan

for any good questions before we zoom

ahead.

And

Rediscovering the role of software

engineering today's world, top three

disciplines you recommend.

Um

Taekwondo is good, I've heard. I've no

I've no idea how to answer this

question. Um

thank you for asking it though. Um Top

three disciplines I recommend.

I mean

Sorry? Plumbing. Plumbing is a good one.

Yeah, yeah, yeah. I don't know if that's

a discipline. I the plumbers I've hired

are not usually very disciplined.

Um

Right.

So, okay. We now have our destination,

okay? Um

Perfect.

So, how do we actually get to our

destination? How do we We have a sort of

vague PRD? How do we split it so that we

don't put things into the dumb zone?

In other words, we have our number four,

how do we split it into this kind of

multi-phase plan? Well, probably what

you would do at this point is you would

say, "Okay, Claude, give me a

multi-phase plan that gets me to this

destination, right?" That sort of makes

sense. This is what we've been doing

before.

But I have um

a sort of better way of doing it now,

which is that

I like

creating a Kanban board out of this.

Raise your hand if you don't know what a

Kanban board is.

Mm, cool. Okay. A Kanban board is

essentially just a set of tickets that

you put on the wall that have blocking

relationships to each other. So, we're

going to see what it kind of looks like

here. This is how we've worked um

as developers for a long time, really

since Agile came around. And what it

does, we can see it here,

it has proposed that we split this setup

into

um five different tasks here.

We have the first one, which is the

schema and the gamification service.

Yeah, well, that looks pretty good. This

is blocked by nothing.

And we can even see here that it's a

it's given it a type of AFK, too. You

remember I talked about human in the

loop and AFK earlier? This is an AFK

task. This is something we can just pass

off to an agent to do its thing.

Streak tracking, okay, that looks good.

Uh

then wire points and streaks into

lessons quiz completion. This is blocked

by one and two.

Retroactive backfill. This is blocked

only by one.

And then this one here is blocked by all

of the tasks. Cool.

Hmm.

Now, I consider this you could say, "Why

don't we just make this sort of

generation of the issues, why don't we

just hand that over to the AI? Why do I

need to be involved here, right?" Cuz

it's given us quite a good selection of

tools here. Why do I need to review this

and sort of

figure out what's next?

Now, my take here is that this is really

cheap to do, like very quick to do once

I've done the PR, and I can immediately

see some issues here.

There's a really, really important

technique when you're kind of figuring

out what the shape of this journey

should look like.

And

it sort of comes to this very classic

idea, uh which comes from the Pragmatic

Programmer called traceable bullets or

vertical slices.

And traceable bullets really transformed

the way I think about actually

getting AI to pick its own tasks.

Systems have layers, right?

There are layers in your system.

These might be different deployable

units. You might have a database that

lives somewhere. You might have an API

that lives maybe close to the database

but in a separate bit. You might have a

front end that lives somewhere totally

different like a CDN.

Or within these deployable units, you

might have different layers within

those. In for instance, the code base

that we're working in, we have a ton of

different services. Service. We have a

quiz service, a team service, a user

service, coupon service, core service.

And these services have dependencies on

each other. So, they're kind of like

individual layers.

Well,

what I noticed is that AI loves to code

horizontally.

So, it loves to code layer by layer.

So, in other words, in phase one, it

will do all of the database stuff, all

of the schema, all of the you know, all

the stuff related to that unit. Then it

will go into phase two and do all of the

API stuff. Then it will add the front

end on top of that.

Does Can anyone tell me what's wrong

with that picture? Why is that not a

good thing to do? Raise your hand if you

have an answer.

Yeah.

>> have that whole feedback loop.

Exactly. You don't get feedback on your

work until you've

really started or completed phase three.

So,

what you really need to do is you you're

not until you get to phase three, you're

not actually testing that all the layers

work together.

You haven't got an integrated system

that you can test against.

And so,

instead you need to think about vertical

layers. You need to think about thin

slices of functionality that cross all

of the layers that you need to.

And this is a much better way to work,

much better way for the AI to work, too,

because it means at the end of phase one

or during phase one it can get feedback

on its entire flow.

So, what this means to me

is inside the PRD to issues skill up

here,

I have got break a PRD into

independently grabbable issues using

vertical slices traceable bullets

written as local markdown files.

[snorts]

We first locate the PRD.

Uh again, explore the code base if this

is a fresh session. We draft vertical

slices.

So, we break the PRD into traceable

issues. A traceable bullet, by the way,

is uh

essentially when you're like an

anti-aircraft gunner. It's quite a

violent idea, actually. Uh

and you're looking up in the sky and

it's night. If you're just shooting

normal bullets, you have no idea what

you're firing at, right? You could just

be you know, you you see the plane but

you don't see where your bullets are

going.

Traceable bullets is they attach a tiny

bit of phosphorescence or phosphor or

something to make it glow as it goes.

So, this means that every sixth bullet

or something you actually see a line in

the sky. So, you have feedback on where

you're aiming. So, this is what this is

the idea here is that we increase our

level of feedback and we get near

instant feedback on what we're building.

Cuz without that the AI is kind of

coding blind until it reaches the later

phases.

We got some vertical slice rules. We

quiz the user.

And then we create the issue files. So,

what I see here

is that even though

I've I've told it to do vertical slices,

it's proposing to

create the gamification service

first on its own. That's just one slice

there. And that to me feels like a

horizontal slice. What I want to see in

the first vertical slice especially is I

want to see the schema changes or some

schema changes. I want to see some new

service being created and I want a

minimal representation of that on the

front end. So, I want it to go through

the vertical slices, not just the

horizontal. Does that make sense?

Okay. So, I'm going to give the AI

a rollicking.

Uh bad boy. No, I'm not.

I'm not going to waste tokens just being

just naming. Um

So, the first slice is too horizontal.

I'll just start with that and see if it

picks it up.

Does that make sense as a concept?

And I think having that um

what I really like about going back to

those old books is that we're really

trying to in this day and age like get

uh

verbalize best software practices in

English.

And these books, 20-year-old books, have

already done that. And it's an absolute

gold mine if you want to throw that into

prompts. But even with that, it's not

going to um not going to do a perfect

job each time.

So,

award points for lesson completion

visible on dashboard. Yes, that's a

beautiful vertical slice because it's

definitely a big chunk of stuff. It's

doing a lot of stories there, but we're

going to see something visible at the

end and the AI will then just be able to

add to that. You see why that's

preferable to the first one. Cool.

Uh looks great.

So, we're getting closer now. Anyone

following at home as well, you know, not

at home but you get the idea.

Um will hopefully see the same thing,

too, and start developing the same

instincts.

Let's open up for questions just while

I'm still creating these GitHub issues.

Uh ba ba ba ba Oh, not GitHub issues. Uh

local issues.

When will I stop using Windows? Never.

What is your Okay, we'll get to that

later.

How does AI um decide when to stop

grilling? Cuz AI can ask incessantly,

can we have a smarter way to decide the

stop point? Yeah, it does tend to really

um

those grilling sessions can be super

intense. And the thing about these

skills is you can tune them if you want

to. If you feel like the AI is just

absolutely hammering you, hammering you,

hammering you, then you can just

tell it to just pull back a little bit

or get it to do, you know, stop points

and that kind of thing. So, if that's a

failure mode that you run into a lot,

then you just, you know, change the

skill.

Uh do I still use uh be extremely

concise, sacrifice grammar for the sake

of concision? Um there was a tip that I

gave folks um

5 months ago, which is that

to basically increase the readability of

your plans. So, when you're using plan

mode,

then you can put it in your Claude.md

and you can say, "Okay, yeah, approve

that."

Let's open up Claude.md.

Uh do I have a Claude.md? Maybe I don't.

I I really don't use Claude.md very

much. I'm just going to put a dummy

inside here.

Um when

No.

When talking to me,

uh sacrifice grammar for the sake of

concision.

And this um prompt was uh really useful

to me when I was reading the plans

because it meant that the plans would

come out and they would be very concise,

really nice, easy to read, often very

concise. But I've

since dropped this idea in preference to

a grilling session because what I

noticed with it just I didn't want to

read the plans. I wanted to get on the

same wavelength as the LLM. I wanted it

to ask aggressive questions to me. And

when I stopped reading the plans, I

stopped needing them to be concise.

So, I think of the plans really in the

destination document as uh the end

state. And I don't need that end state

to be concise.

Hopefully that answers your question.

Uh

What do I think will be the outcome of

the Mexican standoff of future roles of

PMs and other roles converging? Uh I've

no idea. I'm not a pundit. I've no idea.

Uh okay.

So, we should

uh after a couple of approvals,

uh end up with a set of issues.

Now,

these issues that we're creating,

they're designed to be independently

grabbable,

which means that this Kanban board ends

up looking kind of like this.

Where you have

essentially a set of tickets with a

whole load of independent relationships.

So, this one needs to be done before

this one. This one needs to be done

before this one.

And this one, let's say we got another

one over here.

This one needs to be done before this

one.

This means that you can start to

parallelize.

You can start to get agents working at

the same time on these tasks. Because

yeah, this one needs to be done first.

And then

these two

can be grabbed at the same time by

independent agents.

Raise your hand if you've done any kind

of parallelization work with agents.

Okay, cool. So, this allows you

um to turn those plans into to optimally

kind of like into a directed acyclic

graphs essentially, where you just are

able to um

essentially have three phases here.

Where you have

phase one.

Uh let me grab move that.

Uh

above this line here,

you do this one.

Then phase two, you do the two below it.

And then phase three, you do this third

one and add it onto that.

And when you think about there could be

This could This is a relatively simple

plan, but you could have many different

plans operating all at once. It means

that you can do really nice

parallelization. And we'll talk more

about that in a bit. But that's why I

prefer a Kanban board set up like this

to a sequential plan. Because a

sequential plan can really only be

picked up by one agent.

So, this

Where did it go? Over here.

Yeah, this plan here

This is really only one loop, right?

Only one agent can work on these because

we have numbered phases and they're not

parallelizable. Does that make sense?

Cool.

So, we've got our issues. Ah, come on.

Stop asking me for I know it's creating

them on GitHub. I really don't want

that.

Oh, no.

You fool.

Create them in issues instead.

No.

That's not precise enough.

Uh you fool.

Create them in local markdown files

instead, referencing the local version.

Sorry about this.

So, once we get to this point,

we [clears throat] have a bunch of

issues locally

that we can start um looping over and

implementing. And it's at this point

that the human leaves the loop.

So, so far

Let me pull up a a proper overview of

this kind of flow that we're exploring

here.

So far

we have taken an idea.

I'll zoom this in a bit for the folks at

the back.

And we've grilled ourselves about the

idea.

We can skip over research and prototype,

but we turn that into a PRD, into a

destination document.

We then turn that PRD into a Kanban

board. And all of those steps

are human reviewed.

And now

the implementation stage, we step back.

And we let an agent um work through that

Kanban board or multiple agents work

through the Kanban board.

Now, what this means is that yeah, we

spent a lot of time planning here, but

it means that we've queued up a lot of

work for the agent. We can think of this

as kind of like the day shift and the

night shift. This is the day shift for

the human, right? Planning everything,

getting all the all the stuff ready. And

then once we kick it over to the night

shift, the AI can just work AFK. But

what does that look like?

Well,

so I'm just going to Oh, yeah. Just

allow it. It's perfect.

So, this looks like

if we head to the next exercise,

which is

uh in fact, the last exercise here,

running your AFK agent.

Now,

I've called this uh Ralph really cuz it

is a it is essentially a Ralph loop.

And this prompt here, I want to walk

through this really closely.

The first thing it's doing here is we're

essentially going to run Claude

and we're going to basically try to

encourage it to work um

completely AFK.

I'll show you what the sort of script

for this looks like in a minute.

But you say, "Okay, local issue files

from issues are provided at the start of

context."

The way we do that is if you look inside

once.sh here inside the repo,

we have

uh it's essentially just a bash script,

where we grab all of the issues,

um [clears throat] which are inside

markdown files, and we cat them into a

local variable. So, that issues variable

contains all of the issues that are in

our entire backlog.

Then we grab the last five commits. I'll

explain why in a minute.

And then we grab the prompt and we just

run Claude code with permission mode

accept edits.

And then just essentially just pass it

all of the information.

This is what the implementer looks like.

So, that's what a very very simple

version of this sort of loop looks like.

And of course, this is not a loop. This

is just running it once.

The loop

is in the AFK version up here,

which is uh a fair bit more complicated.

And the crucial part here is we're

running it in Docker sandbox as well.

So, I I don't want you to install Docker

on your laptops because we're just going

to be like, "You need to download a

special image and we're going to tank

the conference Wi-Fi if we do that." So,

I'm I am going to demo this to you, but

you um

won't need to run this yourself, but

I'll talk through this in a minute. But

essentially, this once loop here,

and ba ba ba ba boom.

We're just essentially running one

version of the thing that we're going to

loop again and again and again. So, this

is kind of like the human in the loop

version. And this is essential. Running

this again and again is essential

because you're going to see what the

agent does and see how it ends up

working. And any tuning that you need to

add to the prompt, then you can do that.

Let's go to the prompt.

Um

So, local issue files are being passed

in.

You're going to work on the AFK issues

only. That makes sense.

If all AFK tasks are complete, output

this no more tasks thing.

And then the next thing, pick the next

task.

So,

what we're doing here is we're

essentially running a backlog or

curating a backlog that our AFK agent is

going to pick up. That's the purpose of

all of these um setups in the beginning.

In this uh

all the way to this Kanban board here,

we're just essentially creating a

backlog of tasks for the night shift to

pick up.

And the night shift, this sort of Ralph

prompt here, it's got its own idea about

what a good task looks like to next pick

up.

I'm I did talk about parallelization. I

will show you this later, but this is

essentially a sequential loop here.

We're just going to run one coding agent

at a time. This is a good way to just

sort of um get your feet wet

essentially.

So, it's prioritizing critical bug

fixes, development infrastructure, then

trace bullets,

then polishing quick wins and refactors.

And then we just have a very simple kind

of instruction on how to complete the

task.

So, we explore the repo.

Use TDD to complete the task. I'll get

to that later.

And

we then run some feedback loops. So,

let's let's just try this and let's just

see what happens.

So, good. It's created the issue files.

We should be good to go. I'm going to

cancel out of this.

I'll clear and I'm going to run

uh

Where is it? Ralph

once.sh. And you can feel free if you're

following along to do the same thing.

So, we can see it's just running Claude

inside here

with the prompt and with all of the

issues that have been passed in.

And while it's doing its thing,

you probably have some questions about

this setup and about the decisions that

I've made to essentially

delegate all of my coding to AI, right?

So, let's let's do a quick Q&A while

it's getting its feet under it.

Uh okay. Ba ba ba ba ba.

I'm going to just

remove those.

How do you retain negative decisions,

things that you decided against, and

rationales when persisting the results

from the grill me session? Uh great

question.

There's a very simple answer, which is

the in the PRD uh write a PRD section,

there is a stuff at the bottom, a

section of the things that are out of

scope. So, the things we're not going to

tackle in this PRD, which is very

important for giving a definition of

done.

Feel free to ping on the Slido if you've

got any more questions.

Uh what's my front end workflow? Okay,

it's a great question. I'm going to I'm

going to answer that in a minute, I

think.

How to deal with agents producing more

code that we can review? How to properly

parallelize and use multiple agents

separate way. Okay, that's That's two

questions there.

Um

Raise your hand

if you feel like you're doing more code

review now than you used to.

Yeah, definitely.

Um

I don't think there's a way to avoid

this.

If we delegate all of our coding to

agents,

you notice that the implementation here

is really the only AFK bit. We then also

need to QA the work and code review the

work, right?

And if we are

running these loops where it's

essentially going to implement four

issues in one,

it's hard to pair that with the dictum

that you should keep pull requests small

and self-contained, right? Like small

self-contained pull requests means

you're needing to do fewer loops or

shorter loops or something.

Or maybe you do like a big stack of PRs,

but that seems horrible as well. That's

still just more separated code to

review. I don't honestly know what the

answer to this yet.

I think we just need to be ready to be

doing more code review, essentially.

Which is not fun. That's not fun thing

to say. That's not like I don't know. I

don't feel good saying that, but I do

think it's probably the

the way things are going.

It's a great question.

Uh

Can we grab a couple of questions from

the room as well? Let's not We won't do

the mic, but uh raise your hand if

you've got a question for me

immediately.

Yeah.

So, the approach is very linear from an

idea to uh QA code review. Of course,

the real world is a lot more messy. So,

you have all these ideas that are in

parallel and

nobody has the full picture. And

uh while you're working on something,

something else comes in as

a bug. Yeah. How do you deal with the

messiness? How do you tighten that

feedback loop? Great question. So, the

question was

if this all looks great if you're a solo

developer, but actually how do you

implement this in a team? How do you

gather team feedback on this?

And my answer to that is that if you

have an idea up there

and

essentially the sort of journey from the

idea to the destination

is something you need to figure out with

the team, right? So, all of this stuff

up here, this is kind of like team

stuff, you know what I mean? This So, if

you have an idea and you do a grilling

session on it and you have a question

that you don't know how to answer, then

you need to loop in your team as we

described before. Then you might need to

go, "Okay, like we just need to build a

prototype of this. We need to actually

hash this out. We need something that

the domain experts can fiddle with."

Or okay, we might need to integrate a a

third-party library into this. We might

need to do some research. We might need

to actually kind of like um

ping this back and forth and find a

third-party service that we can get the

most out of. We might need to go back

with the information that we gathered

there to the idea phase. So, all the way

up to the sort of PRD in the journey,

that's something you need to involve

your team with. That's something where

these assets are going to be shared over

and you're going to have requests for

comments on them and that that loop is

going to just keep grinding and grinding

until you figure out where you're going.

Once you figure out where you're going,

then you can start doing the Kanban

board implementation. But this is

essentially super arguable and the

you'll be bouncing back and forth

between the phases. Does that make

sense? Yeah.

Would you not need a

PRD for your prototype?

Say again, sorry. Would you not want to

have a PRD for your prototype? The

question was, do you want to go through

this whole session just to sort of

create a prototype? You don't need a PRD

for your prototype as well. Let's just

quickly talk about prototypes for a

second.

Um there was a question about how do you

make this work for front end?

Like how do you cuz front end is like

really sensitive to human eyes. You need

human eyes looking at the front end all

the time to make sure that it looks

good.

AI doesn't really have any eyes. It can

look at code,

but it front end is multimodal.

And so my experiences with trying to

plug AI into um let's say agent browser

or Playwright MCP to give it

You can give it tools to allow it to

look through a front end and sort of

look at images, but in my experience the

um it's not very good at that yet and it

can't create a nice front end in a

mature code base. It can sort of spit

one out. But what it can do is you say,

"Okay, uh I want some ideas on how uh

this front end might look. Give me three

prototypes um that I can click between

in a throwaway uh

throwaway route that I can decide which

one looks best." And you take the asset

of that prototype and you then feed it

back into the grilling session or you

get feedback on it, blah blah blah blah

blah.

Answer your question kind of thing?

The prototype is just, you know, it's

messy. It's supposed to give you

feedback earlier on the process.

So, that's a great way of working with

front end code, great way of looking at

software architecture in general. Let's

go one more question here. Yes.

>> [clears throat]

>> In your system, how do you integrate

respecting an architecture and design

with API contracts and fitting with your

larger system?

Uh security constraints, all kinds of

constraints like that.

Yeah.

There's a lot in that question. The

question was, how do you conform with

existing architecture? How do you do um

how do you make it conform to the code

standards

like of your code base or Yeah, the

architecture design APIs, Yeah. security

rules that constrain your design. Yeah.

I'm going to answer that in a bit.

That's okay.

So, hopefully we have started to get

some stuff cook cooking. Uh it's just

pinging on the explore phase here.

Hmm, tempted to just start running it

AFK.

Maybe I will, maybe I won't.

Um

What it's essentially doing is it's

exploring the repo. It's going to then

start implementing based on what we

wanted.

Let's actually have one more question

just while it's running. Yeah.

Why not AI

QA everything

Yeah.

So, the question was, why do you not get

AI to QA?

AI to QA.

I just got uh jargon overload for a

second. Um why do you not get AI to uh

test its own code? Now, of course, you

absolutely can. And I think while it's

doing while it's cooking here,

okay, it's got a clear picture of the

code base. It's assessing the issues.

It's doing issue 02 as the next task.

I'm again going to show you that in a

bit, I think. The sort of uh cuz you

definitely should do an automated review

step as part of implementation.

So, you have your implementation, you

should then, because tokens are pretty

cheap and AI is actually really good at

reviewing stuff, you should get it to

review its own code before you then QA

it.

I found that that catches a ton of

different bugs

and

the way that works is I will just do a

little diagram is if you have, let's

say, an implementation that sort of like

used up a bunch of tokens in the smart

zone,

if you get it to sort of try to

do its reviewing, it's going to be doing

the reviewing in the dumb zone.

And so, the reviewer will be dumber than

the thing that actually implemented it.

If we imagine this is the

uh let's be consistent. That's the

review.

That's the implementation.

Whereas if you clear the context,

then

you're essentially going to be able to

just review in the smart zone, which is

where you want to be.

Let's see how our implementation is

doing.

Okay, good. It's generating a migration.

That looks pretty nice.

We're getting some code spitting out.

And

while I'm sort of like Aha, here we go.

TDD.

Let's talk about TDD and then I think

we'll have a little another little

break.

TDD I found is absolutely essential for

getting the most out of agents. Uh raise

your hand if uh you know what TDD is.

Cool. Okay. TDD is test-driven

development. What it's essentially doing

is it's doing a something called red

green refactor. And if you look in the

code base, you'll be able to find a um a

skill which really describes how to do

red green refactor and teaches the AI

how to do it.

So, what it's doing is it's writing a

failing test first. So, it's saying,

"Okay, I've broken down the idea of what

I'm doing and I'm just going to write a

single test that fails and then I need

to make the implementation pass."

I have found that

first of all, this adds tests to the

code base and these this tends to add

good tests to the code base. And so,

we've got this kind of gamification

service.

It looks like it's

using some existing stuff to create a

test database. Test fails because the

module doesn't exist yet. Okay, we've

confirmed red. And then it goes and

hopefully runs it and it passes.

I found that uh raise your hand if

you've ever had AI write bad tests.

Yeah.

It tends to try to cheat at the tests

because it's sort of doing it in layers.

It will do the entire implementation and

then it will do the entire test layer

just below it.

Uh

I'm just going to say yes, you're

allowed to use NPX V test.

And using this technique, it generally

is a lot harder to

cheat because it's

sort of instrumenting the code before

it's then writing the code. So, I find

that TDD is so so good for places where

you can pull it off. In fact, it's so

good that I sort of warped my whole uh

technique around getting TDD to work

better.

I can see some dripping eyes. It is so

hot in here.

You can't imagine how hot it is up here.

Let's take another 5-minute comfort

break. Let's come back at quarter to, I

think. Have a nice generous one.

And we'll be back in about 6 7 minutes

and I'll talk about how

uh I think about modules, think about

constructing a code base to make this

possible.

I've just been sort of fiddling with the

AI here and we have ended up with some

with a commit.

So, we have something to test. Issue

number two is complete. Here's what was

done.

This is kind of what it looks like when

a Ralph loop completes is you end up

with a little summary.

Um and we have now something we can QA.

Because we did the feedback loops

because we did the trace bullets because

we were uh said, "Okay, give us

something reviewable at the end of

this." We can immediately go and QA it.

Now, there's nothing uh less exciting

than watching someone else QA something.

But, hopefully we can have a little

play.

Let's just check that it uh works at

all.

In fact, before I go there, I just want

to sort of work through what just

happened.

Which is we see that it's created some

stuff on the dashboard.

And it then ran the feedback loops. So,

it then ran the tests and the types.

Now, TDD is obviously really important.

And it's really important because these

feedback loops are essential to AI,

essential to get AI to produce anything

reasonable.

Because without this, AI is totally

coding blind, right?

You have to have to um

If if your code base doesn't have

feedback loops, you're never ever ever

going to get decent AI decent output out

of AI. And often what you'll find is

that the quality of your feedback loops

influences how good your AI can code,

essentially. That is the ceiling. So, if

you're getting bad outputs from your AI,

you often need to increase the quality

of your feedback loops.

We'll talk about how to do that in a

minute.

Now, so it ran NPM run test, NPM run

type check. It got one type error, and

it needed to fix it with a nice bit of

TypeScript magic. Very good. Yeah, type

of level threshold number. Okay.

Uh you see why I stopped teaching

TypeScript cuz just AI knows everything

now.

Um

So, and it ran the tests, and it passed,

and it's looking good. So, we now end up

with 284 tests in this repo. Pretty

good.

I I do find uh front end really hard to

test here. We're essentially just

testing the service. So, we've created a

gamification service, if we look up

here.

And then we have a test for that

service. You can see that the service

and the test itself.

Now, if I was doing code review here, I

would then go to I would first go to

review the tests, make sure the tests

were testing reasonable things,

and then go and kind of review the code

itself just to make sure that it's it's

not doing anything too crazy, right?

The essential thing is I need to

actually um look at the dashboard.

I'm going to log in as a student.

Oh, if it'll let me. Maybe it won't let

me.

Come on, son. There we go.

Let's log in as Emma Wilson.

Head into courses.

Uh let's say I've got an introduction to

TypeScript.

Continue learning.

Uh yes, I completed this lesson.

And something went wrong. I imagine it's

because I don't have

Uh SQLite error. I don't have the right

table. So, I need a table point events.

Point events is a strange table name.

I'm not sure quite what it was thinking

there.

Uh let's suspend. Let's run uh NPM DB

migrate.

Push, I think.

I can't remember which one it was.

But, you kind of get the idea, right? I

I'm not going to subject you to uh

watching me do QA because it's so dull.

Um but at this point, I would

essentially go back in. I would um

Let me open the project back up.

Uh and I would

This This is a crucial moment, um and

it's so important to um

QA it manually here because QA Oh, dear,

oh dear. What's going wrong? There we

go.

QA is how I then um impose my

uh

opinions back onto the code base, how I

impose my taste.

What you'll often find is that um there

are teams out there who are trying to

automate everything, like every part of

this process. And they will tend to

uh if you try to like automate the sort

of creation of the idea, automate

uh the QA, automate the research,

automate the prototype, you end up with

uh apps that I feel just lack taste

and are bad.

Maybe they just don't work, or they they

don't even work as intended, or there's

just no

You need a human touch when you're

building this stuff because without

that, you just end up with slop.

And we are not producing slop here.

We're trying to produce high-quality

stuff, and so that's what the QA is for.

Mhm.

So, I'm going to do two things in this

final section.

Which is I'm going to first tell you how

to

There's probably a question in your mind

here, which is let's say I have a code

base that I'm working on.

And it's a bad code base. It's a code

base that's like really complicated, uh

that AI just never does good work in,

and maybe actually most humans that go

into that code base don't do good work.

How what How do I improve that code

base?

And the second thing is I'll show you my

setup for parallelization.

So, let's go with um

bad code first.

Now,

where is it? Where's the diagram? Here

it is.

In his book, um The Philosophy of

Software Design,

John Ousterhout talks about

the ideal type of module.

And let's imagine that you have a code

base that looks like this. Each of these

uh blocks here are individual files.

And these files

export things from them. You know, they

have um things that you pull from the

files that you then use in other things.

And so, you might have these weird

dependencies where this file over here

might rely on this file, or might rely

on that file, for instance.

Now, if these files are small and they

don't kind of ex- like

export many things, then John Ousterhout

would call these shallow modules,

essentially. Where they're not very um

They kind of look like uh this, if I No,

actually no. I can't can't make a good

diagram of it.

They're essentially lots and lots of

small chunks. Now, this is hard for the

AI to navigate

cuz it doesn't really understand the

dependencies between everything. It

can't work out where everything is. You

know, it has to sort of manually track

through the entire graph and go, "Okay,

this relies on this. This one relies on

this one. This one relies on this one."

And it's then also hard to test this, as

well, because where do you draw your

test boundaries here?

Do you test each module individually?

Like just literally draw a test boundary

No, don't do that.

Around this one?

And then maybe another test boundary

around the next one, and then the next

one?

Or should you sort of do big groups of

it? Should you say, "Okay, we're going

to test all of these related modules

together, and just sort of, you know,

hope and pray that they work."

Now,

>> [sighs]

>> this means that if I think that bad

tests mostly look like that, where the

AI essentially tries to sort of wrap

every tiny function in its own test

boundary, and then just sort of test

that those individually work. But, what

that does is it means that when, let's

say, this module over here calls those

two,

so it depends on both of these, then

this module might miss order the

functions, or there might be sort of

stuff inside that poor module that's

worth testing on its own. And if you

then wrap this in a test boundary, what

do you do? Do you mock the other two

modules? How does that work?

So, actually figuring out how to um

build a code base that is easy to test

is essential here. Because if our code

base is easy to test, then our code our

feedback loops are going to be better,

and the AI is going to do better work in

our code base. Does that make sense?

So, what does a good code base looks

like look like?

Well, not like that.

It looks like this.

Where you have

what John Ousterhout calls deep modules.

Modules that have a little interface on

there that expose a small, simple

interface that have a lot of

functionality inside them.

Now,

what this means is that these are easy

to test cuz you just Let's say that

there's a dependency between this one

and this one.

My arrow working? Yeah, there we go.

Then,

what you do is you just wrap a big test

boundary around that one module, around

this one up here,

and you're going to catch a lot of good

stuff.

Because there's lots of functionality

that you're testing, and really the

caller, the person calling the module,

is going to have a simple interface to

work from. So, it's not not too tricky.

That makes sense? Deep modules versus

shallow modules. This is good.

This shallow version is bad. And what I

find is that unaided

um or if you don't

uh

if you don't watch AI carefully, it's

going to produce a code base that looks

like this.

So, you need to be really, really

careful when you're directing it.

And that's why, too,

is that if we look inside the PRD,

uh where is the PRD gone? It's inside

the issues. It's inside the gamification

system.

Uh not found. Of course, it's not. Here

it is.

Then I have

uh inside here

data model the modules.

So, it's specifically saying, "Okay,

this gamification service is a new deep

module, which we're going to test

around.

It's going to have this particular

interface.

And it's going to have um Okay, we're

modifying the progress service, too.

We're modifying the lesson route. We're

modifying the dashboard route, etc. So,

it's I'm being really specific about the

modules that I'm editing, and I'm making

sure that I keep that module map in my

mind at all times, throughout the

planning, and then throughout the

implementation. Does that make sense?

Very, very useful.

It's useful for one other reason, too.

Not only does it make your app more

testable,

but you get to do a little mental trick.

And I'm going to refill my water while

you wait for what that is.

Uh let me

Let me get a question from you guys. So,

raise your hands if you feel like

Uh if you feel like you're working

harder than ever before with AI.

Yeah.

Uh raise your hands if you feel like you

know your code base less well

than you used to.

Yeah.

This is a real thing. Um

because we're moving fast, because we're

delegating more things, we end up losing

a sense of our code base. And if we lose

the sense of our code base, we're not

going to be able to improve it, and

we're essentially delegating the shape

of it to AI.

I [snorts] don't think that's good. But

then how do we

how do we make it so that we can move

fast while still keeping enough space in

our brains?

I think that this is a way to do it.

Because what you're doing here is not

only are you thinking about creating big

shapes in your code base, big services.

What I think you should do is

design the interface for these modules,

but then delegate the implementation.

In other words, these modules can become

like gray boxes, where you just need to

know the shape of them, you need to know

what they do, and it's sort of how they

behave, but you can delegate the

implementation of those modules. I found

this is really nice. I don't necessarily

need to code review everything inside

that module. I don't necessarily need to

know everything of what it's doing. I

just need to know that it behaves a

certain way under certain conditions,

and that it does its thing. So, it's

kind of like

okay, I've got a big overview of my code

base, and I understand kind of the

shapes inside it, understand what the

interfaces all do, but

I can delegate what's inside.

I found that has been a really nice way

to retain my sense of the code base

while preserving my sanity.

Make sense?

And so, you might ask, how do I take a

code base

that looks like this

and then turn it into a code base that

looks like this? How do I deepen the

modules?

Well, we have Hopefully, it's in here.

Pretty sure it is. We have a skill.

And that skill is called improve code

base architecture.

Nice and direct.

Uh let's run it.

What this skill is going to do is it's

essentially just going to do it a scan

of our code base and looking for what's

available here. And feel free to run

this yourself if you're um

uh

running the exercises.

And it's exploring the architecture,

exploring um

essentially how to work within this code

base, and it's going to attempt to

uh find places to deepen the modules.

Pretty simple. One really cool um thing

that it found here is part of my uh part

of my course video manager app is a

video editor. A video editor built in

the browser, which is really hardcore.

Uh it's a decent bit of engineering. And

I wanted a way that I could wrap the

entire front end all the way to the back

end in like a single big module, so that

I could test the fact that I press

something on the front end and it goes

all the way to the back end. And so, I

found a way essentially by using a kind

of discriminated union between the two

types here by sort of I was able to use

this uh skill to essentially have a huge

great big module that just tested from

the outside, it was testable from the

outside, this video editor

infrastructure. And it meant that AI

could see the entire flow, could act on

the entire flow, and test on the entire

flow. And honestly, it was just night

and day in terms of the uh ability of AI

to actually make changes, cuz AI working

on a video editor is pretty brutal if

you don't give it good tests. So, that

is

Honestly, I

If you take one thing away from today,

just try running this skill

on your repo and see what happens.

Let's go to Slido. Let's ask a

check a couple of questions as well this

is running.

So, let's see. Have you tried Claude's

auto mode with Claude enable auto mode?

That way you can avoid many of the

obvious permission checks. We'll talk

about permission checks in a second.

Do I keep the markdown plans and issues

for later reference?

Okay.

This is a great question.

So,

let's say

that you uh have a great idea, you turn

it into a PRD,

raise and you then implement that PRD,

and the PRD is essentially done.

Raise your hand if you keep that

information in the repo, so you turn it

into a markdown file. Raise your hand if

you want to keep that around.

Cool. Okay. And raise your hand if you

if you don't want to keep it around. If

you want to get rid of it as soon as

possible. Yeah, this is I think an

a question that doesn't have a clear

answer.

What I'm really scared of

with any documentation decision is that

let's say that we have a PRD for this

gamification system, we keep it in the

repo.

We go on, go on, go on. Let's say a

month later, we want some edits to the

gamification system.

And we go in with Claude, and it finds

this old PRD and says, yes, I found the

original documentation for the PRD

system.

Well, it turns out that the actual code

has changed so much from the original

PRD that it's almost unrecognizable. The

names of things have changed, the um

file structure has changed, even the

requirements may have changed. We might

have actually tested it with users. This

is doc rot, where the documentation for

something is rotting away in your repo

and influencing Claude badly. Or Claude,

agents badly.

So, I tend to not keep it around. I tend

to get rid of it. And for me, because my

setup uses GitHub issues, I just mark it

as closed. It can fetch it if it wants

to, but it's got a visual indicator that

it's done. So, I tend to prefer

ditching these.

Thoughts on the BEADS framework from

Steve. Uh I've not tested it, but it

seems like sort of um another way to

manage Kanban boards and issues. Seems

uh very good, but I've not tried it.

Um

>> [clears throat]

>> Uh let me just quickly check the uh

setup here.

Let's take a couple of questions from

the room. Anybody got any questions at

this point about anything that we've

covered so far, especially this last

bit? Yes.

I thought it was

interesting your answer about like the

markdown files that you delete because

they

create like doc rot.

How about migrations? Like with

migration files, would you also squash

them after that?

Like database migrations? Yeah.

I don't know.

I hope that answers your question. I'm

so sorry. No, no. I think database

migrations are a different thing because

you have a sort of running record of

exactly what changed, and it's more

deterministic. And I think

Yeah, it's an interesting analogy. I'm

not sure. Let's talk about it

afterwards.

That's a good way of saying I've no

idea.

Yeah. Yeah. So, you mentioned that you

don't delete the PRD. You mentioned you

don't review the PRD once it's done.

Sorry, guys. Um I'm just trying to

listen to this guy's question. Have you

considered

uh using a deep think like ChatGPT or

something

to tell it, "Look at this PRD and tell

me if it

It takes about an hour.

Yeah, the question

The question here is um

should I um in the sort of early

planning stage be trying to optimize the

plan?

This is something I actually see a lot

of people doing, and it's a really good

um

idea. So, when you

Let's go back to the phases.

So, let's say that you have all of these

phases here.

And you

uh you get to the point where you've

sort of figured out everything with the

LLM, you understand where you're going,

you've created this sort of uh journey

destination documents here. How do you

then

uh

Like should you then try to optimize and

optimize and optimize that PRD until

it's the perfect PRD you can possibly

imagine?

I don't think there's a lot of value in

that.

Because I think the journey is really

just sort of a hint of where you want to

go, and the place that you need to be

putting the work is in QA.

And you can sort of do that AFK, I

suppose, but in my experience, you're

not going to get a lot of juice out of

it. Like it's the

The thing that really matters is getting

alignment with the AI, which is you do

in the grilling session initially.

Let's have one more question. Anyone got

any more? Yeah. How do you get in in

your workflow to get it to code the way

you want it to code it so by the time

you get to code review, it's at least

familiar, it uses the libraries you

wanted to use, Yeah. Um we had this

question before, actually, which was

like uh how do you uh enforce your

coding standards on the agents,

essentially? How do you get it to code

how you want it to code?

Now, there's essentially two different