Demonstrating "agentic" for production use

Um a few introduction introductory words

about Alan. We didn't start as an AI

company uh and we're not selling AI. So

why are we here?

Alan was created to address the raising

challenges in the healthcare system.

From the start, our mission has been to

leverage cutting it cutting edge

technology to provide the best

experience to our members and help them

navigate the healthcare system that can

sometimes be complex.

Today we support 1 million number

members and we plan to support 10

millions uh in uh by by 2030. So AI is

not an option for us. It is a key

technology to make health simple,

integrated,

personalized for every single member.

2023 and the emergence of Genai were a

real turning point for for us. Um it's

where we decided to invest and go all in

uh with Genai and um we saw the early

signals that Genai could really really

transform uh the value that we provided

our members with. So we fundamentally

reexamined every single decision process

um product through the lenses of what AI

had to offer what it could uh enable and

we basically embedded AI into every

single layer of our company um decision

making daily operations services um

we reinvented everything. So um today AI

is a natural extension uh for our teams.

Uh it accelerates uh processes and it

ensures that we remain at the forefront

of uh the industry um the health

industry. So the results are clear and

measurable. Um well in terms of internal

productivity I I go very quickly on this

because I I think everyone here uses

coding assistance but AI is also

embedded in any single uh function and

process uh internally like sales um like

uh HR etc. in terms of operational

excellence and that's what we're going

to demo in a minute. Um we're using uh

AI um to um bring a better experience

for our members. So for example, we

reimburse every single care in five

minutes. Um sorry, no, most 60% of our

reimbursements are performed under five

minutes. Um we uh detect fraud with very

high precision and uh we prevent most of

fraud before it happens. Um

[clears throat]

and 35% of uh the questions of uh on our

customer service are uh answered uh

automatically

and um the third aspect of it is that

we've embedded uh AI in products in

services that we believe are making a

real difference for our members. So for

example, we've punered um medical

chatbots

um that obviously are always supervised

by doctors um that provide instance and

reliable health advice to our members

and this is really valuable to our

members today for access to health care

um everywhere and anytime.

Um we'll share today learnings and

decisions from our three years geni

journey. So not all of them uh obviously

but um what the message that we want to

convey is that um the journey wasn't

linear. Um we made a lot of experiments.

We made a lot of mistakes. We came back

on our feet. Uh we broke a lot of things

unfortunately without impact uh sorry

fortunately without any impact on our

members and um well

without entering uh into um a detailed

use case. I think a nice example of this

is um how we started uh with workflows

uh to feed uh our agents and we ended up

using playbooks and we're mentioning

that here because we know today the

there is still a a battle between

workflows and playbooks. Uh for instance

the agent builder from OpenAI is based

entirely on workflows. Um and we believe

that the best approach is really uh a

playbook with u inclusions of

deterministic in incursions. So that's a

learning we made along the way. I mean

we started with workflows every single

agent uh that we had um well followed

rules that were super deterministic and

it seemed a very solid starting point

when we started with AI and it works

pretty well uh with older models but now

that uh I mean uh the models have

evolved uh tremendously uh we think that

playbooks is uh a real advantage to

harness the full power of AI and which

doesn't prevent to include a few uh

deterministics um um uh to to blend pro

uh I mean deterministic uh controls

inside uh your playbook.

Another learning we made along the way

is um and we're really happy and curious

to to discuss with you later on the

cocktail to understand your perspective

about that. We started uh with um I mean

we considered uh using offtheshelves uh

agent orchestrators like langraph or uh

padentic AI and uh we finally decided

that it was better to build our own to

move faster, fill our needs and learn as

much uh and as quickly as possible. And

um well um this decision uh basically is

one that we made uh I mean um some time

ago and that we're reconsidering every

six months uh to understand whether uh

the evolution of the industry uh and um

on offshelf frameworks would allow us to

move faster or if we should continue

iterating on our own framework. I will

um leave the floor to um the demo. Um I

mean um happy to discuss uh this content

more extensively during uh the cocktail.

Um, and yeah, by the way, we're hiring a

lot.

>> Great. Thanks a lot, Gab. Um, I'm Alex.

I'm leading engineering at Alan and uh

today I will have the opportunity chance

to show you a bunch of things that have

been built by some engineers that are

with us. Uh today we have Alan's hat. So

feel free to ask any question afterwards

about uh maybe more details that you

would like to cover. Um initially we

also wanted to talk about how we use

Figma MCP but we only have a few minutes

today. So, I'm not going to be able to

cover that because we're thinking that

something that could be more valuable is

to tell you about a real story. A real

story about our first AI agents

currently in productions.

What problem does it solve? How we

operate that agent on a daily basis

and what we learn along the way?

So, one quick note before we dive in.

Everything that we're going to show you

today is from our test environment

because as you can understand uh for

privacy and security reasons uh we're

not going to show you anything from

production.

So first what problem we wanted to

solve. At Alan we support

around 1 million members and we want to

support even more to help our members to

have better life. So navigating health

care can be really stressful when they

reach out to us. We need to

help them sometime in during a difficult

moment and we need to guide them with

clarity, with empathy and with

efficiency.

The question they ask us are not easy

ones. They're not about how to reset a

password like you can see in some demos

or um things that uh can be built in a

few minutes. They cover a wide range of

complex and personal topics that demand

accurate answers.

So our members need instant answer

reliable guidance at scale. It was not

something that traditional approaches

can deliver can deliver efficiently.

That's why we build our AI agent to

solve that problem. Today,

our agent is able to answer between 30

and 35% of the question for our members

with a quality comparable to our care

experts.

So if we zoom in about the agent we had

to build,

we wanted to get multi-turn

conversations so member can provide more

details about the complex question they

have and step by step share more

details. Wanted to get escalation to be

sure that at any moment we can escalate

to a care expert. We wanted smart

direction to be sure that we're not

going to stay in a chat. We're going to

help our members to navigate across our

product to complete their um uh issues

or challenges.

And we wanted to make it personal to be

sure that we're going to be able to

really answer their questions based on

their guarantee the state of based on

the state of their reimbursement for

example. Um, and for that, uh, we wanted

to be able to manage a bunch of

different topics.

So, let's have a look at how it works.

So here you are on Alan and you're going

to ask a questions

as a member. So that's a a quick video

of a question from a members. The first

question they're going to ask is really

generic is about the fact that they they

went to multiple visits and they don't

know exactly what they need to do to get

their reimbursement. Our agent is going

to ask for more details to be sure that

we can really answer their questions as

a member tell us exactly that they want

more details about the fact if whether

they're going to be reimbured and uh for

um a specific GP visit and one about

their child. we can take a step back and

get some data from our back end to be

sure that we can provide a complete

answers and tell them what they need to

do to get here.

We continue to get more details about

their questions and oh it went a bit too

fast. Sorry about that. But there

let's try again.

Yeah. So there I'm going to pause

because we answer to our members about

their questions but we also add a link

so they can continue the discussion by

navigating to the correct part of our

applications. So when the user click on

it they go directly to the right part of

our application and they can complete

their requests. So there is a quick demo

of uh the current agent that we built.

I'm going to go back to

the Let me just check

if I'm right.

Okay, we're good. That's amazing. Um,

so what's under the hood? I guess uh

that's why we are here today.

So under the hood this agent has been

built by uh a multi-disiplinary team

that does great job to iterate and

that's the current architecture and

design we have today.

The first part is when we get the member

question we have a classification agent.

that agent has a clear objective is to

determine if they have enough details to

do an end off to a specialized agent.

The demo I've shown you is about our

claim agent that is able to tell you

about a reimbursement and so on. But the

main thing is that we have a first agent

to classify the questions and to

determine if we can call a specialized

agent.

our specialist agent, our expert agents

um they have a special playbooks

and um

they also have a set of tools.

So for that we use a react loop. So

react is a name for reason and act. So

for each exper agent first we're going

to analyze the conversations where do we

stand? Can we call tools to make the

answer personal? So for example, if we

go back to as the demo just shown you,

it's going to be okay, what are my

current reimbursement? Where do we

stand? Um, can I provide an answer?

Observe the results. Once we get the

results from the tools,

we injected the prompt to go back to

analyze the conversation and decide if

we can exceed that loop and generate a

final response.

Okay.

>> So with that approach we've learned a

lot and that's the current one we use

today.

We learned that for example to bring

more and more tools the agent have

started to sometimes struggle a bit to

make the right call with the right

arguments. You can think about okay as

an LLM how I'm going to provide UU ids

uh in a good way. So one of the best

practice to learn is to remove as many

parameters we can from our uh function

calls and to combine tools when they are

often used.

We also specify as much as possible the

different parameters we expect

and we provide robust error handling so

the agent can know when for example a

vehicle

uh one of the tool with the wrong

arguments.

Another aspect which is really

interesting is that models have also

evolved. So if we compare that loop and

our uh evaluation between now and let's

say six months ago for example um we've

noticed that agents are able to really

make uh tool calling in a way that is

way more efficient and way more reliable

and that makes a lot of sense when we

also uh chat more and more about MCPS

because that's where we want to go and

to give more and or tools to our agents.

So if we take a step back about the

design um there with a classification

agent as a specialist agent um we

started really small with only one agent

and we have been able to iterate really

fast. Um thanks to that we learned a lot

and step by step we added more and more

agents uh to uh our uh agent design

and it was super easy to communicate

also with different stakeholder saying

that okay we're going to take that part

of um the contact we got from our

members and we're going to automate that

part once we are happy with the results

we're going to move to another um wider

range of topics and step by step we have

been able to cover 30% uh 30 between 30

and 35% of uh the members questions we

have on the daily basis. But

architecture alone is not enough. Um,

another aspect which is key and I really

wanted to tell you a bit more about it

tonight is to provide tools

to also involve the domain experts

to configure and also to debug our

different agents and I think it was key

for the success of that agent.

So to do that, we're going to do a live

demo of our internal tooling. Uh because

I think that's not something we show a

lot. Um so I hope that the Wi-Fi is

going to work because we've seen in the

past that uh for live demos it can be a

challenge. And I'm going to let Gregoire

who is a product manager of the team who

has built all those tooling to tell you

more about it.

Cool. Hello. Thank you everyone. Um, so

as Alex said, um, what's very important

when you build AI agents is the overall

architecture, but is also the tooling

that goes around it. And I think you've

heard it a lot over the past two years,

but what matters most with AI agents is

garbage in equals garbage out. And so

very early on in our adventure, we were

quite convinced that we had to focus on

what was going inside our agents. What's

going inside our agents is basically

insurance knowledge and who are the best

people to know about insurance

knowledge. It's our customer service

operators.

And so we built some tooling um to allow

customer service operators to maintain

to improve agents over time and that's

what I'm going to show you. So first two

points before showing the demo as Alex

said it's in a test environment and the

second point all this tooling has been

built fully by engineers so without any

designers involved. So if you have

remarks, feedback on the design, you

have Shion and Dylan over there um that

you can question. So now let's go in the

demo.

Everyone has the same problem.

Okay,

so I told you about design. Um but this

is the first tool. It's called the debug

tool.

So this is a tool that we use u that

customer service operators use when they

want to answer the question what is

wrong with my agent? Why did it answer

in such a way? Um, and so what you see

here is um synthetic conversation

um between a member and our claims agent

and the member is basically asking um

whether they're going to be reimbursed

for their past care but also for their

child's recent care. So what the claim

agent is doing and you can see it here.

Um so what you see on this screen first

is the trace. So it's basically the

latency of each API call the agent

makes. That's mainly useful for

engineers to iterate on the latency and

improve it over time. The second thing

you see is all the steps that our AI

student takes to be able to answer the

member. So as Alex shown the first step

is the qualification agent which takes

as input a prompt carefully crafted by

our data engineers and then the history

of the conversation and based on that it

will output

a chain of thought and a class. You can

see its claims.

What happens next is a bit more complex

and it's related to the React loop that

Alex mentioned, but it's a set of

successive tool calls. So you have

around 10 tool calls here. What the

agent does is that the React loop will

ask the agent, what information do you

need to answer the member? And so what

the claim agent does is it says, okay,

I'm going to call these two tools first,

which is basically what are the care

events for the member? what are the care

events for the child of the member and

it will iterate on the success

successive tool calls until it has

everything it needs to answer the

member. What's quite magical about this

is that if you look at the last

iteration

that's the input that the LM gets. You

can see it's quite rough data. So you

see the different tool calls, you see

lists, you see JSONs and based on all of

that it's able to give a very simple

understandable answer for the member and

that's the magic between behind this

claim agent. What you can see further

down in the debug tool um is this

message

here. It doesn't it doesn't use any tool

call but it uses another interesting

feature which is redirection. And so

what it does here is that it understand

the member needs to upload the doc and

so it sends a redirection to the

reimbursement flow. So that was one tool

the debug tool. It's used when you want

to dig into a given answer. And once

you've had a look at 100 200

conversations that you've understood

what are the strengths and weaknesses of

your agent, you're going to use another

tool. And this other tool is called

agent configurations. I'm very happy to

take any uh recommendation for a better

naming. Um but on this tool

what you do um it's basically a CI like

system that is that allows us to trace

the different changes we make on our

agents and also it allows customer

service operators to test changes in a

safe environment and push them to prod.

So, how do I use this? First, I choose a

branch that's very specific to me.

Okay, I didn't choose the right one.

That's the magic of live demos, of

course. So, that's my branch. Then, I

can choose an agent, the claim agent or

one. And what you can see is that I can

modify different things. The prompts,

the member attributes, the redirections,

the tools. So, I'm going to show you an

example. Um,

I'm going to go into the prompt and I'm

going to delete the chain of thought.

That's not a good thing to do, but I'll

do it. And then I'll save.

So, here I'm still on my branch, so it's

safe. It's not going to be shipped to

prod. And so once I've done this change,

I can do two things. Either I can ship

it directly to broad if I'm very

confident in my change, but I can also

run an eval where I can select a data

set,

select an agent. So here the claims

agent and select the number of

conversations that the offline eval.

So here I've shown you two tools that

customer service operators use on a

dayto-day basis. We have other tools and

we'd be very happy to discuss them at

the cocktail. Um, but yeah, that was it

for the live demo. So now Alex is shown

to take the mic back.

>> Thanks. Uh, I'll try to make it quick as

I think uh we're a bit uh past time. Um

I think um an important part is what

we've shown today is really uh what we

currently use in in in production. I

think that was interesting to also tell

you what we are thinking of what we are

experimenting with. So right now one of

our challenge is uh to manage questions

that on those multiple topics um as you

can imagine as we qualify then pick a

specialist agent is not going to work

super well right now. So we've explored

different ways to address that

challenge. One of them is to use a

different design with um a manager and

orchestrator able to um call the

different agents.

But what is super interesting is um our

experimentation went super well. We've

been able really to solve the issue to

get uh multiple topics managed but it

also brings a lot of complexity a lot of

complexity about how we're going to

manage our tooling how we're going to

manage our eal and when you take a step

back it's only for 4% four to 5% of our

conversation right now so it's really a

question that uh we're thinking about is

it the right problem to solve but I

think it was interesting just us tell

you that right now it's it's available

but we've decided to not put it in

production yet.

Um so to to wrap up uh our presentation

for today I really wanted to highlight

uh three takeaways because what we've

learned by building this first AI agent

is that it's not only about the model

it's not only about the design it's

about combining the fact you have our AI

expert AI um domain expert our eval uh

and our design working together that

have been a su that have been key for

the success of that agent. So as

takeways really it's about what is the

problem you want to solve with AI and

with AI agent there are a lot of uh

discussion about we should use AI for

XYZ but the question is for which

problem if there is no real problem then

it doesn't matter what solution you're

going to get the second is to make it a

team effort because uh as you've seen

we've invested a lot in our internal

tooling to be sure that engineers or

data scientists are not going to be um

bottleneck or just be there to do prompt

engineering because it not makes sense

and we have really our domain experts

able to debug and configure the

different agents that was key for us to

scale and the last one is to invested

tooling. I think right now a bunch of

solutions. Some of them are mature

and what you can build also internally

to build that and you're going to learn

as fast as possible. So hope that was

useful. Thanks a lot uh for having us.