Lessons from building evals in Healthcare at Scale with Clara Matos

[music]

Next speaker is here [music] Clara

Matauss um from Sword Health and I'm

super interested to hear lessons from

building evals [music] in healthcare at

scale. So over to you.

>> So hi everyone, my name is Clara and I

work at Sword Health and today I'm going

to talk about lessons learned from

building with evolves in healthcare at

scale. So when developed and deployed

effectively, large language models have

the potential to increase access and

quality of care. However, we need to

guarantee that models remain safe,

reliable, and useful both before and

after they reach patients. A way to

achieve this is through both uh offline

and online evaluations. So you might

wonder why is evaluation in healthcare

hard? So models need to communicate

naturally with patients. They need to

respect clinical boundaries and they

need to fit within clinicians uh

workflows. So we need uh evaluations

that capture the complexity of care with

messy inputs and long-term outcomes. So

in this talk I'm going to cover how we

apply an eval driven development to

Phoenix our AI care specialist by first

giving an overview about uh Phoenix then

uh covering why are important and then

uh touching on both offline and online

evaluations so evaluations that are

performed before and after release.

So Phoenix is our AI care specialist

that is designed to bring clinical

expertise into every interaction. It

supports patients both both during and

between uh sessions.

Um so during uh sessions, Phoenix acts

as a true companion. It greets the

patient um sets expectations and follows

up on uh previous uh progress. As it as

exercises unfold, Phoenix answers

questions, responds to voice comments,

and provides feedback uh based on uh

motion uh tracking data. Um the outcome

is a smooth and true one-on-one

experience.

So between sessions, Phoenix continues

to play an active uh role by augmenting

clinicians

with by augmenting clinicians allowing

care to scale without compromising

quality among other among other things.

Fenix is able to uh make recommendations

to uh clinicians uh detect signs of of

disengagement and prompt clinicians to

reach out to members at the right time.

So everything that Fenix says uh when it

says and uh what it says is uh validated

both before and after release with

offline and online evaluations.

So first why are uh evolves important?

So the lack of evaluations has been a

key challenging in deploying large

language models to production. So

testing large language models as we all

know is hard and often slows down

development velocity. So we need to uh

measure the performance of these models

in real scenarios to increase confidence

as they are deployed at scale and in

high stake situations.

So what which are the main benefits of

evaluations? So first iterative improve

improvement is my V v2 better than my V1

also quality assurance before and after

um release. So uh has my latest model

introduced any regression?

Also objective model comparison. Can I

switch to another model and maintain or

even improve my current performance? And

finally potential cost savings. So can I

switch to a smaller model um and

maintain or even improve my current

performance.

So this is the process that we follow

and the one that I'm going to touch on

uh in the next slides. So I'm first

going to start by speaking about offline

evaluations, those that are performed

before release and then online

evaluations, those that are performed

after release.

So first offline evaluations. So when uh

developing with large uh language

models, we follow this iterative

process. So we first start by building

the evaluation data set by collecting

examples, defining the evaluation

criteria and uh performing human

annotation. Then we develop uh the a

suit of evaluators which can be called

human or llmas base. And finally before

release we perform a round of b check to

make sure that our release meets the

standards in quality and consistency and

at the same time that new issues that

are not yet covered by the evaluations

appear.

If all checks pass, we release the new

and hopefully improved version to

production.

So starting to speak about how to build

evaluation data sets. So when real data

is available, a common first step is to

collect examples at random. However,

this usually uh results in data sets

that are over represented with simple

and uh common use cases. In order to

avoid this, we try to create evaluation

data sets by gathering uh different

types of examples. So challenging

examples, rare examples, examples that

stress test different parts of the

application such as tool calling uh rag

or different prompt branches and um also

examples that are representative of

issues that have been previously uh

solved

for new features when no real data is

available. This evol data set can also

be built uh using synthetic uh examples.

Even when synthetic examples are uh

available um sorry even when real data

is available synthetic examples can be

used to uh create examples that are long

tail or that are rare and that haven't

yet appeared in uh production.

So this is an example of an item of the

evaluation that the set that we use to

evaluate Fenix conversations. So we

include the same context that is passed

to the model as well as the conversation

transcript between Phoenix and the

patient.

Over time these data sets is

continuously improved with new examples

collected uh either through user

feedback or manual audits.

Then we have we collect our examples.

The next step is to define evaluation

criterion. So for each feature we define

a set of uh criteria that define what

what success uh looks like. We always

try to keep outputs uh binary. So

something like yes or no, pass or fail

to uh reduce ambiguity and improve um

alignment. So this is an example um this

is an example of uh criteria that we use

to make sure that uh Phoenix remains

with an appropriate tone during

conversations.

Um so as we know the success of

evaluation depends on making sure that

these uh evaluators are aligned with

expert judgment. So in order to

guarantee that we perform human

annotation on the collected examples. So

we can use a single annotator or

multiple annotators. In our case usually

the clinical expert or a product

manager. And

um and then for each example of the

evaluation data set, this annotator will

provide uh feedback uh using the uh

evalation criteria that were uh

previously defined.

Uh so once we collect the examples, the

final criteria and perform human

annotation, we have our final uh

evaluation data set.

So the evaluation data set is built. So

what comes next? Uh we need to build the

evaluators. So we can build either human

codes called code cold or ludge evolves.

And each one of these grading uh

approaches is suited for different tasks

with different uh with um with different

failure modes.

So we usually use human evaluations to

evaluate uh things that require complex

reasoning and uh that require complex

reasoning and uh tone. So u this method

although being time consuming and uh

expensive and can suffer from

interotator disagreement is essential to

evaluate things that require expert

judgment and or product taste.

So before uh major uh updates or

releases, we perform pair wise pair-wise

comparison in Gondola, our uh internal

annotation uh tool where we ask uh human

uh evaluators, experts to evaluate the

model's output based on um based on

their um on their scores. We make

deployment decisions. So if a model is

ready or not to uh go to production.

Um then we use codebased evaluators to

evaluate things that have clear and

objective uh criteria. Their key

advantage is scalability. Uh created

once can be used to run thousands of

evaluators

quickly and uh consistently.

So this is an example of a code

evaluator that we use. This is a rather

simple one and that is used to make sure

that the output of uh the model remains

within a specific number of uh

characters.

So um then moving on to evaluators. So

they are a bit of a middle ground

between code based and human based uh

evaluators. Um so we so on one uh side

um they are able to evaluate things that

are more uh complex and subjective than

codebase evaluators and on the other

side they are able to uh process

thousands of evaluators

uh in a short amount of time.

So the key thing here with LLM uh as

judge evaluators is to create the LLM as

judge prompt. So this is an example of

the prompt that we use to evaluate Fenix

conversations. So if I was not standing

in front of it, you were able to see

that we define uh the task, we include

the transcript perform include also the

rubric item which which is which are the

evaluation criteria that I talked about

earlier and then the instructions for

the um evaluation. So in order to

guarantee that this um this evaluation

is aligned with human judgment, we

divide the evaluation data set into

three sets. The um training set,

development sets and the test set. The

training set is used for few short

examples. The development set is used to

iterate on the prompt and measure human

alignment. And then the test set is is

kept untouched until the end. and uh it

is used to have a measure of uh

performance.

So this is the process that we follow.

We start simple by defining the prompt

with clear structure and clear criteria.

Then we evaluate the examples of the

development set using the created

evaluators and then we measure things

such as true positive rate or true

negative rate and then we analyze

mistakes. So the examples where the

human and the model disagree. Then we

refine the evaluation prompt in order to

reduce the number of mistakes. And once

we and we repeat the entire process a

few number of times until we are uh

satisfied with the level of of

agreement.

Once the level of once the desired level

of agreement is uh reached uh we

evaluate performance on the test set.

So once the evaluation uh is created um

once the evalation suit is created, it

can be used to measure and compare the

performance of uh different versions

based on the uh created uh criteria and

they give us uh confidence about uh what

has improved or not on uh the new

version.

So however although we can spend a lot

of time in building uh evaluators

something that we cannot escape escape

from is veing.

So [clears throat]

um before releasing the model to

production we always or most of the

times perform a v check round which is

basically a manual inspection of uh

outputs. Um the goal is to ensure that

no unexpected issues not yet covered by

the evaluators uh surface if false

checks we release the hope the new and

hopefully improved version uh to

production.

However, although uh evaluations allow

us to track performance on different

criteria, they sometimes fail to uh get

the full picture of uh quality. So a new

version can sometimes improve the

evaluation metrics but uh fail the V

check the V check. In that case we hold

the release and perform another

improvement cycle.

So uh to perform V check uh we can use

annotation tools such as Gondola that I

described earlier or Argilla in this

which is the one uh shown uh here. So

using these tools the reviewers can have

the full uh context uh the full context

and provide feedback in a structured

way.

So this is the entire process used for

development once again. So moving on to

online evaluations. So once released to

production the model continues under uh

evaluation. Um so continuous evaluation

ensures that the model performance holds

up in real world conditions that

regressions are caught earlier and that

user experience keeps improving with

every uh iteration. So we keep a tight

feedback loop between production in

development by enforcing guard rails

conducting AB tests performing manual

audit on production traces and running

evaluators on production traces as well.

the the insights collected through this

process are then fed back to the um to

the evaluation data set the criteria and

uh used to make sure that uh they

continue to be reliable and aligned uh

with uh real world conditions.

So regarding guard wells, some uh

evaluators can uh be used as uh guard

wells um and they act as real-time

safety checks that prevent unwanted

content from reaching the end uh user.

So typically they can be implemented as

codebased evolves such as the

uh shown here uh for latency

considerations. However, for more

critical validations like for example

detecting uh medical uh advice, we can

also rely on LLM as judge. Uh so when

the guardrails fail uh you you can

either fall back to a static message or

uh retry the generation a few times.

Nevertheless, the ultimate test of the

new release is conducted in production

through an u AB test. So this validates

if the new version drives the user

behaviors and outcomes that we aim for.

So in practice uh performing AB tests

with large language models is not very

different from performing AB tests with

common uh products. We need to define

success metrics which can be uh user

feedback metrics or usage metrics or

acceptance metrics and then we measure

the lift between the two uh model

versions. However, performing AB tests

with large language models suffers

suffers from the same problems than

usual AB tests. So, first you need to

convince stakeholders to hold on on

releasing a new version and hopefully

better to the large population.

Sometimes your your uh volume of data is

not enough to uh make decisions, right?

No one wants to wait two months to have

the results of an AB test. And also uh

you need to make sure that the test is

statistically sound so you don't uh take

false conclusions. Uh nevertheless even

if everything I just uh spoke checks out

you still need to or most of the times

what happens is that the new version is

not uh different from the previous

version.

So um we set up uh guardrails, conduct

tests and continue to monitor um in

production. How? Through manual

inspection. So as we know manual

inspection although a bit tiredome is

one of the is one of the highest uh

return on investment tasks in uh machine

learning. So by regularly reviewing

production traces, we are able to detect

uh failure patterns early, understand

their root causes, mitigate the issues

and create evaluations that prevent them

from uh resurfacing.

However, to make this effective, we need

you guessed it um extensive

observability. So for each request or a

sample you need to or we usually log

everything from inputs metadata

uh from inputs metadata outputs uh uh

documents retrieved through rag

different tool invocations user feedback

and evaluations when uh they exist. So

this visibility allows us to have a full

picture of what happens in production,

debug faster and solve issues more

efficiently.

Finally, so uh once observability is in

place, the same evaluators that were uh

created um and that are used to evaluate

offline can also be used to evaluate uh

production uh traces. So depending on

the criticality of the issue, these

evaluators can be used to evaluate all

production traces or just a subsample.

So in the case of Phoenix uh which has

and supervised conversations with

patients, the same sets of evaluators

that uh runs in uh production that runs

in development also runs in uh

production. So if pass rates drop below

a specific threshold, the team is prompt

to investigate and uh take corrective uh

measure.

So this is uh once again the entire

process uh that uh we follow. So making

large language models safe, reliable and

useful especially in healthcare means

building trustworthy systems that

improve with every interaction. So

evaluations must be a continuous process

and not a one-time thing. We need to um

define success early create uh data sets

uh build suits of uh evaluators and

continue to run these checks in

production with guardwells AB tests and

observabilities

and [clears throat] observability sorry.

So as large language models grow more

capable uh the bar for safe reliability

and and helpfulness must increase as

well. So the features that endure are

not the ones that move fastest but the

ones that can improve uh with every uh

new interaction. Thank you.

>> [music]