Setting the Launch Bar for AI @Meta
How seven teams agreed on what “good enough to launch” means for an AI agent, and how they'd prove it.
Meta's ad sellers had an AI agent to help with their work, but nothing defined what it should do well, and nothing proved that it did. So the org defined seven workflows for it, gave each one a team, and required every workflow to pass its own evals before it could reach sellers. I was the data scientist on the small team set up over all seven, and my job was to make their evals add up to one standard: one way to define quality, one way to measure it, one bar to clear, and one place to see it all.
Challenges
What I did
- 1ChallengesSeven teams were about to invent seven definitions of good enough, each with its own vocabulary and its own bar.
- 1What I didOne gating process, one set of eval types, and one dashboard, so a number meant the same thing on every team.
- 2ChallengesClose to 100 things could go wrong across the workflows, from the wrong name on an email to a product the recommendation model never picked.
- 2What I didEvery criterion its own pass-or-fail eval: a code check wherever a source of truth existed, a tuned LLM judge where one didn't.
- 3ChallengesA typo and a wrong dollar figure are not the same failure, but a single pass rate treats them as if they were.
- 3What I didThresholds tiered by stakes, set by asking every team the same question: how often is it OK for this to fail?
- 4ChallengesA hand-labeled set can't predict live traffic, and a workflow that launches clean can still drift.
- 4What I didThree gates: a human-labeled golden set, two weeks of shadow traffic, then monitoring that can pull a workflow.
- Role
- Data scientist on the program's alignment team: eval methodology, gating, and the dashboard
- Teams
- 7 workflow teams (a PM, data scientist, data engineer, and engineers each), plus our team of four; 30+ people
- Built for
- An AI agent for thousands of Meta's ad sellers
- Scope
- 7 workflows, close to 100 eval criteria
- Outcome
- Every team building to the same gates, eval types, and dashboard
- Status
- Pre-launch when I left Meta
A note on the examples. The workflows, evals, and dashboard are internal to Meta, so every example, number, and screen on this page is synthetic. The structure is the one the teams built to.
An agent nobody could vouch for
Meta's ad sellers had an AI agent to help with their work, but it had no defined workflows and no evals to say whether it did any of them well. For a tool that drafts what an advertiser reads, that is a risk nobody can size.
So the org defined 7 workflows for the agent to own, three of which carry the examples on this page: drafting a follow-up email to an advertiser after a call, recommending which products to pitch, and building the client presentation itself. Each workflow got its own team, a PM, a data scientist, a data engineer, and at least one software engineer, to build both the workflow and the evals that would gate its launch. An eighth team of four, a PM, a data engineer, a software engineer, and me as its data scientist, sat over all seven to make sure those evals added up to something.
The risk was easy to see. Left alone, seven teams build seven eval systems: different words for the same failure, different bars for the same risk, and no way to say whether one workflow was closer to launch than another.
What good looks like, workflow by workflow
I started with a working session with each team: what goes in, what comes out, and which parts of the output can't be wrong. The teams brought what they'd learned from talking with sellers about what matters in their work, and reviewing the workflows' early outputs surfaced failures nobody had thought to list.
Some criteria were common to every workflow: grammar, tone, factual accuracy. The specific ones mattered most. A follow-up email has to be addressed to the person who was actually on the call, and every number it repeats has to match what was said. A presentation has to read as one argument, not a stack of slides. And the products the agent pitches have to be the ones the recommendation model put at the top, not ones the agent found plausible.
Across the 7 workflows, that came to close to 100 criteria, and not all of them deserved to hold a launch back. Teams could mark a criterion track only: measured and charted on the dashboard, but never gated on. Generic measures like sentiment often landed there, where they could inform the work without blocking it.
Code where there's an answer, a judge where there isn't
Every criterion became its own eval with a pass or fail verdict, and the most consequential decision about each one was what kind of eval it should be.
Where a source of truth exists, the check should be code. An advertiser's spend is in a database, so the figure in the email can be compared against it exactly. The recipient is in the meeting record. The recommendation model's output is logged, so a pitched product is either near the top of it or it isn't. Checks like these are cheap to run, never disagree with themselves, and fail with an exact reason.
Where there's no single right answer, like whether a presentation flows or an email's tone fits the relationship, the check is an LLM judge. Each judge asks one narrow question and returns a binary verdict, and each was tuned against a human-labeled golden set until its verdicts matched people's, then scored on labels held out from tuning, so no judge was ever graded on an example it had been tuned on. Judges handle nuance code can't, but they take several rounds of prompt work to get right and have to be watched for drift, so they were the tool for what code couldn't do, not the default.
Is there a source of truth for the answer? Yes: check it with code. No: ask an LLM judge one pass-or-fail question, tuned against human labels.
- Spend in the email against the billing record
- Recipient against the meeting record
- Pitched products against the recommendation model's output
- The presentation reads as one argument
- The email's next steps match the call
- The tone fits the relationship
- A golden set of labeled examples per workflow
- The labels every judge was tuned on, with a held-out share it was scored on and never tuned against
The same question decides how the workflow itself should be built. If a value can be checked against a source of truth, the agent usually shouldn't be writing it at all. Let code insert the spend figure from the database, and a whole class of failures disappears, along with the eval that was watching for it. The model's job narrows to what only a model can do: the language around the facts, not the facts.
It's the line I draw in my metric forensics work too: the math is deterministic, and the agent is not.
How often is it OK for this to fail?
A single pass rate treats every failure the same, and they aren't. A clumsy sentence and a wrong dollar figure in front of an advertiser are different kinds of mistake, and a launch bar that can't tell them apart is either so strict no workflow can clear it or so loose nobody can trust it.
So I gave every team a rubric to set a threshold for each gated criterion at each stage of the launch process. Instead of asking for a target pass rate, which invites round numbers, it asked one question per criterion: how often is it OK for this not to pass? Framed that way, every team was reasoning about the same thing, the cost of a failure reaching a seller, and a threshold on one team meant the same thing on any other.
“Spend figures match the database” in the follow-up email
“Reads as one argument” in the client presentation
“Grammar and spelling” in every workflow
“Sentiment” in every workflow
The thresholds went into the dashboard beside the live pass rates, so the distance between every criterion and its bar was always visible. That made prioritizing a matter of reading: the next piece of work was the gated criterion furthest below its line, and progress reports wrote themselves.
Three gates to production
No test set predicts live traffic, so the bar had to be cleared on samples that got bigger and more realistic each time. Each workflow's golden set of human-labeled examples was run several times over, so the pass rate reflects how much the agent varies from run to run. Two weeks of shadow traffic then supply the volume a hand-labeled set can't, which is where a threshold set on curated examples meets the real distribution.
- Gate 1Gates launchGolden setHuman-labeled examples, run several times eachRun on every change. Repeating each example shows how much the agent varies from run to run, not just whether it passed once.
- Gate 2Gates launchShadow trafficTwo weeks of live requestsThe workflow runs silently beside sellers, and its outputs are compared against what they actually produced. This is where the volume is, and where the read gets precise.
- After launchCan sideline itProduction monitoringEvery output, for as long as the workflow runsThe same evals keep running. A gated criterion that falls below its bar sidelines the workflow until it's fixed.
One dashboard for seven teams
Close to 100 criteria across seven teams is too much to hold in anyone's head, so I built a dashboard in JavaScript to hold it instead. It tracked how far along each team was in building its evals, charted every eval's pass rate over time against its threshold, and was built for diagnosing regressions: segment by workflow or by a single eval, then open the agent traces and telemetry behind a failing run to see exactly where it went wrong.
scroll the chart sideways →
Judge's reason: The draft says 50%; on the call the advertiser said about 15 percent.
Because every team's evals shared the same verdicts, tiers, and thresholds, one dashboard could show all seven side by side, and “how close is this workflow to launch?” became a question anyone could answer without asking.
Where it stood
By the time I left Meta, all seven teams were building to the same gates, the same kinds of evals, and the same dashboard, and the workflows were still building toward launch. What the program had was the part that's hardest to retrofit: an agreed definition of good enough, a way to measure it that every team trusted, and a path to production that didn't depend on anyone's confidence.
I'm building the same discipline into my own product from the first commit: every change to the agent is versioned, gated by evals, and released in stages. Evals cost the least when they're designed in from the start, rather than retrofitted onto a system people already depend on.
About the figures
The workflows, criteria, and dashboard are internal to Meta, so every figure here was built for this page. The criteria are representative of the kinds the teams wrote, the thresholds are illustrative, and the dashboard is an illustrative example running on synthetic pass rates, with a fictional advertiser in the failing run.