What Moves the Metric @Meta
Sales committed to a new north star metric, then set goals against it, with no evidence behind either.
Leadership needed to know which plays would move it, and the rigorous way to find out took a full quarter per question. As the data science lead on a team of three, I architected an agentic causal inference framework that turned data we already had into direction executives could act on, then took the strongest findings to sales leadership as strategic recommendations with the evidence attached. Sales acted on them, changing which ad solutions it led with in each region and how it reached companies of each size.
Challenges
What I did
- 1ChallengesThe accounts that got a given play were never chosen at random; reps picked the large, healthy ones.
- 1What I didPropensity weighting to rebuild the comparison a randomized experiment would have handed us, with the balance checked in the open.
- 2ChallengesThe rigorous answer, a live experiment, costs a full quarter and buys exactly one hypothesis.
- 2What I didAn estimate in minutes instead, so the quarter goes to the hypothesis that has earned it.
- 3ChallengesOne average effect across a global advertiser base is a real number and a useless instruction.
- 3What I didCausal forests, which estimate a separate effect per subgroup and show where the lever quietly backfires.
- 4ChallengesMaking a causal estimate cheap to produce makes a bad causal estimate cheap to produce too.
- 4What I didBalance, overlap, and interval diagnostics that travel with every brief instead of sitting behind it.
- Role
- Data science lead: framing, method, framework, recommendations
- Team
- Me plus two data scientists
- Context
- Meta Sales analytics, generalized here
- Approach
- Causal forests over observational data, wrapped in an agent
- Outcome
- 50+ hypotheses tested in minutes each; sales plays reprioritized by region and company size
- Status
- Handed off to a team extending it into a multi-method agent
A note on what is and is not here. This work was done inside Meta. No proprietary data, metric name, or result appears on this page. Every figure below runs the real estimation pipeline over a simulated population of 12,000 accounts written for this page, where the true effect is known. The method is exactly what I used. The numbers are not Meta's.
A goal with no map
When I joined, Meta Sales had just moved to a new north star metric. The metric was chosen and the goals against it were set. What nobody had done yet was establish how the metric responded to anything sales actually did. No experiments, no causal work, no sensitivity analysis behind it.
That is a worse position than it sounds. Leadership could see the number move but could not attribute the movement to any decision they had made, so planning had no evidence to draw on about which plays mattered. An organization can be perfectly disciplined about a number it has no way to influence and still miss it.
The rigorous answer takes a quarter
The question landed on my team: which levers can sales pull to move this metric, and by how much? I led that work, with two other data scientists alongside me.
The textbook answer is live experimentation, and it is the right answer. It is also the slow one. A sales experiment needs a full quarter to read out, and a quarter buys you one hypothesis. A business at Meta's scale is not going to hold still for three months per question, and we had dozens of them.
So the requirement was never just "measure this." It was: measure it with data we already had, repeatably, on a timeline where the answer still changes a decision.
Rebuilding an experiment out of history
Causal inference is the family of methods for exactly that constraint. The techniques vary, but they are all variations on one idea: take data where the treatment was never randomly assigned, and reconstruct the comparison a randomized experiment would have handed you.
The obstacle is always selection. The accounts that got a given play were not a random sample. They were the ones a rep chose to spend time on, which means they were larger, healthier, and further along to begin with. Compare them to everyone else and you measure the head start, not the play.
The fix is mechanical: model each unit's probability of having been treated from its covariates, weight the two groups so they look alike on everything you can observe, and then check that they actually do. The check is the part that matters. An effect estimate is worth precisely as much as the balance diagnostics behind it, which is why those diagnostics belong in the figure rather than in an appendix.
The adjustment removes 96% of the naive comparison's error. The remainder is the part no amount of weighting recovers, and it is the reason a live experiment stays the tiebreaker.
On this simulated population the raw comparison reports +6.32 when the truth is +2.33. Nearly two thirds of that headline is the head start. Weighting removes 96% of the error and lands at +2.15, which is close but not exact: the residual bias is larger than the confidence interval around it. That gap is honest and worth stating, because it is the reason a live experiment stays the tiebreaker rather than a formality.
One average effect is a bad instruction
The simplest causal methods return a single number for the whole population: this play lifts the metric by so much. That is a fine sentence and a poor instruction. Meta's advertiser base is not one population. It spans regions, company sizes, industry verticals, and wildly different levels of product maturity, and a play that lands in one segment can do nothing in the next and cost you in a third.
Average those together and you get a recommendation that is technically correct and operationally useless. What a sales team can act on is not one effect. It is which accounts to run the play on.
scroll the chart sideways →
The average here is +2.15. The subgroups underneath it run from -1.47 to +4.95. Acting on the average means running the play across 10% of accounts where it measurably costs you, and under-investing in the segment where it is worth more than twice what the headline promised.
Causal forests: the tree trick, pointed at effects
A causal forest is a random forest aimed one step sideways. An ordinary tree splits the data to separate units with different outcomes. A causal tree splits it to separate units with different responses to treatment. What comes out is not a prediction of the metric. It is a map of where the lever works.
Two properties made it the right choice for this problem:
- Honesty. One half of the sample chooses the splits; the other half estimates the effect inside each leaf. Skip that and a tree will hunt down the subgroup where noise happened to look like a large effect, then report the same noise back as a finding. Honest estimation is what makes the confidence intervals mean anything, and it is the difference between a subgroup analysis and a fishing expedition.
- It is an ensemble. A single tree is fragile: resample the data and the cut points move. Averaging over many trees means what survives is structure rather than one lucky partition, and the parts that do not survive show up as wider confidence intervals instead of quiet errors.
Underneath both sits a third property, borrowed from double machine learning: the outcome and the treatment are each residualized on the covariates before any effect is estimated, so errors in those two nuisance models do not pass straight through into the treatment effect.
scroll the tree sideways →
The same fit on three bootstrap resamples. Every one of them splits on product adoption first and isolates the same underperforming region, and every one of them puts the cut point somewhere different. That is the argument for the forest in one image: trust the structure that repeats, not the thresholds of any single tree.
The framework, and then the agent
The estimator itself is published research. The work is everything around it, and that is where the week actually went: preparing and validating the data, fitting the nuisance models, tuning the forest, running the diagnostics, and turning the output into something a non-specialist could act on. I built a Python framework that handles all of it end to end.
Then I wrapped the framework in a Claude Code skill. An analyst states a hypothesis in plain language, and the skill runs the full analysis and returns an executive brief: what the effect is, where it concentrates, how confident we are, and what the diagnostics say about whether to believe any of it.
The trade-off is the obvious one. Making a causal estimate cheap to produce also makes a bad causal estimate cheap to produce, and a weak analysis dressed in a subgroup table is very persuasive to a room that wants good news. So the diagnostics travel with the answer rather than sitting behind it: covariate balance, overlap between the treated and untreated groups, leaf sizes, confidence interval widths. A brief should let a reader see the estimate and the reasons to doubt it at the same time.
What changed
A hypothesis that took upwards of a week of setup, fitting, and writing became a few minutes of prompting and reviewing. That turned testing into something the team did in parallel rather than in sequence: we worked through dozens of hypotheses, and I took the strongest findings to sales leadership as strategic recommendations with the evidence attached.
Sales acted on them. Different ad solutions were prioritized in different regions, and the way reps reached a company (email, phone, video call, or a mix of them) was matched to its size. Those decisions now had evidence behind them.
None of this replaces experimentation. Live experiments still validate, and where a decision is expensive enough, they should. What the framework changes is which experiments are worth running. Instead of burning a quarter to discover that a hypothesis was never promising, a team can spend an afternoon ranking the whole set and give the quarter to the one that earns it. That makes the experiment queue an informed decision, and it is a capability any business can use.
Handing it off
Causal forests were right for this problem. They are not right for every problem. The shape of the data and the shape of the question decide the method, and choosing it is where the judgment lives. My framework assumes that choice has already been made.
Closing that gap was the natural next step, and it grew into scope for others. I handed the framework to a larger team to build on with more hands, so the technique could be chosen to fit each question rather than assumed. These are the methods that choice runs between.
- Causal forestMany units, rich covariates, and a real chance the effect differs across them.
- Difference in differencesA staged rollout, where some groups get the change before others and you have history on both.
- Synthetic controlOne market or segment switched, and the comparison has to be built from a weighted blend of the ones that did not.
- Regression discontinuityEligibility turns on at a threshold, so units on either side of the cutoff are otherwise alike.
- Instrumental variablesSomething shifts treatment without affecting the outcome by any other route, which identifies the effect where adjusting for covariates alone cannot.
That handoff was the point. A framework only its author can run is a personal productivity gain. One that another team can extend is infrastructure. What made it extensible was the seam between the estimator and everything around it, so swapping causal forests for a different method does not disturb the data preparation, the diagnostics, or the brief.
It is the same problem as the one in my orchestration work, one level up. There, the mistake is picking one model and forcing every step of a task through it. Here, it is picking one estimator and forcing every question through it. Both are solved the same way: choose per problem, and make the choice inspectable afterwards.
About the figures
The tree, the subgroup effects, and the balance diagnostics on this page are not illustrations. They are the output of a working pipeline, run in the browser over a simulated population: a logistic propensity model, an outcome model for local centering, an outcome transformed by the propensity residual as the estimator itself, and an honest causal tree fit on split samples. Inverse probability weights are what the balance plot is drawn with. Because the population is simulated, the true effect is known, which is the only reason a page like this can show you an estimate and its error at the same time.