Metric Forensics @Amazon & @Meta
"Why did the number move?" might be the most expensive question in analytics.
Most organizations answer it by searching: someone opens a notebook and hunts through tables until they find an anomaly large enough to blame. As the worldwide owner of this problem at Amazon, and later rebuilding it for Meta's ads funnel, I replaced that dive with two decomposition frameworks that account for a metric move exactly, then wrapped them in agents so anyone could run the analysis without waiting on my team.
Challenges
What I did
- 1ChallengesA metric moves and the best answer available is a plausible culprit, with no way to know whether it explains most of the change or a tenth of it.
- 1What I didContribution to Change, which splits the move into per-subgroup pieces that sum to the total exactly, with no residual.
- 2ChallengesThe real cause usually hides several dimensions deep, where a top-down dashboard scan dilutes it into ordinary noise.
- 2What I didLayered drilling, from region down to the single cell responsible, reconciling at every level of the descent.
- 3ChallengesHalf the movement comes from shifts in mix that no amount of staring at subgroup rates will ever reveal.
- 3What I didA mix effect reported beside the rate effect, so a subgroup that grew while performing below average becomes visible.
- 4ChallengesRevenue is not a rate, so the framework explained part of a miss and left the rest unattributed.
- 4What I didLMDI for absolute funnel metrics, which decomposes a chain of multiplied factors and still leaves nothing over.
- Role
- Data science lead: framework, tooling, and the executive read
- Ownership
- Worldwide single-threaded owner and coordinator for this work at Amazon
- Context
- Amazon retail pricing, then Meta ads sales. Generalized here
- Approach
- Contribution to Change for rate metrics, LMDI for absolute funnel metrics
- Delivery
- Python modules behind Claude Code skills, self-serve to the org
- Outcome
- Weekly headwinds at Amazon traced to exact causes, then fixed on the roadmap or explained to Jeff Bezos; about 30 reports replaced at Meta, saving roughly 30 weeks of data science time
A note on the numbers. This work was done inside Amazon and Meta, so no real data, target, or result appears here. Every figure below runs the actual decomposition formulas over datasets written for this page. The methods are exactly the ones I used and the arithmetic is live. The businesses in the charts are not.
The question that costs the most to answer
Measurement is the instrument panel of a large business. Amazon runs on how competitively it prices; Meta runs on how efficiently its ads business converts demand into revenue. In both places, the number on the dashboard is not decoration. Funding, headcount, and the shape of next year's plan follow it.
So when a metric moves, one question follows immediately, and it is always the same question: why. It sounds like a small ask. In a business with millions of transactions across dozens of regions, hundreds of categories, and several channels, the number of places the answer could be hiding runs into the thousands.
The cost of not answering it well is not the analyst's time. It is that leadership either acts on a guess or waits, and both of those are decisions with a price tag.
The default answer is a search party
The response I have seen most often from data science teams is to go looking. Someone opens a notebook and starts slicing tables, hunting for an anomaly large enough to be plausibly blamed. Sometimes it works. More often it becomes an open-ended dive with no definition of done, because nothing in the process says when you have found enough.
The deeper problem is that the output of a search is a story. A story names a plausible culprit and stops. It cannot tell you whether that culprit accounts for most of the movement or a tenth of it, and it cannot tell you what accounts for the rest. Presented to an executive, a story invites the only reasonable follow-up question: how do you know that is the whole picture? There is usually no good answer.
What a decision-maker needs is not a suspect. It is a ledger: every part of the business, the exact amount each one moved the number, and a total that reconciles.
Make the arithmetic do the searching
At Amazon I owned this work worldwide, using a framework we called Contribution to Change. The idea behind it is almost embarrassingly simple, which is exactly why it holds up in a room full of executives: a percentage metric is just the weighted average of its parts, so a movement in the whole can only come from two places. Either the parts changed, or the mix of them did.
Split every subgroup's movement into those two pieces and something useful happens. The pieces add up. Not approximately: the contributions across every region, category, and channel sum to the company-wide number exactly, with nothing left over. That turns root cause analysis from a search problem into a sorting problem. You are no longer looking for the cause. You have all of them, and you are ranking them.
Below is a working example on a synthetic retail dataset: 9.4 million product page views, and a price competitiveness rate that fell from 71.65% to 71.16%, a drop of 48 basis points. Click any bar to go a level deeper.
Price competitiveness moved from 71.65% to 71.16%, a change of -48.0 bps. These four regions explain all of it.
scroll the chart sideways →
| Subgroup | Rate | Mix share | Rate effect | Mix effect | Contribution |
|---|---|---|---|---|---|
| APAC | 70.2 → 70.7% | 18.0 → 16.5% | +7.8 | +2.2 | +10.1 |
| North America | 73.8 → 73.9% | 44.8 → 44.0% | +4.0 | -1.7 | +2.3 |
| Europe | 71.4 → 71.2% | 25.7 → 25.0% | -5.2 | +0.1 | -5.1 |
| LATAM | 66.1 → 63.5% | 11.5 → 14.6% | -38.4 | -16.8 | -55.2 |
Two clicks is the whole investigation. LATAM alone accounts for 55 basis points of decline, which is more than the company-wide number, because the other regions partly offset it. Inside LATAM it is electronics. Inside electronics it is the marketplace channel, where third-party sellers set their own prices. That one cell holds 2.3% of all traffic and accounts for 74% of the company-wide decline on its own.
That is a finding an executive can act on the same afternoon, because it names a region, a category, and a channel. It is also a finding that a top-down dashboard scan will never produce, because at every level above it the signal is diluted into something that looks like ordinary noise.
The layering is the part that scales. Because the formula is a single expression rather than a report, dimensions stack freely: region by category by channel by seller tier by whatever else the data holds. The arithmetic does not care how deep you go, and it still reconciles at every level.
The half of the answer nobody goes looking for
Splitting the movement in two is not a bookkeeping nicety. The second half, the mix, is the half that gets missed, and it is routinely the half that matters.
A subgroup can drag the company number down without anything about it getting worse. It only has to grow while performing below average. That is not a hypothetical: in the retail example above, North America improved its own rate and still gave back 1.7 basis points, purely because it is the strongest region and its share of the mix shrank. LATAM did the reverse and was punished twice: its own rate falling cost 38 basis points, and its growth into a below-average position cost another 17.
Taken far enough, this produces a result that looks impossible. Here is a business where every single customer tier improved and the overall number fell hard.
The contradiction
The resolution
Performance was worth +100 basis points. The shift in mix cost 327, leaving a net of -227. An analyst who reports that every tier improved is not wrong. They have simply answered a different question than the one that was asked, and the business would go into the next quarter believing it had a growth problem when what it had was a targeting problem. Those two diagnoses lead to opposite decisions.
At Meta, the metric stopped being a rate
When I moved to Meta, the framework hit its limit on the first question I pointed it at. Ads sales does not run on a percentage. It runs on revenue, and revenue is not an average of anything. It is a chain of factors multiplied together: how many accounts we work, how many opportunities each one yields, how often we win them, and what a win is worth.
Contribution to Change could explain the rate links in that chain and nothing else, which meant it could explain part of a revenue miss and leave the rest unattributed. That is worse than useless in a review, because a decomposition that does not reconcile invites exactly the doubt it was meant to remove.
The method that closes the gap is the Logarithmic Mean Divisia Index. It was developed for energy and emissions accounting, where analysts needed to say how much of a change in national emissions came from output growth versus efficiency versus fuel mix, and have the pieces sum to the actual change. That is the same problem shape as a sales funnel. LMDI weights each factor by the logarithmic mean of the segment's revenue in the two periods, and that specific choice of weight is what makes it come out exact. Older index methods leave a residual that grows with the size of the move, which is to say they fail hardest precisely when the question is most urgent.
What a leader gets out of it is a sentence with no hedging in it: this much of the revenue change came from the size of the book, this much from win rate, this much from deal size.
Revenue moved from $317.4M to $322.1M, a change of +$4.6M. Select any factor to see which segments moved it.
scroll the chart sideways →
The example above is a quarter that grew +$4.6M and should not be celebrated. Accounts contributed +$10.1M while win rate gave back $8.7M. The book grew because more accounts were worked, not because the business got better at converting them, and 67% of the conversion damage sits in SMB, the segment that expanded the most.
A revenue number alone says the quarter was fine. The decomposition says the growth was bought rather than earned, names the segment, and quantifies it. Those are different briefings and they lead to different plans.
Two frameworks, one question, one agent
I built Python modules around both methods, then wrapped them in Claude Code skills. That second step is what turned a technique into a capability, and it is the part I would repeat anywhere.
- Ratio metricsContribution to ChangeAnything shaped like a percentage: conversion, competitiveness, attach rate, on-time delivery. Splits a move into what each subgroup's rate did and what the shifting mix of subgroups did.
- Absolute metricsLogarithmic Mean Divisia IndexAnything that is a chain of factors multiplied together: revenue, bookings, total spend, hours delivered. Splits a move into one exact contribution per factor in the chain.
The skill runs the two in sequence, because that is the order the question decomposes in. LMDI first, to establish which link in the funnel moved the money. Contribution to Change second, aimed at that link, to find the specific corner of the business responsible. The output is a root cause analysis with the waterfall charts and the reconciliation tables already in it, written for someone who will read it once before a meeting.
The design decision underneath it is that the analysis is deterministic and the agent is not. The math is not something a language model is asked to perform or approximate; it runs in tested Python that either reconciles or raises. What the agent contributes is the part it is genuinely good at: understanding a question posed in a business's own vocabulary, picking the right decomposition and the right dimensions, and turning a table of contributions into the four sentences a leader actually needs.
What changed
At Amazon the effect was operational. The market never held still, so every week brought a new headwind. The framework told us exactly which one was behind each move and what it cost, and that turned each headwind into a specific response:
- Sellers learned to game our pricing systems. We adjusted the product roadmap to close the gap.
- Customers saw high prices when stock became low in their region. We rerouted stock from other regions to cover the imbalance.
- Supply chain delays left Amazon priced above retailers that still had stock. We worked with suppliers to prioritize restock shipments and set timeline expectations.
- Limited-edition items sold on Amazon only through resellers, at resale prices. We steered traffic away from them, toward better-priced items.
- Traffic grew in a market where the metric ran lower, pulling the worldwide number down even though no single market got worse. That was outside our control, so we reported it as such and moved on to the next problem without burning cycles on a non-problem.
A reconciling number ends an argument that a story only starts.
At Meta the effect was structural. Recurring reporting became automatic, with the visualizations a leader needed already built into the brief. The skill replaced about 30 reports that would each have taken a data scientist roughly a week to build, about 30 weeks of data science time, and it picked up regular weekly users across the org. More importantly, the analysis stopped being gated on my team. Anyone with an agent could ask what happened to their line of business and get a clear, objective, reconciled answer without filing a request or waiting for a queue.
That is the outcome I care about most. The routine question was the one consuming the most analyst capacity and generating the least insight, and it was the one most amenable to being solved once. Handing it back to the people who own the metrics returned data science time to problems that actually needed a data scientist.
Where decomposition stops
Decomposition is accounting, and it is worth being precise about what that means. It tells you exactly where a change came from. It does not tell you what would have happened otherwise, and it cannot tell you whether a proposed fix will work. Marketplace electronics in LATAM is where the points went; whether repricing that catalog would bring them back is a different question and needs a different method.
That is the seam between this work and my causal inference work. Decomposition tells you what happened, with no assumptions and no residual. Causal inference tells you what to do about it, with assumptions that have to be stated and checked. A measurement organization needs both, and it needs to know which one it is holding at any given moment. Most of the bad decisions I have watched get made came from mistaking one for the other.
About the figures
Nothing on this page is a mockup. Both decompositions are implemented in the browser and run over synthetic datasets: the waterfall re-solves the Contribution to Change formula every time you drill, and the funnel chart is a live LMDI fit over four segments and four factors. The residuals printed under each chart are computed, not typed, which is the only honest way to make the claim these methods rest on.