Finding the Story @IBM
A supply chain can break in a thousand ways. Almost none of them have an alert.
A truck breaks down, a plant catches fire, a port shuts, a strike starts, a storm reroutes a shipping lane. Each of those reaches the world as news, written in words nobody set up a rule for in advance. As the data science lead on a team of six at IBM, I built a system that read a live global news feed and, rather than matching terms, found the stories forming around a client's suppliers and tracked each one as it grew, faded, and came back.
Challenges
What I did
- 1ChallengesThe events worth alerting on run from a factory fire to civil unrest to a currency move. There is no list of terms to watch and nothing to train on.
- 1What I didTreated it as structure discovery, not classification: surface whatever stories are forming around the client, then rank them.
- 2ChallengesAn alert has to say what changed since yesterday, which only means something if a story is the same object today that it was yesterday.
- 2What I didKept a persistent entity graph in Neo4j that accumulates as articles arrive, so a story is a thing in the database with a history rather than the output of the latest run.
- 3ChallengesA real entity belongs to several stories at once, and every clustering method available would force it into exactly one.
- 3What I didBorrowed the duplication idea behind Google's Splitter, so one entity can sit inside many stories without fusing them together.
- 4ChallengesA single wire story republished across dozens of outlets would have dominated the graph on volume alone.
- 4What I didMinHash in the ingestion path, tuned to collapse syndicated reprints while keeping independent coverage of the same event separate.
- Role
- Data science lead: method, algorithm, and the research write-up
- Team
- Two developers, one designer, two product managers
- Context
- IBM engagement for a global energy client
- Approach
- Streaming NER into a persistent entity graph, reorganized into stories on demand
- Outcome
- Proof of concept, written up as an internal IBM paper
A note on the figures. The client is confidential and the original outputs stayed at IBM, so no real feed, entity, or result appears here. The algorithm does: the graph figure below runs the actual betweenness scoring and node splitting in your browser, over a feed written for this page.
An alert for something you cannot list
The client was a global energy company with a supply chain to match. What they wanted was simple to say: tell us when something has gone wrong out there, early enough to do something about it.
The difficulty is in the word something. A supply chain takes damage from a truck breaking down, a factory fire, a dockworkers' strike, a typhoon, a bankruptcy, a border closing, civil unrest. These have nothing in common except their effect. Any list of things to watch for is a list of the disruptions that have already happened to someone, and the expensive ones are usually the ones not on it.
So the requirement was not a classifier. It was closer to the opposite: a system that could notice that something was forming without having been told what to look for, and put it in front of a person while it was still early.
One source wide enough to cover all of it
The best answer would have been the Twitter firehose, the live stream of every tweet being posted. It sees a factory fire before any newsroom does. It was also far outside what the engagement could pay for, so the practical answer was a commercial news feed: several hundred news and blog sources worldwide, arriving live. Slower than the firehose by some hours, and enormously cleaner.
I worked with the two developers on the team to build the path from that feed into the graph. Kafka to move articles between stages, MinHash to collapse near-duplicate wire copy, named entity recognition to tag entities and key phrases, Elasticsearch to hold the articles, and Neo4j to hold the entity graph itself.
- 01FeedSeveral hundred news and blog sources worldwide, arriving continuously.
- 02KafkaOne topic per stage, so a slow consumer applies backpressure instead of dropping articles.
- 03MinHashSyndicated reprints collapsed to one article, before near-identical copies can inflate a co-occurrence count.
- 04NEREntities and key phrases tagged per article. These become the nodes.
- 05ElasticsearchThe articles themselves, so any story component can be traced back to the coverage that produced it.
- 06Neo4jThe entity graph, kept and added to rather than rebuilt. This is what makes a story an object with a history.
A graph you keep
The obvious tool was a topic model, and the obvious objection to one is cost. That objection is weaker than it looks: streaming variants of LDA existed by 2010 and would have kept up with the feed. The real mismatch is what a topic is. It is a corpus-level theme, and it is one of a number of slots you have to fix in advance, which is a strange thing to have to declare about the world's events. I needed the opposite: a discrete thing that began on a particular Tuesday and could be compared against itself a week later. So I built something that produces objects rather than themes.
Nodes are entities and key phrases pulled by the tagger; an edge between two of them means they appeared in the same article, weighted by how often. The graph is never rebuilt. Each batch of articles adds nodes and strengthens edges in the graph already there, and old weight decays, so it accumulates and ages the way the coverage does.
The persistence is the part that makes an alert system possible. The question is never what is being written about, it is what changed since yesterday, and that only means something if the thing you are comparing is the same object it was yesterday. A story in this graph has an identity because it is built out of entities that are still sitting there in the morning.
One entity, many stories
Keeping the graph creates the problem this project actually turned on. Take the entity you care about, pull in everything mentioned alongside it, and you get one dense tangle, because the events overlap. The phrase shipping delays belongs to the port strike and the typhoon and the factory fire all at once. Ordinary clustering has to award it to one of them, and in doing so either fuses three events into one or breaks a real connection.
The idea that unlocked it came from Splitter, from Google's graph mining group. Their observation is that a node deserves more than one representation, because a person sits in a family and a workplace and those are different contexts. Their fix is a persona graph: duplicate a node once per community it belongs to, and let the copies live separate lives.
I did not use Splitter itself. It learns embeddings, and training embeddings on a live feed is exactly the recurring cost I had just designed my way out of. What I took was the underlying move, duplication rather than assignment, and turned it into something that runs directly on the graph: repeatedly find the entity holding the most unrelated things together and hand each of the contexts it was bridging its own copy.
Splitting the tangle
The measure of holding things together is betweenness centrality: the share of all shortest paths that run through a node. An entity with high betweenness is doing duty in more than one place at once, which makes it both the reason the graph is unreadable and the thing to duplicate first.
Below is the ego-network of a fictional client, Meridian Retail: 24 entities and 83 co-occurrence edges, containing 5 unrelated events. Step through the 4 splits and watch it come apart.
The ego-network around Meridian Retail: every entity the feed mentioned alongside it, joined wherever two entities appeared in the same article. Five separate events are in here. None of them is visible.
scroll the chart sideways →
The first split is the instructive one. Duplicating the client itself, by far the highest-betweenness entity, changes nothing: the graph is still a single piece. What was actually holding it together was the shared vocabulary underneath, phrases like shipping delays and component supplier, each of which belongs to several events at once. Splitting those is what makes it fall open.
After 4 splits, 24 entities have become 32 nodes across 5 components, and each component is one of the events that generated the data: port strike, supplier fire, typhoon, chip shortage, carrier bankruptcy. The client appears in all 5 of them, which is correct, and is precisely what a method that had to assign each entity to a single cluster could not have told you.
Nothing is deleted along the way. No edge is cut and no co-occurrence is discarded; entities are only ever copied. It is worth being exact about what that means, because it is easy to overclaim what just happened. The splitting did not create those stories. They were already in the graph, in the sense that every co-occurrence making them up had been sitting there the whole time. They were simply impossible to read, because a handful of shared entities welded them into one mass. All the manipulation does is take those few entities out of a load-bearing role they were never meant to have, and what falls out is readable as evidence: each component is a set of phrases that genuinely arrived together, from multiple independent sources, which is a much better description of a news story than a ranked list of words.
A story is a thing with a life
Splitting the graph once finds stories. Splitting the same accumulating graph again the next day is what turns a story into an object you can watch, because the components can be matched across days by the entities they share. The extraction is redone; the graph underneath it is the same graph, a day heavier. A story then has a size, a direction, and a vocabulary that shifts as new information arrives.
The first real one I found was IBM's acquisition of Red Hat. I was at IBM, so it was the obvious thing to point the system at, and the timeline turned out to be a near-perfect demonstration of why the shape of the method matters.
scroll the chart sideways →
- acquisition closes
- completed acquisition
- Jim Whitehurst
- Red Hat CEO Jim Whitehurst
- IBM CEO Ginni Rometty
- Ginni Rometty
- $190 per share
- closing price
- Nasdaq
- shares of tech stocks
- landmark acquisition
- deal completion
- distinct unit
- combined company
- largest software acquisition
- Armonk
- Raleigh
- investor reaction
- IBM
- Red Hat
- International Business Machines Corp
- open source
- hybrid cloud
- cloud computing
- enterprise software
- software company
The story sits at 8 phrases of background chatter, spikes to 28 when the European Commission clears the deal, then decays over the following week and a half to 9. On any reasonable reading it is over. Then the acquisition actually closes, the two CEOs brief the press, and it comes back larger than it ever was, at 39.
The second peak is the part worth pausing on. 79% of its vocabulary never appeared during the first: the regulatory language of the approval is gone and a product and market language has replaced it. A keyword rule written on the day the story broke would have been watching for the wrong words by the time the story mattered most, and it would have been the analyst's job to notice and rewrite it. Here the component simply grew new nodes.
That is the behavior the client actually needed. A disruption does not arrive fully formed with its final vocabulary attached. It starts as a few local reports, changes words as it develops, and the useful alert is the one that fires on a component growing fast, whatever words happen to be in it that week.
Where it ended up
The project ended at proof of concept, and I wrote the method and the findings up as an internal IBM paper.
It is also one of the most enjoyable technical problems I have worked on, and one I still think about. The IBM and Red Hat timeline came out of a system that had never been told a company acquisition was a thing that happens, and that is the part I would want to build on next.
What I would build now
The weakest part of what I built was the tagging. Off-the-shelf NER then labeled spans without linking them, so a company and a place sharing a name arrived as one node, and the key phrases it pulled were surface strings that knew nothing about two wordings meaning the same thing. That is precisely what a language model is good at now, and it is the piece I would replace outright.
I would go further than swapping the tagger. A model can read a handful of articles and say whether they are covering the same thing, which is a judgement my pipeline had to approximate with co-occurrence statistics, and a team of agents could carry that comparison across days. So the extraction and the topic assignment, the parts that decide what goes into the graph, are where I would put the model and where I expect most of the gain to be.
What I would keep is the graph. Not because a graph is the only way to do this, but because once you are tracking events over weeks you need somewhere to put them that holds identity, shows structure, and can be looked at. A story that grew today, shares four entities with one from last Tuesday, and has picked up a supplier nobody was watching is a natural thing to ask a graph and an awkward thing to ask anything else. The model decides what the graph contains. The graph is still what makes a story a thing rather than an answer.
The thread back to everything else
This is the earliest work on this site and the one that reads least like the others, but it is the same instinct. Give a decision-maker a structure that accounts for what it is showing them, rather than a model output they have to take on faith. Here it is components that keep every co-occurrence that produced them; in the decomposition work it is contributions that sum exactly to the number being explained.
The other thing it taught me, which I have used constantly since, is that a method has to have the right shape, not just the right output. Every technique I could have reached for would have answered what is in the news accurately, and the question was what changed in it. Those need different machinery, and no amount of accuracy on the first converts it into the second. That is the same reasoning I now apply to model choice inside agent workflows.
About the figures
The graph figure is the algorithm, not a picture of it. Betweenness centrality is computed with Brandes' algorithm, the neighborhood clustering and the node duplication run on every step, and the layout is a force simulation over the result, all in the browser over a synthetic feed. The story timeline is a reconstruction: the dates and events are the public record of the acquisition, and the phrase sets are rebuilt from the poster I presented at IBM.