Orchestrating a Team of Models
Most agent systems pick one model and route everything through it.
That works until the graph gets deep, and then the cost shows up everywhere at once: in latency, in spend, and in how much of the system anyone can actually verify. Working on agent evals at Meta taught me you can only trust what you can see, so I built Hyperion around that idea. It treats model choice as a property of each step, and traces every call to measure what that buys.
Challenges
What I did
- 1ChallengesOne prompt fans out into dozens of nondeterministic model calls, each compounding the variance of the one before it.
- 1What I didEvery task planned as an inspectable graph, so any node can be opened, edited, and re-run on its own.
- 2ChallengesWhen every step runs through one model behind one interface, a bad answer can't be traced to the step that caused it.
- 2What I didEvery call traced end to end, with per-node timings, token counts, and failure rates I can audit rather than guess at.
- 3ChallengesRouting every step through one model pays premium latency and cost for work that never needed it.
- 3What I didModel choice treated as a property of each step, with an ordered fallback chain behind every role.
- 4ChallengesA pipeline stalls on its slowest step even when most of the work is independent.
- 4What I didA graph executed in parallel waves, so independent steps run side by side instead of waiting their turn.
- Role
- Creator & architect: design, build, operations
- Project
- Hyperion, the orchestration layer of my personal AI workspace
- Stack
- Python · FastAPI · LiteLLM · Qdrant · Langfuse · Next.js
- Background
- Led evals at Meta across 7 AI workflows built for thousands of sellers
Why I built it
At Meta I worked on AI agents from the measurement side, setting the bar seven workflows had to clear to launch. The lesson that stuck was simple: you can only evaluate what you can see, and you can only trust what you can evaluate.
A single prompt to a modern agent doesn't trigger one model call. It triggers several, sometimes dozens, of sub-agents working together, and because LLMs are nondeterministic, the variance compounds as each step builds on an already-uncertain upstream output. When all of that runs through one model behind one interface, a bad answer is hard to trace to the step that caused it, and every step pays premium rates for work that often didn't need them.
This is a big reason organizations still don't trust AI for high-stakes work like client-facing documents or reported metrics.
Hyperion is my answer, built with my hands on the whole stack: every step is a node you can open, routed to the model that fits it, and traced end to end.
Workflows you can open up
Point Hyperion at a prompt and it drafts a workflow on its own. But a workflow is never a black box: I can open any of them in a graphical builder, rewire which nodes feed which, swap the model or persona behind a node, and save the result as a reusable template.
Below is the real shape of one: five research angles running in parallel, converging into a synthesis step, splitting into specialist review passes, and resolving into a single verdict. Hover or focus a node to see its persona and the model it ran on.
scroll the diagram sideways →
A team of specialists
Every node is a persona, not just a model call. The market-sizing node reasons like a market analyst. The critic argues like someone whose only job is finding the hole in the plan. The assessor weighs critique against advocacy like a partner deciding whether to greenlight a deal.
Stacked together, a workflow behaves less like a single assistant and more like a small, focused team: each member narrowly scoped, briefed with just enough context to do its job, and reporting up to whoever synthesizes the final call.
Routing each step to the model that fits
Research that fans out into five parallel angles rewards breadth over depth, so those nodes run on gpt-4o with a capped search-and-reason loop. The steps that have to hold a lot of context at once and produce long prose (research synthesis, the advocate, the final verdict) run on gemini-2.5-pro.
The planner and developer nodes don't name a model at all. They point at smart and worker, logical roles chosen by intent rather than by vendor. Each role resolves to an alias, and an alias is a pool of interchangeable models spanning providers: smart holds claude-opus-4-6, gemini-2.5-pro, and gpt-4o. The proxy shuffles across whichever members are healthy, retries the others when one fails, and falls through to a different pool only if all of them do.
That indirection is what makes the routing claim more than a preference. Changing a pool's membership re-routes every node pointed at it, across every workflow, without touching code, and a provider outage degrades a run instead of ending it: the request lands on another member and keeps going.
The run below is that fallback caught in the act. No Anthropic key was configured, so both alias nodes were served by whichever of the remaining members answered. The trace store records the target a call asked for and not the provider that served it, which is why those two bars read smart and worker rather than a vendor model. Naming one would be a guess pretending to be a measurement.
Here is what that mix did on the clock across all 4 routing targets in a single run. Bars are colored by target.
scroll the chart sideways →
Same nodes, same models. 2.7× faster because independent work runs at the same time.
Why a graph, not a pipeline
A fixed linear pipeline forces every step through the same path in the same order. Building the orchestrator around an arbitrary directed graph instead means a workflow's shape can match the task: the runner groups nodes into execution waves, fires each wave concurrently, and only advances once every node in it has finished, so independent work runs side by side instead of waiting its turn.
The flexibility has real costs. A graph is harder to reason about than a straight line. Mixing models means each node carries a slightly different voice and judgment style that the synthesis step has to reconcile. And more parallelism means more surface area for one slow node to become the bottleneck everyone waits on. In the run above, the whole first wave waits on market-trends at 135.6s while four sibling nodes sit finished and idle.
What happens when a node misbehaves
Fanning work out to a dozen model calls means a dozen things that can hang, loop, or quietly burn budget, and runs here do fail. The per-agent rates in the monitoring view below are coarser than they look, because a failure is charged to every agent the run selected rather than to the one that caused it, so they rank exposure rather than blame. The three mechanisms here are what keep a misbehaving node from taking the run's budget with it, and each was added because something actually went wrong first.
- A per-request timeout, not just a wall-clock budget. An overall deadline can't interrupt a worker thread already blocked inside a completion call. Each node gets a hard timeout of min(remaining budget, 180s) handed to LiteLLM itself, the only knob that can actually kill a stalled upstream from inside the executor.
- Per-agent token caps checked before the call, not after. A researcher stuck in a search loop is stopped by a pre-flight check against its accumulated input and output tokens, so a runaway node fails fast instead of finishing expensive.
- One fallback retry per node. When a primary completion raises, the node retries once against its fallback target. Combined with the alias groups, a provider outage degrades a run instead of ending it.
Every call (model, tokens, cost, duration, and the node it belongs to) is written to a trace store and grouped into one Langfuse session per run. That store is where every number on this page came from.
What's next
Deciding which model belongs on which node is still mostly judgment. I have cost, latency, and token counts per node; I don't yet have quality per node. The next iteration closes that loop by running evals against each node's output to score models against the specific job that node does, so the routing table becomes a measured argument instead of a defensible guess.
Which lands back on the problem this started with: making sub-agent behavior legible enough that someone could trust the output.
The code
- agents/hyperion ↗the orchestrator
- runner.py: wave grouping ↗the DAG-to-parallel-waves core
- llms.py: per-node routing ↗model handles, caps, fallback