Ten million tokens for one conversation

We had a conversation costing us ten million tokens.

One conversation. A recruitment chatbot walking a candidate through onboarding — somewhere between a hundred and a hundred and fifty turns. In the worst cases, ten million tokens to get to the end of it. That’s not a business. That’s a science experiment with the meter running.

Here’s how we got there, because I think the story is the useful part.

The first version was built with all the bells and whistles and no limits. That was deliberate. When you’re getting a conversational pipeline to hold together structurally — to actually walk someone through onboarding and come out the other end with the right result — you don’t want to be fighting your own tooling at the same time. So we built freely. We handed the agent loose, powerful tools — the bash tool, whatever let it just do the thing without us wiring up plumbing first. And it worked. The pipeline held together and hit its acceptance criteria.

But it had no legs. It couldn’t scale, and it wasn’t anywhere near cost effective. A thing that works once, expensively, in a demo isn’t a product. So once the shape was proven, we pulled it apart and built the real one.

That second build was almost entirely about taking things away.

We went back to function calling — proper schema-based tool calls instead of handing the agent a shell and hoping. A defined function with a defined shape does one job, and you know exactly what it costs. A loose tool is convenient for you today and expensive forever.

We broke the whole thing into routines. Instead of one big agent carrying the entire conversation in its head, each stage became its own small, bounded piece. Then we went stage by stage and boiled each one down to its minimum — the least context, the fewest tools, the smallest prompt that still did the job. The rule was: cut until it breaks, then put back only the one thing you needed. Never compromise the feel, never miss the acceptance criteria the pipeline demanded. Everything else was fair game.

That meant stripping the harness chaff — all the config and scaffolding that rides along by default and quietly costs you tokens on every single turn. MCP servers a stage never touched. Tools it would never call. If a stage didn’t need it, it didn’t load it.

We got serious about caching — structuring the context so the stable parts stay put and actually get cached, instead of reshuffling the prompt every turn and paying full price for tokens we’d already sent.

And we made the whole thing model-agnostic. Once each stage is small and bounded, you can ask a different question of it: does this stage even need the big model? Can it run on something cheaper, or a different provider entirely? When the work is chopped into routines, you can put each routine on the cheapest thing that clears the bar.

None of this is glamorous. It’s the opposite of the bells-and-whistles build. But it’s the difference between a pipeline that demos and a pipeline that ships.

The first version taught us what the conversation needed to be. The second version made it something you could actually run — same feel, same result, a fraction of the cost.

Build it loose to find the shape. Then take out everything the shape didn’t need.