How much a token cost

I've been thinking about a better way to calculate ROI on LLMs, and needless to say that I started to think that tokens are the wrong unit we should be measuring, for the simple reason that a "smarter" LLM should get you to the same outcome with fewer turns (and perhaps also tokens) required.

In most "agentic workloads", most tokens are input tokens. These tokens are memory: the repo, the conversation, tool output, instructions, all re-sent every turn. The bill is the cost of remembering, and the multiplier is turns × context.

The full cost of a task is:

• turns to solve × context per turn × input price
• + generated tokens × output price
• + human turns × loaded cost
• + necessary rework

Let's take a debugging task at rough prices for a frontier model ~ $3 per million input tokens and $15 per million output.

• Fourteen turns at 60k tokens of context each: 840,000 input tokens $2.50
• 20k output tokens: $0.30
• Ninety minutes of an engineer's loaded time: $150

Now take the same task, but:

• Six turns at 40k with a warm cache: 240,000 cached tokens at a tenth of list price, $0.07
• 20k output tokens: $0.30
• Forty minutes of an engineer's loaded time: $65

The two sessions differ on two independent axes: discipline (cache / context) & depth and the token bill can't tell which axis the cheaper session is on.

On tokens alone, the first session costs roughly 8x more in tokens but most importantly, both numbers are noise in the total calculation, until the necessary rework requiring an operator enters the picture. If the cheaper session ships a bug that takes a day to find and revert, it could ends up costing $800 (1 day of engineering time + the original $65).

Writing it that way tells us two things:

• First, prompt caching cuts the price of re-sent context by roughly an order of magnitude on reads, writes pay a premium and a cold session pays list price. An operator who keeps a session cache warm and the context tight is buying a massive discount that compounds across every turn.
• Second, a cheap solve that leaves debt behind is just a worse result with a prettier number. So you can game tokens per solved task by staying shallow.

So the unit that fundamentally matters is cost per solved task, operator cost included. Not tokens used.

The developer as a fund manager

Give a developer a "token" allocation the way you'd give a fund manager capital, and let them deploy it. The job becomes solved tasks per unit of budget, where the budget is really turns, and turns are operator time.

The operator's work is an investment process: context discipline is position sizing, proportional verification is risk management, stopping is cutting losses.

A fund manager who can't explain why a position lost money doesn't get more capital. So naturally, a session that burned two million tokens and produced a revert deserves the same question. So refining the above, what matters is the trendline of cost per solved task, operator cost and rework included.

A thousand times faster

Suppose model prices fall a thousandfold, latency falls with them, and the cost of a solved task becomes a rounding error in the grand scheme of things. Does the operator still matter then?

A thousand times faster makes the operator term the only term there is. Speed doesn't make a result trustworthy and doesn't take responsibility when the change breaks production, however elegant the "loop" or the "software factory" that produced the output was. The obvious rebuttal is to "add a reviewer model", but here's my take. Suppose we always shipped software with machines reviewing machines and it worked nicely. Then production goes down and the postmortem asks who approved the change, and the answer is a log file. Someone then would propose adding a reviewer who might not be smarter than the LLM, and could perhaps even catch fewer bugs, but has to sign their name to the output and will have a worse week if the answer is bad. That would be sold as a breakthrough: a reviewer that's liable when nothing else in the loop can be because a human sign-off does for trust what the token cache does for context.

Those costs stay on the human side at any token speed. The work that becomes mandatory is verification on every commit, adversarial review as a matter of course, exploration that would once have been absurd to bill etc.

We have some consistent signal from DORA which measured across their surveys in 2024/2025 that as AI adoption climbed, delivery stability fell. A model that generates a thousand times faster hands you a flood of plausible changes, and the binding constraint moves to review, verification, and the judgment to reject. If rework is the term everyone forgets today, at a thousand times it's the term that eats the budget. To the question "if the models are a thousand times faster, will the operator matter?" this is my answer: the operator is the only thing left that does. A better operator is a competitive advantage.

Pricing gets stranger rather than simpler though. In Kendrick Lamar's song, a man at a gas station is asked for the smallest unit there is, ten rand, and haggles over it, and the haggling costs him everything the pump never showed. Every buyer of inference is now that man: per-million rates, cache discounts, tokens per task, because that's the number on the pump.

Meanwhile the labs meter something whose marginal cost approaches zero, while the value sits in decisions the same meter can't see, ie the full bill was technically never on the pump. I believe that they will end up selling something else: outcomes, seats, accountability, the assurance that the numbers are real.

So the question to the enterprising mind stops being "how much a token cost" and becomes who certifies the work and how can it be wrapped and packaged into a service.

What is there left to measure

• If inference is effectively free and instant, what does an engineering org optimize for?
• What does it hire for?
• What does the operator do all day when the model produces the artifact faster than anyone can read it?
• And what do we measure then?

The obvious objection is that "task solved" is a lagging metric. Code that passes review today causes the incident in January, so rework gets booked months late and the correct accounting is always behind. The number is slow, unglamorous, and specific to your codebase, which is why nobody sells it. An org that starts keeping it now has two years of track record by the time everyone else starts.

My candidate answers:

• Orgs will hire token portfolio managers, at whatever scale the tokens allow.
• Orgs will measure turns saved, rework avoided, and decisions made well on their own work, with their own people, against their own baseline building up a track record.

The artifact is: N tasks, cost per solved task, rework rate, before and after. It has no baseline and no leaderboard, which is why the number that would maybe settle this debate is still doesn't exist. The people who could perhaps write it have the least incentive to. Put a bit bluntly, the labs grade their own homework with benchmarks that don't represent the work we do so it's past time we graded ours.

The models will keep getting faster but not humans. Somewhere in that gap is the only metric that was ever going to matter, and it has a name: cost per solved task.

Do reach out or share your own take on this, we live in exciting and interesting times.