Summary. A shared inference budget kept hitting zero every ten days or so, and for a month we read that as the price of the work. It was not. Nine-tenths of every dollar was a reasoning model thinking, at eight times the input rate, about a task that needs no thinking. The setting that stops it is one line, and a sister codebase already had it. This is where the money went, the arithmetic, the fix, and the part that matters more than the fix: the only reason we found it is that the budget was small enough to fail.
#The wrong question
When an inference budget runs dry, the question teams ask is "how much should we top up?", as if the bill were a measurement of the work and the only variable were how much work you can afford.
Ours was not a measurement of anything. Two of our systems, Rippli and a second extraction pipeline, bill to one shared prepaid pool. Both read live financial news and pull the companies, people and relationships out of it: structured extraction, in volume, on a schedule. The pool kept draining to zero every ten to twelve days, at which point every extraction call came back 429 RESOURCE_EXHAUSTED and the pipeline quietly stopped. Over thirty days that was roughly 80,000 calls, 250 million tokens and about $358, almost all of it on one small, cheap model.
Cheap by the pricing page. The pricing page has two numbers per model, and we had been reading the wrong one.
A budget that never runs out is not cheaper. It is the same bill with the signal removed.
#Where the money actually went
| What broke | What we saw | The reflex | Actual root cause |
|---|---|---|---|
| The pool hit $0 every ~10 days | 429, extraction stops, top-up, repeat |
Raise the pool, or switch on auto-refill | The model reasons before answering by default, and reasoning is billed as output at 8× the input rate |
| A handful of calls cost hundreds of times the average | Nothing; they landed in the same total | Add retries with backoff, blame the provider | No output ceiling was set, so 57 calls ran to about 65,000 tokens, the model's hard maximum |
| One fallback model failed on every attempt | A circuit breaker cycling, a streak of 4,855 consecutive failures | Tune the breaker thresholds | The model had been decommissioned; the fallback was a dead address that kept being dialled |
| The same fix already existed next door | Nothing; each codebase configured its own calls | Write it in the wiki | The setting is chosen per call site, so one team's discovery never became the other's default |
Read the last column downward and the pattern is not carelessness. Every reflex in the third column is what a competent engineer reaches for first, and every one of them would have changed nothing, because the money was leaving through a setting none of them touch.
#A default is a decision someone else made for you
The model was doing exactly what it ships doing. Current small models from the major providers are reasoning models: before they answer, they generate a hidden chain of thought, and on the one we use that behaviour is on unless you turn it off. The part that costs you is not the thinking. It is where the thinking lands on the invoice. The provider bills those hidden tokens as output, and output is priced at $2.50 per million against $0.30 for input.
Our extraction call sent the model a chunk of article text and asked for a JSON list of the entities in it, at temperature zero, in JSON mode. That is a transcription task. There is nothing to reason about. The model reasoned anyway, at length, and then emitted a few hundred tokens of JSON. Per-call usage told the story once we looked: an average of 1,511 input tokens against 1,598 output tokens, on a task whose useful output is a fraction of that.
Run the arithmetic and the input side of the average call comes to about $0.00045; the output side to about $0.0040. Nine-tenths of every dollar was output, and most of the output was the model thinking about a job that needed none. The cost asymmetry is what makes this worse than it sounds: on a task with no reasoning in it, every thinking token is pure overhead, and it is the overhead that is charged at the premium rate.
#A missing ceiling is not a generous one
If you do not set a maximum, the vendor's maximum is your maximum. We had never passed an output limit on the extraction call, on the reasoning that a JSON list is self-limiting. Fifty-seven calls in the month disagreed, running to roughly 65,000 output tokens each, which is the point at which the model stops because it physically cannot continue.
In money those runaways were small, a few dollars across the month. What they told us was not small: the only bound on our spend per call was one we had never chosen. A ceiling is not a guess about how long the answer should be. It is a statement of how much you are willing to pay to find out, and a call that hits it is a call you want to see.
#The small budget was the diagnostic
We run a deliberately small prepaid pool, not an auto-refilling one. That constraint is the reason this was found at all.
Consider the alternative. With a large pool, or a card on file that tops up whenever the balance dips, the same five-fold waste would have been absorbed every month, indefinitely. The bill would have been "high". High compared to what? A metered bill with no reference point is not a signal; it is a number, and a number nobody has to look at gets paid. There would have been no incident, no date, no failure to explain, and therefore no reason to open the usage table.
Instead the pool hit zero on a visible rhythm, roughly every ten days, and each time it did, extraction stopped and somebody had to notice. A hard, dated failure is annoying in a way a large bill is not, and that is precisely its value. It forced the question "what is consuming this?" instead of the question "how much more do we need?". The answer was sitting in a table we already kept, and had never queried.
We do not throw money at problems, and this is the clearest case we have for why. A tight budget is not only cost control. It is an early-warning system that converts silent, continuous waste into a loud, discrete event you are obliged to investigate. The fix that followed cuts the same workload by an estimated 70 to 85 percent. Topping up would have hidden the bug and paid for it forever.
#A dead fallback is worse than no fallback
A fallback is a claim that something will answer, and claims expire. The extraction chain fell through to a second provider when the first failed. That provider had since retired the model we named, so every call to it returned "model not found", the circuit breaker opened, closed, tried again, and logged a failure streak 4,855 long. No money left through that path, but time did: a retry storm dialling an address that had not existed for weeks, on every extraction that needed a fallback.
The mechanism generalises. A fallback you have not exercised recently is not resilience; it is a belief about resilience, and the two are indistinguishable right up until the primary fails.
#What the fix looks like
Two settings on every extraction call, and one removal from the fallback chain. The shape, not the implementation:
# Structured extraction: a transcription task, not a reasoning one.
params = {
"temperature": 0.0,
"max_tokens": OUTPUT_CEILING, # a bound we chose, not the vendor's
}
if is_reasoning_model(model):
params["reasoning_effort"] = "disable" # -> thinkingBudget: 0 on the wire
What that buys is not a cheaper model. It is the same model, doing the same work, without the part of the work we were never using. The provider still has a reasoning mode; we simply stopped paying for it on calls that cannot benefit from it.
| Per average call | Before | After (projected) |
|---|---|---|
| Input tokens | ~1,500 | ~1,500 |
| Output tokens | ~1,600, mostly hidden reasoning | a few hundred, all answer |
| Output share of cost | ~90% | ~60% |
| Cost | ~$0.0045 | ~$0.0007 to $0.0013 |
| Time until the pool hits zero | ~10 days | ~2 months, on identical work |
One honest caveat on that table. When we shipped the change the pool was already at zero, so no live call could confirm it. We verified it offline instead, by inspecting the exact parameters the client library would put on the wire and confirming the reasoning budget was set to zero. The "after" column is arithmetic on the measured averages, not a measurement. The first live figure to check when credits land is the average output tokens per call; if that has not collapsed, the fix has not taken.
#What we got wrong
Rippli's provider had already set the reasoning budget to zero, with a comment explaining why. The second pipeline, billing to the same pool, doing the same kind of extraction, never adopted it. The fix was a search away in our own repositories for the entire month it was costing us.
The mechanism is worth more than the confession. The setting lives at the call site, so every place that calls a model makes the decision afresh, and a discovery made in one codebase has no path into the next one. We had treated "which model, at what temperature" as the whole configuration of a call, and left the vendor to decide the rest. There is no wiki page that fixes that. What fixes it is a policy: structured extraction runs with thinking off and a ceiling on, everywhere, and a call that wants reasoning has to say so.
The second thing we got wrong is quieter. We had a per-call usage table, with input and output tokens on every row, and for a month we did not run the one query that would have shown the problem. The bill was interpreted as the cost of the work because we had nothing to compare it to. Until you can say what a call should cost, you cannot tell an expensive month from a broken one.
#Before you top up
| Question | If yes | If no |
|---|---|---|
| Is the model a reasoning model with thinking on by default? | Decide, per task, whether you want it | You are not paying this tax; check the next one anyway |
| Does the task actually reason, or does it transcribe, classify or extract? | Keep thinking on, and give it a budget | Turn it off; it is overhead billed at the output rate |
| Is an output ceiling set on every call? | Runaways are bounded by a number you chose | Your ceiling is the model's maximum, and you will find that out on the invoice |
| Do you log input and output tokens per call? | Query them before you top up, every time | Add it first; it is one column and it is where every answer here came from |
| Can you say what one call should cost? | You can recognise a broken month | You can only recognise an expensive one |
| Does the budget refill itself? | Nothing in this document will ever surface for you | The next depletion is an investigation, not an inconvenience |
| Has each fallback model answered a real call in the last week? | It is a fallback | It is a belief, and possibly a retry storm |
The last two rows are the ones people skip, and they are the ones that decide whether any of the others get asked.
#When to skip this
- The task genuinely reasons. Ambiguous classification, multi-step synthesis, anything where the answer is not already in the input. Keep thinking on, set a budget for it rather than a zero, and measure whether the budget buys accuracy.
- A hard stop costs more than the waste. A consumer product mid-launch, a pipeline with a contractual deadline, a system where the outage is what a customer sees. There, a $0 stop is the wrong instrument. Use an alert threshold well below the balance and treat the alert as the incident. The signal is the point, not the outage.
- The cheaper alternative that would have caught most of this. Not a small budget. A single scheduled query, average output tokens per call per model, once a week, with a line in the chat when it moves. It would have shown the thinking tax in week one without ever stopping the pipeline. It would not have found the dead fallback, which never spent a token, and it would not have caught the runaways unless someone thought to look at the maximum as well as the mean.
- Our evidence is one pool, one month, two pipelines. The pricing asymmetry is public and the arithmetic is transferable. The 70 to 85 percent figure is a projection from measured averages, not yet a measured result, and it will differ on any task whose useful output is longer than ours.
#Key takeaways
- Check which side of the invoice hidden reasoning lands on before you choose a model, and assume it is on by default until you have read the parameter that turns it off.
- Disable thinking on every structured-extraction call, and make a call that wants reasoning say so explicitly.
- Set an output ceiling on every call. If you have not chosen a maximum, the vendor has chosen one for you.
- Log tokens per call and know what one call should cost, so that a broken month looks different from an expensive one.
- Keep the budget small enough to fail, or set an alert that fails for you. A bill that never surfaces is a bill you will pay forever.
- Exercise every fallback on a schedule. A fallback nobody has called recently is a retry storm waiting for the primary to blink.
Drawn from one month of shared inference spend across Rippli and a sister extraction pipeline, both billing to a single deliberately small prepaid pool.