Why 96% of Your AI Bill Is Rereading Old Tokens

The tokens you're paying for are almost never the ones you typed.

On a single working day, one operator I follow moved 3.77 billion tokens through their AI workspace. Of those, 3.59 billion were reused input. Ninety-six percent of the bill was the model rereading things it had already read. New content that day was a rounding error.

That's the shape of the problem nobody wants to name. It's also the shape of most operator AI budgets right now.

Why does your AI bill keep climbing when your usage feels flat?

Because language models have no memory. Every message in a conversation resends the entire prior thread from the top. Message one costs what you typed. Message two costs your text plus the reply plus message one. By message thirty, the new content is noise against the accumulated history.

The labs are not going to fix this for you. They are not incentivised to. Higher token consumption is higher revenue, and the pages reward the exact behaviour that wastes your budget. The framing that AI is getting cheaper per token misses the point. Per-token drops. Tokens per task climb faster. The bill goes up.

I've watched this pattern before, in a different wrap. Early cloud compute got cheaper per unit of storage and compute every year. Cloud bills went up every year. The unit got cheaper. Consumption got sloppier. Same story, new decade.

The operator lesson is the one most people skip: when the vendor economics reward waste, discipline is your job, not theirs.

What does token discipline actually look like at the desk?

Think of your AI session as a desk. Every conversation you don't close is paper you left out overnight. Every uploaded PDF is a folder you didn't file. Every tool integration is a stack of reference books piled next to the keyboard whether you're using them today or not.

Nine habits do most of the work at the individual level. None of them require new tools.

Edit your mistakes instead of correcting them in a new message. If your first prompt was unclear, use the edit button. A follow-up saying "that was wrong, try again" doubles the cost of that exchange, because the model now reprocesses the original mistake plus your correction plus its wrong answer.

Bundle related questions and specify the output format. Ambiguity is expensive. If you want 150 words, say 150 words. If you want bullets, say bullets. The model will otherwise burn tokens hedging across formats.

Start a clean thread when the job changes. Long conversations are efficient for the same problem and expensive when the problem shifts. Starting fresh doesn't zero out overhead, because system instructions still load. What it does stop is the last three hours of unrelated work tagging along on every new message.

Carry the answer forward, not the argument. When you move from research to drafting, pass the finished artifact into the next stage. Not the drafts. Not the criticism rounds. Not the rejected sources. The one-page brief, not the twelve rounds it took to produce.

Ask only for the answer you need. Output costs twice. Once when it's generated, then again as input on every subsequent turn. A 50-word answer stays cheap. A 50-page paper haunts the rest of the conversation.

Search your own files when you can. Letting the model rummage through a document folder is convenient and expensive. Pre-identify the relevant passage and paste it in.

Send the lightest useful form of the source. If words matter and layout doesn't, convert the PDF or screenshot to plain text before pasting. Uploading raw PDFs and images is comfortable and burns tokens quickly.

Keep your sources organised before sending them. Same principle as any other workflow. The mess costs money downstream.

Store answers somewhere retrievable. If you keep asking the model the same thing, you're paying to recompute an answer you already had. A basic knowledge store solves this.

None of that is exotic. It's the AI-era version of the same operating discipline that separates firms with margin from firms without it. Which brings me to the harder problem.

Why doesn't cleaning up help as much as you'd expect?

Because a lot of the waste is upstream of anything you type.

Every tool you connect to your AI client sends its full description to the model before your request lands. Name, purpose, arguments, schema. Anthropic has published data showing a typical setup with connectors for GitHub, Slack, Sentry, and Grafana burns roughly 55,000 tokens in tool definitions before Claude has read a single word of your question.

Think about what that means at the operator level. You have a team of fifteen using an AI client. Every one of them has connected the same handful of integrations because it seemed useful last quarter. Every message every one of them sends carries 55,000 tokens of tool preamble that has nothing to do with today's work. That's not a usage problem. That's a configuration problem, and it compounds silently across every seat, every day.

Load only the tools the job can actually use. Turn the rest off. This is one of those settings decisions that quietly determines whether your AI spend is defensible or embarrassing.

Then there are the native features most operators don't touch. OpenAI supports compaction for long-running work, which carries forward state with fewer tokens. Anthropic supports context editing, which clears old tool results and reasoning blocks before the next request. Neither is a perfect reproduction of the full history. Both are far better than hitting a wall or paying to reread a debugging session from four hours ago.

Use the smallest model that will do the job. Not every question needs the frontier tier. Most don't. Operators default to the biggest model out of habit, the way early cloud users defaulted to over-provisioned instances. Same tax, paid the same way, for the same reason.

If you or your team are building on top of APIs, prompt caching is a real lever. Cache the portion of the request that doesn't change. Stop resending the same 20,000-token system prompt on every call. This is developer territory, but the operator implication matters: if your team is building AI features on top of provider APIs and nobody has looked at caching, your infrastructure bill is probably twice what it needs to be.

Where does the real use sit?

The uncomfortable truth is that habits and settings help, but they can't shrink a request that's already been assembled. By the time your prompt reaches the model, the standing instructions, the tool definitions, the conversation history, and your new message have already been packaged and sent. The meter is already running.

The next move is an intermediary layer. Something that sits between your AI client and the model provider, inspects each outgoing request, and decides whether the request even needs to reach the model in its current form.

An intermediary can answer directly from a local knowledge store when the answer already exists. It can run a deterministic recipe locally when the task doesn't need a language model at all. It can forward only the passages that matter instead of the whole document. It can enforce a hard cap on outgoing request size, which is the only real defence against the accidental ten-million-token call that shows up in your billing dashboard the next morning. It can stop a request entirely when the request is malformed.

The interesting rule at this level is not the technology. It's the mental model. Most AI usage today treats the model as the first line of response. Send everything, let the model figure it out. The operator move is the opposite. Treat the model as the last resort. Answer from cache first. Answer with a script second. Answer with a small, tightly scoped model call third. Answer with a frontier model call only when the first three couldn't.

That flip alone probably ten-times the value of your token spend. It's the same discipline any experienced operator applies to any expensive resource. Don't burn the expensive input on work the cheap input could do.

What's the operator takeaway?

Three things, in order of how quickly you can act on them.

First, the biggest waste in your AI bill is almost certainly not what your team is asking. It's the accumulated context they're not clearing, the tools they're not disconnecting, and the sources they're uploading in the heaviest possible format. That's cleanable this week, without new tools, without new vendors.

Second, the settings your team hasn't touched are more expensive than the settings they have. Compaction, context editing, model selection, tool loading. These are configuration decisions with meaningful P&L impact. Somebody on your team should own them the way somebody owns your cloud infrastructure config.

Third, the firms that will run AI cheaply in eighteen months are the ones building an intermediary layer now. Not because they're clever. Because they're doing what every generation of operators has done when a new expensive input arrives: they're putting a governor on it before the bill teaches them to.

The AI labs are going to ship better tools. Compaction will improve. Context management will improve. Caching will get smarter. None of that removes the underlying job, which is the same job it's always been. Know what your inputs cost. Know what you're spending them on. Don't pay for the same work twice.

The desk stays clean because someone decided it would.

---

If this is the kind of operator note you want more of, subscribe to The Operator's AI Brief. One email, written for people running real operations under real constraints.