Uber’s CTO Praveen Neppalli Naga admitted in May the budget earmarked for Claude Code was “blown away already.” Box CEO Aaron Levie warned in March that as AI agents move beyond engineering into legal, sales, and other knowledge work, “compute budgets are just going to monotonically go up over time.” Ramp’s internal data shows enterprise token spend grew 13x between January 2025 and early 2026.
None of this should be happening. Per-token prices are in freefall. Andreessen Horowitz has documented this as “LLMflation.” That is, for equivalent model performance, inference costs decline 10x every year, faster than Moore’s Law, faster than broadband and faster than any prior technology commodity. What cost $60 per million tokens for a GPT-3 quality model in 2021 now costs $0.06, a 1,000x drop in three years.
The economics don’t align with the invoices. And that’s the story enterprise IT strategists need to understand before the next planning cycle.
The Paradox At the Center of Enterprise AI Economics
The same force that bloated software after Moore’s Law and flooded networks after broadband arrived is now at work in AI, presenting the latest example of Jevons paradox. The 19th century economist William Stanley Jevons observed that when steam engines became more efficient, coal consumption didn’t fall; it exploded, because efficiency unlocked entirely new uses. When a resource gets cheaper, people don’t consume less of it. They find ten new uses for it.
(BOY ANTHONY/Shuterstock)
Enterprise AI is a textbook case. Teams that once ran one model in production now run five. RAG pipelines that sent 50 tokens now send 4,000. Agents loop through dozens of calls to complete a single task. Junior developers prompt freely, iterating in ways that would have been cost-prohibitive a year ago. According to Gartner, agentic models require five to 30 times more tokens than standard chatbots.
Per-capita cost is falling, and the appetite for more tokens fueling AI is growing faster.
The Structural Cost Drivers Most Enterprises Aren’t Tracking
Tokens are the atomic unit of AI infrastructure, the way CPU cycles and bandwidth were for prior generations. Input tokens carry what you send. Output tokens carry what the model generates back. Output tokens cost significantly more, and the reason is physical: the model reads your entire input in a single computational pass, but generates output one token at a time, running a full calculation through the entire network for each one. A 1,000-token response requires roughly 1,000 separate passes.
That asymmetry shows up directly in pricing and it’s growing wider. Claude Sonnet 4.6 runs $3 input/$15 output per million tokens, a consistent 5x gap across all current Anthropic models. GPT-5.4 sits at $2.50 input/$15 output, a 6x multiplier, notably wider than its predecessor GPT-4o’s 4x ratio at the same input price. As models invest more compute in higher-quality output, the cost of generation is pulling further away from the cost of comprehension.
But headline token prices are only part of the story. The real budget killers are structural.
RAG pipelines are the biggest offender. A user types a 50-token question. By the time your system appends retrieved context, conversation history and a system prompt, that call is 4,000 tokens, an 80x overhead the user never sees and one your finance team may not be tracking.
Reasoning models add another layer. Models like GPT-5.4 Thinking and Claude’s extended thinking mode generate internal “thinking tokens” befo
re responding. These count toward your bill even though you never see them. A complex reasoning call can silently consume thousands of tokens before producing a single word of output.
Then there’s tokenmaxxing. Tokenmaxxing is a trend where teams treat cheap tokens as free tokens. Prompts grow bloated as developers add context they don’t need, conversations run long because nobody trims history and agents spin in loops chasing answers a single well-formed query could have produced. The Uber example is a textbook example.
(Shutterstock AI)
Model selection is where the multiples compound. Every major provider charges per million tokens, but the range is staggering. At the frontier, OpenAI’s GPT-5.5 Pro runs $30 input/$180 output per million tokens. At the budget end, GPT-4.1 Nano runs $0.10 input/$0.40 output. That’s a 150x spread on input and a 450x spread on output for tasks that are, in many cases, functionally identical. The cost impact in production is rarely that extreme, but it doesn’t need to be. At Acceldata, a recent controlled experiment using the same agentic coding task, same runtime and same sandbox illustrated this sharply. Two different models produced similar token volumes but costs of $1.20 versus $97. The only variable was model choice.
The Optimization Playbook: From Tactical to Architectural
Most enterprises discover their token problem when the bill arrives. By then the habits are set, the architecture is deployed, and optimization feels like retrofitting. It doesn’t have to be. The strategies below range from switches you can flip today to architectural decisions that compound over time.
Prompt caching: reused tokens. This is the highest ROI move for most teams. Caching repeated context delivers up to 90% cost savings on input tokens, with an additional 50% available through batch processing across both Anthropic and OpenAI. If your system prompt exceeds 2,000 tokens and you’re sending it on every call, caching it pays for itself.
Batch processing: discounted tokens. Both providers offer a flat 50% discount on non-real-time workloads. If a task doesn’t need an instant response, there’s no reason to pay real-time prices.
Model routing: right-priced tokens. Not every request deserves a frontier model. Route simple calls to budget models, and reserve frontier models for tasks that genuinely need them. Teams doing this consistently report 40% to 60% cost reduction.
(ultramansk/Shutterstock)
Prompt compression: fewer tokens. Treat every token as earned. Audit system prompts for instructions that no longer serve a purpose, tighten retrieved context so you’re not paying to send the model irrelevant material and write instructions that say exactly what they need to say and nothing more.
The zero-token option: when the best prompt is no prompt. Before optimizing how you prompt, ask whether you should be prompting at all. A web scrape via an MCP server costs thousands of tokens; a direct API call costs nothing. Anthropic’s engineering team has documented a five-server MCP setup consuming roughly 55,000 tokens before the conversation starts. In a recent benchmark of a federated query across Salesforce, Snowflake and ServiceNow under different architectural configurations, CData’s Amit Naik got a 97.6% reduction in token spend, from $0.596 to $0.027 per query, with the same model, data and prompt. MCP is a reasonable starting point for prototypes, but it’s rarely the optimal endpoint. Architecture decisions, such as what goes through the model versus what goes around it, often deliver more savings than any prompt engineering technique.
From Engineering Practice to Boardroom Mandate
If costs are falling 10x a year, why not just wait?
Because consumption is rising faster than prices are falling. Because the enterprises building cost discipline now are developing an engineering culture that compounds through leaner prompts, smarter routing and better instrumentation. And because the window where sloppy token usage goes unnoticed is closing fast.
The Linux Foundation’s newly announced Tokenomics Foundation, backed by Google, Microsoft, IBM, and JPMorgan Chase, signals that token cost governance is moving from engineering best practice to boardroom mandate.
When enterprise AI spend hits 25% to 50% of IT budgets at some firms, as Deloitte estimates, the CFO stops treating tokens as a line item and starts treating them as a strategic reckoning.
About the author: Mahesh Kumar is the CMO of Acceldata. As seasoned marketing leader and company builder, Mahesh has a track record of partnering closely with
CEOs to drive success. Mahesh specializes in taking products from obscurity to market leadership in sectors such as DevOps, Security, Cloud Computing, and Data Science/AI, leveraging his understanding of markets, customers, competition, and various business models like SaaS and Open-Core. Mahesh holds an MBA with a focus on Marketing and Entrepreneurial Management from The Wharton School, and a Master of Science in Engineering from Clemson University.
The post The Enterprise AI Cost Reckoning: Why Falling Per-Token Prices Aren’t Saving You appeared first on AIwire.