The gap between a great LLM demo and a viable LLM product is usually not quality. It is the bill. Cost engineering is how you close it without lowering the quality bar.
TL;DR
- LLM cost scales with tokens × price × volume — and all three are levers.
- The big wins: prompt caching, model routing, and disciplined context budgets.
- Measure cost per request like you measure latency: continuously.
Why cost is an engineering problem
A feature that costs a few cents per call feels free in a demo and becomes a six-figure line item at scale. Unlike most infrastructure, LLM cost is directly proportional to usage, so success makes it worse. Treating cost as something to optimize after launch is how teams get a nasty invoice and a rushed scramble. Build it in from the start.
Lever 1: prompt caching
Much of a prompt is often static — system instructions, few-shot examples, retrieved context reused across calls. Prompt caching lets the provider reuse the processing of that repeated prefix instead of paying for it every time.
- Structure prompts cache-friendly: stable content first (instructions, examples), variable content last (the user's actual input).
- Reuse cached prefixes across requests that share a system prompt.
For high-volume apps with large stable prompts, this alone can cut input cost substantially.
The cheapest tokens are the ones you do not reprocess. Put the stable stuff first and let the cache earn its keep.
Lever 2: model routing
Covered in depth in intelligent model routing: send each request to the smallest model that clears the quality bar. Routing the routine majority to a cheap model while reserving the frontier model for genuinely hard requests is often the single largest cost reduction available — frequently more than half.
Lever 3: context discipline
Every token in the context window is a token you pay for. Bloated context is both a cost problem and a quality problem.
- Retrieve specifically. Pull the three relevant chunks, not the whole document.
- Summarize long histories instead of carrying raw transcripts forever.
- Trim system prompts to what actually changes behavior.
- Cap output length where you can; generation tokens often cost more than input.
Lever 4: avoid the retry spiral
A subtle cost sink is the fix-it loop: an agent produces something wrong, you re-prompt, it tries again, and tokens pile up. Good evals, clear specs, and verifiable targets reduce the number of round-trips it takes to get a correct result — saving cost and time at once.
Measure it like you mean it
You cannot optimize what you do not see. Instrument cost as a first-class metric:
Track per feature: tokens in/out · model used · cache hit rate · cost per request
Alert on: cost-per-request creep · cache-hit-rate drops · retry spikes
The takeaway
Cost engineering is not penny-pinching — it is what makes an AI feature survive contact with real volume. Cache the stable parts, route to the right-sized model, keep context lean, and reduce retries with better evals. Do that and you keep the quality your users feel while keeping the bill your finance team sees under control. That balance is the difference between an AI feature that ships and one that gets cut.