Reduce LLM API Costs in a Live Product: What Works

To reduce LLM API costs, first record tokens and cost on every call, per customer and feature. Then send fewer tokens, route simple tasks to cheaper models, move work that can wait to batch APIs, cache the stable start of each prompt, and enforce limits per plan. Measure first; the costly feature is rarely the one you expect.

TL;DR: Put every model call behind one metered function. Trim context before anything else. Pick models per task and test the switch on an evaluation set. Batch APIs from OpenAI and Anthropic cost half the synchronous price, and cached input tokens cost a fraction of normal ones. Then make plan limits part of the product.

Why do LLM costs surprise software teams?

They surprise teams because the cost grows with use, which most software doesn't. Hosting a web app costs roughly the same whether a customer clicks once or a thousand times. A model call costs money every time, and a feature that's cheap in testing can become the biggest line on the bill once one power user finds it.

This post goes through the levers in the order we'd pull them, with examples from SigmaCPM, Kraftr, and NoteTube. Prices and model names change fast, so we link each provider claim to the provider's own page as of October 2026.

How do you track LLM cost per customer and feature?

Track it by routing every model call through one function that records who, what, and how much. You can't cut what you can't see. SigmaCPM records tokens and cost on every LLM call, and that single habit answers the questions that matter.

1. Which feature costs the most?
2. Which customers cost more than they pay?
3. Did last week's prompt change make things cheaper or dearer?
4. What does one document, one conversation, or one image cost?

Record at least: customer, user, feature, model, input tokens, output tokens, cached tokens, cost, and latency. In NoteTube, AI keys live only in a separate FastAPI service, so every call already passes through one place, which is the natural spot to meter it. We describe that setup on the FastAPI AI backend page.

How do you send fewer tokens?

Send only the context the model needs for this one answer. Input tokens are usually the larger part of the bill, and most products send far more context than necessary.

• Retrieve, don't dump. SigmaCPM runs page-targeted retrieval of candidate pages before extraction, so the model reads the right pages of a tender instead of the whole package.
• Trim conversation history. Long chats resend every earlier message on every turn. Summarize older turns or keep only what's relevant.
• Shorten system prompts. Keep the lines that change behavior and drop the rest.
• Cap output. Set a sensible maximum on output tokens and ask for short answers where short is what the user wants.

Context trimming is the lever we'd pull first, because it also makes answers better. A model given three relevant pages usually answers more accurately than one given three hundred.

Which model should handle which task?

Match the model to the task, and make that choice a setting rather than a constant. Classification, short extraction, and routing are often handled well by smaller, cheaper models. Long reasoning over many sources may need a larger one.

Task	Usually fine on	Test a stronger model when
Classifying a message or document	A small model	Categories are subtle or high-stakes
Extracting fields from a short passage	A small or mid model	Documents are messy or fields are many
Answering from retrieved passages	A mid model	Answers need reasoning across many sources
Writing long-form content	A mid or large model	Quality is the product
Checking another model's output	A different model	The check is the safety net

The right column isn't a rule; it's where to point your evaluation set. NoteTube lets each conversation choose its model across OpenAI, Gemini, Claude, Llama, and Mistral. SigmaCPM calls models through OpenRouter, so changing the model for one step doesn't mean rewriting the pipeline.

Before switching a task to a cheaper model, run the same test questions on both. A cheaper model that gets one answer in ten wrong isn't cheaper once support tickets arrive. We explain how to build that test set in a RAG evaluation set.

When does batch processing cut the bill?

Batch processing cuts the bill whenever nobody is waiting for the answer. Both major providers charge half price for it. OpenAI's Batch API offers a "50% cost discount compared to synchronous APIs," and each batch completes within 24 hours (OpenAI, Batch API). Anthropic's Message Batches API also cuts costs by 50%, with most batches finishing in under an hour (Anthropic, batch processing).

Kraftr turns that into a product choice. A brand can shoot one look now, or queue a whole drop on a batch path that runs at half the credit cost and lands within a day. The customer picks speed or price per drop, and the cheaper path is cheaper for us to run too. Letting price follow cost this way is the clearest pattern we've found for AI products.

Good candidates for batching: overnight document processing, bulk catalogue generation, re-indexing, evaluation runs, and any report nobody reads until morning.

How does prompt caching lower input costs?

Prompt caching lowers costs by billing repeated prompt prefixes at a reduced rate. If the start of your prompt (instructions, tool definitions, a long document) is identical across calls, the provider can reuse its work.

The numbers are large. At Anthropic, cache reads cost 0.1 times the base input price for most models, while 5-minute cache writes cost 1.25 times, and the default cache lifetime is five minutes (Anthropic, prompt caching). OpenAI enables caching automatically on supported models, and the minimum cacheable prompt is 1,024 tokens for GPT-5.6 and later (OpenAI, prompt caching).

The rule both providers give is the same: put stable content first and changing content last. A timestamp or user name near the top of the prompt breaks the match for everything after it. Separately, cache results in your own app: if the same summary of the same unchanged document is requested twice, return the saved one, keyed on the input and the model version.

Plan limits protect the margin

Make limits part of the product and enforce them on the server. Even with every optimization, a few heavy users can dominate the bill.

• Credits or quotas per plan. Kraftr sells credit packs that brands spend on generations.
• Rate limits per user, so a script or runaway loop can't spend a month's budget in an hour.
• Clear messages when a limit is reached, with a path to upgrade.
• Alerts when one customer's daily spend jumps.

Limits must sit where the model is called, not only in the interface. And price features with their cost in mind: if one feature costs ten times more per use than another, it shouldn't sit in the cheapest plan with no limit.

Stop paying for work nobody sees

Small leaks add up. Don't regenerate on every page load what could be generated once and stored. Remove retries that resend a whole long prompt for an error that won't go away. Turn off features nobody uses. And when a user closes the tab mid-answer, stop generating; we show how in SSE streaming for AI chat.

Should you host an open-weight model yourself?

Sometimes, but the GPU bill is only part of the cost. Self-hosting makes sense when volume is high and steady, when data must stay on your infrastructure, or when a small fine-tuned model does a narrow task well. It also adds GPUs, scaling, upgrades, and monitoring to someone's job. Compare the full monthly cost, including people, against the provider bill before deciding.

Once a month, review cost per customer against revenue per customer, cost per feature, the top ten users by cost, and any change in cost per request after a prompt or model change. Those four numbers catch most problems before they reach the invoice.

FAQ

Is the batch API worth it for a small product?

Often yes, if some work can wait a few hours. Batch requests at OpenAI and Anthropic cost half the synchronous price, and the code change is small: collect requests, submit them together, and poll for results. Nightly summaries, re-indexing, and evaluation runs are easy first candidates even for a small team.

Does prompt caching change model answers?

No. Caching reuses the provider's processing of an identical prompt prefix; it doesn't reuse an old answer. The output is generated fresh each time. What it changes is price and speed for the cached part, so it's safe to enable wherever your prompts share a long, stable beginning.

How often should we revisit model choices?

Revisit when a provider releases or reprices models, and at least each quarter. Rerun your evaluation set on the current model and one or two cheaper candidates, compare accuracy and cost per request, and switch per task only where the cheaper model holds up. Keeping model choice as configuration makes this a small change.

Which cost should we cut first?

Cut input tokens first, because they're usually the bigger share and trimming them often improves answers. Then batch whatever can wait, then add caching. Plan limits come next, because they protect margin against the heaviest users. Switching models is powerful but needs an evaluation set, so it often comes later.

Is your AI bill growing faster than revenue? See our LLM cost optimization work and send us a month of usage data for a review.