OpenAI API Cost Optimization: How I Cut Inference Costs by 60%

The first thing I did was not an optimisation. It was instrumentation. Every model call started logging input tokens, output tokens, the feature that triggered it, and an estimated cost. Within a week the bill stopped being one number and became a breakdown, and the breakdown was not what anyone expected.

One background summarisation job that nobody considered important accounted for roughly 40 percent of spend, because it re-summarised records on every write rather than on meaningful change. No amount of prompt tuning would have found that. Attribution by feature is the prerequisite for every other technique in this article.

Teams default to the most capable model for everything, which is like using a lorry for the school run. Classification, routing, short extraction and formatting are handled perfectly well by the cheapest tier, at a fraction of the cost and a fraction of the latency. Reserve the expensive tier for genuinely reasoning-heavy work.

The way to do this without guessing is an evaluation set. Take thirty real cases per feature with known-good outcomes, run them against each tier, and compare. In my own benchmarking the assumption that newer and bigger is better proved wrong more than once — I measured a newer model that failed two of three requests under load and took 54 seconds when it did respond. Without measurement I would have shipped it on the strength of its version number.

Most providers now bill repeated prefix content at a steep discount. That turns prompt structure into a cost decision: put everything stable — the system prompt, the schema, the few-shot examples — at the front, and everything variable at the end. Reorganising prompts so the static block came first was one of the largest single savings I made, and it changed nothing about the output.

The corollary is to stop injecting volatile values into the top of a prompt. A timestamp or a request ID near the start of an otherwise identical prompt breaks the prefix match and quietly forfeits the discount on every call. That is a genuinely invisible cost bug until you are looking at token attribution.

The most common waste is sending an entire document, conversation history, or record dump because it is easier than deciding what matters. Input tokens are the majority of spend in most workloads, so trimming them has a direct effect. Retrieval that returns four well-chosen passages instead of twenty mediocre ones cuts cost and improves answers at the same time.

Conversation history deserves particular scrutiny because it grows quadratically: every turn re-sends everything before it. Summarising older turns into a compact state object, rather than replaying the transcript, flattens that curve. On a long-running assistant this was the difference between a feature that got more expensive the more people used it and one that did not.

An exact-match cache on the normalised prompt is the cheapest win available and it is often skipped because it feels too simple. In practice identical requests are common — the same document classified twice, the same record summarised after an unrelated field changed. Caching those is free money and reduces latency at the same time.

Two smaller levers add up. Batch endpoints, where the provider offers them, trade latency for a substantial discount and suit any overnight or non-interactive job. And setting an explicit maximum output length stops a model from producing a 900-token essay where 200 tokens would do, which is a cost most teams never realise they are paying.

The reduction did not come from one clever trick. Fixing the over-triggering summarisation job was the single biggest item. Moving classification and routing to the cheap tier was next. Prompt reordering for caching, tighter retrieval, an exact-match cache, and output caps made up the rest, each contributing a modest amount that compounded.

Output quality was tracked throughout with the same evaluation sets, which is the part I would insist on. Cost reduction that quietly degrades results is not a saving, it is a deferred complaint. Being able to show that scores held steady while spend fell by roughly 60 percent is what made the changes uncontroversial.