The hidden cost of LLM APIs: how we cut one client's inference bill by 80%

A client came to us with a $340K/month OpenAI bill and a roadmap that would triple usage within two quarters. Nobody on their team had signed up for that number. It had crept up one feature launch at a time, and by the time anyone noticed, it was load-bearing revenue infrastructure. The instinctive fixes (negotiate a better rate, add a caching layer, nag engineers about prompt length) bought them about 12%, nowhere near enough.

Before touching any infrastructure, we spent the first week just instrumenting. Every call to the OpenAI API was already logged, but nobody had ever joined that log against the feature or endpoint that triggered it. We added a request-tagging middleware (feature name, prompt template ID, and token counts on every call) and let it run for five days across production traffic. The breakdown was the actual finding, not a hunch: 70% of spend was going to two endpoints doing document classification and structured field extraction against a fixed JSON schema. Another 20% was semantic search re-ranking on top of a vector store. Only the remaining 10% was genuinely open-ended: a customer-facing support chat feature.

That's the point most cost-reduction efforts skip. 'Reduce our LLM bill' is not one problem, it's three workloads with three different cost curves, and only one of them, the open-ended chat, actually needs a frontier model's general reasoning. The other two were paying GPT-4-class pricing for what is, structurally, a classifier with a fixed label set and an extraction task with a fixed schema.

For the classification and extraction workloads, we built a fine-tuning pipeline on Llama 3.1 8B via Hugging Face. We pulled 90 days of the client's historical requests and responses, filtered for cases where a human had corrected the model's output (their support team logged these as tickets, which turned out to be a labeled dataset nobody had thought to use), and ended up with about 40,000 clean examples after dedup. We ran QLoRA fine-tuning (4-bit base model, rank-16 adapters, roughly six hours on a single A100) rather than a full-parameter fine-tune, since the task was narrow enough that the base model's general knowledge didn't need to move, just its output format and edge-case judgment.

The eval gate mattered more than the fine-tuning itself. We held out 15% of the labeled data and did not touch it during training. Before any checkpoint was eligible for a production rollout, it had to beat the incumbent GPT-4 baseline on this held-out set across three metrics: exact-match accuracy on the extraction schema, F1 on the classification labels, and a hallucination check (did the model ever emit a field value not present in the source document). The first checkpoint we trained actually lost to GPT-4 on the hallucination check. It had learned to 'confidently fill in' missing fields instead of returning null, which the baseline handled correctly. We fixed this by adding explicit negative examples (documents with genuinely missing fields, labeled with nulls) to the training set, and the second checkpoint cleared all three gates.

For serving, we deployed behind vLLM rather than raw HuggingFace `transformers` inference, which is the difference between a demo and something that survives concurrent traffic. Continuous batching let the server pack multiple in-flight requests onto the GPU instead of processing them one at a time, and AWQ 4-bit quantization cut the memory footprint enough to fit comfortably on L40S instances instead of requiring A100s for serving. We benchmarked three configurations against the client's actual traffic replay (not a synthetic load test) before picking a final GPU count: two L40S instances behind a load balancer, autoscaled on queue depth rather than CPU utilization, comfortably cleared their p99 latency SLA with room for the 3x traffic growth already on their roadmap.

The open-ended support chat stayed exactly where it was, on the hosted API. This is worth stating plainly because it's the part that's easy to skip in a 'we saved money' narrative: frontier hosted models are still the right tool for genuinely open-ended reasoning, and trying to force a fine-tuned 8B model to handle unbounded conversational support would have traded a cost problem for a quality problem.

Rollout was staged over three weeks: 10% of classification traffic in week one with the old and new pipelines running in parallel and outputs diffed automatically, 50% in week two once the diff rate on disagreements stayed under 2% (and every disagreement was manually reviewed), and full cutover in week three. We kept the GPT-4 path warm as an automatic fallback for the first month in case of a serving incident, then decommissioned it once the fine-tuned pipeline had a full month of stable p99 latency and error rate data.

End state: the monthly bill dropped from $340K to roughly $71K, the residual spend being the open-ended chat traffic plus a small buffer of extraction requests routed to GPT-4 as a fallback for schema edge cases the fine-tuned model hadn't seen in training. Accuracy on the classification task improved by about 2 points over the GPT-4 baseline, largely because the fine-tuned model had seen the client's actual data distribution and document formats instead of relying on a general-purpose model's priors. p99 latency on extraction calls dropped from 2.1s to 640ms, mostly because the smaller model wasn't reasoning through a long system prompt on every call.

The generalizable lesson isn't 'replace hosted APIs with open-weight models.' For a lot of workloads, especially low-volume or genuinely open-ended ones, hosted frontier APIs remain the right economic and engineering choice. The lesson is that most enterprise LLM spend is not one workload wearing a trenchcoat, it's several workloads with very different cost-to-value ratios, and you can't make a good infrastructure decision about 'the LLM bill' until you've actually measured what's inside it. This kind of workload-shaped cost audit is exactly where our AI engineering practice starts.