Fine-tuning, self-hosted inference and LLM cost control

The problem

A baseline analysis of LLM usage found that two premium models drove about 92% of OpenAI spend, on only 3% of requests. Many tasks did not need a premium model at all. The goal was to keep quality and cut cost.

Fine-tuning

I fine-tuned Llama 8B and 70B with LoRA and QLoRA (Hugging Face PEFT, 4-bit bitsandbytes, multi-GPU). The tasks were KPI analysis, summarization and intent extraction. The tuned models match premium-model quality on these tasks at a fraction of the cost.

Self-hosted inference

The models run on a self-hosted stack built on vLLM:

• PagedAttention and continuous batching keep the GPUs busy under mixed traffic.
• Chunked prefill keeps long prompts from blocking short ones.
• Tensor-parallel sharding splits large models across GPUs.
• An OpenAI-compatible endpoint lets existing code switch models with no rewrite.

I benchmarked AWQ, GPTQ, FP8 and INT8 quantization for latency, throughput and accuracy before choosing a setting.

The cost control plane

I led the design of a company-wide LLM cost control plane:

• Task-aware routing sends each task to the cheapest model that meets its quality bar.
• Per-organization cost attribution shows which customers and features drive spend.
• Runtime model switching changes models without a deploy.
• Self-hosted Gemma and Llama on AWS take over the high-volume tasks.

The analysis projected a 30–40% cut in LLM spend.