Skip to main content
AI & Agents

Inference

Inference is running a trained model on new input to produce output — the serving phase, as opposed to training. Every request to a language model is an inference call, and because training happens once while inference runs on every request, inference economics dominate the cost of AI in production.

A language model call has two phases with different performance profiles. Prefill processes the prompt and is compute-bound and parallelisable; decode generates the answer one token at a time and is bound by memory bandwidth. This is why time to first token and tokens per second are separate metrics, and why a long prompt with a short answer performs nothing like a short prompt with a long one.

Cost is driven by token volume, context length and utilisation. Providers price input and output tokens separately, with output tokens several times more expensive because decode cannot be parallelised across the response. Long contexts consume KV-cache memory, which caps concurrent requests per GPU. Quantisation from 16-bit to 8-bit or 4-bit cuts serving cost substantially for a small quality trade, and is one reason open-weight models are cheap to self-host.

The build-versus-buy decision is a utilisation question. Provider APIs are elastic, operationally empty and priced per token — right for variable or growing load. Self-hosting on a serving engine such as vLLM converts cost to a flat GPU bill and buys control over data residency and model versioning, and it pays off when utilisation is steady enough to keep the hardware busy. The levers that matter at any scale are prompt caching, routing easy steps to smaller models, and streaming to make latency invisible to the user.

Throughput and latency pull against each other, and modern serving engines exist to manage the tension. Continuous batching — popularised by vLLM — admits new requests into a running batch as others finish, keeping the GPU saturated without making individual requests wait for a batch window. Capacity planning then becomes a question of concurrency and KV-cache memory rather than raw request counts, which is why two products with identical traffic can have very different serving bills.

The highest-leverage cost optimisation is usually not faster serving but less serving. A routing layer sends trivial steps — classification, extraction, short rewrites — to a small cheap model and reserves the frontier model for steps that genuinely need it, which routinely cuts cost per task dramatically in systems built naively on one model throughout. Prompt caching, response caching for repeated queries and semantic caching for near-duplicates remove another layer of spend before any hardware decision is even on the table.

Codazz builds this in production — LLM Integration.

FAQ

Inference
FAQ.

Common questions about inference.

Ask Us Anything

Training adjusts a model's weights against data to create capability — expensive, done once or rarely. Inference runs the finished model on a new input to produce output — cheap per call, done on every request. A product's AI bill is almost entirely inference, which is why token pricing and serving efficiency matter more to operating cost than anything about training.

When utilisation is steady and high enough to keep rented GPUs busy, when data residency rules out third-party APIs, or when you need pinned model versions that providers deprecate. Below that threshold, per-token API pricing is cheaper and operationally simpler. The honest comparison is total cost at your actual request volume, not headline price per token against a GPU's theoretical throughput.

For interactive features the number that matters is time to first token, not total generation time — a response that starts streaming within a second feels responsive even if it takes twenty seconds to finish, while a non-streaming call over a few seconds feels broken. Design around it: stream wherever the interface allows, keep prompts short enough that prefill stays fast, and move slow reasoning steps to background jobs instead of blocking clicks.

Related terms

LoRA (Low-Rank Adaptation)

LoRA — Low-Rank Adaptation — is a parameter-efficient fine-tuning method that freezes a model's original weights and trains small low-rank matrices alongside them.

Fine-Tuning

Fine-tuning adjusts a model's weights by training it on examples of desired behaviour, changing how it responds rather than what it knows.

Context Window

A context window is the maximum amount of text a language model can consider at one time, measured in tokens and shared by the system prompt, conversation history, retrieved documents and tool results.

Browse the full glossary →