Learn / What is LLM inference?
What is LLM inference?
LLM inference is the process of running a trained large language model to generate output from a prompt. In production it involves loading model weights, processing the input prompt, and generating output tokens one at a time, while a serving engine manages batching, GPU memory, latency, and throughput across many concurrent requests.
How LLM inference works
LLM inference happens in two phases. In the prefill phase, the model processes your entire input prompt in parallel and builds an internal representation called the KV cache. In the decode phase, it generates output tokens one at a time, each token depending on all the tokens before it. Decode is sequential, which is why streaming speed and how an engine handles many requests at once have such a large effect on real-world performance.
Why the serving engine matters
The same model on the same GPU can be several times faster or slower depending on the serving engine. The engine decides how requests are batched, how work is scheduled, and how GPU memory and the KV cache are managed. In a published benchmark that held the model and a single NVIDIA H200 GPU constant and changed only the engine, output per second varied by up to 4.4x — a reminder that the runtime, not just the hardware, determines production performance.
Key inference metrics
Inference performance is measured with a handful of metrics: time-to-first-token (TTFT), time-per-output-token, end-to-end latency, throughput, and concurrency. Each captures a different part of the user experience. See inference latency and throughput for full definitions and how to improve them.
How Netra helps
Netra Runtime is a high-performance AI inference platform with an OpenAI-compatible API. In a published long-context benchmark it led vLLM and several SGLang configurations on both throughput and latency at every level of load, from a single user up to 512 concurrent users. Learn what Netra Runtime is or see the Netra vs vLLM comparison.
Frequently asked questions
What is the difference between training and inference?
Training teaches a model by adjusting its weights on data; inference runs the finished model to produce outputs from new prompts. Inference is what happens every time a user sends a request in production.
What is prefill and decode in LLM inference?
Prefill is the phase where the model processes the input prompt in parallel; decode is the phase where it generates output tokens one at a time. Decode is sequential, which is why streaming speed and concurrency handling matter so much.
Why is the inference engine more important than just the GPU?
On the same model and GPU, different serving engines can deliver several-fold differences in throughput and latency because of how they batch requests, schedule work, and manage memory. In a published benchmark, swapping only the engine changed output-per-second by up to 4.4x.