Most production inference on open models runs on vLLM or SGLang. These inference platforms are designed to serve a wide range of models, GPUs and workloads. Inference optimization builds on top of these platforms for one specific deployment. Its goal is the highest performance the hardware can physically deliver while preserving quality. This post explains which factors determine the best achievable performance of a deployment, and why reaching that performance takes more than a general platform.
The deployment space
A deployment is defined by four dimensions: model, hardware, workload and objective. The workload is the shape and frequency of the requests. The objective is what the deployment is optimized for, usually one of three: throughput, latency or interactivity.
The four dimensions also depend on each other. The same model on the same GPU is limited by memory bandwidth when few requests run at once. With enough requests, compute becomes the limit. How many run at once depends on the workload and the objective.
Together, the four dimensions form a space, and each deployment is a point in it. The best way to serve a deployment depends on where it sits. For example, these two deployments sit far apart:
| Workload | Model | Hardware | Objective |
|---|---|---|---|
| Agentic coding | GLM 5.3 | 8×B200 | Interactivity |
| Summarization | Qwen3.5 4B | 1×H100 | Throughput |
Their requests differ in size, variability and prefix reuse:
Agentic coding workload
Summarization workload
Inference platforms are general by design
An inference platform has to serve the whole space: every model, GPU, workload and objective its users bring. Its kernels, scheduler and memory management have to work across all of them.
A design that is best for one deployment is often worse for another. One of vLLM’s creators describes this trade-off, and the side vLLM takes:
Maximizing performance often requires specialization, which can conflict with flexibility. […] vLLM intentionally avoids overfitting to present-day workloads.
Any one deployment therefore runs on general-purpose code.
Platforms offer settings to adapt to a deployment, and tuning them helps. But settings only choose among behaviors the platform already implements. What lies between a well-tuned deployment and the limits of its hardware is the subject of inference optimization.
Specializing for one deployment
Reaching the hardware’s limits means replacing general-purpose parts of the serving stack with parts built for the deployment. Two common examples:
Kernels
A consistent workload, such as summarization, sends the GPU the same few batch shapes over and over. A platform’s kernels have to handle every shape, so they are tuned for the general case. Kernels can instead be written for the shapes that recur and the hardware they run on. Sized to fill the GPU in one wave, without padding, they finish each batch sooner.
Quantization
Many parts of a model can run in fewer bits with little or no loss of quality. Which parts, and how few bits, depends on the model and on the task. Tuning quantization for a deployment means testing part by part and keeping only what passes the deployment’s quality gates. The hardware decides which formats are possible at all.
The performance frontier
A deployment’s best achievable performance is a performance frontier between two objectives, most often latency and throughput.
Kernels and quantization usually improve latency and throughput together, so they push the frontier toward the hardware’s limits. Few improvements push it as evenly as Figure 2 suggests; speculative decoding, for example, helps most at light load.
Along the frontier, latency and throughput can only be traded against each other, most simply by changing how many requests are allowed to run at once. The objective decides where the deployment sits, and so which part of the frontier to push.
Conclusion
A deployment’s best achievable performance is set by four things: its model, its hardware, its workload and its objective. No two production deployments are quite the same, so each one needs its own optimization. Inference optimization means understanding the specifics of one deployment, deciding which of the possible optimizations to build first, and spending engineering time and GPU hours building and validating each one until the deployment reaches the best performance its hardware allows.