Method
We optimize one deployment at a time, defined by its hardware, model, workload and objective. Every change is measured on your hardware against a fixed baseline.
“Maximizing performance often requires specialization, which can conflict with flexibility.”
What defines a deployment
- Hardware
- H100, B200
- RTX PRO 6000
- MI300X to MI355X
- Model
- Small or large
- Dense or MoE
- Text, speech or images
- Workload
- Chat and agents
- Summaries and extraction
- Long or short prompts
- Objective
- Latency
- Throughput
- Cost per token
The same model on the same GPU is limited by memory bandwidth at low concurrency and by compute at high concurrency, so the right change depends on where the deployment sits.
What we change
- 01
Speculative decoding. We train drafters on your traffic so more draft tokens are accepted per step.
- 02
Quantization and precision. We test precision layer by layer and keep only what passes your quality checks.
- 03
CUDA graphs. We restructure each step so more of it, up to the whole step, runs as one CUDA graph.
- 04
GPU kernels. We write kernels for your model’s shapes, your GPU and your batch sizes, and draw on a library of 280 we have already built.
- 05
Caching. We keep only the state your workload needs, which frees memory for more concurrent requests.
- 06
Scheduling. We batch prefills so they interrupt decoding less often.
- 07
Input size. We shrink inputs where your quality checks allow it.
- 08
Host-side work. We remove the CPU work that leaves the GPU waiting between steps.
- 09
Engine. We tune engine settings and code paths in vLLM or SGLang for your workload.
How we work
- Step 1
Profile
We measure each part of a step against its hardware limit with overshoot-roofline, including the time the GPU waits on the CPU.
- Step 2
Rank
We order the levers by the time each one can recover on your deployment.
- Step 3
Build and measure
We build each change and measure it on your hardware against the same baseline, across repeated server restarts.
- Step 4
Check the output
We compare outputs before and after each change. A change ships only if the difference stays within the variation between two restarts of the unchanged server.
- Measured
- 9.2 ms
- Limit
- 5.0 ms
- Headroom
- 46%
Hatched SMs mark the share of the step spent above the hardware limit. The share is measured; which SMs carry the hatching is illustrative.
A deployment typically takes 12 to 42 of these changes to reach its target.