On one deployment of a 4B model on a B200, our GPU changes shortened the decoding step until the CPU work that prepares each step was 75–80% as long as the GPU’s. More than half of all steps then waited on the CPU, and a faster kernel left the step time unchanged. Running the whole step as one CUDA graph cut the CPU’s busy time per step by 60%, and the same kernel then took 9.3% off every step. On the public Qwen3.5 4B with a fast host, the same kinds of change behaved differently. Its decode step stayed GPU-bound, the graph removed a wait of 5% of the step, and our decode kernels helped as much without the graph as with it. Whether the CPU sets the pace depends on the deployment, so we measure it on each one.
The deployment
The model is a fine-tune of a public 4B model that mixes linear-attention and full-attention layers. A block draft model proposes up to 31 tokens per step. The deployment serves 1 to 8 requests at a time on one B200, and the objective is speed per request. The two models run as CUDA graphs. Between them the engine’s CPU thread schedules requests, builds the metadata for the next step, samples, runs the acceptance check and updates each request. With asynchronous scheduling the CPU prepares step n + 1 while the GPU runs step n, so a step takes as long as the longer of the two. This CPU work is the same from 1 to 8 requests, so it weighs most on small models at low concurrency.
How to tell whether your step is CPU-bound
Put timers on the engine thread, with the profiler off, and record for each step how long the thread is busy and how long it waits for the GPU. A step in which the thread never waits is CPU-bound, and GPU savings will not reach it. In our case, once the CPU’s busy time per step reached about three quarters of the GPU’s, the variation between steps made more than half of them CPU-bound. A profiler trace shows the GPU’s side of the same question, the time the GPU sits idle waiting for the next launch, and our profiler, overshoot-roofline, reports it per step. For the CPU’s busy time we use the timers, because in our measurements the PyTorch profiler added CPU time to every operation.
GPU work first
On an earlier build, with a GPU step about 1.5× longer, 10% of steps were CPU-bound. Our GPU-side changes cut the GPU’s step by about a third, the CPU’s work stayed the same, and 54% of steps became CPU-bound. A fused MLP kernel that saved about 7% of a step in its isolated test then left the step time unchanged when served. Four pairs of runs on two hosts ranged from −3.5% to +4.5%.
One CUDA graph per step
We captured the whole speculative step as one CUDA graph. The target and draft graphs are its child nodes, with the metadata kernels, the sampler, the acceptance check and the bookkeeping between them. The graph for each number of running requests is captured on the second step, after one ordinary step, so first-use setup does not happen during capture. Each later step runs the CPU bookkeeping, copies its inputs into fixed device buffers and replays the graph. The fixed buffers matter because six per-step inputs sit in pinned host memory that alternates between two slots, so a graph that read them directly would read a stale address on every other step.
- The engine thread’s busy time per step fell by 60%.
- CPU-bound steps fell from 54% to 7%.
- Outputs were bit-identical at one request (200 of 200), and within the usual batching differences at 8.
On a fast host the step moved by 3%, because the GPU’s step now set the pace. Its effect showed in the next GPU change.
A general engine keeps parts of the step outside its graphs so that it can support every sampler, drafter and scheduling feature. On the public Qwen3.5 4B, vLLM 0.31 runs the linear-attention core operation between its CUDA graphs. One graph per step works here because the deployment fixes the drafter and the sampler, and plain sampling is a requirement of the capture.
GPU work again
The fused MLP kernel then took 9.3% off every step and raised speed per request by 6.3%, in one pair of runs on a slower host, where single pairs resolve differences above 5%. It also slowed the steps that process new prompts, raising mean time to first token by 7–18%, and it changed output numerics more than the graph did.
The same check on a public model
We ran the same changes on the public Qwen3.5 4B with its public DFlash drafter at 15 draft tokens, one B200, 512 output tokens per request on output that copies the prompt, one server lifetime per arm. We measured each decoding step with overshoot-roofline at one request:
| configuration | decode step | GPU idle waiting on the CPU |
|---|---|---|
| vLLM 0.31 default | 5.49 ms | 5% |
| our GPU-side change | 4.34 ms | 3% |
| + our decode kernels | 4.24 ms | 5% |
| + one CUDA graph per step, no decode kernels | 4.18 ms | 0.1% |
| + one CUDA graph per step and our decode kernels | 3.93 ms | 0.05% |
On this host and model the decode step stayed GPU-bound, so the graph removed a wait that was already small, and speed per request did not change. At one request our decode kernels added 5.8% to speed per request without the graph and 5.3% with it, the same within noise, so the order that mattered on our deployment did not appear here. The CPU did set the pace in steps that admit a new prompt. On vLLM 0.31 at 8 requests such a step took 34.9 ms, of which the GPU waited on the CPU for 16.5 ms. The draft length, the drafter, the host’s CPU and how far the GPU work has already been cut all decide whether decode steps become CPU-bound.
To hold output lengths fixed in such a test, use ignore_eos with max_tokens. In our engine min_tokens counts as
logits processing and silently turns off the step graph, so check the graph’s replay count in the server log before
comparing.
After each GPU change, measure the share of CPU-bound steps again, because every GPU saving lowers the point where the CPU sets the pace.