Book a call

Blog/Post

Building linear-attention metadata once per step

On a hybrid model, vLLM prepared the same inputs for each of its 30 linear-attention layers in every step.

Before each step, the engine prepares a small amount of metadata for every layer: which requests are decoding and which are being prefilled, where each request’s tokens start, and where each request’s state lives. On a hybrid model with 30 Gated DeltaNet layers, vLLM built that metadata 30 times per step, and parts of each build waited on the GPU. Under load the preparation took 21 ms of a 110 ms step, and the GPU was busy for only about 2 ms of those 21. We now build it once per step and give each layer only the part that differs. Preparation fell to 2 ms, the output text stayed the same in our checks, and one B200 served 7.4% more pages within the same latency target.

The deployment

We serve the public Qwen3.6-35B-A3B in FP8 on one B200, with a DFlash draft model proposing 7 tokens per step. Each request is one document page: a vision encoder turns the page image into about 3,000 tokens, and the model writes out the page’s text. We optimize the number of pages per second the GPU sustains while 90% of pages finish within 10 s.

The language model has 40 layers. 10 are standard attention, and 30 are Gated DeltaNet, a form of linear attention that keeps a fixed-size recurrent state per request instead of a KV cache. Under load many steps are mixed: one page is being prefilled while dozens of requests decode and verify their drafts.

Where the step’s time went

We profiled a mixed step at 64 requests in flight: one page prefilled alongside the requests that were decoding. From the start of the step to the first GPU kernel of the vision encoder took 21.0 ms. In that time the GPU was busy for only about 2.4 ms. The rest was the CPU preparing the step.

Most of the preparation was the Gated DeltaNet metadata builder. vLLM builds attention metadata once per KV-cache group, and on this model each Gated DeltaNet layer is its own group, so the builder ran 30 times per step. Each build sorted requests into those verifying drafts and those that were not, counted tokens and computed cumulative offsets, with many small operations each time:

operation per build per build per step, 30 builds
nonzero 9 270
item 9 270
cumsum 6 180
async memory copies (cudaMemcpyAsync) 14 420

Some of them made the CPU wait. The build selected rows of GPU tensors with a boolean mask held on the CPU, and each such selection needs a count back from the GPU before it can continue.

All of this repeats across layers. Which requests verify drafts, where each request’s tokens start and where its chunks begin are the same in every layer. Only one thing differs: the slot that holds each request’s state in that layer.

Removing the waits

The first change keeps the build per layer but stops it waiting on the GPU. The request masks are already on the CPU, so we compute the integer indices of draft-verifying and other requests there once, along with the token order and the cumulative query lengths, then copy the results to the GPU without blocking. The GPU selects rows with an index instead of a boolean mask.

That removed 480 of the 868 kernels in the preparation and took it from 21.0 to 15.9 ms (Figure 1). It did not change the number of pages per second on its own, because all 30 builds still ran.

Fig. 1CPU preparation per mixed step, 64 requests in flight
Stock builder: 868 kernels
21.0 ms
Without the waits: 388 kernels
15.9 ms
Built once per step: 156 kernels
2.0 ms

Building once per step

The second change builds the full metadata for the first Gated DeltaNet layer only. Each of the other 29 layers reuses it and computes just its own state slots from its block table. For steps that replay a CUDA graph, each layer writes its slots into its own fixed buffer, so the graph still reads the right addresses.

Preparation fell to 2.0 ms and 156 kernels per mixed step, and the mixed step from 110.5 to 98.8 ms. In a mixed step at low load on the same server, preparation went from 9.3 ms to 1.9 ms. The CPU’s time per step varies with how busy the host is, but the kernel counts do not.

Decoding steps also ran the 30 builds, so they got faster too. At one request at a time, the time per output token fell from 1.27 to 1.12 ms, 11.8% less, and a whole page from 2.27 to 2.01 s.

What it changed end to end

At a 10 s p90 per page, one B200 went from 4.17 to 4.48 pages per second, 7.4% more, both measured with the same sweep. The change only moves where integers are computed and stops recomputing them, and in our check (ten runs of one page, one request at a time) the output text was byte-identical to the stock builder’s.

What carries over

A general engine builds metadata per group of layers. Its cost then grows with the number of groups, not with the work in the step, and on a hybrid model with many layers of one kind most of that work repeats.

You can check for it in a profile. Count how often the metadata builder runs in one step, and look for blocking device-to-host copies between the end of one step and the first kernel of the next. Raw counts of nonzero and item are not enough, because many of them run on the CPU alone. When the step is short and the host work runs once per layer, building it once per step is often among the cheapest gains available.

Settings

Qwen/Qwen3.6-35B-A3B-FP8 on one B200, vLLM nightly 27a94d1c, DFlash drafter with 7 draft tokens, greedy sampling, FP8 KV cache, BF16 recurrent state, FlashInfer attention in the vision encoder, token counts padded to a multiple of 8, at most 128 requests and 16,384 batched tokens per step, CUDA graphs up to 8,192 tokens. Profiles were taken at 64 requests in flight; latency per token at one request at a time. Pages per second come from Poisson arrivals at 4, 4.5 and 5 pages per second for 120 s each, with the achieved rate interpolated to a 10 s p90 per page.

We do this work for one deployment at a time.