vLLM runs attention with FlashAttention wherever it can, because on most models and GPUs it is the fastest kernel available. Gemma 4 on NVIDIA’s RTX PRO 6000 is an exception. FlashAttention has no kernel for the model’s attention layers with the largest heads on this GPU, so they run on a general kernel that leaves most of the GPU’s matrix throughput unused. We wrote a kernel for those layers, and it cut their attention time to about a quarter.
The deployment
We serve Gemma 4 31B with FP8 weights and a small draft model on one RTX PRO 6000, and we optimize time to first token on long prompts. By default, a 20k-token prompt took 4.5 s to the first token, most of it in prefill.
Five of every six layers in the model attend only to a window of recent tokens. The sixth, which we call a global layer, attends to the whole prompt with heads of 512 dimensions, twice the size of the others. Its cost per token grows with the length of the prompt, so on long prompts the 10 global layers carry most of the attention work.
Finding the layer type to rewrite
We profiled the prefill of one 20k-token prompt and split its GPU time by kind of work, against the hardware limit for each kind (Figure 1). Global attention took the largest share, 38% of the GPU time. It ran at 17% of the rate we measured for BF16 matrix multiplication on this GPU, so it also had the most time to gain.
vLLM default: 4,595 ms per prefill against a 1,781 ms limit.
Ours: 2,562 ms per prefill against a 1,747 ms limit.
Ours also runs FlashAttention 2 on the sliding-window layers and the draft model on the last position only.
Existing kernels first
No FlashAttention kernel covers heads of 512 dimensions on this GPU: FlashAttention 2 stops at 256, and FlashAttention 4 runs only on B200-class GPUs. With one attention backend for the whole model, all 60 layers run on vLLM’s general Triton kernel.
We tuned that kernel first. Its launch settings had been tuned only for the B200 and smaller heads, and doubling the query rows per block cut its time per call by 41%, with bit-identical output. We also tried two libraries. FlashInfer matched the tuned Triton kernel. cuDNN’s new kernel for this GPU was faster, but less accurate and slow on its first call at each new prompt length, so we did not use it.
Fitting the kernel to the RTX PRO 6000
The RTX PRO 6000 is a Blackwell GPU without the B200’s tensor-core instructions, so a kernel for it uses Ampere-style
mma.sync instructions and at most 99 KB of shared memory per thread block. Our kernel fits those limits with the
layout llama.cpp, FlashMLA and cuDNN use for large heads (Figure 2).
- Each thread block takes 8 tokens of the 8 query heads that share a key-value head, so every key and value tile it loads into shared memory serves 64 query rows, twice as many as in the Triton kernel.
- The 512 dimensions of Q and the output do not fit in one warp’s registers, so each pair of warps splits them. The pair adds its two halves of the scores through shared memory, and each warp then runs the softmax and multiplies by its half of V.
- Two warp groups run in ping-pong, so one group’s matrix multiplications overlap the other’s softmax.
Per call, the kernel runs 2.5–2.6 times faster than the tuned Triton kernel, at about 65% of the measured tensor-core rate. Its output is as accurate as the Triton kernel’s: against an fp64 reference both have the same mean and maximum error. It runs only for prefill steps of 64 or more tokens, so short prompts are unchanged by it.
What it changed end to end
Time to first token for the 20k-token prompt went from 4.5 s by default to 3.3 s with the tuned Triton kernel and two smaller changes, and to 2.7 s with our kernel, 18% less than with the tuned Triton kernel.
The two smaller changes are a shorter first pass of the draft model and exact parallel tokenization. The configuration we ship adds an 8-bit version of the kernel and other changes, and reaches the first token in 2.3 s against 4.5 s by default. We accepted the 8-bit version because its output drift from the BF16 model matched that of the FP8 weights.
The time left inside the kernel
Timers inside the first version of the kernel show where the time for each tile of keys goes. The matrix multiplications take two thirds of it and run at about 92% of the tensor-core rate. The other third goes to waiting for the next tile, exchanging scores and the softmax, which all warps reach at the same moment. Registers and shared memory leave little room to hide that third: Q and the output fill most of each thread’s registers, so only one block fits per SM, and shared memory holds only two stages of keys and values. Serving every key and value from L2 gained only 1–2%, so memory bandwidth does not set the limit. TMA loads, multicast across a cluster and a four-way split of the head’s dimensions all ran slower.
What carries over
Global attention now takes 1.6 times its hardware limit, down from six times by default. The kernel is sized to Gemma 4’s 512-dimension heads with 8 query heads per key-value head, and to the RTX PRO 6000’s registers and shared memory, so a different model or GPU needs a new version. A general engine chooses kernels that serve every model on every GPU, and for this model on this GPU it falls back to a general kernel.
The same profile works on other deployments. Profile one long request, compare each kind of work with its hardware limit, and start with the largest gap. When no library covers that shape on your GPU, a kernel written for the one model and the one GPU you serve can close most of the gap.