Book a call

The Inference Engineering Blog

Measurements from the inference deployments we optimize: what each step cost before, what it costs now and, where we measured it, the hardware limit.

By Younes El Hjouji and Zakaria El Hjouji. 8 posts and 4 findings since February 2026.

Latest, Oct 9, 2026

A custom attention kernel for Gemma 4 on the RTX PRO 6000

No version of FlashAttention runs Gemma 4’s attention over the whole prompt on the RTX PRO 6000, so we wrote our own kernel.

Read the post

Global attention per 20k-token request, Gemma 4 31B, one RTX PRO 6000
vLLM default
1,770 ms
Our kernel
464 ms
floor 294 ms
Before
1,770 ms
After
464 ms
Hardware floor
294 ms
Left to the floor
170 ms
  1. vLLM’s Triton fallback underuses the tensor cores on Gemma 4 attentionfinding
    17% of measured peak
  2. Three launch settings speed up Gemma 4 attention with identical outputfinding
    −41% per call
  3. The case for inference optimization
    essay
  4. Compounding GPU and CPU work on a small model
    6.4 → 3.4 ms
  5. A fused MLP kernel paid off only after whole-step CUDA graphsfinding
    −9.3% per step
  6. Caching less to serve more requests
    2.6× requests per GPU
  7. On Qwen3.5 35B-A3B, state copies fill most of each request’s memoryfinding
    85% of memory
  8. TCP for server-to-GPU communication
    325 → 72 ms
  9. Multimodal processor rewrite: 5× lower latency
    428 → 81 ms
  10. Gemma 4: 20× lower vision latency under load from one fix
    20× p95 TTFT
  11. Day-zero Gemma 4 support
    4 models, day zero