Book a call

The Inference Engineering Blog

Measurements from the inference deployments we optimize: what each step cost before, what it costs now and, where we measured it, the hardware limit.

By Younes El Hjouji and Zakaria El Hjouji. 9 posts and 4 findings since February 2026.

Latest, Oct 9, 2026

A custom attention kernel for Gemma 4 on the RTX PRO 6000

FlashAttention has no kernel for Gemma 4’s largest attention heads on the RTX PRO 6000. Ours cuts global attention from 1,770 ms to 464 ms per 20k-token request, against a hardware floor of 294 ms.

Read the post

Global attention per 20k-token request, Gemma 4 31B, one RTX PRO 6000
vLLM default
1,770 ms
Our kernel
464 ms
floor 294 ms
Before
1,770 ms
After
464 ms
Hardware floor
294 ms
Left to the floor
170 ms
  1. vLLM’s Triton fallback underuses the tensor cores on Gemma 4 attentionfinding
    17% of measured peak
  2. Three launch settings speed up Gemma 4 attention with identical outputfinding
    −41% per call
  3. The case for inference optimization
    essay
  4. Compounding GPU and CPU work on a small model
    54% → 7% CPU-bound
  5. A fused MLP kernel paid off only after whole-step CUDA graphsfinding
    −9.3% per step
  6. Caching less to serve more requests
    105 → 274 per B200
  7. On Qwen3.5 35B-A3B, state copies fill most of each request’s memoryfinding
    85% of memory
  8. TCP for server-to-GPU communication
    325 → 72 ms
  9. Multimodal processor rewrite: 5× lower latency
    428 → 81 ms
  10. Gemma 4: 20× lower vision latency under load from one fix
    20× p95 TTFT
  11. Day-zero Gemma 4 support on Overshoot
    4 models, day zero
  12. Deploying VLMs through vLLM: a field guide
    30 models