Real-time inference for vision-language models

Break latency into useful measurements

End-to-end latency starts when the application decides to inspect video and ends when it can use the answer. That path includes frame capture, WebRTC transport, frame selection, image decoding, visual preprocessing, model scheduling, prefill, generation, SSE transport, parsing, and any validation. A single server number hides these components. Instrument timestamps at the publisher, request client, first token, final token, and workflow boundary.

Time to first token describes responsiveness for streaming interfaces and early routing. Inter-token latency describes generation cadence after the first token. Full-response latency describes when a complete answer or JSON object is available. Report p50, p90, or p95 with the offered load and run duration. A percentile without model, media shape, concurrency, and region cannot support a deployment decision.

Control the visual workload

Resolution, frame count, and segment sampling affect preprocessing and model prefill. Use frame_index=-1 for a current-state question. Use a short video interval only when temporal order contributes to the answer. Stream segments default to max_fps 1.0. Raise that ceiling deliberately and measure the result. A longer segment can add many visual tokens while producing no useful accuracy gain.

Keep prompts and output budgets equally controlled. A long system prompt increases prefill work. A large max_completion_tokens value increases the possible generation tail. Stable thread_id values can support prompt-cache behavior for related requests. Cache observability appears in the Overshoot response where supported. Run comparisons with identical prompt prefixes and report whether the measured request used a warm related thread.

Read the Qwen preprocessing result correctly

In a workload-specific engineering test, Qwen3.6-27B-FP8 processed 15 frames at 480p on one H200. The stock sequential CPU processor measured 428 ms p90. An optimized implementation reduced that stage to 81 ms p90 while preserving output. At 5 QPS, stock p90 rose to 11,574 ms and the optimized path stayed near 80 ms. These values isolate preprocessing for that model, input, hardware, and load.

The study shows how CPU serialization can dominate a GPU-backed serving path. It does not establish a universal end-to-end response for every model or request. Use its method: identify a stage, preserve output equivalence, sweep offered load, and measure tail latency. When your workload differs in resolution, frame count, model family, hardware, or scheduling, collect a new result.

Read the Gemma batching result correctly

A second workload-specific study used Gemma with six 480p frames, about 500 visual tokens, and roughly 500 output tokens. After encoder batching work, p95 time to first token stayed near 120 ms at 20 QPS. Median inter-token latency stayed near 5 ms through 10 QPS. The test examined how vision-encoder work interacts with decode scheduling under increasing load.

TTFT and ITL answer separate questions. Encoder batching can protect request startup while decode contention influences generation cadence. The numbers do not predict a browser result or a different model. Use a load sweep that reaches the concurrency expected in production, then watch queueing and tails. Report achieved throughput alongside offered QPS so overload is visible instead of hidden in averages.

Build for model and service state

GET /v1beta/models reports ids and ready or unavailable status. Refresh availability instead of assuming a model always serves. Overshoot-hosted ids contain a provider/model-name path and run on managed infrastructure. Proprietary ids without a slash can represent slower passthrough options. Select based on measured task quality and latency. Define whether the product waits, chooses a tested fallback, or asks the user to retry when the preferred model is unavailable.

Handle non-success responses as typed operational events. Bound retries and avoid retry storms during load. SSE clients should tolerate partial network chunks, process complete data lines, stop on [DONE], and cancel readers when the user leaves. Keep authentication server-side in products where the key belongs to your organization. The public playground is a separate bring-your-own-key context.

Create a decision-grade benchmark

Freeze a representative source, prompt, model id, region, frame-selection rule, max_fps, and output limit. Warm or cold behavior should match the intended product path. Increase concurrency in controlled steps and collect enough requests for stable tail percentiles. Log failures and achieved throughput. Measure at the real client when network and parsing are part of the experience.

Pair performance with correctness. A faster request that misses the event does not meet the task. Score the same labeled set for every model and media envelope, then choose the smallest input that reaches the acceptance threshold. Re-run after model or prompt changes. Real-time inference is an operational property of a complete workload, and the benchmark should preserve enough detail for another engineer to repeat it.

References

• Benchmark methodology
• Model discovery
• Real-time inference guide