Real-time vision benchmarks and methodology
A real-time vision benchmark measures a defined inference workload under disclosed conditions. Overshoot-hosted vision models respond in 200ms. The engineering results here explain two serving optimizations and keep their boundaries explicit. The Qwen study isolates multimodal preprocessing for 15 480p frames on one H200. The Gemma study examines time to first token and inter-token latency with six 480p frames under load. Neither result is a universal application guarantee. Use the methodology to benchmark your selected model, media, region, prompt, output, and client path.
- Product response
- 200msTypical response time for Overshoot-hosted vision models.
- Qwen p90
- 81 msOptimized preprocessing for 15 480p frames on one H200.
- Qwen load
- 5 QPSOptimized preprocessing remained near 80 ms in this specific test.
- Gemma p95 TTFT
- ~120 msSix 480p frames at 20 QPS after encoder batching.
- Gemma ITL
- ~5 msMedian inter-token latency through 10 QPS in this specific test.
Qwen preprocessing before and after optimization
The Qwen workload used Qwen3.6-27B-FP8, one H200, and 15 frames at 480p. The stock sequential CPU processor measured 428 ms p90 for visual preprocessing. The optimized implementation measured 81 ms p90 while preserving output. This is a stage-level comparison. It does not include every publisher, network, model, generation, and client component.
The improvement came from treating multimodal preprocessing as a serving bottleneck worthy of its own profile. GPU capacity alone did not remove serialized CPU work. The result demonstrates why teams should measure decoding, resizing, normalization, token construction, and transfer around the model instead of assuming model execution dominates every request.
Qwen behavior at sustained load
At 5 QPS in the same documented workload, stock preprocessing p90 rose to 11,574 ms. The optimized processor remained near 80 ms. Queueing exposed a large difference that a single-request test would have hidden. The comparison holds model, frame count, resolution, hardware, implementation family, and metric in view.
Do not convert 5 QPS into a service throughput statement. It is the offered load in this specific engineering test. Capacity for another model, media envelope, prompt, output length, region, or serving configuration must be measured separately. Report achieved throughput and errors so an overloaded run cannot appear successful through latency filtering.
Gemma vision-encoder batching result
The Gemma workload used six 480p frames, about 500 visual tokens, and roughly 500 output tokens. After batching changes, p95 time to first token stayed around 120 ms at 20 QPS. Before the work, the measured value under the documented load was about three seconds. The test focused on scheduling vision-encoder work so it did not block request progress under load.
Median inter-token latency stayed around 5 ms through 10 QPS after the change, compared with about two seconds in the earlier behavior. TTFT measures when generation begins. ITL measures cadence after that point. A product may care about one or both depending on whether it needs an early label, streamed explanation, or complete structured answer.
Define a reproducible workload
Name the model id, serving version, region or hardware, source resolution, frame count, segment duration, max_fps, visual token estimate, prompt length, output limit, response mode, cache state, offered load, achieved throughput, and run duration. Publish the percentile and clock boundary. Keep source media fixed when comparing implementation changes.
For browser measurements, start the clock at the user or scheduling event and end at first useful token or validated completion. This includes network and client work. For server-stage measurements, name the exact function boundaries. Both are valuable when labeled correctly. Avoid combining them into one unlabeled latency claim.
Pair performance with task quality
Reducing frames, resolution, or output can improve speed and degrade the answer. Build a labeled evaluation set and measure quality for every media envelope under consideration. Select the smallest input that reaches the task threshold. Track unknown behavior and parse failures alongside correctness. A fast invalid JSON response does not complete the workflow.
Compare models with the same prompt, media, and output contract. If a model requires a different prompt, report that difference. Separate provider errors and unavailable states from wrong answers. Repeat measurements after model or prompt changes because performance and quality can move independently.
Read the 200ms product response responsibly
Overshoot-hosted models respond in 200ms. Use that statement as a starting point for your own acceptance test. Capture-to-answer latency also includes publisher transport, source freshness, selected media, network distance, SSE parsing, and application action. Proprietary passthrough models can have different latency characteristics from hosted provider/model-name ids.
Run the expected concurrency in us-west1 or us-central1, record p50 and tail results, and retain the request metadata required to explain outliers. Test model unavailability, retries, expired Streams, and interrupted SSE. A benchmark becomes decision-grade when another engineer can reproduce the workload and when its boundary matches the product interaction.
Frequently asked questions
Are these end-to-end customer results?
No. The Qwen result isolates preprocessing, and the Gemma result covers serving TTFT and ITL for disclosed workloads.
Does 5 QPS state Overshoot capacity?
No. It is the offered load in one Qwen preprocessing test.
References
Run a workload-specific benchmark
Publish the model, media shape, load, percentile, timing boundary, errors, and task score with every result.