Computer Vision Inference Benchmarks 2026: Multimodal Preprocessing and Video Latency

Computer vision inference benchmarks measure how a serving system processes visual inputs under defined hardware, model, request, concurrency, and sampling conditions. In 2026, multimodal systems require more than a single images-per-second number. Video decoding, CPU preprocessing, vision encoding, batching, language-model prefill, token generation, and queueing can each set the latency floor. This guide reports published Overshoot engineering measurements and explains how to reproduce comparable tests.

Scope and benchmark integrity

This article covers multimodal preprocessing and serving latency for Qwen3.6-27B-FP8 and Gemma 4 workloads. It contains no detector mAP results and no Jetson benchmark results. Overshoot has no Jetson certification or edge runtime, so inventing device throughput would misrepresent the product and the evidence. Detector accuracy also answers a different question from VLM time to first token. The useful scope here is published cloud serving data and a method readers can apply to their own systems.

Every row below comes from an engineering benchmark described in an existing Overshoot case study. Rows are not extrapolated to other GPUs, resolutions, models, or request rates. Approximate values remain marked with a tilde. A result without the model, hardware, input shape, offered load, metric, and percentile is too ambiguous for architecture decisions.

Metrics that describe multimodal serving

Time to first token, or TTFT, starts when the client issues a request and ends when the first generated token arrives. Depending on instrumentation, it can include network and API overhead or only server time. State the boundary. TTFT captures queueing, multimodal preprocessing, vision encoding, and language-model prefill. It is often the primary latency metric for an interactive visual query.

Inter-token latency, or ITL, measures spacing between generated tokens after the first one. It reflects decode behavior and interference from other work scheduled into the serving loop. A response can have a good TTFT and poor ITL, or the reverse. For structured outputs with few tokens, TTFT may dominate the user-visible delay. Longer descriptions need both metrics.

Preprocessing latency covers CPU work that turns decoded media into model inputs. Typical steps include resize, normalization, patch extraction, prompt tokenization, placeholder construction, and framework validation. Offered QPS is the arrival rate applied by the load generator. It differs from achieved throughput when the service cannot keep up. Report p50 and tail percentiles because queue instability appears in the tail first.

Qwen3.6 preprocessing benchmark

The Qwen benchmark ran Qwen3.6-27B-FP8 on an H200 with 15 frames at 480p. Stock multimodal preprocessing measured 428 ms p90 for sequential requests. An optimized, bit-identical processor reduced that result to 81 ms p90. The implementation owned the hot path, removed repeated framework traversal, preallocated outputs, and fused suitable operations while retaining validation against the stock processor.

The load result exposed the larger operational problem. At 5 QPS, stock preprocessing reached 11,574 ms p90. The optimized path stayed near 80 ms. A single stock processor worker needed roughly 428 ms in the tail for this input shape, so requests arrived faster than it could drain them. Queue wait then dwarfed the actual math. The optimized service used enough processing capacity to absorb the offered load without recreating contention elsewhere.

This is a capacity boundary, not a general claim that all Qwen requests take either value. A single image, six frames, another resolution, another processor version, or a different CPU allocation creates a different workload. The case study also found a runtime thread limit that made the optimized processor slower inside vLLM until OMP_NUM_THREADS=8 restored the expected performance. Hidden runtime defaults belong in the benchmark manifest.

Representative serving controls from the optimized Qwen test

export OMP_NUM_THREADS=8

vllm serve QWEN_MODEL_PATH \
  --renderer-num-workers=2 \
  --mm-processor-kwargs '{"do_sample_frames": false}'

The final flag prevents a subtle workload change. The default Qwen and vLLM path can downsample video to roughly 2 FPS before processing. A test intended to contain 15 frames can become a smaller effective input unless frame sampling is disabled. Reproducibility requires confirming the processor actually received the expected frame count.

Gemma 4 vision encoder benchmark

The Gemma workload used six 480p frames, corresponding to roughly 500 visual tokens, and generated a short response. The original vision path processed frames serially inside each batch. One video prefill could therefore delay decode work sharing the same serving iteration. As offered load increased, video TTFT and ITL rose sharply even though a text workload of similar token size remained stable.

The serving change batched compatible vision-encoder work while retaining a sequential pooler where full batching introduced numerical drift. After the change, p95 TTFT stayed near 120 ms at 20 QPS for the six-frame workload. Median ITL stayed near 5 ms through 10 QPS. These figures show the behavior of that implementation and input shape under the reported load sweep. They are not accuracy scores and do not establish performance for arbitrary Gemma deployments.

The lesson differs from the Qwen processor result. Qwen exposed a CPU preprocessing queue before GPU inference. Gemma exposed serial work in the GPU vision-encoder path and its interaction with continuous batching. Both appear to a client as slow visual inference. Stage-level instrumentation was necessary to identify which optimization could change the curve.

Visual token estimates and input shape

Frame count alone does not describe a multimodal request. Resolution controls the number of image patches, and model processors can resize, tile, or sample media differently. For planning Qwen workloads, approximate visual-token counts are 200 tokens for one 480p frame, 450 for one 720p frame, and 1,010 for one 1080p frame. Fifteen 480p frames are therefore a very different input from fifteen 1080p frames.

More visual tokens can increase processor work, encoder work, prefill cost, memory pressure, and queue time. The relationship is not guaranteed to be linear because kernels, batching, tiling boundaries, and cache behavior change with shape. Benchmark the resolutions and frame counts the application will send. Avoid testing high-resolution media when the production publisher downscales it, or testing sparse frames when production sends dense segments.

Live video latency with Overshoot Streams

Overshoot Live Stream ingest uses WebRTC through LiveKit from a browser, native, or server publisher. The service retains 600 seconds of recent visual history. A Stream has a 300-second lease that the publisher renews with keepalive. Chat completions reference frames or segments through the ovs:// scheme and return tokens with SSE. Overshoot answers in 200ms.

The 200ms answer does not represent camera-to-action latency. A production benchmark should start at capture or at a known event timestamp and include publish delay, regional network transit, request scheduling, first token, output completion, parsing, and application action. Available regions are us-west1 and us-central1. Measure from the intended deployment network because route quality and uplink conditions affect the result.

For recent video queries, specify the interval and sampling cap. The default segment max_fps is 1.0. Increasing it changes the benchmark workload by adding frames. Direct RTSP, RTMP, and ONVIF ingest are absent from the public API. If an external gateway bridges an RTSP camera to WebRTC, report its decode, buffer, and encode contribution separately.

Build a reproducible benchmark

1. Pin the model id, weights, serving engine commit, processor code, driver, and container image.
2. Record GPU, CPU allocation, memory, NUMA placement, thread settings, and worker counts.
3. Freeze the media set, decoded resolution, frame count, sampling behavior, prompt, and output token limit.
4. Disable caches when measuring uncached behavior, or report cache hit ratio when caches are part of production.
5. Warm the service with a declared procedure, then run long enough to observe steady-state queues.
6. Apply a defined arrival process and report offered QPS, achieved QPS, errors, and concurrency.
7. Measure preprocessing, vision encoding, prefill, TTFT, ITL, and end-to-end latency at useful percentiles.
8. Publish raw samples or enough histogram detail to reveal variance and saturation.

Use a fixed-rate load generator when the production system receives independent arrivals. A closed-loop client that waits for one response before sending the next can hide overload by reducing its own offered rate. Run a QPS sweep rather than one point. The knee where tail latency accelerates indicates that a stage is approaching saturation. Continue far enough to distinguish graceful degradation from an unstable queue, while enforcing safety limits on the test environment.

Request shape for an uncached streaming latency run

curl -N https://api.overshoot.ai/v1beta/chat/completions \
  -H "Authorization: Bearer $OVERSHOOT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID",
    "stream": true,
    "max_tokens": 10,
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Return one concise visible-state label."},
        {"type": "image_url", "image_url": {"url": "https://example.com/fixed-frame.jpg"}}
      ]
    }]
  }'

Prevent common benchmark errors

First, confirm the media path. A URL may include download and decode time, a data URL may increase request size, and an ovs:// reference uses already ingested Stream media. These paths answer different operational questions. Overshoot chat requests support HTTPS and data media in addition to Stream references. State which path the benchmark uses.

Second, control output length. Total response time for ten tokens cannot be compared with a detailed paragraph. Third, separate cold start from steady state. Fourth, do not mix percentiles from different sample sets. Fifth, preserve errors in the report. Dropping timed-out requests can make an overloaded service appear faster. Sixth, verify actual frames and visual tokens after preprocessing, since automatic sampling or resize can silently change the test.

Finally, keep quality evaluation separate and paired. A latency optimization is useful only if output behavior remains acceptable. The Qwen processor work used bit-identical validation. The Gemma implementation retained a sequential pooling step after fully batching it caused numerical drift. When exact identity is unavailable, run a frozen task evaluation and report the tolerance and observed changes.

How to interpret the published results

The Qwen 11,574 ms p90 at 5 QPS demonstrates queue amplification from a serialized CPU stage. It does not mean Qwen3.6 model execution inherently takes eleven seconds. The optimized ~80 ms result shows that removing the processing bottleneck and provisioning the runtime appropriately kept that stage stable at the tested load. A service can still encounter another bottleneck at higher QPS.

The Gemma ~120 ms p95 TTFT at 20 QPS and ~5 ms median ITL through 10 QPS demonstrate the effect of batching vision-encoder work in the tested server. They do not imply the same figures on another GPU, with more frames, longer outputs, or a different model revision. Reproduce the curve on the intended stack before using any value in a service-level objective.

What these benchmarks do not claim

There are no detector mAP measurements here. There are no Jetson latency, throughput, power, or thermal measurements here. The article makes no claim about object-detector accuracy, edge-device performance, offline operation, or industrial customer outcomes. Overshoot performs cloud VLM inference and does not offer local detectors or Jetson certification. Any table that supplied such rows without a real test would be fabricated.

A detector benchmark should specify dataset, split, class mapping, input size, confidence threshold, non-maximum suppression, precision, recall, and mAP definition. An edge benchmark should specify exact device SKU, power mode, clocks, runtime, precision, batch size, thermals, and sustained duration. Those are valuable measurements when they exist. They cannot be inferred from the cloud multimodal results in this article.

Summary

• Qwen3.6-27B-FP8 preprocessing fell from 428 ms p90 to 81 ms for 15 480p frames on an H200.
• At 5 QPS, the same optimized Qwen stage stayed near 80 ms while stock processing reached 11,574 ms p90.
• Gemma 4 batching held p95 TTFT near 120 ms at 20 QPS and median ITL near 5 ms through 10 QPS.
• Resolution, frame count, visual tokens, sampling, and queue behavior must appear in every useful video benchmark.
• No detector mAP or Jetson results are reported; reproducibility is preferable to fabricated coverage.

Review the complete engineering evidence: read the Qwen multimodal processor case study and the Gemma 4 batching case study, then use the Overshoot quickstart to measure your own fixed workload.