Real-time inference for computer vision is the process of turning a current image or recent video interval into a model result before that result loses operational value. The time budget depends on the application: an interactive assistant, a monitoring workflow, and an offline indexer have different deadlines. Overshoot provides cloud-hosted VLM inference with 200ms answers, live WebRTC ingest through LiveKit, and an OpenAI-compatible API. A production design still needs to measure its own capture, network, queueing, model, and output costs.
Real-time is therefore an end-to-end property. A fast model cannot compensate for a stale frame, an overloaded preprocessor, or a long output. Likewise, high camera frame rate does not guarantee fast decisions. The useful metric starts when the relevant visual evidence exists and ends when the application has enough validated output to proceed.
What real-time inference means
A deadline should come from the action window. If a user is waiting for an answer, time to first token often governs perceived responsiveness. If software needs a complete structured result, full-response latency matters. If the system observes a brief event, freshness and sampling coverage matter before inference even starts. Define the event timestamp, the usable completion point, and the maximum acceptable age of the selected frame.
| Metric | Definition | Use |
|---|---|---|
| Capture-to-request | Evidence timestamp to request admission | Detect stale capture, buffering, and transport |
| TTFT | Request admission to first generated token | Measure interactive response and prefill delay |
| ITL | Time between generated tokens | Measure decode smoothness under load |
| Full-response latency | Request admission to validated completion | Measure machine-action readiness |
| Freshness at action | Action time minus evidence timestamp | Measure whether the result still describes the scene |
Report percentiles alongside averages. A p50 describes the ordinary request. A p95 or p99 reveals queueing, cold paths, and contention that an average can hide. The chosen percentile should match the product risk. Also report failures and timeouts separately; dropping slow requests from a latency chart creates an unrealistically clean result.
The end-to-end vision inference pipeline
A live request starts with capture and encoding. The publisher sends a WebRTC video track through LiveKit. Overshoot has no direct RTSP, RTMP, ONVIF, or USB API, so sources using those interfaces need a gateway that publishes WebRTC. The service stores recent Stream history, and application code selects a frame or interval with an ovs:// anchor. Static inputs can instead use HTTPS or data media.
After admission, media passes through CPU preprocessing, the vision encoder, language-model prefill, and token decode. CPU preprocessing may decode, resize, normalize, patch, and arrange frames. The vision encoder converts those patches into visual representations. Prefill processes visual and text input and emits the first token. Decode emits subsequent output tokens. Network transfer and request queueing surround those compute stages.
Instrument each boundary with a monotonic clock where possible. Server timing can describe remote stages, while the client records capture time, request start, first SSE chunk, completed response, and parsed result. Preserve a trace id across the controller and action layer. This turns a vague latency regression into a stage that can be profiled.
Frame selection, resolution, and visual tokens
Visual input size strongly affects preprocessing and prefill. For Qwen, estimated visual token counts are about 200 for one 480p frame, 450 for 720p, and 1,010 for 1080p. The 1080p estimate is roughly five times the 480p estimate. These are planning estimates for that model family, not a universal conversion across VLMs. Dynamic resizing, aspect ratio, and model-specific patching can change the count.
Use resolution that preserves task evidence. Coarse scene state may survive at 480p. Reading small text, distinguishing fine parts, or inspecting a distant region may require more pixels or a crop. A crop can retain local detail while reducing irrelevant image area. Test accuracy at each candidate shape rather than assuming the highest resolution is required.
Frame count needs the same discipline. An image question should generally use one relevant frame. Motion and sequence require a bounded interval. Overshoot video references accept start and end anchors and an optional max_fps, which defaults to 1.0. Increasing sampling increases the chance of covering a short event, but also adds visual work. Choose it from measured event duration and task accuracy.
How to issue and time a streaming request
Overshoot exposes /v1beta/chat/completions with an OpenAI-compatible request shape. Streaming responses use SSE. The example below records time to first SSE data and full stream duration in a browser or Node environment. In a real application, parse each JSON chunk and assemble the content rather than only counting bytes.
Measure first event and full stream time
const started = performance.now()
const response = await fetch(
'https://api.overshoot.ai/v1beta/chat/completions',
{
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.OVERSHOOT_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'provider/model-name',
stream: true,
max_tokens: 32,
messages: [{
role: 'user',
content: [
{ type: 'text', text: 'Describe the current workstation state.' },
{ type: 'image_url', image_url: {
url: 'ovs://streams/STREAM_ID?frame_index=-1',
} },
],
}],
}),
},
)
if (!response.ok || !response.body) throw new Error(`HTTP ${response.status}`)
const reader = response.body.getReader()
let firstEventMs: number | undefined
for (;;) {
const { done } = await reader.read()
if (done) break
firstEventMs ??= performance.now() - started
}
console.log({
firstEventMs,
fullStreamMs: performance.now() - started,
})This client measurement includes network time and is useful for product behavior. It does not isolate model TTFT because the first byte can include SSE framing or service metadata. For controlled inference studies, combine client timing with server-side stage measurements and use a fixed request body.
Benchmark methodology that survives review
A reproducible benchmark identifies the exact model id, hardware, quantization, software revision, prompt, output token limit, media shape, frame count, region, concurrency pattern, and percentile. It also distinguishes offered load from achieved throughput. A closed-loop test, where each worker waits for a response before sending another request, can conceal overload. A constant-rate test continues offering the target QPS and exposes queue growth.
Warm the deployment, then run long enough to cover ordinary scheduler variation. Use different media per request or explicitly disable caches when measuring uncached work. Confirm that a framework is not silently resampling the video. Save the raw request timestamps and percentile calculation method with the result so another engineer can recompute the chart. Record accuracy on a fixed evaluation set beside latency. An optimization that drops the frame containing the event may produce an impressive chart and a worse system.
Measure cold and warm paths separately. Connection setup, process startup, model loading, and empty caches can dominate the first request while disappearing from a long steady-state run. Both views matter, but combining them into one distribution makes diagnosis harder. Keep client clocks synchronized when evidence and completion timestamps come from different machines, and record clock uncertainty for freshness calculations. During load tests, graph queue depth and resource utilization beside latency. CPU saturation, accelerator utilization, memory pressure, and active request count help explain where the system crosses from stable throughput into accumulating delay.
Keep product claims and engineering benchmark results separate. Overshoot answers in 200ms. The studies below diagnose specific inference paths on named workloads. They should not be read as a universal service-level agreement, a guarantee for another model, or a prediction for different media.
What the Qwen preprocessing benchmark found
A Qwen3.6-27B-FP8 study on an H200 used 15 frames at 480p and disabled the framework behavior that would otherwise downsample the video. Stock CPU preprocessing took 428 ms p90 for sequential requests. Under a sustained offered load of 5 QPS, the same stock path reached 11,574 ms p90. Profiling showed that the delay occurred before GPU inference.
The optimized processor preserved bit-identical output while reducing the 15-frame sequential p90 to 81 ms. At 5 QPS it remained around 80 ms p90 instead of accumulating an 11-second queue. The engineering lesson is broader than the specific numbers: multimedia preprocessing is part of the serving system, and it must be load-tested independently from GPU execution.
What the Gemma batching benchmark found
A separate Gemma study used six 480p frames representing about 500 visual tokens. Before the fix, the vision encoder processed frames serially and delayed continuous batching. Video prefills could therefore block decode work from other requests. Low single-request latency did not predict behavior once requests overlapped.
After batching the appropriate vision-encoder work, p95 TTFT stayed around 120 ms at 20 QPS for that workload. Median ITL stayed around 5 ms through 10 QPS. These results describe the tested Gemma deployment and request shape. They demonstrate why TTFT and ITL should be plotted against offered QPS, rather than presented as one number measured at concurrency one.
Throughput, queueing, and backpressure
When arrival rate approaches service capacity, latency can rise much faster than throughput. Queueing may appear in CPU media processing, the model scheduler, or an application worker. Bound every queue. If the result concerns the current scene, dropping an obsolete request can be better than processing it after newer evidence has arrived. Carry the evidence timestamp so the consumer can reject stale results.
Cadence should respond to product need, not camera FPS. A 30 FPS publisher does not imply 30 VLM calls per second. A timer, user action, state transition, or upstream event can trigger a query. Debounce repeated triggers, cap concurrent requests per Stream, and set timeouts from the action window. Backoff for rate limits, but do not retry a stale visual question indefinitely.
Operating live inference in production
Live transport and model serving have separate lifecycles. A Stream retains 600 seconds of history and has a 300-second lease. Call keepalive before expiry and update the LiveKit room with the fresh token. Treat an ended Stream as terminal. Health reporting should distinguish publisher disconnected, Stream ended, model unavailable, request timed out, and response rejected by the parser.
Query GET /v1beta/models rather than hard-coding an assumption that every model is ready. The model inventory is dynamic. Evaluate fallbacks ahead of time because two VLMs may format answers differently or support different media. Overshoot serves two regions; run acceptance tests from the region and network path used by the application.
Set production gates for both quality and latency. A useful release check might require a task-specific accuracy threshold, a percentile freshness target, a maximum timeout rate, and correct behavior during disconnect and overload tests. Keep model output away from safety-rated controls. Use deterministic application validation and human review where an incorrect action has meaningful cost.
Summary
- Define real-time performance from evidence capture through validated application output.
- Measure TTFT, ITL, full response, freshness, failures, and percentiles under stated offered load.
- Control resolution, frame count, interval length, and sampling from task evidence rather than camera rate.
- Profile CPU preprocessing and vision encoding as well as language-model execution.
- Use bounded queues, stale-result rejection, dynamic model discovery, and explicit Stream lifecycle handling.
Set up a reproducible live test: use the chat completions API reference, then compare your stage measurements with the Qwen multimodal processor case study.