Vision-language models for live video

VLMs answer flexible visual questions

A detector is usually trained around a fixed label space. A VLM can follow a language instruction that changes by workflow, explain visible context, read text in relation to the scene, compare images, or summarize a short sequence. This flexibility works well for operator assistance, exception triage, visual question answering, and products where users ask their own questions.

Flexibility does not remove the need for evaluation. VLMs can miss small details, infer unsupported context, or answer confidently when the view is ambiguous. Build examples from the real source and score task outcomes. Ask for visible evidence and an unknown state. Keep calibrated measurement, geometric localization, and safety interlocks in systems designed for those functions.

Discover model availability at runtime

GET /v1beta/models returns a list with model ids and ready or unavailable status. Pass the chosen id verbatim in the chat-completions request. Ids containing provider/model-name identify Overshoot-hosted models on the fast managed path. Ids without a slash can represent proprietary passthrough options with different latency and capability characteristics. Availability changes, so applications should refresh the catalog and define a tested fallback policy.

Model selection should combine task accuracy, media support, latency, output behavior, and operational availability. A ready model may still have limits on video, response formats, tools, or context. Test each feature through the exact API request you plan to use. Avoid ranking models from general reputation alone. A smaller model that reliably answers one bounded camera question may be a stronger product choice.

Represent visual media precisely

Use an image_url part for one image and a video_url part for temporal context. Each can carry an HTTPS URL or data URL where supported. An ovs:// Stream reference resolves retained live media inside Overshoot. For an image, frame_index=-1 selects the latest frame. For video, use start and end anchors with an optional max_fps. The default max_fps is 1.0.

The model sees only the media resolved for that request. Choose the smallest interval that contains the evidence. Exact frame indexes make follow-up analysis stable while the frame remains retained. Offsets are convenient near the live edge. Multiple content parts can provide separate views, and Stream references in a single request must belong to the same region. The documented regions are us-west1 and us-central1.

Budget visual and text tokens

Visual tokenization varies by model family, resolution, and processor. Frame count and image size generally increase preprocessing and prefill work. The Qwen engineering material estimated roughly 200 visual tokens for one 480p frame, 450 for 720p, and 1,010 for 1080p in that specific family. These estimates are useful for workload thinking and should not be treated as universal conversion rates.

Start with lower resolution and fewer frames, then measure task quality. Set max_completion_tokens according to the output contract. A one-label response needs a smaller budget than a detailed explanation. Reuse a stable thread_id for related prompts when cache behavior is relevant. Log model, input shape, output length, and cache observability with latency results so later changes can be explained.

Stream output and validate it

Set stream to true to receive server-sent events. Parse complete data lines, extract each delta, and stop at data: [DONE]. SSE improves perceived responsiveness because the interface can show the answer before generation completes. It also requires cancellation, partial-chunk handling, and a clear state for network interruption. A truncated stream should never be accepted as complete structured output.

Validate every machine-consumed result. Check JSON syntax, required fields, enums, ranges, and evidence requirements. response_format can help on supported models, although support is provider-specific. For free text, constrain length and parse only a small declared portion. Treat provider and service errors separately from valid unknown answers. This distinction prevents an outage from looking like a visual judgment.

Evaluate quality and speed together

Overshoot-hosted vision models respond in 200ms. Measure your full client path with the selected region, prompt, frames, and load. Record time to first token, completion latency, inter-token latency, failures, and task score. Keep the evaluation set fixed while comparing models. Then test varied lighting, occlusion, motion, and uncommon scenes to expose brittle behavior.

The published Qwen and Gemma studies report detailed stage-level results for specific workloads. Qwen preprocessing reached 81 ms p90 for 15 480p frames on one H200 after optimization. Gemma p95 TTFT reached about 120 ms at 20 QPS for six 480p frames after batching. Those benchmarks explain engineering methods and do not substitute for your model evaluation.

References

• Current model catalog
• List models endpoint
• VLMs and computer vision