Computer vision for live and historical context

Choose the visual method from the output

Start with the decision your software needs. A detector fits a stable set of classes and often returns boxes, scores, or masks. Optical character recognition fits text extraction. A tracking system fits identity across frames. A VLM fits questions whose labels change, whose answer needs surrounding context, or whose result is naturally expressed in language. Production systems can combine these methods and route each observation to the least complex component that meets the requirement.

Overshoot does not claim detector accuracy, object geometry, or a universal task score. Evaluate the selected model against your own labeled examples. Ask for concise evidence and a bounded output. If an existing detector already identifies an event cheaply, use that event to schedule a richer VLM query. This architecture limits unnecessary model calls and gives the language model a specific question to answer.

Connect live video through WebRTC

Create a Stream through /v1beta, then use the returned LiveKit URL and token from a browser, native client, or server publisher. Browser capture supports the media tracks exposed by the browser, including camera and screen share. A server publisher can produce a LiveKit track from another internal source. The Overshoot API contract begins at that published WebRTC track and does not advertise direct camera-protocol ingest.

Keep the Stream alive while the publisher is in use. The lease is 300 seconds and each keepalive returns a fresh publish token. Monitor frame arrival independently from room connectivity. Ended Streams never resume. A robust integration creates a replacement Stream, reconnects the publisher, and updates the application reference when an expiration or explicit deletion ends the old session.

Query recent context without uploading clips

An active Stream retains a rolling 600 seconds. For current-state computer vision, reference frame_index=-1 in an image_url part. For incident review, choose an exact frame, timestamp, or offset. For motion, use video_url with start and end anchors. The default segment max_fps is 1.0. This reference model avoids creating a new media file for each question and keeps selection close to inference.

History is temporary working memory. Frames leave the window as time advances, and deletion or expiration ends access. Store an artifact in your own system when audit or long-term review requires one. Log the Stream id, anchor, model id, prompt version, and response so an engineer can understand which observation produced a result. Avoid logging sensitive imagery or answers beyond the policy for that application.

Use prompts as versioned application logic

Write prompts with the same discipline as code. Define the observation scope, output fields, allowed values, and unknown state. Keep a version identifier next to each prompt. Test changes against a fixed evaluation set before release. Natural language can express a changing requirement quickly, although that flexibility also creates room for ambiguity. Short sentences and concrete visual criteria usually outperform broad requests for analysis.

When downstream code needs JSON, request a small schema and validate the parsed result. response_format support varies by model, so confirm capability during model evaluation. Set max_completion_tokens to the minimum useful budget. Do not treat fluent explanation as confidence. If confidence matters, derive operational confidence from measured task performance, agreement checks, visible evidence, or human review.

Understand the cloud deployment boundary

Overshoot runs VLM inference in the cloud. Publishers need network access to us-west1 or us-central1. Applications that must continue offline, keep every pixel on a device, or meet a hard local control deadline need a local runtime for those requirements. A connected design can still keep fast deterministic logic near the camera and request cloud VLM interpretation for selected events.

Measure capture-to-answer behavior on the target network. Overshoot-hosted models respond in 200ms, while application latency also contains capture, uplink, model request, SSE delivery, parsing, and workflow work. Use percentiles and representative concurrency. Record frame count and resolution because visual preprocessing grows with the media envelope. Compare model candidates using the same input and output budget.

Build a complete request and response path

A practical integration starts on a trusted server. Call GET /v1beta/models, filter for status ready, and select a tested model id. Create the Stream on that server so the organization API key never enters browser code. Return only the LiveKit room URL, short-lived publish token, and stream id to the authorized publisher. After frames arrive, send a user message with text plus an image_url whose URL is ovs://streams/{stream_id}?frame_index=-1. Set stream to true and cap the answer with max_completion_tokens.

Read the response as SSE instead of assuming network chunks contain complete events. Buffer partial text, parse complete data lines, append choices[0].delta.content, and stop at data: [DONE]. Cancel the reader when the user changes sources or closes the view. If a structured answer ends before [DONE], reject it as incomplete. Keep API errors distinct from an unknown visual answer. This request path gives the interface enough state to show source readiness, model availability, inference progress, completion, and recoverable failure.

Move from prototype to controlled production

Begin with one camera, one question, and one reversible outcome. Build an evaluation set from normal operation and difficult edge cases. Run the proposed workflow in shadow mode and compare answers with a trusted process. Track model availability from /models, request failures, empty-frame conditions, time to first token, parse failures, and task errors. Define a visible degraded state when inference is unavailable.

Keep people in the loop for consequential decisions. Preserve deterministic safety systems. Review privacy, access, and evidence retention before adding more sources. A useful production computer vision feature has a narrow purpose, observable inputs, measurable acceptance criteria, and a recovery path. The API makes visual context programmable; engineering practice determines whether the surrounding product is reliable.

References

• Current models
• LiveKit publishing flow
• Vision-language models and computer vision