Real-time AI vision through one streaming API

One interface covers live and supplied media

Live media starts with POST /v1beta/streams. Connect a LiveKit SDK to the returned room and publish a camera, screen share, generated canvas, or a video track produced by your server. The public ingest surface is WebRTC through LiveKit. Sources in other transport formats require a bridge that acts as a LiveKit publisher. Overshoot does not expose direct RTSP, RTMP, ONVIF, or USB ingest in the public API.

For request-scoped media, place an HTTPS URL or data URL in an image_url or video_url content part. For live and recently retained media, use ovs://streams/{stream_id} with a query anchor. The ovs:// value is an identifier parsed inside the API. It is not a network URL and should never be fetched by the browser. This common request envelope lets one application handle camera state, uploaded evidence, and remote media with consistent response code.

Choose a single frame or a temporal window

A single frame is the fastest way to answer questions about visible state, text, count, layout, or a static condition. frame_index=-1 selects the latest retained frame. Exact frame indexes support repeatable analysis. timestamp_ms and offset_ms provide time-based selection. Include only the visual context the model needs. Extra frames increase preprocessing and prefill without improving every task.

A video segment supports questions about sequence, direction, entry, removal, or change. Choose start and end anchors around the event. The default max_fps is 1.0, so explicitly raise it only when the motion needs finer sampling and the selected model accepts the resulting input. A ten-second segment at one sampled frame per second conveys a different workload from six carefully selected frames. Record the effective media shape in tests.

Use the OpenAI-compatible request shape

POST a model id, messages array, and multimodal content parts to /v1beta/chat/completions. Set stream to true to receive SSE chunks ending with data: [DONE]. Read content from each choice delta and update the interface as tokens arrive. Cap output with max_completion_tokens when a concise answer is enough. Smaller outputs reduce generation work and simplify validation.

Compatibility makes existing client concepts familiar, while media and model support still depend on the selected provider. Query /v1beta/models and filter for status ready. Test response_format, video support, tools, and long contexts against the exact model you plan to use. Keep the model id and request shape in logs. A provider error should be surfaced as an operational state, not converted into an apparently valid visual answer.

Latest-frame content part

{
  "type": "image_url",
  "image_url": { "url": "ovs://streams/{stream_id}?frame_index=-1" }
}

Keep live sessions healthy

The Stream lease lasts 300 seconds. Renew it before the deadline with POST /streams/{stream_id}/keepalive. Each response includes a fresh LiveKit publish token. Update the room token when possible and retain reconnect logic for transient network failures. An expired or deleted Stream enters the ended state permanently. Code should create and publish to a replacement instead of retrying keepalive forever.

GET /streams/{stream_id} helps diagnose a session. It reports state, recent frame information, retained counts, lease timing, and the oldest available frame. A query that references evicted history cannot recover that media. If the product requires a durable record, save the relevant evidence separately under an explicit policy. DELETE the Stream during normal teardown and make deletion idempotent in cleanup code.

Place regions and network time in the design

The documented regions are us-west1 and us-central1. Keep related Streams in one region when a request compares multiple live sources, since one request cannot mix Stream regions. Select the closest practical region to publishers and users, then measure the complete route. Browser timing includes local capture, uplink, internet path, service processing, downlink, and rendering.

Overshoot-hosted models respond in 200ms. Treat that as a product response statement and verify your own task at realistic load. Capture p50 and tail percentiles. Separate time to first token from completion time. Include cold application paths, reconnects, long prompts, varying frame counts, and concurrent requests. A single fast local test does not describe the operator experience on a busy network.

Build an evaluation set before automation

Collect examples from the real visual environment, including blur, glare, partial views, empty scenes, unexpected objects, and ambiguous cases. Write the desired answer and the acceptable unknown behavior before tuning prompts. Score task-level correctness. Treat prose quality as secondary. For structured output, validate syntax and allowed values before another system consumes the response.

Start with operator assistance or a review queue. Show the queried frame or segment beside the answer. Add automated action only after measuring errors and defining recovery. Deterministic safety controls should remain outside a VLM path. Real-time AI vision is most useful when language-level interpretation shortens a human workflow or gives software context that a fixed label set cannot express.

References

• Quickstart
• Chat completions
• Computer vision benchmarks