Real-time video agents for live, queryable streams

The agent loop has four explicit stages

Start by creating a Stream with POST /v1beta/streams. The response contains a stream id plus a LiveKit room URL and short-lived publish token. A browser, native client, or server publisher connects to that room and sends a video track over WebRTC. Overshoot retains a rolling 600 seconds of frames while the Stream remains active. The agent then chooses a visual anchor, asks a focused question, and handles the model output in ordinary application code.

This separation keeps agency visible. The API does not decide when to inspect video or run continuously on every frame. A motion event, user click, timer, or upstream business event can schedule a request. The application can parse the response, request review, call another service, or discard a low-confidence result. Query cadence and output budgets stay under developer control, which matters for latency, cost, and operational clarity.

1. Create a leased Stream
2. Publish WebRTC video with LiveKit
3. Reference a frame or segment
4. Stream model output over SSE
5. Validate and act in application code

Address the exact visual context

Use an image_url content part for a single moment. frame_index=-1 resolves to the latest retained frame at request time. A fixed frame_index supports repeatable follow-up questions while that frame remains in history. timestamp_ms selects by Stream time, and offset_ms selects relative to the live edge. These anchors let an agent distinguish a current-state check from an investigation of a known moment.

Use video_url when motion or ordering matters. start_offset_ms=-5000 selects the trailing five seconds, while an end anchor closes a bounded interval. max_fps controls how densely the segment is sampled, and its default is 1.0. More frames can expose a transition, yet they also add visual preprocessing and model work. Choose the shortest window and lowest useful sampling rate for the task.

Shape requests for dependable answers

A useful prompt names the observation, expected format, and uncertainty behavior. Ask for visible evidence before an action label. For example: "Describe the current station state in one sentence, list the visible evidence, and return unknown when the view is blocked." Keep max_completion_tokens small for routing decisions. Use response_format only after confirming that the selected model supports it, since provider capabilities differ.

Send related requests with a stable thread_id when prompt-cache support applies. Discover model availability from GET /v1beta/models at runtime and select entries whose status is ready. Model ids use provider/model-name for Overshoot-hosted models and should be passed verbatim. Availability can change, so a hard-coded catalog becomes stale. Define an explicit fallback or a clear unavailable state in the product.

Operate the Stream lifecycle

A Stream has a 300-second lease. Call POST /streams/{stream_id}/keepalive before expiration, often every 90 to 120 seconds, and pass the fresh token to the LiveKit room when the client supports token updates. If the lease expires, the Stream moves from active to ended and cannot resume. Create a new Stream and republish after that terminal transition. Delete a Stream when the session closes so resources do not linger.

Treat publisher connection state and Stream state as separate signals. A connected publisher can lose network access, and an active Stream can temporarily have no recent frame. GET /streams/{stream_id} exposes lifecycle and frame-history fields that help a UI explain what is happening. Before querying, check that recent frames exist. On shutdown, stop local tracks, disconnect LiveKit, stop keepalive scheduling, and delete the Stream.

Design safety and review into the agent

Vision-language output is probabilistic. Keep hazardous machine interlocks, access control, and other irreversible safety functions in deterministic systems designed for those duties. A video agent can summarize context, prioritize review, draft an explanation, or request a second check. Give people the source moment, prompt, model id, and model answer when a decision needs review. Preserve evidence under your own retention policy because Stream history rolls off after 600 seconds.

Evaluate the actual camera, lighting, occlusion, motion, language, and rare failure cases. Measure false positive and false negative rates against a written acceptance rubric. Add an unknown path instead of forcing every observation into a decision. Run shadow mode before enabling workflow actions. These practices turn an impressive demonstration into a system whose limits are visible to operators and engineers.

Measure a complete response path

Overshoot-hosted vision models respond in 200ms. For your own product decision, measure from the scheduling event through the first useful token at the client. That interval includes capture, WebRTC transport, frame selection, visual preprocessing, model prefill, generation, SSE delivery, parsing, and application work. Record region, model, frame count, resolution, max_fps, prompt, output limit, concurrency, and percentile with every result.

Time to first token is useful for interactive agents. Full-response latency matters when an action requires complete structured output. Inter-token latency affects how quickly a longer explanation appears. Report all three when they answer different product questions. Compare candidates with identical media and prompts. The published Qwen and Gemma engineering studies isolate specific serving stages; they are workload-specific evidence and do not promise universal end-to-end performance.

Frequently asked questions

Does Overshoot run the whole agent?

No. Overshoot provides live Stream memory and cloud-hosted VLM inference. Your application schedules requests, validates answers, and invokes tools.

Can the agent inspect recent history?

Yes. An active Stream retains 600 seconds and supports frame, timestamp, offset, and bounded video references.

References

• Core API flow
• Stream media references
• Video agent architectures