Video agents that reason over live camera context

Separate observation from orchestration

Overshoot resolves visual context and runs cloud-hosted VLM inference. Your orchestrator decides when an observation is needed. It might respond to a person asking a question, a timer, a detector event, a workflow transition, or a state change elsewhere in the product. The orchestrator also owns retry policy, tool permissions, durable conversation state, and the final interpretation of a model response.

This boundary makes an agent inspectable. Each inference call has a model id, prompt, media reference, and output. Each application action has a separate rule and audit event. Avoid a hidden loop that queries continuously without a stated purpose. Event-driven scheduling reduces unnecessary calls and lets engineers trace why the agent looked at a particular moment.

Use Stream history as short-term visual memory

The 600-second rolling history lets an agent revisit a recent moment without uploading a clip. frame_index=-1 means the latest frame. Negative offsets select time relative to the live edge. A bounded video reference captures a short transition, and max_fps defaults to 1.0. The agent can first ask about current state, then inspect the preceding interval when the answer indicates a change.

Treat this history as ephemeral working context. Save durable state as text or application records, and preserve media evidence separately when policy allows and the workflow requires it. Once frames leave the retained window, an ovs:// reference cannot recover them. Keep the exact anchor with every result so a follow-up request uses the intended visual moment instead of silently moving to the newest frame.

Design prompts around tools and decisions

Ask the model for observations that your tool policy can evaluate. A response such as "A sealed carton is visible at station two" is easier to review than a broad instruction to manage the station. Define allowed labels, required evidence, and an unknown value. A second application rule can decide whether to notify an operator, request another frame, or create a review item.

Keep tool arguments narrow and validate them before execution. Model support for tool and structured response features varies, so test the current ready model from /models. For consequential actions, require confirmation or independent evidence. The model should never receive credentials or permissions that exceed the one operation under consideration. Record the prompt version and parsed arguments with the action result.

Manage sessions and reconnects

A Stream lease lasts 300 seconds. Schedule keepalive calls before expiry and update the connected LiveKit room with the fresh publish token returned by each successful call. GET /streams/{stream_id} can confirm active state and recent frame availability. When a Stream ends, stop queries against that id and create a replacement. The state transition to ended is terminal.

Network recovery should distinguish a LiveKit reconnect from a Stream replacement. A temporary publisher interruption may recover within the same active Stream. Expiration requires a new id and new media references. Expose these states in the product so users know whether the agent is seeing current video, waiting for frames, reconnecting, or unavailable. Delete active Streams during deliberate shutdown.

Keep latency aligned with the interaction

Overshoot-hosted vision models respond in 200ms. The agent experience also includes scheduling, media transport, request construction, token streaming, parsing, tool execution, and user-interface updates. Measure from the event that triggered observation to the first output the user or workflow can use. Use small output limits for classification and routing. Longer explanations can stream after an early status appears.

Frame and segment choices are part of performance engineering. A latest-frame question is smaller than a sampled ten-second interval. Test task accuracy before increasing frame count. At load, collect tail latency and failure rates across many requests. The Qwen preprocessing and Gemma batching studies demonstrate useful stage-level methods, while their exact values remain specific to the documented workloads.

Ship a narrow first agent

Choose one source, trigger, observation, and reversible action. Build a labeled evaluation set with ordinary scenes and failure cases. Run in shadow mode. Compare the model result with the existing human or software process. Review unknown answers, occlusions, stale frames, disconnections, and model unavailability. Decide how the interface communicates each state.

Expand only after the first loop has measurable value and a clear owner. Add another tool or visual question as a separate capability with its own acceptance criteria. Keep deterministic controls outside the agent. This incremental approach produces a video agent whose behavior can be explained from trigger through source evidence to action, which is more important than maximizing autonomy on the first release.

References

• What is a Stream?
• Media reference syntax
• What are real-time video agents?