A real-time video agent is a software system that observes a live video source, selects relevant visual context, asks a model to interpret it, and uses the response while the observed event is still actionable. The word agent describes the control loop around the model: application code decides when to look, which frames matter, what question to ask, and whether a response may trigger another operation. A vision-language model, or VLM, supplies visual reasoning inside that loop. Overshoot provides cloud-hosted VLM inference and returns answers in 200ms through an OpenAI-compatible API.
This architecture differs from batch video analytics. A batch job can upload a completed recording and wait for a report. A real-time agent works against a moving live edge, so transport, frame selection, inference latency, and action policy all affect whether its answer remains useful. It also differs from a fixed detector. A detector emits a predefined label or bounding box. A VLM can answer an application-specific question in natural language or structured text, including questions that combine objects, relationships, state, and recent motion.
How a real-time video agent works
The basic loop has four responsibilities. First, a publisher sends live video over WebRTC through LiveKit. Overshoot does not expose a direct RTSP, RTMP, ONVIF, or USB ingest API. A camera or gateway using one of those interfaces must convert its output into a WebRTC video track before publishing. Second, application code chooses an image or video interval from the Stream. Third, it calls /v1beta/chat/completions with a prompt and media reference. Finally, it parses the streamed answer and applies an action policy.
| Layer | Responsibility | Important interface |
|---|---|---|
| Publisher | Capture, encode, and send live video | WebRTC track published through LiveKit |
| Stream | Retain addressable recent visual history | 600-second history and 300-second lease |
| Agent controller | Schedule queries and select visual context | Timers, user events, or application events |
| VLM inference | Interpret the selected image or interval | OpenAI-compatible /v1beta/chat/completions |
| Action layer | Validate output and invoke allowed operations | Application-owned policy and tools |
These layers should remain explicit in production. The Stream is visual memory, not the agent itself. The chat completion is one observation, not a durable workflow. The controller owns state such as the last accepted answer, cooldowns, retries, and task progress. The action layer owns authorization. Keeping these boundaries clear makes it possible to change models or query cadence without rewriting capture and business logic.
Live video, Stream history, and visual anchors
After creating a Stream, the publisher connects to the returned LiveKit room and publishes a video track. A Stream lease lasts 300 seconds. A long-running publisher renews it with the keepalive endpoint before expiry and gives LiveKit the fresh publish token returned by that request. Stream states move forward: an ended Stream cannot be resumed. Delete the Stream when the session is complete rather than leaving lifecycle cleanup implicit.
Overshoot retains 600 seconds of recent Stream history. Agent code addresses that history with an ovs:// reference. For an image, frame_index=-1 means the latest available frame. An image can also be selected by timestamp or a negative offset from the live edge. A video reference uses start and end anchors plus an optional max_fps; its default is 1.0. The reference is interpreted by the service. It is not a URL that a browser should fetch.
The same chat API also accepts HTTPS and data media. Those forms are useful for a stored image or an application-generated snapshot. Use an ovs:// reference when the source is a live Stream or when a question needs recent Stream history. This avoids repeatedly downloading, clipping, and uploading the same live media from application code.
Image questions and temporal questions
Choose the smallest visual context that contains the evidence. A latest-frame image is appropriate for questions such as whether a doorway is blocked, which screen is visible, or whether a work surface is clear. A short video interval is appropriate when direction, order, duration, or change matters. Examples include whether an item was placed and then removed, whether a person crossed a boundary, or which step happened first.
Sending a longer interval is not automatically safer. Every additional sampled frame adds visual processing and model input. Sparse sampling can miss a brief event, while dense sampling can increase latency and cost without improving the answer. Select an interval from the event timescale, test the default 1.0 max_fps, then raise it only when evaluation footage shows that the event is otherwise missed.
A minimal agent query
The following request asks about the latest frame and consumes the response as server-sent events, or SSE. The exact model id should come from GET /v1beta/models because the available model set and readiness are dynamic. Model ids containing a slash identify Overshoot-hosted models. The request keeps the answer narrow so application code can validate it before use.
Query the latest Stream frame
curl -N https://api.overshoot.ai/v1beta/chat/completions \
-H "Authorization: Bearer $OVERSHOOT_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "provider/model-name",
"stream": true,
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Return CLEAR or BLOCKED for the doorway."},
{"type": "image_url", "image_url": {
"url": "ovs://streams/STREAM_ID?frame_index=-1"
}}
]
}],
"max_tokens": 12
}'SSE sends partial completion chunks and ends with data: [DONE]. A controller can render those chunks for an interactive user, but a machine action should wait for a complete, valid response. If the response must become an enum or JSON object, reject extra fields and unknown values. A low-latency invalid answer is still invalid.
Agent scheduling and state
There is no single correct query cadence. User-driven agents query when a person asks a question. Event-driven agents query after another system reports a relevant change. Periodic monitors query on a timer. A workflow agent may query after each accepted step. In every case, the controller should make cadence visible in code and logs. Hidden loops are difficult to cost, debug, and stop.
State should be compact and task-specific. Store the prior decision, its anchor, the prompt version, and any cooldown needed to prevent repeated actions. Use a stable thread_id when the application intentionally benefits from prompt caching, and record the cache metadata returned by Overshoot. Do not treat conversation history as a substitute for retrieving the right frames. Text claiming that an event happened is weaker evidence than a visual anchor covering the event.
Define concurrency beside cadence. If only the latest scene matters, allow one active request and discard a trigger that has already become stale. If every event requires review, place event timestamps in a bounded durable queue and accept that the agent is processing history. Cancellation alone does not prove remote work stopped, so attach a decision id and make downstream actions idempotent. This prevents a retried request or late completion from producing the same action twice. The policy should also say what happens when a newer observation contradicts a response still waiting for validation.
Where video agents are useful
Interactive assistance is a natural fit. A remote operator can ask what is visible, request a summary of a recent interval, or verify that a step was completed. A consumer application can guide a user through a physical task while checking the current scene. A developer tool can inspect an application preview or device screen and answer a narrowly scoped question.
Monitoring is also possible when the task tolerates probabilistic interpretation. The agent can classify scene state, compare a current view with an expected condition, or produce a concise explanation for a human reviewer. These systems work best when the output informs a reversible software action or queues a review. A VLM should not sit directly in a safety-rated control path. Physical interlocks and deterministic safety systems remain separate.
Production design checks
Evaluate the whole loop on footage from the intended camera position. Include lighting changes, occlusion, motion blur, empty scenes, and plausible hard negatives. Accuracy should be measured on the exact prompt, model id, frame selection policy, and output parser. Changing any of those elements changes the system under test. Measure capture-to-first-token latency from the application region instead of quoting model execution alone.
Plan for disconnects and terminal Stream states. Renew leases before 300 seconds, handle a failed keepalive by recreating the Stream, and expose whether the publisher is still sending frames. Handle unavailable models by consulting /models and choosing an evaluated fallback. Overshoot operates in two regions, so test network behavior from the deployment location that will actually make requests.
Finally, log enough context to reproduce each decision: Stream id, selected anchor, model id, prompt version, timing, parsed result, and action outcome. Correlate retries under one decision id so repeated network attempts do not look like independent observations. Track response rejection and uncertain outcomes beside successful actions. Avoid logging sensitive frames or generated text without a retention policy. Observability should explain both what the model saw and what the controller did with its answer.
Summary
- A real-time video agent combines live transport, visual context selection, VLM inference, controller state, and an application-owned action policy.
- Overshoot ingests live video over WebRTC through LiveKit and exposes recent frames through 600-second Stream history and ovs:// anchors.
- Use latest-frame queries for current state and bounded video intervals for motion or sequence.
- Validate complete model output before acting, and keep safety controls outside the probabilistic loop.
- Benchmark the full loop on representative footage, load, region, and failure conditions.
Build the first observation loop: follow the Overshoot quickstart, then read the Gemma 4 inference case study for a concrete account of latency under concurrent video load.