Video Agent Architecture Patterns: How to Design Real-Time CV Systems

A video agent architecture is the arrangement of live video transport, visual memory, inference requests, controller state, and action policies used to turn a changing scene into timely software decisions. The architecture determines when a model observes the scene and what it can do with the answer. Overshoot supplies cloud-hosted VLM inference with 200ms answers, WebRTC ingest through LiveKit, and an OpenAI-compatible API. Application code remains responsible for scheduling, state, validation, and tools.

Most systems do not need a free-running autonomous loop. They need a clear trigger, a bounded piece of visual evidence, and a narrow response contract. Choosing that control pattern early makes latency, cost, failure behavior, and evaluation much easier to reason about.

The common architecture underneath every pattern

The media plane starts with a publisher that captures and encodes video, then publishes a WebRTC track through LiveKit. Overshoot does not offer direct RTSP, RTMP, ONVIF, or USB ingest. A camera gateway must translate those sources into WebRTC. The resulting Stream retains 600 seconds of recent history and has a 300-second renewable lease.

The control plane creates Streams, schedules model requests, selects anchors, parses responses, and calls allowed tools. It queries /v1beta/chat/completions with HTTPS media, data media, or an ovs:// Stream reference. SSE returns generated tokens incrementally. The current model inventory comes from GET /v1beta/models, so model availability should be discovered rather than assumed.

Pattern 1: On-demand observer

The on-demand observer runs only when a user or calling service asks a question. It usually selects the latest frame with frame_index=-1, though the caller can request a recent interval for a temporal question. This pattern has simple cost control because every inference maps to an explicit request. It also gives the user a natural place to clarify an ambiguous answer.

Keep a freshness timestamp beside the response. A question can wait in a client queue while the scene changes. If the user asks about the current state, select the frame when the request is admitted rather than reusing an older snapshot. For a follow-up question, decide explicitly whether it concerns the same anchor or the new live edge.

Pattern 2: Periodic monitor

A periodic monitor asks a fixed question at a controlled interval. It suits conditions that change more slowly than the query cadence, such as whether a work area is occupied or whether a visible process appears complete. The camera can publish continuously while the VLM is called much less frequently. Camera FPS and inference QPS are independent design variables.

Use a single-flight loop so a new tick does not create an unbounded backlog. If a previous request is still running, skip or replace it according to the task. Add hysteresis before changing application state: require repeated agreement, a minimum duration, or a human confirmation when a false transition is costly. Store the anchor behind each accepted state so an operator can inspect the evidence.

Pattern 3: Event-triggered semantic review

An event-triggered reviewer runs after another system provides a timestamp or state change. That event might come from application logic, a device signal, or a user action. The controller uses Stream history to select visual evidence before and after the event, then asks the VLM a focused semantic question. This pattern avoids continuous model calls and makes temporal context precise.

Overshoot retains 600 seconds of Stream history, so the request can use timestamp or offset anchors after the trigger arrives. A video reference can specify start and end anchors and max_fps, with 1.0 as the default. Set the interval from event timing and expected duration. A narrow interval is easier to evaluate and usually costs less than asking the model to search a long clip.

Pattern 4: Workflow verifier

A workflow verifier combines visual observations with an explicit state machine. The state machine knows which step is expected, while the VLM checks whether visible evidence satisfies that step. Only an accepted result advances the workflow. This is more reliable than asking a model to infer the entire process and maintain all state inside conversation history.

Prompts should be state-specific. Ask one question about the expected condition, define allowed answers, and include an uncertain outcome. Persist the workflow transition, prompt version, model id, anchor, and parsed answer together. On restart, the application can reconstruct state without relying on model memory. For irreversible or high-cost transitions, require deterministic checks or human approval.

Pattern 5: Evidence-first human escalation

Some decisions should end at a review queue. The VLM can summarize the scene, identify why a case may need attention, and attach the exact visual anchor. A reviewer then sees the same evidence and chooses the action. This pattern is useful when errors are expensive, policy requires oversight, or evaluation shows a meaningful ambiguous region.

Do not show generated prose without its evidence and age. Include the Stream timestamp, frame or interval selection, and model result. Because Stream history is finite, retain approved artifacts through the application storage policy when review may occur after 600 seconds. Avoid copying sensitive media by default; make retention a deliberate product and compliance decision.

Implementing a bounded periodic controller

This example demonstrates the scheduling property of a periodic monitor: at most one request is active, and a timeout prevents stale work from occupying the loop forever. The completion function would call the Overshoot chat endpoint, parse SSE through data: [DONE], and validate the final result before returning it.

Single-flight monitor loop

const intervalMs = 1_000
let running = false

async function tick(streamId: string) {
  if (running) return
  running = true

  const controller = new AbortController()
  const timeout = setTimeout(() => controller.abort(), 800)

  try {
    const result = await classifyLatestFrame({
      mediaUrl: `ovs://streams/${streamId}?frame_index=-1`,
      signal: controller.signal,
      allowedValues: ['CLEAR', 'BLOCKED', 'UNCERTAIN'],
    })

    await recordObservation({
      streamId,
      observedAt: new Date().toISOString(),
      result,
    })
  } finally {
    clearTimeout(timeout)
    running = false
  }
}

const timer = setInterval(() => void tick(streamId), intervalMs)
// Call clearInterval(timer) when the session ends.

The 1-second interval and 800 ms timeout are example policy values, not service recommendations. Set both from the application action window and measured latency distribution. If every observation must be processed, use a bounded durable queue and accept that the result may describe historical evidence. If only current state matters, discard superseded work.

Visual memory and agent memory

Visual memory and agent memory serve different purposes. Stream history holds recent media that can be selected by frame index, timestamp, or offset. Agent state holds accepted facts, workflow position, prior actions, and cooldowns. Keep references between them, but avoid reducing visual evidence to an unverified text summary too early.

Use thread_id when repeated chat requests intentionally benefit from prompt caching, and record the returned cache observability. It should not become hidden long-term state. Important state belongs in the application database with a schema and lifecycle. When a later decision depends on what happened visually, query the relevant interval while it remains in the Stream history or store evidence according to an explicit retention policy.

Choosing image or video context

Current-state questions need an image. Sequence, direction, and change need video. This distinction matters because video adds frames and visual tokens. For Qwen, estimated input is about 200 visual tokens for a 480p frame, 450 for 720p, and 1,010 for 1080p. Use these as model-specific planning estimates, then measure the chosen model.

Resolution should follow the smallest relevant feature. A global scene check may work at lower resolution. Small labels or distant details may need a crop or higher resolution. Sampling should follow event duration. Evaluate the frame policy as part of the agent, since a strong model cannot answer from an interval that omitted the event.

Reliability, safety, and failure handling

Every pattern needs explicit failure states. Distinguish no fresh frame, publisher disconnected, Stream ended, model unavailable, request timeout, malformed output, and uncertain result. A Stream lease lasts 300 seconds, and keepalive returns a fresh LiveKit publish token. If an ended Stream returns a keepalive failure, create a new Stream rather than trying to resume the old one.

Model output should pass schema validation and task-specific policy before any tool call. Use idempotency keys or application records for actions that might be retried. Add cooldowns for repeated observations of the same condition. Keep safety-rated controls independent from VLM output, and require review for decisions where a false positive or false negative has unacceptable cost.

Discover current readiness with /v1beta/models. Evaluate any fallback with the same prompts and footage before enabling it. Overshoot serves two regions, and cloud latency includes the application network path, so test from the actual deployment location. The Qwen and Gemma engineering benchmarks published by Overshoot are useful serving studies, but their metrics are tied to named hardware, models, media shapes, and offered load rather than a universal SLA.

How to choose an architecture pattern

Start from the trigger and consequence. If a user asks, use an on-demand observer. If a condition changes slowly, use a periodic monitor with single-flight scheduling. If another system can narrow the time window, use event-triggered review. If the application has known steps, put a workflow state machine around focused visual checks. If errors are costly, end the loop in a review queue.

Measure the selected design end to end. Record evidence time, request time, TTFT, complete response time, parser result, and action time. Test representative scenes, hard negatives, disconnects, queue pressure, and model unavailability. Architecture quality appears in predictable behavior under those conditions, not in how autonomous the diagram looks.

Summary

• Separate the WebRTC media plane from the controller, inference, and action policy.
• Choose an on-demand, periodic, event-triggered, workflow, or review pattern from the real trigger and consequence.
• Keep visual evidence in Stream anchors and durable task state in application storage.
• Bound concurrency, reject stale work, validate complete output, and model failures explicitly.
• Use independent safety controls and measure the full loop on representative footage and load.

Implement one bounded pattern: start with the Overshoot quickstart, then study the Qwen preprocessing case study to see why the media path must be profiled under load.