Vision-Language Models for Computer Vision: What They Can and Cannot Do in Production

A vision-language model, or VLM, is a model that interprets visual input together with text and generates a text response grounded in the supplied image or video. This interface lets computer vision applications express tasks as prompts instead of training a separate classifier for every question. Overshoot runs cloud-hosted VLM inference, accepts live Stream references and ordinary media, and returns answers in 200ms through an OpenAI-compatible chat-completions endpoint.

The flexible interface is useful, but it changes the engineering problem. A VLM response is generated text with uncertainty, even when the output looks precise. Production quality depends on the model, prompt, selected frames, resolution, scene distribution, and parser. Teams should treat a VLM as an evaluated component inside a controlled workflow rather than a general visual oracle.

How vision-language models process visual input

A typical VLM pipeline prepares the media, passes image patches through a vision encoder, projects the resulting representations into the language model, and performs language-model prefill and decode. The model then produces tokens conditioned on both the prompt and visual representations. This explains why image shape and frame count affect latency even when the text prompt remains unchanged.

Visual tokens provide a useful approximation of input size. For Qwen, one 480p frame is estimated at about 200 visual tokens, 720p at about 450, and 1080p at about 1,010. These figures are model-specific estimates. Another VLM may resize, tile, pool, or sample input differently. Measure actual behavior with the model and media form you intend to deploy.

What VLMs can do well

VLMs are effective when a task requires semantic interpretation that is awkward to represent as a fixed label set. They can describe scene state, compare visible conditions, identify relationships among objects, read prominent text, and answer focused questions. A prompt can ask whether a workspace matches an expected setup or whether the latest visible step appears complete. The same model can support several related questions without a separate training pipeline for each one.

They can also combine visual evidence with task instructions. For example, an application can supply a short checklist and ask which step is visible. A support tool can ask for a concise description using product terminology. A reviewer can request the evidence behind a classification. This language interface helps product teams iterate quickly, provided every prompt revision is evaluated like a code change.

Short video input adds temporal evidence. A VLM can reason about order, change, and coarse motion when the sampled frames contain the relevant progression. Live Stream history is especially useful here because application code can query a bounded interval after an event instead of uploading a separate clip.

What VLMs cannot guarantee

A VLM cannot guarantee that every statement is supported by the image. It may infer a plausible detail, overlook a small object, misread text, or answer from prompt expectations when the visual evidence is ambiguous. Asking for confidence does not calibrate the answer by itself. Calibration requires a labeled evaluation set and a decision rule tested against it.

A VLM also cannot recover evidence absent from the selected media. A frame taken before an event, an interval sampled too sparsely, or a crop that excludes the key region leaves the model without the fact it needs. Longer video is not a complete solution because context limits, preprocessing, and attention still constrain what can be considered. Retrieval and frame selection remain application responsibilities.

Generated text is not inherently a deterministic control signal. Even a prompt requesting one word may return punctuation, an explanation, or an unexpected label. Parse against a strict schema, reject unknown output, and define a safe fallback. Do not put a VLM directly in a safety-rated control path. Human review, physical interlocks, and deterministic controls serve different functions and should remain independent.

VLMs compared with traditional computer vision models

A task-specific classifier or detector can be a better fit when the label space is stable, training data exists, and the output needs consistent numeric structure. Such models can expose class scores, boxes, masks, or keypoints directly. They are easier to validate for one narrow task and may run with smaller compute requirements. Their cost appears in data collection, training, deployment, and maintenance when requirements change.

A VLM reduces task-specific training and supports open-ended semantics, but adds prompt sensitivity and generative variability. The choice is not exclusive. A workflow can use deterministic geometry or a specialized model to identify a region, then ask a VLM to interpret context inside it. Another workflow can use a VLM to route uncertain cases to a human without making the final decision.

Image input, video input, and live Streams

Use a single image when the answer depends on current state. The Overshoot API accepts HTTPS URLs and data media for ordinary images, plus ovs://streams/{stream_id}?frame_index=-1 for the latest live frame. Historical image anchors can select a frame index, timestamp, or offset from the live edge. An ovs:// reference is a service-side media reference, not a fetchable web URL.

Use video when order or change carries the answer. A Stream video reference specifies start and end anchors and can set max_fps. The default is 1.0. Start with the shortest interval and lowest sampling rate that preserve task accuracy. This controls visual tokens and makes the evidence easier to inspect when an answer is wrong.

Live ingest uses WebRTC through LiveKit. There is no direct Overshoot RTSP, RTMP, ONVIF, or USB ingest endpoint. Existing camera infrastructure needs a publisher or gateway that turns the source into a WebRTC track. Streams retain 600 seconds of history and use a 300-second renewable lease.

Calling a VLM with structured output expectations

The following TypeScript function asks one narrow question and validates the completed SSE content against an application enum. It deliberately treats anything else as an error. Production code should use a proper SSE parser because network chunks do not necessarily align with event boundaries; the example accumulates text first to show the validation boundary.

Validate a live-frame classification

type DoorState = 'CLEAR' | 'BLOCKED' | 'UNCERTAIN'

function parseDoorState(text: string): DoorState {
  const value = text.trim().toUpperCase()
  if (value === 'CLEAR' || value === 'BLOCKED' || value === 'UNCERTAIN') {
    return value
  }
  throw new Error(`Unexpected model output: ${text}`)
}

const body = {
  model: 'provider/model-name',
  stream: true,
  max_tokens: 8,
  messages: [{
    role: 'user',
    content: [
      { type: 'text', text: 'Answer CLEAR, BLOCKED, or UNCERTAIN.' },
      { type: 'image_url', image_url: {
        url: 'ovs://streams/STREAM_ID?frame_index=-1',
      } },
    ],
  }],
}

function acceptCompletion(completedText: string): DoorState {
  // Call this only after parsing SSE deltas and joining their content.
  return parseDoorState(completedText)
}

Schema validation controls syntax. It does not establish that the classification is visually correct. Accuracy still comes from evaluation. Include an explicit uncertain state when the application can use it, and make the prompt describe the decision boundary. Avoid asking for long reasoning when only a small action field is consumed; output length adds decode time and more surface area for parsing errors.

How to evaluate a VLM for production

Build an evaluation set from the intended camera, scene, and event distribution. Include ordinary positives, hard negatives, empty scenes, occlusion, blur, lighting changes, unusual viewpoints, and ambiguous cases. Preserve the original media so competing models receive identical evidence. Label the outcome that the application consumes, not the eloquence of the explanation.

Pin the prompt, model id, media selection policy, sampling, and parser for each run. Score false positives and false negatives separately because their costs may differ. For structured answers, report exact parse success and task accuracy. For descriptions, use a task-specific rubric with human review. Keep a holdout set that prompt authors do not repeatedly inspect.

Measure latency and cost on the same requests. Report p50 and tail TTFT, full-response time, timeout rate, and offered load. A model that is accurate at concurrency one may queue under production traffic. Repeat evaluation after model, serving, prompt, or camera changes. Model quality is a property of the deployed system and its data, not a permanent leaderboard rank.

Choosing a model and serving configuration

Start with the smallest shortlist that appears capable of the task, then use measured results. GET /v1beta/models reports the current dynamic model inventory and whether a model is ready or unavailable. Model ids with a slash identify Overshoot-hosted models; names without a slash identify proprietary passthrough models. Media support and response behavior can differ, so exercise the actual image or video request form.

Keep benchmark context attached to every number. In one Qwen3.6-27B-FP8 study on an H200, CPU preprocessing for 15 480p frames improved from 428 ms p90 to 81 ms; at 5 QPS, the optimized path stayed around 80 ms while stock processing reached 11,574 ms. In a Gemma study using six 480p frames and about 500 visual tokens, p95 TTFT was around 120 ms at 20 QPS after batching, while median ITL stayed around 5 ms through 10 QPS. These are specific engineering workloads, not universal SLAs.

Overshoot runs inference in the cloud rather than on an edge runtime. Network path is therefore part of the product measurement. Overshoot serves two regions, so test from the application deployment and user geography. If an application requires fully offline execution, cloud-hosted inference does not satisfy that requirement.

Production guardrails and observability

Version prompts and parsers together. Log the model id, visual anchor, timing, parser result, and action outcome for each decision, subject to media and data retention policy. Monitor answer distribution as well as request errors. A sudden shift toward one label can indicate camera movement, scene drift, prompt regression, or model behavior change.

Design fallbacks before deployment. A timeout might produce no action, a cached last-known state with an age marker, or a human review request. An unavailable model might route to an evaluated alternative discovered through /models. Never silently reinterpret malformed output. Visible failure is easier to operate than an untracked guess.

Summary

• VLMs provide flexible visual question answering and semantic interpretation without task-specific training for every prompt.
• They cannot guarantee grounding, event coverage, calibrated confidence, or deterministic output.
• Choose image or video context from the evidence required, then control resolution, interval length, and sampling.
• Evaluate exact prompts, models, media policies, and parsers on representative footage under production load.
• Keep strict validation, fallbacks, observability, and safety controls around the model response.

Evaluate models on your own media: review the Overshoot model documentation, then use the 2026 VLM survey to form a testable shortlist.