Edge Inference vs. Cloud Inference for Computer Vision: Which Should You Use?

Edge inference and cloud inference are two deployment patterns for running computer vision models near the image source or in a remote managed service. The right boundary depends on the deadline, network, data policy, model size, fleet burden, and failure mode of one specific workload. Overshoot is a cloud VLM service. It receives live video over WebRTC through LiveKit and returns answers in 200ms. It does not ship an edge or offline runtime.

What edge inference means

Edge inference runs model execution on hardware at or near the camera. The edge may be an embedded accelerator, industrial computer, local server, mobile device, or on-premises GPU cluster. Frames can remain within the site, and the inference path can continue when the wide-area network fails. Local execution also permits tight coupling to a real-time application when the hardware, operating system, and model stack can satisfy its deadline.

That control creates operational work. The team selects hardware, converts and packages models, manages drivers, monitors thermals and memory, rolls out updates, and handles hardware-specific failures across a fleet. Performance is limited by the accelerator and power envelope. Large VLMs may need quantization, reduced input resolution, fewer frames, or a smaller model to fit.

What cloud inference means

Cloud inference sends visual input or a live video stream to remotely operated model servers. The device can stay small while the service runs larger accelerators and centralizes model serving. A managed API simplifies capacity changes and model access, but the request depends on network connectivity and the service region. End-to-end latency includes capture, encoding, transport, queuing, preprocessing, inference, token streaming, and application handling.

Overshoot accepts HTTPS or data media in chat requests and provides a Live Stream path for repeated queries against current and recent video. The live publisher may run in a browser, native application, or server. Streams retain 600 seconds of recent history, use a 300-second renewable lease, and expose visual context through ovs:// references. Responses stream with SSE.

Latency is a system measurement

A local accelerator does not guarantee a lower application latency, and a cloud endpoint does not guarantee that the network dominates. A small edge model may execute quickly but spend time decoding, resizing, and copying frames. A cloud VLM may return in 200ms while a camera gateway adds buffering before upload. Measure from the physical event or frame timestamp to the first usable application output.

Separate latency into stages so optimization has a target. Record camera exposure and capture delay, encode queue, uplink, server time to first token, generation time, validation, and final action. Use percentiles under sustained load. A median hides queue growth, packet loss, cold paths, and occasional large visual inputs. Test the same frame count, resolution, prompt, and output length for both architectures.

Measure time to first SSE token from application code

import json
import time
import requests

started = time.perf_counter()
with requests.post(
    "https://api.overshoot.ai/v1beta/chat/completions",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "model": MODEL_ID,
        "stream": True,
        "messages": [{"role": "user", "content": [
            {"type": "text", "text": "Return one visible state label."},
            {"type": "image_url", "image_url": {
                "url": f"ovs://streams/{STREAM_ID}?frame_index=-1"
            }},
        ]}],
    },
    stream=True,
    timeout=10,
) as response:
    response.raise_for_status()
    for line in response.iter_lines():
        if line.startswith(b"data: ") and line != b"data: [DONE]":
            chunk = json.loads(line[6:])
            print(f"first token: {(time.perf_counter() - started) * 1000:.1f} ms")
            break

Choose edge inference for local guarantees

Choose an edge path when the requirement explicitly says the system must operate without wide-area connectivity, pixels cannot leave the site, or a local deadline cannot include network variance. Edge also fits high-rate fixed tasks such as counting, tracking, presence detection, and geometric inspection when a compact model has already met the accuracy requirement. Local execution can reduce upstream bandwidth when every frame would otherwise leave the facility.

Treat these benefits as design properties that must be verified. Offline behavior still depends on local power, storage, camera connectivity, and software health. Data residency still depends on logs, update channels, and support tooling. A deterministic control deadline still needs a suitable scheduler and control architecture. Overshoot does not provide local detectors, Jetson certification, or an installable edge runtime, so those components come from another stack.

Choose cloud inference for managed VLM capacity

Choose cloud VLM inference when the task needs language-driven visual reasoning, the instruction changes often, or the available edge device cannot serve the chosen model. Cloud deployment is also useful when model operations should stay centralized and the source devices should remain thin publishers. Teams can evaluate multiple hosted model ids through one OpenAI-compatible request shape without packaging each model for each hardware target.

The workload must tolerate connectivity to us-west1 or us-central1 and the applicable data handling policy. Test both routes from the actual deployment network. Design explicit behavior for timeouts, rate limits, publisher disconnects, and ended Streams. Cloud VLM output remains probabilistic, so safety-rated actions and hard real-time loops belong in deterministic systems.

Use a hybrid architecture when tasks have different deadlines

Many camera systems contain several workloads. A local rule or detector can screen every frame, maintain a safety boundary, or identify an event. The application can then query a cloud VLM for description, flexible classification, or recent context. This arrangement keeps the fixed high-rate path local and reserves cloud requests for events that need broader reasoning.

The local component is external to Overshoot. Overshoot does not install or operate that detector. The event bridge is also application code: there are no public webhooks, PLC adapters, SCADA connectors, or MES connectors. A trigger can call the chat-completions API and reference a recent Stream segment. With a 600-second history, the application can include frames from before the trigger as long as the Stream was already publishing.

Ask about the five seconds around a local event

{
  "model": "google/gemma-4-26B-A4B-it",
  "stream": true,
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "Summarize the visible change and return unclear when occluded."},
      {"type": "video_url", "video_url": {
        "url": "ovs://streams/STREAM_ID?start_offset_ms=-5000&end_offset_ms=0&max_fps=1.0"
      }}
    ]
  }]
}

Compare privacy and security boundaries

Start with a data-flow diagram rather than a generic claim that one deployment is private. List raw frames, sampled frames, prompts, outputs, credentials, logs, support access, and retained artifacts. Edge inference can keep visual data on site, but remote logs or fleet tooling may still export derived data. Cloud inference transmits selected media to a provider and therefore requires review against organizational policy and applicable regulation.

Minimize data at the source. Crop only when the crop preserves task evidence, choose the shortest useful video interval, and avoid unnecessary frame rates. The default Overshoot segment sampling cap is 1.0 FPS. Store API keys in a server or secured publisher environment, rotate them, and restrict which application component can issue requests. An ovs:// reference is a service-side media address and cannot be fetched as a public URL.

Compare cost and operational ownership

Edge cost includes devices, spares, installation, power, cooling, field service, model conversion, testing, monitoring, and fleet updates. Cloud cost includes ingestion, inference usage, network connectivity, and the application services around the API. Compare costs over the planned lifetime and expected request rate. A purchased accelerator is not free after installation, and a low request volume may not justify dedicated hardware.

Sampling policy strongly affects both paths. More frames raise decode and preprocessing work, visual token count, and possibly model latency. Resolution matters as well. For Qwen workloads, useful visual-token estimates are about 200 tokens per 480p frame, 450 per 720p frame, and 1,010 per 1080p frame. These are estimates for planning, not universal counts across every model processor.

Design for connectivity and failure

Write the failure policy before selecting a deployment. If a network outage must never interrupt the task, use a local path for that task. If manual review is acceptable, a cloud workflow can disable itself visibly and queue metadata until service returns. Avoid silently treating a timeout as a negative detection. Timeout, unavailable, unclear, and observed-negative are different states.

For an Overshoot Live Stream, renew the 300-second lease well before expiry and update LiveKit with the fresh publish token returned by keepalive. Handle a failed keepalive and an ended Stream by creating a new Stream. Monitor frame freshness alongside connection state. A connected publisher can still send frozen, black, delayed, or incorrectly oriented video.

Run a fair deployment comparison

1. Define one task, input distribution, output schema, and acceptance threshold.
2. Use the same camera footage and equivalent sampling for edge and cloud candidates.
3. Measure end-to-end p50, p95, and p99 latency under representative concurrency.
4. Score false positives, false negatives, abstentions, and malformed output.
5. Test loss of internet, local power interruption, camera restart, and service errors.
6. Estimate three-year hardware, cloud, network, and engineering costs.
7. Document who owns updates, incidents, credentials, and end-of-life replacement.

Do not compare a small local detector and a large VLM as if they implement the same function. Compare complete candidate systems against the same operational requirement. If their outputs differ, evaluate whether each output is sufficient for the decision. A compact label can be better than a rich description when the action only accepts a fixed state.

Decision checklist

• Use edge inference when offline operation, on-site pixels, or a strict local deadline is mandatory.
• Use cloud VLM inference when managed larger models and flexible language instructions justify connectivity.
• Use a hybrid design when continuous fixed screening and occasional contextual reasoning have different requirements.
• Measure event-to-output latency and tail behavior instead of relying on model execution time.
• Assign ownership for hardware, publishers, model versions, failure policy, and downstream actions.

Summary

• Edge and cloud are deployment boundaries with different failure modes and operating responsibilities.
• Overshoot provides cloud VLM inference in 200ms and has no edge or offline runtime.
• Live Overshoot ingest uses WebRTC through LiveKit; direct RTSP, RTMP, and ONVIF ingest are absent.
• A hybrid system can trigger cloud reasoning from a separately operated local detector.
• Choose with measured latency, task quality, outage tests, policy review, and lifecycle cost.

Test the cloud side of the boundary: use the Overshoot quickstart to publish a Stream, then review the Gemma 4 serving case study before setting a production latency budget.