Embedded computer vision through a cloud VLM API

Define the device and cloud boundary

The embedded side captures video, produces a LiveKit-compatible track, manages connectivity, renews the Stream, and schedules requests. Overshoot receives the WebRTC publication, retains recent frames, and runs cloud-hosted VLM inference. Your application parses responses and controls local behavior. This boundary should be explicit in a system diagram and in failure handling.

No specific board, camera, accelerator, or operating system is certified by the public contract. Start by verifying that the target environment can run an appropriate LiveKit client or communicate with a nearby server publisher. Measure CPU, memory, bandwidth, reconnect behavior, and power on the actual product. Avoid assuming that a browser example proves suitability for an embedded deployment.

Create and maintain the Stream

POST /v1beta/streams returns a stream id, LiveKit URL, token, and 300-second lease. Connect and publish the video track. Schedule keepalive every 90 to 120 seconds and use the fresh token returned by the endpoint. Track successful frame arrival. A transport connection can remain open while the capture pipeline is stalled, so health needs both network and media signals.

An ended Stream cannot resume. After expiry or deletion, create a new Stream, republish, and replace every stored media reference. During normal shutdown, delete the Stream and release local capture resources. Bound reconnect attempts and expose a degraded state. Embedded systems often operate unattended, making explicit lifecycle telemetry essential for diagnosis.

Limit bandwidth and model work

Choose capture resolution and publication behavior based on measured task quality and network conditions. For current-state questions, reference one latest frame. For motion, select the shortest useful segment. Segment max_fps defaults to 1.0. More sampled frames increase visual processing and can lengthen requests. Query on an event or user action instead of running an unnecessary timer at high frequency.

Keep prompts short and cap output. A device status label with evidence needs fewer generated tokens than a broad description. Use a stable thread_id for related requests when prompt caching applies. Log request timing without storing sensitive visual content by default. Bandwidth, model input, output length, and cadence all contribute to the deployed operating profile.

Plan for regions and disconnections

Overshoot documents us-west1 and us-central1. Measure round trips from the networks where devices will operate. Cellular, shared wireless, and mobile uplinks can change rapidly. Test packet loss, handoff, suspended application state, clock changes, and long offline periods. Decide which product functions remain available when cloud inference is unreachable.

If the product must interpret video without connectivity, use an on-device runtime for that requirement. A hybrid system can keep a small local rule or detector for immediate behavior and ask the cloud VLM for richer context when connected. Keep safety and hard deadlines local. Do not represent cloud response as a deterministic control signal.

Secure device credentials and media

Do not bake a shared long-lived organization API key into distributable firmware or browser code. Put credential exchange behind an application-controlled service and limit what reaches the device. The short-lived LiveKit publish token is designed for joining the specific room. Protect logs, device identity, Stream mappings, and any durable visual evidence according to the product threat model.

Use TLS endpoints as documented and rotate compromised credentials. Define who can create Streams and issue inference requests. Avoid sending extra visual area when a crop or camera position can reduce exposure. Stream history lasts 600 seconds, while any application copy follows your own retention. Security review should cover publisher authentication, API authorization, physical device access, and update behavior.

Validate on real devices and networks

Build an evaluation set from production optics, motion, mounting, lighting, and compression. Measure task correctness, unknown rate, frame freshness, reconnect time, bandwidth, battery or power impact, and capture-to-answer latency. Overshoot-hosted models respond in 200ms. The complete embedded path includes capture, uplink, cloud processing, SSE return, parsing, and local action.

Run long sessions to exercise keepalive and recovery. Test an unavailable model from /models, expired Stream, stale camera, malformed output, and interrupted SSE response. Start with an assistive feature whose failure is visible and reversible. This evidence determines whether managed VLM inference fits the embedded product more reliably than assumptions about a named device platform.

References

• LiveKit client flow
• Stream lifecycle
• Edge inference versus cloud inference