Manufacturing computer vision with live context

Define a bounded manufacturing question

Select a task with a visible answer and an existing review process. Examples include describing why a station appears blocked, checking whether the visible arrangement matches a work instruction, summarizing the final seconds before an exception, or sorting review items by visible condition. Avoid starting with complete line monitoring. A bounded question makes labeling, prompt design, and operator feedback manageable.

Document the user, source, trigger, expected labels, required evidence, and unknown state. Identify the business step after the answer. If the result creates a review ticket, specify what the reviewer sees and how they correct it. If no action follows, the inference may add complexity without improving the process.

Publish live video through LiveKit

POST /v1beta/streams returns the LiveKit room URL, publish token, stream id, and lease timing. A browser, native application, or server process publishes a video track. Other plant video transports require an application or gateway that produces the documented LiveKit WebRTC track. The public API makes no claim of direct camera, PLC, SCADA, or equipment-controller integration.

Renew the 300-second lease every 90 to 120 seconds and update the publisher with the fresh token. Observe frame timestamps and retained counts through GET /streams/{stream_id}. A room connection alone does not prove current imagery is available. When a Stream expires or is deleted, it enters ended state permanently. Recovery creates a new Stream and republishes.

Use recent history for exception review

A current frame can answer whether a visible condition exists now. A short video segment can explain how that state developed. Reference a trailing interval with start_offset_ms and omit the end to reach the live edge. max_fps defaults to 1.0. Choose a tighter interval or explicit frame set when the task does not need every sampled moment.

The Stream retains 600 seconds. This gives the application time to react to an event and inspect preceding context. It does not provide long-term recording. If a quality record requires source evidence, copy approved material into your own controlled storage before it leaves the window. Associate the anchor and prompt with the review result.

Combine VLMs with existing inspection

Keep fixed detectors, gauges, barcode readers, and process sensors where they already perform well. Their event can trigger a VLM request that explains surrounding visible context. The application can present both signals to an operator. This division reduces the number of language-model calls and avoids asking a VLM to recreate a precise measurement.

For a language-based check, request a small set of states and visible evidence. Validate JSON or enum output. Require unknown when the view is obscured, the relevant area is outside frame, or the instruction cannot be verified visually. Never translate model fluency into a confidence score without measured calibration on the actual task.

Test line variation and failure modes

Collect normal cycles, product changes, packaging changes, tool changes, cleaning, maintenance, shift lighting, glare, blur, occlusion, empty scenes, and rare failures. Split examples by time or production run to reduce leakage. Score the workflow outcome, including unknown handling and reviewer effort. Review false positives and false negatives separately because their operational costs differ.

Measure latency on the actual network and region. Overshoot-hosted models respond in 200ms. Record capture-to-first-useful-token, complete structured output, failures, frame count, resolution, prompt version, and offered load. The published engineering benchmarks describe specific model-serving workloads and should not be presented as manufacturing line throughput.

Run a production-minded pilot

Start with one station and a human-reviewed queue. Show the queried image or segment, answer, evidence, model id, and timestamp. Give reviewers a correction path. Monitor missing frames, ended Streams, unavailable models, parsing errors, latency percentiles, and task errors. Define who responds to each operational state.

Keep hard control and safety paths independent. Review regional data handling, access, and retention. Version prompts and repeat the evaluation after every camera or process change. A useful pilot produces a clear before-and-after workflow measurement without claiming an invented customer outcome. The goal is evidence from your own line and your own acceptance criteria.

References

• Stream lifecycle
• API best practices
• Industrial computer vision guide