Florence-2 on live video streams

What Florence-2 is good at

Florence-2 is a task-specific tool rather than a chat model: a single architecture handles captioning, object detection, phrase grounding, OCR, and segmentation, selected by the prompt format rather than by fine-tuning separate models for each task.

Its small size, 0.23B or 0.77B parameters depending on variant, means it runs cheaply and quickly, which matters for pipelines that need to run one of these vision tasks on every frame of a live stream rather than on occasional still images.

• Dense captioning and region description from a single prompt format
• Object detection, phrase grounding, and OCR without a separate model per task
• Segmentation mask generation on live-frame pipelines

Florence-2 and Overshoot's live video API

Florence-2 is not currently in Overshoot's live model catalog. The catalog changes over time, and GET /v1beta/models is the authoritative list of what is available, so check there before building around a specific model.

On Overshoot, live-video requests work by publishing a camera or screen-share track over WebRTC through LiveKit to open a Stream, then sending a chat-completions request whose image_url or video_url is an ovs:// reference: the latest frame, an exact timestamp, or a recent segment from the 600 seconds of retained history. A prompt-based model like Florence-2 would map naturally onto that shape: because it is not conversational, requests would use a fixed task prompt per call rather than accumulating a long chat history, with structured output streamed back over SSE the way every Overshoot model responds.

How it fits next to PaliGemma 2

Florence-2 and PaliGemma 2 28B occupy the same niche, task-specific vision tools rather than general chat models, but Florence-2 is far smaller and handles its task set out of the box through prompt selection, while PaliGemma 2 28B is meant to be fine-tuned onto one task at a time. Teams that need several vision tasks covered without a fine-tuning step tend to reach for Florence-2 first.

Frequently asked questions

Can Florence-2 analyze live video?

Florence-2 processes individual frames, so it can run its captioning, detection, and OCR tasks on frames sampled from live video. It is not currently in Overshoot's live model catalog, however; the catalog changes over time, so check GET /v1beta/models for the models that can be pointed at a live WebRTC stream today.

Is Florence-2 open source?

Yes. It ships under the MIT license, which permits unrestricted commercial use, modification, and self-hosting of the weights in either the 0.23B or 0.77B variant.

Is Florence-2 a chat model?

No. Florence-2 is a prompt-based, task-specific vision model, not a chat model. A request selects a task, such as captioning, detection, or OCR, and Florence-2 returns a structured result for that task rather than an open-ended conversational reply.

How fast is Florence-2?

Florence-2's small size, at 0.23B or 0.77B parameters, keeps inference fast and cheap. That efficiency is what makes it practical for pipelines that run a vision task on every frame of a stream rather than on occasional still images.

Related models

• PaliGemma 2 28B
• Phi-4 Multimodal
• Phi-3.5 Vision
• Gemma 3 27B