Efficient Video Sampling: Reducing Redundancy in Video VLMs

The Problem With Video

Before a vision-language model can reason about a video, the video first has to be converted into a representation that the model can process.

Typically, this means sampling frames from the video, passing those frames through a vision encoder, and converting the resulting representations into visual tokens that are eventually passed to the language model.

However, how efficient is this tokenization process for all frames when videos inherently contain a significant amount of repeated information?

Given consecutive frames in a video, much of the meaning and importance in a scene might be shared and even redundant between back to back stills. For instance, small movements or background consistency might render two or more frames in a row both visually and semantically the same.

Moreover, as frames are added, this redundancy can quickly increase the number of visual tokens passed to the language model. Additional visual tokens therefore mean more computation during prefill, longer time to first token, and ultimately higher inference costs.

Despite this, a VLM may still process visual tokens from these frames.

So how can we reduce the amount of repeated visual information a VLM processes without removing the information that actually changes over time?

Comparing Entire Frames

A natural solution to this problem is to look at whole neighboring frames to determine whether they are visually similar.

Suppose a vision encoder divides every video frame into visual patches and produces an embedding for each patch.

A simple approach would be to average all of these patch embeddings into one global representation of the frame.

Given two frames (A) and (B), we could then compare these averaged representations using cosine similarity:

If the cosine similarity is high, the two frames are likely visually similar.

At the same time, though, averaging an entire frame introduces another problem.

Imagine a video where almost everything stays the same, but a person raises their hand.

Most of the patch embeddings may remain extremely similar. When all of these embeddings are averaged together, the small region containing the hand movement can be outweighed by the much larger part of the frame that did not change.

As a result, the two frame-level representations may still have a very high cosine similarity even though something important happened.

If we compress the whole frame into a single representation, we may lose important information about where a change actually occurred.

Comparing Corresponding Patches

Instead of comparing the entire frame at once, what if we compared the same regions across time?

Conveniently, Vision Transformers divide visual inputs into a grid of patch embeddings.

Given two neighboring frames, we can therefore compare a patch in one frame with the patch at the corresponding spatial location in the next frame.

For patch (i), this can be written as:

where (pᵢᵗ) represents patch (i) at time (t), and (pᵢᵗ⁺¹) represents the corresponding patch in the next frame.

If the cosine similarity is high, that region likely changed very little.

If the cosine similarity is lower, that region contains more visual change.

Rather than asking:

Did the entire frame change?

we can instead ask:

Which parts of the frame changed?

Approaching sampling from this perspective allows a static background to be treated differently from a smaller region containing motion or some new visual event.

Efficient Video Sampling

Efficient Video Sampling, or EVS (Bagrov et al., 2025), uses this idea to reduce redundant visual information before it reaches the language model.

At a high level, EVS compares corresponding visual patches across neighboring video frames and uses their cosine similarity to estimate which information is redundant.

Patches that remain highly similar over time are likely to contain relatively little new information. In comparison, patches that change more significantly may contain new visual information that is important for understanding the video.

Using these similarity scores, EVS can reduce the number of visual tokens passed to the language model.

This is different from simply comparing two full frames and deciding whether one of them should be removed.

A single frame may contain both highly redundant regions and a small region where something important changes.

By comparing patches instead of collapsing the entire frame into one representation, EVS can make a more localized decision about where redundancy actually exists.

EVS on Qwen3-VL

We tested EVS on Qwen/Qwen3-VL-32B-Instruct using vLLM with a video pruning rate of 0.5.

For latency, we tested on VideoMME with streaming time to first token and a fixed output length of 64 tokens.

At a 0.5 pruning rate, we observed the following results:

The largest change was in the number of prompt tokens.

EVS reduced the average prompt size from 10,101 tokens to 5,152 tokens, corresponding to approximately a 2× reduction.

This also translated into lower latency.

Time to first token decreased from 3,336 ms to 2,695 ms, corresponding to roughly a 19% reduction in TTFT, or a 1.24× speedup. End-to-end latency decreased from 4,582 ms to 3,920 ms, corresponding to approximately a 14% reduction, or a 1.17× speedup.

Interestingly, while we implemented a 50% pruning rate, we actually did not observe a decrease in accuracy on this evaluation.

VideoMME accuracy increased from 62.5% to 66.7%, corresponding to a 4.2 percentage point improvement.

Less Is More

Based on these results, EVS demonstrates that pruning can remove redundant visual information without necessarily hurting model performance. As we saw in our results, moderate video pruning did not lead to performance degradation and instead corresponded with higher VideoMME accuracy on this evaluation.

The key takeaway is that efficient sampling not only reduces redundancy in video processing for VLMs, but can also help filter out unnecessary information without an observed loss in baseline model performance.