An Introduction To Token Pruning For VLMs

Why VLM Efficiency Matters

Vision-language models provide powerful insights by reasoning across language and visual inputs, including images and video. They can identify objects, read visible text, or answer questions about a scene. But this flexibility also comes with a cost. Large model sizes and long sequences of visual tokens often raise issues such as increased memory usage, processing demands, and inference latency.

Given this, can we make VLMs more efficient by removing irrelevant or redundant visual information while preserving the evidence needed to answer a question?

Do We Even Need All These Parameters?

A traditional approach to efficiency is parameter pruning. Parameter pruning looks at whether all the weights in a pretrained model are necessary. Research on parameter pruning has revealed a compelling insight: there is often substantial "redundancy" in a neural network. You can remove a significant number of weights while retaining most of the performance of the original model (Han et al., 2015). This suggests many models are likely to be bigger than they need to be. But what about input redundancy? Can this same principle be applied to the input image itself?

Do We Even Need All These Visual Tokens?

Modern vision-language models do not directly process an image or video as a single object. Instead, an image is typically divided into patches, with each patch representing a visual token generated by a vision encoder.

At a high level, they work like this: image → patches → visual tokens → VLM → answer

Once the image has been transformed into a token sequence, it can be combined with a text query to perform visual reasoning. Generally, every visual token has to be processed alongside the text. And most of the input representation for a VLM is typically vision tokens, not text tokens.

Given this large number of tokens, is every visual token necessary to answer a question?

Probably not.

This suggests that image tokens may be pruned, just as parameters can be pruned.

Why Visual Tokens Might Be Redundant

An image or video often contains a lot of visual information that is irrelevant to a given question. For example, an image may contain background scenery, repeated patterns, or visual details that do not help answer the question. Processing every visual token can be wasteful, especially since many vision-language models use attention mechanisms whose cost grows quadratically with sequence length. We can see this issue when asking different questions about the same image. For example, one question might ask, "What sport is being played?" while another asks, "How many players are there?" In the first example, the model may need to focus on the ball, players, and field. In the second example, it may need to focus more specifically on the players. PuMer, a token reduction framework, is designed around this idea.

PuMer proposes a method called text-informed image pruning, which helps remove image tokens that are less relevant to the text query while preserving more useful ones (Cao et al., 2023). To do this, PuMer does not discard visual information randomly. At selected cross-modal layers, it uses text-to-image cross-attention scores to estimate how relevant each visual token is to the text. It keeps higher-scoring tokens and prunes lower-scoring ones. This reduces the number of tokens processed by subsequent layers and retains the visual information most relevant to the prompt.

PuMer also uses token merging. This is useful because not every redundant token should be deleted. Some tokens may contain overlapping information, such as neighboring patches that represent the same object, texture, or region. Rather than keeping all of these tokens separately, PuMer uses the dot-product similarity between their key vectors to identify and merge similar tokens separately within the image and text modalities.

Under this framework, PuMer achieves substantial improvement in efficiency, reporting 1.7x to 2.1x higher throughput and a 38% to 50% lower memory footprint across evaluated vision-language models and benchmarks. The performance drop? Less than 1% accuracy degradation. This demonstrates that token importance is not inherently fixed, and that the visual information that matters depends on the text query.

Do Visual Tokens Need Every Layer?

PuMer asks which visual tokens are relevant to the prompt. However, even if visual tokens are useful early in the model, do they need to be processed through every layer? Researchers have observed that visual tokens often receive much less attention in deeper layers of VLMs, which means many of them may no longer need to be processed through the rest of the model (Chen et al., 2024). This is the intuition behind FastV. Rather than discarding visual tokens before they have been processed by the model, FastV prunes the stream of visual tokens at a certain depth within the LLM, removing lower-scoring ones from later LLM layers.

FastV uses the average attention each visual token receives from other tokens as an importance score, then discards the lowest-scoring visual tokens from subsequent processing. Early layers process all visual tokens, while later layers only process the highest-scoring ones. The practical implications are significant.

First, by discarding tokens, the model reduces the number of tokens that later attention layers need to process. Second, because each remaining token also passes through feed-forward layers, removing tokens reduces computation beyond attention alone. The results demonstrate the effectiveness of this method. FastV reports a 45% reduction in FLOPs for LLaVA-1.5-13B while maintaining performance across various image and video understanding tasks. It also reports that, on A-OKVQA, LLaVA-13B with FastV achieves lower latency than LLaVA-7B while outperforming it. More importantly, token pruning enables a different tradeoff between model size and inference cost. In this case, a larger model may still have more parameters, but if it can process fewer visual tokens during inference, it can become more practical to deploy.

Efficiency Versus Evidence

Although token pruning has its benefits, it is also vital to understand which tokens are safe to remove. This is especially important for fine-grained tasks such as OCR, counting, spatial reasoning, or robotics. If pruning removes tokens containing relevant details, the model may no longer have enough visual context. Pruning can improve throughput and memory, yet it can’t be applied indiscriminately. It is helpful to think of pruning as a knob that can be turned up or down based on the situation. For a more general question, you may be able to prune aggressively. But for a precise question involving thorough visual analysis, you will want to reduce the level of pruning to avoid removing critical information.

The Bigger Lesson: Not All Tokens Matter Equally

Token pruning is an important step toward more efficient vision-language models. Earlier methods focused on parameter pruning, which asked whether all the model's parameters are needed. By contrast, token pruning focuses on the input to the model: are all the visual tokens necessary for this particular input and query? That is a critically important difference. Efficiency is no longer only a function of model design. A model can become more efficient by pruning unnecessary parameters, layers, or components. But VLMs can also become more efficient by reducing redundancy in the input representation itself. The larger lesson connects to visual grounding. The best VLMs will not merely process more visual information at higher resolution, but also identify the most valuable information for the question being asked.