Are All Parameters Made Equal?
Joseph Scharpf
The Cost of Capacity
Modern neural networks are large, and that size is part of what makes them powerful. The parameters in a neural network, the weights and biases, enable the network to learn relationships, associations, and representations. Additional parameters can give a model more capacity to learn expressive patterns. Yet, that extra capacity comes with tradeoffs: higher memory usage, greater compute requirements, increased latency, higher energy consumption, and greater cost. A neural network that performs well in a research setting may be difficult to use in production because of these drawbacks.
Naturally, this raises the question:
If neural networks are large, and have millions or billions of parameters, are all of these parameters equally necessary?
The answer, in many cases, is no.
Why Efficient Models Matter
A more efficient model has many benefits. It may be faster, cheaper, or have lower latency. These properties are essential for many applications. Large language models and vision-language models are expensive to run at scale. Making them more efficient can make them more widely applicable. For example, a faster model can enable real-time applications. A lower-latency model can be used in robotics or embedded systems. A more portable model can run on a phone or in a browser. A cheaper model can serve more requests or be more accessible to the public.
Large neural networks are expensive not just in terms of arithmetic operations, but also in terms of memory access. Research into efficient neural networks has demonstrated that memory access can dominate energy costs, particularly when weights need to be accessed from off-chip memory. This limits the applicability of large neural networks to mobile or embedded systems [Han et al., 2015].
Given this, can we remove unnecessary parameters while retaining desirable properties of the model?
This is the problem of pruning.
Weight Pruning: Removing Unnecessary Connections
A common form of pruning is weight pruning, or connection pruning. The motivation behind this approach is simple: after training, many weights in the network may be close to zero. These weights are often good candidates for removal because they may have little effect on the model's output. A standard approach to weight pruning is as follows:
- Train a dense neural network.
- Identify low-importance weights.
- Set low-importance weights to zero.
- Retrain the network with masked weights.
- Repeat until further pruning causes unacceptable performance degradation
Early approaches to learning both weights and connections demonstrated that CNNs can contain substantial redundancy. By first training a dense network, then pruning unimportant connections and retraining the sparse network, AlexNet was reduced from 61 million parameters to 6.7 million, about a 9x reduction, with no loss in accuracy. Similar improvements were seen in VGG-16, with a 13x reduction in parameters from 138 million to 10.3 million, also with no loss in accuracy [Han et al., 2015].
This demonstrates that model size is not strictly indicative of model performance. Large networks like AlexNet and VGG-16 contained many connections that could be pruned without degrading performance.
Thus, neural networks often contain substantial redundancy that can be removed by pruning.
Weight pruning, however, has a major limitation: it introduces irregular sparsity. For example, a layer may have the same input and output dimensions, but many individual weights are set to zero. This can reduce memory requirements, but it does not always lead to substantial improvements in speed, since most deep learning hardware is optimized for dense matrix multiplication. This motivates structured pruning.
Structured Pruning: Removing Bigger Pieces
Structured pruning removes bigger pieces from the network. Instead of removing individual weights, structured pruning can remove channels, neurons, attention heads, or entire layers.
This is an important distinction, because structured pruning can lead to more practical improvements in speed. For example, a CNN with fewer channels requires less memory and less computation. A Transformer with fewer attention heads can also require less memory and computation. In many cases, these changes can be directly applied to standard deep learning hardware, since the computation is no longer irregular.
OSSCAR is one such approach to structured pruning. It focuses on one-shot structured channel and neuron pruning, where specific structures are removed from the network in one pruning step. This approach focuses on removing structures such as channels from convolutional layers, neurons from dense layers, and attention heads from multi-head attention layers. The motivation for structured pruning is that the speedups from such pruning can be directly applied to standard deep learning hardware, unlike irregular sparsity patterns from weight pruning [Meng et al., 2024].
The question then becomes: which structures can be removed?
OSSCAR approaches this problem from the perspective of layer-wise reconstruction. For each layer, the pruned layer should stay as close as possible to the original layer. In other words,
original layer output ≈ pruned layer output
This is related to the intuition behind pruning, but it is more nuanced than simple magnitude-based pruning. If we remove some heads from a multi-head attention layer, the pruned layer should still behave roughly like the original layer.
A particular structure may have small weights, but still be needed for reconstruction. Another structure may have large weights, but be redundant with other structures. Structured pruning is about removing structures that are not needed for preserving the behavior of the layer.
OSSCAR formalizes this as an optimization problem, where the goal is to find structures to prune while minimizing reconstruction error. The search space for structures to prune can be very large, so OSSCAR uses a local combinatorial search strategy to find good structures to prune [Meng et al., 2024].
As models become more complex, pruning individual weights is not always enough. To achieve practical efficiency gains, it can be more useful to prune meaningful structures.
Why Vision-Language Models Are Different
In this context, a vision-language model is different from a standard vision model or a standard language model. A vision-language model must encode both vision and language, and align the two representations. In particular, CLIP-style models have a vision encoder and a text encoder:
vision encoder → image embedding
text encoder → text embedding
The goal of CLIP is to bring together related images and text. Images of dogs should be close to text about dogs, and images of cars should be close to text about cars. This means that vision-language models require special consideration when it comes to pruning.
In particular, a vision-language model has two encoders that must be aligned. A module may be unimportant for the vision encoder alone, or the text encoder alone, but important for keeping the two encoders aligned. Similarly, a module may appear less important for one encoder in isolation, but still matter for keeping representations of similar concepts close together.
Pruning for Cross-Modal Alignment
For vision-language models, the definition of importance changes. A module is not only important if it is useful for the vision encoder or the text encoder. A module is also important if it helps keep the vision encoder and text encoder aligned.
MoPE-CLIP is one such approach to pruning vision-language models. It evaluates the importance of each module based on how much performance degrades when the module is removed. Specifically:
- temporarily remove a module
- evaluate how much performance degrades
If removing a module leads to substantial performance degradation, the module is important and should be kept. If removing a module has little effect on performance, the module can be pruned. Formally, MoPE-CLIP measures the importance of a module as the difference between the performance of the full CLIP model and the performance of the pruned CLIP model:
MoPE = performance(full) - performance(pruned)
Higher MoPE values indicate that a module is more sensitive to pruning and should be kept [Lin et al., 2024].
MoPE-CLIP can be applied to many types of modules, including attention heads, FFN neuron groups, and Transformer layers. This enables both width pruning and depth pruning. Width pruning refers to pruning inside a layer, such as attention heads or FFN neurons. Depth pruning refers to pruning entire layers. MoPE-CLIP evaluates the effect of pruning attention heads and FFN neurons for width pruning, and the effect of pruning Transformer layers for depth pruning [Lin et al., 2024].
This is different from OSSCAR, which focuses on structured pruning by evaluating how well a pruned layer can reconstruct the original layer:
original layer output ≈ pruned layer output
MoPE-CLIP focuses on a different signal: how much performance degrades when a module is removed.
Another difference is that MoPE-CLIP uses knowledge distillation to further improve performance after pruning. The original CLIP model acts as a teacher, and the pruned CLIP model acts as a student. The student model is distilled from the teacher model by matching image-text similarities, final features, and intermediate hidden states [Lin et al., 2024].
This is important because pruning reduces the model's capacity. By distilling the pruned model from the original model, the pruned model can recover some of the performance lost during pruning.
Ultimately, for vision-language models, pruning is not only about model compression, but also about alignment preservation.
What Pruning Teaches Us
Neural networks are large, but they are not always using all their parameters equally. Pruning can reveal substantial redundancy in neural networks, and this redundancy can often be removed without affecting performance.
However, pruning has become more sophisticated as models have evolved. For CNNs, early pruning approaches demonstrated that neural networks could be made substantially smaller with no loss in performance. For modern architectures, structured pruning has become more important, since it can lead to more practical improvements in speed and memory. For vision-language models, pruning has to take into account the importance of modules for cross-modal alignment.
As neural networks grow in size and complexity, these insights will become even more important. Larger neural networks may enable more capabilities, but efficient neural networks will determine where these capabilities can actually be deployed.