The Impact of Rarely-firing Nodes in Neural Networks on Representational Geometry and Predictions of Human Similarity Judgments

Vision deep neural networks (DNNs) learn many sparse features that are activated rarely or not at all. It is however unknown how these features structure the representational geometry of a DNN's embedding space. Previous research used supervised iterative-magnitude pruning (``lottery-ticket") to remove less-important features, and concluded that even minor feature-pruning strongly alters layer representations. Here we investigate DNN's representational geometry, but using an unsupervised approach, where pruning is guided by how frequently a node is inactivated across samples. Using representational similarity analysis, we find that for CIFAR-10 and MNIST, 20% of the features can be removed without any impact on the representational space. However, these redundant features do contain distributed information: when used alone, they account for 10%-50% of the variance in the non-pruned embeddings. Additionally, we find that for some natural image-sets, the removal of sparse features improves the prediction of human similarity judgments. Finally, we show that for a given set of images belonging to an object category, never-activated features encode meaningful semantics that is irrelevant for representing the category. Overall, our findings contribute to the understanding of how sparse features shape objects' representations in DNNs and how they impact their effectiveness as a model of human behavior.