Konstantin Schürholt
Last Name
Schürholt
First name
Konstantin
Email
konstantin.schuerholt@unisg.ch
Phone
+41 71 224 34 54
9 results
Now showing 1 - 9 of 9
- Some of the metrics are blocked by yourconsent settings
Item type:Publication, A Model Zoo on Phase Transitions in Neural Networks(2025-09-29); ; - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Towards Scalable and Versatile Weight Space LearningLearning representations of well-trained neural network models holds the promise to provide an understanding of the inner workings of those models. However, previous work has either faced limitations when processing larger networks or was task-specific to either discriminative or generative tasks. This paper introduces the SANE approach to weight-space learning. SANE overcomes previous limitations by learning task-agnostic representations of neural networks that are scalable to larger models of varying architectures and that show capabilities beyond a single task. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights, thus allowing one to embed larger neural networks as a set of tokens into the learned representation space. SANE reveals global model information from layer-wise embeddings, and it can sequentially generate unseen neural network models, which was unattainable with previous hyper-representation learning methods. Extensive empirical evaluation demonstrates that SANE matches or exceeds state-of-the art performance on several weight representation learning benchmarks, particularly in initialization for new tasks and larger ResNet architectures.Type:conference paperVolume:Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria. - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Models(2023); With growing size of Neural Networks (NNs), model sparsification to reduce the computational cost and memory demand for model inference has become of vital interest for both research and production. While many sparsification methods have been proposed and successfully applied on individual models, to the best of our knowledge their behavior and robustness has not yet been studied on large populations of models. With this paper, we address that gap by applying two popular sparsification methods on populations of models (so called model zoos) to create sparsified versions of the original zoos. We investigate the performance of these two methods for each zoo, compare sparsification layer-wise, and analyse agreement between original and sparsified populations. We find both methods to be very robust with magnitude pruning able outperform variational dropout with the exception of high sparsification ratios above 80%. Further, we find sparsified models agree to a high degree with their original non-sparsified counterpart, and that the performance of original and sparsified model is highly correlated. Finally, all models of the model zoos and their sparsified model twins are publicly available: modelzoos.cc.Type:conference paperJournal:ICLR 2023 Workshop on Sparsity in Neural Networks - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Hyper-Representations for Pre-Training and Transfer Learning(2022); - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights(Curran Associates, Inc., 2022-11); - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Model Zoos: A Dataset of Diverse Populations of Neural Network Models(36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks., 2022-11); - Some of the metrics are blocked by yourconsent settings
Item type:Publication, Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction(Neural Information Processing Systems (NeurIPS), 2021-11-09); - Some of the metrics are blocked by yourconsent settings
Item type:Publication, The Impact of Model Zoo Size and Composition on Weight Space Learning(2025-04-14); ; Re-using trained neural network models is a common strategy to reduce training cost and transfer knowledge. Weight space learning - using the weights of trained models as data modality - is a promising new field to re-use populations of pre-trained models for future tasks. Approaches in this field have demonstrated high performance both on model analysis and weight generation tasks. However, until now their learning setup requires homogeneous model zoos where all models share the same exact architecture, limiting their capability to generalize beyond the population of models they saw during training. In this work, we remove this constraint and propose a modification to a common weight space learning method to accommodate training on heterogeneous populations of models. We further investigate the resulting impact of model diversity on generating unseen neural network model weights for zero-shot knowledge transfer. Our extensive experimental evaluation shows that including models with varying underlying image datasets has a high impact on performance and generalization, for both in- and out of-distribution settings. Code is available on github.com/HSG-AIML/MultiZoo-SANE.Type:conference contributionJournal:ICLR Workshop on Neural Network Weights as a New Data Modality - Some of the metrics are blocked by yourconsent settings
Item type:Publication, A Model Zoo of Vision Transformers(2025-04-14); ; ; The availability of large, structured populations of neural networks-called "model zoos"-has led to the development of a multitude of downstream tasks ranging from model analysis, to representation learning on model weights or generative modeling of neural network parameters. However, existing model zoos are limited in size and architecture and neglect the transformer, which is among the currently most successful neural network architectures. We address this gap by introducing the first model zoo of vision transformers (ViT). To better represent recent training approaches, we develop a new blueprint for model zoo generation that encompasses both pre-training and fine-tuning steps, and publish 250 unique models. They are carefully generated with a large span of generating factors, and their diversity is validated using a thorough choice of weight-space and behavioral metrics. To further motivate the utility of our proposed dataset, we suggest multiple possible applications grounded in both extensive exploratory experiments and a number of examples from the existing literature. By extending previous lines of similar work, our model zoo allows researchers to push their model populationbased methods from the small model regime to state-of-the-art architectures. We make our model zoo available at github.com/ModelZoos/ViTModelZoo.Type:conference contributionJournal:ICLR Workshop on Neural Network Weights as a New Data Modality