A Model Zoo on Phase Transitions in Neural Networks
Journal
Journal of Data-centric Machine Learning Research
Type
journal article
Date Issued
2025-09-29
Author(s)
Abstract
Using the weights of trained Neural Network (NN) models as data modality has recently gained traction as a research field-dubbed Weight Space Learning (WSL). Multiple recent works propose WSL methods to analyze models, evaluate methods, or synthesize weights. Weight space learning methods require populations of trained models as datasets for development and evaluation. However, existing collections of models-called 'model zoos'-are unstructured or follow a rudimentary definition of diversity. In parallel, work rooted in statistical physics has identified phases and phase transitions in NN models. Models are homogeneous within the same phase but qualitatively differ from one phase to another. We combine the idea of 'model zoos' with phase information to create a controlled notion of diversity in populations. We introduce 12 large-scale zoos that systematically cover known phases and vary over model architecture, size, and datasets. These datasets cover different modalities, such as computer vision, natural language processing, and scientific ML. For every model, we compute loss landscape metrics and validate full coverage of the phases. With this dataset, we provide the community with a resource with a wide range of potential applications for WSL and beyond. Evidence suggests the loss landscape phase plays a role in applications such as model training, analysis, or sparsification. We demonstrate this in an exploratory study of the downstream methods like transfer learning or model weights averaging.
Funding(s)
SNF Grant 10001118
Language
English
Keywords
Model Zoo
Weight Space Learning
Neural Networks
Phase Transition
HSG Classification
contribution to scientific community
Refereed
Yes
Volume
2
Official URL
Subject(s)
Division(s)
File(s)![Thumbnail Image]()
open.access
Name
105_A_Model_Zoo_on_Phase_Trans.pdf
Size
7.86 MB
Format
Adobe PDF
Checksum (MD5)
da7ad0626f87e52b26cb81eed05771da