Model Zoos: A Dataset of Diverse Populations of Neural Network Models

Item Type	Conference or Workshop Item (Paper)
Abstract	In the last years, neural networks (NN) have evolved from laboratory environments to the state-of-the-art for many real-world problems. It was shown that NN models (i.e., their weights and biases) evolve on unique trajectories in weight space during training. Following, a population of such neural network models (referred to as model zoo) would form structures in weight space. We think that the geometry, curvature and smoothness of these structures contain information about the state of training and can reveal latent properties of individual models. With such model zoos, one could investigate novel approaches for (i) model analysis, (ii) discover unknown learning dynamics, (iii) learn rich representations of such populations, or (iv) exploit the model zoos for generative modelling of NN weights and biases. Unfortunately, the lack of standardized model zoos and available benchmarks significantly increases the friction for further research about populations of NNs. With this work, we publish a novel dataset of model zoos containing systematically generated and diverse populations of NN models for further research. In total the proposed model zoo dataset is based on eight image datasets, consists of 27 model zoos trained with varying hyperparameter combinations and includes 50’360 unique NN models as well as their sparsified twins, resulting in over 3’844’360 collected model states. Additionally, to the model zoo data we provide an in-depth analysis of the zoos and provide benchmarks for multiple downstream tasks. The dataset can be found at www.modelzoos.cc.
Authors	Schürholt, Konstantin; Taskiran, Diyar; Knyazev, Boris; Giro-i-Nieto, Xavier & Borth, Damian
Research Team	AIML Lab
Language	English
Subjects	computer science
HSG Classification	contribution to scientific community
Date	November 2022
Publisher	36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks.
Event Title	36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks.
Depositing User	Konstantin Schürholt
Date Deposited	08 Nov 2022 08:22
Last Modified	28 Feb 2023 12:41
URI:	https://www.alexandria.unisg.ch/publications/267857

Download

Text
model_zoos_dataset_camera_ready.pdf - Published Version
Available under License Creative Commons Attribution No Derivatives.
Download (674kB)

Citation

Schürholt, Konstantin; Taskiran, Diyar; Knyazev, Boris; Giro-i-Nieto, Xavier & Borth, Damian: Model Zoos: A Dataset of Diverse Populations of Neural Network Models. 2022. - 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks..

Statistics

https://www.alexandria.unisg.ch/id/eprint/267857

Edit item