Model Zoos: A Dataset of Diverse Populations of Neural Network Models

Item Type Conference or Workshop Item (Paper)
Abstract In the last years, neural networks (NN) have evolved from laboratory environments to the state-of-the-art for many real-world problems. It was shown that NN models (i.e., their weights and biases) evolve on unique trajectories in weight space during training. Following, a population of such neural network models (referred to as model zoo) would form structures in weight space. We think that the geometry, curvature and smoothness of these structures contain information about the state of training and can reveal latent properties of individual models. With such model zoos, one could investigate novel approaches for (i) model analysis, (ii) discover unknown learning dynamics, (iii) learn rich representations of such populations, or (iv) exploit the model zoos for generative modelling of NN weights and biases. Unfortunately, the lack of standardized model zoos and available benchmarks significantly increases the friction for further research about populations of NNs. With this work, we publish a novel dataset of model zoos containing systematically generated and diverse populations of NN models for further research. In total the proposed model zoo dataset is based on eight image datasets, consists of 27 model zoos trained with varying hyperparameter combinations and includes 50’360 unique NN models as well as their sparsified twins, resulting in over 3’844’360 collected model states. Additionally, to the model zoo data we provide an in-depth analysis of the zoos and provide benchmarks for multiple downstream tasks. The dataset can be found at www.modelzoos.cc.
Authors Schürholt, Konstantin; Taskiran, Diyar; Knyazev, Boris; Giro-i-Nieto, Xavier & Borth, Damian
Research Team AIML Lab
Language English
Subjects computer science
HSG Classification contribution to scientific community
Date November 2022
Publisher 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks.
Event Title 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks.
Depositing User Konstantin Schürholt
Date Deposited 08 Nov 2022 08:22
Last Modified 08 Nov 2022 08:22
URI: https://www.alexandria.unisg.ch/publications/267857

Download

[img] Text
model_zoos_dataset_camera_ready.pdf - Published Version
Available under License Creative Commons Attribution No Derivatives.

Download (674kB)

Citation

Schürholt, Konstantin; Taskiran, Diyar; Knyazev, Boris; Giro-i-Nieto, Xavier & Borth, Damian: Model Zoos: A Dataset of Diverse Populations of Neural Network Models. 2022. - 36th Conference on Neural Information Processing Systems (NeurIPS 2022) Track on Datasets and Benchmarks..

Statistics

https://www.alexandria.unisg.ch/id/eprint/267857
Edit item Edit item
Feedback?