Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights
Type
conference paper
Date Issued
2022-11
Author(s)
Research Team
AIML Lab
Abstract
Learning representations of neural network weights given a model zoo is an emerg- ing and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation. Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation, which captures intrinsic and extrinsic properties of the models in the zoo. In this work, we ex- tend hyper-representations for generative use to sample new model weights. We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and several sampling methods based on the topology of hyper-representations. The models generated using our methods are diverse, per- formant and capable to outperform strong baselines as evaluated on several down- stream tasks: initialization, ensemble sampling and transfer learning. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.
Language
English
HSG Classification
contribution to scientific community
HSG Profile Area
None
Book title
Advances in Neural Information Processing Systems
Publisher
Curran Associates, Inc.
Volume
35
Event Title
Conference on Neural Information Processing Systems
Event Location
New Orleans
Subject(s)
Division(s)
Eprints ID
267695
File(s)![Thumbnail Image]()
Name
HyperRepresentations_as_Generative_Models-camera_ready.pdf
Size
4.6 MB
Format
Adobe PDF
Checksum (MD5)
3da6f3545b87a083fa7fb5f17f3b393f