Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights

Item Type Conference or Workshop Item (Paper)
Abstract Learning representations of neural network weights given a model zoo is an emerg- ing and challenging area with many potential applications from model inspection, to neural architecture search or knowledge distillation. Recently, an autoencoder trained on a model zoo was able to learn a hyper-representation, which captures intrinsic and extrinsic properties of the models in the zoo. In this work, we ex- tend hyper-representations for generative use to sample new model weights. We propose layer-wise loss normalization which we demonstrate is key to generate high-performing models and several sampling methods based on the topology of hyper-representations. The models generated using our methods are diverse, per- formant and capable to outperform strong baselines as evaluated on several down- stream tasks: initialization, ensemble sampling and transfer learning. Our results indicate the potential of knowledge aggregation from model zoos to new models via hyper-representations thereby paving the avenue for novel research directions.
Authors Schürholt, Konstantin; Knyazev, Boris; Giro-i-Nieto, Xavier & Borth, Damian
Research Team AIML Lab
Language English
Subjects computer science
HSG Classification contribution to scientific community
HSG Profile Area None
Date November 2022
Publisher Curran Associates, Inc.
Volume 35
Title of Book Advances in Neural Information Processing Systems
Event Title Conference on Neural Information Processing Systems
Event Location New Orleans
Depositing User Konstantin Schürholt
Date Deposited 25 Oct 2022 08:20
Last Modified 25 Oct 2022 08:20
URI: https://www.alexandria.unisg.ch/publications/267695

Download

[img] Text
HyperRepresentations_as_Generative_Models-camera_ready.pdf - Accepted Version
Available under License Creative Commons Attribution No Derivatives.

Download (4MB)

Citation

Schürholt, Konstantin; Knyazev, Boris; Giro-i-Nieto, Xavier & Borth, Damian: Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights. 2022. - Conference on Neural Information Processing Systems. - New Orleans.

Statistics

https://www.alexandria.unisg.ch/id/eprint/267695
Edit item Edit item
Feedback?