Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction
Type
conference paper
Date Issued
2021-11-09
Author(s)
Research Team
AIML Lab
Abstract
Self-Supervised Learning (SSL) has been shown to learn useful and information- preserving representations. Neural Networks (NNs) are widely applied, yet their weight space is still not fully understood. Therefore, we propose to use SSL to learn neural representations of the weights of populations of NNs. To that end, we introduce domain specific data augmentations and an adapted attention architecture. Our empirical evaluation demonstrates that self-supervised representation learning in this domain is able to recover diverse NN model characteristics. Further, we show that the proposed learned representations outperform prior work for predicting hyper-parameters, test accuracy, and generalization gap as well as transfer to out-of-distribution settings.
Language
English
HSG Classification
contribution to scientific community
Publisher
Neural Information Processing Systems (NeurIPS)
Publisher place
Sydney, Australia
Volume
35
Event Title
Neural Information Processing Systems (NeurIPS)
Subject(s)
Division(s)
Contact Email Address
konstantin.schuerholt@unisg.ch
Eprints ID
264718
File(s)![Thumbnail Image]()
Name
neurips_2021.pdf
Size
4.92 MB
Format
Adobe PDF
Checksum (MD5)
051973b9912939dc857e17bd2aa23d46