Rethinking Gauss-Newton for learning over-parameterized models

Michael Arbel, Romain Ménégaux, Pierre Wolinski

NeurIPS 2023

Proceedings · arXiv · PDF

Romain Ménégaux and Pierre Wolinski contributed equally.

Abstract

This work studies the global convergence and implicit bias of Gauss Newton's (GN) when optimizing over-parameterized one-hidden layer networks in the mean-field regime. We first establish a global convergence result for GN in the continuous-time limit exhibiting a faster convergence rate compared to GD due to improved conditioning. We then perform an empirical study on a synthetic regression task to investigate the implicit bias of GN's method. While GN is consistently faster than GD in finding a global optimum, the learned model generalizes well on test data when starting from random initial weights with a small variance and using a small step size to slow down convergence. Specifically, our study shows that such a setting results in a hidden learning phenomenon, where the dynamics are able to recover features with good generalization properties despite the model having sub-optimal training and test performances due to an under-optimized linear layer. This study exhibits a trade-off between the convergence speed of GN and the generalization ability of the learned solution.

BibTeX

@inproceedings{arbel2023rethinking,
  title     = {Rethinking Gauss-Newton for learning over-parameterized models},
  author    = {Arbel, Michael and Ménégaux, Romain and Wolinski, Pierre},
  year      = {2023},
  booktitle = {Advances in Neural Information Processing Systems 36 (NeurIPS 2023)},
  pages     = {33379--33402},
  publisher = {Curran Associates, Inc.},
  url       = {https://proceedings.neurips.cc/paper_files/paper/2023/hash/6a14c7f9fb3f42645cfa6bd5aa446819-Abstract-Conference.html}
}