Learning with Random Learning Rates

Léonard Blier, Pierre Wolinski, Yann Ollivier

ECML PKDD 2019

Publisher · arXiv · PDF · Code

Léonard Blier and Pierre Wolinski contributed equally.

Abstract

In neural network optimization, the learning rate of the gradient descent strongly affects performance. This prevents reliable out-of-the-box training of a model on a new problem. We propose the All Learning Rates At Once (Alrao) algorithm for deep learning architectures: each neuron or unit in the network gets its own learning rate, randomly sampled at startup from a distribution spanning several orders of magnitude. The network becomes a mixture of slow and fast learning units. Surprisingly, Alrao performs close to SGD with an optimally tuned learning rate, for various tasks and network architectures. In our experiments, all Alrao runs were able to learn well without any tuning.

BibTeX

@inproceedings{blier2019learning,
  title     = {Learning with Random Learning Rates},
  author    = {Blier, Léonard and Wolinski, Pierre and Ollivier, Yann},
  year      = {2019},
  booktitle = {Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2019)},
  pages     = {449--464},
  publisher = {Springer},
  doi       = {10.1007/978-3-030-46147-8_27}
}