Structural Learning of Neural Networks
PhD thesis, Université Paris-Saclay, 2020
Supervised by Guillaume Charpiat and Yann Ollivier. Defended on March 6, 2020.
Abstract
The structure of a neural network determines to a large extent its cost of training and use, as well as its ability to learn. These two aspects are usually in competition: the larger a neural network is, the better it will perform the task assigned to it, but the more it will require memory and computing time resources for training. Automating the search of efficient network structures -of reasonable size and performing well- is then a very studied question in this area. Within this context, neural networks with various structures are trained, which requires a new set of training hyperparameters for each new structure tested. The aim of the thesis is to address different aspects of this problem. The first contribution is a training method that operates within a large perimeter of network structures and tasks, without needing to adjust the learning rate. The second contribution is a network training and pruning technique, designed to be insensitive to the initial width of the network. The last contribution is mainly a theorem that makes possible to translate an empirical training penalty into a Bayesian prior, theoretically well founded. This work results from a search for properties that theoretically must be verified by training and pruning algorithms to be valid over a wide range of neural networks and objectives.
BibTeX
@phdthesis{wolinski2020structural,
title = {Structural Learning of Neural Networks},
author = {Wolinski, Pierre},
year = {2020},
school = {Université Paris-Saclay},
url = {https://theses.fr/2020UPASS026}
}