BIOMEDICAL IMAGING GROUP (BIG)
Laboratoire d'imagerie biomédicale (LIB)

On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networks

S. Neumayer, L. Chizat, M. Unser

Journal of Machine Learning Research, vol. 25, no. 15, pp. 1–24, 2024.

In supervised learning, the regularization path is sometimes used as a convenient theoretical proxy for the optimization path of gradient descent initialized from zero. In this paper, we study a modification of the regularization path for infinite-width 2-layer ReLU neural networks with nonzero initial distribution of the weights at different scales. By exploiting a link with unbalanced optimal-transport theory, we show that, despite the non-convexity of the 2-layer network training, this problem admits an infinite-dimensional convex counterpart. We formulate the corresponding functional-optimization problem and investigate its main properties. In particular, we show that, as the scale of the initialization ranges between 0 and +&8734#;, the associated path interpolates continuously between the so-called kernel and rich regimes. Numerical experiments confirm that, in our setting, the scaling path and the final states of the optimization path behave similarly, even beyond these extreme points. 

@ARTICLE(http://bigwww.epfl.ch/publications/neumayer2401.html,
AUTHOR="Neumayer, S. and Chizat, L. and Unser, M.",
TITLE="On the Effect of Initialization: The Scaling Path of 2-Layer
	Neural Networks",
JOURNAL="Journal of Machine Learning Research",
YEAR="2024",
volume="25",
number="15",
pages="1--24",
month="",
note="")
© 2024 The Authors. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from The Authors. This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.