Neural Networks and Minimum-Norm Ridge Splines
M. Unser
Keynote address, Proceedings of the HCM Workshop: Synergies Between Data Sciences and PDE Analysis (HCM'22), Bonn, Federal Republic of Germany, June 13-17, 2022, pp. 1.
A powerful framework for supervised learning is the minimization of a cost that consists of a data fidelity term plus a regularization functional. In this talk, I investigate a Radon-domain regularization functional that depends on a generic operator L. The proposed formulation yields a solution that takes the form of a two layer neural network with an activation function that is determined by the regularization operator. In particular, one retrieves the popular ReLU networks by taking L to be the Laplacian. The proposed setting offers guarantees of universal approximation for a broad family of regularization operators or, equivalently, for a wide variety of shallow neural networks including cases (such as ReLU) where the activation function is increasing polynomially. It also explains the favorable role of bias and skip connections in neural architectures.
@INPROCEEDINGS(http://bigwww.epfl.ch/publications/unser2202.html,
AUTHOR="Unser, M.",
TITLE="Neural Networks and Minimum-Norm Ridge Splines",
BOOKTITLE="Proceedings of the {HCM} Workshop: {S}ynergies Between Data
Sciences and {PDE} Analysis ({HCM'22})",
YEAR="2022",
editor="",
volume="",
series="",
pages="1",
address="Bonn, Federal Republic of Germany",
month="June 13-17,",
organization="",
publisher="",
note="Keynote address")