EPFL | Biomedical Imaging Group | DeepSpline Networks

Lipschitz Function Approximation using DeepSpline Neural Networks

S. Neumayer

EPFL-CIS & RIKEN-AIP Joint Workshop on Machine Learning (JWML'22), Virtual, September 7-8, 2022.

In this talk, we investigate NNs with prescribed bounds on the Lipschitz constant. One possibility to obtain Lipschitz-constrained NNs is to impose constraints on the architecture. Here, it turns out that this significantly limits the expressivity if we use the popular ReLU activation function. In particular, we are unable to represent even simple continuous piece-wise linear functions. On the contrary, using learnable linear splines instead fixes this problem and leads to maximal expressivity among all component-wise activation functions. From the many possible applications of Lipschitz-constrained NNs, we discuss one in more detail to see that the theoretical observations also transition into improved performance.

@INPROCEEDINGS(http://bigwww.epfl.ch/publications/neumayer2203.html,
AUTHOR="Neumayer, S.",
TITLE="{L}ipschitz Function Approximation using {DeepSpline} Neural
	Networks",
BOOKTITLE="{EPFL-CIS} \& {RIKEN-AIP} Joint Workshop on Machine Learning
	({JWML'22})",
YEAR="2022",
editor="",
volume="",
series="",
pages="",
address="Virtual",
month="September 7-8,",
organization="",
publisher="",
note="")

© 2022 EPFL. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from EPFL. This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.