EPFL | Biomedical Imaging Group

Deep Spline Networks with Control of Lipschitz Regularity

S. Aziznejad, M. Unser

Best student paper award, Proceedings of the Forty-Fourth IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'19), Brighton, United Kingdom, May 12-17, 2019, pp. 3242-3246.

The motivation for this work is to improve the performance of deep neural networks through the optimization of the individual activation functions. Since the latter results in an infinite-dimensional optimization problem, we resolve the ambiguity by searching for the sparsest and most regular solution in the sense of Lipschitz. To that end, we first introduce a bound that relates the properties of the pointwise nonlinearities to the global Lipschitz constant of the network. By using the proposed bound as regularizer, we then derive a representer theorem that shows that the optimum configuration is achievable by a deep spline network. It is a variant of a conventional deep ReLU network where each activation function is a piecewise-linear spline with adaptive knots. The practical interest is that the underlying spline activations can be expressed as linear combinations of ReLU units and optimized using ℓ₁-minimization techniques.

@INPROCEEDINGS(http://bigwww.epfl.ch/publications/aziznejad1901.html,
AUTHOR="Aziznejad, S. and Unser, M.",
TITLE="Deep Spline Networks with Control of {L}ipschitz Regularity",
BOOKTITLE="Proceedings of the Forty-Fourth IEEE International Conference
	on Acoustics, Speech, and Signal Processing ({ICASSP'19})",
YEAR="2019",
editor="",
volume="",
series="",
pages="3242--3246",
address="Brighton, United Kingdom",
month="May 12-17,",
organization="",
publisher="",
note="Best student paper award")

© 2019 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from IEEE. This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.