Deep Spline Networks with Control of Lipschitz Regularity
S. Aziznejad, M. Unser
Best student paper award, Proceedings of the Forty-Fourth IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP'19), Brighton, United Kingdom, May 12-17, 2019, pp. 3242–3246.
The motivation for this work is to improve the performance of deep neural networks through the optimization of the individual activation functions. Since the latter results in an infinite-dimensional optimization problem, we resolve the ambiguity by searching for the sparsest and most regular solution in the sense of Lipschitz. To that end, we first introduce a bound that relates the properties of the pointwise nonlinearities to the global Lipschitz constant of the network. By using the proposed bound as regularizer, we then derive a representer theorem that shows that the optimum configuration is achievable by a deep spline network. It is a variant of a conventional deep ReLU network where each activation function is a piecewise-linear spline with adaptive knots. The practical interest is that the underlying spline activations can be expressed as linear combinations of ReLU units and optimized using ℓ1-minimization techniques.
@INPROCEEDINGS(http://bigwww.epfl.ch/publications/aziznejad1901.html,
AUTHOR="Aziznejad, S. and Unser, M.",
TITLE="Deep Spline Networks with Control of {L}ipschitz Regularity",
BOOKTITLE="Proceedings of the Forty-Fourth IEEE International Conference
on Acoustics, Speech, and Signal Processing ({ICASSP'19})",
YEAR="2019",
editor="",
volume="",
series="",
pages="3242--3246",
address="Brighton, United Kingdom",
month="May 12-17,",
organization="",
publisher="",
note="Best student paper award")