Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions
Stanislas Ducotterd

Meeting • 2022-11-22

Abstract
Lipschitz-constrained neural networks have several advantages over unconstrained ones and can be applied to a variety of problems, making them a topic of attention in the deep learning community. Unfortunately, it has been shown both theoretically and empirically that they perform poorly when equipped with ReLU activation functions. This observation has justified the development of new activation functions specifically tailored for Lipschitz-constrained architectures. The current state-of-the-art is GroupSort which is multivariate and consists of splitting the input into groups and outputting each group sorted in ascending order. In this work, we propose an alternative way to boost the performance of Lipschitz-constrained networks via the use of 1-Lipschitz learnable linear spline activation functions. We compare our framework with other architectures including GroupSort on various tasks and show that we usually outperform them.