On Radon-Domain BV Spaces: The Native Spaces for Shallow Neural Networks
Rahul Parhi

Meeting • 2022-09-27

Abstract
Neural networks are not well understood mathematically and their success in many science and engineering applications is usually only backed by empirical evidence. In this talk, we will discuss studying neural networks from first principles. We use tools from variational spline theory to mathematically understand neural networks. In particular, we view neural networks as a type of spline. We propose and study a new family of Banach spaces, which are bounded variation (BV) spaces defined via the Radon transform. These are the “native spaces” for neural networks. We show that finite-width neural networks are solutions to data-fitting variational problems over these spaces. Moreover, these variational problems can be recast as finite-dimensional neural network training problems with regularization schemes related to weight decay and path-norm regularization, giving theoretical insight into these common regularization methods for neural networks. The Radon-domain BV spaces are also interesting from the perspective of functional analysis and statistical estimation. The best approximation and estimation error rates of these spaces are (essentially) independent of the input dimension, while the best linear approximation and estimation error rates suffer the curse of dimensionality. The Radon-domain BV spaces contain functions that are very smooth in all directions except (perhaps) a few directions. The anisotropic nature of these spaces distinguishes them from classical function spaces studied in analysis.