Splines for Image Processing

Spline Methods for the Continuous/Discrete Processing of Images

Principal Investigator: Prof. Michael Unser

Participants: Mehrsa Pourya, Vincent Guillemet, Rahul Parhi

Period: November 1, 2023 - October 31, 2027.

Abstract

Our aim is to develop a comprehensive framework to address a whole class of problems that are usually formulated in the continuous domain, but call for a digital implementation. Our practical focus is on image processing while the theoretical context is provided by splines. This project started in 1998 and its scope has continued to broaden and to become more mathematical as fundamental aspects of splines (related to differential equations, estimation and regularization theory, stochastic processes, and inverse problems) have been uncovered.

The specific spline-related research topics that will be investigated during the 2023-2027 period are:

1. Splines and inverse problems

Splines can be specified as solutions of data-fitting problems subject to a Hilbertian regularization, which involves a defining operator L and a native space (reproducing-kernel Hilbert space). During the last term, we have generalized the concept to linear inverse problems. Specifically, by substituting the L2-norm of the classical regularizer with its proper L1 counterpart, we could prove that the solution is achieved by an L-spline (of a type matched to the operator L) with a minimal number of adaptive knots. We shall extend our formulation by considering additional regularization norms and domains, as well as broader classes of inverse problems. We shall apply our framework to the resolution of nonlinear inverse problems in optical imaging.

2. Box splines and generalized total variation

With the advent of compressed sensing, Hilbertian norms and regularizers have been supplanted by their sparsity-promoting counterparts (see also Item 1). To accommodate this shift in paradigm, we shall revamp our spline toolbox by replacing the cubic B-splines by continuous piecewise-linear box splines. We shall use these to derive exact discretizations of the total-variation (TV) energy and its second-order extension (HTV) in any number of dimensions. We shall apply the framework to the resolution of concrete inverse problems in the continuum (imaging), using a coarse-to-fine strategy to progressively refine the discretization grid such as to ensure convergence to the continuous-domain solution.

3. Vector-valued splines

We shall extend our spline framework to the handling of vector-valued signals. To do so, we shall rely on matrix-valued digital filters and suitable matrices of differential operators, the idea being to exploit inter-channel dependencies. We shall construct vector-valued B-splines/interpolators and characterize the solution of variational vector-valued problems. We shall also make the connection with vector-valued stochastic processes. We shall apply our techniques to the reconstruction and processing of vector fields.

4. Splines and the k-plane Radon transform

We shall investigate a novel representation of functions on R^d that involves a special kind of spline atoms: the rotated and translated versions of a common (d - k)-dimensional profile, which we call a k-ridge. We intend to prove that these new splines are solutions of variational problems involving a defining operator L and the k-plane Radon transform. The two extreme cases of this extension are connected to the two most popular representations used in machine learning: the radial-basis functions for k = 0, and neural networks with one hidden layer for k = d - 1.

5. Native spaces for splines

The specification of a variational spline has two fundamental ingredients: a spline energy (regularization) and a native space. The latter is the largest possible collection of functions over which the defining energy-minimization problem admits a solution. While the theory of native spaces is well-established for Hilbertian energies, the construction of native spaces has remained elusive for other regularization norms, especially for the scenarios where the null space of the regularization operator is nontrivial. We intend to fill this gap by developing a generic construction of native spaces which should cover all the cases that are of practical interest to us.