BIG > Tutorials, Reviews and Recent Talks 
CONTENTS 
DOCUMENTS 
Sparsity 
Representer theorems for illposed inverse problems with sparsity constraints 

M. Unser 


One World Seminar: Mathematical Methods for Arbitrary Data Sources (MADS), June 8, 2020 

Regularization addresses the illposedness of the training problem in machine learning or the reconstruction of a signal from a limited number of measurements. The standard strategy consists in augmenting the original cost functional by an energy that penalizes solutions with undesirable behaviour. In this presentation, I will present a general representer theorem that characterizes the solutions of a remarkably broad class of optimization problems in Banach spaces and helps us understand the effect of regularization. I will then use the theorem to retrieve some classical characterizations such as the celebrated representer theorem of machine leaning for RKHS, Tikhonov regularization, representer theorems for sparsity promoting functionals, as well as a few new ones, including a result for deep neural networks. 



New representer theorems: From compressed sensing to deep learning 

M. Unser 

Mathematisches Kolloquium, Universität Wien, October 24, 2018. 

Regularization is a classical technique for dealing with illposed inverse problems; it has been used successfully for biomedical image reconstruction and machine learning. In this talk, we present a unifying continuousdomain formulation that addresses the problem of recovering a function f from a finite number of linear functionals corrupted by measurement noise. We show that depending on the type of regularizationTikhonov vs. generalized total variation (gTV)we obtain very different types of solutions/representer theorems. While the solutions can be interpreted as splines in both cases, the main distinction is that the spline knots are fixed and as many as there are data points in the former setting (classical theory of RKHS), while they are adaptive and few in the case of gTV. Finally, we consider the problem of the joint optimization of the weights and activation functions in a deep neural network subject to a secondorder total variation penalty. The remarkable outcome is that the optimal configuration is achieved with a deepspline network that can be realized using standard ReLU units. The latter result is compatible with the stateoftheart in deep learning, but it also suggests some new computational/ optimization challenges. References 1. M. Unser, J. Fageot, J.P. Ward, "Splines Are Universal Solutions of Linear Inverse Problems with Generalized TV Regularization," SIAM Review, vol. 59, no. 4, pp. 769793, December 2017. 2. M. Unser, "A Representer Theorem for Deep Neural Networks," arXiv:1802.09210 [stat.ML] 

Representer theorems for illposed inverse problems with sparsity constraints 

M. Unser 


BLISS Seminar, June 28, 2017, University of California, Berkeley 

Abstract: Illposed inverse problems are often constrained by imposing a bound on the total variation of the solution. Here, we consider a generalized version of totalvariation regularization that is tied to some differential operator L. We then show that the general form of the solution is a nonuniform Lspline with fewer knots than the number of measurements. For instance, when L is the derivative operator, then the solution is piecewise constant. The powerful aspect of this characterization is that it applies to any linear inverse problem. Bio: Michael Unser is currently a Professor and the Director of the Biomedical Imaging Group, EPFL, Lausanne, Switzerland. His primary area of investigation is biomedical image processing. He is internationally recognized for his research contributions to sampling theory, wavelets, the use of splines for image processing, stochastic processes, and computational bioimaging. He has authored over 250 journal papers on those topics.




Biomedical Image Reconstruction 

M. Unser 


12th European Molecular Imaging Meeting, 57 April 2017, Cologne, Germany. 

A fundamental component of the imaging pipeline is the reconstruction algorithm. In this educational session, we review the physical and mathematical principles that underlie the design of such algorithms. We argue that the concepts are fairly universal and applicable to a majority of (bio)medical imaging modalities, including magnetic resonance imaging and fMRI, xray computer tomography, and positronemission tomography (PET). Interestingly, the paradigm remains valid for modern cellular/molecular imaging with confocal/superresolution fluorescence microscopy, which is highly relevant to molecular imaging as well. In fact, we believe that the huge potential for crossfertilization and mutual reenforcement between imaging modalities has not been fully exploited yet. The prerequisite to image reconstruction is an accurate physical description of the imageformation process: the socalled forward model, which is assumed to be linear. Numerically, this translates into the specification of a system matrix, while the reconstruction of images conceptually boils down to a stable inversion of this matrix. The difficulty is essentially twofold: (i) the system matrix is usually much too large to be stored/inverted directly, and (ii) the problem is inherently illposed due to the presence of noise and/or bad conditioning of the system. Our starting point is an overview of the modalities in relation to their forward model. We then discuss the classical linear reconstruction methods that typically involve some form of backpropagation (CT or PET) and/or the fast Fourier transform (in the case of MRI). We present stabilized variants of these methods that rely on (Tikhonov) regularization or the injection of prior statistical knowledge under the Gaussian hypothesis. Next, we review modern iterative schemes that can handle challenging acquisition setups such as parallel MRI, nonCartesian sampling grids, and/or missing views. In particular, we discuss sparsitypromoting methods that are supported by the theory of compressed sensing. We show how to implement such schemes efficiently using simple combinations of linear solvers and thresholding operations. The main advantage of these recent algorithms is that they improve the quality of the image reconstruction. Alternatively, they allow a substantial reduction of the radiation dose and/or acquisition time without noticeable degradation in quality. This behavior is illustrated practically. In the final part of the tutorial, we discuss the current challenges and directions of research in the field; in particular, the necessity of dealing with large data sets in multiple dimensions: 2D or 3D space combined with time (in the case of dynamic imaging) and/or multispectral/multimodal information.




Sparsity and Inverse Problems: Think Analog, and Act Digital 

M. Unser 


IEEE International Conference on on Acoustics, Speech, and Signal Processing (ICASSP), March 2025, 2016, Shanghai, China. 

Sparsity and compressed sensing are very popular topics in signal processing. More and more researchers are relying on l1type minimization scheme for solving a variety of illposed problems in imaging. The paradigm is well established with a solid mathematical foundation, although the arguments that have been put forth in the past are deterministic and finitedimensional for the most part. In this presentation, we shall promote a continuousdomain formulation of the problem ("think analog") that is more closely tied to the physics of imaging and that also lends it itself better to mathematical analysis. For instance, we shall demonstrate that splines (which are inherently sparse) are global optimizers of linear inverse problems with totalvariation (TV) regularization constraints. Alternatively, one can adopt an infinitedimensional statistical point of view by modeling signals as sparse stochastic processes. The guiding principle it then to discretize the inverse problem by projecting both the statistical and physical measurement models onto a linear reconstruction space. This leads to the specification of a general class of maximum a posteriori (MAP) signal estimators complemented with a practical iterative reconstruction scheme ("act digital"). While the framework is compatible with the traditional methods of Tikhonov and TV, it opens the door to a much broader class of potential functions that are inherently sparse, while it also suggests alternative Bayesian recovery procedures. We shall illustrate the approach with the reconstruction of images in a variety of modalities including MRI, phasecontrast tomography, cryoelectron tomography, and deconvolution microscopy. In recent years, significant progress has been achieved in the resolution of illposed linear inverse problems by imposing l1/TV regularization constraints on the solution. Such sparsitypromoting schemes are supported by the theory of compressed sensing, which is finite dimensional for the most part. 



Sparsity and the optimality of splines for inverse problems: Deterministic vs. statistical justifications 

M. Unser 


Invited talk: Mathematics and Image Analysis (MIA'16), 1820 January, 2016, Institut Henri Poincaré, Paris, France. 

In recent years, significant progress has been achieved in the resolution of illposed linear inverse problems by imposing l1/TV regularization constraints on the solution. Such sparsitypromoting schemes are supported by the theory of compressed sensing, which is finite dimensional for the most part. In this talk, we take an infinitedimensional point of view by considering signals that are defined in the continuous domain. We claim that nonuniform splines whose type is matched to the regularization operator are optimal candidate solutions. We show that such functions are global minimizers of a broad family of convex variational problems where the measurements are linear and the regularization is a generalized form of total variation associated with some operator L. We then discuss the link with sparse stochastic processes that are solutions of the same type of differential equations.The pleasing outcome is that the statistical formulation yields maximum a posteriori (MAP) signal estimators that involve the same type of sparsitypromoting regularization, albeit in a discretized form. The latter corresponds to the loglikelihood of the projection of the stochastic model onto a finitedimensional reconstruction space.




Sparse stochastic processes: A statistical framework for compressed sensing and biomedical image reconstruction 

M. Unser 


Plenary. IEEE International Symposium on Biomedical Imaging (ISBI), 1619 April, 2015, New York, USA. 

Sparsity is a powerful paradigm for introducing prior constraints on signals in order to address illposed image reconstruction problems. In this talk, we first present a continuousdomain statistical framework that supports the paradigm. We consider stochastic processes that are solutions of nonGaussian stochastic differential equations driven by white Lévy noise. We show that this yields intrinsically sparse signals in the sense that they admit a concise representation in a matched wavelet basis. We apply our formalism to the discretization of illconditioned linear inverse problems where both the statistical and physical measurement models are projected onto a linear reconstruction space. This leads to the specification of a general class of maximum a posteriori (MAP) signal estimators complemented with a practical iterative reconstruction scheme. While our family of estimators includes the traditional methods of Tikhonov and totalvariation (TV) regularization as particular cases, it opens the door to a much broader class of potential functions that are inherently sparse and typically nonconvex. We apply our framework to the reconstruction of images in a variety of modalities including MRI, phasecontrast tomography, cryoelectron tomography, and deconvolution microscopy. Finally, we investigate the possibility of specifying signal estimators that are optimal in the MSE sense. There, we consider the simpler denoising problem and present a direct solution for firstorder processes based on message passing that serves as our goldstandard. We also point out some of the pittfalls of the MAP paradigm (in the nonGaussian setting) and indicate future directions of research. 



Sparse stochastic processes: A statistical framework for compressed sensing and biomedical image reconstruction 

M. Unser 


4 hours tutorial, Inverse Problems and Imaging Conference, Institut Henri Poincaré, Paris, April 711, 2014. 

We introduce an extended family of continuousdomain sparse processes that are specified by a generic (nonGaussian) innovation model or, equivalently, as solutions of linear stochastic differential equations driven by white Lévy noise. We present the functional tools for their characterization. We show that their transformdomain probability distributions are infinitely divisible, which induces two distinct types of behavior‐Gaussian vs. sparse‐at the exclusion of any other. This is the key to proving that the nonGaussian members of the family admit a sparse representation in a matched wavelet basis. Next, we apply our continuousdomain characterization of the signal to the discretization of illconditioned linear inverse problems where both the statistical and physical measurement models are projected onto a linear reconstruction space. This leads the derivation of a general class of maximum a posteriori (MAP) signal estimators. While the formulation is compatible with the standard methods of Tikhonov and l1type regularizations, which both appear as particular cases, it open the door to a much broader class of sparsitypromoting regularization schemes that are typically nonconvex. We illustrate the concept with the derivation of algorithms for the reconstruction of biomedical images (deconvolution microscopy, MRI, Xray tomography) from noisy and/or incomplete data. The proposed framework also suggests alternative Bayesian recovery procedures that minimize t he estimation error. Reference




Sparse stochastic processes: A statistical framework for modern signal processing 

M. Unser 

Plenary talk, Int. Conf. Syst. Sig. Im. Proc. (IWSSIP), Bucharest, July 79, 2013. 

We introduce an extended family of sparse processes that are specified by a generic (nonGaussian) innovation model or, equivalently, as solutions of linear stochastic differential equations driven by white Lévy noise. We present the mathematical tools for their characterization. The two leading threads of the exposition are




Towards a theory of sparse stochastic processes, or when Paul Lévy joins forces with Nobert Wiener 

M. Unser 

Mathematics and Image Analysis 2012 (MIA'12), Paris, January 1618, 2012 

The current formulations of compressed sensing and sparse signal recovery are based on solid variational principles, but they are fundamentally deterministic. By drawing on the analogy with the classical theory of signal processing, it is likely that further progress may be achieved by adopting a statistical (or estimation theoretic) point of view. Here, we shall argue that Paul Lévy (1886 1971), who was one of the very early proponents of Haar wavelets, was in advance over his time, once more. He is the originator of the LévyKhinchine formula, which happens to be the perfect (nonGaussian) ingredient to support a continuousdomain theory of sparse stochastic processes. Specifically, we shall present an extended class of signal models that are ruled by stochastic differential equations (SDEs) driven by white Léevy noise. Léevy noise is a highly singular mathematical entity that can be interpreted as the weak derivative of a Lévy process. A special case is Gaussian white noise which is the weak derivative of the Wiener process (a.k.a. Brownian motion). When the excitation (or innovation) is Gaussian, the proposed model is equivalent to the traditional one. Of special interest is the property that the signals generated by nonGaussian linear SDEs tend to be sparse by construction; they also admit a concise representation in some adapted wavelet basis. Moreover, these processes can be (approximately) decoupled by applying a discrete version of the whitening operator (e.g., a finitedifference operator). The corresponding loglikelihood functions, which are nonquadratic, can be specified analytically. In particular, this allows us to uncover a Levy processes that results in a maximum a posteriori (MAP) estimator that is equivalent to total variation. We make the connection with current methods for the recovery of sparse signals and present some examples of MAP reconstruction of MR images with sparse priors. 



Wavelets, sparsity and biomedical image reconstruction 

M. Unser 

Imaging Seminar, University of Bern, Inselspital November 13, 2012. 

Our purpose in this talk is to advocate the use of wavelets for advanced biomedical imaging. We start with a short tutorial on wavelet bases, emphasizing the fact that they provide a sparse representation of images. We then discuss a simple, but remarkably effective, imagedenoising procedure that essentially amounts to discarding small wavelet coefficients (softthresholding). The crucial observation is that this type of “sparsitypromoting” algorithm is the solution of a l1norm minimization problem. The underlying principle of wavelet regularization is a powerful concept that has been used advantageously for compressed sensing and for reconstructing images from limited and/or noisy measurements. We illustrate the point by presenting waveletbased algorithms for 3D deconvolution microscopy, and MRI reconstruction (with multiple coils and/or nonCartesian kspace sampling). These methods were developed at the EPFL in collaboration with imaging scientists and are, for the most part, providing stateoftheart performance. 



Stochastic Models for Sparse and PiecewiseSmooth Signals 

M. Unser 

Sparse Representations and Efficient Sensing of Data, Schloss Dagstuhl, Jan. 30  Feb. 4, 2011 

We introduce an extended family of continuousdomain stochastic models for sparse, piecewisesmooth signals. These are specified as solutions of stochastic differential equations, or, equivalently, in terms of a suitable innovation model; this is analogous conceptually to the classical interpretation of a Gaussian stationary process as filtered white noise. The nonstandard aspect is that the models are driven by nonGaussian noise (impulsive Poisson or alphastable) and that the class of admissible whitening operators is considerably larger than what is allowed in the conventional theory of stationary processes. We provide a complete distributional characterization of these processes. We also introduce signals that are the nonGaussian (sparse) counterpart of fractional Brownian motion; they are nonstationary and have the same ωtype spectral signature. We prove that our generalized processes have a sparse representation in a waveletlike basis subject to some mild matching condition. Finally, we discuss implications for sampling and sparse signal recovery. 



Recent Advances in Biomedical Imaging and Signal Analysis 

M. Unser 

Proceedings of the Eighteenth European Signal Processing Conference (EUSIPCO'10), Ålborg, Denmark, August 2327, 2010, EURASIP Fellow inaugural lecture. 

Wavelets have the remarkable property of providing sparse representations of a wide variety of "natural" images. They have been applied successfully to biomedical image analysis and processing since the early 1990s. In the first part of this talk, we explain how one can exploit the sparsifying property of wavelets to design more effective algorithms for image denoising and reconstruction, both in terms of quality and computational performance. This is achieved within a variational framework by imposing some ℓ_{1}type regularization in the wavelet domain, which favors sparse solutions. We discuss some corresponding iterative skrinkagethresholding algorithms (ISTA) for sparse signal recovery and introduce a multilevel variant for greater computational efficiency. We illustrate the method with two concrete imaging examples: the deconvolution of 3D fluorescence micrographs, and the reconstruction of magnetic resonance images from arbitrary (nonuniform) kspace trajectories. In the second part, we show how to design new wavelet bases that are better matched to the directional characteristics of images. We introduce a general operatorbased framework for the construction of steerable wavelets in any number of dimensions. This approach gives access to a broad class of steerable wavelets that are selfreversible and linearly parameterized by a matrix of shaping coefficients; it extends upon Simoncelli's steerable pyramid by providing much greater wavelet diversity. The basic version of the transform (higherorder Riesz wavelets) extracts the partial derivatives of order N of the signal (e.g., gradient or Hessian). We also introduce a signaladapted design, which yields a PCAlike tight wavelet frame. We illustrate the capabilities of these new steerable wavelets for image analysis and processing (denoising). 


© 2020 EPFL • webmaster.big@epfl.ch • 02.11.2020