PERFORMANCE METRICS
To assess the quality of the reconstruction we provide several scripts
which implement the quality metrics that will be used in the deconvolution
challenge. Below we briefly describe these quality metrics and the usage
of the corresponding MATLAB scripts.
Peak Signal-to-Noise Ratio (PSNR)
The PSNR is defined as
$$
\textrm{PSNR}=10log_{10}(\textrm{peak}^2/\textrm{MSE})
$$
where \(\textrm{peak}\) is the peak value used in the computation and
\(\textrm{MSE}\) is the mean squared error between the reconstruction and the
reference image-stack.
To compute it we provide the
psnr.m script.
Required Input Arguments
- \(x\) : Reconstructed image stack (3D MATLAB array).
- \(f\) : Ground-truth image stack (3D MATLAB array).
Optional Input Arguments
- \(p\) : Peak value used in the computation of PSNR (Default: maximum value of \(f\)).
Output Arguments
- \(\textrm{psnr}\) : Peak signal to noise ratio.
- \(\textrm{mse}\) : Minimum mean squared error between \(x\) and \(f\).
Normalized Mean Integrated Squared Error (NMISE)
We use a modified version of the original NMISE metric, which is better
suited to the image restoration problem. It is defined as
$$
\textrm{NMISE}=\frac{1}{N}\sum_{i=1}^N(x_i - f_i)^2/{f_b}_i.
$$
To compute it we provide the
nmisec.m script.
Required Input Arguments
- \(x\) : Reconstructed image stack (3D MATLAB array).
- \(f\) : Ground-truth image stack (3D MATLAB array).
- \(f_b\) : Intermediate result, \(f_b=\mathbf{K}\mathbf{x}+\mathbf{b}\), of the observation model (3D MATLAB array). Output of the
ForwardModel3D.m script.
Output Arguments
- \(\textrm{nmise}\) : Normalized mean integrated squared error between \(x\) and \(f\).
Structure Similarity Index (SSIM)
The structural similarity (SSIM) index is a metric for measuring the
similarity between two images. A description of this metric can be found
on
Wikipedia.
To serve the needs of the 3D reconstruction we provide the
ssim3D.m
script which can be applied to image volumes.
Required Input Arguments
- \(x\) : Reconstructed image stack (3D MATLAB array).
- \(f\) : Ground-truth image stack (3D MATLAB array).
Optional Input Arguments
- \(K\) : constants in the SSIM index formula (see ssim_index.m) (Default : \(K\) = [0.01 0.03]).
- \(\textrm{window}\) : local window for statistics (see ssim_index.m). The default window is Gaussian given by
window = fspecial('gaussian', 11, 1.5).
- L : dynamic range of the images (Default: maximum value of \(f\)).
Output Arguments
- \(E\) : The mean ssim index over all the slices of the image stacks.
- \(M\) : The minimum ssim index over all the slices of the image stacks.
Total Variation Score
In this challenge we are using a quality metric based on Total Variation.
The metric is defined as:
$$
R=\frac{\sum_{n=1}^{N}\|(\nabla x)_n-(\nabla f)_n\|}{\sum_{n=1}^N \|(\nabla f)_n\|}
$$
To compute it we provide the
TVscore.m script.
Required Input Arguments
- \(x\) : Reconstructed image stack (3D MATLAB array).
- \(f\) : Ground-truth image stack (3D MATLAB array).
Optional Input Arguments
- grid : A \(3\times 1\) vector which consists of the spatial sampling rates in the three dimensions of the image stack. If a different sampling rate is used for every dimension of the volume, then it has to be taken into account for the correct computation of the derivative operators. (Default value : [1 1 1])
- bc : A string which specifies the boundary conditions that will be used in the computation of the derivatives. Available inputs 'reflexive' or 'circular' or 'zero'. (Default: 'reflexive').
Output Arguments
- R : The TV score as defined above.
Structure Tensor Score
This quality metric is based on the eigenvalues of the 3D Structure
Tensor. It is defined as
$$
R=\frac{\sum_{n=1}^{N}\mid\|(S x)_n\|_1-\|(S f)_n\|_1\mid}{\sum_{n=1}^N \|(S
f)_n\|_1},
$$
where \(|(S x)_n\|_1=\sqrt{\lambda_1}+\sqrt{\lambda_2}+\sqrt{\lambda_3}\),
with \(\lambda_k, k=1,2,3\) the eigevalues of the Structure Tensor at the
nth voxel of the image stack.
To compute it we provide the
STensor_score.m script.
Required Input Arguments
- \(x\) : Reconstructed image stack (3D MATLAB array).
- \(f\) : Ground-truth image stack (3D MATLAB array).
Optional Input Arguments
- \(G\) : Smoothing kernel used for the computation of the Structure Tensor. (Default: 3D normalized Gaussian of support 3x3x3).
- grid : A \(3\times 1\) vector which consists of the spatial sampling rates in the three dimensions of the image stack. If a different sampling rate is used for every dimension of the volume, then it has to be taken into account for the correct computation of the derivative operators. (Default value : [1 1 1])
- bc : A string which specifies the boundary conditions that will be used in the computation of the derivatives. Available inputs 'reflexive' or 'circular' or 'zero'. (Default: 'reflexive').
Output Arguments
- \(R\) : The Structure Tensor score as defined above.
Curvature Score
This quality metric is based on the eigenvalues of the 3D Hessian
operator. It is defined as
$$
R=\frac{\sum_{n=1}^{N}\mid\|(H x)_n\|_F-\|(H f)_n\|_F\mid}{\sum_{n=1}^N
\|(H f)_n\|_F},
$$
where \(\|(H x)_n\|_F=\sqrt{\lambda_1^2+\lambda_2^2+\lambda_3^2}\), with \(\lambda_k, k=1,2,3\) the eigevalues of the Hessian operator at the nth voxel of the image stack.
To compute it we provide the
Curvature_score.m script.
Required Input Arguments
- \(x\) : Reconstructed image stack (3D MATLAB array).
- \(f\) : Ground-truth image stack (3D MATLAB array).
Optional Input Arguments
- grid : A \(3\times 1\) vector which consists of the spatial sampling rates in the three dimensions of the image stack. If a different sampling rate is used for every dimension of the volume, then it has to be taken into account for the correct computation of the derivative operators. (Default value : [1 1 1])
- bc : A string which specifies the boundary conditions that will be used in the computation of the derivatives. Available inputs 'reflexive' or 'circular' or 'zero'. (Default: 'reflexive').
Output Arguments
- \(R\) : The curvature score as defined above.
Wavelet Sparsity Index
This quality metric measures the sparsity of the input in a wavelet
transform domain.
To compute it we provide the
wavsparseidx.m script.
Required Input Arguments
- \(x\) : Input image stack (3D MATLAB array).
- \(J\) : Decomposition depth of the wavelet transoform.
- \(d\) : A boolean entry which indicates whether we are using a non-redundant (true) or redundant (false) wavelet transform.
- \(f\) : Wavelet family (see
BiorthWavFilters1D.m routine for the available wavelet transforms).
Output Arguments
- \(R\) : The wavelet sparsity index of the input.
Fourier Shell Correlation
The Fourier shell correlation measures the normalized cross-correlation coefficient between two image stacks over corresponding shells in the Fourier space, and is defined as
$$ \mbox{FSC}\left( r \right)=\frac{\sum_{r_i\in r}\hat{f}_1(r_i)\cdot \hat{f}_2^*(r_i)}{\sqrt{\sum_{r_i\in r}{|\hat{f}_1(r_i)|}^2\cdot\sum_{r_i\in r}{|\hat{f}_2(r_i)|}^2}}, $$
where \(\hat{f}_1(r_i), \hat{f}_2(r_i)\) are the Fourier components of the two image stacks at the given spatial frequency \(r_i\). To compute it we provide the
FourierShellCorrelation.m script.
Required Input Arguments
- m : Output of the
FourierMetricsConstructor.m script.
- \(\hat{f}_1\) : The Fourier transform of the reconstructed image stack.
- \(\hat{f}_2\) : The Fourier transform of the ground-truth image stack.
Output Arguments
- \(\mbox{FSC}\) : The Fourier shell correlation as defined above.
Relative Energy Regain
The relative energy regain is a Fourier-based quality metric which measures the recovery of information at a range of absolute spatial frequency. For more informations about this metric we refer to the following article: Heintzmann. Estimating missing information by maximum likelihood deconvolution. Micron 38 (2007) 136-144.
To compute it we provide the
RelativeEnergyRegain.m script.
Required Input Arguments
- m : Output of the
FourierMetricsConstructor.m script.
- \(\tilde{I}\) : The Fourier transform of the reconstructed image stack.
- \(\tilde{O}\) : The Fourier transform of the ground-truth image stack.
Output Arguments
- \(\mbox{G}_R\) : The relative energy regain.
