Single-Molecule Localization Microscopy  •  Software Benchmarking

Second edition of the challenge focussed on 3D

The challenge has been presented at Single Molecule Localization Microscopy Symposium (SMLMS) at Lausanne, Switzerland, August 28-30, 2016. This edition will be focussed on 3D localization techniques, astigmatism, biplane, or double-helix.



1. Structure Sample

2. Photo-activation Model

3. Image Formation Model (PSF)

4. Noise Model

5. Wobble Correction

6. Metrics for the assessment

7. Tool: Baseline Performances

8. Tool: Comparison Localization

1. Structure Sample

The synthetic datasets were designed to be as similar as possible to images derived from cellular structures in real experimental conditions To achieve the high degree of realism, we defined mathematical models for biological structure that try to imitate microtubules and endoplasmic reticulum/mitochondria. These structure have a tubular shape in the 3D space. Typically, microtubules are defined with their central axis elongating in a 3D space having an average outer diameter of 25 nm with an inner, hollow tube of 15 nm diameter.

The underlying sample structure is formalized in a continuous space which allows rendering of digital images at any scale, from very high resolution (up to 1 nm/pixel) to low resolution (camera resolution: 100 nm). The continuous-domain 3D curve is represented by means of a polynomial spline. The sample is imaged in a limited field of view, i.e., less than 6.4 × 6.4 μm2, and the center lines of the microtubules have limited variation along the z (vertical) axis, i.e., less than 1.5 μm. The fluorescent markers are uniformly distributed over the structure according to the required density. The photon emission rate of each fluorophore is controlled by a photo-activation model (see below).

The exact locations of all fluorophores are therefore stored at high precision, as floating point numbers expressed in nanometers. This ground-truth file is useful for conducting objective evaluations without human bias.

Reference: Sage et al. Quantitative evaluation of software packages for single-molecule localization microscopy, Nature Methods 2015.

2. Photo-activation Model

Flux of photons

The flux of photons is given by the relation: F = Φ . P . σ / e in photons/seconds

Φ is the quantum yield og the dye

P is power of the laser (not spatially uniform) in W/cm2

e = h . c / λ is the energy of 1 photon is power of the laser (it is not spatially uniform)

σ = 1000 . ln(10) . ε / NA is the absorption cross section in cm2

ε is the molar extinction coefficient (EC) or absorptivity in cm2/mol

4-states photophysics model

Given a list of source locations from the structure simulator, fluorophore blinking was modelled by a 4 state process:

The OFF to ON transition is not Poisson distributed it is uniform random, to reflect that in normal experimental conditions constant imaging density is maintained by tuning the photoactivation rate during the experiment. All other transitions are Poisson distributed. All switching is calculated at sub-frame resolution and then total fluorophore on-time was integrated over each frame.

The actual mean lifetime in On state is 1/(1/Ton + 1/Tbleach) due to two decay paths. Switching rates were chosen as "fluorescent-protein-like". For the training datasets: Ton = 3; Tdark = 2.5; Tbl=1.5; (units of frames)

Fractional fluorophore on-times per frame (between 0 and 1) were then multiplied by the mean photon emission. At the end of this process a list of XY positions, on-frames and (noise-free) intensities for all activated fluorophores was obtained.

Code is here:

3. Image Formation Model: PSF

In order to achieve as realistic a simulation as possible, model point spread functions were derived from experimentally measured PSFs. For each modality, images of fluorescent beads were recorded the conditions below. Signal to noise of recorded PSFs was maximised in all cases by maximizing exposure time and averaging over several frames to increase dynamic range.

Sample (all modalities)

PSF imaging conditions Optics Camera Pixelsize Z-stack
2D 2D PSF Nikon NA 1.49 TIRF oil objective on Nikon N-STORM commercial microscope Andor iXon EMCCD
Pixel size at sample plane: 43 nm (1.5x microscope and 2.5x SIM magnifiers in place) Z-stack step size: 10 nm, 3 μm Z-range
AS Astigmatic PSF Nikon NA 1.49 TIRF oil objective EMCCD
160μm pixel size
Pixel size at sample plane: 100 nm Z-stack step size: 10 nm, 3 μm Z-range
DH Double-Helix PSF Nikon NA 1.49 TIRF oil objective EMCCD
Pixel size at sample plane: 100 nm Z-stack step size: 20 nm, 3 μm Z-range
BP Biplane PSF

The biplane model PSF was constructed from the 2D PSF data – see below.

Model PSF construction

This yielded 3 high SNR model PSFs (2D, astigmatism and double-helix) with a voxel size of 10x10x10 nm3. A central Z-range of 1.5 μm was selected. The biplane PSF was constructed by duplicating the 2D PSF and offsetting it by -250 nm and 250 nm for each Z-plane. For the DH-PSF, the transmission of the combined phase mask/ 4f system was measured as 96 %, which was approximated as 100 % brightness relative to the 2D-SPF and AS-PSF. The ground truth XY=0 was defined as the image centre of mass of the in-focus frame of the model PSF, and Z=0 was defined as the in-focus frame. Accounts for shifts in the fitted XY centre of the model PSF by localization software due to systematic offsets and Z-dependent variation of the model PSF centre of mass are dealt the wobble correction.

PSF generation code is here:

4. Noise Model

// Pseudo-code noise model  for LM challenge 2016.
// This assumes all input light is fluorescence 
// (background, signal) and thus follows poisson statistics
// The camera is an EMCCD, specifically the
// Photometrics Evolve Delta 512

for each pixel
  n_photIn is the input electrons
    // n_ie, apply noise model 
    // poisson noise including shot noise and spurious 
    // charge plus binomial quantum efficiency conversion
    // is just a Poisson distribution as per Hirsch.
  n_ie = poisson(QE*n_photIn + c);
    // emccd model, shape param k=n_ie, 
    // scale param theta=EMgain
    // after Basden et al Mon Not R Astron Soc 2003
  n_oe = gammadistribution(n_ie,EMgain);
  n_oe = n_oe + gaussian(read_noise);
    // analog to digital converter
  ADU_out = int(n_oe/e_per_adu)+ baseline;
    //restrict to 16 bit
  ADU_out = min(ADU_out,65535)

A constant mean autofluorescent background was added to the noise-free simulated images, and these images were then fed through the noise model representing Poisson distributed fluorescence emission recorded on a high quantum efficiency back-illuminated EMCCD.

Reference: model is inspired by Hirsch et al., PLoS One 2015.

Quantum Efficiency QE = 0.9 Evolve quantum efficiency @700 nm)
Readout noise read_noise = 74.4 Manufacturer measured rms electrons for Evolve
Spurious charge c = 0.0002 Manufacturer quoted spurious charge (CIC only, dark counts negligible) for Evolve
EM Gain Register EMgain = 300 Manufacturer quoted spurious charge (CIC only, dark counts negligible) for Evolve
Baseline baseline = 100 Typical value
Electron per analog-to-digital units e_per_adu = 45 ADC conversion factor, arbitary value similar to typical
Total system gain gain = 6 0.9 * 300 / 45 = 6

Note: Quantization noise not an issue as noted by lots of astronomers and e.g. [1] Hirsch et al, PLoS One 2015.

Java code of the simulator is available on Github

5. Wobble Correction


Comparison of 3D localization to ground truth for experimentally derived model PSFs requires correction of of depth-dependent lateral distortion, here called wobble. This is due to arbitrary systematic offset since definition of PSF centre is arbitrary. It was mentioned in several sources but discussed most fully here: Carlini et al., Correction of a Depth-Dependent Lateral Distortion in 3D Super-Resolution Imaging, PLOS One, 2015.

In order to correct for this, competitors should either:

Example data is provided here for the competition beads dataset, 2D PSF, analysed using RapidSTORM
A typical plot of wobble as a function of Z is shown here. As can be seen, a small but significant (~50nm) Z-dependent offset is observed on both X- and Y-axes for Z-values <400 nm. At extreme Z-values, the wobble correction, ie the shift in detected PSF centroid shifts dramatically. This essentially indicates that beyond this limit fitting to the (extremely defocussed) PSF has failed. These large correction values at extreme Z are generally nothing to worry about as fitting to the PSF within this range has failed - ie spots outside this range will not be detected with meaningful accuracy - so the correction value is not relevant.

If the wobble/ offset stuff is confusing – please contact us. In any case for submitted data, there is the option for us to automatically calculate the correction based on the uploaded bead localizations, without the need to worry.

Matlab code and example are available on Github

6. Metrics of the Assessment

7. Tool: Baseline Performances

Objectives: This tool is a minimalist SMLM software that performs localizations of bright emitters on the 4 modalities of the challenge 2016: 2D, 3D-Astigmatism, 3D-Double-Helix, and 3D-Biplane. This SMLM_BaselineLocalization software is only designed to establish the performance baseline for the SMLM challenge.

It is fast and very inaccurate! It has intentionally limited lines of code and relies only on two threshold parameters.

3D Calibration: there is a very simple calibration tool that has to run on a z-stack of beads to find the linear relation between the axial position Z and the shape of the bead.

Distribution: The open-source software SMLM_BaselineLocalization is released as a Java ImageJ plugin.

8. Tool: Compare Localization 3D

Objectives: This is a piece of software to compute the rate of detection and the accuracy in 3D based on 2 sets of localization (x, y, z).

Wobble correction: 3D localization requires correction of depth-dependent lateral distortion, called wobble, see above.

Assessment: CompareLocalization3D performs 8 assessments:

The open-source software CompareLocalization3D is released as a stand-alone Java application.

© 2017 Biomedical Imaging Group, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
Last update: 31 Mar 2017