Single-Molecule Localization Microscopy  •  Software Benchmarking

3D SMLM Software Challenge

The 3D SMLM challenge is an ongoing competition to assess SMLM software on both simulated and real reference datasets using objective assessment metrics.

The 3D challenge remains continuously open to new submissions from both 2D and 3D SMLM software.

The first round of the challenge was presented in the Special Session at SMLMS 2016.


Note: We are currently in the process of updating this page. The latest and most detailed description of methods is contained in the 3D SMLM challenge preprint on bioRxiv.


1. Structure Sample

2. Photo-activation Model

3. Image Formation Model (PSF)

4. Noise Model

5. Wobble Correction

6. Metrics for the assessment

7. Tool: Baseline Performances

8. Tool: Comparison Localization

1. Structure Sample

The synthetic datasets were designed to be as similar as possible to images derived from cellular structures in real experimental conditions To achieve the high degree of realism, we defined mathematical models for biological structure that try to imitate microtubules and endoplasmic reticulum/mitochondria. These structure have a tubular shape in the 3D space. Typically, microtubules are defined with their central axis elongating in a 3D space having an average outer diameter of 25 nm with an inner, hollow tube of 15 nm diameter.

The underlying sample structure is formalized in a continuous space which allows rendering of digital images at any scale, from very high resolution (up to 1 nm/pixel) to low resolution (camera resolution: 100 nm). The continuous-domain 3D curve is represented by means of a polynomial spline. The sample is imaged in a limited field of view, i.e., less than 6.4 × 6.4 μm2, and the center lines of the microtubules have limited variation along the z (vertical) axis, i.e., less than 1.5 μm. The fluorescent markers are uniformly distributed over the structure according to the required density. The photon emission rate of each fluorophore is controlled by a photo-activation model (see below).

The exact locations of all fluorophores are therefore stored at high precision, as floating point numbers expressed in nanometers. This ground-truth file is useful for conducting objective evaluations without human bias.

Reference: Sage et al. Quantitative evaluation of software packages for single-molecule localization microscopy, Nature Methods 2015.

2. Photo-activation Model

Flux of photons

The flux of photons is given by the relation: F = Φ . P . σ / e in photons/seconds

Φ is the quantum yield og the dye

P is power of the laser (not spatially uniform) in W/cm2

e = h . c / λ is the energy of 1 photon is power of the laser (it is not spatially uniform)

σ = 1000 . ln(10) . ε / NA is the absorption cross section in cm2

ε is the molar extinction coefficient (EC) or absorptivity in cm2/mol

4-states photophysics model

Given a list of source locations from the structure simulator, fluorophore blinking was modelled by a 4 state process.

The Off to Om transitions have an uniform random distribution. It reflects normal experimental conditions where constant imaging density is maintained by tuning the photoactivation rate during the experiment. All other transitions are Poisson distributed. All switching is calculated at sub-frame resolution and then total fluorophore on-time was integrated over each frame.

The actual mean lifetime in On state is 1/(1/Ton + 1/Tbleach) due to two decay paths. Switching rates were chosen as "fluorescent-protein-like". For the training datasets: Ton = 3; Tdark = 2.5; Tbl=1.5; (unit is frames)

Fractional fluorophore on-times per frame (between 0 and 1) were then multiplied by the mean photon emission. At the end of this process a list of XY positions, on-frames and (noise-free) intensities for all activated fluorophores was obtained.

The Matlab code that implements this photo-activation model is available at:

3. Image Formation Model: PSF

In order to achieve as realistic a simulation as possible, model point spread functions were derived from experimentally measured PSFs. For each modality, images of fluorescent beads were recorded the conditions below. Signal to noise of recorded PSFs was maximised in all cases by maximizing exposure time and averaging over several frames to increase dynamic range.

Sample (all modalities)

PSF modality 10x10x10nm
3D TIFF file
Optics Camera Pixelsize at sample plane Z-stack step size
2D 2D PSF 2D-Exp †
(127 Mb)
Nikon NA 1.49 TIRF oil objective on Nikon N-STORM commercial microscope Andor iXon EMCCD
43 nm (1.5x microscope and 2.5x SIM magnifiers in place) 10 nm, 3 μm Z-range
AS Astigmatic PSF AS-Exp †
(111 Mb)
Nikon NA 1.49 TIRF oil objective EMCCD
160μm pixel size
160 nm 10 nm, 3 μm Z-range
DH Double-Helix PSF DH-Exp †
(150 Mb)
Nikon NA 1.49 TIRF oil objective EMCCD
160 nm Z20 nm, 3 μm Z-range
BP Biplane PSF

BP-250 †
BP+250 †
(2x127 Mb)
The biplane model PSF was constructed from the 2D PSF data – see below.

† Making available PSF 01 July 2018

At the stage of the challenge, the PSF are not yet making publicly available at 10x10x10nm resolution. The PSF can be estimated from the beads z-stack which have a 100x100x10nm resolution. The 10x10x10nm PSFs will be open later on.

Model PSF construction

This yielded 3 high SNR model PSFs (2D, astigmatism and double-helix) with a voxel size of 10x10x10 nm3. A central Z-range of 1.5 μm was selected. The biplane PSF was constructed by duplicating the 2D PSF and offsetting it by -250 nm and 250 nm for each Z-plane. For the DH-PSF, the transmission of the combined phase mask/ 4f system was measured as 96 %, which was approximated as 100 % brightness relative to the 2D-SPF and AS-PSF. The ground truth XY=0 was defined as the image centre of mass of the in-focus frame of the model PSF, and Z=0 was defined as the in-focus frame. Accounts for shifts in the fitted XY centre of the model PSF by localization software due to systematic offsets and Z-dependent variation of the model PSF centre of mass are dealt the wobble correction.

The Matlab code of the PSF generation is available at:

4. Noise Model

// Pseudo-code noise model  for LM challenge 2016.
// This assumes all input light is fluorescence 
// (background, signal) and thus follows poisson statistics
// The camera is an EMCCD, specifically the
// Photometrics Evolve Delta 512

for each pixel
  n_photIn is the input electrons
    // n_ie, apply noise model 
    // poisson noise including shot noise and spurious 
    // charge plus binomial quantum efficiency conversion
    // is just a Poisson distribution as per Hirsch.
  n_ie = poisson(QE*n_photIn + c);
    // emccd model, shape param k=n_ie, 
    // scale param theta=EMgain
    // after Basden et al Mon Not R Astron Soc 2003
  n_oe = gammadistribution(n_ie,EMgain);
  n_oe = n_oe + gaussian(read_noise);
    // analog to digital converter
  ADU_out = int(n_oe/e_per_adu)+ baseline;
    //restrict to 16 bit
  ADU_out = min(ADU_out,65535)

A constant mean autofluorescent background was added to the noise-free simulated images, and these images were then fed through the noise model representing Poisson distributed fluorescence emission recorded on a high quantum efficiency back-illuminated EMCCD.

Reference: model is inspired by Hirsch et al., PLoS One 2015.

Quantum Efficiency QE = 0.9 Evolve quantum efficiency @700 nm)
Readout noise read_noise = 74.4 Manufacturer measured rms electrons for Evolve
Spurious charge c = 0.0002 Manufacturer quoted spurious charge (CIC only, dark counts negligible) for Evolve
EM Gain Register EMgain = 300 Manufacturer quoted spurious charge (CIC only, dark counts negligible) for Evolve
Baseline baseline = 100 Typical value
Electron per analog-to-digital units e_per_adu = 45 ADC conversion factor, arbitary value similar to typical
Total system gain gain = 6 0.9 * 300 / 45 = 6

The Java code of the simulator is available at:

5. Wobble Correction

Example data is provided here for the competition
beads dataset, 2D PSF, analysed using RapidSTORM

Comparison of 3D localization to ground truth for experimentally derived model PSFs requires correction of of depth-dependent lateral distortion, here called wobble. This is due to arbitrary systematic offset since definition of PSF centre is arbitrary. It was mentioned in several sources but discussed most fully here: Carlini et al., Correction of a Depth-Dependent Lateral Distortion in 3D Super-Resolution Imaging, PLOS One, 2015.

In order to correct for this, competitors should either:

Provide the localizations file for the simulated beads data, and we will automatically correct the wobble/ offset using the tool discussed below.

Calculate and upload the wobble/offset using the tools provided. This also allows competitors to accurately compute their performance for the training data. The following code allows automatic calculation of the wobble/offset compared to ground truth data. For the competition, this should be performed for the simulated z-stack of beads for the PSF modality to be submitted (including 2D). The code generates a CSV file containing a list of x,y offset corrections as a function of Z, from -750 to 750 nm. This can be used by CompareLocalization Tool to calculate corrected performance metrics.

A typical plot of wobble as a function of Z is shown here. As can be seen, a small but significant (~50nm) Z-dependent offset is observed on both X- and Y-axes for Z-values <400 nm. At extreme Z-values, the wobble correction, ie the shift in detected PSF centroid shifts dramatically. This essentially indicates that beyond this limit fitting to the (extremely defocussed) PSF has failed. These large correction values at extreme Z are generally nothing to worry about as fitting to the PSF within this range has failed - ie spots outside this range will not be detected with meaningful accuracy - so the correction value is not relevant.

If the wobble/ offset stuff is confusing – please contact us. In any case for submitted data, there is the option for us to automatically calculate the correction based on the uploaded bead localizations, without the need to worry.

Matlab code and MS-Windows runtime to perform the wobble correction

Example of wobble correction, results and figures

Example of wobble file


6. Metrics of the Assessment

7. Tool: Baseline Performances

Objectives: This tool is a minimalist SMLM software that performs localizations of bright emitters on the 4 modalities of the challenge 2016: 2D, 3D-Astigmatism, 3D-Double-Helix, and 3D-Biplane. This SMLM_BaselineLocalization software is only designed to establish the performance baseline for the SMLM challenge.

It is fast and very inaccurate! It has intentionally limited lines of code and relies only on two threshold parameters.

3D Calibration: there is a very simple calibration tool that has to run on a z-stack of beads to find the linear relation between the axial position Z and the shape of the bead.

Distribution: The open-source software SMLM_BaselineLocalization is released as a Java ImageJ plugin.

Download the plugin: SMLM_BaselineLocalization.jar (Put this jar file into the plugin folder of ImageJ or Fiji)

Download the Java source code:

See a screenshot: screenshot.png

8. Tool: Compare Localization 3D

Objectives: This is a piece of software to compute the rate of detection and the accuracy in 3D based on 2 sets of localization (x, y, z).

Wobble correction: 3D localization requires correction of depth-dependent lateral distortion, called wobble, see above.

Assessment: CompareLocalization3D performs 8 assessments:

The open-source software CompareLocalization3D is released as a stand-alone Java application.

Download the jar file: CompareLocalization3D.jar (Double-click on jar file to start the application)

Download the Java source code:

See a screenshot: screenshot.png

© 2018 Biomedical Imaging Group, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
Last update: 30 Nov 2018