3D SMLM Software ChallengeThe 3D SMLM challenge is an ongoing competition to assess SMLM software on both simulated and real reference datasets using objective assessment metrics. The 3D challenge remains continuously open to new submissions from both 2D and 3D SMLM software. The first round of the challenge was presented in the Special Session at SMLMS 2016.

Note: We are currently in the process of updating this page. The latest and most detailed description of methods is contained in the 3D SMLM challenge preprint on bioRxiv.
3. Image Formation Model (PSF)
4. Noise Model
7. Tool: Baseline Performances
8. Tool: Comparison Localization
The synthetic datasets were designed to be as similar as possible to images derived from cellular structures in real experimental conditions To achieve the high degree of realism, we defined mathematical models for biological structure that try to imitate microtubules and endoplasmic reticulum/mitochondria. These structure have a tubular shape in the 3D space. Typically, microtubules are defined with their central axis elongating in a 3D space having an average outer diameter of 25 nm with an inner, hollow tube of 15 nm diameter.
The underlying sample structure is formalized in a continuous space which allows rendering of digital images at any scale, from very high resolution (up to 1 nm/pixel) to low resolution (camera resolution: 100 nm). The continuousdomain 3D curve is represented by means of a polynomial spline. The sample is imaged in a limited field of view, i.e., less than 6.4 × 6.4 μm2, and the center lines of the microtubules have limited variation along the z (vertical) axis, i.e., less than 1.5 μm. The fluorescent markers are uniformly distributed over the structure according to the required density. The photon emission rate of each fluorophore is controlled by a photoactivation model (see below).
The exact locations of all fluorophores are therefore stored at high precision, as floating point numbers expressed in nanometers. This groundtruth file is useful for conducting objective evaluations without human bias.
Reference: Sage et al. Quantitative evaluation of software packages for singlemolecule localization microscopy, Nature Methods 2015.
Flux of photons
The flux of photons is given by the relation: F = Φ . P . σ / e in photons/seconds
Φ is the quantum yield og the dye
P is power of the laser (not spatially uniform) in W/cm^{2}
e = h . c / λ is the energy of 1 photon is power of the laser (it is not spatially uniform)
σ = 1000 . ln(10) . ε / N_{A} is the absorption cross section in cm^{2}
ε is the molar extinction coefficient (EC) or absorptivity in cm^{2}/mol
4states photophysics model
Given a list of source locations from the structure simulator, fluorophore blinking was modelled by a 4 state process.
The Off to Om transitions have an uniform random distribution. It reflects normal experimental conditions where constant imaging density is maintained by tuning the photoactivation rate during the experiment. All other transitions are Poisson distributed. All switching is calculated at subframe resolution and then total fluorophore ontime was integrated over each frame.
The actual mean lifetime in On state is 1/(1/Ton + 1/Tbleach) due to two decay paths. Switching rates were chosen as "fluorescentproteinlike". For the training datasets: Ton = 3; Tdark = 2.5; Tbl=1.5; (unit is frames)
Fractional fluorophore ontimes per frame (between 0 and 1) were then multiplied by the mean photon emission. At the end of this process a list of XY positions, onframes and (noisefree) intensities for all activated fluorophores was obtained.The Matlab code that implements this photoactivation model is available at:
https://github.com/SMLMChallenge/Challenge2016
In order to achieve as realistic a simulation as possible, model point spread functions were derived from experimentally measured PSFs. For each modality, images of fluorescent beads were recorded the conditions below. Signal to noise of recorded PSFs was maximised in all cases by maximizing exposure time and averaging over several frames to increase dynamic range.
Sample (all modalities)
PSF modality  10x10x10nm 3D TIFF file 
Optics  Camera  Pixelsize at sample plane  Zstack step size  
2D  2D PSF  2DExp † (127 Mb) 
Nikon NA 1.49 TIRF oil objective on Nikon NSTORM commercial microscope  Andor iXon EMCCD 
43 nm (1.5x microscope and 2.5x SIM magnifiers in place)  10 nm, 3 μm Zrange 
AS  Astigmatic PSF  ASExp † (111 Mb) 
Nikon NA 1.49 TIRF oil objective  EMCCD 160μm pixel size 
160 nm  10 nm, 3 μm Zrange 
DH  DoubleHelix PSF  DHExp † (150 Mb) 
Nikon NA 1.49 TIRF oil objective  EMCCD 
160 nm  Z20 nm, 3 μm Zrange 
BP  Biplane PSF  BP250 † BP+250 † (2x127 Mb) 
The biplane model PSF was constructed from the 2D PSF data – see below. 
At the stage of the challenge, the PSF are not yet making publicly available at 10x10x10nm resolution. The PSF can be estimated from the beads zstack which have a 100x100x10nm resolution. The 10x10x10nm PSFs will be open later on.
Model PSF construction
This yielded 3 high SNR model PSFs (2D, astigmatism and doublehelix) with a voxel size of 10x10x10 nm^{3}. A central Zrange of 1.5 μm was selected. The biplane PSF was constructed by duplicating the 2D PSF and offsetting it by 250 nm and 250 nm for each Zplane. For the DHPSF, the transmission of the combined phase mask/ 4f system was measured as 96 %, which was approximated as 100 % brightness relative to the 2DSPF and ASPSF. The ground truth XY=0 was defined as the image centre of mass of the infocus frame of the model PSF, and Z=0 was defined as the infocus frame. Accounts for shifts in the fitted XY centre of the model PSF by localization software due to systematic offsets and Zdependent variation of the model PSF centre of mass are dealt the wobble correction.
The Matlab code of the PSF generation is available at:
https://github.com/SMLMChallenge/Challenge2016
// Pseudocode noise model for LM challenge 2016. // This assumes all input light is fluorescence // (background, signal) and thus follows poisson statistics // The camera is an EMCCD, specifically the // Photometrics Evolve Delta 512 for each pixel n_photIn is the input electrons // n_ie, apply noise model // poisson noise including shot noise and spurious // charge plus binomial quantum efficiency conversion // is just a Poisson distribution as per Hirsch. n_ie = poisson(QE*n_photIn + c); // emccd model, shape param k=n_ie, // scale param theta=EMgain // after Basden et al Mon Not R Astron Soc 2003 n_oe = gammadistribution(n_ie,EMgain); n_oe = n_oe + gaussian(read_noise); // analog to digital converter ADU_out = int(n_oe/e_per_adu)+ baseline; //restrict to 16 bit ADU_out = min(ADU_out,65535) end
A constant mean autofluorescent background was added to the noisefree simulated images, and these images were then fed through the noise model representing Poisson distributed fluorescence emission recorded on a high quantum efficiency backilluminated EMCCD.
Reference: model is inspired by Hirsch et al., PLoS One 2015.
Quantum Efficiency  QE = 0.9  Evolve quantum efficiency @700 nm) 
Readout noise  read_noise = 74.4  Manufacturer measured rms electrons for Evolve 
Spurious charge  c = 0.0002  Manufacturer quoted spurious charge (CIC only, dark counts negligible) for Evolve 
EM Gain Register  EMgain = 300  Manufacturer quoted spurious charge (CIC only, dark counts negligible) for Evolve 
Baseline  baseline = 100  Typical value 
Electron per analogtodigital units  e_per_adu = 45  ADC conversion factor, arbitary value similar to typical 
Total system gain  gain = 6  0.9 * 300 / 45 = 6 
The Java code of the simulator is available at:
https://github.com/SMLMChallenge/Challenge2016
Comparison of 3D localization to ground truth for experimentally derived model PSFs requires correction of of depthdependent lateral distortion, here called wobble. This is due to arbitrary systematic offset since definition of PSF centre is arbitrary. It was mentioned in several sources but discussed most fully here: Carlini et al., Correction of a DepthDependent Lateral Distortion in 3D SuperResolution Imaging, PLOS One, 2015.
In order to correct for this, competitors should either:
Provide the localizations file for the simulated beads data, and we will automatically correct the wobble/ offset using the tool discussed below.
Calculate and upload the wobble/offset using the tools provided. This also allows competitors to accurately compute their performance for the training data. The following code allows automatic calculation of the wobble/offset compared to ground truth data. For the competition, this should be performed for the simulated zstack of beads for the PSF modality to be submitted (including 2D). The code generates a CSV file containing a list of x,y offset corrections as a function of Z, from 750 to 750 nm. This can be used by CompareLocalization Tool to calculate corrected performance metrics.
A typical plot of wobble as a function of Z is shown here. As can be seen, a small but significant (~50nm) Zdependent offset is observed on both X and Yaxes for Zvalues <400 nm. At extreme Zvalues, the wobble correction, ie the shift in detected PSF centroid shifts dramatically. This essentially indicates that beyond this limit fitting to the (extremely defocussed) PSF has failed. These large correction values at extreme Z are generally nothing to worry about as fitting to the PSF within this range has failed  ie spots outside this range will not be detected with meaningful accuracy  so the correction value is not relevant.
If the wobble/ offset stuff is confusing – please contact us. In any case for submitted data, there is the option for us to automatically calculate the correction based on the uploaded bead localizations, without the need to worry.
Matlab code and MSWindows runtime to perform the wobble correction
wobblewobble_files.zip
Example of wobble correction, results and figures
wobble_correction_example.zip
Example of wobble file
wobbleexample.csv
Objectives: This tool is a minimalist SMLM software that performs localizations of bright emitters on the 4 modalities of the challenge 2016: 2D, 3DAstigmatism, 3DDoubleHelix, and 3DBiplane. This SMLM_BaselineLocalization software is only designed to establish the performance baseline for the SMLM challenge.
It is fast and very inaccurate! It has intentionally limited lines of code and relies only on two threshold parameters.
3D Calibration: there is a very simple calibration tool that has to run on a zstack of beads to find the linear relation between the axial position Z and the shape of the bead.
Distribution: The opensource software SMLM_BaselineLocalization is released as a Java ImageJ plugin.
Download the plugin: SMLM_BaselineLocalization.jar (Put this jar file into the plugin folder of ImageJ or Fiji)
Download the Java source code: SMLM_BaselineLocalizationsrc.zip
See a screenshot: screenshot.png
Objectives: This is a piece of software to compute the rate of detection and the accuracy in 3D based on 2 sets of localization (x, y, z).
Wobble correction: 3D localization requires correction of depthdependent lateral distortion, called wobble, see above.
Assessment: CompareLocalization3D performs 8 assessments:
The opensource software CompareLocalization3D is released as a standalone Java application.
Download the jar file: CompareLocalization3D.jar (Doubleclick on jar file to start the application)
Download the Java source code: SMLM_CompareLocalization3Dsrc.zip
See a screenshot: screenshot.png
© 2018
Biomedical Imaging Group, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
Last update: 30 Nov 2018