3D SMLM Software Challenge
The 3D SMLM challenge is an ongoing competition to assess SMLM software on both simulated and real reference datasets using objective assessment metrics.
The 3D challenge remains continuously open to new submissions from both 2D and 3D SMLM software.
The first round of the challenge was presented in the Special Session at SMLMS 2016.
Note: We are currently in the process of updating this page. The latest and most detailed description of methods is contained in the 3D SMLM challenge preprint on bioRxiv.
3. Image Formation Model (PSF)
4. Noise Model
The synthetic datasets were designed to be as similar as possible to images derived from cellular structures in real experimental conditions To achieve the high degree of realism, we defined mathematical models for biological structure that try to imitate microtubules and endoplasmic reticulum/mitochondria. These structure have a tubular shape in the 3D space. Typically, microtubules are defined with their central axis elongating in a 3D space having an average outer diameter of 25 nm with an inner, hollow tube of 15 nm diameter.
The underlying sample structure is formalized in a continuous space which allows rendering of digital images at any scale, from very high resolution (up to 1 nm/pixel) to low resolution (camera resolution: 100 nm). The continuous-domain 3D curve is represented by means of a polynomial spline. The sample is imaged in a limited field of view, i.e., less than 6.4 × 6.4 μm2, and the center lines of the microtubules have limited variation along the z (vertical) axis, i.e., less than 1.5 μm. The fluorescent markers are uniformly distributed over the structure according to the required density. The photon emission rate of each fluorophore is controlled by a photo-activation model (see below).
The exact locations of all fluorophores are therefore stored at high precision, as floating point numbers expressed in nanometers. This ground-truth file is useful for conducting objective evaluations without human bias.
Reference: Sage et al. Quantitative evaluation of software packages for single-molecule localization microscopy, Nature Methods 2015.
Flux of photons
The flux of photons is given by the relation: F = Φ . P . σ / e in photons/seconds
Φ is the quantum yield og the dye
P is power of the laser (not spatially uniform) in W/cm2
e = h . c / λ is the energy of 1 photon is power of the laser (it is not spatially uniform)
σ = 1000 . ln(10) . ε / NA is the absorption cross section in cm2
ε is the molar extinction coefficient (EC) or absorptivity in cm2/mol
4-states photophysics model
Given a list of source locations from the structure simulator, fluorophore blinking was modelled by a 4 state process.
The Off to Om transitions have an uniform random distribution. It reflects normal experimental conditions where constant imaging density is maintained by tuning the photoactivation rate during the experiment. All other transitions are Poisson distributed. All switching is calculated at sub-frame resolution and then total fluorophore on-time was integrated over each frame.
The actual mean lifetime in On state is 1/(1/Ton + 1/Tbleach) due to two decay paths. Switching rates were chosen as "fluorescent-protein-like". For the training datasets: Ton = 3; Tdark = 2.5; Tbl=1.5; (unit is frames)Fractional fluorophore on-times per frame (between 0 and 1) were then multiplied by the mean photon emission. At the end of this process a list of XY positions, on-frames and (noise-free) intensities for all activated fluorophores was obtained.
The Matlab code that implements this photo-activation model is available at:
In order to achieve as realistic a simulation as possible, model point spread functions were derived from experimentally measured PSFs. For each modality, images of fluorescent beads were recorded the conditions below. Signal to noise of recorded PSFs was maximised in all cases by maximizing exposure time and averaging over several frames to increase dynamic range.
Sample (all modalities)
3D TIFF file
|Optics||Camera||Pixelsize at sample plane||Z-stack step size|
|2D||2D PSF||2D-Exp †
|Nikon NA 1.49 TIRF oil objective on Nikon N-STORM commercial microscope||Andor iXon EMCCD
||43 nm (1.5x microscope and 2.5x SIM magnifiers in place)||10 nm, 3 μm Z-range|
|AS||Astigmatic PSF||AS-Exp †
|Nikon NA 1.49 TIRF oil objective||EMCCD
160μm pixel size
|160 nm||10 nm, 3 μm Z-range|
|DH||Double-Helix PSF||DH-Exp †
|Nikon NA 1.49 TIRF oil objective||EMCCD
||160 nm||Z20 nm, 3 μm Z-range|
|BP||Biplane PSF||BP-250 †
|The biplane model PSF was constructed from the 2D PSF data – see below.|
At the stage of the challenge, the PSF are not yet making publicly available at 10x10x10nm resolution. The PSF can be estimated from the beads z-stack which have a 100x100x10nm resolution. The 10x10x10nm PSFs will be open later on.
Model PSF construction
This yielded 3 high SNR model PSFs (2D, astigmatism and double-helix) with a voxel size of 10x10x10 nm3. A central Z-range of 1.5 μm was selected. The biplane PSF was constructed by duplicating the 2D PSF and offsetting it by -250 nm and 250 nm for each Z-plane. For the DH-PSF, the transmission of the combined phase mask/ 4f system was measured as 96 %, which was approximated as 100 % brightness relative to the 2D-SPF and AS-PSF. The ground truth XY=0 was defined as the image centre of mass of the in-focus frame of the model PSF, and Z=0 was defined as the in-focus frame. Accounts for shifts in the fitted XY centre of the model PSF by localization software due to systematic offsets and Z-dependent variation of the model PSF centre of mass are dealt the wobble correction.
The Matlab code of the PSF generation is available at:
// Pseudo-code noise model for LM challenge 2016. // This assumes all input light is fluorescence // (background, signal) and thus follows poisson statistics // The camera is an EMCCD, specifically the // Photometrics Evolve Delta 512 for each pixel n_photIn is the input electrons // n_ie, apply noise model // poisson noise including shot noise and spurious // charge plus binomial quantum efficiency conversion // is just a Poisson distribution as per Hirsch. n_ie = poisson(QE*n_photIn + c); // emccd model, shape param k=n_ie, // scale param theta=EMgain // after Basden et al Mon Not R Astron Soc 2003 n_oe = gammadistribution(n_ie,EMgain); n_oe = n_oe + gaussian(read_noise); // analog to digital converter ADU_out = int(n_oe/e_per_adu)+ baseline; //restrict to 16 bit ADU_out = min(ADU_out,65535) end
A constant mean autofluorescent background was added to the noise-free simulated images, and these images were then fed through the noise model representing Poisson distributed fluorescence emission recorded on a high quantum efficiency back-illuminated EMCCD.
Reference: model is inspired by Hirsch et al., PLoS One 2015.
|Quantum Efficiency||QE = 0.9||Evolve quantum efficiency @700 nm)|
|Readout noise||read_noise = 74.4||Manufacturer measured rms electrons for Evolve|
|Spurious charge||c = 0.0002||Manufacturer quoted spurious charge (CIC only, dark counts negligible) for Evolve|
|EM Gain Register||EMgain = 300||Manufacturer quoted spurious charge (CIC only, dark counts negligible) for Evolve|
|Baseline||baseline = 100||Typical value|
|Electron per analog-to-digital units||e_per_adu = 45||ADC conversion factor, arbitary value similar to typical|
|Total system gain||gain = 6||0.9 * 300 / 45 = 6|
The Java code of the simulator is available at:
Comparison of 3D localization to ground truth for experimentally derived model PSFs requires correction of of depth-dependent lateral distortion, here called wobble. This is due to arbitrary systematic offset since definition of PSF centre is arbitrary. It was mentioned in several sources but discussed most fully here: Carlini et al., Correction of a Depth-Dependent Lateral Distortion in 3D Super-Resolution Imaging, PLOS One, 2015.
In order to correct for this, competitors should either:
Provide the localizations file for the simulated beads data, and we will automatically correct the wobble/ offset using the tool discussed below.
Calculate and upload the wobble/offset using the tools provided. This also allows competitors to accurately compute their performance for the training data. The following code allows automatic calculation of the wobble/offset compared to ground truth data. For the competition, this should be performed for the simulated z-stack of beads for the PSF modality to be submitted (including 2D). The code generates a CSV file containing a list of x,y offset corrections as a function of Z, from -750 to 750 nm. This can be used by CompareLocalization Tool to calculate corrected performance metrics.
A typical plot of wobble as a function of Z is shown here. As can be seen, a small but significant (~50nm) Z-dependent offset is observed on both X- and Y-axes for Z-values <400 nm. At extreme Z-values, the wobble correction, ie the shift in detected PSF centroid shifts dramatically. This essentially indicates that beyond this limit fitting to the (extremely defocussed) PSF has failed. These large correction values at extreme Z are generally nothing to worry about as fitting to the PSF within this range has failed - ie spots outside this range will not be detected with meaningful accuracy - so the correction value is not relevant.
If the wobble/ offset stuff is confusing – please contact us. In any case for submitted data, there is the option for us to automatically calculate the correction based on the uploaded bead localizations, without the need to worry.
Matlab code and MS-Windows runtime to perform the wobble correction
Example of wobble correction, results and figures
Example of wobble file
Objectives: This tool is a minimalist SMLM software that performs localizations of bright emitters on the 4 modalities of the challenge 2016: 2D, 3D-Astigmatism, 3D-Double-Helix, and 3D-Biplane. This SMLM_BaselineLocalization software is only designed to establish the performance baseline for the SMLM challenge.
It is fast and very inaccurate! It has intentionally limited lines of code and relies only on two threshold parameters.
3D Calibration: there is a very simple calibration tool that has to run on a z-stack of beads to find the linear relation between the axial position Z and the shape of the bead.
Distribution: The open-source software SMLM_BaselineLocalization is released as a Java ImageJ plugin.
Download the plugin: SMLM_BaselineLocalization.jar (Put this jar file into the plugin folder of ImageJ or Fiji)
Download the Java source code: SMLM_BaselineLocalization-src.zip
See a screenshot: screenshot.png
Objectives: This is a piece of software to compute the rate of detection and the accuracy in 3D based on 2 sets of localization (x, y, z).
Wobble correction: 3D localization requires correction of depth-dependent lateral distortion, called wobble, see above.
Assessment: CompareLocalization3D performs 8 assessments:
The open-source software CompareLocalization3D is released as a stand-alone Java application.
Biomedical Imaging Group, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
Last update: 30 Nov 2018