Single-Molecule Localization Microscopy  •  Software Benchmarking

3D SMLM Software Challenge

The 3D SMLM challenge is an ongoing competition to assess SMLM software on both simulated and real reference datasets using objective assessment metrics.

The 3D challenge remains continuously open to new submissions from both 2D and 3D SMLM software.

The first round of the challenge was presented in the Special Session at SMLMS 2016.

banner

Real Experiments • 3D Astigmatism

The experimental datasets consist of sequences of frames from real stained biologicial samples. A Z-stack of beads is provided for the axial calibration.

Tubulin-A647-3D

3D astigmatic image of microtubules imaged using dSTORM with a cylindrical lens (112'683 frames).
Microtubules in U-2 OS cells, labeled with anti-alpha tubulin primary and Alexa Fluor 647-coupled secondary antibodies.
Calibration data for the tubulin dataset: fluorescent beads adsorbed to a glass coverslide in water are stepped in Z at high SNR.

Courtesy of Jonas Ries (EMBL)

Reference: Li et al., Nature Methods 15, 2018. (raw data of Figure S6)

NPC-A647-3D

3D astigmatic dSTORM image of nuclear pore complex.
Nup107-SNAP-BG-AF647 in U-2 OS cells, imaged using dSTORM with a cylindrical lens (150'000 frames).
Calibration data for the dataset: fluorescent beads adsorbed to a glass coverslide in water are stepped in Z at high SNR.

Courtesy of Jonas Ries (EMBL)

Reference: Li et al., Nature Methods 15, 2018. (raw data of Figure S9)

Calibration • Z-stack of beads

2D • AS • DH • BP

Published June 7, 2016.

banner

Experimental conditions
  • z-stack of 6 beads in 2D, 3D-Astigmatism, 3D-Double-Helix, 3D-Biplane
  • Z-step = 10nm, Z-range = -750nm to 750nm, Z-focal plane at 0nm (151 slices)

Download page

Ground-truth of the position of the beads

Preliminary Results

Realistic Simulations • 3D • Experimental PSF

The experimental datasets consist of sequences of frames from real stained biologicial samples. A Z-stack of beads is provided for the axial calibration. The same 3D simulated structure is imaged in different experimental conditions and for different modality: 2D, 3D astigmatism (AS), 3D biplane (BP), and 3D double-helix (DH). As a testing phase, we have also the quad-plane (QP) datasets for high-density conditions. The training datasets are provided with ground-truth localizations.

picture1
Microtubules • Contest datasets

Published June 7, 2016 as training dataset for the SMLM challenge 2016

Three crossing microtubules in a field of view of 6.4 x 6.4 x 1.5 μm

Two experimental imaging conditions:

N1: Typical photon counts and background levels for Alexa647 labelled STORM sample

N2: Photoswitchable fluorescent protein labelled sample such as mEos2 or Dendra2

MT1.N1.LD
Low density Molecule density: 0.2 High SNR (N1) 19'996 frames AS • DH • BP
MT2.N1.HD
High density Molecule density: 2 High SNR (N1) 3'125 frames AS • DH • BP • QP
MT3.N2.LD
Low density Molecule density: 0.2 Low SNR (N2) 20'000 frames 2D • AS • DH • BP
MT4.N2.HD
High density Low SNR (N2) 3'020 frames 2D • AS • DH • BP • QP

picture1
Endoplasmic Reticulum • Contest datasets

Published June 7, 2016 as training dataset for the SMLM challenge 2016

Simulation of cellular organelle (endoplasmic reticulum/mitochondria) inspired structure in the field of view of 6.4 x 6.4 x 0.7 μm

One experimental imaging conditions:

N3: Typical photon counts and level of background for site specific dye-labelled live cell STORM sample such as ER Tracker

ER1.N3.LD
Low density Molecule density: 0.2 Very low SNR (N3) 19'620 frames 2D
ER2.N3.HD
High density Molecule density: 5 Very low SNR (N3) 3'020 frames 2D

picture1
Microtubules • Training datasets

Published June 7, 2016 as training dataset for the SMLM challenge 2016

Three crossing microtubules in a field of view of 6.4 x 6.4 x 1.5 μm

Two experimental imaging conditions:

N1: Typical photon counts and background levels for Alexa647 labelled STORM sample

N2: Photoswitchable fluorescent protein labelled sample such as mEos2 or Dendra2

Ground-truth
localizations
MT0.N1.LD
Low density Molecule density: 0.2 High SNR (N1) 2DASBPDH
MT0.N1.HD
High density Molecule density: 2 High SNR (N1) 2DASBPDH
MT0.N2.LD
Low density Molecule density: 0.2 Low SNR (N2) 2DASBPDH
MT0.N2.HD
High density Molecule density: 2 Low SNR (N2) 2DASBPDH

3D Datasets x Modality

Structure Signal/Noise
Ratio
Density Name Release (last update) Access
Data
Access
Ground-thruth
2D 3D
Astigmatism
3D
Double-Helix
3D
Biplane
Calibration Z-stack
Beads Very high 6 beads Beads 25.10.2016 Open Open Yes Yes Yes Yes
Training Datasets
Microtubules High Low MT0.N1.LD 07.06.2016 Open Open Yes Yes Yes Yes
Microtubules High High MT0.N1.HD 07.06.2016 Open Open Yes Yes Yes Yes
Microtubules Low Low MT0.N2.LD 07.06.2016 Open Open Yes Yes Yes Yes
Microtubules Low High MT0.N2.HD 07.06.2016 Open Open Yes Yes Yes Yes
Contest Datasets
Microtubules High Low MT1.N1.LD 25.10.2016 Open No - Yes Yes Yes
Microtubules High High MT2.N1.HD 11.03.2017 Open No - Yes Yes Yes
Microtubules Low Low MT3.N2.LD 25.10.2016 Open No Yes Yes Yes Yes
Microtubules Low High MT4.N2.HD 25.10.2016 Open No Yes Yes Yes Yes
Endoplasmic reticulum Medium Low ER1.N3.LD 07.06.2018 Open No Yes - - -
Endoplasmic reticulum Medium High ER2.N3.HD 07.06.2016 Open No Yes - - -
Real Datasets
Microtubules High Low Tubulin-A647-3D 04.10.2018 Open Unknown - Yes - -
Nuclear pore complex High Low NPC-A647-3D 04.10.2018 Open Unknown - Yes - -

© 2018 Biomedical Imaging Group, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
Last update: 30 Nov 2018