Support by the official sponsors of
the Single-Molecule Localization Microscopy Symposium

Single-Molecule Localization Microscopy  •  Software Benchmarking

The Challenge 2013 is turned to an online permanent challenge

The Grand Challenge Localization Microscopy was organized as Workshop in the conference IEEE International Symposium on Biomedical Imaging. Nearly 30 software have been run, mostly by the authors of the software, thus constituting the first world-wide effort in benchmarking the localization software in a comprehensive review. The results are now published.

The 2013 ISBI challenge is now re-open as a online permanent challenge and a new SMLM challenge will be held in 2016.

banner

Be Involved

Goals of the challenge

The goal of this challenge is to benchmark currently available localization software. In particular, comparing their performance in terms of detection rate, localization accuracy, and image rendering. Accessibility, usability for the end-user, and computational time will be evaluated, too. The benchmark data consists of simulated images of various biological structure such as tubulins. The simulated data accounts for fluorophores activation and excitation, image formation, and known perturbations models. As the ground truth exact positions of the fluorophores are known, the benchmarking metrics will use use objective measures.

  • Provide reference datasets
  • Establish of a metrics
  • Gather information about the available software
  • Identify of the new directions

Pre-registration is now permanently open!

Access to the datasets

Pre-registration gives access to the ground-truth of the training datasets and to the frame sequence of the contest datasets.

Open to everybody

The challenge is open to all individuals or teams, academic or corporate, existing or newly developed localization microscopy software. Participation is free.

How to register?

To announce your participation, just send an email to Daniel Sage indicating the name of the software that you want to represent or to test and the name of a contact person.

Account

Pre-registration will create your own account to submit your localization results.

Committee • Challenge 2013

Daniel Sage

daniel.sage@epfl.ch

Biomedical Imaging Group (BIG)
Ecole Polytechnique Fédérale de Lausanne (EPFL)

Hagai Kirshner

hagai.kirshner@epfl.ch

Biomedical Imaging Group (BIG)
Ecole Polytechnique Fédérale de Lausanne (EPFL)

Thomas Pengo

Center for Genomic Regulation
Barcelona, Spain

Nico Stuurmann

Vale Lab
University of California, San Francisco (UCSF)

Junhong Min

Bio Imaging Signal Processing
Korea Advanced Institute of Sci. and Tech. (KAIST)

Suliana Manley

Laboratory of Experimental Biophysics (LEB)
Ecole Polytechnique Fédérale de Lausanne (EPFL)

Industrial Sponsors • Challenge 2013

Hamamatsu, Camera Products

www.hamamatsucameras.com

Mad City Labs

www.madcitylabs.com

International Institutions • Challenge 2013

Euro-BioImaging

Euro‐BioImaging is a large‐scale pan‐European research infrastructure project on the European Strategy Forum on Research Infrastructures (ESFRI) Roadmap. www.eurobioimaging.eu

Open Bio Image Alliance

www.openbioimage.org

ISBI Workshop

April, 8, 2013, San Francisco
Presentation Speaker
IntroductionDaniel Sage
PSF models Hagai Kirshner
Photon quantification Nico Stuurman
Datasets Daniel Sage
Rendering Thomas Pengo
File format Nico Stuurman
Assessment Daniel Sage
Software presentation: B-recsHerveé Raoult
Software presentation: FacePALM Eelco Hoogendoorn
Software presentation: FALCON Junhong Min
Software presentation: Fast-ML-HD Kyungsang Kim
Software presentation: FPGA Manfred Kirchgessner
Software presentation: GPUgaussMLE Keith Lidke
Software presentation: Insight3Ryan McGorty
Software presentation: MicroManager LM Nico Stuurman
Software presentation: QuickPALM Christophe Zimmer
Software presentation: RadialSymmetry Raghuveer Parthasarathy
Software presentation: simpleSTORM Luca Fiaschi
Software presentation: SOSpluginIhor Smal
Software presentation: ThunderSTORMPavel Krizek
Software presentation: Wavelet FluoroBrancroft Trevor Ashley
Additional software (a-livePALM, DOASTORM, PeakFit, RapidSTORM)Thomas Pengo
Results - Real datasets Junhong Min
Results - Localization Thomas Pengo
Round-table, discussions Everybody

Methods

Realistic simulation of datasets

The datasets simulate biological samples that are imaged by a microscope with an EMCCD camera. Different structures have been generated to produce many experimental scenarios that vary from ideal acquisition scenarios to highly perturbed scenarios. Two categories have been proposed:

  • LS: Long sequence, few actived fluorophores per frame
  • HD: High-density of actived fluorophores per frame over short sequence

Sample factory: Biological sample are composed of extruded tubes that simulate microtubules (around diameter 25 nm). The tubes are defined as 3D geometric shapes in the continuous-spatial domain, allowing for arbitrarily high precision of the ground-truth data. For instance, the central axis of the tubes is parameterized by a set of cubic spline knots.

Fluorophores factory: The sample compounds are decorated with hundreds of thousands of fluorophores according to a specified density for each compound. Every fluorophore has an independent photoactivation behavior: it can be switched on at random times and emit a random quantity of photons according to a lifetime model (constant emission, exponential-decay emission). The fluorophore positions are stored as double precision lists.

Frame Sequence factory: The whole acquisition chain is simulated. Starting at a given time and for a specified frame rate, it computes thousands of synthetic images containing only the active fluorophores. The computation is performed at high resolution (5 nm/pixel) then the images are down-sampled to a camera resolution (100 nm/pixel or 150 nm/pixel). The simulation includes the following acquisition perturbations:

  • PSF: 3D optical PSF using the Gibson and Lanni model or a z-defocussed 2D-Gaussian.
  • Noise: Shot noise, read-out noise, EM-CCD noise, hot pixels and dark pixels.
  • Background: Dark current, auto-fluorescence sources, spatial inhomogeneity.
  • Camera: Resolution, gain, saturation, quantization, file-format (uncompressed 16-bits TIFF).

For the challenge, the number of pertubations are limited to those that directly related to the localization procedure: 1) 2D localization, recover the lateral positions (XY), 2) flat sample (less than 1 μm), 3) no drift 4) a single channel, 5) low density of fluorophores or high density of fluorophores per volume unit.

Metrics of assessment

Towards standardization of the file format Interactive Creation of the XML Description File

The results of the localization are given in a delimiter-separated values text file. Every localization position of each frame is stored as row in this file. The description file is a XML that helps to decode the localization file format and it gives the spatial reference allowing comparaison.

Required information for each row

  • Mandatory - Frame number starting from 1
  • Mandatory - Position in X axis (in nm)
  • Mandatory - Position in Y axis (in nm)
  • Optional - Position in Z axis (in nm)
  • Optional - Measured intensity
  • Optional - Confidence in the measurement (%)
Description XML File
Organization
Separator of columns Required
Unit for the XY position default in nm
Row of the first localization
 
Column index • Important note: column index starts at 0
Column of the frame Required
Column for the X position Required
Column for the Y position Required
Column for the Z position Optional
Column for the intensity Optional
Column for the confidence (%) Optional
 
Shift of the origin • Check the figure
X shift default value: 0
Y shift default value: 0
Z shift default value: 0
Unit shift default in pixel
 
Shift in the numbering of the frames • Convention: 1 for the first frame
Frame shift default value: 0

Predefined settings

Rendering

PALM-siever • Visualization for Single-Molecule Localization Microscopy

palm-siever

For this challenge, we use PALM-siever as tool to render the reconstruction image from the localization results. PALM-siever is hosted by Google Project.

The PALM-siever platform covers both visualization and analysis of single-molecule localization microscopy data. Built on MATLAB, it enables data to be modified and displayed interactively.

Features

  • histogram, smoothed histogram, scatter, delaunay, color-coded 3D
  • smart import of tabular data
  • interactive slicing with any variable
  • plugins
  • palm-siever

Results

Comparative results

Publication of the results in the Nature Methods article

D. Sage, H. Kirshner, T. Pengo, N. Stuurman, J. Min, S. Manley & M. Unser, Quantitative evaluation of software packages for single-molecule localization microscopy, Nature Methods 12, 717–724 (2015).

Read full PDF
Results of a software

The detailled results of the evaluation for a specific software has been communicated to the the participants for every submitted localization files corresponding to a dataset. It is included the full statistical analysis, some rendering images, some comparative rendering image (tested software localization vs ground-truth), and some cross-section profiles.

Example of comparitive close-ups

Rendering images by sum up of 2D Gaussian functions. The composite color image cobimes red channel (Ground-truth localizations) and the green channel (tested software localizations).

Scale bar: 4400 nm
Rendering pixelsize: 44 nm/pixel
Scale bar: 400 nm
Rendering pixelsize: 4 nm/pixel
Scale bar: 100 nm
Rendering pixelsize: 1 nm/pixel

Example of cross-section

Rendering images by sum up of 2D Gaussian functions. The composite color image cobimes red channel (Ground-truth localizations) and the green channel (tested software localizations).

Detection answer in function of the local SNR

Java tool for evaluation

CompareLocalization: Java application

A Java application to compare two sets of localization is given at CompareLocalization.jar. The positions of the molecules have to be stored in a column-separated text file, as CSV file. Information to parse this file has to provided in a XML description file or provided using the graphical user interface. A tool to create the XML file available

Examples

This application can run on any machine with at least Java 1.5 installed.

Java code to compute the Jaccard index, the F-Score index, and the RMSD

	
	

© 2017 Biomedical Imaging Group, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
Last update: 31 Mar 2017