Comprehensive assessment of SMLM software.
The single molecule localization microscopy software challenge is an ongoing competition to assess SMLM software on both simulated and real reference datasets using objective assessment metrics. The 3D challenge remains continuously open to new submissions from both 2D and 3D SMLM software.
This challenge is based on realistic simulated datasets for 4 popular SMLM modalities:
3D astigmatism, biplane, double-helix and 2D.
Simulations include a 4-states photophysics model and
Qualitative assessments against real 3D STORM test structures, microtubules and NPCs, are also performed.
The challenge remains continuously open to new submissions.
News and views of Nature Photonics: Seamus Holden and Daniel Sage Nature Photonics 10, 2016.Read full PDF
The Grand Challenge Localization Microscopy was presented in the conference IEEE ISBI 2013. Nearly 30 software have been run, mostly by the authors of the software, thus constituting the first world-wide effort in benchmarking the SMLM software. The results are published in Nat. Method 2015 (PDF).Read more
These challenges are the first attempt to foster the community to use the same common reference datasets to evaluate the performances of the localization software.
Collaboration through competition: Nature Methods Editorial, Nature Methods 11, 695 (2014).
Grand Challenge: These SMLM challenges are part of the Grand-Challenge in Biomedical Image Analysis.
NATURE METHODS | ANALYSIS
D. Sage, H. Kirshner, T. Pengo, N. Stuurman, J. Min, S. Manley & M. Unser, Quantitative evaluation of software packages for single-molecule localization microscopy, Nature Methods (2015). doi:10.1038/nmeth.3442
Abstract The quality of super-resolution images obtained by single-molecule localization microscopy (SMLM) depends largely on the software used to detect and accurately localize point sources. In this work, we focus on the computational aspects of super-resolution microscopy and present a comprehensive evaluation of localization software packages. Our philosophy is to evaluate each package as a whole, thus maintaining the integrity of the software. We prepared synthetic data that represent three-dimensional structures modeled after biological components, taking excitation parameters, noise sources, point-spread functions and pixelation into account. We then asked developers to run their software on our data; most responded favorably, allowing us to present a broad picture of the methods available. We evaluated their results using quantitative and user-interpretable criteria: detection rate, accuracy, quality of image reconstruction, resolution, software usability and computational resources. These metrics reflect the various tradeoffs of SMLM software packages and help users to choose the software that fits their needs.
Multiple updates to the challeng including assessment against real 3D STORM datasets, interactive graphing and ranking tools and more detailed software reports. Check it out here.
A data exploration system is on-line to analyse the results of the assessment across software x modalities x metrics. interactive leaderboard.
The first results are presented on the bioRxiv: Super-resolution fight club: A broad assessment of 2D & 3D single-molecule localization microscopy software
Thanh-an Pham has a talk at the SMLMS 2017 Symposium in London: Developments of the ongoing 3D SMLM software challenge.
Biomedical Imaging Group, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
Last update: 30 Nov 2018