Journal of Microscopy, Vol. 00, Issue 0 2018, pp. 1–16
doi: 10.1111/jmi.12714
Received 19 October 2017; accepted 27 April 2018
A multi-emitter fitting algorithm for potential live cell super-resolution imaging over a wide range of molecular densities T . T A K E S H I M A ∗ , T . T A K A H A S H I ∗ , J . Y A M A S H I T A ∗ , Y . O K A D A †, ‡
∗ System Division, Hamamatsu Photonics K.K., Hamamatsu City, Japan
& S. WATANABE∗
†RIKEN Center for Biosystems Dynamics Research, Suita, Osaka, Japan ‡Department of Physics, Universal Biology Institute and International Research Center for Neurointelligence, University of Tokyo, Tokyo, Japan
Key words. Localization microscopy, multi-emitter fitting algorithm, super-resolution microscopy, template matching. Summary Multi-emitter fitting algorithms have been developed to improve the temporal resolution of single-molecule switching nanoscopy, but the molecular density range they can analyse is narrow and the computation required is intensive, significantly limiting their practical application. Here, we propose a computationally fast method, wedged template matching (WTM), an algorithm that uses a template matching technique to localise molecules at any overlapping molecular density from sparse to ultrahigh density with subdiffraction resolution. WTM achieves the localization of overlapping molecules at densities up to 600 molecules μm–2 with a high detection sensitivity and fast computational speed. WTM also shows localization precision comparable with that of DAOSTORM (an algorithm for high-density super-resolution microscopy), at densities up to 20 molecules μm–2 , and better than DAOSTORM at higher molecular densities. The application of WTM to a high-density biological sample image demonstrated that it resolved protein dynamics from live cell images with subdiffraction resolution and a temporal resolution of several hundred milliseconds or less through a significant reduction in the number of camera images required for a high-density reconstruction. WTM algorithm is a computationally fast, multiemitter fitting algorithm that can analyse over a wide range of molecular densities. The algorithm is available through the website. https://doi.org/10.17632/bf3z6xpn5j.1
Introduction The diffraction-limited resolution of a lens-based optical microscope is d = λ/ (2 NA) 200 nm, where d is the minimum spacing of two point emitters that can be individually resolved (Hell, 2009; Huang et al., 2010), λ is the wavelength of the light used, and NA is the numerical aperture of the objecCorrespondence to: Shigeo Watanabe, System Division, Hamamatsu Photonics K.K. 812 Joko-cho, Higashi-ku, Hamamatsu City 431–3196, Japan. Tel: +81-53-4310124; fax: +81-53-435-1574; e-mail:
[email protected] C 2018 The Authors. Journal of Microscopy
tive lens. Recent super-resolution single-molecule localization methods have overcome this limitation by switching fluorescent molecules on and off so that adjacent molecules within the diffraction limit can be separated (Betzig et al., 2006; Hess et al., 2006; Rust et al., 2006; Shroff et al., 2008). The main drawback of this method is that each fluorescent molecule within the diffraction spot needs to be switched on separately, which places a restrictive requirement on the experimental conditions: any molecules simultaneously emitting during a given frame must be adequately separated. Consequently, a lot of image frames are required to create the final super-resolution image, resulting in low temporal resolution. This prevents conventional sparse-emitter localization super-resolution methods from being widely applied to living cell imaging. In addition, the long exposure times and relatively high illumination intensities required for switching the fluorophores can result in phototoxicity and the need to address sample drift (Cox & Jones, 2013; Small & Stahlheber, 2014). One solution for these problems is to use a multi-emitter fitting algorithm, which identifies the coordinates of each molecule even when the molecules’ point spread functions (PSFs) overlap. Several methods have been developed for multi-emitter fitting including SSM BIC (Quan et al., 2011), DAOSTORM (Holden et al., 2011), PALMER (Wang et al., 2012), MLEsCMOS (Huang et al., 2013), L1-homotopy (Babcock et al., 2013), FALCON (Min et al., 2014), ADCG (Boyd et al., 2015), TVSTORM (Huang et al., 2017) and SRRF (Gustafsson et al., 2016). Compressed sensing (Zhu et al., 2012) and the 3B method (Cox et al., 2012) have also been developed for estimating fluorophore locations and super-resolution imaging. Although these multi-emitter fitting algorithms are powerful tools that reduce the number of frames needed, thereby increasing the temporal resolution for live cell super-resolution, they have limitations. With SSM BIC (Quan et al., 2011), the number of overlapping molecules that can be handled by the algorithm is severely limited. A common limitation is the computational time taken by the fitting algorithm, which is often too slow for widespread and convenient use, although
published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society.
This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.
2
T. TAKESHIMA ET AL.
Fig. 1. The concept of wedged template matching and flowchart of the WTM algorithm. (A) Schematic illustration of wedged templates for separating overlapping molecules. Top: Four examples of wedged templates. Middle: Illustrations of two overlapping molecules and the wedged area for the wedged template matching. Four different degrees of overlap are shown. Bottom: Side view of the two overlapping molecules in the respective middle image. (B) Schematic illustration of the steps in the WTM algorithm to identify individual molecules from clusters of overlapping molecules.
PALMER and L1-homotopy (Wang et al., 2012; Babcock et al., 2013) are exceptions. The computational infrastructure required for the current implementation of a specific fitting algorithm may be intensive. We have specifically addressed the problem of the long execution time of multi-emitter fitting algorithm by using a simple IT (information technology) infrastructure without a graphics processing unit (GPU); this nevertheless achieves the localization of a high number of molecules with good localization accuracy even with an ultrahigh density condition. In this paper, we introduce a computationally fast method for resolving overlapping molecules, at any density from sparse to ultrahigh, with subdiffraction limit resolution. To estimate molecular coordinates from overlapping PSFs with a reasonable execution speed, we developed an algorithm based on template matching, a processing technique used to find parts of a digital image that match a template image. Template matching has been used in other fields (Thomann et al., 2003; Brunelli, 2009), including astronomy, where it has been C 2018 The Authors. Journal of Microscopy
applied to resolve overlapping images of stars in constellations (Noordmans & Smeulders, 1998). Unlike straightforward template matching to fit single molecules, our algorithm uses a wedge-shape template to identify and localise molecules that overlap with others (Fig. 1). We therefore named our algorithm ‘wedged template matching’ (WTM). Like the single-molecule localization algorithm, the WTM algorithm is basically a localization algorithm; but this algorithm has the additional ability to resolve the biological structure even with overlapping PSFs from multiple molecules. The WTM algorithm is relatively simple; the computational speed is considerably faster than that of other multi-emitter fitting algorithms, even without a GPU, yet the algorithm yields comparable precision of localization at any overlapping molecular density from sparse to ultrahigh. These advantages of WTM could make it possible to analyse live cells with a high temporal resolution and a reasonable spatial resolution. The WTM algorithm software and instruction are made publicly available with the online version of this paper. published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
MULTI-EMITTER FITTING ALGORITHM FOR POTENTIAL LIVE CELL SUPER-RESOLUTION IMAGING
Material and methods Computer simulation image using a camera simulation engine To evaluate the performance of the WTM algorithm, we generated simulated images using the camera simulation engine we had previously developed (Fullerton et al., 2012). This generated statistically accurate camera images that incorporated photon shot noise in addition to camera noise, including readout noise, excess noise [for electron multiplying chargecoupled devices (EM-CCDs)], and pixel quantum efficiency, as well as pixel-to-pixel variations in these parameters. Images were simulated based on the pixel size and information about the sample and imaging optics. To generate the simulated images, the optical intensity on the camera was set according to the airy disk model with the Lommel function for 3D images and Bessel functions for focused images of the PSF of an individual fluorophore through an aberration-free 60× (NA 1.45) objective lens optics with a 1.2× tube lens in perfect focus. The diameter of the 1/e1/2 intensity point, σ , of a single-molecule PSF is close to 76 nm when fitted by a circular Gaussian function. To test the performance of the WTM algorithm in different scenarios, the camera simulation engine simulated several sets of images in which N molecules were randomly placed in the field of view (Fig. 2A). We used a 4 × 4 μm2 area to place the molecules within a 5.04 × 5.04 μm2 field of view. To avoid edge effects, a 3.7 × 3.7 μm2 area of the reconstructed images was analysed and the molecules positioned in a 3.5 × 3.5 μm2 area were used for the evaluation of the algorithm (3.2 × 3.2 μm2 and 3.0 × 3.0 μm2 areas for 150 nm and 300 nm defocused molecules, respectively). The object-referenced pixel size was 90 nm. Single molecules were positioned with a 1 nm resolution. The high-intensity molecule model used a log-normal distribution with a mean of 3000 (±1700) photons molecule–1 and a uniform optical background of 70 photons pixel–1 . The mid-intensity molecule model used 750 ± 460 photons molecule–1 with a background of 50 photons pixel–1 , and the low-intensity molecule model used 200 ± 77 photons molecule–1 with a background of 10 photons pixel–1 . The emission wavelength of the molecules was 540 nm. The simulations used the specification of the ORCA-Flash4.0 camera (quantum efficiency = 0.70 at 540 nm) and the measured distribution of pixel readout noise [median = 1.3 e− , rms (root mean square) = 1.9 e− ] and the shot noise was also implemented. The number of simulated images varied with the molecular density: 9600 images at 0.0625 molecules μm–2 , and then 300, 150, 100, 75, 60, 50, 38, 30, 10, 6, 2 and 1 images at 2, 4, 6, 8, 10, 12, 16, 20, 60, 100, 300 and 600 molecules μm–2 , respectively. To evaluate WTM with molecules of different intensities and assess the dependency of localization precision on the number of iterations of the WTM cycle, simulated images with a 10-fold greater number of molecules were prepared. One model with three different C 2018 The Authors. Journal of Microscopy
3
focuses, based on the airy disk, were used to evaluate how well the WTM algorithm recognized defocused molecules; these described a molecule located perfectly at the focus of the objective, or out of focus by 150 or 300 nm. Evaluation of the algorithm The WTM algorithm was evaluated in terms of the total number of localizations and the localization precision (Figs. 2B, C, D and 3A, B, C). The evaluation of localization precision was based on calculating the distances between the algorithm’s localization of each recognized molecule and the nearest true position of a molecule (within 100 nm). These distances were then summarised by two indices: the root mean square distance (RMSD) and a box plot derived from a histogram of these distances without the minimum and maximum values. Cramer-Rao lower bound (CRLB) calculation was used to demonstrate the statistical limit of localization precision (Mortensen et al., 2010; Fig. S3). We also compared the WTM algorithm with DAOSTORM from another point of view. The two graphs (Figs. S8A, B) show the false detection rate to detected molecules and the correlated detection rate to true molecules. A nearest neighbour analysis was performed at each molecular density for every true molecule position (Figs. 3D–E). For this, the distance between each true molecule position and the nearest recognized molecule position, and the distance between each true position and the nearest true position, were calculated and plotted against each other for the different molecular densities. The WTM algorithm could be considered as a deconvolution algorithm. Therefore the correlation between the convolved algorithm results with PSF, and camera resolution original images with the equation of SNR (signalto-noise) analysis used in algorithm contest paper (Sage et al., 2015) was calculated (Fig. S8C) . We also evaluated the execution time of the WTM algorithm on a laboratory computer with the specifications shown in Figure 3(G), comparing this with the execution time of DAOSTORM, another multi-emitter fitting algorithm, on the same computer. The parameters for each algorithm are shown in Table 1. Preparation of biological samples All reagents were purchased from FUJIFILM Wako Pure Chemical Corporation (Osaka, Japan), unless otherwise noted. Madin–Darby Canine Kidney (MDCK) II cells (Noda et al., 2001) were obtained from European Collection of Authenticated Cell Cultures (ECACC, Salisbury, UK) (0062107). MEFK cells were derived from primary mouse embryonic fibroblast cells by introducing the SV40 large T antigen for immortalisation (Ueno et al., 2011). Both cell lines were maintained in Dulbecco’s Modified Eagle Medium (044-29765) supplemented with 10% fetal bovine serum (FBS; Gibco, ThermoFisher Scientific, Waltham, MA, USA). The cells were plated to a cover
published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
4
T. TAKESHIMA ET AL.
Fig. 2. Evaluation of the WTM algorithm using computer simulation images. (A) Left: Simulation image for demonstrating the WTM algorithm’s capability of identifying molecules efficiently at a density of 6 molecules μm–2 . Scale bar 1 μm. Right: Image showing the true molecule positions (white circles) and those recognized by the WTM algorithm (blue circles). Intensity difference is indicated by circle size. (B) Intensity-weighted histograms of the distribution of the localised points relative to the true molecular positions at various molecule densities. The negative distance and positive distance represent the left and right half area from the true molecular position. (C) Comparison of the total number of molecules (without molecular intensity information) detected by the WTM algorithm and the M2LE single-molecule fitting algorithm with an ellipticity filter. (D), (E) Comparison of the localization precision of the WTM and M2LE algorithms using the root mean squared distance (RMSD) index and box plots without the first and fourth quartiles values. C 2018 The Authors. Journal of Microscopy
published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
MULTI-EMITTER FITTING ALGORITHM FOR POTENTIAL LIVE CELL SUPER-RESOLUTION IMAGING
5
Fig. 3. Comparison of the WTM algorithm to another multi-emitter fitting algorithm, DAOSTORM, using computer-simulated images. (A) Comparison of the efficiency of localization (the total number of recognized molecules). (B) Comparison of localization precision using the root mean squared distance (RMSD). (C) Comparison of localization precision using box plots without the first and fourth quartiles. (D) Schematic illustration of the nearest neighbour analysis. For each true position, the distances to the nearest recognized molecule position and the nearest neighbouring true position are measured at different molecular densities and plotted against each other. (E) The results of the nearest neighbour analysis. The colour scale shows the number of cases. The same log-scale look-up table was applied at each density, but not between the different densities. The area indicated by the red square is magnified in the lower graphs. The number in each image indicates the normalised detection rate of molecules. (F) Result images of the WTM and DAOSTORM analyses at different molecular densities, showing the true positions of the molecules (white circles), and the positions of molecules as identified by WTM (blue circles) and DAOSTORM (red circles). Intensity difference is shown by circle size. (G) Comparison of the execution times of WTM and DAOSTORM.
C 2018 The Authors. Journal of Microscopy
published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
6
T. TAKESHIMA ET AL.
Table 1. Parameters for each algorithm. WTM Area intensity 2283 1141 304 636 553 2327 4000 1000
Threshold
Sigma
Upsampling
Mask size
400 400 200 1000 2000 1000 6000 5000
0.876 0.876 0.876 2.2 2.2 2.72 1.4 3
15 15 15 15 15 15 5 15
5 5 5 5 5 5 5 5
For 3000 photons data For 750 photons data For 200 photons data For Figure 4(B) For Figures 4(C) and (D) For Figure 4(E) For Figure 5 For Figure 6
MLE Signal noise cutoff
Lowest noise estimate
Saturation point
Eccentricity threshold
Third moment threshold
Usable pixel
1 10
65 535 65 535
0.9 to 1 0.9 to 1
0.95 to 1 0.95 to 1
100% 100%
Intensity threshold
Width threshold
Maximum iterations
Max noise multiplier
Min noise bound
0.01% 0.01%
0.0001 0.0001
50 50
2 2
1 1
1.5 1.5 Position threshold 0.0001 0.0001 MaxWidth
MinWidth
10 10
1.2 1.2
For simulation image For Figure 4(B)
DAO-STORM SigmaBG
Thresh
FrameNo
Fwhmpsf
Datamin
Datamax
Gain
4 4
0 0
2.0 6.4
200a 1
10 000 14 000
N.A. 2.2
15 6
For simulation image For Figure 4(E)
Compressed sensing Eps 1.25
Div
Boxsize
Extramargin
Scanoverlap
Margin
8
5
1
2
4
a Datamin was changed depending on the intensity level of the target sample image.
glass (Matsunami, Osaka, Japan) coated with a thin layer of collagen (type I, Nitta Gelatin, Osaka, Japan). The cells were transfected with TransFectin (Bio-Rad, Hercules, CA, USA). The plasmids for the Golgi apparatus were the fusion of enhanced yellow fluorescent protein (EYFP) with the N-terminal 81 amino acids of human beta 1, 4-galactosyltransferase (Clontech, Takara Bio, Shiga, Japan). The plasmids for the microtubules were made by ligating the cDNA of mouse beta1tubulin or mouse EB1 to the backbone vector pEYFP-N1 (Clontech). The plasmids were purified with EndoFree Plasmid Maxi Kit (Qiagen, Venlo, Netherlands). The cells were observed for 16–24 h after transfection and then were rinsed with Hanks Balanced Salt Solution (HBSS+). The medium was replaced with HBSS+ supplemented with 10% FBS, 20 mM HEPES at pH 7.4 and GLOX (0.5 mg mL–1 glucose oxidase, 40 μg mL–1 catalase and 10% glucose). For im C 2018 The Authors. Journal of Microscopy
munostaining of the microtubules, the cells were stained with antitubulin antibody DM1A (Sigma-Aldrich, St. Louis, MO, USA) and DyLight 488-labelled anti-mouse IgG secondary antibody (ThermoFisher Scientific) after fixation with cold methanol at −20°C. The stained cells were mounted with phosphate buffered saline supplemented with GLOX and 100 mM cysteamine. Microscopy and data acquisition The cells were observed with an IX81 inverted microscope (Olympus, Tokyo, Japan) (Figs. 4–6). A 100× NA1.40 objective lens (UPLSAPO100× O,Olympus) was used with total internal reflection illumination optics (Olympus). The image was projected to a GenII scientific complementary metal oxide semiconductor (sCMOS) camera (C11440-22CU, published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
MULTI-EMITTER FITTING ALGORITHM FOR POTENTIAL LIVE CELL SUPER-RESOLUTION IMAGING
7
Fig. 4. Analysis of biological sample images using the WTM algorithm. (A) GSDIM activation images of enhanced yellow fluorescent proteins (EYFPs) tagged to the Golgi apparatus in MDCK cells. Four different timeframe images are presented. The image sequences were captured with stream acquisition at 30 ms intervals (30 ms exposure time) with a sCMOS camera (Hamamatsu Flash4.0). The contrast of each frame has been enhanced. (B) Analysis results for frames 1–100 and frames 1–9957 with WTM algorithm or the single maximum likelihood estimation (MLE) algorithm, M2LE, with or without an ellipticity filter. The numbers in each image refer to the numbers of molecules identified. (C) Example image of microtubules tagged with EYFPs in an MDCK cell, activated by GSDIM. The image sequences were captured with stream acquisition at 30 ms intervals (30 ms exposure time). Left: First frame of the GSDIM image sequence. Right: WTM reconstruction image from 100 frames. (D) Line profile graphs of the same region in both images shown in (C). (E) Comparison of WTM and DAOSTORM analysis of a biological sample. Top: Frame 1 and 100 of the sample sequence. Middle: The images reconstructed by the WTM and DAOSTORM analyses. Bottom: The line profile analysis results; the analysed line is shown in yellow in the top left image.
C 2018 The Authors. Journal of Microscopy
published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
8
T. TAKESHIMA ET AL.
Fig. 5. Analysis of live cell enhanced yellow fluorescent protein (EYFP)-EB1 protein dynamics using the WTM algorithm. (A) Top: Average intensity image from 1000 frames (238 × 160 pixels, 25.4 × 17.1 μm) of the GSDIM activation of EYFPs tagged to EB1 proteins in an MDCK cell. The image sequences were captured with stream acquisition at 11.4 ms exposure time. The area indicated by the red box was cropped and analysed by the WTM algorithm, as shown in (B). Bottom: Projected intensity image of the WTM results. Scale bar, 5 μm. (B) Time-lapse image sequences of WTM results. Each image was reconstructed from 25 raw frames. The temporal resolution of the WTM results was 285 ms. Scale bar, 1 μm. (C) Top: Average intensity image from 1000 frames of the original images (43 × 30 pixels, 4.5 × 3.2 μm). Middle: Projected intensity images of WTM results. Bottom: The WTM results are colored according to their recording time. Scale bar, 1 μm. (D) Top: Projected intensity images of the WTM results, which included two analysed lines. Bottom: Two examples of line profile analyses against time. The location of the EB1 protein along the yellow and blue line area in the top image plotted against time, showing EB1 translocation along the microtubule.
ORCA-Flash4.0, Hamamatsu Photonics K.K., Hamamatsu, Japan) at a full 4 megapixel frame size at a rate of 30–50 Hz. For the EB1 data (Fig. 5), an EM-CCD camera (C9100-13, ImagEM, Hamamatsu Photonics K.K., Hamamatsu, Japan) was used with a 1.5× magnification lens. For the excitation of molecules, an activation laser (Sapphire, Coherent, Santa Clara, CA, USA) was used at power levels in the range 20– 50 mW measured at the back focal plane of the objective lens, corresponding to 800–2000 W cm–2 at the sample plane. The
C 2018 The Authors. Journal of Microscopy
activation laser intensity was adjusted so that a high density of EYFP molecules was activated in each camera frame. We captured image sequences using the ground state depletion microscopy followed by individual molecule return (GSDIM) excitation procedure with total internal reflection fluorescence microscopy. GSDIM pumps the molecular state of the fluorescence protein into a triplet and dark state, resulting in individual molecules being sparsely activated in the presence of an oxygen scavenger (Folling et al., 2008). All of the
published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
MULTI-EMITTER FITTING ALGORITHM FOR POTENTIAL LIVE CELL SUPER-RESOLUTION IMAGING
9
Fig. 6. Analysis of the microtubule structure image captured by a sCMOS camera using the WTM algorithm. An example of an analysis of a 4 megapixel image by WTM. DyLight 488 tagged to antibodies for tubulin in an MEFK cell was activated using GSDIM. Image sequences were captured with stream acquisition at 50 ms intervals (50 ms exposure time). (A) A 2048 × 2048 pixel image of the original raw data (left) and the WTM analysis result calculated from 100 frames (right). (B) Magnified images of the WTM analysis results. Top: Magnified image (512 × 512 pixels, left) and WTM analysis result (right) of the yellow box in the full-size image (A). Bottom: Magnified image (100 × 100, left) and WTM analysis result (right) from the blue box in the 512 × 512 image. (C) Line profile of the blue and green region of interest at the bottom image of (B).
image acquisition was performed using HCImage Live software (Hamamatsu Photonics K.K., Hamamatsu, Japan). The microscopy system and peripheral equipment were controlled by Metamorph software (Molecular Devices, San Jose, CA, USA). Statistical analysis Mann–Whitney U-test was used to compare the precision of the WTM and DAOSTORM algorithms (Fig. 3B). At each molecular density the distances between the algorithm’s localization of each recognized molecule and the nearest true position of a molecule within 100 nm are calculated. These distance values from WTM and DAOSTORM are used for Mann– Whitney U-test. From 0.0625 to 60 molecules μm–2 , there were significant differences between WTM and DAOSTORM at significance level of 0.05. Dynamic sample analysis with WTM algorithm An image sequence of a dynamic sample was analysed with the WTM algorithm. The GSDIM experimental procedure C 2018 The Authors. Journal of Microscopy
was used to capture images of EYFP-tagged EB1 proteins in living MDCK cells (Fig. 5A). The GSDIM images were captured every 11.4 ms with an exposure time, which was also 11.4 ms, using an EM-CCD camera (electron multiplying gain, 200). From these image sequences, 1000 frames with relatively high molecular densities were extracted and analysed with the WTM algorithm applied to every raw frame. Groups of 25 analysed result images were then used to reconstruct super-resolution images; the temporal resolution of these images was 285 ms (i.e. 11.4 × 25 ms) (Fig. 5B).
Reconstructing a nanoscopy image from large files To test the WTM algorithm ability to analyse large files, we demonstrated the analysis of 2048 × 2048 pixel 100 frame images of microtubule structures tagged with DyLight 488 (Fig. 6). Each frame of 2048 × 2048 pixel images was analysed using WTM algorithm. Then the final nanoscopy image was reconstructed from those results.
published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
10
T. TAKESHIMA ET AL.
Real-time analysis using different numbers of CPU threads To test the algorithm’s real-time capability, we assessed whether it could analyse an image during image acquisition without dropping any image frames. We tested this with different numbers of assigned central processing unit (CPU) threads (Fig. S7). Results The concept behind the WTM algorithm The concept of WTM is summarised in Figure 1(A). Like singlemolecule localization nanoscopy, WTM is a deconvolution method that uses a template for single-molecule PSFs and attempts to make the best estimate of the positions of individual molecules, thereby creating the final image. Unlike conventional template matching methods, where the template used for matching is the same as the entire object to be recognized, WTM is based on a variation of feature-based template matching; this uses a partial image template (the wedged template, Fig. 1A, top) for the recognition and localization of a molecule, and then uses the full template to remove the detected molecule from the molecule cluster. The essence of the WTM algorithm is to match the intensity in a segment of the model function of the emitters in an overlapped molecule cluster with the wedgeshaped template (Fig. 1A). Even when there is severe overlapping of molecules, the intensity distribution of a segment of the model function will closely match the intensity distribution in a small area at the edge of the cluster, and this intensity information is used by the algorithm to identify an emitter, localise its centre position and then remove the whole single emitter model function from the cluster of overlapping molecules. Overview of the WTM algorithm process The steps taken by the WTM algorithm are summarised in Figure 1(B). The WTM algorithm consists of four steps: (1) preparation of the single-molecule model; (2) background subtraction; (3) WTM: identification of the pixel in which the centre of a molecule is located by fitting the wedged template at a coarse (camera pixel) level and then a fine (subcamera pixel) level and (4) removal of the identified molecule model from the clustered image. These four steps are iterated until all the molecules are identified. The WTM algorithm first needs two parameters for the single-molecule model: the 1/e1/2 intensity point, σ of the PSF and the total intensity of a single-molecule emission. Next, for each molecular identification and removal step, the WTM algorithm executes template matching in a coarse and a fine phase. In the coarse phase (camera-pixel WTM), the algorithm fits the image intensity to the template at a resolution of a camera pixel, to determine the camera pixel in which the centre of the emitter PSF is located; then, in the second phase (sub-pixel WTM), the emitter location is estimated at a sub C 2018 The Authors. Journal of Microscopy
pixel resolution. After the WTM algorithm localises each molecule in a cluster of molecules with overlapping PSFs, it subtracts that molecule from the cluster image using the whole single-molecule model with a single value of Gaussian spatial width σ and a total intensity. In this subtracting process, the WTM algorithm subtracts double the value of the input molecular intensity as a subtracting unit. The reason for this is that the WTM algorithm was first designed to use the peak intensity of molecular intensity distribution as the subtracting unit, but, during the evaluation of the algorithm, using double the peak intensity as the subtracting unit resulted in the highest number of recognized molecules without overcounting. Therefore, the subtracting unit was designed to be double the input intensity value. In this paper, we used the expected value of molecular intensity distribution as the subtracting unit for the evaluation of the WTM algorithm. This is because during actual applications, we found that when the input intensity was half the expected intensity of the molecular distribution, using the expected intensity as the subtraction unit, the WTM analysis showed a slight overcounting at low molecular densities but a higher number of recognized molecule than when the input intensity was peak intensity (Fig. S1). Considering that the WTM algorithm was intended for high-density molecule samples, overcounting at low molecule densities could be considered marginal. Instead, we preferred the higher number of recognized molecules. We therefore used the expected value of molecular intensity distribution as the subtracting unit for the evaluation of the WTM algorithm. The WTM algorithm treats brighter clusters as multiple molecules and repeats the molecular detection and subtraction processes until the location of every molecule is identified. This simple process results in visually relevant images; however, when combined with the background subtraction, it leads to a counting error in the estimate of the number of molecules, at least when compared to other single-molecule methods, and this also causes an estimation error of the total number of molecules due to blinking of the fluorescent molecules. Preparation of the single-molecule model The first step of the WTM algorithm is preparation of the singlemolecule PSF model. We used a Gaussian PSF with only two parameters, the diameter of PSF (i.e. the 1/e1/2 intensity, σ ) and the total intensity of a single-molecule emission. This simple single-molecule model is incorporated into the algorithm, enabling fast computation. The WTM algorithm should use the optimised size of template depending on the magnification of the microscope. Template size should cover most of the area of the PSF, which is about six-sigma area of PSF. Image processing to find the candidate pixels After the background subtraction (Figs. S2A–D), the WTM algorithm convolves the image with a convolution mask published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
MULTI-EMITTER FITTING ALGORITHM FOR POTENTIAL LIVE CELL SUPER-RESOLUTION IMAGING
(typically 5 × 5 pixels) and then uses the pixels that consist higher intensity compared to the defined intensity, as the candidate pixels for template matching. This process roughly sums the intensity of the area covered by the convolution mask and puts this summed value in the centre pixel. After applying the convolution mask, the pixels with an intensity above a defined threshold are marked as candidate pixels for fitting, that is as possible pixels in which the centre of a molecule may be located. This process substantially reduces the number of pixels over which the template matching needs to be applied, resulting in less computational time. Choice of the wedged template To identify a single molecule from a cluster of overlapping molecules, the WTM algorithm uses a wedged template, that is a wedged-shaped 2D Gaussian single-molecule model (Fig. S2E). In practice, the choice of the appropriate wedged template is important for identifying single molecules in the cluster because the area used for template matching varies according to the degree of molecular overlap (Fig. 1A, bottom). Usually this area diminishes as emitters overlap to a greater degree. The WTM algorithm prepares four different wedged templates for different matching areas (Fig. S2E), each with a different wedge angle (90°, 120°, 180° and 240°). The WTM algorithm chooses the appropriate wedged template according to the degree of molecular overlapping. The degree of molecular overlapping is estimated from the intensity of the candidate pixel in the molecular cluster. When the candidate pixel has an intensity >85% of the peak intensity of all the emitter clusters in the image, the WTM algorithm uses the 90° wedged template for fitting. Similarly, the algorithm uses the 120° wedged template for the intensity range 60%–85%, the 180° wedge for the intensity range 55%–60% and the 240° wedge for the intensity range 35%–55%. A template for the full PSF is used for intensity ranges less than 35% (Fig. S2E). Wedged template matching at the camera pixel level For each candidate pixel, the WTM algorithm first applies coarse WTM at the camera pixel level. The template is positioned so that the centre of the molecule in the template matches the candidate pixel. By rotating the wedged template through steps of 45° (up to three different directions in the case of a 5 × 5 pixel PSF model), the algorithm calculates the normalised cross correlation coefficient (RNCC ) at each angle as the ‘fitting value’, defined as follows:
where Iij is the intensity of each pixel after background subtraction, Tij is the intensity of each pixel of the template, i and j are pixel indices, and M and N are the dimensions (in pixels) of the region of interest, the ‘matching area’, containing the cluster. The dark level is the lowest value of the pixel intensity within the matching area after the background subtraction; it represents a local offset, for example, due to the tails of nearby emitters. The fitting value, RNCC, estimates the similarity between the image and the template. The highest fitting value across all angles θ is assigned as the fitting value of the candidate pixel. The pixel (x, y) with the highest fitting value (for any θ ) in any of the candidate pixels in the cluster is selected as the pixel of the emitter in the cluster. Wedged template matching at the sub-pixel level For the pixels selected through the camera pixel-level process, higher-precision template matching is applied to estimate the coordinates of the molecule at a sub-pixel resolution. Before applying sub-pixel WTM, the desired sub-pixel resolution is defined, such as 9 × 9 or 15 × 15 sub-pixels within a single camera pixel (corresponding to a 12-nm or 7.2-nm sub-pixel width for a 6.5 μm camera pixel with 60 × magnification). As with the camera pixel WTM process, the algorithm uses the wedged template with the most appropriate wedge angle for the intensity of the target camera pixel (Fig. S2E). The wedged templates for sub-pixel WTM are specially prepared. For each camera pixel fitting, the same number of wedged templates as sub-pixels are created, each with a differently located centre, and these centres are assigned to each target sub-pixel. Examples are shown in Figure S2(F). To increase the algorithm speed, the WTM algorithm restricts the fitting angle to only those regions with the lowest intensities instead of all directions, because in most cases, the native single emitters that do not overlap with other emitters are located in regions of lower intensity. The algorithm fits the wedged template with the angle of lowest intensity and angles of ±45° to this. The highest fitting value among all the tested angles becomes the fitting value of the sub-pixel. From a fitting value map of the sub-pixels in one camera pixel, the coordinates of the sub-pixel with the highest fitting value is defined as the location of the single emitter. After identifying the coordinates of a molecule, the WTM algorithm subtracts the single-molecule model, incorporating both the size of the PSF and its intensity, from the original image data. The molecular intensity for this subtraction is twice the input intensity. If this subtraction results in a negative value, the algorithm
N−1 M−1
j =0 i =0 {(Iij − dark level)(Tij(x, y, θ ))} RNCC (x, y, θ ) = , N−1 M−1 N−1 M−1 2 (Iij − dark level)2 j =0 i =0 j =0 i =0 Tij(x, y, θ )
C 2018 The Authors. Journal of Microscopy
11
published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
12
T. TAKESHIMA ET AL.
uses instead the pixel intensity value of the image to avoid the negative value as the intensity of the subtracting molecule. The algorithm repeats this process until no single-molecule remains (Fig. 1B).
Image correction Subtracting the single-molecule model from the cluster using the WTM algorithm described above leads to an accumulation of localization errors due to the iterative process of WTM, increasing localization errors during the later steps of subtraction. To compensate for these errors, the algorithm executes localization error correction after all the molecules have been identified, using the residual intensity information after subtracting the intensities of the identified single molecules. The reasoning for this is that if the localization of every single molecule were perfect, without any noise, the residual intensity distribution would be flat or zero; thus, in case residual intensity is not uniform, this indicates significant localization error. The WTM algorithm therefore identifies the molecules that affect the residual intensity distribution and corrects the locations of those molecules so that the residual intensity is as low and uniform as possible. Example images with and without image correction are shown (Fig. S9).
Presentation of the final image The output of the WTM algorithm is a list of coordinates of the individual molecules with the intensity of each molecule normalised to the intensity of the model. The final resolution of WTM is the grid (camera pixel size/up-sampling factor) without any interpolation. In the final image, the algorithm plots each molecule according to its coordinates, with normalised intensity. To improve the appearance, the WTM algorithm also has the capability to render each molecule with a Gaussian blur.
Evaluation of WTM using simulated images We evaluated the WTM algorithm by using computersimulated images, in which molecules were randomly distributed (Fig. 2A; see ‘Evaluation of the algorithm’ subsection in ‘Material and methods’ section). We used two criteria to evaluate the algorithm: (1) the total number of localizations and (2) the precision of localization of recognized molecules relative to their true molecular position. First, we compared the WTM algorithm to M2LE, a maximum likelihood estimation methods-based single (isolated) molecule fitting algorithm with elliptical rejection filter (Starr et al., 2012) (Figs. 2B–E). Based on the distances from the recognized molecules to the closest position of a true molecule located within 100 nm (Fig. 2B), the RMSD and box plot profile were calculated (Figs. 2D, E). C 2018 The Authors. Journal of Microscopy
At lower molecular densities, the parameters of WTM was set so that WTM recognized almost the same number of molecules (Fig. 2C). As the molecular density increased, the total number of localizations gradually decreased. However, M2LE with the elliptical rejection filter did not recognise most of the molecules. At the highest molecular density (20 molecules μm–2 ), the WTM algorithm was still able to localise 85% of the molecules (total numbers). At this density, the RMSD observed with WTM was 35 nm. Overall, these results suggested that WTM had the capability of identifying more molecules than the single-emitter fitting algorithm from high-density overlapping molecular images at a resolution beyond the diffraction limit. We also evaluated the WTM algorithm with mid- and low-intensity molecular models, as well as defocused, isotropic molecules (Figs. S3, S4). The dependency of WTM localization precision on the number of iterations was also investigated (Fig. S5). Comparison of the WTM algorithm with other multi-emitter fitting algorithms Among the multi-emitter fitting algorithms in this field, DAOSTORM is well-regarded; for example it has performed well in software competitions (Sage et al., 2015). DAOSTORM is based on an astronomy software, and simultaneously fit overlapping molecular PSFs with multiple model PSFs (Holden et al., 2011). We therefore considered DAOSTORM to be the best multi-emitter algorithm against which to assess the potential of the WTM algorithm (Fig. 3). The WTM algorithm found more molecules than DAOSTORM, especially at the higher molecule density up to 600 molecules μm–2 (Fig. 3A). With regard to the localization precision parameters, RMSD and the box plot results, DAOSTORM performed slightly better than WTM at molecular densities less than 20 molecules μm–2 , whereas both indices were better with WTM at 20 molecules μm–2 (Figs. 3B, C). A comparison of the raw data for the distances between recognized molecule positions and the true positions showed a significant difference between WTM and DAOSTORM at up to 20 molecules μm–2 molecular density (p < 0.05, Mann–Whitney U-test). These results suggest that DAOSTORM localised the recognized molecules closer to their true position than WTM at less than 20 molecules μm–2 , whereas localization was more accurate with WTM at 20 molecule μm–2 . However, RMSD analysis is not appropriate with higher molecular density images (>20 molecules μm–2 ) because the distance between the recognized and true molecule positions is so small that pairing becomes difficult. We therefore applied the nearest neighbour analysis method at different molecular densities up to 600 molecules μm–2 , plotting for each true molecule the distance to the nearest recognized molecule position against the distance to the nearest true molecular position (Fig. 3D). At the highest density (600 molecules μm–2 ), the distances between recognized molecule and true molecule positions was published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
MULTI-EMITTER FITTING ALGORITHM FOR POTENTIAL LIVE CELL SUPER-RESOLUTION IMAGING
longer with DAOSTORM than with WTM, suggesting that localization was more accurate with WTM at higher molecular densities (Fig. 3E). To compare WTM and DAOSTORM from a different point of view, we also show the false detection rate to detected molecules and the correlated detection rate to true molecules (Figs. S8A, B). WTM results show worse false detection rate and correlated detection rate than DAOSTORM. At more than 16 molecules μm–2 WTM show better correlated detection rate. These results are consistent with other results. However, the RMSD analysis showed better results with DAOSTORM than with WTM. Therefore, to confirm that this RMSD result could be reproduced in the nearest neighbour analysis, we modified the nearest neighbour analysis to include the pairing method used in RMSD for recognized and true molecule positions, considering only those located within 100 nm, and plotting these against the closest true molecule distances (Fig. S6). The result was consistent with the earlier RMSD result, showing that molecules recognized by DAOSTORM were localised closer to the true molecules was the case with WTM at molecular densities less than 20 molecules μm–2 . These results show that WTM can reconstruct nanoscopy images with imaging performance comparable to DAOSTORM from sparse to middle molecular densities, and that WTM showed better localization at higher molecular densities close to 600 molecules μm–2 . Images analysed by WTM and DAOSTORM are shown at different molecular densities in Figure 3(F), and nearest neighbour analysis graphs are presented in Figure 3(E). These graphs indicate better localization with WTM than with DAOSTORM at higher molecular densities and comparable localization at lower molecular densities. Especially at 100 and 300 molecules μm–2 , DAOSTORM tended to localise only a few molecules with extremely high intensity from the overlapping molecule cluster; this explained why the WTM nearest neighbour results were better. We also compared both algorithms in SNR image quality (Fig. S8C). The results also show that WTM is better than DAOSTORM at higher molecular densities. We compared the computational speed of the two algorithms, testing them on simulated images of different molecular densities and plotting the execution time for one frame against the molecular density (Fig. 3G). The execution time per frame was calculated from the computational time for 300 frames at each molecular density. At the highest molecular density analysed (20 molecules μm–2 ), the computational times were 0.079 s frame–1 for WTM and 8.15 s frame–1 for DAOSTORM, that is the WTM algorithm was approximately 100 times faster than DAOSTORM. Together, these results suggest that the WTM algorithm is relatively fast and has comparable localization precision and imaging ability to DAOSTORM at molecular densities less than 20 molecules μm–2 . At higher molecular densities the localization precision is more accurate with WTM. C 2018 The Authors. Journal of Microscopy
13
Biological application Finally, we examined the performance of the WTM algorithm for analysing biological samples. Image sequences captured using the GSDIM excitation procedure with total internal reflection fluorescence microscopy are presented in Figure 4(A). In the GSDIM time series image sequence, the density of activated molecules gradually decreased due to fluorophore bleaching. We extracted two image sequences in which the Golgi apparatus with EYFP appeared in MDCK cells from this image sequence: frames 1–100, at high molecular density, and frames 1–9957, which included high, middle, and low molecular densities, and applied the WTM algorithm or the M2LE algorithm with or without the ellipticity rejection filter (Fig. 4B). The M2LE algorithm, even without the ellipticity rejection filter, could not detect enough molecules to reconstruct the Golgi structure from both image sequences. In contrast, the WTM algorithm reconstructed the same Golgi structure image from the first 100 frames of high molecular density as from the 9957 frames. These data showed the WTM algorithm could reconstruct images that have the structure of the Golgi apparatus from only 100 frames of the high-density image set, whereas the M2LE algorithms failed to reconstruct the same structural images. We then applied the WTM algorithm to EYFP-tagged tubulin sample images (Figs. 4C, D). The original image sequence was captured under the GSDIM experimental condition. Figure 4(C) presents the first frame of the GSDIM image sequence and the WTM image reconstructed from 100 raw frames. The line profiles of these two images were measured and compared (Fig. 4D). The WTM algorithm clearly resolved the structure of the two parallel tubulin molecules, but this structure could be less resolved from the GSDIM image, which is comparable to a conventional microscope image. This suggested that WTM analysis was able to clearly resolve structures which are less resolved by conventional microscopy. We compared WTM to DAOSTORM for biological sample image analysis (Fig. 4E). Both algorithms recognized the molecules from the high-density image sequence and reconstructed the tubulin structure. However, the molecules recognized by DAOSTORM were somewhat aggregated, whereas the molecules recognized by WTM were smoothly distributed. This difference was shown by a line profile analysis of the reconstructed images (Fig. 4E, bottom). The aggregated structure of the molecules recognized by DAOSTORM reflected that algorithm’s behaviour at ultrahigh molecular densities (Fig. 3F); it may be an artefact because the structure of tubulin is not thought to be aggregated. We estimated the molecular density of sample image sequences from the numbers of molecules detected by the WTM algorithm: the mean molecular density was 323.8 molecules μm–2 and the maximum molecular density was 1363 molecules μm–2 . This difference between WTM and DAOSTORM could be due to the algorithms’ different responses to the extremely high molecular
published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
14
T. TAKESHIMA ET AL.
density of biological images, as shown in the nearest neighbour analysis (Fig. 3E). The WTM algorithm was used to analyse an image sequence of a dynamic sample and to reconstruct a series of superresolution images. This allowed the dynamic translocation of EB1 proteins on a microtubule to be observed with a reasonable temporal resolution (285 ms) (Fig. 5; Supplementary Movie). The temporal resolution could be improved by using a higher excitation intensity, a smaller field of view, and a correspondingly higher camera frame rate. Finally, we demonstrated two beneficial properties of the WTM algorithm: its ability to analyse large files and its realtime capability. The WTM algorithm was able to analyse 100 frames of 2048 × 2048 pixels and reconstruct a nanoscopy image, although this took approximately 20 min (Fig. 6). Figure 6(C) shows that even with 4 megapixels image, the WTM algorithm can analyse images. Regarding image quality, the WTM algorithm recognized the locations of the molecules and resolved the fine structure of microtubule, in the same way when the WTM algorithm is applied to small size image. For the algorithm’s real-time capability, with six CPU threads assigned, the WTM calculation took less than the camera exposure time, meaning that WTM could analyse the previous frame during the time the camera was capturing the next frame (Fig. S7). Not surprisingly, the WTM analysis with one CPU thread failed to analyse the image in time. By assigning several threads, the WTM algorithm could at least analyse an image in less time than the frame interval, indicating the possibility of real-time analysis. Discussion WTM algorithm is computationally fast, multi-emitter fitting algorithm that can analyse over a wide range of molecular densities. We showed in this study that WTM has the capability to analyse images for a wide range of molecular densities, from sparse to ultrahigh, and to reconstruct nanoscopy images using a considerably lower number of frames than other algorithms, within a reasonable amount of time. The algorithm can therefore enable the analysis of the dynamics of biological phenomena in living cells, using widely available and inexpensive PC-based computing. We compared the performance of WTM with that of another well-regarded multi-emitter fitting algorithm, DAOSTORM, for simulated images over a range of molecular densities and for an ultrahigh-density biological sample image. WTM recognized more molecules than DAOSTORM with reasonable precision, especially at ultrahigh molecular density; this difference was probably due to computational differences in the algorithms. DAOSTORM performs global minimisation of the error by simultaneously fitting all model PSFs to the whole image (Holden et al., 2011), whereas WTM uses a process of template matching at the edge of an overlapping molecular cluster. This difference resulted in DAOSTORM’s failure to recognise molecules in a very high-density molecular cluster. C 2018 The Authors. Journal of Microscopy
For an ultrahigh-density biological sample image of tubulin (at a density of about 300 molecules μm–2 ), the image reconstructed by DAOSTORM showed an aggregated structure, which was not the molecular structure of the tubulin. This may also have been due to the algorithm computational difference. One drawback of recent multi-emitter fitting algorithms is their execution time. WTM’s computational time was considerably faster than that of DAOSTORM, even without a GPU, due to the more extensive iteration process in the DAOSTORM algorithm. Even though DAOSTORM splits overlapping molecules into subgroups, the need for large-area optimisation resulted in a longer execution time than that of WTM. The algorithms MLEsCMOS and PALMER are also computationally fast, although a GPU is required for the fastest computation (Wang et al., 2012; Huang et al., 2013). We did not compare WTM to MLEsCMOS and PALMER in terms of computational speed; however, the WTM algorithm does not require a GPU, thereby simplifying practical implementation. L1-Homotopy is another fast algorithm that does not need GPU implementation (Babcock et al., 2013), and has been reported to be 300-fold faster than compressed sensing based on the interior point method. However, we found that WTM was more than 1000-fold faster than compressed sensing. From this, we assume that WTM retains a computation speed advantage over L1-Homotopy. The WTM algorithm uses a single-molecule model with a single PSF spatial profile and a single intensity. Although the single-molecule model helps to reduce the computational time of the algorithm, it limits the application of the algorithm to total internal reflection fluorescence images in which the variation of PSF is restricted. At the same time, this model causes fitting errors on images with defocused, elliptical, or unexpected PSFs because of imperfect optics or fixed dipole emitters. Another disadvantage of this single-molecule model is its inaccurate estimation of the number of molecules. Fluorescence molecules need to be blinking for localization microscopy. In this study, we used the GSDIM method to blink fluorescence molecules for high molecular density images. However, the GSDIM method requires a high-power laser, which is expensive and causes photobleaching, limiting its practical use. Recently reported methods for localization microscopy may solve these problems. These include spontaneously blinking fluorophores (Uno et al., 2014) and the DNAPAINT super-resolution imaging technology (Jungmann et al., 2014). The spontaneously blinking fluorophore HMSiR does not need a high-power laser, and DNA-PAINT is photobleaching free. These new methods make it easier for the biological researcher to acquire high molecular density blinking sample images for WTM analysis. The relatively fast computational speed of WTM algorithm helps to analyse large files like 2048 × 2048 pixel images with reasonable time. Recently, sCMOS cameras, which offer high speed and a high signal-to-noise ratio at low light levels, have published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
MULTI-EMITTER FITTING ALGORITHM FOR POTENTIAL LIVE CELL SUPER-RESOLUTION IMAGING
increasingly been used for biological imaging. These cameras (include the Hamamatsu ORCA-Flash4.0 series used in this study) have a resolution of 2048 × 2048 pixels, and algorithms that can analyse this image size would be of benefit to many researchers. An analysis using MLEsCMOS (Huang et al., 2013) showed a video rate temporal resolution better than the 285 ms resolution that we achieved with WTM; however, the field of view and the number of recognized molecules were very limited. Another group reported that an analysis using DAOSTORM showed temporal resolutions down to 1–2 s (Shim et al., 2012). Although a comprehensive comparison of WTM with all of the other algorithms was not possible, the ability of WTM to resolve meaningful images with as few as 25 frames of raw image data indicated the possibility of fast temporal resolution with WTM, limited by the camera frame rate and fluorophore activation. The combination of WTM’s ability as a multi-emitter fitting algorithm and its real-time analysis capability suggest it could be used for real-time monitoring of nanoscopy images during or immediately after an experiment (Kechkar et al., 2013). Usually, localization microscopy requires many steps, including the acquisition, storage, and analysis of many frames, and then rendering; immediate examination of the results is therefore impossible. A major part of the problem is the large number of images involved and the time required to reconstruct the final super-resolution image. The WTM algorithm can identify molecules even at a relatively high density, requiring fewer images for reconstruction. The computational speed of the WTM algorithm is relatively fast; seamlessly integrating image acquisition software with the WTM algorithm would make it possible to check results approximately in real time or shortly after the acquisition of image sequences. Similarly, the WTM algorithm could be used for the optimisation of experimental conditions, including of sample preparation, the region of interest, and focusing level. This application has the potential to bring many benefits to biological researchers. In conclusion, in this study we described and evaluated the WTM multi-emitter fitting algorithm, which can successfully process images of overlapping molecules over a wide range of molecular density from sparse to ultrahigh density for nanoscopy. Compared with other algorithms, WTM achieved a higher temporal resolution and a reasonable spatial resolution for visualising live cell nanostructures. As WTM is a computationally fast algorithm, it has the capability to reconstruct localization microscopy images in real time, potentially making possible the real-time monitoring of localization images in living cells in future investigations. Competing interests T.T., T.T., J.Y. and S.W. are employees of Hamamatsu Photonics K.K., a manufacturer of scientific cameras for life science applications. Y.O. declares no competing financial interests. C 2018 The Authors. Journal of Microscopy
15
Funding This work was partly supported by the following grants to Y.O.: Ministry of Education, Culture, Sports, Science and Technology, Grant-in-Aid for Scientific Research (KAKENHI, grant number 25113723), the Uehara memorial Foundation, and the Naito Foundation. Acknowledgements The authors thank Katsuhide Ito and Keith Bennett, HPK for generously reading manuscript and providing them useful comments. Patents Hamamatsu Photonics K.K. holds a patent JP2011288390 issued, a patent US13/726916 pending, a patent GB1407785.3 pending, a patent DE112012005521.7 pending and a patent CN201280065612.8 issued regarding this algorithm.
References Babcock, H.P., Moffitt, J.R., Cao, Y. & Zhuang, X. (2013) Fast compressed sensing analysis for super-resolution imaging using L1-homotopy. Opt. Express 21, 28583–28596. Betzig, E., Patterson, G.H., Sougrat, R. et al. (2006) Imaging intracellular fluorescent proteins at nanometer resolution. Science 313, 1642–1645. Boyd, N., Schiebinger, G. & Recht, B. (2015) The alternating descent conditional gradient method for sparse inverse problems. arXiv:1507.01562v1. Brunelli, R. (2009) Template Matching Techniques in Computer Vision: Theory and Practice. Chichester, West Sussex, U.K.: Wiley. Cox, S. & Jones, G.E. (2013) Imaging cells at the nanoscale. Int. J. Biochem. Cell Biol. 45, 1669–1678. Cox, S., Rosten, E., Monypenny, J., Jovanovic-Tailsman, T., Brunette, D.T., Lippincott-Schwartz, J., Jones, G.E. & Heintzmann, R. (2012) Bayesian localization microscopy reveals nanoscale podosome dynamics. Nat. Methods 9, 195–200. Folling, J., Mariano Bossi, M., Bock, H. et al. (2008) Fluorescence nanoscopy by ground-state depletion and single-molecule return. Nat. Methods 5, 943–945. Fullerton, S.M., Bennett, K., Toda, E. & Takahashi, T. (2012) Camera simulation engine enables efficient system optimization for super-resolution imaging. Proceedings of the SPIE, 8228. http://adsabs.harvard.edu/abs/2012SPIE.8228E..11F Hell, S.W. (2009) Microscopy and its focal switch. Nat. Methods 6, 24–32. Gustafsson, N., Culley, S., Ashdown, G., Owen, D.M., Pereira, P.M. & Henriques, R. (2016) Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations. Nat. Commun. 7, 12471 Hess, S.T., Girirajan, T.P.K. & Mason, M.D. (2006) Ultra-high resolution imaging by fluorescence photoactivation localization microscopy. Biophys. J. 91, 4258–4272. Holden, S.J., Uphoff, S. & Kapanidis, A.N. (2011) DAOSTORM: an algorithm for high-density super resolution microscopy. Nat. Methods 8, 279–280.
published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
16
T. TAKESHIMA ET AL.
Huang, B., Babcock, H. & Zhuang, X. (2010) Breaking the diffraction barrier: super-resolution imaging of cells. Cell 143, 1047– 1058. Huang, F., Hartwich, T.M.P., Rivera-Molina, F.E. et al. (2013) Video-rate nanoscopy using sCMOS camera–specific single-molecule localization algorithms. Nat. Methods 10, 653–658. Huang, J., Sun, M., Ma, J. & Chi, Y. (2017) Super-resolution image reconstruction for high-density 3D single-molecule microscopy. IEEE Trans. Comput. Imaging 3, 763–773. Jungmann, R., Avendano, M.S., Woehrstein, J.B., Dai, M., Shih, W.M. & Yin, P. (2014) Multiplexed 3D cellular super-resolution imaging with DNA-PAINT and Exchange-PAINT. Nat. Methods 11, 313–318. Kechkar, A., Nair, D., Heilemann, M., Choquet, D. & Sibarita, J.B. (2013) Real-time analysis and visualization for single-molecule based super resolution microscopy. PLoS One 8, e62918. Min, J., Vonesch, C., Kirshner, H. et al. (2014) FALCON: fast and unbiased reconstruction of high-density super-resolution microscopy data. Sci. Rep. 4, 4577–4585 Mortensen, K.I., Churchman, L.S., Spudich, J.A. & Flyvbjerg, H. (2010) Optimized localization analysis for single-molecule tracking and superresolution microscopy. Nat. Methods 7, 377–381. Noda, Y., Okada, Y., Saito, N., Setou, M., Xu, Y., Zhang, Z. & Hirokawa, N. (2001) KIFC3, a microtubule minus end-directed motor for the apical transport of annexin XIIIb-associated Triton-insoluble membranes. J. Cell Biol. 155, 77–88. Noordmans, H.J. & Smeulders, A.W.M. (1998) Detection and characterization of isolated and overlapping Spots. Comp. Vis. Image Understand. 70(1), 23–35. Quan, T., Zhu, H., Liu, X., Liu, Y., Ding, J., Zeng, S. & Huang, Z. (2011) High-density localization of active molecules using structured sparse model and Bayesian information criterion. Opt. Express 19, 16963– 16974. Rust, M.J., Bates, M. & Zhuang, X. (2006) Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3, 793–795. Sage, D., Kirshner, H., Pengo, T., Stuurman, N., Min, J., Manley, S. & Unser, M. (2015) Quantitative evaluation of software packages for single-molecule localization microscopy. Nat. Methods 12, 717–724. Shim, S.H., Xia, C., Zhong, G. et al. (2012) Super-resolution fluorescence imaging of organelles in live cells with photoswitchable membrane probes. Proc. Natl. Acad. Sci. U.S.A. 109, 13978–13983. Shroff, H., Galbraith, C.G., Galbraith, J.A. & Betzig, E. (2008) Live-cell photoactivated localization microscopy of nanoscale adhesion dynamics. Nat. Methods 5, 417–423. Small, A. & Stahlheber, S. (2014) Fluorophore localization algorithms for super-resolution microscopy. Nat. Methods. 11, 267–279.
C 2018 The Authors. Journal of Microscopy
Starr, R., Stahlheber, S. & Small, A. (2012) Fast maximum likelihood algorithm for localization of fluorescent molecules. Opt. Lett. 37, 413– 415. Thomann, D., Dorn, J., Sorger, P.K. & Danuser, G. (2003) Automatic fluorescent tag localization II: improvement in super-resolution by relative tracking. J Microsc. 211, 230–248. Ueno, H., Huang, X., Tanaka, Y. & Hirokawa, N. (2011) KIF16B/Rab14 molecular motor complex is critical for early embryonic development by transporting FGF receptor. Dev. Cell 20, 60–71. Uno, S.N., Kamiya, M., Yoshihara, T. et al. (2014) A spontaneously blinking fluorophore based on intramolecular spirocyclization for live-cell super-resolution imaging. Nat. Chem. 6, 681–689. Wang, Y., Quan, T., Zeng, S. & Huang, Z. (2012) PALMER: a method capable of parallel localization of multiple emitters for high-density localization microscopy. Opt. Express 20, 16039–16049. Zhu, L., Zhang, W., Elnatan, D. & Huang, B. (2012) Faster STORM using compressed sensing. Nat. Methods 9, 721–723.
Supporting Information Additional supporting information may be found online in the Supporting Information section at the end of the article. Fig. S1. Comparison of the input intensity parameters Fig. S2. Background subtraction Fig. S3. Applying the WTM algorithm to low-and mid-intensity molecule models Fig. S4. WTM algorithm on defocused molecule models Fig. S5. Iteration dependency of the WTM algorithm Fig. S6. Nearest neighbor analysis following the RMSD analysis method Fig. S7. Evaluation of the feasibility of real-time analysis during image capture Fig. S8. Comparison of WTM algorithm and DAOSTORM Fig. S9. Example images with and without image correction Supporting information Supplementary information is also available through the web site. http://doi.org/10.17632/266rp8rvzg.3 (1) Supplementary information manuscript (2) Supplementary information figures (3) WTM analysed result movie (4) Example data for algorithm test
published by JohnWiley & Sons Ltd on behalf of Royal Microscopical Society., 00, 1–16
