ACsN algorithmic framework

ACsN combines camera calibration, noise estimation and sparse filtering to correct the most relevant noise sources generated by a sCMOS camera (Fig. 1a and Supplementary Notes 1 and 2.1). In particular, ACsN first corrects the fixed-pattern noise using a map of the offset and gain of the sCMOS pixels. The presence of the fixed-pattern noise in sCMOS cameras generates in different pixels (p) a different number of photoelectrons from the same number of impinging photons (S p ). This effect is proportional to the illumination level and can be modeled as a multiplicative factor γ p applied to the parameter of the Poisson-distributed variable S p . At the same time, during the analog-to-digital (AD) conversion, the voltage produced by each pixel is read as the difference from a reference level, which represents the absence of light. In practice, this reference voltage is assigned a positive value that is responsible for a bias (β p ) in the measured intensity values. Therefore, the acquisition of a sCMOS camera can be modeled by the equation:20

$$Z_p = \gamma _p{\mathrm{Pois}}\left\{ {S_p\left( \tau \right)} \right\} + N\left( {0,\sigma _R} \right) + \beta _p,$$ (1)

where Z p is the value of the pixel p, τ the exposure time, and N (0, σ R ) the Gaussian-distributed readout noise of mean μ R = 0 and standard deviation σ R . Considering the practicality of fluorescence microscopy, in this model we have omitted the contribution of dark current, which can be disregarded for exposure times below 1 s, and the quantization noise due to the AD conversion, which is negligible compared to the readout noise3,21 (Supplementary Note 2.2).

Fig. 1: ACsN concept and performance. a Concept of the ACsN algorithm. The input image is scaled with the pixel gain and offset maps of the camera in order to remove the fixed-pattern noise (FP). Then, using the experimental parameters, the OTF boundary is calculated and used to produce a high-pass filtered image, from which the noise estimation (NE) is obtained. Finally, sparse filtering (SF) is performed to generate the denoised image. b Comparison of noise variations before (gray squares) and after (red circles) noise correction. All data were divided by the expected value for pure Poisson noise. The dashed line represents the ideal camera performance. To generate this plot three different sets of images of HeLa microtubules were used. The error bars represent the temporal standard deviation (STD) evaluated over 100 images. c, d Fluctuation maps, i.e., STD evaluated over 100 sCMOS images acquired at a 10-ms exposure time before (c) and after (d) ACsN denoising. Intensities are expressed in analog-to-digital units (ADU). e, f Zoomed-in images of the areas marked by the white squares in c and d, respectively. g Temporal fluctuation of the intensity values of the pixels corresponding to the circled areas (1 and 2) in e and f, respectively. The values from the original and denoised images are plotted in gray and red, respectively. Scale bars: 500 nm (a), 1 µm (b), 3 µm (d), 300 nm (f). Full size image

Since the fixed-pattern noise depends only on the camera circuitry, β p and γ p can be estimated through a one-time calibration (see Methods). However, a careful assessment of both the Gaussian-distributed readout noise, N(0, σ R ), and the fluctuation due to the Poisson-distributed photon shot noise, Pois{S p (τ)}, is necessary to obtain an accurate estimate of the underlying signal S p . To perform this assessment, we devised a noise model that allows for a joint estimation of the noise variance by analyzing the frequency response of the microscopy system. This is based on the fact that the Poisson distribution of the photon shot noise can be feasibly approximated by a Gaussian distribution when the photon flux is >3 photons per pixel22. In particular, the error introduced by approximating the Poisson variance, \(\sigma _P^2\), with a Gaussian variance, \(\sigma _G^2\), becomes <1% when the photon flux is more than 5 photons per pixel (Supplementary Note 2.3). Notably, the abovementioned conditions on the photon flux are usually satisfied for many applications in fluorescence microscopy23,24. Therefore, we consider the camera-related noise as the result of the sum of two independent Gaussian-distributed random variables, whose variance is \(\sigma _N^2 = \sigma _R^2 + \sigma _G^2\). Such a distribution consists of a constant power spectral density, while the signals coming from the sample are contained within the optical transfer function (OTF)25. Therefore, we take advantage of the knowledge of the optical system to evaluate the pixel fluctuation outside the OTF, which is due to noise only, and then we use the value obtained to derive σ N in the original image (Supplementary Note 2.3).

Next, the algorithm uses these noise statistics for a non-local assessment of the self-similarity of the sample and to perform collaborative sparse filtering on the input sequence. Unlike previous implementations of collaborative filtering, we adopted a layered approach that sequentially probes the image self-similarity in space and time in order to enhance noise correction without sacrificing accuracy and runtime. In brief, the filter decomposes the image in patches and sorts them into three-dimensional (3D) groups according to their similarity26. Then, it employs a 3D transform to process each group all at once. The denoising is performed by hard-thresholding and enhanced by the fact that, due to the similarity between the patches, the 3D transform results in an even sparser representation of the original patches, whereas the noise power spectrum remains constant27. Afterwards, the denoised patches are returned to their original locations to form an intermediate image. At this point, the collaborative filter is run a second time but replacing the hard-thresholding with a Wiener filter. The filter is performed using both the noisy and intermediate images and generates the final denoised image (Supplementary Note 2.4). It should be noted that the spatial variation of the noise across the image may affect the performance of the Wiener filter. However, this is considerably mitigated by the use of patch-based processing, which, compared to the whole image, enhances the intensity uniformity within individual patch groups, exhibiting a great stability against spatially variant noise9.

Finally, another collaborative filter is performed looking for similar patches also in the neighboring frames. This way, lingering noise can be further reduced taking advantage of the sample self-similarity in time while preserving the temporal resolution18 (Supplementary Note 2.5).

Characterization of ACsN

Next, we characterized the performance of ACsN using both numerical and experimental data. Notably, ACsN collaborative filtering depends on the estimation of σ N , as well as on the choice of the parameters in the algorithm28, which were chosen in order to optimize both the noise correction and runtime (Supplementary Note 3.1). We observed that our strategy can significantly attenuate the detrimental effect of camera noise, avoiding loss of image resolution, especially in presence of highly spatially variant noise (Supplementary Note 3.2). Moreover, the camera noise can induce temporal fluctuations of the pixel values that are not related to the sample, thus affecting the quantitative analysis of time-lapse data. ACsN denoising reduces this effect by approximately one order of magnitude, with residual fluctuations comparable to that of an ideal camera (Fig. 1b–g and Supplementary Note 3.3). Furthermore, it should be noted that at low-photon counts, the sample’s details start to be comparable with the noise fluctuations and become harder to retrieve. Thus, the performance of image restoration is intrinsically related to the photon flux of the input image. Nonetheless, using both simulations and experimental data, we verified a robust ACsN noise correction at low-light levels down to 5–10 photons per pixel (Supplementary Note 3.4).

Furthermore, we validated the performance of ACsN under various sampling rates normally adopted for fluorescence microscopy. In practice, a sampling rate close to the Nyquist criterion represents a good tradeoff between signal to noise ratio (SNR) and detail preservation. Here, examining numerically and experimentally across a wide range of sampling rates, we demonstrated the viability of ACsN for low SNR with oversampling and no noticeable loss of signals with under-sampling (Supplementary Note 3.5).

Unlike natural images, fluorescent images of biological samples are highly specified, exhibiting precisely labeled molecular targets or structures in cells. Therefore, each fluorescent image usually features specific objects recurring across the field of view, which supplies sufficient non-local self-similarity to make the algorithm notably efficient for fluorescence microscopy. With numerical and experimental data, we characterized the dependence of the ACsN performance on the usage of self-similarity of an input image (Supplementary Note 3.6). Furthermore, as shown in the following, we quantitatively assessed a variety of non-biological and biological samples to verify the viability of the method, spanning various dimensionality, morphology, randomness and density, such as caliber targets, fluorescent particles, single molecules, microtubules, actin filaments, mitochondria, filopodia, lamellipodia, and small animals.

Wide-field microscopy

Wide-field microscopy, especially total internal reflection fluorescence (TIRF) microscopy, is one of the most widely used techniques in cell imaging29. TIRF uses the phenomenon of total internal reflection of light at the glass/water interface in order to create an evanescent wave that propagates only for a few hundreds of nanometers across the coverslip. This allows the selective excitation of the fluorescent labels at the bottom of the sample (Supplementary Fig. 1a). However, in case of weak fluorescent emitters, low-light intensity or a short exposure time, sCMOS-related noise becomes severe and deteriorates image quality (Supplementary Fig. 1b). ACsN denoising can effectively reduce such contribution and recover the undistorted signals from the noise, allowing faster acquisition without compromising the underlying signal (Supplementary Fig. 1c, d).

We demonstrated ACsN denoising of wide-field microscopy in both epi-fluorescence and TIRF configurations using various fixed, live and multi-color sub-cellular samples, including microtubules (Fig. 1 and Supplementary Fig. 1), mitochondria (Fig. 2 and Supplementary Movies 1 and 2), and F-actin (Fig. 2). The use of ACsN can maintain the same image quality with a shorter exposure time (i.e., better temporal resolution) and a lower excitation level (i.e., less photo-damage). The performance is, thus, limited primarily by the photo-physics of the fluorescent emitters. Using quantitative metrics, we showed that the method can recover wide-field images with a photon budget two orders of magnitude lower with no loss of image quality (Supplementary Table 1).

Fig. 2: ACsN noise correction improves wide-field fluorescence microscopy. a Epi-fluorescence imaging of mitochondria in fixed bovine pulmonary artery endothelial (BPAE) cells at an exposure time of 1 ms. b The same image in a after ACsN denoising. c–f Zoomed-in images of the corresponding boxed regions in a and b. Quantitative results and analysis are reported in Supplementary Table 1. g Representative frame from a time-lapse of 100 images of mitochondria in live human embryonic kidney (HEK) cells recorded at 50 Hz (exposure time: 20 ms). h The corresponding representative frame of the image sequence (g) obtained after ACsN processing. The insets in g and h show zoomed-in images of the corresponding regions marked in the dashed white box in g. i–n Zoomed-in images of the corresponding regions marked in the solid yellow box in g at different time points of 200 ms (i, l), 800 ms (j, m), and 1200 ms (k, n). o, p Dual-color image, respectively, before (o) and after (p) ACsN denoising of F-actin (cyan) and mitochondria (orange) in fixed BPAE cells obtained by TIRF microscopy with an exposure time of 2 ms. q, r Cross-sectional intensity profiles of (o) and (p) along the corresponding dashed line in o, respectively, showing substantially denoised and better resolved cellular structures. Scale bars: 10 μm (b), 3 μm (f), 4 μm (h, p), 1 μm (h, inset) and (l). Full size image

Deconvolution and light-field microscopy

Image deconvolution is widely used in optical microscopy, from the restoration of low-quality images to the improvement of super-resolution techniques30. However, noise can easily degrade the performance of many common algorithms by producing deconvolution artifacts. Instead, we observed a remarkable reduction of such artifacts in deconvolved images by employing ACsN denoising prior to different methods based on Richardson–Lucy algorithm31, machine learning32, and radial fluctuation33 (Supplementary Note 4.1). The enhancement of image restoration is reflected also by an improvement of the global image quality, evaluated using metrics such as the Resolution Scaled Pearson’s coefficient (RSP)34. For example, combining ACsN and radial fluctuation, we generated super-resolution images with a better RSP value at a temporal resolution up to two orders of magnitude higher than currently reported33 (Supplementary Fig. 2).

Image deconvolution is also at the basis of three-dimensional reconstruction in light-field microscopy (LFM). LFM employs a microlens array in a microscopy system to obtain both the two-dimensional (2D) spatial and 2D angular information of the incident light, allowing for computational reconstruction of the full 3D volume of a specimen from a single camera frame35. However, the deconvolution-based reconstruction process is highly sensitive to the SNR, especially due to LFM’s wide-field, volumetric, and fast imaging scheme. For this reason, the use of ACsN to correct the noise in the raw images (Fig. 3a, b) results in clearly noticeable improvement in the 3D light-field reconstructions (Fig. 3c, d). Indeed, the presence of the noise leads to the miscalculation of the 3D object or the propagation of non-fluorophore-associated peaks. The former affects the sampling along the axial dimension and can result in an uneven axial resolution (Fig. 3e, f). The latter produces additional background that covers the fluorescence signal, impairing also the lateral resolution (Fig. 3g–i). Using ACsN, both deficiencies can be mitigated, resulting in substantially improved 3D volumetric rendering of cellular structures.

Fig. 3: ACsN denoising improves the quality of 3D reconstruction in light-field microscopy. a, b Raw light-field images of microtubules in a HeLa cell before (a) and after (b) ACsN processing. Insets show the zoomed-in microlens images of the corresponding boxed regions, where noise has been substantially reduced as seen in b. c, d Three-dimensional (3D) reconstructed images obtained from a and b, respectively. The depth information is coded according to the color scale bar. Insets show the zoomed-in images of the corresponding white dashed boxed regions, where better image quality and improved 3D resolution are observed after ACsN denoising. e, f Cross-sections on the YZ plane corresponding to the red dashed lines in c and d, respectively, where microtubule structures are better resolved with reduced artifacts using ACsN. g, h Zoomed-in images of the red solid boxed regions in c and d, respectively, at z = 1.4 μm, where microtubule structures are better resolved using ACsN. i Cross-sectional profiles of (g, gray) and (h, red) corresponding to the white dashed lines in g, h, respectively. Filaments covered by non-fluorophore-associated background noise are resolved using ACsN. Scale bars: 8 µm (b, d), 800 nm (b, inset), 3 µm (d, inset), 1 µm (e, g). Full size image

Single-molecule localization microscopy

To validate the feasibility of ACsN for single-molecule localization microscopy (SMLM)36, we performed STORM imaging of mitochondria in HeLa cells (Supplementary Fig. 3). The effect of sCMOS-related noise in single-molecule localization can be seen in two aspects: the presence of false negatives, due to the loss of weakly emitting molecules covered by noise (Supplementary Fig. 3c, d), and the presence of false positives, due to the hot pixels or simply the noise distribution (Supplementary Fig. 3e, f). Removing the noise from the raw single-molecule data allows for suppression of both types of localization errors, resulting in significantly improved STORM image quality and metrics such as the RSP and the Resolution Scaled Error (RSE)34 (Fig. 4a, b). Also, such improved efficiency of localization leads to a better contrast and the appearance of features not clearly visible in the reconstruction without denoising (Fig. 4c–f). Furthermore, the reduction of pixel fluctuations unrelated to the sample permits to obtain a map of the fluorophores’ blinking rate that can be used to alleviate the effects of imperfect labeling (Supplementary Fig. 4).

Fig. 4: ACsN improves localization performance in STORM and single-particle tracking. a STORM image of mitochondria in a fixed HeLa cell (RSP: 0.81, RSE: 40.6). b STORM image reconstructed after ACsN denoising of raw single-molecule data of a (RSP: 0.85, RSE: 36.7). In both cases, 5000 single-molecule frames were used. Representative frames of the raw data before and after denoising are shown in Supplementary Fig. 3. Quantitative image analysis with NanoJ-SQUIRREL assessed an improvement of both RSP ( + 0.04) and RSE (−3.9) values in b compared to a. It is observed that the number of localizations in b is increased in comparison with a, which leads to a better contrast in the former and to the appearance of features not visible in the latter (c–f). g Single-particle tracking of a fluorescent bead recorded with a 1 ms exposure time. A representative frame is shown in the inset. Each color corresponds to one of the six different tracks detected. h Single-particle tracking of the same bead in g after ACsN denoising (inset). The improved SNR yields a better localization accuracy, which results in a single, smooth trajectory (black line). i Representative frame for biplane single-particle tracking at 1 kHz frame rate (exposure time: 1 ms) before (left) and after (right) ACsN denoising. Scale bars: 4 µm (a), 2 µm (c, e, i), 1 μm (g, inset), 250 nm (h). Full size image

Like single-molecule imaging, the localization precision in single-particle tracking (SPT) is closely related to the number of photons detected. Therefore, one critical factor affecting the performance of SPT is the SNR of the image data37. We showed that ACsN can be used to minimize the localization errors responsible for misidentification of particles and erroneous trajectories (Fig. 4g, h and Supplementary Movie 3). This SNR improvement results in a better particle localization accuracy, i.e., a better estimation of the bead’s lateral displacement with sub-pixel sensitivity. This can be of great use also in biplane SPT, where the accuracy of the 3D tracking depends on the quality of the out-of-focus image38 (Fig. 4i, Supplementary Movie 4, and Supplementary Note 4.2).

Fluorescence microscopy with low-cost CMOS cameras

Recently, the advances of high-end industrial-grade CMOS cameras have sparked the interest of the scientific community at the possibility to approach the performance of sCMOS cameras at a more affordable price39,40,41,42. It has been shown that such CMOS cameras can be utilized for SMLM imaging41,42. However, the lower quantum efficiency and the higher readout noise limit the image quality and the general usability for quantitative biomedical research in many areas. Addressing the challenge with a proper denoising strategy would provide a critical and timely solution to transform the industrial-grade cameras for broader imaging applications. Here, we first implemented ACsN with a high-end industrial-grade camera for wide-field microscopy using both epi- and TIRF illumination (Fig. 5a–h). In both configurations, ACsN denoising substantially improved the image quality, achieving prominent agreement with the images obtained by the sCMOS camera (Supplementary Figs. 5 and 6, and Supplementary Table 2).

Fig. 5: ACsN improves fluorescence microscopy with low-cost CMOS cameras. a TIRF image of F-actin in a fixed BPAE cell, taken at a frame rate of 38 Hz (exposure time: 26 ms). b The same image in a after ACsN denoising. c Epi-fluorescence imaging of mitochondria in a fixed bovine pulmonary artery endothelial (BPAE) cell, taken at a frame rate of 38 Hz (exposure time: 26 ms). d The same image in c after ACsN denoising. e–h Zoomed-in images corresponding to the boxed regions in a–d, showing the improvement of image quality after ACsN denoising. In particular, such improvement is comparable to the images taken with sCMOS sensors, as shown in Supplementary Figs. 5 and 6. i, j Images of GFP-stained calcein in live Adipocytes (lipocytes) taken with low-cost CMOS for miniaturized microscopy before (i) and after (j) ACsN denoising. The data were taken by immersing a miniscope in live-cell culture. k–n Zoomed-in images of the corresponding boxed regions in i and j. o, p Plots of the cross-sectional intensity profiles of cellular structures before (gray) and after (red) ACsN denoising along the dashed lines in k, l and m, n, respectively. Scale bars: 10 μm (a, c), 4 μm (e, g), 50 μm (i), 20 μm (k). Full size image

The single-photon-excitation-based miniaturized microscope, or miniscope, has been developed to perform wide-field calcium imaging in freely behaving animals43,44,45. The required miniaturization was achieved by replacing compound objective lenses with a gradient-index (GRIN) rod lens, which offers several advantages, including low cost, light weight, and relatively high-numerical aperture. These features of the miniscope enable minimally invasive imaging of a significant volume of the brain with a cellular-level resolution during complex behavioral, cognitive and emotional states46,47,48. However, the low-cost CMOS sensor (MT9V032C12STM, ON Semiconductor, price ~$15) currently adopted yields a poor image quality in order to obtain a relatively high imaging speed, which can be severely restrictive for broader applications in cell imaging. Here, we validated the feasibility of ACsN for the miniscope sensor by performing single-photon-excitation-based, wide-field imaging of GFP-stained calcein in live Adipocytes (Fig. 5i–p).

Selective plane illumination microscopy

In contrast to wide-field microscopy, selective plane illumination microscopy (SPIM) illuminates the sample with a sheet of light perpendicular to the direction of observation. This avoids unnecessary illumination, permitting an unparalleled long-term imaging of dynamic biological specimens49,50,51. Lattice light-sheet microscopy (LLSM) further optimizes the optical system by illuminating the sample with multiple plane waves that sculpt a propagation-invariant optical lattice52. However, while new strategies are being investigated to deal with sample-related issues53,54, camera noise remains the most relevant limitation to SPIM and LLSM imaging capabilities due to their relatively low-background signal.

We first demonstrated that ACsN denoising can overcome this limitation by performing a SPIM volumetric scan of a fixed brine shrimp. Here, we enhanced the self-similarity using 3D sparse filtering along the scan direction. After ACsN processing, we observed that noise-canceling makes the sample’s details stand out better in each individual slice (Supplementary Fig. 7). In particular, the correction of the fixed-pattern noise is especially noticeable in the maximum intensity projection images (Fig. 6a, e and Supplementary Movie 5). In addition, it is remarkable to observe a clear improvement in the orthogonal cross-sections of the scanned volume (Fig. 6b–d, f–h), allowing for a better assessment of the sample’s 3D structures.

Fig. 6: ACsN processing of volumetric data obtained with SPIM and LLSM. Maximum intensity projections (MIP) of SPIM images of a fluorescently labeled adult brine shrimp before (a) and after (e) ACsN denoising. Orthogonal views along the XZ plane of the raw (b–d) and denoised (f–h) volumetric scans at y = 237 mm (b, f), y = 904 mm (c, g), and y = 1491 mm (d, h). Slices along the XY and YZ plane have been provided in Supplementary Fig. 7. i Three-dimensional rendering of live human lung cancer cells (NCI-H1299 NSCLC) acquired with LLSM and processed with ACsN denoising. Zoomed-in images of the area corresponding to the white box in i before (j) and after (k) ACsN denoising. The corresponding time-lapse sequence has been provided in Supplementary Movies 6 and 7. MIP images and representative slices are depicted in Supplementary Figs. 9 and 10. Scale bars: 400 μm (a, e), 100 μm (b, f), 10 μm (i), 4 μm (k). Full size image

To validate ACsN processing for LLSM, we first imaged fixed skin cells stained for Keratin with EGFP at different exposure times (5, 10, and 20 ms) using a constant laser illumination power of 27 mW (measured at the back focal plane of the illumination objective). These images were acquired using the sample scan mode and, accordingly, the slices had to be deskewed to retrieve the original positions (see Methods). We performed such operation before ACsN denoising in order to utilize the self-similarity along z for 3D sparse filtering. We observed that the image quality can be well maintained by denoising even after a fourfold reduction of the exposure time (Supplementary Fig. 8 and Supplementary Table 3).

Furthermore, we demonstrated ACsN image restoration of time-lapse live-cell LLSM imaging. First, we imaged live human lung cancer cells (NCI-H1299 NSCLC) in the sample scan mode with intervals of 18.4 s over more than 30 min (Fig. 6i–k, Supplementary Fig. 9, and Supplementary Movies 6 and 7). As stated above, the sample scan mode requires deskewing of the volumetric slices, which increases the size of the dataset and, then, the processing complexity. In contrast to the previous case, however, for time-lapse imaging we were able to utilize the temporal self-similarity, which yields a more efficient noise correction compared to the volumetric one55. Therefore, we denoised the time-lapse volumetric scans by processing the corresponding temporal stacks of each individual slice. This way, ACsN could be used before deskewing, effectively preserving the denoising performance while saving the computational time (Supplementary Fig. 10). Next, we observed the movement of endogenous F-actin in live mouse embryonic fibroblasts using LLSM in the sheet scan mode (see Methods). Notably, this mode does not produce any shift between the slices, and the volumetric information can be retrieved without deskewing (Supplementary Fig. 11). In particular, the movement of filopodia all around the cell can be observed with higher clarity after denoising (Supplementary Movie 8).