High resolution single photon sensitive 3D imaging

Our 3D imager utilizes an upconversion single photon detector (USPD) capable of QPMS due to carefully selected pump and probe pulses. The setup is shown in Fig. 1 (see Methods section for details). The pump and probe pulses are carved from a 50 MHz femtosecond mode-locked laser (MLL) using separate sets of cascaded spectral filters. The probe pulses are sent out to raster scan the target scene via an optical transceiver and programmable scanning MEMS mirror. Meanwhile, the pump pulses are sent to a programmable optical delay line (ODL). They are then combined with the backscattered probe signal and sent to the USPD. The internal conversion efficiency of the waveguide is 121% W−1 cm−2, and the total detection efficiency of the USPD is 3.6% with total dark count rate of 250 Hz (as compared with about 1000 Hz for a typical 1-ns gated InGaAs APD)29. The dark counts of USPD are primarily attributed to Raman noise photons generated in the upconversion waveguide22 as the Si-APDs 50 Hz dark count level is negligible. Note that Raman noise photons are maximally filtered out by QPMS due to the mode selectivity of our imager29,31. The robust all-fiber design of our imager reduces the number and footprint of required optical components while simplifying the optical alignment procedure. Finally, a field-programmable-gate-array (FPGA) is employed as the central processor for controlling the MEMS mirror and ODL, and to collect data from the USPD.

Fig. 1: Experimental setup of noise tolerant 3D single photon imaging. MLL, mode-locked fiber laser (Repetition rate = 50 MHz, Center wavelength = 1560 nm); MEMS, micro-electro-mechanical systems; ODL, optical delay line; ASE, amplified spontaneous emission (1520–1570 nm); USPD, upconversion single photon detector; Si-APD, silicon avalanche photodiode; FPGA, field-programmable-gate-array. Pump (1565.5 nm) and probe (1554.1 nm) pulses are carved out from the MLL using off-the-shelf 200 GHz telecom wavelengths dense-wavelength-division-multiplexing (DWDM) filters. The FPGA is the central processor for controlling the programmable MEMS mirror and programmable ODL, and acquiring the data from the USPD. The noise, which is covering the identical spectral-temporal region as probe signal, is carved out from an ASE source with a DWDM filter. Full size image

To reconstruct the 3D image of the target scene, we measure the time-resolving upconversion signal to retrieve the time-of-flight (ToF) information of the backscattered signal while raster scanning the probe beam across the scene. We use a pixelwise maximum-likelihood value (MLV) approach where a time-resolving measurement is performed for each pixel by counting the number of upconverted photons as a function of the temporal delay between the synchronous probe and pump pulses. The temporal delay is scanned using the programmable ODL where backscattered signal photons that are temporally aligned with the pump pulses in the PPLN will be upconverted efficiently and detected by the Si-APD. In contrast, photons distributed in orthogonal time-frequency modes will be upconverted with very low efficiency, even when they temporally overlap with the pump pulses29,32. The scanning of the ODL creates a photon-counting histogram for the ODL points, and the MLV estimate is the point with the most detections. For post-processing, a standard MATLAB median filter is applied to smooth the reconstructed image. The filter converts every pixel value to the median of a 3 × 3 region consisting of the pixel and its eight nearest neighbors. This commonly used image processing tool is preferred over mean filtering because it is robust against bias from outlier values. An example of a complete time-resolved (longitudinal) upconversion signal from two different transverse locations is shown in Fig. 2a. Note that the time-resolving upconversion photon counting reflects the intensity-correlation of pump and backscattered probe pulses, where the measured width is about 9 ps (versus ~500 ps for typical InGaAs APDs). This shows the ultrahigh timing resolution of our system upon a single-detection event, which is defined by the phase matching of the upconversion and the full-width-half-maximum (FWHM) of optical pulses. It represents almost one order of magnitude improvement over the resolution achievable with Si-APDs whose timing jitter is typically capped at 50 ps. Sub-picosecond timing resolution is achievable with shorter optical pulses that can be created using spectrally wider filters and waveguides with broader phase matching33.

Fig. 2: High resolution single photon sensitive 3D imaging. a Time-resolving photon counting from two different longitudinal positions. b MLV histogram for two regions (A and B) within the dotted lines in (c). c 5 cm × 7 cm CNC machined aluminum depth resolution calibration chart. d Reconstructed image of the depth resolution chart in (c). Full size image

Our system can achieve even higher ranging resolution by using the ODL to acquire the full upconversion histogram. We experimentally determine that a peak of 150 detections (equivalent to 0.006 photon detections per pulse over 500 μs dwell time) is sufficient to reach the minimum standard deviation of 0.5 ps for a single time-resolving measurement (see Supplementary Note 5). Reduced standard deviation in temporal measurement is translated into improvement in the ranging resolution15 as depicted in reconstructed 3D image (Fig. 2d).

To assess the depth resolving capability of our imager, we carry out 3D imaging with 2400 pixels on a machined depth resolution chart placed 1.5 m from the transceiver. The chart is a 50 mm × 70 mm aluminum block with 20 depth varying circles of 7 mm diameter (see Fig. 2c). The bottom row contains four reference circles of 1 mm depth. The depth of the remaining circles ranges from 1 to 2.5 mm in steps of 0.1 mm. The reconstructed 3D image is shown in Fig. 2d, where the ODL was scanned in 1 ps steps over a range of 30 ps. The dwell time for each pixel is 1 ms per ODL sample, rendering a total data acquisition time of 72 s. This time can be substantially reduced by using shorter dwell time and fast ODLs, such as those based on solid-state switches. Shown in Fig. 2b, we plot the MLV histograms for all pixels contained within the dotted lines (regions A and B) of Fig. 2c. Each histogram contains two features, with the left-most from pixels on the surface around the indented circle and the right-most from pixels within the indented circle. The separation between these two features gives a measured ToF difference corresponding to the depth of the circle compared to the surrounding surface. Additionally, the depth profiling accuracy of our 3D imager is determined to be approximately 0.09 mm by measuring it against a calibrated linear translational stage and a certified gauge (see Supplementary Note 4). The transverse spatial resolution of this imager is currently limited by the collimated optical beam diameter (2.2 mm).

Imaging through an obscuring object

Our 3D imager is advantageous in discriminating objects of interest in a complex environment with multiple reflecting surfaces owing to the picosecond pump pulses that are physically time-gating the backscattered photons. Therefore, we manage to accept photons from the target while discounting photons arriving at different times, even as close as several picoseconds away. This also effectively removes the distorting pileup effect14 that can arise from repeated detection of undesired photons within a larger detection window. The performance of our technique shows improvement in terms of sensitivity, timing resolution, accuracy, bias, and dead time14.

We perform 3D imaging through an obscurant where a ceramic mannequin head is located 2 mm behind an obscuring aluminum wire mesh with 1 mm2 holes, as depicted in Fig. 3a, c. Note that the ToF difference between the mesh and the target is well below the 50 ps timing resolution of a typical Si-APD. The highly reflective mesh reduces the backscattered photons from the mannequin by an average of 5 dB while inducing backscattering ahead of the desired target. Nonetheless, our 3D imager is capable of resolving both the mannequin head and the aluminum mesh as shown in Fig. 3b. Additionally, time resolving feature of our imager allows us to isolate mannequin head as depicted in Fig. 3d. Our system distinguishes between the two reflecting surfaces by identifying separate peaks in the upconversion signal. This is demonstrated in Fig. 3e–g which show the time-resolving photon counting measurements for different pixels along the nose of the mannequin head. The left peak represents the backscatter from the mesh and the right peak represents the backscattered signal from the mannequin. The short time-gate created by picosecond pulses enables the immaculate retrieval of 3D information from the mannequin head despite the presence of backscatter from the obscurring material. The optical-gating advantage of our 3D imaging technique can be extended for use in distinguishing objects in a complex environment19,20 and other applications where precise detection of faint optical pulses is needed34,35.

Fig. 3: Imaging through obscuring object. a Close-up photo showing the milimetric distance of obscuring aluminum wire mesh (1 mm2 opening) and target (head mannequin). b Reconstructed 3D image showing the obscurant and target, resembling a. c Obscuring aluminum grid and target looking at probe signal's propagation direction. d Reconstructed 3D image of target isolated from obscurant. e–g Time-resolving photon counting obtained at different raster scanning position, showing distinguishable backscattering signal peaks from the obscurant and target. Full size image

Noise-tolerant imaging

To demonstrate the noise tolerance of our technique, we perform 3D imaging in a scenario where the backscattered signal photons spectrally and temporally overlap with strong background noise. The noise photons are generated by using a DWDM filter to spectrally carve the amplified spontaneous emission (ASE) from an erbium doped fiber amplifier (EDFA). The filter is similar to the one used to carve the signal from the MLL to ensure that the signal and noise match well in spectrum. Then, we mix the noise with the backscattered signal using a 99:1 fiber beam splitter, as shown in Fig. 1.

We test active imaging under two different levels of background noise, marked by the number of ASE noise photons per period of the signal pulses (20 ns). The results for 350 and 1700 background photons per period are shown in Fig. 4. These noise levels amount to incident signal to background ratios of 1/23,000 and 1/113,000, which is well beyond the capabilities of computational imaging post-processing36. For a typical direct detection system using an InGaAs APD with 1 MHz maximum counting rate, 7.5% detection efficiency, and 1 ns time-gating window, those levels correspond to 1.3 and 6.4 mean detected photons per detection window. In both cases the InGaAs detector would be saturated by the noise and therefore unable to detect the low-flux signal. In contrast, the mean photon number detected by QPMS per pulse is only 0.0004 and 0.0016, respectively. Hence, the current QPMS implementation gives an average of about 36 dB advantage in background suppression over direct detection.

Fig. 4: Noise-tolerant imaging. a Noise free raw 3D image of target, acquired with mean photon number of 15 detections per pixel. b Reconstructed 3D image acquired with 34 times noise photons after a median filter has been applied. c Error map of reconstructed 3D image (b), with noise free 3D image (a) as ground truth. d, e The raw 3D images acquired with the presence of 7 and 34 times noise. f Time-resolving photon counting at different noise levels, where the dashed lines mark the average number of 7 (green) and 34 (red) times noise photons counted at each ODL point. Full size image

This substantial advantage is attributed to the reduced detection mode number, polarization sensitive upconversion, and mode selectivity against spectrally and temporally overlapping noise17,28,29. In the case of direct detection, the DWDM filter has the spectral bandwidth of B = 250 GHz and detector’s gating window T = 1 ns, which combined to a total detected time-frequency modes of πBT∕2 ≈ 392. For QPMS, the phase matching profile of upconversion waveguide limits the spectral bandwidth to B = 90 GHz while the effective detection window of T = 6 ps is defined by the FWHM of pump pulse. Evidently, the QPMS reduces the number of detection modes to ≈1, resulting a total noise suppression advantage of 25.9 dB. Theoretically, similar noise suppression can be achieved by using a combination of ideally matched time-frequency filters. However, it will be difficult to implement in practice for such ultrashort and faint optical pulses. Additionally, upconversion is efficient only along a single polarization, which offers another 3 dB noise suppression by rejecting half of randomly polarized noise photons. Finally, the mode selectivity contributes an additional 7.1 dB to the total 36 dB advantage in background suppression observed, agreeing well with the 7.4 dB mode selectivity predicted by simulation result (see Supplementary Note 3). Even higher selectivity can be achieved by using a longer PPLN waveguide, tailored phase matching profile or optimally designed pump pulses27,28,37.

The exceptional noise rejection by QPMS not only defeats the detector saturation, but also helps accurate image reconstruction. This is because high levels of detected noise necessarily leads to image distortion and errors by virtue of its Poissonian variance. Figure 4a, d, e shows how the number of erroneous pixels on the raw 3D images grows quickly with the background level. Figure 4c shows that considerable features on the mannequin head are concealed by the noisy pixels. This highlights the importance of background rejection when the backscattered signal photons are scarce.

Thanks to QPMS’ exceptional signal to noise, our 3D imager is able to recover the salient details of the target scene without the use of any post-selective filtering, despite overwhelming noise. In Fig. 4a, b, we compare the reconstructed 3D image without noise and with 1700 background photons per period, acquired with the same signal photon detection rate. Figure 4c shows the depth error (standard deviation) map between these two figures. As seen, the error occurs mainly around high contrast features of the mannequin, e.g., the nose and edge of the face, whose average depth (longitudinal) error is 1 mm. In general, the higher error in those locations is caused by the sharp angle of incidence and thus the dramatically reduced backscattered signal. Consequently, noisy pixels that are not reflecting the true surface morphology are dominant in those areas. Figure 4f gives an example of the time-scanned photon counts acquired by QPMS, where the photon counting peak is still identifiable despite orders of magnitude higher noise. Suppressing noise on the detection end will reduce the number of noisy pixels, allowing for image reconstruction with reduced error. Nevertheless, the errors in the reconstructed 3D image can be further minimized through post-selective filtering that distinguishes between signal and background detections by exploiting the scene’s transverse spatial correlation15.