Quanta Camera

[Aiko Bartels]

Single-photon avalanche diodes (SPADs) are an emerging sensor technology capable of detecting individual incident photons, and capturing their time-of-arrival with high timing precision. While these sensors were limited to single-pixel or low-resolution devices in the past, recently, large (up to 1 megapixel) SPAD arrays have been developed. These single-photon cameras (SPCs) are capable of capturing high-speed sequences of binary single-photon images with no read noise. We present quanta burst photography, a computational photography technique that leverages SPCs as passive imaging devices for photography in challenging conditions, including ultra low-light and fast motion. Inspired by recent success of conventional burst photography, we design algorithms that align and merge binary sequences captured by SPCs into intensity images with minimal motion blur and artifacts, high signal-to-noise ratio (SNR), and high dynamic range. We theoretically analyze the SNR and dynamic range of quanta burst photography, and identify the imaging regimes where it provides significant benefits. We demonstrate, via a recently developed SPAD array, that the proposed method is able to generate high-quality images for scenes with challenging lighting, complex geometries, high dynamic range and moving objects. With the ongoing development of SPAD arrays, we envision quanta burst photography will find applications in both consumer and scientific photography. Source code: -burst-photography

Conventional image sensors need to collect 100-1000 photons in order to generate a clear image. Single-photon image sensors are novel sensors that are sensitive to the arrivals of single photon arrivals. They capture binary frames which record if a photon arrives at each pixel during the exposure time. If we take a continuous burst of binary frames, the average number of detected photons depends nonlinearly on the incoming light intensity, and a linear intensity image can be recovered by inverting this nonlinear response curve. However, this does not take into account possible scene/camera motion between binary frames, which results in motion blur.

We propose quanta burst photography, a computational photography technique that computationally re-aligns the photons along motion trajectories, for achieving high-quality images in challenging scenarios, including low-light and high-speed motion. We develop algorithms that align the binary frames, thus creating a high-bit-depth, high-dynamic-range, super-resolved image, while minimizing noise and motion blur.

We show experimental results for a range of challenging scenes, including depth variation, specular highlights, complex geometry and fine structures. Naive averaging without motion compensation results in heavily-blurred images. The proposed method is able to align the frames accurately and generate high-quality images with low blur and noise.

We capture a scene where the light source (the lamp) is directly visible in the image. Quanta burst photography achieves very high dynamic range and is able to recover the details of the filament and the text on the plaque at the same time.

Here we show a person plucking the lowest two strings of a guitar. Averaging the captured binary sequence results in either ghosting artifacts or a low SNR. Our method is able to reconstruct a clear, sharp image despite fast and non-rigid scene motion.

Beyond empirical results, we also propose a theoretical framework to analyze the SNR performance of conventional and quanta burst photography for different lighting conditions and motion speeds. Here we show some example plots showing the SNR difference between current/projected SPADs and commercial sub-electron CMOS sensors. This framework can be used to direct future development of single-photon cameras for best performance under certain light levels and amount of motion.

Quanta burst photography (QBP) is a computational photography technique that measures the motion between binary frames captured by single-photon cameras and merge them into a high-quality intensity image with low motion and blur and high dynamic range.

In this work we focus on scenes that contain significant motion. If we know a priori that the scene is perfectly still, then quanta burst photography will perform as well as a conventional camera assuming sufficient light in the scene. In case of extremely low illumination conditions, below the noise floor of conventional camera pixels, quanta burst photography can still provide improved performance due to its extreme sensitivity and low noise.

We have recently shown that, surprisingly, sensing individual photons can provide extended dynamic range under extremely high illumination conditions. By relying on the non-linear film-like response of these single-photon sensors, we show that it is possible to capture much higher brightness levels, orders of magnitude beyond the saturation limit of conventional image sensors. For more details please visit this project page: From One Photon to a Billion: High Flux Passive Imaging with Single-Photon Sensors.

We compare the performance of SPADs and jots-based quanta burst photography in simulated sequences. Under fast motion, the merged image from jots contains motion blur, while SPADs are able to register the binary images and merge them into a sharp image. On the other hand, when the motion is slow, jots are able to generate a sharper image due to their high spatial resolution. Therefore, we envision these two technologies to complement each other: SPADs achieve higher performance in high-speed scenarios, while jots with projected high resolution will achieve better image quality for scenes with relatively slow motion and high-frequency texture details. Comparison between the two technologies on real data remains for future work.

It is indeed difficult to match the binary frames directly due to their low dynamic range (1-bit) and highly randomness. Therefore, we divide the entire sequence into temporal blocks (e.g. every consecutive 100 frames), compute the sum image of each block and align the sum images. Then the motion field between blocks is further interpolated to get the intra-frame motion. Furthermore, a robust frequency-space merging algorithm is used to reduce the visual artifacts introduced by alignment errors.

The proposed algorithm assumes certain level of spatio-temporal smoothness on the image-space motion field (patchwise 2D translation and constant velocities within blocks). When this assumption does not hold, the discrepancies between the true deformation of the patch and the translation approximation can be mitigated by the robust merging algorithm. The proposed algorithm has shown robust performance for global 6DoF handheld camera motion and moderate amount of local, non-rigid scene motion (e.g. guitar player example). An interesting future research direction is to design optical flow algorithms for aligning images for challenging scenes including fast changing global motion (e.g. drone) and highly non-rigid scene motion (e.g. cloth, air turbulence).

Currently, our algorithm is not optimized for real-time implementation. Our MATLAB code takes about 30 minutes to process a sequence with 10,000 binary frames. One important future direction is to implement quanta burst photography in a fast and also energy-efficient way, which will be critical for consumer photography applications.

Our quanta burst camera prototype captures about 100,000 frames per second. The high dynamic range and temporal resolution of SPADs comes at the cost of large bandwidth requirement. Currently, the captured binary images are stored on a local memory on the camera board, and later transferred to a PC via USB 3.0 for offline processing. The bandwidth requirement can be relaxed to some extent by capturing multi-bit images and sacrificing some temporal resolution. The bandwidth in future SPAD sensors can also be improved by using faster interfaces such as PCI Express or CameraLink.

In this work we focus on imaging in passive illumination conditions where the camera does not have its own light source. Indeed, when used in conjunction with an active light source such as a pulsed laser, the high sensitivity and timing resolution of single-photon sensors can be used to capture high resolution 3D information about the scene. For more details about our work in 3D imaging with single-photon cameras, please visit our project page:

The hardware demonstration in our paper was limited to black & white photos because our prototype camera only captures monochrome images. This is not a fundamental limitation of our technique. Our technique can be easily extended to color imaging using standard color capture techniques widely used in conventional cameras (e.g., an RGB Bayer filter pattern on the image sensor).

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

To get more information about the recommended connection for using multiple cameras, I recommend reaching out to Quanta, the manufacturer. They should be able to provide guidance on the optimal setup for your specific camera configuration.

Considering the bandwidth limitation, it would be advisable to connect the three 8MP cameras to different camera groups to distribute the data load. Specifically, I suggest connecting one camera to group A, another camera to group C, and the third camera to group D.

csi-ab worked for a few minutes with 3 cameras then crashed. Trying other options as well, outside of the vendor letting us know, would that be stored in some nvsipl metadata we could look at on the system?

Choreographed camera movements recorded the energy of moving light. While wandering around reacting to light and choreographing camera movements during long exposures I lost track of the activity around me and felt like I was dancing with light. The experience was immersive. There was a mixture of both planning and spontaneity.

c80f0f1006