Playing fast and dim

As the super-resolution microscopy revolution continues apace, the use of CMOS cameras is attracting a lot of attention… but to really progress the field, we need to extract more information from the available photons, enabling us to characterise fast and dim biological processes. An international team of scientists explain...  

Super-resolution microscopy encompasses a diverse range of technologies and techniques aimed at resolving structures below the diffraction limit of visible light.

Current solutions are based around detection using sensitive, but conventional, sCMOS or EM CCD cameras

Although much recent progress has been made in this field, many aspects of the technology remain to be improved. Continuous advances in the equipment and knowledge base has enabled scientists to resolve smaller structures in less time, detecting faster and dimmer events (e.g. weak fluorophore blinking).  Current solutions are based around detection using sensitive, but conventional, sCMOS or EM CCD cameras.

Here we introduce a novel approach using the Non-Destructive-Read (NDR) mode of CMOS (Complementary Metal–Oxide–Semiconductor) technology and demonstrate its advantages in fast and low-light applications. The focus will be specifically on its relevance to stochastic super-resolution microscopy and we present STORM data that was generated with a SciMeasure DaVinci-2K CMOS camera in Dr Cadby’s lab at Sheffield University in the UK. The data from Sheffield lab demonstrates that, compared to the conventional Correlated-Double-Sampling (CDS) readout mode, NDR can more than double the signal-to-noise (S/N) ratio and more than triple the number of events detected from data acquired during the same time window.¹

It’s all about the photons

Non-Destructive-Read (NDR) mode in CMOS cameras is in fact well known, and has existed since the technology first came about. When reading CMOS sensors, the largest source of noise is associated with clearing the charge from each pixel (resetting) before each integration period.

To reduce the overall noise, the reset voltage level needs to be sampled prior to data sampling and subtracted from each pixel. This process is known as Correlated-Double-Sampling (CDS), and the subtraction can be done either before or after digitisation. The NDR mode is an extension of this CDS concept, but involves only resetting the pixels once every preset number of frames.

As long as the accumulated photoelectrons don’t saturate the wells of the pixels, each data point accurately reflects the photon count at the time of sampling. The advantage of this approach is that it allows over-sampling over the integration period without destroying the photon signals by reset², offering ample freedom in post-acquisition analyses. Unfortunately, most newer high sensitivity CMOS, so called sCMOS, sensors implement “in-pixel CDS” which precludes the possibility of NDR.

Photon statistics dictate that the shot-noise is the square root of the number of detected photons. The total noise is the sum of the shot-noise and read-noise added in quadrature. When effective read-noise is low as in Electron Multiplied CCDs and low-noise CMOS or sCMOS sensors, shot-noise will be the dominant source of noise. Generally, more photons give a better S/N ratio. Due to problems of photobleaching and phototoxicity, biological fluorescence is inherently a low-light application, although in stochastic techniques such as STORM a very high intensity is required to generate weak blinks. This leaves the researcher with the burden of having to determine a frame integration period that will allow sufficient signal accumulation while providing adequate temporal resolution.

For many biological applications that are photon-starved, analog photoelectron accumulation with post-acquisition flexibility will be the most effective way to improve S/N ratio

Non-Destructive-Read (NDR), on the other hand, does not require such compromise. It allows analog accumulation of photoelectrons between resets and offers the flexibility of selecting the desired integration period in post-acquisition analysis to optimise the S/N ratio. For many biological applications that are photon-starved, analog photoelectron accumulation with post-acquisition flexibility will be the most effective way to improve S/N ratio.

Localisation microscopy

In STORM imaging (Stochastic Optical Reconstruction Microscopy) using EMCCD or sCMOS, a typical integration time is 30 msec per frame with approximately 10,000 frames needed to construct one super-resolution image. As mentioned, the integration time needs to be set in advance and a duration in the region of 30 msec is usually deemed to be a sensible compromise between S/N ratio and temporal resolution, even though sCMOS cameras are capable of being driven much faster. At an integration time of 30 msec per frame, one rarely sees blinks of the fluorophore that last longer than one frame, but rather we see frames with varying brightness, as the blink is often distributed over two frames.

At this frame rate, it is difficult to determine the time course of the blinking events or how many blinks there are. Although understanding the blinking behaviour of a fluorophore may not be critical to reconstructing the super-resolution image using existing algorithms, it is extremely important in helping us to improve the technology so we can achieve a better S/N ratio in less time.

 This requirement to identify a large number of brief, dim, and discrete events, combined with the need to repeat the process over thousands of frames suggested to us that stochastic super resolution imaging could be improved by using NDR

The resolution of localisation microscopies such as STORM is determined by the S/N ratio³. With the current trend in localisation microscopy moving towards the use of fluorescent proteins, which are intrinsically dimmer than dye based fluorophores, the ability to achieve greater S/N ratio becomes increasingly important. This requirement to identify a large number of brief, dim, and discrete events, combined with the need to repeat the process over thousands of frames suggested to us that stochastic supersuper-resolutioning could be improved by using NDR. Blink events are thought to produce in the range of 100 photoelectrons at the detector and the frequency of events on any given pixel is low, therefore there is ample well capacity (~15Ke-) to accommodate accumulation of a large number of frames between resets.

We tested the application of NDR mode in STORM using the new SciMeasure DaVinci-2K CMOS camera at a frame rate of 2,500 frames-per-second (fps). We were able to visualise the time course of blinks of various durations. The shortest blinks were as brief as 2-3 msec. In general, such lone brief blinks will fall below the noise floor in conventional CDS readout.

Figure 1 is a schematic illustration of how each blink would be sampled under two different readout modes and speeds, CDS at 33fps and NDR at 2500fps. Clearly, the CDS at slow frame rate can sometimes split a blink into 2 subsequent frames (as Blink A) and most often combines both the blinking and non-blinking periods within each sample (as in the blue CDS sample point that includes both Blink A and B and non-blinking period). On the other hand, NDR going at much higher frame rate can precisely determine the blinking period and allow us to use only these frames in post-acquisition analysis.

[caption id="attachment_61172" align="alignnone" width="890"] Figure 1 is a visualisation of a fluorophore's blink effects[/caption]

Since the signals were not destroyed by reset in NDR mode, subtracting the two bookend frames of this period is equivalent to having an integration period finely fitted to the blinking period. In this schematic figure, we did not include any read-noise from either camera, or background photon and its shot noise.

The idea is to introduce the concept of temporal oversampling using NDR mode and then choosing the integration period in post-acquisition analysis. And, at the end of this article, we will discuss in more technical detail how this method can help researchers to extract faster and weaker signals, such as briefer and dimmer blinks in STORM and PALM imaging, which would be missed using conventional sampling methods. Ideally, by implementing this method to extract more information from datasets, it will be possible to reconstruct super-resolved images in a much shorter time, or with better S/N ratio. By acquiring images at KHz rates would also dramatically reduce the risk of focal or sample drift.

Pick your moments

Obviously, the raw NDR data does not inherently offer the conventional movie display in the first place, but it does offer a large amount of useful data to researchers who are trying to characterise fast and dim biological events. The simple summary of the NDR approach is “oversampling and picking the best moment” in post-acquisition analysis. NDR allows high frame rate without clearing photoelectrons or losing photons. The continuous accumulation brings most data frames into the shot-noise limited regime. This could prove to be extremely useful in any application recording dim and fast events. Correctly implemented it would allow the researcher to make multiple post-acquisition analyses with different optimisations between temporal resolution and S/N ratio. These analyses could be varied either through the time series or within the field of view to allow different structures to be resolved and different time resolutions in the same image.

Dynamic fluorescence calcium and voltage sensitive dye measurements could benefit from this approach. Particle tracking would be especially interesting for systems where the time course of movement was highly variable, as the temporal resolution could be varied across the data set.  A second major benefit for particle tracking is the possibility to have subsequent frames separated by an interval shorter than the exposure time and hence truly overlapped exposure.  We would expect there to be many other applications and methods of interrogating the data set that we haven’t yet considered.

The main limitation on the use of this technology is in the handling of the large data sets. But with modern PCs, graphics cards and GPU programming the data processing will become much faster in the near future.

Authors: 

Chun X. Bleau, RedShirtImaging, Decatur, GA, USA

Jeremy Graham, Cairn Research, Faversham, UK

Ashley Cadby, Sheffield University, Sheffield, UK

Samuel F.H. Barnett, Sheffield University, Sheffield, UK

Charles Bleau, SciMeasure, Decatur, GA, USA

References

1. Barnett, S. F. H. et al. A Novel Application of Non-Destructive Readout Technology to Localisation Microscopy. Sci. Rep. 7, 42313; doi: 10.1038/srep42313 (2017). 2. Fowler B. Low Noise Wide Dynamic Range Image Sensor Readout using Multiple Reads During Integration (MRDI) 3. Thompson, R., Larson, D. & Webb, W. Precise Nanometer Localization Analysis for Individual Fluorescent Probes. Biophysical Journal 82, 2775–2783, 10.1016/S0006-3495(02)75618-X (2002).

Additional Information:

Methods and Results

The DaVinci-2K CMOS camera with NDR readout mode is provided by SciMeasure, Decatur, GA, USA. This camera can run at 100fps CDS and 200fps NDR in full frame mode. The sensor has 2048x2048 15um² pixels which are read out at 68MHz through 16 parallel ports, with 2.8e^- read noise. This is a traditional CMOS sensor with independent Reset and Sample control which allows it to be read non-destructively. The Quantum Efficiency (QE) is 65% without micro-lenses for better uniformity and Modulation Transfer Function (MTF). In our experiment, we mainly used a windowed-down configuration of 2048x180 at 2500fps NDR mode. This allows us to visualise the time course of brief blinks of a few milliseconds. The reset for the NDR mode was delivered every 100, 500 or 1000 frames, and this can be programmed differently according to the anticipated or measured light levels.

The camera has 16 parallel readout channels. Each channel has one amplifier and one 14 bit A/D outside the CMOS sensor. This architecture is designed to achieve high speed, uniformity, linearity, and monotonicity at the same time, and prevent spurious charges without any artificial image filtering. Experiments were conducted on a Nikon TiE inverted microscope with Perfect Focus System (PFS) hardware autofocus and a 100X 1.49 NA TIRF lens. The laser illumination was provided using a prototype LDI 630 – 640nm multimode laser Cairn Research / 89 North capable of delivering over 2W of power through a multimode fibre. The end of the fibre was focused onto the backplane of the objective lens and gave a uniform illumination over the sample with a maximum illumination intensity of 400 mW. A standard CY5 Nikon filter cube was used to separate the laser light from the resulting fluorescence signal. The biological samples used were tagged with Alexa Fluor 647 (Thermo Scientific), a common organic dye used in STORM microscopy.

Results and Discussion

CDS vs NDR

Figure 2 shows the raw NDR data (unfiltered, unprocessed) from 2 ROIs and 1 dark area within one NDR cycle. The data set was acquired at 2500fps in NDR mode with reset every 500 frames in a 2048x180 windowed-down configuration. The ramp of the signals in each waveform shows the accumulation of photons (ROI-red and ROI-green) as well as the background (the blue trace) over time after the reset of this NDR cycle. Consider ROI-red, which shows two short but distinct bumps on its time course waveform. These two bumps represent two blinks displayed in Frame B-A and Frame D-C as the bright spot corresponding to the ROI position marked by the red circle. Both blinks lasted about 5-6 frames and the first blink has about 120e-/pixel and the second has about 60e-/pixel, which is respectively 20e- and 12e-/pixel/frame. The background light level between the two blinks is 6e-/pixel/frame. The dimmer of the two blinks is only 30e- (6e-/pixel/frame) above the background fluorescence during the 2.5msec of blinking period. The best way to detect such dim blinking is to place the integration period right at the time when the blink occurs.

[caption id="attachment_61174" align="alignnone" width="999"] Figure 2 shows raw data from Non-Destructive-Read mode in CMOS cameras[/caption]

When using EMCCD and sCMOS, 30-msec integration duration is a typically elected frame interval as a trade-off between S/N and temporal resolution.  Here are three possible scenarios of how sampling occurs with fixed 30 msec integration time.

1. Single or partial brief blink (<integration time) occurred during the 30 msec period as depicted with Blink A and C in Figure 1.
2. Several short blinks occurred during the 30 msec period as depicted with Blink A and B in Figure 1.
3. A long blink (>integration time) occurred over more than one 30 msec integration time.

In all scenarios, NDR has the advantage over CDS approach. In both scenario 1 and 2, there are non-blinking periods integrated with the blinking period(s), adding the background shot noise with no improvement in the signal. In the scenario 3, the blinking will be split into two frames. Even though the two frames can be added digitally, the read-noise will be added as well in quadrature. In addition, part of both frames most likely includes non-blinking period which also adds background shot noise. The NDR readout mode, on the other hand, allows us to exclude these non-blinking periods which don’t bear any relevant signals. Hence, for the best-case scenario for conventional imaging where the blink occurs during a single frame:

(SN_b = Shot Noise during blinking period, SN_nb = Shot Noise between blinks, RN = Read Noise, n = number of frames averaged)

(Note: in the Da Vinci camera RN is equivalent between CDS and NDR modes as both modes involve subtracted reads)

Experiments carried out in Dr Cadby’s lab supported the notion that NDR mode at high frame rate allows us to see more events and to separate events better both spatially and temporally than non-NDR mode. They demonstrated that, compared to the conventional CDS readout mode, NDR can double the S/N ratio and more than triple the number of events detected from data acquired during the same time period¹. The increase in S/N ratio can be translated into the increase in resolution, which peaks at 10nm for NDR vs 20nm for CDS (Figure 3). Dr Cadby’s lab also compared the NDR readout with Andor Zyla sCMOS and showed that the S/N ratio using NDR is several fold better depending on the sampling rate (Figure 4).

[caption id="attachment_61251" align="alignnone" width="668"] Figure 3 shows how much more data is collected when using NDR to capture images compared to Correlated Double Sampling.[/caption]

 

Figure 4 shows the difference between CMOS and NDR images.

So far we have compared CDS at a slow frame rate with NDR at a very fast frame rate. One might raise the question as to what if CDS is used at high frame rate and a temporal averaging is applied post-acquisition (post-digitisation). The reality is, if CDS is used at 2500fps, the photon counts from individual frames would be so low that it is no longer shot-noised limited. Averaging these CDS frames will not help to extract the signal because read-noise from all the frames will be added in quadrature, while subtracting two bookend frames using NDR will only involve one pair of read-noise from the bookends as illustrated below:

(SN_b = Shot Noise during blinking period, RN = Read Noise, n = number of frames averaged)

Here we reiterate that the key advantage of NDR mode is analog summation over digital summation and ability of selecting integration period with precision after acquisition. During the NDR readout process, each read (without reset) does not destroy the charge in the pixel, therefore the data is preserved at the analog level. Each data point represents the photoelectron accumulation since the reset of the NDR cycle. In post-acquisition analysis, one can determine which frame is the beginning of an event and which is the end, and get the difference as the integration period. On the other hand, CDS destroy the charge after each read. In order to accumulate the same number of photons, all the frames will needed to be summed up digitally post-acquisition. The read noise will be added in quadrature.

Having stated the advantages of NDR mode, there are also challenges that come with it. Large amount of data from oversampling and unconventional data components are the two main issues. Powerful computers and software with automated analysis are needed to extract the information for real time super resolution imaging. This automation is currently being developed in Dr Cadby’s lab in Sheffield.

The current systems for STORM imaging – including dyes, laser setting etc – were tailored to have infrequent blinking rate for slow frame rate acquisitions. With the promise of high frame-rate temporal oversampling, it becomes desirable to search for new fluorescent markers, different laser setting, or buffer solutions that could boost blinking rate (faster and more frequent blinking), so that the imaging system can be improved as a whole.

Other factors when comparing camera architectures

1). Quantum Efficiency.

The DaVinci-2K camera does not score the highest in terms of QE, because it doesn’t use microlenses and is not back-thinned. It achieves relatively high QE by having large pixels with high fill-factor, but without the distortion or optical cross-talk associated with microlenses.  The technology could be improved further by back-thinning to improve QE and SciMeasure are investigating this possibility.

Even though a back-thinned chip is desirable for high QE, any improvement in S/N ratio achieved from higher QE is only a few percent. On the other hand, NDR mode can offer several fold of improvement depending on the frame rates that are compared. Clearly, higher QE increases S/N ratio by using more of the available photons. But NDR increases S/N ratio by excluding the background photons that do not bear relevant signals and only add shot-noise.

In addition, it has become increasingly evident that the pixel size is important in achieving good S/N ratio. And often the manufacturer’s quoted QE measurement, especially from cameras with microlenses, can be misleading, and does not necessarily reflect the practical performance of the camera in a microscope system.

2). Frame rate.

Before sCMOS cameras were developed, EMCCDs had been the primary choice of camera for super resolution imaging.  They are certainly very sensitive detectors, but the serial CCD readout makes them fundamentally frame-rate limited.

While sCMOS sensors can be driven at much higher frame rate, around 30 msec integration time is typically used in practice, mainly because higher frames will result in detection of fewer events and lower S/N ratio. Since sCMOS sensors have in-pixel CDS, they cannot be read with NDR readout mode. There is no other way to improve S/N ratio other than slowing down the frame rate or improving the QE by back-thinning.

3). Single A/D.

One of other differences between the new DaVinci-2K CMOS and existing sCMOS sensors is that DaVinci-2K has one 14 bit A/D per channel (16 channels in total) while sCMOS has two A/Ds each column, one for high gain and one for low gain. This is an effort to reduce read noise while preserving dynamic range. The disadvantage of such an approach is that the stitching of two A/Ds causes non-linearity, and sometimes, spurious charge, which will require calibration and filtering.