A time-domain projection-based registration-scene-based ...

OPTICAL REVIEW Vol. 21, No. 1 (2014) 17–26

A Time-Domain Projection-Based Registration-Scene-Based Nonuniformity Correction Technology and its Detailed Hardware Realization Ning LIU1
and Hang QIU2 1 2

Nanjing XiaoZhuang University, College of Physics and Electronics, Nanjing, Jiangsu Province 211171, China Nanjing Institute of Technology, Industrial Center, Jiangsu Province 211167, China

(Received September 5, 2013; Accepted October 22, 2013) Non-uniformity (NU) in infrared images can cause great degradation of the image quality and appearance. Scene-based nonuniformity correction (SBNUC) has become a very effective way to deal with NU. Although many SBNUC methods have been developed by researchers worldwide, few of them have a good correction performance and can be applied to small-package real-time devices. In this paper, we propose a time-domain projection-based registrationscene-based NU correction technology and its detailed hardware realization. We developed a new projection estimator to calculate the relative displacement of neighboring frames. The estimator uses a column and row projection vector to calculate the displacement separately without reducing the accuracy. We also developed an improved gain coefficient correction method using the corrected bias coefficient to correct the gain coefficient by clarifying the intrinsic relationship between these two coefficients instead of correcting them separately. We have also thoroughly analyzed how this technology performs with an actual infrared video sequence containing both low-frequency and highfrequency NUs. The hardware realization of this technology in a single-FPGA-core real-time system is also described in detail. We have successfully realized this technology in a real engineering application. Detailed flow charts for the hardware implementation of this algorithm are also provided. # 2014 The Japan Society of Applied Physics Keywords: time domain, projection estimator, non-uniformity correction, hardware realization

1.

advantage of simple calculation, but it requires too many raw images to participate in the correction process. It cannot correct the gain coefficient of NU, which may cause serious image degradation and artifacts.2) The neural network NUC algorithm uses steepest-descent iterations to update the correction coefficient. It adopts a neural network structure and sets the average of neighboring pixels as the expectation. This algorithm has the advantage of convenient calculation, but it also has the defects of image degradation and contour artifacts, not to mention a slow convergence speed and poor correction ability for low spatial frequency noise.6–8,11) Because of these defects, many similar algorithms with additional restrictions derived from a neural network become less effective when the scene has not changed for a long time. The CS algorithm was established with the knowledge that the statistical mean value and deviation of all the pixels in statistical frames tend to be equal. It has a good correction performance for low spatial frequency noise, but it takes much longer calculation time to conduct the statistical process for a scene and also has the problem of artifacts.13–15,17,19,25,26) Meanwhile, artifacts appear as the reverse image of the previous scene when using the CS algorithm, which has a serious impact on the visual performance. In contrast to these algorithms, registration NUC uses image registration technology to set up a correspondence between pixels in the image sequence, while simultaneously updating the correction coefficients. The convergence speed of the registration algorithm is much faster than the other methods. It only takes dozens of frames or even less to estimate the correction coefficients. Moreover, it has much better image restoration quality than the other methods.20,24)

Introduction

Focal-plane array (FPA) sensors have become the most commonly used detector in infrared (IR) and visible-light imaging systems in recent years. However, FPA sensors, especially IRFPA sensors are strongly affected by the nonuniform spatial response of each pixel sensor which arises owing to the different responses of each photodetector within an FPA sensor.1,2) This nonuniformity (NU) problem is the main cause of fixed pattern noise (FPN) which is easily noticeable in the acquired images. Furthermore, for uncooled IR cameras, the problem is even worse because the NU of the sensor response is not stationary and often drifts with time. For this type of camera, scene-based nonuniformity correction (SBNUC) is required throughout the sensor operation to eliminate the influence of temporal drift without interrupting its normal operation. Scene based NU algorithms have been developed rapidly in recent years. They can be divided into several categories, such as the time-domain high-pass-filter NUC algorithm,1–4) the neural network NUC algorithm,5–12) the constant statistical NUC algorithm,13–19) and the image registration NUC algorithm.20–24) All these algorithms have their own advantages and disadvantages. For example, the timedomain high-pass-filter NUC algorithm is focused on correcting the offset coefficient of NU. It is based on constructing a time-domain high-pass filter to calculate the mathematical expectation of raw images. It is similar to some types of constant statistical (CS) method, and only increases the expectation value. This algorithm has the


E-mail address: [email protected] 17

18

OPTICAL REVIEW Vol. 21, No. 1 (2014)

N. LIU and H. QIU

Although the registration NUC algorithm has its own advantages, it has still not been realized on a single-FPGAcore, small-package and low-power-consumption system because of some defects: 1. the currently used registration algorithms are mostly based on the Fourier transform, which is used to calculate the cross-correlation coefficient of the neighboring frames.21) Once the relative displacement of the neighboring frames has been calculated, the post-registration image can be acquired through the inverse Fourier transform. This requires too much computation for a singleFPGA-core system. Thus far, no report has indicated that a registration-scene-based NUC algorithm can be realized in a single-FPGA-core system. 2. If a digital signal process (DSP) is used for the computation, the system will require greater power consumption and a larger structure, making miniaturization difficult. With this background, we have developed a new projection-based estimator to calculate the relative displacement. We have also devised an effective way to correct the gain of raw images. This algorithm is easier to perform than the traditional two-dimensional (2D) registration algorithm owing to its computational method. More importantly, it has been fully realized in a singleFPGA-core system. This paper is organized as follows: in Sect. 2, the correction model is outlined and the algorithm is discussed; in Sect. 3, the registration accuracy of our algorithm and the gain adjustment performance are analyzed; in Sect. 4, the hardware implementation of this algorithm is given in detail, and in Sect. 5, we give a conclusion and perspective of the study. 2.

Here wn ði; jÞ and bn ði; jÞ are respectively the NUC gain and offset of the linear correction model of the ði; jÞth detector. Their relations with the real gain and offset can be represented by 1 ; ð3Þ wn ði; jÞ ¼ gn ði; jÞ on ði; jÞ bn ði; jÞ ¼  : ð4Þ gn ði; jÞ Thus, we can conduct the NUC using Eq. (2) when we acquire the ideal estimation of wn ði; jÞ and bn ði; jÞ or gn ði; jÞ and on ði; jÞ. 2.2 New time-domain projection estimator In the registration scene based NUC algorithm, the most important task is calculating the relative displacement of neighboring frames. As mentioned above, in the traditional 2D registration NUC, the 2D Fourier transform is used to calculate the cross-correlation coefficient to determine the displacement.21) This method is very difficult for an FPGA to conduct. However, Cain et al.25) provided us with inspiration by performing a 1D cross-correlation calculation to determine the displacement using the column and row projection of a raw image. Following this inspiration, we describe our new projection estimator as below: 1. If the dimensions of an image are M  N, let the projections of the image, px , x ¼ 1; 2, in the horizontal and vertical directions be defined, respectively, by Px ðiÞ ,

Discussion

In this section, we focus on two aspects: 1. the new timedomain projection estimator; 2. the new gain adjustment method. They are discussed as follows. 2.1 Correction model of non-uniformity Generally, the relationship between the signal response and the incident IR photon flux is nonlinear, especially when an FPA operates in a wide dynamic incident flux range.16,27) For SBNUC, to simplify the problem formulation, the photo responses of the individual detectors in an FPA are commonly approximated using a linear irradiance-voltage model17) and their output is given by Yn ði; jÞ ¼ gn ði; jÞ  Xn ði; jÞ þ on ði; jÞ:

ð1Þ

The subscript n is the frame index. gn ði; jÞ and on ði; jÞ are respectively the real gain and offset of the ði; jÞth detector. Xn ði; jÞ stands for real incident IR photon flux collected by the respective detector. This model is reasonable especially for some SBNUC methods with a fast convergence rate, because during a short period of time, the temperature of an object can be ensured to remain in a small range so as to satisfy the linear response model. We apply a linear mapping to each observed pixel value to provide an estimate of the true scene value so that the detectors appear to be performing uniformly. Xn ði; jÞ ¼ wn ði; jÞ  Yn ði; jÞ þ bn ði; jÞ:

ð2Þ

Py ð jÞ ,

M X

i¼M

Pði; jÞ 

j¼N X

j¼1

i¼1 j¼1

N X

j¼N X

i¼1

Pði; jÞ=M;

ð5Þ

Pði; jÞ=N:

ð6Þ

i¼M

Pði; jÞ 

i¼1 j¼1

Here, Pði; jÞ is the 14-bit pixel value of the raw image. Px ðiÞ is the total sum in the column direction, and Py ð jÞ is the total sum in the row direction. 2. In any real imaging system, there will be new information entering the field of view. The projection estimator is sensitive to this new information which causes the registration to be less valid. Cain et al. designed a sliding window filter and added it to the projection estimator to enable the algorithm to ignore the new information. We find that the computation of this filter is still too demanding for an FPGA and it is difficult to realize only using the hardware description language (HDL). In this case, we use a cosinesine fixed window filter as a substitution. This window filter can be described as follows. For column projection: 8   1 þ cosð  ðrow  1  iÞ=row Þ > > px ðiÞ  > > > 2 > > < i < row   ; ð7Þ pix ¼ > 1  sinð  ðrow  1  iÞ=row Þ > > px ðiÞ  > > 2 > > : i > N  row

N. LIU and H. QIU

OPTICAL REVIEW Vol. 21, No. 1 (2014)

For row projection: 8   1 þ cosð  ðcol  1  jÞ=col Þ > > p ð jÞ  > y > > 2 > > < j 1  sinð  ðcol  1  jÞ=col Þ > > py ð jÞ  > > 2 > > : j > M  col Here, row and col are respectively the preset maximum shifts in the vertical and horizontal directions. Assuming the column and row displacements of the two neighboring frames are within the preset shifts row and col , it means that the new information mentioned above would appear in this region. We use the cosine-sine fixed window filter to lower the interference probability for the upcoming correlated calculation. The picture illustration is demonstrated in Fig. 1: 3. We need two neighboring frames to determine the relative displacement. That is to say, we have to acquire two frames and set one of them as the reference frame and the other as the current frame. We use their projection estimators to calculate the cross-correlation matrixes as follows: Cof row ¼

i¼2row þ1 j¼N2 Xrow þ1

19

(a)

ðref row ð j þ i  1Þ

i¼1; j¼1

 currow ðrow þ iÞÞ2 ; Cof col ¼

ð9Þ

i¼2col þ1 j¼N2 Xcol þ1

ðref col ð j þ i  1Þ

i¼1; j¼1

 curcol ðcol þ iÞÞ2 :

ð10Þ

Every possible displacement is stored in the matrixes Cofrow and Cofcol . Once the cross-correlation matrixes have been determined, we can obtain the relative displacement by choosing the minimum value among these two matrixes: drow , arg minðCof row Þ;

ð11Þ

dcol , arg minðCof col Þ:

ð12Þ

We use some figures to illustrate the above discussion. Let us take the column direction as an example. Supposing the blue line is the column projection estimator of the reference frame and the red line is the column projection estimator of the current frame. The result in Fig. 2 is acquired from Eqs. (5) and (6). It can be seen from Fig. 2 that the two frames have a displacement in the column direction. Then we calculate the correlation coefficient to determine the relative displacement using Eqs. (9) and (10). The results are shown in Fig. 3, in which all the possible coefficients are calculated and plotted. The green line is determined as the most precise one. The coefficient of the green line should satisfy Eq. (12). Thus, we use Eqs. (11) and (12) to determine the final displacement between the reference frame and the current frame. The calculation of the row displacement is the same as that of the column displacement. 2.3 New gain correction method After the relative displacement of the neighboring frames has been determined, we can continue updating the

(b) Fig. 1. (Color online) Picture illustration. (a) Projection illustration of Eqs. (5) and (6). (b) Filter illustration of Eqs. (7) and (8).

correction coefficient in the overlap area of these two frames. The traditional 2D registration NUC algorithm and the neural network algorithm both use the steepest descent method21–23) to correct the gain and offset coefficients. Here, we consider how to execute this method in the 2D registration NUC algorithm. The basic description of the 2D registration NUC algorithm is that it uses the amplitude spectrum of the Fourier transform of the two neighboring frames to determine the overlap area. Then it conducts the NUC within the area using the least mean squares (LMS) algorithm to converges the gain and offset coefficients. The high-frequency NU can be corrected rapidly by this method in the case of precise registration. However, the lowfrequency NU is difficult to correct at the same time. This is because the high-frequency NU usually exists in the offset

20

OPTICAL REVIEW Vol. 21, No. 1 (2014)

N. LIU and H. QIU

(a)

(b)

Fig. 2. (Color online) Column projection of reference and current frame.

(c) Fig. 4. (Color online) The correction result of steepest-descent method with different steps in 2D registration NUC algorithm. (a) Raw image. (b) Convergence step is set to 0.01 using the steepest-descent method. (c) Convergence step is set to 0.1 using the steepest-descent method.

the neighboring frames should have the same output in the overlapping area. These assumptions can be written as the following equations. y1 ¼ y2 ; y 1 ¼ k 1  x 1 þ b1 ; Fig. 3. (Color online) Correlation calculation and determination of displacement.

coefficients while the low-frequency NU usually exists in the gain coefficients. However, when using similar equations to correct the gain and offset coefficients, the correction offset coefficients is faster than that of the gain coefficients. When the high-frequency NU has been corrected, the lowfrequency NU is still remains in the image. Figure 4 shows a set of images we acquired by using the 2D traditional registration NUC algorithm with the steepest-descent method. The convergence step in Fig. 4(b) is set to 0.01, while in Fig. 4(c) is set to 0.1. We preset the convergence step to different values and then correct 200 frames. It can be seen from Fig. 4 that the convergence step is set to 0.01, there is a large amount of low-frequency NU remaining in the image as plotted in red circles. If the step is set to 0.1, there is still low-frequency NU remaining in the image, although some of it has been corrected. We can find that, in Fig. 4(c), the red plots circled some newly income NU. This is because the convergence step of 0.1 is set too high for the correction. In our research, we use the following new method to correct the gain coefficient. Supposing the two neighboring frames have an overlapping area in a scene, that is to say, the scene radiation in this area should theoretically be the same. After the NUC,

ð13Þ

y 2 ¼ k 2  x 2 þ b2 : Here, x1 and x2 are the same scene in the overlapping area of two neighboring frames, y1 and y2 are the outputs of the neighboring frames in the overlapping area, and k1 , k2 , b1 , and b2 are the gain and offset coefficients of the two neighboring frames. Following these equations, the following can be deduced easily: k2 ¼ k1 

b2  b 1 : x^

ð14Þ

The correction of offsets b1 and b2 in Eq. (14) must carried out before the gains k1 and k2 , which can be easily realized. Supposing the image represents by y2 is displaced towards the image represents by y1 , and the overlapping area of these two frames is defined as ERR. Then, the offset correction can be given as follows: 8 < bn ði; jÞ þ   ERRði; jÞ when pixel ði; jÞ is in the overlapping area : bnþ1 ði; jÞ ¼ : bn ði; jÞ otherwise ð15Þ Here, x^ is the average of x1 and x2 . From Eq. (14), we can see that the gain adjustment can be completed with the offset adjustment. We complete the equation by adding a convergence step to it, and finally we have the following equation:

OPTICAL REVIEW Vol. 21, No. 1 (2014)

(a)

N. LIU and H. QIU

(b)

(a)

(b)

(c)

(d)

(e)

(f)

21

Fig. 5. Correction result using Eq. (16) when convergence step is set to 0.1. (a) Raw image. (b) Correction result.

knþ1 ði; jÞ ¼

8 bnþ1 ði; jÞ  bn ði; jÞ > > < kn ði; jÞ    xð^; jÞ

: when pixel ði; jÞ is in the overlapping area > > : kn ði; jÞ otherwise ð16Þ The coefficient  is the convergence step in Eqs. (15)– (16). We redo the above correction by substituting the gain adjustment in Eq. (16) to obtain the result shown in Fig. 5. In Fig. 5, we can see that the correction performance of low-frequency NU is better than that shown in Fig. 4, especially the NU in the central part of the image. The performance will be analyzed in Sect. 3. 3.

Analysis of Performance of Projection Estimator and Gain Adjustment Method

The main differences between our algorithm and the traditional 2D registration-scene-based algorithm are that we develop a new projection estimator to determine the relative displacement instead of using the Fourier transform and we use a new way to adjust the gain coefficient. In this section, we analyze the performance of these two aspects. 3.1 Analysis of the performance of the projection estimator Our research group has already analyzed the registration accuracy of the 2D registration NUC algorithm using an MAE index.21) In this part, we mainly focus on comparing the results of our projection estimator and the 2D registration algorithm for the same video sequence and analyze its performance with another index. First, we compare the registration performances of our projection estimator and the 2D registration for test image pairs with different levels of NU as shown in Fig. 6, and we artificially shift the reference image by a fixed displacement. It is well known that, NU will causes a computational error when conducting the registration. Here, we consider three different NU levesl to compare the registration accuracy of the 2D registration and our new projection estimator. The results are shown in Table 1. It can be seen from Table 1 that two registration algorithms have almost the same registration accuracy. They both perform well when the NU is weak. Meanwhile, when the NU is strong, both of them have a certain error. Figure 7 shows the real image sequence. We find that the error for the two algorithms is still almost the same. This

Fig. 6. Test image pairs with different levels of NU. (a), (b) Test image pair without NU. (c), (d) Test image pair with weak NU. (e), (f ) Test image pair with strong NU.

means that our new algorithm can work as well as the traditional 2D registration algorithm. However, more importantly, our new method has the advantages of less computation and ease of realization in a hardware system. In Fig. 7, the Z-axis is the frame number, the X-axis is the column displacement, and the Y-axis is the row displacement. The blue dots are acquired using the 2D registration algorithm and show the displacement between neighboring frames. The red dots are the comparative results acquired using our projection estimator. (a) shows a comparison of the results NU, and (b) shows the results under strong NU. It can be seen in Fig. 7 that regardless of the level of NU, the red dots almost coincide with the blue dots. That means that our projection estimator has the same accuracy as the 2D registration algorithm. Although the estimator is calculated in the time domain, it still has the same stability as that calculated using the 2D Fourier transform in the frequency domain. Next, we use another index to describe the registration accuracy of our projection estimator. We use two neighboring frames to conduct the registration; thus, the two frames must have an overlapping area that is facing the same scene. We subtract the overlapping area in the neighboring frames and obtain an error matrix. Then we calculate the root-mean-

22

OPTICAL REVIEW Vol. 21, No. 1 (2014) Table 1.

N. LIU and H. QIU

Registration results of image pairs with different NU levels in Fig. 6.

Image pair

NU level

Real displacement

2D-NUC

Our method

Figs. 4(a) and 4(b) Figs. 4(c) and 4(d) Figs. 4(e) and 4(f )

none weak strong

row ¼ 8, col ¼ 11 row ¼ 8, col ¼ 11 row ¼ 8, col ¼ 11

row ¼ 8, col ¼ 11 row ¼ 8, col ¼ 11 row ¼ 8, col ¼ 12

row ¼ 8, col ¼ 11 row ¼ 8, col ¼ 11 row ¼ 8, col ¼ 12

(a)

(b)

(c)

(d)

(a)

(b) Fig. 7. (Color online) Comparison of displacement between our projection estimator and the 2D registration algorithm.

square value of the difference between all the pixels in the error matrix and the average of the error matrix using Eq. (17). vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi u M;N u 1 X ð17Þ ðXði; jÞ  Xavg Þ2 : RMSERR ¼ t M  N i; j In this equation, Xði; jÞ is the subtracted pixel value of the error matrix. Xavg is the average of the error matrix, and M and N are the dimensions of the error matrix, which vary with the displacement. The calculation of this index is based on the assumption that if the registration of neighboring frames is accurate, then RMSERR should be small. An example is shown as follows: Figures 8(a) and 8(b) show a test image pair with displacement both in the column and row directions. We artificially simulate an incorrect displacement to calculate the error matrix, whose RMSERR value is 401.1823. If the displacement is correct, the value drops to 50.4369. This index can also be used as a criterion to judge whether the displacement of the neighboring frames is correct. If the value exceeds a preset threshold, the correction should not be carried out at this time.

Fig. 8. RMSERR calculation of an overlapping area. (a), (b) Test image pairs with displacement in both column and row directions. (c) Error matrix with wrong displacement (RMSERR ¼ 401:1823). (d) Error matrix with correct displacement (RMSERR ¼ 50:4369).

3.2 Performance of the non-uniformity correction In this part, we analyze the NU correction performance of our algorithm by comparing it with the traditional 2D registration NUC algorithm. Usually, the RMSE index is used to determine the performance of an SBUC algorithm.21) The calculation of the RMSE index requires two types of image, that is, the raw image with no NU and the same image with NU added to it. Then, the RMSE calculated as. sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 X ðXði; jÞ  Xð^; jÞÞ2 ; ð18Þ RMSE ¼ M  N i; j where M and N are the dimensions of the images, Xði; jÞ is the raw value of pixel ði; jÞ, and Xð^; jÞ is the corrected value of pixel ði; jÞ. In our study, the image sequence we use is raw data with NU that is directly captured using a thermal imager without any correction. Thus, if we use Eq. (18) to determine the RMSE index, the curve should have a rising trend, shown in Fig. 9. In Fig. 9, we corrected nearly 200 frames. The green curve is plotted using our correction algorithm with a convergence step of 0.1. The red one is plotted using the traditional 2D registration algorithm with a convergence step of 0.1, while the blue one is plotted using the traditional 2D registration algorithm with a convergence step of 0.01. We can see from Fig. 9 that increasing the step by 10 times in the 2D algorithm, the RMSE value has not increased much

OPTICAL REVIEW Vol. 21, No. 1 (2014)

Fig. 9.

N. LIU and H. QIU

23

(Color online) RMSE index of our algorithm and the 2D registration NUC algorithm.

Fig. 10. (Color online) RMSE for different convergence steps in our algorithm.

better. This is because although the high-frequency NU is corrected rapidly, the low-frequency NU is not corrected significantly at the same time. The speed of gain adjustment is lower than the offset adjustment. However, in our algorithm, the gain adjustment follows the offset adjustment. We set the convergence step to control the adjustment speed so that it can match the pace of offset adjustment. This is highly effective in correcting the low-frequency NU. We can also see from Fig. 9 that when the convergence step is set to the same value, the correction performance of our algorithm is still better than that of the tradition 2D registration NUC algorithm. We set the convergence step  used in Eq. (15) and finally obtain Eq. (16) as the new gain adjustment equation. Here, we consider the effect of various values of the convergence

step. We capture a 600-frame video sequence with real NU as the test sequence. The original NU level is the same as that in Figs. 4 and 5. We set the convergence step to 0.5, 0.1, and 0.01 to correct the test video. The RMSE results are plotted in Fig. 10. In Fig. 10, we can see that the convergence step affects the convergence speed and correction stability under the same level of NU. Although a larger step increase the correction speed, it also degrades the stability of the curve. That is to say, when we use this algorithm in a real system, we have to choose a suitable convergence step depending on the NU level. If the initial NU level is high, a larger step should be chosen. After correcting dozens of frames, the NU becomes weak and the step should be decreased. If the initial NU is weak, a smaller step should be chosen.

24

OPTICAL REVIEW Vol. 21, No. 1 (2014)

(a)

(b) Fig. 11. (Color online) Hardware system. (a) Size of our hardware system in comparison with a one dollar coin. (b) Location of our system.

4.

Hardware Realization of Our Algorithm

In this section, we focus on the hardware realization of this algorithm. As mentioned in Sects. 1 and 2, although there are many scene-based NUC algorithms, we still face some problems when applying them. Some of these algorithms are less effective, some of them can hardly be realized such as the traditional 2D registration. Nowadays, the development tools available for researchers are becoming increasingly smart, making that algorithms increasingly complex. They can hardly be used without an intelligent terminal such as a PC. Let us take the traditional 2D registration-scene-based NUC as an example. We can hardly realize the 2D Fourier transform of an image on a platform with only one FPGA as a CPU. However, in real applications, we often have to build a small-package, lowpower-consumption system to meet the needs of a natural environment. Our new registration-scene-based NUC algorithm can meet these needs. 4.1 Hardware system description Before we detail the realization of our algorithm, we first describe the hardware system that we built. Figure 11 shows a photograph of our system. In this system, an EP2C20F484I8 FPGA is used as the core to conduct the image processing. Two SRAMs are used to store the temporal image and correction coefficient data. The frame frequency is set to 50 Hz under a 10 MHz pixel clock. The total power consumption of this system is less than 2.4 W under 6 V external voltage input. 4.2

Detailed realization of our algorithm in the hardware system The whole data stream and algorithm structure is shown in Fig. 12.

N. LIU and H. QIU

When the image stream enters in the algorithm module, the algorithm starts. 1. Each raw pixel value is sent to the display module and archived in the Image fifo SRAM after it is corrected using the parameters in the Coefficient SRAM. These two operations are parallel to fit the data valid time in a single frame. 2. The Image fifo SRAM has been divided into two regions to archive the incoming neighboring frames. Once the incoming frames have been archived in either of these regions, the algorithm is applied to it to continue the computation. We use a 40 MHz clock which is 4 times to the incoming clock (10 MHz) for the image stream to run the algorithm. Thus, the computation is only processed in the invalid time of each frame So that the algorithm can be real-time realized. 3. As discussed in Sect. 2 and from the data flow shown in Fig. 11, the image has to be projected horizontally and vertically. After the cosine-sine fixed window filter, the column and row projection is archived in four internal RAMs for the cross-correlation calculation to acquire the relative displacement between the reference frame and the current frame. In the real-time hardware system, we use a structure to greatly increase the calculation efficiency. Since the image dimensions are M  N, we can obtain a column vector with dimensions of M  1 and a row vector with dimensions of 1  N. Taking the column vector as an example, every time we obtain a projection point in the M  1 vector after 320 master clock cycles (40 MHz), we immediately conduct filtering by applying the data in the look up table (LUT) in one master clock cycle. We preset the value of the cosinesine filter in the LUT. Then we archive the filtered projection point in the internal RAM in one master clock cycle. The depth of the internal RAM is set to exactly the dimensions of the column and row vectors. Thus, for an image with dimensions of M  N, it only takes M  2 þ N  2 more master clock cycles to finish step 3. The simplified timing scheme is shown in Fig. 13. 4. Once two neighboring frames are being archived into the Image fifo SRAM, we will use their filtered projection vectors to determine the relative displacement by Eqs. (9)–(12). We find that each frame can be used as the reference frame during the calculation. Therefore, once we obtain a new pair of filtered projection vector, we can conduct the displacement calculation, and thus can keep the algorithm working in every frame. After the displacement has been calculated, we determine the overlap area of the neighboring frames and then update the correction coefficient in the Coefficient SRAM. Here, we use a pipeline structure to increase the processing speed. Once the overlap area has been determined, the address of the area in both regions in the Image fifo SRAM is also determined. We read a pair of matching points in the two regions

OPTICAL REVIEW Vol. 21, No. 1 (2014)

N. LIU and H. QIU

25

Fig. 12. Data stream and structure design of the time-domain registration-scene-based NUC algorithm.

Fig. 13.

Simplified timing scheme of step 3 with the column projection as an example.

Fig. 14.

(Color online) Process of step 4.

simultaneously, and then immediately calculate the error matrix. After the calculation, we read the original coefficient from the Coefficient SRAM to finish the update. The process is shown in Fig. 14. The simplified timing scheme is shown below: In Fig. 15, 01, 02. . .N represent the matching points and T is the error matrix threshold. In the FPGA, we can only process the integral values. However, in Eq. (16), the step  should be a decimal. Thus, we have to multiply the step by

32 to convert the decimal to an integer. Then we carry out signed-multiple (signed-mult) operation to complete the part after the minus sign in Eq. 16. After this, we have to shift the signed-mult result to fit in the original gain coefficient scale and conduct a signed-add operation to complete the coefficient update. 5. After updating the correction coefficient, the next incoming frame will uses it to conduct the correction, and steps 1, 2, 3, and 4 are repeated again.

26

N. LIU and H. QIU

OPTICAL REVIEW Vol. 21, No. 1 (2014)

Fig. 15.

Simplified timing scheme of step 4.

in the application. For example, when the background is too clean with very few scenes in it or there are too many blind pixels on the focal plane array, the effectiveness of the algorithm is reduced and the algorithm may fail. Also, this algorithm is modeled on the Cartesian coordinate system. Thus, if a scene is rotated or scaled, the technology cannot perform correctly. Moreover, as mentioned in Sect. 3, the convergence step should be chosen carefully considering the original non-uniformity level of the scene. We are still working on establishing a criterion for judging the nonuniformity level from the very first raw image when conducting the algorithm in a real system. These defects need to be overcome in future work. References

(a)

(b)

Fig. 16. Performance of the registration scene based NUC algorithm in a real-time hardware system. (a) Raw image. (b) Correction result.

4.3 Hardware performance of this technology Next, we demonstrate how this algorithm performs in a real-time hardware system. Figure 16 shows that even though the NU is strong in the raw image, this algorithm can still work effectively. In the real-time hardware system, we set the learning step to 0.1 in Eq. (16). Figure 16 is captured by the hardware and transferred to a PC for storage. This figure shows a comparison of the first and last frames in the hardware process. The total numbers of frames in the hardware process is less than 200. We can find from Fig. 16 that the original NU level is very strong. It not only has the high-frequency NU but also has strong lowfrequency NU. When the correction process starts, the highfrequency NU is first corrected and then the low-frequency NU. Figure 16 shows that both two types of NU have been corrected within 200 frames. After that, as the algorithm continues, the coefficient maintains high stability without divergence. 5.

Conclusions

In this paper, we proposed a registration-scene-based nonuniformity correction algorithm using a new projection shift estimator and a new gain adjustment method. This algorithm can meet the needs of a real-time hardware system with the characteristics of a small package and low power consumption. In fact, our research group has recently designed and manufactured a sophisticated IR guided system for an particular application using this algorithm, and achieved great success. However, this algorithm still has a few defects

1) D. A. Scribner, K. A. Sarkaday, J. T. Caulfield, M. R. Kruer, G. Katz, and C. J. Gridley: Proc. SPIE 1308 (1990) 224. 2) W. Qian, Q. Chen, and G. Gu: Opt. Rev. 17 (2010) 24. 3) W. X. Qian, Q. Chen, and G. H. Gu: Chin. Opt. Lett. 051103 (2011) 1. 4) W. Qian, Q. Chen, J. Bai, and G. Gu: Appl. Opt. 50 (2011) 1. 5) D. A. Scribner, K. A. Sarkady, M. R. Kruer, J. T. Caulfield, J. D. Hunt, and C. Herman: Proc. SPIE 1541 (1991) 100. 6) D. A. Scribner, K. A. Sarkaday, J. T. Caulfield, M. R. Kruer, J. D. Hunt, M. Colbert, and M. Descour: IEEE 93 (1993) 1955. 7) E. Vera and S. Torres: EURASIP: J. Appl. Signal Process. 2005 (2005) 1994. 8) S. N. Torres, E. M. Vera, R. A. Reeves, and S. K. Sobarzo: Proc. SPIE 5076 (2003) 130. 9) E. Vera and S. Torres: ICIP 7803 (2003) 8. 10) A. Rossi, M. Diani, and G. Corsini: Proc. SPIE 7834 (2010) 78340D. 11) T. Zhang and Y. Shi: Opt. Eng. 45 (2006) 1. 12) A. Rossi, M. Diani, and G. Corsini: Electron. Lett. 46 (2006) 348. 13) J. G. Harris and Y. M. Chiang: IEEE 8 (1999) 1148. 14) J. G. Harris and Y.-M. Chiang: Proc. SPIE 3061 (1997) 895. 15) M. M. Hayat, S. N. Torres, E. Armstrong, S. C. Cain, and B. Yasuda: Appl. Opt. 38 (1999) 772. 16) B. M. Ratliff, M. M. Hayat, and R. C. Hardie: J. Opt. Soc. Am. A 19 (2002) 1737. 17) J. G. Harris and Y.-M. Chiang: Proc. SPIE 3377 (1998) 106. 18) B. M. Ratliff, M. M. Hayat, and J. S. Tyo: J. Opt. Soc. Am. A 20 (2003) 1890. 19) B. M. Ratliff, M. M. Hayat, and J. S. Tyo: J. Opt. Soc. Am. A 22 (2005) 239. 20) C. Zuo, Q. Chen, G. Gu, and X. Sui: Opt. Commun. 285 (2012) 2293. 21) C. Zuo, Q. Chen, G. Gu, and X. Sui: J. Opt. Soc. Am. A 28 (2011) 1164. 22) C. Zuo, Q. Chen, G. Gu, X. Sui, and W. Qian: Opt. Eng. 50 (2011) 087006. 23) C. Zuo, Q. Chen, G. Gu, and W. Qian: Opt. Rev. 18 (2011) 197. 24) Y.-J. Liu, H. Zhu, and Y.-G. Zhao: Appl. Opt. 48 (2009) 2364. 25) S. C. Cain, M. M. Hayat, and E. E. Armstrong: IEEE 10 (2001) 1860. 26) C. Zuo, Q. Chen, G. Gu, X. Sui, and J. Ren: Infrared Phys. Technol. 55 (2012) 263. 27) C. Zuo, Y. Zhang, Q. Chen, G. Gu, W. Qian, X. Sui, and J. Ren: Infrared Phys. Technol. 60 (2013) 190.