Practical calibrations for a real-time digital omnidirectional camera Harsh Nanda Department of Computer Science University of Maryland
[email protected]
Ross Cutler Microsoft Research Microsoft
[email protected]
Abstract
timal image quality (as opposed to conservatively fixing the camera parameters and post-processing the image with the CPU).
Newly available high quality inexpensive digital 1394 cameras enable cheap and easy construction of omnidirectional cameras. We address several important issues for creating real-time panoramas, including user-assisted focusing, devignetting, and computer controlled brightness, contrast, and white balance. We demonstrate remarkable improvements in panoramic image quality using methods requiring very little CPU usage.
Our system runs on a 1GHz Pentium III PC in Windows 2000, and is implemented using DirectShow. The color calibration program is written so that the user only need press a single button to color calibrate the system.
1. Introduction
2. Related Work
There has been recent interest in the use of omnidirectional cameras for computer vision applications such as surveillance, collision avoidance, and office meeting recording. Newly available high quality inexpensive digital 1394 cameras, such as those based on the TI TSB15LV01, enable cheap construction of omnidirectional cameras. For example, Figure 1 shows an omnidirectional camera, the RingCam, built using five $60 (retail) 1394 board cameras. The RingCam uses daisy-chained 1394 cameras, so that only a single 1394 cable is required to interface to the PC for both data and control signals, which is both convenient and inexpensive. In this paper, we address several important issues for creating real-time panoramas with the RingCam. First, because the 1394 cameras we utilize use inexpensive microlenses which are difficult to focus, we develop an objective measure to focus the cameras. We next provide an efficient method for devignetting images, which allows for more seamless panoramic images. We then address the issues of brightness, contrast, and white balance. Our 1394 board cameras support automatic modes for each of these features, but the resulting panoramic images are not very pleasing due to discontinuities across image boundaries (see Figure 2). We develop computer controlled methods that automatically set these parameters for a single camera and propagate the values across all other cameras so that the image boundaries are smooth. This method utilizes the camera hardware, which is not only computationally efficient, but also results in op-
The Computer Vision community had long ago recognized the importance of the geometric calibration in accounting for geometric distortions, but work on multicamara radiometric calibration is limited. Recently, a large amount of work has been done in the direction of geometric registration [4], making panoramas look seamless. In some cases the color uniformity is taken for granted, which is not always true. Others have used techniques to make the change gradual rather than abrupt, techniques known as feathering or blending [4]. Our approach is to use the digital camera controls to get images which are radiometrically similar to start with, and then apply standard blending techniques. Some research has been done to analyze, classify and correct photometric non-uniformities in and across tilted projectors [2]. The work that has been done in the field of radiometric calibration has mainly dealt with noise estimation and modelling [7, 5] or with single camera self calibration [1]. Autofocus has been a topic of interest for camera manufacturers for a long time now. Robust active focus techniques have been developed which work rather well. Passive focus techniques depend mainly on maximizing a measure of focus, and are not as robust as active methods. Some frequency based focusing techniques are discussed in [11]. [9] discusses two new measures which can be used in passive autofocusing. Wavelet transform based autofocus is discussed in [10]. 1
3. Method
the camera an absolutely black scene and then adjusted he brightness parameter to make the mean intensity of the image reach a value which we call “black value” and is criterion of being black. To provide a black scene, rather than making the user to put a lens cap on all the lenses, to simulate that, we make the exposure of the camera zero and then adjust the brightness in order to make the mean of the image reach the “black value”.
3.1. Autofocus We have used a common approach to autofocus, where we define a measure which exhibits a maximum when the image is at best focus. Some examples of focus measures that have been considered in the literature are: 1) Variance of pixel values, 2) Energy of image gradient and 3) Selfentropy of the phase of the Fourier transform. We have chosen contrast as the measure of focus. A focused camera gives sharp images hence the output image has a high contrast. A search is made through the range of possible focal lengths, and when the optimal position is found the focus is set to that. Our measure of contrast of an image is the absolute difference of a pixel with its eight neighbors, summed over all the images of the image. The positive features of this focus measure are that it exhibits a sharp and well defined peak at the position of best focus, it decreases monotonically as the de-focus increases and it is resistant to noise in the image and hence is not very sensitive to small changes in the scene. The fact that the measure decreases monotonically allows us an easy implementation of an intelligent search procedure to find the best focus. The application provides both the contrast value at an instant and also the direction (focusing or de-focusing) in which we are going. Thus we can either look for the point where the contrast is maximum or else the point where the system shifts from the focusing to de-focusing mode, indicating that the system after reaching the best focus is now starting to de-focus. This provides us an objective measure of focus rather than a subjective measure which we get when we try to find the state of the focus using our naked eye. When focusing needs to be done for different cameras, a subjective measure of focus is not sufficient to use the cameras together. To add to its utility, the application allows the user to choose a region of interest which is used for focusing.
3.3. AutoContrast The contrast parameter, c as shown in (1), controls the gain that is provided to the input pixel values. In other words, it is the scale factor applied to the input before it is offset by the brightness. Thus, in effect contrast decides the mean intensity of the image and hence the dynamic range. Our application adjusts the contrast so as to make the mean close to a user specified value “mean brightness”. The default value for the same is 127, which gives the maximum dynamic range to the pixel values. This parameter can be changed by the user to meet the needs of the application.
3.4. Auto White Balance A white piece of paper should appear to be white under sunlight as well as under incandescent light. From a physical point of view, the power spectrum coming from the white piece of paper is same as the illuminants and is different in the two situations. Nevertheless, it will appear white to an observer, because of chromatic adaptation [3]. White balancing is done to keep the color of a white piece of paper the same under different illuminants. Our cameras give us control of red and blue gain to do white balance, which are the gain values that are applied to the red and blue channel. Thus by adjusting these values we try to make a white object appear white in the output of the camera. To be able to do white balance dynamically, without any input from the user, we use the “Grey World” assumption and try to make the amount of green, blue and red in the scene equal, by adjusting the two parameters, red and blue gain. This assumption holds good for most of the real life scenes and gives satisfactory results. Another way we can do white balance once is to allow a user to click an area which is white and then we try to make the red, blue and green content of that patch equal by adjusting the two white balance parameters.
3.2. AutoBrightness The relation between the observed pixel value I (x; y ) at any given point, the brightness b, contrast c and the accumulated charge on the CCD i(x; y ) can be expressed as I (x; y ) = b + ci(x; y ):
(1)
The brightness parameter c is the offset that is provided to the intensity at each pixel. In other words, if the camera looks at a “absolutely black” object, it still won’t have an absolute zero pixel value, but a non-zero value equal to the brightness setting. Thus by adjusting brightness we are actually adjusting what a black object will appear in an image. In the ideal case what we would have done is shown
3.5. Vignetting Vignetting is the radial drop of intensity as we go away from the center of the image. In a typical image formation system the image irradiance E is related to scene radiance L as [1] d E = L ( )2 cos4 ; 4 h
2
(2)
where, h is the focal length of the imaging lens, d is the diameter of its aperture, and is the angle subtended by the principal ray from the optical axis. Because of this, if the object were a plane of uniform brightness exactly perpendicular to the optical axis, the illuminance of its image can be observed to fall off with distance away from the image center. In most compound lenses vignetting also increases with the aperture size [1]. Also, from (2) we can see that lenses with high field of view show stronger vignetting effects. In the RingCam cameras, the effect is not symmetric around any axis and no parametric function fitting the fall off could be found. This is most likely due to the lens mount not being properly centered on these inexpensive board cameras. Hence, we use a generic correction matrix to compensate for vignetting. To determine this matrix, we take several images of a uniformly lit white surface, average them, and find the best fitting plane image P . The correction matrix elements are C i;j = Pcx;xy =Pi;j , where Pcx;cy is the value at the center of P . New images are corrected by simply multiplying elementwise with C . We have written a DirectShow filter which multiplies all incoming image with this correction matrix, which removes the effect of vignetting.
balance parameters have to be set, we calculate the total amount of red, blue and green. As we have no control on the amount of green, we try to propagate the ratio of red w.r.t. green and the ratio of blue w.r.t green. We try to make these two ratios the same for the overlapping strip of the reference camera and the next camera by adjusting the red and blue.
4. Propagation of parameters
In the RingCam’s 1394 cameras, the hardware implementation for the Brightness parameter is incorrect. Rather than just changing the intensity, a change in brightness causes a color shift. Thus, currently we are not able to propagate brightness correctly, and each cameras sets its brightness independently.
4.2 Contrast Propagation The basic idea for contrast propagation is same as that for white balance propagation, i.e., to adjust contrast parameter for the current camera in order to make the contrast in the overlapping strips of the current camera and the reference camera same. For contrast our measure is the average intensity on the overlapping strip. We calculate the mean intensity on the strip on the reference camera and then by adjusting the contrast for the next camera we try to make the average intensity on its corresponding strip equal to that of the reference camera.
4.3 Brightness Propagation
In our application we make a panorama out of images taken from different cameras. Hence the cameras should not only be color calibrated independently, but should also be mutually color calibrated. This is to ensure that they look alike and there is a radiometric uniformity across cameras. We choose any one camera as the reference camera, which can be changed dynamically. This reference camera is color calibrated using the methods mentioned above. Each camera has some overlap with the adjacent cameras. The basic idea is to make the overlapping regions have same white balance, brightness and contrast characteristics. Thus, using the overlapping regions, the camera settings are adjusted for the neighboring cameras and this change is propagated down the chain to the last camera on each side of the reference camera. This processes is continuously repeated, as the reference camera may change its parameters (e.g., due to lighting changes or object motion).
5. Results and Conclusions Figure 2 shows the RingCam’s panorama before color calibration has been done, where each camera is doing white balance, contrast and brightness adjustments independently. Figure 3 shows the RingCam’s panorama after the application of the ColorCalibration including devignetting, which is greatly improved. This paper provides practical, real-time calibration of digital omnidirectional cameras in the areas of focusing, devignetting, brightness, contrast, and white balance control. We have presented novel solutions for the color calibration of an omnidirectional camera. We have also presented an efficient method for devignetting images; we are not aware of any previous work on devignetting multicamera omnidirectional images.
4.1 White Balance Propagation For white balance propagation, we adjust the camera parameters of the current camera so as to make the ratio of red and green and the ratio of blue and green to be the same for the region of overlap between the current camera and the reference camera. In the reference camera, for the strip that overlaps with the current camera, i.e., the camera for which the white
References [1] S.B.Kang, R.Weiss,“Can we calibrate a camera using an image of a flat, textureless Lambertian surface”, ECCV, 2000.
3
Figure 2: RingCam panoramic image before color calibration (1/3 full resolution)
Figure 3: RingCam panoramic image after color calibration (1/3 full resolution)
[5] G.Kemberova, R.Bajcsy, “The Effect of Radiometric Correction on Multicamera Algorithms” Technical Report MS-CIS97-17, 1997. [6] Edited by H.John Durrett, Interactive Systems Laboratory, San Marcos, Texas, Color and Computer, Academic Press, INC, 1986. [7] E.Glenn, Healey, K.Raghava, “Radiometric CCD Camera Calibration and Noise Estimation”, IEEE Trans. PAMI, 16(3):267–276, 1994. [8] M.Charfi, A.Nyeck, A.Tosser, “Focusing Criterion”, Electronic Letters, Vol. 27, No. 14, Page 1233-1235. [9] M.Subbarao, J.Tyan, “Selecting the Optimal Focus Measure for Autofocusing and Depth-from-Focus”, IEEE Transactions on PAMI, Vol. 20, No. 8, AUGUST 1998.
Figure 1: RingCam
[10] J.Widjaja, S.Jutamulia, “Use of wavelet analysis for improving autofocusing capability”, Optics Communications 151, pp. 12-14, 1998.
[2] A.Majumder, Z.He, H.Towles, G.Welch, “Achieving Color Uniformity Across Multi-Projector Displays”, IEEE Visualization 2000.
[11] K.Choi, S.Ko, “New Autofocusing Technique Using the Frequency Selective Weighted Median Filter for Video Cameras”, IEEE Transactions on Consumer Electronics, Vol. 45, No. 3, August 1999.
[3] D.Hasler, S.Susstrunk, “Colour Handling in Panoramic Photography”, Proceedings of SPIE Vol. 4309, 2001. [4] R.Szeliski and H.Shum, “Creating Full View Panoramic Image Mosaics and Environment Maps”, Computer Graphics (SIGGRAPH ’97), pp. 251–258, 1997.
4
