Developing a multi-spectral HDR imaging module for

Developing a multi-spectral HDR imaging module for a BRDF measurement system Duck Bong Kim, Myoung Kook Seo, Kang Yeon Kim and Kwan H. Lee∗ Intelligent Design and Graphics Laboratory Department of Mechatronics, Gwangju Institute of Science and Technology (GIST) 1 Oryong-dong, Puk-gu, Gwangju, 500-712, South Korea ABSTRACT Most recent bidirectional reflectance distribution function (BRDF) measurement systems are the image-based that consist of a light source, a detector, and curved samples. They are useful for measuring the reflectance properties of a material but they have two major drawbacks. They suffer from high cost of BRDF acquisition and also give inaccurate results due to the limited use of spectral bands. In this paper, we propose a novel multispectral HDR imaging system and its efficient characterization method. It combines two promising technologies: high dynamic range (HDR) imaging and multispectral imaging to measure BRDF. We perform a full spectral recovery using camera response curves for each wavelength band and its analysis. For this, we use an HDR camera to capture HDR images and a liquid crystal tunable filter (LCTF) to generate multi-spectral images. Our method can provide an accurate color reproduction of metameric objects as well as a saturated image. Our multi-spectral HDR imaging system provides a very fast data acquisition time and also gives a low system setup cost compared to previous multi-spectral imaging systems and point-based commercial spectroradiometers. We verify the color accuracy of our multi-spectral HDR imaging system in terms of human vision and metamerism using colorimetric and spectral metric. Keywords: Multi-spectral HDR imaging, spectral characterization, metamerism, liquid crystal tunable filter

1.

INTRODUCTION

An accurate description of color reproduction1 is a key step in computer vision and computer graphics. Ideally, a good color reproduction must be physically accurate, numerically advantageous, and flexible to generate real objects in a synthesized image. But, many conventional color imaging systems are based on RGB triplets, and they are difficult to reproduce the original color of an object. Even though the color management technology is greatly progressed, still there remain limitations in color reproducibility using a conventional color imaging system. The advancements in high dynamic range image (HDRI) techniques2,3 allow a greater dynamic range of luminances between light and dark areas of a scene using off-the-shelf digital cameras. The aim of HDRI is to accurately represent a wide range of intensity levels in a real scene that ranges from sunlight to shadows. Debevec et al.2 proposed a technique that uses multiple images of different exposures to produce a single HDR image. These techniques are widely used in image-based BRDF measurement methods4,5 to estimate the radiance of the HDR images. However, many previous BRDF systems based on multiple images with different exposures suffer from a long acquisition time, a huge size of raw data, and inaccuracy due to the limited number of spectral bands. Still the RGB triplets remain the most common method for conventional imaging and rendering systems. For most cases, describing a color in the tristimulus space is sufficient to communicate color information to a human viewer. But, if the whole computation of light reflection and transfer is performed in the tristimulus space, a significant color distortion can be introduced due to the device dependent property. Moreover, the use of *

To whom correspondence should be made: [email protected] Reflection, Scattering, and Diffraction from Surfaces II, edited by Zuhan Gu, Leonard M. Hanssen, Proc. of SPIE Vol. 7792, 77920M · © 2010 SPIE CCC code: 0277-786X/10/$18 · doi: 10.1117/12.860071 Proc. of SPIE Vol. 7792 77920M-1 Downloaded from SPIE Digital Library on 07 Sep 2010 to 203.237.44.136. Terms of Use: http://spiedl.org/terms

tristimulus representations lead to metameric colors6, i.e. colors with the same tristimulus values but with different spectral power distributions. Thus, the RGB triplets prove to be a poor choice when accurate synthesizing of color images is needed. To address these limitations, multispectral imaging techniques have been widely used in many areas such as image processing, computer graphics, and color reproduction. In this paper, we propose a multi-spectral HDR imaging system and its efficient characterization method that combines the use of HDRI and multispectral imaging. We use a HDRC HDR camera (IMS-Chips system) to capture HDR images and a liquid crystal tunable filter (LCTF, CRI, Inc.; Cambridge, Ma) to generate multispectral images. Our method can provide an accurate color reproduction of metameric objects as well as saturated images. Our multi-spectral HDR imaging system provides a very fast data acquisition time and also gives a low system setup cost compared to previous multi-spectral imaging systems and point-based commercial spectro-radiometers. We verify the color accuracy of our multi-spectral HDR imaging system in terms of human vision and metamerism using colorimetric and spectral metric.

2.

RELATED WORK

Many researchers have dealt with high dynamic range imaging2,3,7,8,9 and multi-spectral imaging10-13. But few research work focused on developing a multi-spectral HDR imaging system14 and its characterization. 2.1 Characterization of a HDR imaging system Absolute luminance levels can be acquired by re-scaling the luminance value according to a luminance meter 7,8. Inanici et al.7 proposed a HDRI technique which is a luminance mapping tool that uses multiple exposure images to capture a wide luminance variation within a same scene. In contrast, Krawczyk8 proposed a characterization method which maps the pixel values of a HDR image onto the luminance without multiple exposure images. They performed an absolute calibration of a HDR camera system to allow the recovery of real-world luminance values by comparing the luminance values from the HDR camera to the measurements performed by a luminance meter for a grayscale patch. Kautz et al.9 proposed an efficient characterization method for HDR images which can measure a physically accurate radiance of a real object by using a back-lit transparent color target and a spectroradiometer. Recently Kim et al.16 performed an efficient characterization method for a HDR imaging system to estimate the reflectance of a given target. They can estimate the real physical reflectance of a given object using an HDR image which is generated from only one image, in contrast to the Kautz et al’s method that requires multiple exposure images. Their method16 has strength in terms of calibrating luminance and color at the same time, compared to the previous methods by Krawczyk8 and Inanici7 that calibrate the luminance only. However, it is still limited in dealing with metameric objects, since they use RGB triplets instead of using radiance in each wavelength. 2.2 Characterization of a multi-spectral imaging system Many researchers10-13 have worked on a multi-spectral imaging system since it can provide an accurate color reproduction as well as generation of a device-independent profile such as CIE XYZ values. Many multispectral imaging systems10-13 have been developed using monochromatic imaging sensors such as charged coupled devices (CCDs) and liquid crystal tunable filter (LCTF) to generate multi-spectral images. Hardeberg et al.10-11 proposed an efficient characterization method of a spectral imaging system, composed of a LCTF and a monochromatic CCD camera. They converted the camera outputs to a device independent color space such as CIEXYZ or CIELAB by inverting the model using a principal eigenvector approach. But, the common CCDs offer a single shot having a dynamic range of 50 dB to 80 dB depending on applications. A single image can be easily saturated or image details can be lost when acquiring images in a complex lighting situation. Therefore, it requires multiple images with different exposures to avoid unwanted images, which leads to a lot of time and cost. 2.3 Characterization of a multi-spectral HDR imaging system Johannes et al.14 proposed an multi-spectral HDR imaging system which is composed of seven band-pass filters

Proc. of SPIE Vol. 7792 77920M-2 Downloaded from SPIE Digital Library on 07 Sep 2010 to 203.237.44.136. Terms of Use: http://spiedl.org/terms

on a computer controlled filter wheel. Recently, Matthias et al.17 developed a bi-spectral BRRDF (bidirectional reflectance and re-radiation distribution function) measurement system. Martin et al.18 developed a BTF (bidirectional texture function) measurement system. The detector module of the systems is a multispectral HDR imaging system which consists of a LCTF and a monochromatic CCD camera. However, the computation time and storage cost of their system are quite high since it needs multiple images with different exposures as well as multiple different wavelength images. Kim et al.19 developed a multispectral HDR imaging system and performed a spectral characterization by fitting the pixel values of the multispectral HDR imaging system to the measured reflectance from a spectroradiometer. However, it does not perform the spectral recovery, and only provides colorimetric information in their work.

3.

BACKGROUND

3.1 Metamerism In colorimetry, metamerism occurs when two object have the same tristimulus value while they have different spectral power distributions. For example, the different objects show the same color under a certain illumination while they show different colors under a different illumination. There are four different types of metamerism: illuminant, observer, field size and geometric metamerism which is caused by different illuminants, persons, distances and viewing angles, respectively. In this paper, we focus on illuminant metamerism as shown in Fig. 1. We test the Gretag Macbeth metamerism kit #3 under a color viewing booth (The Judge II, GretagMacbethTM). Fig. 1 (left) shows the color appearance under day light while Fig. 1 (right) shows the color appearance under ultra violet light. We can observe the difference of colors between four patches as shown in Fig 1. (right).

Fig. 1. Photographs of Gretag Macbeth Daylighting Metamerism Test Kit #3 under different illuminants: under day light (left), under U.V. (right).

3.2 Metric for error evaluation There are two kinds of error evaluation metrics: colorimetric and spectral. Colorimetric metrics such as CIELUV, CIELAB, CIE94, and CIE2000 approximate color differences observed by the human eye. To calculate the metric, we need to know the tristimulus values of a given color signal, but we cannot distinguish the spectral difference between two objects. The colorimetric CIE2000 ΔE*00 will be used to evaluate the colorimetric between two spectra. Spectral metrics measure the distance between two spectra. The root mean square error (RMSE) is a simple metric that is commonly used to evaluate spectral error. Another spectral metric is the goodness-fit-coefficient (GFC), which is a widely used index of similarity between two spectra6. It is based on the inequality of Schwartz. E(λ) represents the original spectrum while ER(λ) indicates the recovered spectrum. The GFC is described by Eq. 1 as follows:

Proc. of SPIE Vol. 7792 77920M-3 Downloaded from SPIE Digital Library on 07 Sep 2010 to 203.237.44.136. Terms of Use: http://spiedl.org/terms

GFC =

∑ E (λ ) E (λ ) j

j

∑ [E (λ )]

2

j

j

1/ 2

R

j

∑ [E (λ )]

1/ 2

(1)

2

j

R

j

In this paper, the CIE2000 is used as a colorimetric cost function and the RMSE and GFC are used as a spectral evaluation metric.

4.

CHARACTERIZATION OF A MULTI-SPECTRAL HDR IMAGING SYSTEM

4.1 The setup of a multi-spectral HDR imaging system The multispectral HDR imaging system19 is designed to acquire the spectral and radiometric information over a wavelength of 400 to 720 nm. It consists of a monochromatic HDR camera (by IMS-Chips system), a VariSpec VIS liquid crystal tunable filter (by CRI, Inc.; Cambridge, Ma), and a manual zoom video lens (by Edmund optics), as shown in Fig. 2. Due to the technological advances in CMOS sensors, an HDR image sensor with an array of 640×480 has been developed, which acquires HDR images with a dynamic range of up to 8 orders of magnitude at video frame rates. We can acquire seventeen multi-wavelength bands from 400 nm to 720 nm at 20nm intervals.

Fig. 2. Multispectral HDR imaging system: LCTF mounted in front of an HDR camera with a long back working distance optics.

We design the light source to give highly bright and uniform emission across the wavelength from 380 to 780 nm and be collimated, unpolarized, and spectrally and radiometrically stable over time without flickering. It must be compact and generate a small amount of heat for safety. It must also have a good color rendering index (CRI). Fig. 3 shows the photograph (left) and a schematic view (right) of the specially designed light source. The Xenon arc lamp (OSRAM XBO 300W) with a reflector provides a continuous spectrum in the desired range while minimizing infrared emission and reducing heat and focusing the beam. To reduce the excessive heat and unwanted light, we use an aperture in front of a lamp. To eliminate residual polarization from the light source, we use an UV holographic diffuser. The light is gathered by two aspheric condenser lenses and passed through a small aperture to approximate a point light source, and then collimated by a camera lens (Nikon, f =85 mm 1.4).

Fig. 3. The developed light source: photograph (left) and a schematic view of its optical components (right).

Proc. of SPIE Vol. 7792 77920M-4 Downloaded from SPIE Digital Library on 07 Sep 2010 to 203.237.44.136. Terms of Use: http://spiedl.org/terms

4.2 Experimental setup for characterization To perform a spectral recovery from the seventeen multispectral HDR images, we should estimate the camera responses of the multispectral HDR imaging system. To calculate the camera response, we prepare an experimental setup that includes a reflective color checker (ColorChecker® Color Rendition chart by Xrite), a transparent color checker (ESSER TE226 by Imaging Engineering), a spectroradiometer (CS-1000 by Konica Minolta), the light source, and the multi-spectral HDR imaging system, as shown in Fig. 4. In this experiment, we acquire 69 samples for the training and test targets (64 samples for the training, 5 samples for the test).

Fig. 4. Experimental setup for characterization of the multi-spectral HDR imaging system: a setup to measure the spectral reflectance of the reflective color chart using a multispectral HDR imaging system (left) and a setup for a transparent color chart using a spectroradiometer (right).

To characterize the multispectral HDR imaging system, we use a reflective color checker and a transparent color checker. They are supposed to give a good representation of natural reflectance spectra. The reflective chart is shown in Fig. 5 (left). We can acquire a number of samples with a wide dynamic range using the two color charts. The spectral reflectance of each color in the reflective color chart measured by a spectroradiometer (CS1000 by Konica Minolta) is shown in Fig. 5 (right), respectively.

Fig. 5. The Xrite ColorChecker® Color Rendition chart (left) and its measured spectral reflectance (right).

To conduct a spectral recovery from the multispectral HDR images, we acquire seventeen multispectral images of the two color rendition charts while varying the peak wavelength of the LCTF from 400 nm to 720 nm with a 20-nm interval. Seventeen channels of these multispectral images of the reflective color chart are shown in Fig. 6. We obtain a good spectral sensitivity for most wavelength bands except near 400 nm and 420 nm although the exposure time of the HDR camera is fixed.

Proc. of SPIE Vol. 7792 77920M-5 Downloaded from SPIE Digital Library on 07 Sep 2010 to 203.237.44.136. Terms of Use: http://spiedl.org/terms

Fig. 6. The seventeen multispectral HDR images of the color rendition chart.

5.

SPECTRAL RECOVERY OF MULTISPECTRAL HDR IMAGES

5.1 Overall procedure of the experiments Fig. 7 shows a schematic diagram to generate the transformation matrix from digital signals to reflectance using the training targets and the spectral estimation and accuracy evaluation for verification. The acquired multispectral HDR images are masked to extract the coordinates and digital signals of areas corresponding to the uniform patches. It will result in k band images (k=17 for our multispectral HDR imaging system) with r patches (r=64 for our training targets including the two color charts) giving s pixels per patch. The digital signals of pixels are averaged for each patch resulting in k bands with r digital signals. Most previous spectral recovery methods used the camera digital signals directly to estimate the transform matrix. A direct transform from the camera digital signal to the measured radiance is not appropriate for the HDRC CMOS image sensor used here since it has an algorithmic response. Therefore, we use estimated radiance values from the digital signals using the logarithmic camera response curves for each spectral band which can be estimated in the previous work19. Several spectral reconstruction techniques such as Imai-Berns method20, Weiner estimation21, and the pseudoinverse transformation matrix22 have been proposed in multispectral imaging. Among these techniques, the pseudo-inverse transformation method is widely used due to its mathematical simplicity. The method establishes a direct relationship between the camera response and the spectral reflectance of a target by using a direct 17 by 321 transformation from 17 estimated radiances to 321 dimensional reflectance spectra. After obtaining a transformation matrix, the five test targets are estimated and compared with the measured reflectances by a spectroradiometer (CS-1000, Konica-Minolta). The results of the estimated reflectances are shown and compared in terms of colorimetric and spectral quality metrics in Section 6.

Proc. of SPIE Vol. 7792 77920M-6 Downloaded from SPIE Digital Library on 07 Sep 2010 to 203.237.44.136. Terms of Use: http://spiedl.org/terms

Fig. 7. Schematic diagram of characterization based on the pseudo-inverse transformation method from digital signals to reflectance using the training targets and its validation test for 5 patches.

5.2 Pseudo-inverse transformation method We denote the spectral power distribution of an illuminant by l(λ), the spectral reflectance of a target by r(λ), the spectral transmittance of an optical system o(λ), the spectral transmittance of the kth channel by Φ(λ), and the spectral sensitivity of a camera a(λ). The relationship between the responses (ck) of the multispectral HDR imaging system for each channel and spectral reflectance of the target can be represented as

c = ∫ l (λ )r (λ )o(λ )φ (λ )a(λ )dλ.

(2)

Here l(λ), o(λ), Φ(λ), and a(λ) are the system unknowns. When they are merged into a single term w(λ) as spectral responsivity, then Eq. (2) can be written as

c = ∫ r (λ ) w(λ )dλ.

(3)

In practical computation, Eq. (3) can be rewritten as a scalar product in matrix notation by sampling the visible spectrum uniformly at k wavelength intervals:

c = rtw ,

(4)

where r= [rλ1 rλ2 rλ3 · · · rλk]t and w=[wλ1 wλ2 wλ3 · · · wλk]t. Let ck be a sensor response vector that is obtained from the ith learning sample in the learning chart with a known spectral reflectance ri. Let C be an M×k matrix that contains the sensor responses [c1 c2 c3 · · · ck] and let R be an N×k matrix that contains the corresponding spectral reflectances [r1 r2 r3 · · · rk], the pseudo-inverse method is to find a matrix W(N×M) that minimizes ||RWC||. The matrix W is then given by W = RC+ = RCt(CCt)-

(5)

+

where C represents the Moore-Penrose pseudo-inverse matrix of C. By applying a matrix W to a sensor response vector, the spectral reflectance of the target is estimated as follows:

rˆ = Wc .

(6)

Thus, this method does not need the spectral sensitivity of the image sensor, the LCTF, the optical system and the spectral power distribution of an illuminant, but only uses the spectral reflectances of learning samples.

Proc. of SPIE Vol. 7792 77920M-7 Downloaded from SPIE Digital Library on 07 Sep 2010 to 203.237.44.136. Terms of Use: http://spiedl.org/terms

6. RESULTS AND DISCUSSION Fig. 8 shows the comparison between the measured spectra and the estimated spectral reflectances from the measured camera responses of five test targets. Table 1 shows the results of colorimetric error evaluation (CIEDE2000 ΔE*00) and spectral error evaluation (RMSE, GFC) between the measured spectra and the estimated spectral reflectances using the measured camera responses for each test target. The color differences (

ΔE00) vary from 1.82 to 5.70, while the RMSE and GFC vary from 0.0718 to 0.3191 and from 0.9789 to 0.9983, respectively. This demonstrates that the proposed system gives accurate results in terms of colorimetric and spectral errors. Our system also gives better efficiency than the spectroradiometer, since it can measure the radiance of a given material by the number of image pixels, while a spectroradiometer can measure it by a point. As mentioned in Hardeberg et al.11, the nominal full width at half maximum (FWHM) of the LCTF is not constant. For example, it varies from 15 to 80 nm. Moreover, there are unwanted secondary peaks at long wavelengths for peak wavelengths