Optimal Illumination Stectrum for Endoscope Moon-Hyun Lee, Dong-Kyun Seo, Byung-Kuk Seo, and Jong-Il Park∗ Department of Electronics and Computer Engineering, Hanyang University, 17 Haengdang-dong, Seongdong-ku, Seoul, Korea, 133-791 ∗ Corresponding author:
[email protected]
Camera Spectral Sensitivity
Illuminant Spectral Power
Abstract—Color differences are determined by illumination, the spectral reflectance of objects, and the spectral sensitivity of the imaging sensor. In this paper, we explore the optimal illumination conditions that best separate one object from another. Given two objects with distinct spectra, we derive the optimal illumination spectrum to maximize their color distance with a plain RGB camera. In practice, it is crucial to compose the most appropriate illuminations using available lighting sources since creating arbitrary illumination spectrum is unrealistic. Therefore, we derive the optimal linear combination of the provided illumination sources. Finally, we verify the effectiveness of the methods through experiments.
l(λ)
c(λ)
Wavelength
Wavelength
1 Any
color model can be used in principle.
Surface s(λ)
Image (RGB)
B
G
R
Wavelength
Spectral Power
Mechanism of color image formation
Spectral Reflectance
Fig. 1.
Cancer
Synthetic Illumination LA
700
680
660
640
620
600
580
560
540
520
500
480
460
440
420
400
680
660
640
620
600
580
560
540
520
500
480
460
440
420
700
RGB Distance: 115.86
700
680
660
640
620
600
Camera Red Channel
580
560
540
500
Camera Green Channel
480
460
440
420
400
Camera Blue Channel
520
Spectral Sensitivity
400
Normal
Spectral Power
RGB Distance: 92.85
700
680
660
640
620
600
580
560
540
520
500
480
460
440
Xenon Lamp
420
700
680
660
640
620
600
580
560
540
520
500
480
460
440
420
Halogen Lamp
400
Spectral Power
Wavelength(nm)
400
Object distinction from a scene is an important task in many computer vision and recognition systems. In general, an object in a scene is distinguished from its background through its color differences. Such color differences are directly determined by the illumination, each individual object’s spectral reflectance, and the spectral response of camera. As shown in Fig. 1, light emitted from an illumination source is reflected from a scene element and then enters an imaging device. Thus, the color of a scene element is obtained by integrating the spectral power distribution of the illumination source, the spectral reflectance of the object, and the spectral sensitivity of the digital imaging device. Illumination plays a key role in determining the appearance of an object. Different objects which look the same under a specific illumination can be revealed different under another illumination. Figure 2 shows how a scene is changed by different illuminations. A scene consisting of two materials with cancer and normal tissue spectra (character region and background region) is captured by an RGB camera with the presented spectral sensitivity [9]. Under illumination with halogen lamp, the two regions distance in the RGB color space is 92.85. When the scene is illuminated by a xenon lamp, the distance is 98.12. Suppose we attempt to discriminate an object from others. The best illumination spectrum would be the one that maximize the color separation in the RGB space1 . The optimal illumination using our method in this paper results in the distance of 135.2, better than other illuminations. We can more clearly discriminate the two materials with ease in the image. In this paper, we focus on illumination spectrum that distinguishes an object from a scene and derive the optimal illumination spectrum to maximize the color distance. Moreover,
Spectral Reflection
I. I NTRODUCTION
RGB Distance: 98.12
Fig. 2. The effect of illumination in object distinction. (from top-left counter-clockwise) Object spectra and camera spectral sensitivity curves, power spectrum of a halogen lamp and result image, power spectrum of a xenon lamp and result image, and power spectrum of the optimal illumination and result image.
we look for the optimal linear combination of the available light sources and verify by applying to endoscopic imaging which is a practical application. 1) Related Works: Color is the most important factor for low-level image processing such as classification or segmentation in computer vision. Three channel color models are widely used in most computer vision systems but they have inherent limitations because they are crude approximation of continuous spectrum. Thus, many researchers have been interested in spectrum-based imaging as a good solution for overcoming the limitations. There have been many researches associated with multispectral or hyperspectral imaging that computationally estimates the spectral reflectance of an object and synthesizes images of the object under a variety of illumination conditions. Moreover, the characteristic of illumination that significantly affects the color of an object has popularly been used for object recognition based on spectrum and several approaches have proposed for color distinction of objects by varying illumination conditions [2], [4], [7], [8], [10]. In particular, the spectrum-based imaging has steadily developed for diagnosis of disease in the field of medical imaging. For instance, [3] used an illumination with infrared band for distinguishing irregular tissues with normal tissues and [5] proposed a method that enhances the sensitivity of the spectral reflectance of the scattering feature in the superficial tissue layers. There have been several previous researches using differences of spectral reflectance for object distinction [6], [9], but they mainly focused on estimation and analysis of differences of the spectral reflectance and did not paid attention to illumination. 2) Contribution: The key contributions of our paper can be summarized as follows. •
•
•
Optimal illumination spectrum: Given two objects with distinct spectra, we derive a solution for optimal illumination spectrum that maximizes the color distance between objects. Maximum color separation using limited sources: Since creating arbitrary illumination spectrum is unrealistic, it is important to compose the most appropriate illuminations using given lighting sources. We present a solution to find the most appropriate illumination using linear combinations of the provided illumination sources. Illumination controllable endoscope system: We present a practical endoscope system with a set of controllable LED light sources.
3) Organization of the paper: In Section 2, we formulate a problem of optimal spectral illumination to maximize color distance and derive solutions for unrestricted illumination and restricted illumination, respectively. In Section 3, we present some experimental results using real endoscopic imaging system and discuss our findings in investigation of optimal spectral illumination. Then, we conclude our work in Section 4.
II. I LLUMINATION S PECTRUM FOR M AXIMUM C OLOR S EPARATION A. Optimal Illumination Denote ck (λ), s(λ), and l(λ) represent the spectral response of a camera in channel k, the spectral reflectance of a scene point, and the spectral power distribution of an illumination, respectively. Then the value Ik measured at a pixel is given by ∫ Ik = ck (λ)s(λ)l(λ)dλ. (1)
The equation can be rewritten in a vector-matrix form as follows: Ik = cTk Sl.
(2)
where S is a diagonal matrix consisting of spectral reflectance s. Let two objects with distinct spectra Sa and Sb be given, the color distance d can be obtained by d2 =
∑[ ]2 cTk (Sa − Sb )l .
(3)
k
To derive the optimal illumination spectrum that maximize the distance in the color space, we can formulate the following maximization problem: lopt = arg max l
∑[ ]2 cTk (Sa − Sb )l , ∥l∥ = 1.
(4)
k
where the constraint ∥l∥ = 1 is enforced for fair comparison. Denoting qTk = cTk (Sa − Sb ), we can rewrite the Eq. 4 as [ lopt = arg max l l
T
∑
] qk qTk
l.
(5)
k
Because ∑ the Eq. 5 is a quadratic form, the largest eigenvalue of k qk qTk is the maximum d2 and its corresponding eigenvector is lopt , according to the Constrained Extremum Theorem [1]. Since any spectral illumination function must be positive, the condition l ≥ 0 must be enforced. Therefore, we can obtain the solution that satisfies both conditions ∥l∥ = 1 and l ≥ 0 by constraining negative elements of eigenvector to zero and renormalize the vector: ˜1 . lopt = e
(6)
˜1 is a nonnegative renormalized eigenvector of where e ∑ T q q k k k corresponding to the largest eigenvalue.
B. Maximum Color Separation Using Limited Light Sources Since creating arbitrary illumination spectrum is unrealistic, it is crucial to compose the most appropriate illuminations using available lighting sources. Therefore, it is useful to find the most appropriate illumination using linear combinations of the illumination sources at hand. If controllable light sources whose spectra are l1 , l2 , ..., ln , are given, we can formulate a problem of finding the best linear combination of illumination sources as follows: ∑[ ]2 xopt = arg max cTk (Sa − Sb )Lx . (7) x
Endoscope
Camera Optical Fiber to Endoscope
k
where L is a matrix consisting of source vector l1 , l2 , ..., ln and x represents, a weight vector for light sources. Here, the conditions, ∥x∥ = 1 and xi ≥ 0 for all i, must be satisfied because the illumination power is limited and nonnegative. Then we have [ ] ∑ T T xopt = arg max x (8) wk wk x, x
LEDs
Lens Array
k
where, wkT = cTk (Sa −Sb )L. The optimal weight vector x can be derived by the same process as the previous section. Finally, we obtain the optimal linear combination of light sources: lcomposed = Lxopt .
LED Controller
(9)
III. E XPERIMENTAL R ESULTS The proposed method is suitable for endoscopic imaging because it aims at diagnosis of disease by distinguishing abnormal tissues from normal tissues. Moreover, illumination of endoscope is easily controllable inherently since it is completely isolated from exterior lights. In our experiments, we designed a new endoscope system to verify the effectiveness of our method. We used LEDs as illumination sources because they were cheap, small, and controllable. The LEDs consist of power LEDs (Z-Power LED P4T M ) of five different colors (”white”, ”red”, ”amber”, ”green”, and ”blue”) and their spectra were measured by a spectroradiometer (Luchem SPR4001T M ). The light of LEDs was concentrated on an optical fiber through lens array and the optical fiber delivered the light to the tip of endoscope. An RGB camera (PointGrey Dragonfly ExpressT M ) was used for imaging and an LED controller was used for regulating the power ratio of LEDs. B. Experiments for Optimal Illumination If the spectral reflectance of target objects is known, it is easy to compute an optimal illumination. Otherwise, the spectral reflectance should be measured or estimated first using spectral reflectance measuring devices such as spectrophotometer. However, it is difficult to apply those devices to endoscopic imaging where work space (=measuring environment) is isolated like internal body. Thus, we circumvented
Fig. 3. Our endoscope system using controllable illumination. It consists of an endoscope, an RGB camera, lens array, LEDs(5 types), and an LED controller.
White Red Amber Green Blue Camera Blue Camera Green Camera Red
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
A. Endoscope System Using Controlled LEDs
Fig. 4. The spectra of the 5 types of LEDs (solid lines) and the spectral responses of the three color channels of the PointGrey Dragonfly Express camera (dashed lines) used in our system.
Halogen Lamp
White LED
400 425 450 475 500 525 550 575 600 625 650 675 700 400 425 450 475 500 525 550 575 600 625 650 675 700
Xenon Lamp
400 425 450 475 500 525 550 575 600 625 650 675 700 400 425 450 475 500 525 550 575 600 625 650 675 700
Derived Optimal Illumination
Skin
80
Blue
Blue
100
120
120
100
100
60
60
40
40 Blood Vessel
200 150 100 Green
200
RGB Distance 113.64
80
Skin
100
80 60
Blood Vessel
RGB Distance 100.25
200
200
200 100
Red
Green
40
Blood Vessel 150
100
80 60
40 200
100 Red
120
Skin
150
100
RGB Distance 126.27 Blue
RGB Distance 132.33
Blue
Skin
120
Green
Blood Vessel 200
200 150
100
100 Red
Green
100 Red
Fig. 6. Comparison of relighting results using optimized illumination and conventional illuminations. Top row: The illumination spectra for relighting. These spectrums are normalized for fair comparison. Middle row: Relighting results using the derived optimal illumination, xenon lamp, halogen lamp, and white LED(left to right). Bottom row: RGB distributions of corresponding images.
Blood Vessel
400 435 470 505 540 575 610 645 680
400 435 470 505 540 575 610 645 680
Skin
Fig. 5. Multispectral image using an endoscope system shown in Fig. 3. The scene is taken of human throat. Spectral reflectance of skin is shown on left and blood vessel reflectance spectrum is shown on right.
this problem by using one of multispectral imaging methods for estimating spectral reflectance, proposed by Park et al [8]. Figure 5 is a multispectral image of a part of throat surface captured by the system shown in Fig. 3. The image consist of two regions, internal skin and blood vessels, and their spectral reflectance was respectively estimated as shown Fig. 5. With the estimated spectral reflectance and the camera spectral sensitivity of Fig. 4, we obtained the optimal illumination spectrum and showed on the top-left in Fig. 6. When we relit the multispectral image using the computed optimal illumination spectrum, the blood vessels are more cleanly distinguished compared with using a xenon lamp, a halogen lamp, and white LEDs which are commonly used
for endoscope(Fig. 6-(middle row)). When we calculated the color distances between skin and vessel regions, the optimal illumination resulted in greater distances consistently. For example, the distances between the O-marked pixel(blood vessel) and the X-marked pixel(skin) are 132.33, 113.64, 126.27, and 100.27 for the optimal illumination, a xenon lamp, a halogen lamp, and white LEDs, respectively. These results demonstrate the validity of the proposed method. C. Experiments for Optimized LED Illumination LEDs are widely used as efficient light source. We used a variety of LEDs to show that the proposed optimization with a given set of illumination sources is efficient and practical. To distinguish blood vessels from internal skin, we computed the optimal linear combination of the LEDs using the Eq. 9. The coefficients for the LED colors are shown in Table I and the optimal LED spectrum is shown in Fig. 7. Figure 8 shows the multispectral relit images using the optimal illumination spectrum and the optimized LED illumination spectrum. We see that the skin and blood vessel is well distinguished in both cases. Careful examination reveals that is expected, the optimal illumination shows slightly better results than the optimized LEDs. Because the optimized LED illumination is constrained in the spectrum shape. Figure 9 is real captured images for oral skin using the
0.1
0.1 0.05
Spectral Power 400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700 400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700 400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700 Spectral Power 400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
0.4
Pink Flower
0.3 0.2 0.1 0
0.2
Spectral Power
Buttercup Flower
0.1 0 -0.1
Spectral Power
0.3
0.5
Spectral Difference
0.6
Spectral Power
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700 400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
Spectral Power
0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
Almond Walnut
Spectral Difference
0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
0
Spectral Power
0.1 0.05
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
0.15
0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 -0.01
Spectral Difference
Red Bush Leaf Red Maple Leaf
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
0
0.2
-0.2
Difference of Object Spectra
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
Object Spectra
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
-0.3 400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
Spectral Reflectance
Spectral Power
0.2
0.15
Spectral Power
0.3
0
Spectral Reflectance
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
0.2 Cabbage Lettuce
0.4
Spectral Difference
Spectral Reflectance
0.5
Spectral Reflectance
Spectral Power
Spectral Difference
Banana Lemon
0.25 0.2 0.15 0.1 0.05 0 -0.05 -0.1 -0.15 -0.2
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
Spectral Reflectance
0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
Optimal Illumination
Optimized LED
Fig. 10. Examples of optimal illumination spectra for some materials. (from left) Material spectra, difference of spectrum, optimal illumination, and optimized LED illumination. These are derived for the camera and LEDs shown in Fig. 4.
LED Color White Red Amber Green Blue
Optimal Weight 0.6721 0.0981 0.1988 0.6515 0.2732
White Red Amber Green
TABLE I O PTIMAL WEIGHTS FOR OUR LED S TO MAXIMIZE RGB SKIN TO BLOOD VESSEL .
DISTANCE FROM
Blue
400 420 440 460 480 500 520 540 560 580 600 620 640 660 680 700
Optimized LED Illumination
optimized LED illumination. We see the blood vessel is distinguished from background clearly.
Fig. 7. Optimized LED spectra. Dashed line is the weighted sum of LED spectra.
D. Discussion We have done a number of simulations for a variety of cases. Some of the results are shown in Fig. 10. Findings from extensive simulation are as follows. The shape of optimal illumination spectrum is roughly
similar to the difference of two spectra of interest but not necessarily because of the dependency on camera spectral sensitivity. In our experiments, most of optimal illumination
Optimal Illumination
Optimized LED
distinct materials are considered for optimization. It would be interesting to extend and generalize the proposed method to multiple objects of interest. One possible solution would be to exploit multiplexed illumination. Currently, we are trying to resolve all these problems by using multispectral imaging and flexible virtual re-illumination. ACKNOWLEDGMENT
Fig. 8. Comparison of the multispectral relighting result using the optimal illumination and optimized LED spectrum.
This work was supported by the IT R&D program of MKE/MCST/IITA. [2008-F-031-01, Development of Computational Photography Technologies for Image and Video Contents] R EFERENCES
Fig. 9. Real captured images for oral skin using the optimized LED illumination shown at Fig. 7.
tends to have peak power in the green channel (around 535 nm) of camera because the quantum efficiency of the camera being used is highest in the channel. When we change the white balance, we obtained different results. The spectrum of optimized LED illumination showed similar overall shape to that of optimal illumination in most cases. IV. C ONCLUSION AND F UTURE W ORK We proposed spectrum-based optimal illumination to efficiently discriminate objects with distinct spectra. In our approach, we derived the optimal illumination spectrum to maximize the color distance when captured with a plain RGB camera. We also derived the optimal linear combination of practically available illumination sources. To verify the effectiveness of our approach, we designed an endoscope system capable of controlling illumination and applied the proposed method to endoscopic imaging. Experimental results showed that the color of a target object was clearly discriminated from others. One big advantage of the proposed approach is its flexibility. We can apply target-specific illumination easily with the proposed method. If we combine the proposed technique with multispectral imaging, more flexible detecting methodology would be possible, which we are currently working on. The limitation of the method in this paper is that only two
[1] H. Anton and R. C. Busby. Contemporary Linear Algebra. Jhon Wiley & Sons, Inc., 2003. [2] C. Chi, H. Yoo, and M. Ben-Ezra. Multi-spectral imaging by optimized wide band illumination. International Journal of Computer Vision, 2008. [3] L. Chiriboga, P. Xie, H. Yee, V. Vigorita, D. Zarou, D. Zakim, and M. Diem. Infrared spectroscopy of human tissue. i. differentiation and maturation of epithelial cells in the human cervix. Biospectroscopy, 4:47–53, 1998. [4] D. W. Fletcher-Holmes and A. R. Harbvey. Real-time imaging with a hyperspectral fovea. Journal of Optics: Pure application Optics, 7:S298– S302, 2005. [5] K. Gono, M. Igarashi, T. Obi, M. Yamaguchi, and N. Ohyama. Multiplediscriminant analysis for light-scattering spectroscopy and imaging of two-layered tissue phantoms. OPTICS LETTERS, 29:971–973, 2004. [6] R. Levenson and J.R.Mansfield. Multispectral imaging in biology and medicine: slices of life. Cytometry Part A, 69:748–758, 2006. [7] A. Mohan, R. Raskar, and J. Tumblin. Agile spectral imaging; programmable wavelength modulation for cameras and projectors. In Proc. Eurographics, 2008. [8] J. Park, M. Lee, M. D. Grossberg, and S. K. Nayar. Multispectral imaging using multiplexed illumination. In Proc. IEEE ICCV, 2007. [9] M. Sambongi, M. Igarashi, T. Obi, M. Yamaguchi, N. Ohyama, M. Kobayashi, Y. Sano, S. Yoshida, and K. Gono. Analysis of spectral reflectance using normalization method from endoscopic spectroscopy system. Optical Review, 9:238–243, 2002. [10] M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama. High-fidelity video and stillimage communication based on spectral information: Natural vision system and its applications. In Proc. SPIE-IS&T Electronic Imaging, 2006.
