Portable Real-Time Color Night Vision Alexander Toet*, Maarten A. Hogervorst TNO Human Factors, P.O. Box 23, 3769 ZG Soesterberg, the Netherlands ABSTRACT We developed a simple and fast lookup-table based method to derive and apply natural daylight colors to multi-band night-time images. The method deploys an optimal color transformation derived from a set of samples taken from a daytime color reference image. The colors in the resulting colorized multiband night-time images closely resemble the colors in the daytime color reference image. Also, object colors remain invariant under panning operations and are independent of the scene content. Here we describe the implementation of this method in two prototype portable dual band realtime night vision systems. One system provides co-aligned visual and near-infrared bands of two image intensifiers, the other provides co-aligned images from a digital image intensifier and an uncooled longwave infrared microbolometer. The co-aligned images from both systems are further processed by a notebook computer. The color mapping is implemented as a realtime lookup table transform. The resulting colorised video streams can be displayed in realtime on head mounted displays and stored on the hard disk of the notebook computer. Preliminary field trials demonstrate the potential of these systems for applications like surveillance, navigation and target detection. Keywords: image fusion, false color, natural color, real-time fusion, lookup tables
1. INTRODUCTION Night vision cameras are a vital source of information for a wide-range of critical military and law enforcement applications related to surveillance, reconnaissance, intelligence gathering, and security. The two most common nighttime imaging systems cameras are low-light-level (e.g., image-intensified) cameras, which amplify the reflected visible to near infrared (VNIR) light, and thermal infrared (IR) cameras, which convert invisible thermal energy from the midwave (3 to 5 microns) or the long wave (8 to 12 microns) part of the spectrum into a visible image. Until recently a gray- or greenscale representation of nightvision imagery has been the standard. However, the increasing availability of fused and multi-band infrared and visual nightvision systems has led to a growing interest in the color display of night vision imagery 5,9,10,10,14,20,20. In principle, color imagery has several benefits over monochrome imagery for surveillance, reconnaissance, and security applications. For instance, color may improve feature contrast, which allows for better scene recognition and object detection. When sensors operate outside the visible waveband, artificial color mappings generally produce false color images whose chromatic characteristics do not correspond in any intuitive or obvious way to those of a scene viewed under natural photopic illumination. This type of false color imagery may disrupt the recognition process, resulting in an observer performance that is even worse compared to that obtained with singleband imagery alone11. Several different techniques have been proposed to display night-time imagery in natural daylight colors12-15,18,20, some of which have been implemented in realtime nightvision systems1,4,16,17,19. Most of these techniques are computationally expensive and/or do not achieve color constancy. In a companion paper we introduce a new color mapping that was developed to display night-time imagery in natural daytime colors2. This technique is simple and fast, and can easily be deployed in realtime. Moreover, it provides stable colorization under variations in scene content2,3. Here we describe the implementation of this color mapping in two prototype portable dual band realtime color night vision systems.
*
[email protected]; phone +31-346-356237; fax +31-346-353977; http://lextoet.googlepages.com/
Multisensor, Multisource Information Fusion: Architectures, Algorithms, and Applications 2008, edited by Belur V. Dasarathy, Proc. of SPIE Vol. 6974, 697402, (2008) · 0277-786X/08/$18 · doi: 10.1117/12.775405 Proc. of SPIE Vol. 6974 697402-1 2008 SPIE Digital Library -- Subscriber Archive Copy
2. COLOR MAPPING The principle of the new lookup-table based color mapping color mapping technique is explained in detail in a companion paper2. For the sake of completeness we will now briefly describe this procedure. First, a false color image is constructed by mapping the different bands of a multisensor nightvision system to respectively the R, G, and B channels of an RGB image (set channel B to zero when only 2 bands are available, and use only the first three principal components when the system provides more than 3 bands). Second, transform the false color image thus obtained into an indexed image using a color lookup table containing a set RGB triples (this is a 3D lookup table, which reduces to 2D when only 2 bands are available). Finally, replace the false color lookup table of the input multiband nightvision image with a new color lookup table that maps the false colors onto natural colors. The new color lookup-table can be obtained either by applying a statistical transform to the entries of the original lookup-table, or by a procedure that replaces entries of the original lookup-table by their corresponding natural color values. The statistical transform method transfers the first order statistics (mean and standard deviation) of the color distribution of a representative natural color daytime reference image to the false color multiband nighttime image2,13. This mapping is usually performed in a perceptually decorrelated color space (e.g. l 8).The sample-based method deploys a set of corresponding samples from the combination of a multi-band sensor image of a given scene and a registered naturally colored (RGB) daytime reference image of the same scene to derive a color lookup table transform pair that transfers the color characteristics of the natural color reference image to the false color nighttime image2,3. For an 8-bit multi-band system providing 3 or more bands the 3D color lookup table contains 256x256x256 entries (for a 2 band system the 2D table contains 256x256 entries). When the color lookup table contains fewer entries, the color mapping is achieved by determining the closest match of the table entries to the observed multi-band sensor values. Once the color transformation has been derived and the pair of color lookup tables that defines the mapping has been created, they can be used in a real-time application. The lookup table transform requires minimal computing power. An additional advantage of the color lookup transform method is that object colors only depend on the multi-band sensor values and are independent of the image content. As a result, objects keep the same color over time when registered with a moving camera. In the next sections we first describe two prototype portable dual band real-time hardware implementations of nighttime image acquisition systems that deploy the lookup-table color transform method. The first system creates a color nightvision image by fusing the visual and near-infrared bands of two identical image intensifiers. The second system presents a color fused image of the signals of an image intensifier and a longwave infrared thermal camera. Then we will present the results of some preliminary field trials.
3. THE GECKO SYSTEM: COMBINATION OF VISUAL AND NEAR-INFRARED The Gecko sensor module provides co-axially registered visual and NIR images. This system is named after nocturnal geckos that still have color vision at very dim light levels7. The Gecko system includes 2 image intensifiers (night vision goggles, Fig.1a), 2 compact EO cameras (Fig.1b), a heat reflecting (hot) mirror, and a near-infrared reflecting mirror (Fig. 2). The image intensifiers are two GEN III type Mini N/SEAS monocular night vision goggles (NVGs) from International Technologies Lasers Ltd (ITL). They provide a 1x magnification, and have a circular field-of-view (FOV) with a diameter of 40 deg, corresponding to about 2000 pixels. They are sensitive in the visual and near infrared part of the spectrum. Both image intensifiers are place side by side. A distinctive characteristic of the construction of our acquisition unit is the hardware registration of both NVG images. A co-aligned view is achieved through the use of a hot mirror in combination with a NIR reflecting mirror. The hot mirror is an Edmund Optics (www.edmundoptics.com) NT43-958 3.3 mm thick mirror, intended for an angle of incidence of 45°, with a multi-layer dielectric coating that reflects infrared radiation (heat), while allowing visible light to pass through (Fig. 3a). The NIR radiation is reflected by a Melles Griot 01 MFG 011 mirror (www.mellesgriot.com) that is covered with a protected aluminum coating, which has an average reflectance greater than 87% over the spectral range from 400 to 800 nm (fig. 3b). As shown in Fig. 2a, the hot mirror acts as a dichroic beam splitter, transmitting the visual part of the incoming radiation to the upper NVG, while reflecting the NIR part via the NIR reflecting mirror into the lower NVG. The image from each NVG is registered by a PixeLINK PL-A741 MV FireWire 1.3 megapixel monochrome camera with a 2/3" CMOS detector with a resolution of 1280 x 1024 pixels (www.pixelink.com). Both cameras are equipped with a 16 mm Pentax C1614A C-mount lens, yielding a horizontal FOV of 30º72'T. They operate either at 33 fps at an image size of 1k x 1k, or 105 fps at 640 x 480 pixels.
Proc. of SPIE Vol. 6974 697402-2
The firewire signals of both PixeLINK cameras, representing respectively a visual and near-infrared representation of the viewed scene, are combined into an RGB image stream for further processing in a Dell Inspiron 9300 notebook computer (the B channel is set to zero). As a result of the co-axial image registration parallax problems are eliminated and only minimal spatial alignment is needed before image/video exploitation. The previously described color mapping is implemented as a color lookup table transform.
1 :/
(b) (a) Fig. 1. Sensors comprising the Gecko dual band system. (a) Mini N/SEAS monocular night vision goggle (ITL Ltd). (b) PixeLINK PL-A741 MV camera.
HOT
mirror
/2
}
NIR
______________
mirror
Visible radiation NIR radiation
(b)
(a)
4
(c)
(d)
Fig. 2. The dual band Gecko system. (a) Schematic representation. (b) 3D view of the system design. (c) Front view of the system and its casing. (d) Side view.
Proc. of SPIE Vol. 6974 697402-3
g5 4
100
80
Transmission
Reflection
F-
90 85
Reflection 400
Transmission
500 600 700 800 900 1000
Wavelength (nm)
80
400
incidence
s plane p plane
450 500
550
600
650
700 750
WAVELENGTH IN NANOMETERS
(b)
(a)
Fig. 3. Reflectance and transmission characteristics of (a) the Edmund Optics NT43-958 hot mirror (source www.edmundoptics.com) and (b) the Melles Griot 01 MFG 011 NIR reflecting mirror (source www.mellesgriot.com).
4. THE VIPER SYSTEM: COMBINATION OF VISUAL AND LONGWAVE INFRARED The Viper sensor module provides co-axially registered visual and longwave infrared (LWIR) or thermal images. This system is named after a species of snake that fuses in its optic tectum the visual images from its eyes with thermal images from infrared sensitive organs that function like a pinhole cameras6. The Viper sensor module includes a compact infrared microbolometer (Fig.4a), a digital image intensifier (Fig. 4b), and 2 hot mirrors (Fig. 5). The FLIR Systems ThermoVision A10 infrared microbolometer has a 160 x 128 pixel focal plane array, and a spectral sensitivity range of 7.5 – 13µm, which is the range of most interest for outdoor applications. It is equipped with an 11mm (f/1.6) lens providing a 40° x 30° wide angle view. The ThermoVision A10 delivers wide dynamic range (14-bit) analog video output at 30 Hz (for RS-170) or at 25 Hz (for CCIR). It has an NETD of
