Bio-Inspired Adaptive Hyperspectral Imaging for Real ... - IEEE Xplore

IEEE SENSORS JOURNAL, VOL. 10, NO. 3, MARCH 2010

647

Bio-Inspired Adaptive Hyperspectral Imaging for Real-Time Target Tracking Tao Wang, Student Member, IEEE, Zhigang Zhu, Senior Member, IEEE, and Erik Blasch, Senior Member, IEEE

Abstract—In this paper, we present an efficient and novel approach to embed hyperspectral imaging (HSI) capability in an intelligent panoramic scanning system for real-time target tracking and signature acquisition. The sensor platform we propose consists of a dual-panoramic peripheral vision component and a narrow field-of-view (FOV) HSI component. The panoramic HSI design optimizes the tradeoff of a wide FOV, a high-spatial/spectral resolution in real-time imaging, and bandwidth limitations. The dual-panoramic scanners with a hyperspectral fovea sensor platform improves some existing designs in literature in three aspects: 1) a panoramic view is provided instead of just a normal wide-angle view; 2) a dual scanning system is designed to obtain moving targets in a very effective and efficient manner; and 3) active control of the hyperspectral sensor is added to facilitate signature acquisition of targets of various locations that is required in real time for target tracking. The data collection of the sensor platform is closely coupled with real-time algorithms of target detection, tracking, and identification. The intelligent hyperspectral data acquisition is performed in real time so that tracking of one or more targets and switching between multiple candidate target types is possible. Important issues such as single-view-point constraint of the multimodal imaging components, target detection and tracking, real-time HSI, and target signature detection and target identification are discussed. Index Terms—Active sensing, field of view (FOV), hyperspectral imaging (HSI), multimodal, target tracking.

I. INTRODUCTION

H

YPERSPECTRAL IMAGING (HSI) involves of collecting data simultaneously in dozens or hundreds of narrow, adjacent spectral bands. With extended information of the recorded data image classification can be further improved. Manuscript received August 31, 2008; revised June 09, 2009; accepted June 19, 2009. Current version published February 24, 2010. This work was supported in part by the Air Force Office of Scientific Research under the Discovery Challenge Trusts Program, Award FA9550-08-1-0199, in part by the Air Force Research Laboratory Sensor Directorate under Award FA8650-05-1-1853, and in part by the National Science Foundation under Grant CNS-0551598. The associate editor coordinating the review of this paper and approving it for publication was Prof. Yangiiu Li. T. Wang and Z. Zhu are with the City College Visual Computing Laboratory, Department of Computer Science, City College of New York, New York, NY 10031 USA, and also with the Department of Computer Science, Graduate Center at City University of New York, New York, NY 10016 USA (e-mail: [email protected]; [email protected]). E. Blasch is with the Air Force Research Laboratory/RYAA Evaluation Branch, Wright-Patterson Air Force Base, Dayton, OH 45433 USA, and also with the Department of Electrical Engineering, Wright State University, Dayton, OH 45435 USA (e-mail: [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/JSEN.2009.2038657

However, recording hyperspectral data is very time consuming. Also, for those applications that are time varying and transient and when there is a relative motion between objects and surroundings, the use of time-sequential HSI introduces the possibility of spatial misregistration of images and a consequent distortion of the recorded spectra [1]. Thus, tracking moving objects with HSI in real time using conventional HSI techniques are problematic. To ease the spectral distortion, all voxels of the spectral data need to be recorded simultaneously. The simultaneous recording can be achieved using HSI snapshot techniques [1]. But the spectral resolution is limited to few tens of bands for most techniques except integral field spectroscopy [2], which has multiple high-cost detector arrays. The increased number of bands is directly related to the information bottleneck constituted by the space-bandwidth product of the detector arrays. To demonstrate 2-D, video-rate, HSI (i.e., hundreds of spectral bands) with megapixel spatial resolution requires images to be recorded at the order of 10 G [3]. It is not practical to process such high-rate data pixels in real time. Recently, a great deal of effort has been put into an adaptive and tunable multispectral or hyperspectral sensor design with goals to address the challenging problems of detecting, tracking, and identifying (ID) targets in dynamic scenes. Representative programs include: the Defense Advanced Research Projects Agency’s Adaptive Focal Plane Array Program [4], Army Research Laboratory’s Advanced Sensor Collaborative Technology Alliance [5], National Science Foundation’s Center for Mid-Infrared Technologies for Health and Environment [6], and Air Force Office of Scientific Research’s Adaptive Multimode Sensing and Ultra-High Speed Electronics [7]. In the design of a useful sensor, field of view (FOV) is another important factor. Real-time omnidirectional vision has been implemented by catadioptric designs using hyperbolic, parabolic, or ellipsoidal mirror designs, as in [8] and [9]. Real-time panchromatic peripheral view and hyperspectral fovea have been designed by [3] to balance the problems of FOV and real-time HSI. All these designs represent the state-of-the-art of adaptive sensor designs that have advanced the conventional sensor concepts. However, most of the sensor designs only have very limited data exploitation capabilities for optimizing hyperspectral data acquisition. In this paper, we present a novel single sensor design that has both omnidirectional view and hyperspectral fovea, and that integrates sensing and processing for real-time target detection and signature acquisition. We propose a novel and low-cost adaptive sensor design concept: HSI data are only captured in the regions of interest (ROIs) that are provided by a peripheral vision component in real time.

1530-437X/$26.00 © 2010 IEEE

648


Here, we will mainly focus on tracking and ID moving targets in real time with such an HSI system. We will show the design of a particular sensor platform with multimodalities that both covers wide FOV and has real-time HSI capability. This design is one example of bio-inspired sensor designs that are useful for a wide range of applications in real-time target recognition [10] and surveillance [3]. The panoramic HSI sensor platform improves or differs from previous designs [1], [4], [5] in literature in three aspects: 1) a panoramic view is provided instead of a normal wide-angle view; 2) a dual scanning system is designed to obtain moving targets in a very effective and efficient manner; 3) active control of the hyperspectral sensor is added to facilitate signature acquisition of targets of various locations that can only be determined in real time. In different environments, different sensors or combinations of various types of sensors need to be designed to achieve the tracking and ID tasks. It is impractical to fit all application scenarios with one kind of sensor design. Therefore, as an ultimate goal, our team has proposed a system approach to adaptive multimodal sensor designs [11] in order to reduce development time and system cost while achieving optimal results through an iterative process that incorporates simulation, evaluation, and refinement of critical elements. Therefore, in addition to presenting a novel sensor design, this paper also provides discussions of the design example that follow the philosophy by using scene and sensor simulation and integrating data exploitation into data acquisition. The paper is organized as the following. Section II shows the design of the bio-inspired adaptive multimodal sensor platform—the dual-panoramic scanners with a hyperspectral fovea (DPSHF) for the task of tracking and ID moving targets in real time. Section III describes the simulation environment and the parameter configuration of the sensor platform. Section IV presents the image exploitation algorithm for detecting and tracking moving targets. Section V discusses a spectral classification method in recognizing a simple moving object. Conclusions and discussions will be provided in Section VI. II. SENSOR DESIGN Our sensor platform design is inspired by the biological concept of the human vision system. The structure of a human eye is to focus light onto the retina. The retina contains two forms of photosensitive cells important to vision—rods and cones [12]. Rods cells are highly sensitive to light, allowing them to respond in dim light and dark condition, e.g., in detecting moving targets. However, they cannot sense color differences. Rods are found primarily in the periphery of the retina. The wide FOV peripheral vision is good at detecting motion. Cone cells are found primarily in the center (or fovea) of the retina that has a high visual acuity in a very narrow FOV. They are used primarily to distinguish colors. Like the eye, we desire a system that tracks objects in the periphery and identifies them in the fovea. We propose a sensor platform—DPSHF (see Fig. 1), which consists of a dual-panoramic (or omnidirectional) peripheral vision and a narrow FOV hyperspectral fovea. The DPSHF sensor design is an extension of the human vision system in that the peripheral vision is not just wide FOV but omnidirectional, and

the foveate vision is not just high-resolution color but hyperspectral. The intelligent sensor works as follows. At the first step, two panchromatic images with 360 FOV are generated by rotating two line scanners around a common rotating axis, pointing apart to two slightly different directions. The angular distance between two scanners is determined during the sensing process and should be sufficient for detecting and tracking both slow and fast moving objects in real time. One concern is the geometric distortion of a line scanner caused by sensor platform motion and scan motor speed that is not matched to the rotation speed. The DPSHF is not subject to distortion since the platform is motor controlled/stabilized, and the vertical scan speed is faster than the platform horizontal rotation (panning) speed. There are two advantages of using line scanners that will be further amplified. First, line scanner can have large FOV. Second, resulted images are inherently registered. Moving targets can then be easily and quickly determined by the differences of the two panoramic images in low resolution generated from two scanners, respectively. The next position and the time of the moving target can be predicted from the difference of previous two ROIs. The detailed processing algorithm will be discussed in Section IV. Then, we turn the hyperspectral imager with a specific focal length calculated based on the size of the object to the predicted region that includes the moving targets. Thus, HSI data are recorded in an efficient way for only those ROIs. The two line scanners and the hyperspectral imager are aligned so that they all share a single viewpoint. The spectral data can be efficiently recorded with a foveal hyperspectral imager (FHI) [3]. The FHI uses a pick-off mirror that intercepts the light onto a fiber optical reformatter (FOR) [13], which turns a 2-D spatial array of light into a 1-D array and projects it onto a dispersive hyperspectral imager (DHI) [14]. The DHI then produces a 2-D hyperspectral data array, one spatial dimension, and one spectral dimension. Then the DHI result is combined with the panchromatic images generated by the dual-panoramic scanners, to produce a coregistered spatial-spectral image. The FOR is constructed from a coherent array of ribbon array of fibers into a fibers. The reformatter maps an 1-D 1 array. The fibers diameter determines the maximum spatial resolution of FHI. With 50 m in diameter and arranged on a 56- m pitch, the spatial resolution can be 14 14 pixels. With reduced diameter of 10 m, the spatial resolution can be significantly improved to 70 70 pixels. The blurring effect from cross-coupling of optical fibers was not a significant magnitude and proved from experience by Harvey et al. [1]. The advantage of using FHI is that it contains a high spectral resolution up to 1 nm. However, the spatial resolution of FHI is limited by the FOR to 14 14 pixels (using currently available fibers) or improved to 70 70 pixels (using the next generation fibers). It is sufficient for tracking targets far away from the sensor, but if the target is close the small FOV might not be enough. One solution is to use image replication imaging spectrometer (IRIS) [15], which gives modest FOV with tradeoff of a small number of spectral bands. In using IRIS, one approach is to select narrow spectral band of spectrometer to be sampled. Therefore, we can only retain spectral bands of interest and analyzing the object spectral profile and then determining

WANG et al.: BIO-INSPIRED ADAPTIVE HYPERSPECTRAL IMAGING FOR REAL-TIME TARGET TRACKING

649

Fig. 1. Design concept of DPSHF. The dash lines indicate the single viewpoint of both the FHI and the two line scanners. Note that the actually configuration is not shown in diagram. It can be adjusted during simulation and will be demonstrated in Section III.

the significant bandwidth to be selected for the spectral filter. An acoustooptic tunable filters (AOTF) [16], [17] can be one of the solutions that can be rapidly tuned to pass various wavelengths of light by varying the frequency of an acoustic wave propagating through an anisotropic crystal medium [18]. AOTF operates very fast, usually in microseconds, so that it can be used in real-time processing. Indeed, we only need to tune the filter once when the tracking target is decided. In order to track objects, the next section will detail the scene analysis and sensor modeling from which HSI target exploitation is accomplished via time-space-frequency segmentation. III. SCENE SIMULATION AND SENSOR MODELING In order to test the sensor design concept and to integrate sensing and processing before a real sensor is fabricated, we first simulate a realistic scene and model our sensor platform in digital imaging and remote sensing image generation (DIRSIG) tool [19], developed at the Rochester Institute of Technology. Utilizing scene simulation and sensor modeling, we reduce the cost and development time for new sensor designs with complementary exploitation algorithms. The DIRSIG is a complex synthetic image generation application, which produces simulated imagery in the visible through thermal IR regions [20]. DIRSIG is designed to produce broadband, multispectral, and hyperspectral imagery through the integration of a suite of first principles-based radiation propagation submodels. One important submodel DIRSIG use is MODTRAN [21], which is a radiation propagation model to generate a complex set of lookup tables to characterize the exoatmospheric irradiance, emitted, and scattered radiances and path transmission. Before scene simulation and sensor modeling, we need to set up different scenarios and configure the sensor parameters. One simple scenario we used involves a moving Humvee in an urban scene (see Fig. 2). The average speed of the Humvee is about

33.6 mi/h moving from east to west, passing through the main building in the scene. Our sensor platform is placed in front of this building. The scan speed of each line scanner can be set from 60 to 100 Hz selectable, thus one entire 360 scan take from 6.0 down to 3.6 s. This time constraint is not a problem for real-time target detection since detection and scanning are continuous and simultaneous. The number of pixels per line in the vertical direction is set to 512 to match the horizontal scanning resolution. Only one spectral channel is needed for dual line scanning. The focal length is fixed at 35 mm for both line scanners, and the angle between the pointing directions of the two scanners is 10 so that the time the second scan reaches the position of the first scan is only about 0.1 s. In theory, the time difference between two scans should be much less than moving object speed to avoid uncertainty (i.e., Nyquist sampling). Two scanners are used so that 1) the more accurate direction and focal length of the hyperspectral fovea can be estimated and 2) moving target detection can still be performed when background subtraction using a single scanner fails due to cluttered background, multiple moving targets, and the sensor platform ego-motion. The hyperspectral imager focal length is automatically adjusted according to the target detection results generated from two line scanners. The focal length adjustment is accomplished our image processing algorithm described in the next section. To simulate the hyperspectral imager, we use a frame array sensor with small spatial resolution at 70 70 pixels. The spectral resolution is 0.01 m ranged from 0.4 to 1.0 m. Different portion of bandwidth can be selected and determined by analyzing model spectral profile. The results are analyzed using ENVI and MATLAB. The processing part is programmed in C++. IV. DETECTION AND TRACKING The data exploitation bio-inspired adaptive HSI tracking (BAHT) algorithm we developed is to find ROIs to control the

650


= 43 0

= 77 0

Fig. 2. Simulated urban scene image at latitude : and longitude : , captured at 1000 m above the scene. A Humvee is placed in the center road moving from right to left. The bigger red circle shows the starting position, and the red lines show the moving trajectory. The sensor platform showing in blue square is placed in front of the main large building and about 5-m high from ground. The sun position at 14:00 is 21.8 from the z axis and 299.4 clockwise from the y axis, where the y axis points from left to right and the z axis points out of the paper in the aforementioned figure.

pointing directions of hyperspectral imager. BAHT mainly consists of two components: background subtraction for moving target detection and ROIs determination for target localization. A. Background Subtraction The first step is background acquisition. The average of all images captured by the first scanner of the scene without any moving targets is considered as the starting background [see Fig. 3(a)]. Each of two line scanners will generate a sequence of 1-D image lines at each rotation angle. The panorama is constructed by combining all image lines. The advantage of using line scanner here is to avoid registration problem. Fig. 3(b) and (c) shows the panoramas generated from the first and the second scanners with a moving target in view, respectively. The moving target can be easily extracted from subtracting each panorama from the background whenever the following relation holds for a pixel: (1) where is a “predefined” threshold, which should be relatively small so that any small region including the moving target will be detected. However, the background subtraction result (the difference image) has a lot of noise due to the changes of time and the reflections of the light over time. In Fig. 3(d) and (e), the pixel intensities of the difference images are amplified so that the shapes of both the target and the noise can be viewed clearly. Most significant noisy regions are resulted from the reflections

of tree leaves and shadow boundaries. We apply a morphological noise removal technique [22] that consists of the opening (i.e., dilation) and the closing (i.e., erosion) operations to remove spare points and irregular shapes. Therefore, small sparse noises can be removed using the opening operation, while small holes can be filled using the closing operation. In order to have better moving object extraction over time, we keep updating the background scene model after each 360 rotation with (2) where is kept small to prevent artificial “tails” forming behind moving objects [23]. A more efficient method is to only update those pixels belonging to the background using (2). The background subtraction approach using two scans at different times determine the speed of a moving target easily, which will be described in the next section. Note that real-time target detection is achieved since the scanning and detection are performed simultaneously and continuously. Furthermore, with the dual-panoramic scanners, even if background subtraction fails, targets can still be detected from the difference of two scans scanning the scene at slight different times. B. ROI Determination After background subtraction together with noise filtering, we find two ROIs of the same target viewed at different times resulting from the two different scans. The differences of the two


651

2

Fig. 3. All 360 panoramic images (512 3600) shown here are integrated from line scans captured by the dual-panoramic scanners. (a) Background scene simulated in color. (b) Panoramic image from the first scanner, with the moving target indicated inside a red circle. (c) Panoramic image from the second scanner, again the same moving target indicated inside a red circle. (d) Background subtraction result for the image captured by the first scanner. (e) Background subtraction result for the image captured by the second scanner. [Note: in order to view the shapes of the targets clearly and make the background noise more visible an offset of 50 was added to the intensities of all pixels in (d) and (e) except the pure black (i.e., zero intensity)].

Fig. 4. ROIs extracted from two scans. The same small potions of the two previous panoramas [see Fig. 3(d) and (e)] are cropped and zoomed to bring out a closed view of those ROIs where the targets are detected. Red boxes show the ROIs from the first scan. Blue boxes show the ROIs from the second scan.

regions can determine the relative bearing angle of the hyperspectral fovea imager to zoom in on the moving target. For example, in Fig. 4, the red box shows the ROI having the target at the first scan, and the blue box shows the ROI having the target at the second scan. From ROI exploitation, we estimate whether the object is moving left to right and moving closer to our sensor platform. Therefore, we predict the next position of the target

will be from the motion estimate. Also, the ratio of the previous two regions indicates whether we should increase or decrease the focal length of the hyperspectral imager. The angle difference of two scans for two ROIs at different can be used to predict the position of the next times and , when the hyperROI having the moving target at the time , spectral imager can be in place. Therefore, given the time

652


Fig. 5. Angle estimation of a moving target based on different times. The platform is rotating clockwise where the target is moving in the opposite direction. The formula is also applied for a target moving in other directions.

we estimate the panning and tile angles of the hyperspectral imager. Note that only the angles relative to the center of regions are needed. The turning angles (i.e., panning and titling) of the hyperspectral imager should be

2

(3) correspond to panning (in the -direcwhere the superscript tion) and the tilting (in the -direction), respectively. The angle is corresponding to a ROI at a time , as shown in Fig. 5. The focal length of the hyperspectral fovea is inversely proportional to the desired FOV of the hyperspectral imager in order to have the target in the full view of the FOV. The FOV angle can be estimated as

Fig. 6. (a) Foveal shot (70x 70) that has moving Humvee inside. The image is presented at red = 680 nm, green = 540 nm, blue = 460 nm. The panning angle, the tilt angle, and the focal length of hyperspectral imager are calculated as 155.7 , 2.1 , and 115.0 mm, respectively. (b) Spectral profile of the foveal image. Seven significant shapes of spectral curve are demonstrated here. Spectral curves of all other pixels were averaged to those seven curves.

(4) is the predicted size of the target region at , where is the number of line scan per angle defined in sensor and and the previous configuration. The relationship between two regions of the same target at different times can be expressed as (5) Fig. 6(a) shows an example of hyperspectral foveal shot of a ROI from the calculation. Thus, hyperspectral data are recorded only for ROI [see Fig. 6(b)]. It is possible for some regions that do not have true moving targets, e.g., the left two boxes shown in Fig. 4, but have two large shadows. Shadow verification is determined by simply subtracting ROIs of the two scans from the matching regions in the background. If one has no motion change, we confirm that there is no moving object. Note: the additional advantage of having two panoramic scanners is demonstrated through shadow verification V. SPECTRAL CLASSIFICATION Recognizing moving targets compares a target spectrum associated with each pixel to a set of training spectrums. The spectral

Fig. 7. (a) Humvee model in DIRSIG database. (b) Spectral profile of Humvee model shows two major spectral curves of two significant parts.

library was prebuilt with existing models (see Fig. 7). An absorption-feature-based method, spectral feature fitting, is used for matching a target spectrum to the reference endmembers that are selected from spectral library. To reduce the effects of variation in illumination intensities and directions, the magnitudes of all targets and training spectra were normalized using the continuum removal technique [24]. Continuum is a mathematical function used to isolate a particular absorption feature for analysis. High points of a spectral curve are defined as local maxima. The high points are connected by straight lines, which generate a


653

Fig. 8. Spectral matching result of target pixels in Fig. 6 and model spectral in Fig. 7. (a) Matching result of the first curve in model spectral library. The brighter a pixel is, the less the difference is. (b) Matching result of the second curve in model spectral library. The darker a pixel is, the larger the difference is. (c) Combination of all matching result, binary decision is made at the threshold 0.6.

continuum. The final spectrum is the result of dividing the original spectrum from its continuum. A least square difference is calculated between each target spectrum and the reference endmembers. The total RMS error is used to form an RMS image for each endmember (see Fig. 8). One general RMS that uses the arithmetic mean is the minimum Euclidean distance (MED) [25]. (6) where is an unclassified pixel, is the mean of the class , and is the number of spectral dimensions. The MED represents the Euclidean distance between the current sample and the and the sample is assigned mean for each class to the class with the minimal MED. The problem with the MED is that it does not account for the data variance. More advanced methods [26] may be applied for certain cases, but will not be discussed here. As a result, each pixel is classified either to a known object if the target spectrum was matched with the library spectrum of that object, or to unknown object, e.g., the background. Finally, the false-color image was generated for each pixel that belongs to the known object in the database. The resulting falsecolor image was embedded into the panchromatic peripheral image, which was captured with HSI simultaneously to produce a co-registered composite image that indicated the next position of the moving target. Fig. 8 demonstrated a simple spectral matching result of spectral curve of target pixels and sample pixels. In this case, only two spectral curves are needed to recognize the target. The results of matching differences of the two curves in spectral library were shown in Fig. 8(a) and (b). To recognize the whole object, all matching results were combined to a single image profile, and binary decision was made using a threshold found by experience [see Fig. 8(c)]. It shows that even though we had large area of shadow on the top of the Humvee from trees, we were still able to detect large part of the Humvee body. Therefore, we can embed the matched result into the peripheral image and produce a snapshot of the moving target. Sequences of peripheral image snapshots with embedded false-color foveal images were taken as the next moving position of the target. Thus, tracking and ID moving objects can be done in real time. In Fig. 9, we show a single image snapshot captured at the same time the HSI was recorded, and the result was integrated to produce a coregistered composite image.

Fig. 9. Spectral classified image was embedded into a panchromatic peripheral image snapshot (512 512), showing the next movement of the target Humvee. Pixels inside the fovea are assigned false-color red and black in watershed to indicate similarity to sample spectra of the Humvee model and nonrecognized object.

2

VI. CONCLUSION We described our BAHT algorithm, using the multimodal sensors design that enables hyperspectral data to be recorded efficiently for tracking moving targets in real time. We also showed simulated and tested examples of the DPSHF sensor platform design with the DIRSIG simulation tool that enables both hyperspectral scene generation and image data exploitation for performance-driven sensing. The BAHT image processing algorithm demonstrates the basic idea of how to effectively capture hyperspectal data in the ROI. Needless to say, more sophisticated algorithms can be developed for more challenging tracking tasks over complex scenarios. In the future, we will design and evaluate similar platform that can be mounted on a mobile vehicle (ground or airborne) and develop algorithms that work with imaging data collected from a moving platform. Additional constraints will be addressed such as the geometric distortion of line scanner, complex geometry calculations, and occluded objects. We want to note that the advantage of using simulation allows us to reconfigure the sensor platform parameters reducing the developing cost and time before fabrication. The system-sensor design approach leverages expertise from users, sensor engineers, and exploitation scientists for a more efficient and effective design of adaptive multimodal sensors. ACKNOWLEDGMENT The author would like to thank Dr. H. Rhody at Rochester Institute of Technology for the advice on using the digital imaging and remote sensing image generation tool and his collaborations in the system approach for multimodal sensor designs. REFERENCES [1] A. R. Harvey, J. Beale, A. H. Greenaway, T. J. Hanlon, and J. Williams, “Technology options for imaging spectrometry,” in Imaging Spectrometry VI Proc. SPIE, M. Desour and S. Shen, Eds. Bellingham, WA: SPIE, 2000, vol. 4132. [2] D. Q. Ren and J. Allington-Smith, “On the application of integral field unit design theory for imaging spectroscopy,” Publ. Astron. Soc. Pac., vol. 114, pp. 866–78, 2002.

654


[3] D. W. Fletcher-Holmes and A. R. Harvey, “Real-time imaging with a hyperspectral fovea,” J. Opt. A: Pure Appl. Opt., vol. 7, pp. S298–S302, 2005. [4] R. Balcerak, DARPA’s Adaptive Focal Plane Array (AFPA) Program 2007. [Online]. Available: http://www.arpa.mil/mto/programs/afpa/index.html [5] A. C. Goldberg, B. Stann, and N. Gupta, “Multispectral, hyperspectral, and three-dimensional imaging research at the U.S. army research laboratory,” in Proc. 6th Int. Conf. Inf. Fusion, 2003, vol. 1, pp. 499–506. [6] MIRTHE, Center for Mid-Infrared Technologies for Health and the Environment, Last Visited: 2008. [Online]. Available: http://www.mirthecenter.org/ [7] K. Reinhardt, Adaptive Multi-Mode Sensing and Ultra-High Speed Electronics, Last Visited: 2009. [Online]. Available: http://www.wpafb.af.mil/library/factsheets/factsheet.asp?id=12580 [8] Z. Zhu, E. M. Riseman, and A. R. Hanson, 1999 Geometrical modeling and real-time vision applications of panoramic annular lens (PAL) camera, Comput. Sci. Dept. ,Univ. Massachusetts Amherst, Amherst, MA, Tech. Rep. TR99-11, Feb. 1999. [Online]. Available: http://www-cs.engr.ccny.cuny.edu/~zhu/UMASSTR99-11.ps.gz [9] S. K. Nayar and S. Baker, “Catadioptric image formation,” in Prof. DARPA Image Understanding Workshop, May 1997, pp. 1431–1437. [10] H. Ren and C.-I. Chang, “Automatic spectral target recognition in hyperpectral imagery,” IEEE Trans. Aerosp. Electron. Syst., vol. 39, no. 4, pp. 1232–1249, Oct. 2003. [11] Z. Zhu, A System Approach to Adaptive Multi-Modal Sensor Designs, 2008. [Online]. Available: http://www-cs.engr.ccny.cuny.edu/~zhu/ Research/MultimodalSensors.html [12] V. Bruch, P. R. Green, and M. A. Georgeson, Visual Perception: Physiology, Psychology and Ecology. Florence, KY: Psychology Press, 1996. [13] Fibreoptic Systems Inc. 60 Moreland Rd, Unit A. Simi Valley, CA 93065, Last Visited: 2008. [Online]. Available: http://www.fibopsys.com [14] Headwall Photonics, Inc. Fitchburg, MA 01420, Last Visited: 2008. [Online]. Available: http://www.headwallphotonics.com/ [15] A. R. Harvey, D. W. Fletcher-Holmes, A. Gorman, K. Altenbach, J. Arlt, and N. D. Read, “Spectral imaging in a snapshot,” Spectral Imag.: Instrum., Appl., Anal. III Proc. SPIE, vol. 5694, pp. 110–119, 2005. [16] N. Gupta, “Acousto-optic tunable filters,” Opt. Photon. News, vol. 8, pp. 23–27, 1997. [17] M. S. Gottlieb, “Acousto-optic tunable filter,” in Designs and Fabrication of Acousto-Optic Devices, A. P. Goutzoulis and D. R. Pape, Eds. New York: Marcel Dekker, 1994, pp. 197–283. [18] H. R. Morris, C. C. Hoyt, and P. J. Treado, “Imaging spectrometers for fluorescence and raman microscopy: Acousto-optic and liquid crystal tunable filters,” Appl. Spectrosc, vol. 48, no. 7, pp. 857–866, 1994. [19] DIRSIG. [Online]. Available: http://dirsig.cis.rit.edu/ Aug. 2008 [20] J. R. Schott, S. D. Brown, R. V. Raqueño, H. N. Gross, and G. Robinson, “An advanced synthetic image generation model and its application to multi/hyperspectral algorithm development,” Can. J. Remote Sens., vol. 25, no. 2, pp. 211–220, Jun. 1999. [21] MODTRAN4, 1999. [Online]. Available: http://www.kirtland.af.mil/ library/factsheets/factsheet.asp?id=7915 [22] E. R. Dougherty, An Introduction to Morphological Image Processing. Bellingham, WA: SPIE, 1992. [23] A. M. McIvor, “Background subtraction techniques,” in Proc. Image Vis. Comput., 2000, pp. 143–153, 2000. [24] R. N. Clark and T. L. Roush, “Reflectance spectroscopy: Quantitative analysis techniques for remote sensing applications,” J. Geophys. Res., vol. 89, no. B7, pp. 6329–6340, 1984. [25] J. Blackburn, M. Mendenhall, A. Rice, P. Shelnutt, N. Soliman, and J. Vasquez, E. O. Drummond and D. R. Teichgraeber, Eds., “Feature aided tracking with hyperspectral imagery,” in Proc. SPIE Signal Data Process. Small Targets, 2007, vol. 6699, pp. 66990S-1–66990S-12. [26] , C.-I. Chang, Ed., Hyperspectral Data Exploitation: Theory and Applications. New York: Wiley-Interscience, Apr. 6, 2007.

Tao Wang (S’09) received the B.S. degree in computer science from Stony Brook University, Stony Brook, NY, in 2002, and the M.Eng. degree in civil engineering from Cornell University, Ithaca, NY, in 2004. He is currently working towards the Ph.D. degree at the Department of Computer Science, City University of New York Graduate Center, New York. Since 2006, he has been a Research Assistant in the City College Visual Computing Laboratory, Department of Computer Science, City College of New York, where he is currently engaged in research on multimodal sensor design and integration, and video surveillance.

Zhigang Zhu (S’94–A’97–M’99–SM’05) received the B.E., M.E., and Ph.D. degrees, all in computer science from Tsinghua University, Beijing, China, in 1988, 1991, and 1997, respectively. He is currently a Full Professor in the Department of Computer Science, City College of New York. He is also with the Department of Computer Science, Graduate Center at the City University of New York. He is also the Director of the City College Visual Computing Laboratory, and a Codirector of the Center for Perceptual Robotics, Intelligent Sensors and Machines at the City College of New York. He is also with the Department of Computer Science, City University of New York Graduate Center. He was an Associate Professor at Tsinghua University, and Senior Research Fellow at the University of Massachusetts, Amherst. He is the author or coauthor of more than 100 technical papers published in various international journals and conference proceedings. His research interests include 3-D computer vision, human–computer interaction, virtual/augmented reality, video representation, and various applications in education, environment, robotics, surveillance, and transportation. Prof. Zhu is a Senior Member of the ACM and has been served as an Associate Editor of the Machine Vision Applications Journal.

Erik Blasch (SM’06) received the B.S. degree in mechanical engineering from Massachusetts Institute of Technology, Cambridge, the Master’s degrees in mechanical, health science, and industrial engineering from Georgia Tech, Atlanta, the MD/Ph.D. degrees in mechanical engineering/neurosciences from the University of Wisconsin, Milwaukee, and the M.B.A., M.S.E.E., M.S. (in economics), M.S./Ph.D. [in psychology (ABD)], and Ph.D. degrees (in electrical engineering) from Wright State University (WSU), Dayton, OH. He is, currently with the Air Force Research Laboratory (AFRL)/RYAA Evaluation Branch, Wright-Patterson Air Force Base, Dayton, OH, an Information Fusion Evaluation Tech Lead for the AFRL—Comprehensive Performance Assessment of Sensor Exploitation Center (AFRL/RYAA), Adjunct EE and BME Professor in the Department of Electrical Engineering, WSU, and Air Force Institute of Technology, and a Reserve Major with the Air Force Office of Scientific Research (AFRL/AFOSR). He is the author or coauthor of more than 300 scientific papers and book chapters. He was a Team Member of the Winning 1991 American Tour del Sol Solar Car Competition, the 1994 AIAA Mobile Robotics Contest, and the 1993 Aerial Unmanned Vehicle Competition, where they were first in the world to automatically control a helicopter. Since that time he has been engaged in research on automatic target recognition, targeting tracking, and information fusion research. Dr. Blasch is an active member in IEEE AES and editor for the IEEE SYSTEMS, MAN, AND CYBERNETICS. He is a Fellow of the International Society for Optical Engineers. He was a Founding Member of the International Society of Information Fusion (ISIF) in 1998 and the 2007 ISIF President.