3D Display Applications for Defense and Security

Invited Paper

3D Display Applications for Defense and Security Scott D. Robinson and Patrick J. Green (Planar Systems, Beaverton OR 97006) ABSTRACT Displays used for defense and security can often be a critical component in analysis where image data needs to be converted to actionable information with accuracy and speed. In situations where complex and/or time-critical image data are being processed, 3D displays have been historically used to maximize accuracy, comprehension and efficiency in the analysis process, and could be used more widely today. While the ideal 3D display technology does not yet exist, viable 3D display products have become available in recent years for many applications. These new products are driven by advancements in display technology at large. This paper will provide a brief overview of 3D perception and imaging, an overview of current 3D display technology and a discussion of current and potential near term applications for 3D displays. KEYWORDS: displays

3D displays, stereoscopic 3D displays, military 3D applications, geospatial analysis, medical 3D

1. INTRODUCTION The process of converting visual data to usable information has become increasingly challenging in recent years due to the increasing complexity of available imagery and in some case, its sheer volume and time-critical nature. Whether acquired directly or computer generated (CG), much of these data can contain three dimensional information. However, while so-called “3D graphics” are currently widespread and are definitely an enhancement over two dimensional representations, they do not make full use of the human visual system. Viewing complex image data on a 3D display provides the possibility of extracting information faster and more accurately, saving time and improving efficiency. The importance of image analysis for defense and security applications makes it critical that professionals have the best possible tools making full use of human perception. In addition, the proven attention-getting nature of content shown on 3D, as demonstrated in its past and present use in entertainment; make utilization of 3D display technology also potentially attractive for simulation, education and training of defense and security professionals. In most computer graphics applications sophisticated algorithms use depth cues such as relative size, interposition, perspective, light shading and others to stimulate the perception of depth. These widely-used monoscopic depth cues do not employ the most powerful source of human depth perception, a subconscious process called stereopsis that takes advantage of the fact that our two eyes receive slightly different views of the world because of their horizontal separation. The human visual system compares these differences and translates them into perception of depth. See Figure 1. With the eyeball to eyeball spacing, the so-called interocular distance, of about 65 mm, depth perception extends to a distance of approximately 13 meters in normal human vision . Stereopsis was first described by Wheatstone 1 in 1839 and interest in stereoscopic imaging has existed since the birth of photography in the 1840’s. Since then there have been several periods of interest in 3D imaging, including the popularity of the Holmes Viewer in the 1890s, the use of 3D viewing with the ViewMaster™ toy from the 1930’s, and the 3D movie boom in the 1950’s that has been revived recently 2. However, these sporadic uses for 3D imaging have not been widespread and primarily limited to entertainment. Until relatively recently the additional computing burden and lack of suitable content and the ability to visualize it have made the everyday professional use of electronic stereo 3D visualization difficult and of limited productivity. These factors have largely been reduced or eliminated with the availability of affordable and powerful personal computers containing high-capacity CPUs with a large amount of low-cost memory. Equally important is the dramatic advances in graphic processing units (GPUs) critical to 3D graphics processing 3. Also, the use of OpenGL and DirectX-based software has facilitated adapting a wide range of applications to stereo 3D viewing. There are three primary methods to obtain 3D content suitable for use with 3D displays. The most common and straightforward path is to directly acquire a stereo pair of images, either as a still or as video. Ideally the stereo pair

Display Technologies and Applications for Defense, Security, and Avionics II, edited by John Tudor Thomas, Andrew Malloy, Proc. of SPIE Vol. 6956, 69560B, (2008) · 0277-786X/08/$18 · doi: 10.1117/12.785009 Proc. of SPIE Vol. 6956 69560B-1 2008 SPIE Digital Library -- Subscriber Archive Copy

would be acquired simultaneously, but for aerial or satellite images this may not be possible or practical. Images for 3D presentation have been obtained in the infrared, x-ray and microwave region (conventional radar and SAR, synthetic aperture radar), with visible laser light (LIDAR), and of course, in the visible spectrum. The field of view for stereoscopic 3D perception can be extended beyond the normal human range by increasing the separation between the two acquisition sources. The imaging device (a pair of cameras, for example) are separated by about 63 mm (the human inter ocular distance) for realistic stereo viewing. The separation may be quite large for satellite imaging where terrestrial relief is being analyzed or it may be quite small for stereo microscopy. A second approach to 3D content involves use of computer generated imagery. This can be created either from an image engine, as is the case of computer animation, or from volumetric data such as that acquired via tomography or some other 3D sensor array, and subsequently rendered. CG content is inherently three dimensional but in conventional “3D graphics” is presented only with monoscopic depth cues (perspective, light shading, etc). Modern CPUs and the GPUs in graphics cards are fully capable of rendering dual, simultaneous views of a given CG figure in real time. A stereo pair of simultaneous views is presented with an angular separation typically within a range from a few degrees up to a maximum of about 10 degrees (often adjustable within the application). Less parallax may not provide sufficient depth perception. A majority of people are not able to fuse a stereo pair that exceeds ten degrees of separation. Most contemporary PC video games can be viewed in 3D stereo with this technique using readily available firmware drivers 4. There is also freeware and more sophisticated commercially-available software suitable for use with images acquired using computed tomography (CT), magnetic resonance imaging (MRI) 5, 6 or arrays of seismic sensors 7. A third path to 3D image data is creation of a stereo pair from a single monoscopic image. To do this, a fairly complete knowledge of the scene at the time of acquisition is needed, including feature-to-camera dimensions. This method is being employed to create stereoscopic 3D recreations of existing 2D feature films 8. The process uses the existing film content for one eye’s view and creates the complimentary stereo pair image for the other eye using proprietary software. Records from the filming of the movie provide the needed dimensional data. Historically one significant issue with use of stereoscopic 3D displays has been viewer discomfort. This can be caused by shortcomings in the display or by the use of content that is inappropriately prepared or presented. The human visual system interprets horizontal disparity as a potential depth cue, but there is nothing in the process of stereopsis that deals with vertical disparity. Vertical misalignment in the projection of stereoscopic 3D movies in the 1950s and the discomfort it caused has been described as one of the reasons these presentations were not more successful. Minimizing vertical disparity is especially important in the acquisition of the stereo image pair. As mentioned previously, presenting stereo image pairs with separation beyond the user’s ability to fuse the stereo pair (>10 degrees) is a certain path to discomfort. Another source of discomfort is flicker, a by-product in the operation of some time modulated stereo 3D displays. Flicker can come directly from the display in use, from other displays in peripheral view or from pulsed, e.g. fluorescent, lighting in the area. It is important to note that at least 5% of the population either has difficulty or is incapable of stereopsis-based depth perception. Tests for depth perception are available 9 and should be used to test professionals examining 3D content.

2. 3D DISPLAY TECHNOLOGY Providing a high quality digital 3D image in a manner that is user friendly and cost effective to the viewer is a challenging display design exercise. Historically most professional 3D displays have been desktop CRT-based, but with the advancements in AMLCDs (active matrix liquid crystal displays) and MEMs (micro electromechanical) technology several new approaches have been introduced. The following table summarizes performance attributes pertinent specifically to 3D displays with a comment regarding ideal performance:

Proc. of SPIE Vol. 6956 69560B-2

Key Performance Traits of 3D Displays Parameter Resolution in 3D

Goal Same as 2D display

Image quality

Chromaticity, response time, etc same as 2D mode

Stereo crosstalk between left and right eye

Less than 0.5%

User comfort

Same as 2D displays, e.g. no flicker

Viewing angle

Same as 2D displays, i.e. multi-user

Luminance

Sufficient for use in normal room light

Screen size

Same as 2D displays

Ease of interfacing

Same as 2D displays

Ability to convert between 2D and 3D

Required

Footprint

Same as 2D displays

Need for eyewear

No eyewear would be preferred

Note that it is very difficult to quantitatively compare different 3D display technologies for some key parameters such as overall contrast, stereo contrast (essentially a crosstalk ratio between the stereo pair of images), user fatigue/discomfort, and in some cases, resolution, because of the various means in which the 3D presentation is achieved. This can make a parametric comparison between some products very difficult. The authors believe this is a consequence of the relative newness of the market that will be resolved as 3D displays are used more widely. The International Committee for Display Metrology - Society for Information Display (ICDM-SID) has established a subcommittee focused on the metrology of 3D displays to initiate a solution to this problem. There is a summary available that covers the display requirements of the National Geospatial-intelligence Agency (NGA) that includes stereo 10. This standardization effort for 3D displays is very much a work in progress. To date no 3D display offers all the optimum characteristics listed in the table above, although shortcomings are being diminished as new designs are introduced. In the remainder of this section we will attempt to summarize the leading 3D display technologies with some comments about these parameters. The discussion will include 3D desktop and projection technology. Electronic displays that present volumetric imagery can be divided into two classifications: stereoscopic 3D and volumetric 3D. Stereoscopic 3D displays, the more common of the two, present a stereo pair of images, one for the left eye and one for the right, in a fashion that segregates the two views between the eyes of the viewer. This can be accomplished using polarization, color separation, mechanical parallax, optical parallax, and/or synchronized frame sequential separation, as will be discussed below. A volumetric 3D display creates real spatial separation in presenting an image, typically via projection or with a moving screen, or both. The most obvious distinction between these two approaches is that when the viewer moves with respect to a volumetric 3D display, he/she sees a new perspective on the displayed image. Thus, a group of viewers surrounding a volumetric 3D monitor will each have their own viewing perspective. In a stereoscopic 3D display, where the technology allows multiple viewers, all see the same image. The most successful volumetric 3D display currently is a so-called swept-volume design 11 where a 10-inch circular screen is rotated at 900 rpm and illuminated by a DLP™ projection engine. This approach provides a full color image composed of 198 slices projected at XGA resolution (1024x768 pixels) in a 60Hz frame (or volume) time. The display has a 360° viewing angle where each observer has their own unique perspective. The transfer of voxel (volumetric pixel) data is a severe challenge that had to be overcome in the product development. This data rate potentially limits the availability of higher resolution in this design. Other limitations include cost and visual artifacts that occur when different colors are presented in the viewer’s line of sight through the image. The viewer perception in the overlap is the additive color. Other volumetric displays have been offered as prototypes 12, 13, but to the authors’ knowledge, are not commercially available.

Proc. of SPIE Vol. 6956 69560B-3

Time-multiplexed displays using CRTs with fast-switching liquid crystal shutters have been the most widely used 3D displays until recently. These present alternating left eye/right eye images frame sequentially at twice the standard refresh rate 14. Two approaches are commonly used. In one design an LC shutter is placed in front of the CRT screen that switches between clockwise and counter-clockwise circular polarizations. Wearing passive, crossed circular polarizing glasses permits the segregation of the left eye/right eye images for stereo viewing. In the other approach glasses containing LC shutters as eyepieces are optically synchronized with the frame sequential CRT presentation of the stereo images. These shutter glasses are battery powered and historically have been bulky and somewhat heavy, although more recent designs are not much more obtrusive than passive glasses. Frame-sequential, CRT-based stereo monitors typically have low luminance, requiring use in a darkened room. They are also prone to flicker which can cause discomfort. A significant logistical problem has arisen of late in that most CRT monitors used in these systems have gone end of life in their production due to the emergence of competitive flat panel display alternatives. A recent innovation in AMLCD design has brought products nominally driven at 120Hz to market to minimize motion blur. These displays may eventually be applied to frame sequential stereo. Note that the same principles of employing framesequential presentation of stereo pairs to achieve 3D viewing are also used in projection 3D stereo. This will be discussed below. Active matrix liquid crystal displays (AMLCDs) have desirable characteristics that include excellent image quality, high resolution, flicker-free viewing, ease of computer interfacing and wide viewing angle. AMLCDs are currently available in a sizes from less than one-inch diagonal to over 70-inch in screen resolution up to 16 MPixels. Prototypes exceeding 100-inch diagonal have been demonstrated. This technology is being used in several stereoscopic 3D products described in the following paragraphs. So-called autostereo displays provide the very desirable quality of providing stereo 3D viewing without the use of eyewear. AMLCD-based designs provide a spatial separation of the stereo image pairs through use of a converging pair of optical paths that project the stereo images to a specific location (one for each eye) relative to the display. When the user’s eyes are positioned appropriately in this viewing zone, depth perception is created with appropriate content. This is accomplished using an AMLCD with a lenticular lens 15 or a parallax barrier 16 as shown in Figure 2. The designs using the parallax barrier or lenticular lens place these optical elements in the path of backlight illumination to create the stereo viewing zones. In both designs, stereo 3D image pairs are thus generated at the expense of display resolution. The highest resolution autostereo AMLCD has one viewing zone, but in this approach the 3D viewing angle is severely restricted to head location normal to the display. It is possible to program the displays with several viewing zones along with appropriate optics to increase viewing angle, but this further reduces display resolution 15. These autostereo monitors seem to be well-suited for gaming and advertising applications; however, they are a more difficult fit for professional uses where higher resolution and artifact-free image quality is required. There are autostereo display designs based on dual light paths that employ LCOS 17 or dual CRTs 18 image sources and separate optical paths. These designs take advantage of the excellent image quality of their respective display technology where full resolution and excellent color rendering is made available in stereo. However, viewer head movement is typically restricted in order to maintain a stereo 3D view and use is limited to a single user. In addition the cost of the optical path components makes these displays fairly expensive. An additional variation on the dual-optical path approach is to use a head-mounted display (HMD) where the miniature displays designated for each eye are driven with the stereo image pair. AMLCD 19 and organic light emitting diode (OLED) 20 miniature displays have been used. These displays provide excellent chromaticity at full resolution, mobility and a potentially rugged approach to 3D stereo. The immersive nature of HMDs can cause discomfort with extended use for some viewers. Also the isolation inherent in the use of a HMD can be a disadvantage for some applications where interaction with co-workers is needed. The polarized light-emitting nature of LCDs has been exploited for use in stereo 3D displays. A relatively recent approach, called the StereoMirror™, combines the output of two AMLCDs into a 3D stereo image using a novel beamsplitter design 21. The two AMLCDs are oriented at a fixed angle with the beamsplitter mirror bisecting the two monitors. This product is shown in Figure 3. The polarization in the reflective path is effectively rotated 90° with respect to its origin while the polarization in the transmitted light path is relatively unaffected. This results in the stereo pair of images being directed to the viewer with one view at crossed polarization with respect to the other. When seen with similarly cross-polarized glasses, a high contrast 3D stereo image is perceived. The design provides the flicker-free image quality at the full resolution of the AMLCD and with performance attributes equivalent to 2D AMLCDs. The design can be readily extended to most any AMLCD; stereo monitors up to 26-inch diagonal and 5MPixel have been

Proc. of SPIE Vol. 6956 69560B-4

demonstrated. This StereoMirror approach provides the highest image quality in stereo currently available. Since the display uses linear polarization, there is the possibility of increased stereo crosstalk with head tilt and the form factor, with its protruding mirror may not be advantageous for all environments. Another recent stereo 3D display design offered by several vendors makes use of dual layered AMLCDs with modified optical films. One panel modulates the pixel intensity and the other controls the distribution of light between the two eyes 22, 23, 24 via a high level of image processing to prepare the stereo pair images for display. See Figure 4. A collimated backlight is used with circularly polarized glasses. This design provides roughly the form factor of a conventional AMLCD or a thin CRT with very good image quality and wide viewing angle. A privacy mode is available in one design 23 where the screen content is only visible to a user wearing polarizing glasses. There currently is not much experience in the market with these displays but it is likely the need for a backlight collimation will limit resolution in product offerings without excessive pixel to pixel crosstalk. Currently the highest resolution available is SXGA (1280x1024). The backlight collimation requires a scattering film as the final optical element in the optical path. This film can cause image degradation in some designs. A relatively recent 3D stereo design using an AMLCD makes use of a proprietary process for creating relatively high resolution, patterned polarizers 25. Orthogonal polarization is alternated in horizontal stripes corresponding to match the pitch of the pixel rows in the LCD. The stereo information is alternated vertically, row by row between the two eyes, corresponding to the pitch of alternating polarization. Orthogonally polarizing glasses are worn to perceive 3D. While this approach reduces the vertical resolution of the native display at least in half and may be limited to a minimum pitch and viewing angle range, the resultant stereo image achieves optical performance similar to the native AMLCD. The form factor is also that of the original AMLCD. This design may find wide use for larger displays in PC gaming applications that might be less sensitive to the loss of resolution and vertical viewing angle. Currently there are at least four commercial approaches for the projection of 3D stereo content, principally based on DLP™ engines 26. The most straightforward path is to employ two separate projectors, driving each with one stereo view as shown in Figure 5. This is analogous to the dual-film projector system used for stereo cinema in the 1950’s. As with the film projectors, orthogonal polarizers are placed in front of the projection lens with crossed polarizing glasses to differentiate the left eye data from right eye. There is a need to use a non-depolarizing screen, such as a metallized plastic film, so that depolarizing scatter does not reduce or destroy stereo contrast. With the dramatic reduction in cost of small projectors of reasonable quality, this is an easy way to project 3D stereo. Because of the offset of the two projectors with respect to the normal, a keystone correction may be needed to achieve alignment.. The most common two stereo 3D projection systems available commercially are analogous to the two stereo CRT designs. In both designs the projection engine is driven at twice the standard frame rate and as with the CRT approach, the left eye/right eye images are interposed at an equal rate. One method, described in Figure 6, uses a fast-switching liquid crystal cell positioned in front of the projection lens to alternate between opposing circular polarizations. Passive circular polarized glasses are employed to see stereo. As with the dual-projector system, a non-depolarizing screen must be used and this can make a conversion from a conventional, scattering screen an expensive proposition, especially for large venues. The other method makes use of powered shutter glasses 27 and requires that the synchronizing emitters be dispersed throughout the hall. See Figure 7. The attenuation of the projected light for both these projection systems is quite high, resulting in projected luminance less than one third that of the standalone projector. Both of these techniques are currently in use in commercial 3D cinema and can also be used in smaller settings. A newer approach to stereo projection has recently been introduced for 3D cinema 28 that may find wide use. Stereo separation is accomplished through color separation by using two sets of narrow red, green and blue (RGB) filters for each eyepiece, as shown in Figure 8. The filter sets, fabricated using thin dielectric films, are designed to have no spectral overlap. This provides stereo separation when video is transmitted through the eyepieces frame sequentially. A pair of RGB filters with the same spectral properties is used in the DLP color wheel, alternating the left eye/right eye data between the two. See Figure 9. There is a significant advantage in this design in that there is no requirement to make extensive modifications to an existing theater, either to the screen or with the addition of an emitter array. We are not aware of any published data regarding the performance of this design, but one of the authors (PJG) has attended a demonstration and the image quality was excellent in stereo 3D. The overall luminance is no higher than the other designs, but the ability of this approach to be used without a special screen or sensor array is very attractive.

Proc. of SPIE Vol. 6956 69560B-5

3. APPLICATION FOR 3D DISPLAYS Geospatial Analysis One of the earliest uses for professional 3D imaging was geospatial imaging. The analysis of stereo 3D aerial photographs during the Cuban missile crisis in 1962 lead to the identification of Russian intermediate range nuclear missile sites only 90 miles from Florida. The Russians had placed a ring of defensive surface-to-air missiles surrounding the offensive missile launch area that was camouflaged with netting. The stereo images rendered the camouflage ineffective, revealing the true purpose of the site. As a result of this analysis, using film-based stereo images, President Kennedy was able to confidently expose the intentions of Russians and defuse the crisis 29. A task as simple as determining whether a geographic element is a ridge or valley can be very difficult to determine from a single aerial image, but can be resolved with a stereo view. More complex tasks, such as making accurate measurements in three dimensions of geographic features or terrain are also enabled by 3D imaging. Stereo pairs of aerial or satellite images provide data for the extraction of measurement information based on the principles of photogrammetry 30. Three dimensional coordinates on an object or terrain feature are determined using measurements made with two or more images taken from different positions. Common points from each image are used to merge the data employing commercially-available software. Line of sight rays are constructed connecting the camera point to the given points on the object. From the intersection of these rays the three dimensional location of the point can be determined via a process of triangulation. More recently 3D data for geospatial analysis is routinely acquired using (SAR synthetic aperture radar), LIDAR (light detection and ranging), as well as with optical imaging in the IR and UV. Use of stereoscopic 3D imaging permits analysis and visualization of complicated spatial relationships that would be difficult or impossible to decipher in a 2D analysis only. These can be critical for an accurate situational analysis assessment, especially in an urban environment 31. Here are examples of tasks enabled or enhanced by 3D stereo images: - Development of a DTEM (Digital Terrain Elevation Model) for cartography - Creation of mobility assessment, i.e. identification of ingress and egress options for mobile land forces using 3D terrain models and flight paths for manned or unmanned aircraft, including helicopters - Threat analysis, including lines of fire, location and elevation of hostile versus friendly forces, etc. - Identification of cell phone or other communication paths or obstructions - Accurate estimation of the size and access to buildings - Evidence of recent activity e.g. identification of the relief in snow or gravel along a freshly-plowed road - Simulation of mission plans, e.g. mock-up of an extraction, threat assessment

Medicine The potential medical advantages of stereoscopic viewing were appreciated very early in the development of radiography. Only a few months after the discovery and public disclosure of x-rays by Röentgen in 1895, there appeared an article by E. Thomson describing the acquisition and viewing of stereoscopic x-ray images 32. The medical value of stereoscopic x-ray imaging for localization of tissues and seeing structures in depth was soon appreciated by Sir James Mackenzie Davidson, a prominent British physician, who published an article 33 in the British Medical Journal in 1898, and later, in 1916, published a book containing many illustrative stereo x-rays that demonstrated the utility of stereoscopic x-ray imaging 34. There is also a long history of using 3D stereo in the teaching of anatomy 35. A recent revolution in medicine has involved the adoption of minimally invasive surgical techniques that significantly reduce surgically-caused injury to the body by eliminating or minimizing large entry incisions in a given surgical procedure. This approach reduces recovery times dramatically and thus potentially requires less infrastructure to support the post-operative patient, an outcome important both to military and civilian medical care. However, the degree of difficulty for the surgeon is amplified because they must perform most or all the procedure using real time images taken from a small probe inserted into the body and viewed (typically) on a 2D video monitor. For routine procedures such as appendectomies and gall bladder removal, 3D imaging is not needed for the experienced surgeon. However, for delicate procedures involving the heart, the prostate or in delicate wound intervention, specially designed 3D displays have been developed to compliment remotely controlled robotic surgical systems 36. These robotic surgical procedures have attracted strong interest from the military as well and although the current cost of these systems, shown in Figure 10, is

Proc. of SPIE Vol. 6956 69560B-6

prohibitive, the intention is to provide immediate medical care to wounded warfighters while keeping physicians removed from the threat of battle. Stereoscopic 3D HUDs are also available 20 for minimally invasive surgery and offer the advantage of ready mobility. See Figure 11. Reduction of patient time in surgery and/or in recovery can provide a critical advantage in a military medical situation. Improving surgeon endurance and limiting fatigue with improved efficiency can be equally important. Another revolution in medical imaging over the past few decades has resulted in the availability of remarkable internal views of the body on a routine basis. However, one of the challenges of medical imaging today is determining the best method to present volumetric image data to the caregiver in an efficient manner and in a way that makes full use of the information contained in the image data. Currently volumetric images are available from a variety of modalities such as computed tomography (CT), and magnetic resonance imaging (MRI), ultrasound (UL), and others, but the output is typically viewed in 2D, often only as slices of a particular study. As the resolution of these acquisition techniques improves, the number of slices to be examined increases, adding to the workload and reducing the efficiency of the medical professional responsible for reading the images. Clinical trials making use of 3D stereoscopic displays are needed to provide data demonstrating the improvement possible with stereoscopic 3D visualization. One such trial 37 will be briefly referenced here to illustrate the potential value of stereo 3D imaging in radiology. Mammography is considered one of the most difficult radiographic exams to interpret. The complexity of breast x-ray images makes it difficult to discern features of interest among over- or underlying breast tissue. A stereoscopic 3D analysis of mammographic images holds the promise of improving the early detection of breast cancer by providing depth cues to mammograms which currently are 2D projections. A clinical trial of stereoscopic digital mammography versus standard digital mammography in a screening setting has just been completed at the Emory University Breast Imaging Center in Atlanta, Georgia 38. The preliminary results from the trial are striking. In the case sample presented at the 2007 Radiological Society of North America (RSNA) meeting, stereo mammography reduced false negative readings by 40%. This result strongly suggests that stereo mammography is more sensitive than standard mammography in detecting true lesions. Equally impressive, stereo mammography reduced false positive lesion detections in the described sample by 49%. This result is also clinically significant. A false positive finding is a diagnostic result that is found to be incorrect in a subsequent exam, while a false negative is a situation where a potentially cancerous lesion is not detected. A 5MPixel grayscale version of the Planar’s StereoMirrorTM was used in the Emory study. The improvement in screening mammography that could be afforded by stereo mammography would relieve many women from the considerable stress and anxiety produced by unnecessary recalls, result in substantial annual financial savings, and ease the load on already overburdened systems for screening mammography. Extending the advantages of stereo 3D imaging demonstrated in the Emory study will potentially benefit both military and civilian medicine. Simulation and Training Stereoscopic 3D visualization of complex datasets from simulation, experimentation and modeling results has long been utilized by the US weapons laboratories. A stated mantra is “to see is to know” and “to know is to understand” 39. For example, a 43 million pixel stereo 3D projection system composed of 33 projectors has been used to maintain the U.S. government nuclear stockpile without underground testing. The immersive visualization system comprises 15-ft. wide by 10-ft. deep and 12-ft. high room in which images are rear-projected onto three walls, the ceiling and the floor. In addition to extensive use of desktop 3D stereoscopic displays, these so-called CAVES™ or CAVE Automatic Virtual Environment are widespread in the US National Labs 40. Recently a six-sided 100MPixel visualization center was installed at Iowa State University 41, supported by the US Air Force Office of Scientific Research. The so-called C6 facility will be used in part to develop a control interface for the military's next generation of unmanned aerial vehicles. The researchers are building a virtual environment that allows operators to see the vehicles, the surrounding airspace, the terrain they're flying over as well as information from instruments, cameras, radar and weapons systems. The system would allow a single operator to control several vehicles simultaneously. Stereo 3D presentations are often featured at theme parks and museums because the impact is typically attention-getting and memorable. These two outcomes are also the goals in a training situation. In recent years the phenomenal advancements in PC game software engines that have lead to highly realistic PC games are being used to create software for simulation and training situations. These so-called “Serious Games” represent a substantial emerging market 42. While the “3D graphics” displayed on a conventional 2D display are typically quite stunning, the ease with which this same content can be presented in 3D stereo using available firmware drivers 4 with available projection or desktop 3D

Proc. of SPIE Vol. 6956 69560B-7

stereo should not be ignored. Presentation of training materials in stereo 3D could potentially provide significant improvements in performance and content retention. Luggage and Container Inspection Occasional long delays for x-ray inspection of carryon luggage are a reality for the frequent airline traveler today,. In addition to the inconvenience and expense these delays cause, the Transportation Safety Administration (TSA) employees assigned to visually analyze the x-ray images are placed under a sustained level of stress. In real time they must make go/no go decisions in viewing a 2D projection of 3D objects from the X-ray system. Under- or over-lying parcels may have been intentionally placed to hide or mislead the inspector in an attempt to conceal contraband and/or dangerous materials. The mental concentration required to sustain this task during an entire workshift, day after day, can be quite taxing, and worse, potentially prone to fatigue-caused missed detections. Proposals have been made to replace the conventional 2D x-ray systems with scanners capable of creating a volumetric image using a dual source system 43. Computed tomography (CT), a widely-used medical modality, could be adapted to this application as well 44 with the assumption that this will improve detection accuracy and reduce stress. The results from the Emory stereo mammography study mentioned earlier can be used to make a case for stereo 3D visualization here. The significant reduction in both false positives and false negatives for mammograms, another very complex analysis currently viewed in 2D, could indicate similar improvements may be realized with 3D stereo inspection of luggage. In addition it is expected this type of system would improve throughput, reduce inspector stress and its side effects, and most importantly, increase detection probability. The stereo display used for this application would have to be capable of providing excellent image quality, resolution comparable or exceeding the sensor array, and no issues with user comfort. More than 11 million shipboard containers move through US ports annually 45. This influx makes individual container inspection impractical. While only a small fraction of these are currently inspected, the threat of importing a WMD into the heart of a major city is very real. Scanning of these containers by x-ray or gamma-ray irradiation has been proposed as a method to mitigate this threat 46. As in the case of x-ray luggage inspection, the challenge is to present a volumetric image to the inspector in a manner that will cause minimal fatigue and yet provide the perceptual context with which to effectively screen for contraband and threatening cargo. See an example of a gamma-ray container scan in Figure 12. A prototype dual-scanning, gamma-ray system has been proposed that would create a stereoscopic 3D representation of the contents of the container volume 47. The stereo 3D data potentially allows for accurate measurement and representation of the contents while still presenting a realistic volumetric view to the inspector. Vehicular Vision Currently an excellent example of a vehicular navigation system using 3D stereo visualization exists more than 50 million miles from Earth. Both of the Martian Rovers, Spirit and Opportunity, are equipped with several stereo camera pairs for navigating the Martian landscape 48. See Figure 13 for a picture of the Rover showing the stereo camera. A stereo pair of photographs from Mars 49 is shown in Figure 14. Not evident without a stereo viewer or by showing the images on a stereo display, is a prominent ridge and precipice that runs at about the vertical midpoint horizontally across the image. This sort of feature in the landscape is the sort of hazard the stereo navigation system is intended to identify and avoid. The use of remotely operated terrestrial vehicles has increased rapidly in recent years. The term “remotely operated” here includes unmanned vehicles (land, air and underwater) as well as those where human occupants are not allowed a direct view of the outside world. Examples of the latter case include situations where the operator’s eyes must be protected, when the vehicle occupants must isolated from a chemical or radiological threat, or when night vision is required. These situations usually require the operator to control the vehicle while viewing the outside world either through a viewport with a narrow field of view, such as the vision blocks used in the M1 Abrams tank, or using a monitor, the Driver Vision Enhancer (DVE), for example. Hostile environments involving smoke, fog and dust make a monoscopic view all the more hazardous. Whether manned or unmanned, the disadvantage of limiting the operator’s view to that of a narrow angle or a conventional display is the loss of depth perception. Lack of situational awareness, for example of topographic features shown in Figure 14, can have catastrophic consequences to personnel and mission outcomes. The logical solution is the use of a stereo video acquisition system and a stereoscopic 3D display to restore the operator’s sense of depth. To the authors’ knowledge no stereo 3D display has been qualified to date for use in a military vehicle. An HMD-based stereo display may be a good candidate here. A version of the layered AMLCD approach has been proposed for use in US

Proc. of SPIE Vol. 6956 69560B-8

Army vehicular applications 23. The form factor, similar to that of a standard AMLCD, appears well-suited for this use. Image quality and ruggedization appears to be key concerns. The StereoMirror design would provide the needed image quality, but the form factor and the potential vulnerability of the beamsplitter mirror would have to be dealt with. A key issue beyond the scope of this paper is the design of a ruggedized stereo image acquisition system capable of providing sufficient stereo separation in real time at the maximum speed encountered in the unmanned vehicle. There have been interesting results from the DARPA-sponsored Grand Challenge Race for remote-guided vehicle making use of stereo vision systems that provide some promise of this capability 50. One of the most challenging and potentially dangerous military aircrew tasks involves aerial refueling. In the current designs employed in the KC-135 and KC-10, the aerial refueling operator (ARO) lies on his stomach, peering through a window in the back of the airplane in order to “fly” the refueling boom into place so the receiver aircraft can fill up with fuel. The ARO views the operation directly through a window in the back of the plane. A new system, proposed by Boeing using the 767 Tanker Transport 51, locates the ARO at a workstation directly behind the cockpit in the front of the plane where he/she will be assisted by a automatic “fly by wire” system that locates the refueling boom. The actual refueling process is visualized using a stereo 3D display allowing the ARO to manipulate the boom using a joystick. The display will either be a HMD or a stereo 3D monitor mounted in the workstation. To acquire the real-time stereo 3D view of the operation, two digital cameras will be mounted side by side in front of the where the boom is attached to the 767’s tail. The cameras are synchronized and the image is processed to maintain proper depiction of depth to the operator. Use of the remote-control system eliminates the need for costly airframe modifications otherwise needed to create a viewing window are part of the retrofitting process that will turn the 767’s into tanker aircraft.

4. SUMMARY We have presented several examples where 3D displays currently contribute significantly in defense and security applications and have offered possibilities where this technology might be used in the future. It is the opinion of the authors that 3D displays are currently an underutilized resource in applications where timely analysis of complex images is required. We also believe 3D imaging can be instrumental in advancing the efficiency of simulation and training. Improved awareness of the benefits and ongoing progress in performance and availability of 3D displays are anticipated. While volumetric 3D displays are desirable for the long term to provide more realistic 3D representations, stereoscopic displays will predominate for the foreseeable future. Unfortunately large, high definition holograms of Princess Leia from the first Star Wars™ movie will not be reduced to practice anytime soon, but hopefully this will happen before the 23rd century!

REFERENCES [1] Wheatstone C, "Contributions to the physiology of vision. Part the first. On some remarkable, and hitherto unobserved phenomena of binocular vision", Phil. Trans. Royal Soc, London, 128, 371 (1839) [2] http://www.stereoscopy.com/ [3] http://www.nvidia.com [4] http://www.nvidia.com/object/3d_stereo.html [5] http://www.slicer.org/ [6] http://www.3mensio.com/ [7] http://www.geosoft.com/ [8] http://www.in-three.com/ [9] http://www.stereooptical.com/html/stereo-test.html [10] Grotte, M. D., “NGA Softcopy Exploitation Display Hardware Performance Guideline” SID ADEAC 05, 63 (2005) [11] http://www.actuality-systems.com/ [12] http://www.holografika.com/ . [13] http://www.lightspacetech.com/ [14] Buzak, T.S., “A Field-Sequential Discrete-Depth-Plane Three-Dimensional Display”, SID International Symposium Digest of Technical Papers, p. 345 (1985) [15] Hiddink MGH, de Zwart ST, OH Willemsen, OH Dekker T, "Locally Switchable 3D Displays", SID Int Symp Dig Tech Papers, 37, 1145 (2006) [16] Woodgate, G.J., Harrold, J., Jacobs, A.M.S., Mosely, R.R., Ezra, D., “Flat panel autostereoscopic displays – characterization and enhancement”, SPIE Vol. 3957, and reference therein

Proc. of SPIE Vol. 6956 69560B-9

[17] Cobb J.M., Kessler D., Agostinelli J.A.,. Waldman M, High-Resolution, "Autostereoscopic immersive imaging display using a monocentric optical system", Proc SPIE, 5006, 92 (2003) [18] Ballantyne G.H., Moll F., "The da Vinci Telerobotic Surgical System: the Virtual Operative Field and Telepresence Surgery", Surg Clin North Am, 6, 1293 (2003) [19] http://www.vikingsystems.com/endosite3di.htm [20] http://www.emagin.com/products/index.php [21] http://www.planar3d.com/ [22] Kirsch, J.C, Jones, B.K., “Compact 3D Display Using Dual LCDs”, SID ADEAC 04, 35 (2004) [23] Boyd J.E., Gaudreau M, Bechamp M, "Innovative Stereoscopic Display using variable polarizing angle", Proceedings of SPIE, Vol. 6055, (2006) [24] http://iz3d.com [25] http://www.arizawa.co.jp/en/product/3d.html [26] http://www.dlp.com/projectors/default.aspx [27 Ward C, "A Single Lens, Single-Chip 3D Projector", SID ADEAC 05, 183 (2005) [28] http://www.dolby.com/professional/motion_picture/solutions_d3ddc.html [29] http://www.nationalmuseum.af.mil/factsheets/factsheet.asp?id=1876 [30] McClone, C., “Manual of Photogrammetry, 5th Ed.”, ASPRS pub. (2004) [31] Krauss, T., Lehner, M., Reinartz, P., “Modeling of Urban Areas from High Resolution Stereo Satellite Images“, IPIworkshop, Land Cover and Change Detection II, Hanover, Germany, February 29, 2007 [32] Thomson E., "Stereoscopic Röentgen Pictures", "The Electrical Engineer", 21, 256 (1896) [33] Davidson S.J.M., "Remarks on the value of stereoscopic photography and skiagraphy: records of clinical and pathological appearances", British Medical Journal, 1669-1671 (1898) [34] Davidson S.J.M.. Localization by X rays and stereoscopy. London: H.K. Lewis, (1916) [35] Cunningham DJ, M.D., Edinburgh Stereoscopic Atlas of Anatomy, Keystone Publishing Company, Meadville PA, (1900) [36] http://www.intuitivesurgical.com/index.aspx [37] Getty, D. Green, P., “Clinical Applications for Stereoscopic 3D Displays”, J. of SID, 15(6) 377 (2007), and Getty, D. et al, “Improved Accuracy of Lesion Detection in Breast Cancer Screening with Stereoscopic Digital Mammography”, in press, 2008 [38] http://www.bbn.com/news_and_events/press_releases/2007_press_releases/pr_mammography_112807 [39] http://www.theory.lanl.gov/CINT/capabilities.html [40] http://www.fakespace.com [41] http://www.public.iastate.edu/~nscentral/news/06/may/c6update.shtml [42] http://www.seriousgamessummit.com/ [43] http://www.techbriefs.com/content/view/1049/34/ [44] http://www.analogic.com/security/name/checked.html [45] http://rpc.senate.gov/_files/Sep0606PortSecurityLB.pdf [46] Hitachi Corp, “Cargo inspection imaging X-ray inspection systems”, Hitachi Review, 53(2) 97 (2004) [47] Zhu, Z., Hu, Y.-C., “Stereo Matching and 3D Visualization for Gamma-Ray Cargo Inspection”, Proceedings of the Eighth IEEE Workshop on Applications of Computer Vision, Feb 21st-22nd, (2007) Austin, Texas, USA [48] Smith, G.H, et al., “Optical Designs for the Mars ’03 Rover Cameras”, Proceedings of SPIE, 4441, 118 (2001) [49] http://marsprogram.jpl.nasa.gov/MPF/parker/highres-stereo.html [50] Stiller, C., Färber, G., Kammel, S., “Cooperative Cognitive Automobiles”, Proc. IEEE Intelligent Vehicles Symposium, Seiten 215-220, Istanbul, Turkey, June, 2007 [51] http://www.boeing.com/news/frontiers/archive/2006/november/cover.pdf

Proc. of SPIE Vol. 6956 69560B-10

Figure 1 – The mental process of stereopsis

Parallax Barrier

Lenticular Lens

iI •1 AMLCD Subpixels

AMLCD Backlight

Barrier

Lenslets AMLCD Subpixels

Figure 2 – Two approaches to autostereo AMLCD displays

Figure 3 – Planar SD2420W StereoMirror™ stereo 3D monitor

Proc. of SPIE Vol. 6956 69560B-11

AMLCD Backlight

AMLCD #1

Collimated Backlight

AMLCD #2

Passive Polarizing Glasses Diffuser

Polarizers

Figure 4 – Schematic description of the operating mode for layered AMLCD stereo 3D displays

Dual DLP Projection at Std Refresh Rate

Non-depolarizing Screen

Crossed Polarizers

Polarized (Passive) Glasses

Figure 5 – Dual projector approach to stereo 3D projection

Proc. of SPIE Vol. 6956 69560B-12

Non-depolarizing Screen

DLP Projection at 2x Std Refresh Rate

Circularly Polarizing Optical Switch

Circularly Polarized (Passive) Glasses

Figure 6 – Polarization-switching approach to stereo 3D projection

DLP Projection at 2x Std Refresh Rate

LC Shutter (Active) Glasses

Std Screen

Synchronizing Emitter

Figure 7 – Shutter glass approach to stereo 3D projection

Proc. of SPIE Vol. 6956 69560B-13

Wavelength

Left eye spectrum

Right eye spectrum

Figure 8 –Schematic description of stereo separation for the segregated RGB bandpass filter approach to stereo 3D projection

DLP Projection at 2x Std Refresh Rate

Std. Screen

RGB Bandpass Filter Wheel

RGB Bandpass Passive Glasses

Figure 9 – Segregated RGB bandpass filter approach to stereo 3D projection

Proc. of SPIE Vol. 6956 69560B-14

—

Figure 10 – The da Vinci® System for remote-guided, minimally invasively surgery (Intuitive Surgical, Inc). Note the surgeon using the stereo 3D workstation in the foreground.

Figure 11 – Minimally invasive surgery using head-mounted stereo 3D displays (Viking Systems, Inc.)

Proc. of SPIE Vol. 6956 69560B-15

—C7

.4.p. Th

A':

y

1

Figure 12 – Gamma-ray image of packed cargo containers 45

I

Figure 13 – Artist rendering of a Martian Rover. Note the mast-mounted stereo 3D camera used for navigation

Figure 14 – Stereo photographic pair of the “Twin Peaks” area on Mars 48. There is a significant ridge running horizontally at the vertical midpoint of the photographs. This is made prominent using stereo 3D visualization

Proc. of SPIE Vol. 6956 69560B-16