POAC (Programmable Optical Array Computer) Applied for Target Recognition and Tracking Szabolcs Tőkés*, László Orzó, Ahmed Ayoub, Tamás Roska Analogical and Neural Computing Laboratory, Computer and Automation Research Institute, Hungarian Academy of Sciences, H-1111 Kende u. 13-17, Budapest, Hungary
ABSTRACT A portable programmable opto-electronic analogic CNN computer (Laptop-POAC) has been built and used to recognize and track targets. Its kernel processor is a novel type of high performance optical correlator based on the use of bacteriorhodopsin (BR) as a dynamic holographic material. This optical CNN implementation combines the optical computer’s high speed, high parallelism (≈106 channel) and large applicable template sizes with flexible programmability of the CNN devices. Unique feature of this optical array computer is that programming templates can be applied either by a 2D acousto-optical deflector (up to 64x64 pixel size templates) incoherently or by an LCD-SLM (up to 128x128 size templates) coherently. So it can work both in totally coherent and partially incoherent way, utilizing the actual advantages of the used mode of operation. Input images are fed-in by a second LCD-SLM of 600x800 pixel resolution. Evaluation of trade-off between speed and resolution is given. Novel and effective target recognition and multiple-target-tracking algorithms have been developed for the POAC. Tracking experiments are demonstrated. Collision avoidance experiments are being conducted. In the present model a CCD camera is recording the correlograms, however, later a CNN-UM chip and a high-speed CMOS camera will be applied for post-processing. Keywords: Optical target recognition, optical tracking, optical correlator, optical computing, optical CNN
1. INTRODUCTION Although digital technology manifests extraordinary development in the recent decades, optical correlators - thanks to their enormous inherent parallelism and the technical advancement of the applied sensors and displaying devices - still can provide competitive alternative in special application fields. Optical correlators can rapidly identify partially covered objects in noisy environment; therefore these devices are frequently used in target recognition and tracking applications1,2. However, the optical correlator architectures used so far do not ensure flexible programmability. This drawback is considerably hindering the application of these devices in tracking tasks. VanderLugt (VLC) type of scheme is able to identify patterns at very high speed, but it requires predefined matched filters. Determination of a matched filter is a time-consuming process3. Therefore, the VLC architecture can be useful only in those applications, where a fixed, limited set of patterns are looked for. Contrarily, Joint Fourier transform correlators (JTC) can be regarded flexibly programmable, as it does not require any matched filter4. However, the achievable speed in a JTC system is considerably lower, then that of the VLC one, because all correlation operations involve a joint power spectrum recording step. This hologram-recording step sets considerable demands against the recording medium resolution (space-bandwidth product) and sensitivity. In this paper an original optical correlator architecture will be introduced. We will show that it is superior to the architectures used so far. We combine our optical correlator's vast parallelism with a Cellular Neural Network (CNN) VLSI device's on chip sensing and flexibly programmable processing capabilities5. This hybrid device is called programmable opto-electronic array computer (POAC)6. We have built a portable model version of this device and give some details of its design and outline the features of the applied components. We will demonstrate that it is flexibly programmable and efficiently applicable for adaptive target-tracking task.
*
[email protected] Phone: +36 1 279-6135; Fax: +36 1 466-7503 or 209-5264; http://lab.analogic.sztaki.hu/index
2. NEW OPTICAL CORRELATOR ARCHITECTURE We developed an optical correlator architecture that can be regarded as a modified JTC system7, but similarly as an altered VLC arrangement, too. Additionally, this system can work either in a coherent or in a semi-incoherent way8. The system works as it follows: In the first step we record the joint power spectrum of the input image and a single reference pixel. That is, we record the Fourier hologram of the input image. By an appropriate parallel laser beam we can reconstruct the original input image from this hologram. In the second step we modulate this beam according to the Fourier transform of the applied template that is the pattern we are looking for in the input image, we can get correlation. This Fourier transform can be easily implemented by appropriate positioning of a Fourier optic lens and the displayed template. This way, every single template pixel corresponds to a parallel beam with an appropriate deflection angle. These beams reconstruct the original input images’ weighted and shifted versions. Coherent or incoherent superposition of these reconstructions results in a correlogram. This system resembles to a dual axis JTC system, where the reference object is a single pixel, while the read-out beam is not a simple parallel beam, but a modulated beam bundle determined by the Fourier transform of the template. This architecture can be considered as a modified VLC system, hence first the hologram of the input image is recorded and later this ‘matched filter’ is filtering the template’s Fourier transform. A great advantage of this architecture is that the optical elements’ precise, scrupulous positioning is not required. Even the hologram can be recorded slightly out of the lens focal plane. The only requirement is to ensure the same viewing angle of the template and the input image pixels from the view-point of the hologram9. As the size of the applied templates are much smaller than that of the input image, they can be displayed at a much higher speed on an appropriate device. Furthermore, the laser power requirement of the system is reduced, since each hologram-recording step can be followed by dozens of reconstructions, correlation operations. Therefore, the system performance is limited primarily by the speed of the template-displaying device, the applied read-out laser power and the sensor speed and sensitivity. Scheme of the introduced new correlator architecture can be seen in Fig. 1.
Read out
Write in Template t1 (Single pixel)
Template t2
Square law detector (hologram) Lens
Correlogram
Lens
Coherent plane wave
Input, object
f
f
f
f
Fig. 1. Scheme of the unconventional, flexibly programmable optical correlator architecture. The hologram of the input image applying a t1 reference object (or a single reference pixel) is dynamically fixed as in the case of JTC. Additional programming (by template t2) is carried out in the read-out step. In the second step(s) a great number of different templates can be used, so extremely high correlation speed can be achieved.
3. THE LAPTOP POAC MODEL Based on the experiments with previous breadboard models of this new optical correlator architecture we have designed and constructed an improved, portable size version (Laptop POAC). Although, not all the final conceptually optimal elements of the system are built-in yet, its feasibility, applicability and special advantages have been successfully demonstrated. The schematic structure of the two-wavelength optical correlator can be seen in the next Fig. 2. The space-bandwidth product of the optically addressable spatial light modulator (LC OASLM), applied in our earlier experimental setup as a square law detector (holographic recording device), was too low. Therefore, alternative highresolution dynamic holographic materials were considered to increase the resolution of the system. First a special photochromic polymer film was examined, but due to this material’s wavelength and laser power requirements, an alternative dynamic holographic medium, the Bacteriorhodopsin had been chosen. It turned out to be an excellent choice (details of this material will be given later).
A 2D acousto-optical deflector was used to display the read-out t2 template. By the use of this device it was easy to set the required angular deflections. The speed of this device is superior to other micro display devices for relatively small resolution templates. An appropriate telescopic system simultaneously transformed the spot size and the angle of the pixel-beams impinging onto the hologram. As in this setup we use two different wavelengths for hologram recording (532 nm) and reconstruction (658 nm), and the thickness of the recording medium cannot be considered thin, we recorded Bragg type holograms. Earlier, when only single wavelength had been used, the reflection type (Lippmann-Denisyuk) hologram recording technique was also successfully tested. The reason of using different color lasers for recording and for reconstruction will be given in section 4.3. Output
Input Programm
Fig. 2. Architecture of the tow-wavelength Laptop POAC system
In this correlator setup we applied a slow CCD camera as an output device. Although this sensor was an adequate measuring device, it limited the whole setup’s speed to 30 frame/seconds. In the further development of the POAC this sensor will be replaced by a visual CNN sensor and processor array. Hence, not only the sensor speed limitations will be alleviated, but also an adequate post-processing unit will be included. Input, output and other parts of the POAC system require proper control. These controls are accomplished by a PC. Outline of the required tasks and controls can be seen in the next figure (Fig. 3). Displaying results (output) Microdisplay (input) Microdisplay (template) Control and Timing Camera (input) Camera (output) AOD control (template) Lasers controls
Fig. 3. Schematic view of the Laptop POAC control.
Arrangement and parts of the POAC system implementing the hologram-recording step can be seen in the next figure (Fig. 4). Two different type of template displaying technique, correlation have been implemented. Partially incoherent correlation can be achieved by the application of a 2D acousto-optic deflector. In this case, the reconstruction using each template pixel is coherent, but their summation is incoherent. Coherent reconstruction is also possible in the Laptop
POAC system switching to the built-in template-LCD for coherent template display implementation. The setup of the template arms can be seen in Fig. 5. Filter
CCD or CNN Camera
BR
Dichroic Mirror
Mirror Beam Expander
Green Laser
Microdisplay
Fourier optic Lens
Mirror
Polarizer
Template Beam
Polarization Beam Splitter
Polarization Rotator
Mirror 5 Beam splitter
Polarization rotator
Beam Expander
Fig. 4. Object (hologram recording) arm architecture of the Laptop POAC system. Appropriate beam attenuation, expansion and path lengths are set in both image and reference paths to assure the high quality dynamic hologram recording into the Bacteriorhodopsin film. Read out (template path) beams are denoted by dashed lines. Before the sensor (CNN VLSI chip or CCD camera) an appropriate filter cut the green laser of the object path. dichroic mirror
mirror
bacteriorhodopsin
polarization rotator template SLM polarization rotator
mirror 2nd lens of telescope and FO lens of SLM 2D AOD polarizing beamsplitter
red laser
red laser 1st lens of telescope
Fig. 5. Coherent and incoherent template arms of Laptop-POAC system.
The photograph of the final architecture is shown in Fig. 6. Correlation (output)
Template (program) Object (Input) Fig. 6. Photo of the laptop POAC architecture. Red and green lines denote the paths of the laser beams. Input (LCD), programming (AOD or LCD for templates) and output (CCD) devices are also indicated.
Simple measurement results of the POAC system can be seen in the next figure (Fig. 7). A
B C
D
Fig. 7. Computed (C) and experimentally measured correlation (D) of the input image (500x500) (A) and the template (32x32) (B) can be compared. Cross correlation terms show a high degree of similarity. When applying an appropriate threshold, the correlation peaks become separable. The system performance can be estimated from the signal-to-noise-ratio.
4. APPLIED COMPONENTS AND METHODS To implement the new architecture we have to esteem the speed, resolution and performance of the required and available tools and devices. Although, the current laptop size model is constructed using only limited performance devices, we present here our estimations of our architecture’s achievable performance, too. We will outline the possible further improvements, as well. 4.1. Cellular Neural Networks Optical correlators usually require a sensor array, which is able to detect and possibly post-process the measured correlogram in real time. However, currently available sensors are not appropriate for this. The locally connected architecture of the Cellular Neural Networks (CNN)10,11, makes it possible to build VLSI sensor arrays, with relatively high resolution and on chip processing capabilities. As the POAC system integral part is a CNN chip, here, we provide a short introduction. The CNN is a large array of simple analog circuits with local interactions (see Fig. 8). The dynamics of a single cell can be defined as: x& ij (t) = − xij + ∑ Aij ,kl y kl (t) + ∑ Bij ,kl u kl + z ij kl∈S r
kl∈S r
where i,j denotes the cell position in a grid, x, y, u, z are called state, output, input, and bias of cell (i,j), respectively. A and B are called the feedback and input synaptic operators or templates. The nonlinear output y is defined by the following equation:
yij = f ( xij ) =
1 ( xij + 1 + xij − 1 ) . 2
Fig. 8. The interconnection pattern of the Cellular Neural Network
An elementary operation in this architecture is called template operation11, which means the execution of some operation on the whole image. By using templates, different image processing, morphological and wave-metric operations can be implemented. These template operations are convolutions and that is why an optical correlator can be regarded as an optical implementation of the CNN. Several optical and electro-optical implementation of the CNN has been suggested so far12,13,14. Our solution provides good results with feed-forward-only, space-invariant template operations. The CNN Universal Machine architecture is an analog-and-logic, 3D array computer; a stored programmed one, with a CNN array embedded5. It contains, however, additional units: local continuous (analog) and logic memory, local analog and logic units as well as a global analogic programming unit (GAPU), see Fig. 9. Hence, continuous valued spatiotemporal dynamics is embedded in a logic structure, locally and globally. The ACE4K visual microprocessor15,16 , is a physical implementation of the CNN-UM. VLSI CNN implementations have high sensitivity optical sensors on the chip, besides the stored programmable processing capabilities. These features make these VLSI devices optimal sensors for optical correlators.
LCCU L A M
CNN nucleus LAOU
L L M
LLU
GAPU
Global Analogic Programming Unit
APR LPR SCR GACU
global Analog Program Register global Logic Program Register Switch Configuration Register Global Analogic Control Unit
LCCU LAOU LLU LAM LLM
Local Communication and Control Unit Local Analog Output Unit Local Logic Unit Local Analog Memory Local Logic Memory
GAPU
Fig. 9. The architecture of CNN Universal Machines
4.2. Input A liquid crystal micro-display device (LC SLM) provides the input for the optical correlator (Three-Five Systems). The resolution of the so far utilized device is only SVGA (800x600) but is able to update the image at 300 frames per second speed. Later on, if it is required, much higher resolution and faster micro-displays can be applied (1000x1000-2000x2000 resolution and up to 450 Hz frame rate). High-resolution optically addressable spatial light modulators (OASLM) can also bee used. The main advantage of the OASLM would be the simplification of the whole system, as it would not be necessary to use a complicated camera-display system for real-time input. The input image resolution and speed could also be higher, as there are extremely high-resolution OASLM devices. Furthermore, OASLMs can be used for direct feedback in our optical correlator. 4.3. Dynamic holographic recording media Bacteriorhodopsin (BR) is a fast, high-resolution dynamic holographic material. BR is a stable transmembrane protein of an archea, Halobacterium Salinarium (formerly Halobacterium halobium). After absorption of a photon BR goes through a sequence of chemical and physical changes. It is including the retinal molecule’s all-trans – 13-cis transformation, shifts of charges within the protein, movement of protons, that is protonation, deprotonation, and several conformational changes as well. The above characteristic changes are reflected - and manifested primarily - in the changes of the BR’s
absorption spectrum and in the related refractive index changes. Different states have different lifetimes and characteristic absorption spectra. After photon absorption and the proton transfer steps, sooner or later BR returns to its initial conformation. The whole sequence of transitions is called the photocycle of the BR17,18. Above-mentioned characteristic changes in the absorption spectra can be used for analog optical storage. Knowing the BR’s structure and its detailed molecular dynamics, even its molecular engineered modification is possible. Different biologically, chemically and physically modified BR forms are available. For practical application several different types of utilization of BR have been suggested. BR has extremely high spatial resolution: higher than 5000 lines/mm. This resolution even exceeds the commonly applied optical system’s capabilities. Sensitivity of the material is relatively low (0.1-80 mJ/cm2), but it is even better than any other holographic-materials with comparable resolution (e.g. photo-polymers). Speed of BR writing can be remarkably high, but obviously, it depends on which states or transitions of the photocycle are utilized. Usually the BR’s initial state (bR) and another relatively stable intermediate (M) state are used. The bR to M transformation takes about 40 µsec. This speed seems to be satisfactory for our transient holographic recording purposes, but using other states and different modified forms of BR even much higher speed can be achieved. One of the main problems with the utilization of BR as a temporary holographic material is the relatively long thermal recovery time. M to bR transition can take several hundreds of milliseconds and even seconds depending on the different physical and chemical conditions of modified BR samples. Using a properly chosen wavelength (e.g. 658 nm) non-volatile read-out is possible. Absorption at this wavelength is negligible but the refractive index modulation, resulting in high diffraction efficiency, is still satisfactory. An intensive flash of blue light (λ
