EAGLE, a compact vision system with two

FONDAZIONE GIORGIO RONCHI http://ronchi.isti.cnr.it

WALTER ALLASIA, PAOLO PRINETTO, ALBERTO RIVA

EAGLE, a compact vision system with two simultaneous field of view

Estratto da: Atti della Fondazione Giorgio Ronchi Anno LXIX, n. 4 - Luglio-Agosto 2014

Edizioni Tassinari - Viale dei Mille 90 - Firenze 2014

ANNO LXIX

LUGLIO-AGOSTO 2014

ATTI DELLA

FONDAZIONE

GIORGIO RONCHI FONDATA DA VASCO RONCHI ISSN: 0391 2051

Papers presented at

AITA-12 Castello del Valentino, Polythecnic of Turin, Italy 10-13 September 2013

Pubblicazione bimestrale - Prof. LAURA RONCHI ABBOZZO Direttore Responsabile La responsabilità per il contenuto degli articoli è unicamente degli Autori Iscriz. nel Reg. stampa del Trib. di Firenze N. 681 - Decreto del Giudice Delegato in data 2-1-1953 Edizioni Tassinari - Viale dei Mille 90 - Firenze - Agosto 2014

N. 4

Atti della “Fondazione Giorgio Ronchi”

Anno LXIX, 2014 - N. 4

SENSORS

EAGLE, a compact vision system with two simultaneous field of view WALTER ALLASIA (*), PAOLO PRINETTO (**), ALBERTO RIVA (***) SUMMARY. – In this paper we describe EAGLE, an application of the Solid Telescope principle. The idea stems from a particular scenario (search and rescue), where it is essential to have two fields of view with two different information acquired simultaneously in order to reduce the calibration issues. This is the case when the user needs to recognize different features using different wavelengths (e.g. infrared and visible) or fields of view. We report a preliminary study of two solid telescopes with different optical design supported by the same mechanical structure.

1. Introduction Solid Telescope is an optical tool proposed by Schmidt (1) and DeVany (2) in past century. The idea has been recently refined by Riva (3) in 2011. It is essentially a non-traditional use of a lens, with a zonal coating on its surfaces. Its features of stability and self-alignments offer a unique opportunity for those applications where robotization and reduced human intervention are one of the key issues. Even if the idea was conceived initially for astronomical science cases, where the efficiency in photon collection must be stressed at maximum, this tool appears to be promising for all civil engineering cases where photon sources to be monitored have a huge Signal to Noise ratio. In this paper we will describe EAGLE, one of those possible civil engineering application case, such as search and rescue (SAR), car plate recognition, and temperature monitoring.

(*) EURIX, Via Carcano, 26, Torino, I-10153, Italy, [email protected] (**) Politecnico di Torino, C.so Duca d. Abruzzi, 24, Torino, I-10129, Italy, [email protected] (***) INAF Oss. Astrof. di Torino, Via Osservatorio, 20, I-10025, Pino Torinese TO, [email protected]

514

W. Allasia - P. Prinetto - A. Riva

2. EAGLE principle EAGLE concept can be described through two main subsystems, that require adequate engineering phase: optomechanic (OME) and software and electronic (SAE). The principle of the instrument has been inspired by the eagle’s eye. Indeed the eagle has a particular detection system with two simultaneous focal ratios. In one portion of the eye, the bird can see a global scene with a big Field of View (FOV), while in the eye central portion the lens system focuses over a small FOV with a bigger detail, in order to recognize and track the pray. In the EAGLE device we propose to have two solid telescopes with different optical parameters. Indeed, as the eagle, we need to acquire different information at the same time and looking at the same scene (Fig. 1). In the figure the front solid telescope (FST), is optimized for an overall view, since the photons collected are fewer than the rear solid telescope (RST), which is optimized on the desired details we want to detect from the same scene.

Fig. 1 EAGLE principle. The setup is composed of two different solid telescopes with a detector each. The front telescope (FST, left, blue rays) acquires information from the scene that are complementary to the ones coming from the rear telescope (RST, right, green rays).

Each solid telescope focuses its light over a dedicated detector connected to specific processing FPGA-based boards. The proposed solution make use of standard technologies in order to improve the reliability of calibration processes (extrinsic and intrinsic): the couple of optical systems is embedded in a same mechanical envelope, looking at the same scene onto the same optical axis.

EAGLE, a compact vision system …

515

This compact solution represents a good candidate for being mounted on unmanned aerial vehicle (UAV) for search and rescue (SAR). In this context the compactness, flexibility and lightness are key issues. On the one hand, each telescope is vibration free, because the adoption of the configuration proposed in (3) allows a single lens layout, even for deeper FOV usually requiring multiple elements design. On the other hand, the mechanical coupling lead to compact systems, providing two different data acquisition with high level alignment. Specially in the SAR with UAV, the recent development is adopting two wavelengths camera acquisition, IR and visible: the rationale is the separation between the human thermal contribution from the background. Currently the available systems have separated devices, needing specific extrinsic calibrations and mechanical set up. 3. OME: Opto Mechanic stage of EAGLE EAGLE represents a big innovation in the field of multiple cameras supported by the same backbone. Indeed it exploits one of the main features of the Solid Telescope concept: its alignment stability. Since each solid telescope can be summarized as a “complex” telescope built with a single lens, it can tolerate relatively extreme environmental conditions, such as unpredicted vibration spectra that can affect the stability of each detecting unit. It is the case of an UAV dedicated to the SAR, where the flying unit must act in unpredictable conditions with different weather trends (i.e. wind magnitude and direction) or different engine of the UAV itself (quadricopter, jet propulsion, etc…). In such scenario, the flexibility of the Solid Telescope allows to imagine various tailoring of the payload, depending on the specific requirements. One of the possibilities is the coupling of a front telescope (FST) designed for the infrared contribution, with a rear telescope (RST) dedicated to the visible imaging. With such device, the RST can be used for the detection of the global scene and the tracking of the UAV, while the FST can be dedicated to the detection of the thermal source given by the blackbody contribution of the human body (the rescue target). Detecting thermal sources (animals and persons) is a key feature for the search and rescue, since the infrared contribution gains a bigger Signal to Noise ratio. This is even more crucial in situation like the night-time where the optical camera cannot provide the desired quality of the information. An alternative configuration of the two solid telescopes can be the coupling of two Field of View (FOV) values on the same optical axis. It is the scenario where the FST has a big FOV and it is dedicated to the recognition of bigger landscape visibility (e.g., rivers, mountains, forests, etc…), while the RST uses the greater collection area in order to focus over a smallest FOV, detailing particular

516


features (e.g. human bodies, small artificial objects, signs…). This configuration allows to have different values of NIIRS (National imagery interpretability rating scales) (4), the numeric scale that codifies the images interpretation. Such configuration could be complementary with the previous. In principle, there are no show stoppers for the combination of two FOV values and two bandwidth at the same time. The combination of the two concepts can stress instrumental features like the signal to noise of the thermal FST. All the aforementioned concepts are based on the design and production of the two telescopes with different optical properties. In the perspective of mass production of the EAGLE device, one can stress the optical production cost. For the reduction of the optical design complexity (the most delicate in any case), there is the possibility of having both the FST and the RST made up with the same parameters (FOV, wavelengths, etc…). The solutions stands in the different pointing axis of the two telescopes. If the RST is mounted on a Gimbal device (5), it is possible to keep fixed the FST scene and to use the RST to scan the surrounding scene (with a little overlapping for calibration). The benefits introduced by the replication of the two telescopes are balanced by the complexity given by the rotational movement of the Gimbal mounting that moves the RST. It introduces potential uncertainties on the vibrations given by a motor, and tolerances due to the movement and aging of the rotational parts. The solution for this potential problem stands in an increased complexity of the software calibration, that take into account the potential additional mismatch between the two scenes. 4. SAE: Software and electronic stage of EAGLE The SAR scenario requires intensive image processing. As previously introduced, a couple of wavelengths are currently adopted, the IR and visual, in order to be able to detect human beings in wilderness, as described in Refs. 6 to 9. Many of these works are making use of several common methodologies for image processing, that our approach is allowing and improving. With the optical infrastructure we are proposing with EAGLE it will be possible to adopt several image processing techniques, such as the pattern detection based on Haar-like features (10,11), used in the SAR context in (6). They are nowadays available as open software libraries for computer vision, such as (12). Having the same optical axes, the couple of cameras require less effort in spatial calibration such as extrinsic and intrinsic, usually representing a big issue with cameras bundled together on UAV for SAR (13). Consolidated techniques for extrinsic calibration making use of reference grids acquired simultaneously by both cameras such as in (7) can be improved adding data fusion and integration of different information gathered. EAGLE allows also the low level features extraction (as in (14)), enabling the search of specific patterns and textures in the processed frames. In (15) and


517

(16) the Speed-up RobustFeatures (SURF) have been used for pattern matching of human beings. Equipped with the EAGLE OEM and SAE modules it will be possible to improve the approach by introducing a cascade of different classifiers, based on low level descriptors of the numerical image. Indeed it is needed because the above approach is limited to few patterns for training the match, onto which the scale invariant key points have been evaluated. They are deeply depending on several external factors not always predictable, such as sun light and UAV attitude and speed, that are subject to rapid variation during a SAR mission. The SAE module can hence be made of a degenerate decision tree based on different classifiers making use of several different low level features. The introduction of a tree allows to keep high recall for each classifier and even if the respective precision is low, the overall chain will improve it after every classifier step. This method has been adopted in similar works (17) for traffic sign detection and recognition with a single visual camera mounted on a patrolling vehicle. In order to apply a filter to the several classifier and come up with an average result, it will be possible to adopt numerical methods based on conditional probabilities associated with every element in the image processing flow, hence following the Bayes’ rules. SAE module can improve the images to be processed applying subtraction techniques, for example making use of a silhouette approach as in (18) for detecting human beings in movement. Even if the silhouette (18) cannot be applied to a SAR context where human beings must be detected especially when they are not moving, some solutions can be evaluated in order to automatically subtract the background, exploiting the UAV motion and pattern tracking techniques. A particular methodology for pattern tracking and matching suitable for the SAR context can be based on similarity search (19), a well-established technique adopted in information retrieval. It can be embedded in the SAE module in order to improve the evaluation and comprehension of image background and to enable automatic recognition of specific zones and environments (20). The above mentioned image processing algorithms are heavily computational intensive and they requires hardware acceleration for allowing real time applications. Hence the SAE module can be equipped with ad-hoc designed circuits, implemented on reprogrammable hardware (FPGA), tailored to the UAV context needs. Many feature extraction algorithms (21,22) and pattern detection (23,24) have been published and implemented already by several researchers on reprogrammable devices such as FPGA, mainly in contexts of assisted and automatic navigation and guide control. Unfortunately most of current implementations are not taking into account the requirements on reliability of the digital system, but they are just implementing the algorithms that are needing acceleration. In the EAGLE SAE module we are able to plan for fault-detection and faulttolerance FPGA implementation, improving the overall robustness of the bundle

518


of complete system (OEM and SAE). A similar approach already developed in (25) has demonstrated the efficient adoption of FPGA software components in image pre-processing, such as SAFE (26), in order to limit the computation workload carried out on a SAE module and moving the low level pixel elaboration to FPGA (e.g. edge detection, etc.). The EAGLE capability to have two FOV simultaneously enables a new video descriptor for the scene, the interested geographic area that the UAV is patrolling. As shown in Fig, 2, the large FOV of FST can provide the HLF High Level Features whilst the small FOV of RST can provide the LLF Low Level Features. Both features are characterizing the picture seen by the UAV, enabling a fast indexing due to the HLF that can be used as root index and LLF as leaf.

Fig. 2 SAE candidate modules overview: FST: FrontTelesSope RST: RearTeleScope; FG: FrameGrabber FE: FeatureExtraction; HLF: HighLevelFeatures LLF: Low LevelFeatures; BoW: Bag of Words.

The interested geographic area can be acquired separately and simultaneously with the two telescopes. The Software and Electronic Module (SAE) grabs the fames, extract the features, summarize high and low level features in order to produce a Bag of Words (BoW) that can be used for indexing the interested scene. A similar approach already developed in (25) has demonstrated the efficient adoption of FPGA software components in image pre-processing, such as SAFE (26), in order to limit the computation workload carried out on a SAE module and moving the low level pixel elaboration to FPGA (e.g. edge detection, etc.). The EAGLE capability to have two FOV simultaneously enables a new video descriptor for the scene, the interested geographic area that the UAV is patrolling. As shown in Fig, 2, the large FOV of FST can provide the HLF High Level Features whilst the small FOV of RST can provide the LLF Low Level Features.


519

Both features are characterizing the picture seen by the UAV, enabling a fast indexing due to the HLF that can be used as root index and LLF as leaf. Both features can then be merged together in a BoW (Bag of Word) that can be considered as the characterization of the entire scene: it is no more a specific single frame but the combination of two frames captured with different FOVs. A similar approach with high and low descriptors of the image, has recently been investigated and adopted by the MPEG (27) CDVS (28), where features are mainly based on SIFT (29) They are usually a huge amount of data for each image and in order to allow a fast retrieval the CDVS group has extracted features from the image itself (for the LLF) and from a somehow blurred and down sampled image (for the HLF). With EAGLE, it is not needed to rescale to a lower resolution the image for extracting the HLF because the two FOV are providing already the two images. Moreover, being “different” images, the descriptors extracted are more suitable for a better characterization of the scene compared to other techniques. 5. Future work Concerning the OEM, at the time of writing the authors started to design the EAGLE lens and to contact the factories for placing an order to produce a first prototype. Concerning the SAE, it is planned to evaluate the BoW approach and check it against the ground truth provided by the sample sets available in the CDVS research group.As soon as the lens prototype will be available, a specific test plan will be organized for checking the overall EAGLE chain and get fundamental data for evaluating the proposed system. The cited SAR scenario as well as new potential ones, will be investigated within the research projects currently involving the authors. Acknowledgments The activity is partially supported by the Italian Space Agency funds under contract to INAF I/058/10/0 (Gaia Mission - The Italian Participation to DPAC).

REFERENCES (1) W.H. Christie, D.O. Hendrix, The Schmidt principle in optical design, Public. of the Astronomical Soc. of the Pacific, 51,ì (302), 223 (1939). (2) A.S. De Vany, Optical design for two telescopes, part B: an f = 5 solid telescope, Appl. Opt. IP, 2 (2), 201 (1963). (3) A. Riva, New version of solid telescope, Appl. Opt., 50 (21), 4148-4160 (2011). (4) J.M. Irvine, National imagery interpretability rating scales (NIIRS): overview and methodology, Proc. SPIE, 3128, 93-103 (1997).

520


(5) F.C. Moon, The Machines of Leonardo da Vinci and Franz Reuleaux: Kinematics of Machines from the Renaissance to the 20th Century, (Springer, 2007), p.314. (6) P. Rudol, P. Doherty, Human Body Detection and Geolocalization for UAV Search and Rescue Missions Using Color and Thermal Imagery‚ Proc.IEEE Aerospace Conference, (Big Sky, Montana, 2008), pp. 1-8. (7) N. Rasmussen, B. Morse, M. Goodrich, D. Eggett, Fused Visible and Infrared Video for Use in Wilderness Search and Rescue, Proc.of WACV2009, IEEE Workshop on Applications of Computer Vision (WACV), pp. 1-8 (2009). (8) J.A. Adams, J.L. Cooper, M.A. Goodrich, C. Humphrey, M. Quigley, B.G. Buss, B.S. Morse, Camera-Equipped Mini UAVs for Wilderness Search Support: Task Analysis and Lessons from Field Trials, Byuhcmi Technical Report (2007). (9) S. Waharte, N. Trigoni, Supporting Search and Rescue Operations with UAV, Emerging Security Technologies (EST), 2010 Intern. Conf., 2010, pp.142-147. (10) P. Viola, M. Jones, Rapid object detection using a boosted cascade of simple features, Computer Vision and Pattern Recog., 1, 511-518 (2001). (11) R. Lienhart, J. Maydt, An extended set of haarlike features for rapid object detection, Proc. Intern. Conf. on Image Processing, 1, 900-903 (2002). (12) G. Bradski, A. Kaehler, Learning OpenCV: Computer Vision with the OpenCV Library, (O’Reilly-Media, 2008), 1st ed. (13) J.-Y. Bouguet, P. Perona, Camera calibration from points and lines in dual-space geometry, California Inst. of Techn., Tech. Rep., (1997). (14) D. Rodriguez, Robust Harris-SURF features for robotic vision based navigation, Intelligent Transportation Systems (ITSC), 13rd Intern. IEEE Conf. (2010), pp. 1160-1165. (15) A. Symington, S. Waharte, S. Julier, N. Trigoni, Probabilistic target detection by camera-equipped UAVs, Robotics and Automation (ICRA), 2010 IEEE Intern. Conf. on, 40764081 (2010). (16) I. Mondragon, P. Campoy, J. Correa, L. Mejias, Visual model feature tracking for uav control, Proc. IEEE Intern. Symp. on Intelligent Signal Processing, pp. 1-6 (2007). (17) W. Allasia, C. Culeddu, M. Ferraro, F. Gallo, M. Vigilante, Automatic video processing for traffic sign recognition, Proc.of the 5th IASTED European Conf. on Internet and Multimedia. Systems and Applications (EuroIMSA 2009), (Cambridge, UK, Acta Press, 2009). (18) J. Han, B. Bhanu, Detecting Moving Humans Using Color and Infrared Vide, Proc. of MFI2003‚ IEEE Intern. Conf. on Multisensor Fusion and Integration for Intelligent Systems, (Tokyo, Japan, 2003), pp. 228-233. (19) P. Zezula, G. Amato, V. Dohnal, M. Batko, Similarity Search. The Metric Space Approach, in: Advances in Database Systems. (233 Spring Street, New York, USA, 2006), Vol. 32. (20) W. Allasia, F. Falchi, F. Gallo, MounaKacimi, A. Kaplan, J. Mamou, Y. Mass, N. Orio, Audio-visual content analysis in P2P: the SAPIR approach, EBU P/SCAIE, First AIEMPro 2008 Workshop in conjunction with DEXA (2008). (21) J. Svab, T. Krajnik, J. Faigl, L. Preucil, FPGA based Speeded Up Robust Features, Technologies for Practical Robot Applications (TePRA IEEE Intern. Conf., 2009), pp.35-41. (22) R. Harinarayan, R. Pannerselvam, M.M. Ali, D.K. Tripathi, Feature extraction of Digital Aerial Images by FPGA based implementation of edge detection algorithms, Proc. 2011 Intern. Conf. on Emerging Trends in Electrical and Computer Technology (ICETECT), pp.631-635. (23) Y. Tanida, T. Maruyama, Real-time detection of rotated patterns using FPGA, FieldProgrammable Technology, FPT Intern. Conf., pp.316-319, (2009). (24) T. Miyoshi, T. Shibata, A hardware-friendly object detection algorithm based on variable-block-size directional-edge histograms, World Automation Congress (WAC), pp.1-6 (2010). (25) W. Allasia, E. Licata, F. Rosso, F. Gallo, D. Rolfo, P. Prinetto, P. Trotta, A. Favetto, M. Paleari, P. Ariano, Proc. ISBN ECSI Media 979-10-92279-02-3 IEEE Xplore 97910-92279-01-6 (2013). (26) S. Di Carlo, G. Gambardella, P. Lanza, P. Prinetto, D. Rolfo, P. Trotta, SAFE: a Self Adaptive Frame Enhancer FPGA-based IP-core for real-time space applications, IEEE 7th


521

Intern. Design and Test Symp. (IDT), Doha, QR 15-17, pp. 6 (2012). (27) MPEG, Moving Picture Expert Group, the official name of the International Organization for Standardization, JTC1, Sub Committee 29, Working Group 11. (28) CDVS, Compact Descriptor for Visual Search (2014), G. Francini, S. Lepsy, M. Balestri, Selection of Local Features for Visual Search, Image Commun., DOI 10.1016/j.image.2012.11.002 Elsevier (2013). (29) SIFT, Scale Invariant Feature Transform, D.G. Lowe, Distinctive image features from scale-invariant keypoints, Intern. J. Computer Vision, DOI:10.1023/B:VISI.0000029664.99615.94 (2004).

Atti della “Fondazione Giorgio Ronchi”

Anno LXIX, 2014 - N. 4

INDEX

Advanced technologies

U. GALIETTI, D. PALUMBO, R. DE FINIS, F. ANCONA, Fatigue damage evaluation with new thermal methods M.N. LITVINOVA, V. KRISHTOP, N. KIREEVA, Y. PONOMARCHUK, Conversion of incoherent IR-radiation into visible radiation in nonlinear crystals A. LIBBRA, A. MUSCIO, Reverse use of an infrared emissometer for measurement of thermal conductivity

Bio applications

I. BENKO, Histographic method as a tool of thermal image processing V. REDAELLI, G. PIZZAMIGLIO, P. SELLERI, N. DI GIROLAMO, P.A. MARTINO, F. LUZI, Environmental assessment of terrarium in the main species of reptiles through the thermography technique. Preliminary results

Pag. 387 » 405 » 413 » 423 » 433

Buildings and infrastructures

A. BORTOLIN, G. FERRARINI, P. BISON, G. CADELANO, Thermal performance measurement of the building envelope by infrared thermography E.C. ROBISON, C. ESATRADA, The application of Infrared Thermography to Condition Assessments of Guastavino Vaulting G. FERRARINI, P. BISON, A. BORTOLIN, G. CADELANO, M. GIROTTO, U. NAZZALI, F. PERON, M. VOLINIA, Energy survey of a historical building by infrared thermography

» 441 » 447 » 453

Cultural heritage

A. TAVUKCUOGLU, Quantative Infrared Thermography and Ultrasonic Testing for In-Situ Assessment of Nemrut Dag Stone Statues

» 459

Image and data analysis

W. SWIDERSKI, D. NESTERUK, V. VAVILOV, D. DERUSOVA, Data Fusion in » 473 IR Thermographic Detection of Landmines and NDT of Composites A.A.S. MOHAMMED, S.M. HAMEED, A Development Formula to Evaluate the » 479 Band Width of an Image for a digital Camera

Industrial applications

A. KHALIFA, M. MARCHETTI, L. IBOS, V. FEUILLET, J. DUMOULIN, M. BUÈS, L. BOUILLOUD, Monitoring of traffic incidence on pavement surface temperature

» 487

Nondestructive evaluation

L. VEGA, P. VENEGAS, I. SAEZ DE OCARIZ, M. VANDEWAL, A. BROOK, E. CRISTOFANI, P. MOUNAIX, J.P. GUILLET, S. WOHNSIEDLER, C. MATHEIS, F. OSPALD, Validation of a Novel NDT Technique with Infrared Thermography P. VENEGAS, L. VEGA, J. GUEREDIAGA, I. SAEZ DE OCARIZ, Instrumentation Technology based on Infrared Thermography for the Control of Aeronautical Composite Components Subjected to Certification Load Tests A. ELHASSNAOUI, S. BELATTAR, S. SAHNOUN, Determination of defects thermal effusivities by pulsed thermography

Sensors

W. ALLASIA, P. PRINETTO, A. RIVA, EAGLE, a compact vision system with two simultaneous field of view

» 493 » 499 » 505

» 513