Performance of obstacle detection and collision warning system for civil helicopters Naruto Yonemoto*a, Kazuo Yamamotoa, Kimio Yamadaa, Hidemi Yasuib, Naohiro Tanakab, Claire Migliaccioc, Jean-Yves Dauvignacc, and Christian Pichotc a
Electronic Navigation Research Institute (ENRI), 7-42-23, Jindaiji-Higashi, Chofu, Tokyo, 182-0012 JAPAN ; b IHI Aerospace Co. Ltd. , 900 Fujiki, Tomioka, Gunma, 370-2398, JAPAN ; c Electronics, Antennas and Telecommunications Laboratory (LEAT), University of Nice-Sophia Antipolis-CNRS, FRANCE ABSTRACT Some helicopters strike the power lines under the good weather conditions. Helicopter pilots sometimes have some difficulties to find such long and thin obstacles. We are developing an obstacle detection and collision warning system for civil helicopters in order to solve such problems. A color camera, an Infrared (IR) camera and a Millimeter Wave (MMW) radar are employed as sensors. This paper describes the results of different flight tests that show good enhancement of radar detection over 800m range for power lines. Additionally, we exhibit the processed fusion images that can assist the pilots in order to recognize the danger of the power lines. Keywords: obstacle detection and collision warning, millimeter wave radar, image fusion
1. INTRODUCTION According to the aircraft accident reports in Japan, helicopters and small aircraft often strike against obstacles while they are flying at low altitude in visual flight rules. Many collisions are caused by long, thin artificial objects such as power lines. It is because pilots are often very difficult to find these objects even for good weather flight conditions. A lot of work has been done to avoid such collisions by developing obstacle sensors such as Hogg1, Sadovonik2 and Sarabandi3. However, their sensors are mainly designed for military applications to expand the operational conditions. We are developing an obstacle detection and warning system for civil helicopters. A color camera, an IR camera and an MMW radar are applied as the sensors of this system. The primary detection target is a long and thin object such as power lines or ropeway cables. Former results showed that the IR camera with fine resolution would be useful to detect such thin obstacles4,5. We also showed the validity of the MMW radar sensor to measure the target distance or to compensate it under the poor visibility such as in the morning or evening6-8. In addition to these studies, we are conducting a collaborative research to improve the MMW radar performance such as , antenna return loss reduction9-11 and antenna size reduction12 in parallel. We showed the validity of the real-time warning method using a color, IR and radar data fusion image13,14. Major objective of this paper is to demonstrate the validity of our obstacle detection and warning system that we have developed for helicopter flight test. First, the concept, configuration and feature are described about our on-board obstacle detection and warning system. Flight tests were conducted three times, September 2004, February 2005 and March 2006 to collect data in different weather conditions. The procedure of the flight test is presented. Then, measured results are presented to evaluate the basic performance of the system as coverage, data rate and display.
The antenna has been replaced with a new reflector one for the MMW radar of our obstacle detection and *
[email protected]; phone +81-422-41-3174; fax +81-422-41-3176; http://www.enri.go.jp/
warning system to reduce the radar size. Detection performance of both two antennas has been evaluated by the measured results. Finally, typical examples of the processed display in which power lines are enhanced are presented.
2. THE OBSTACLE DETECTION AND WARNING SYSTEM FOR HELICOPTERS The obstacle detection and warning system under development is for small civil aircraft in visual flight rules. It must be low in cost and compact and light in size. According to the requirements from helicopter operators, the maximum detection range is about 800 m and the surveillance coverage should be as wide as possible. Fig. 1 shows an overview of the total system. Color
CCD Camera (Color Image)
Infrared Camera
Imag e
Proce ssed
Imag
e
Display
Infra r Imag ed e
PC (Data Dista n Proccessor) Infor ce mati on
MMW Radar
Scan n Infor ing mati on
PC (Controller) Warn i & Dis ng play
Contr ol Scan ning Cont rol
Scanning Equipment (Gimbal Ring)
Figure 1:
Block diagram of the obstacle detection and warning system for helicopters
The system consists of a color camera, which is a small CCD camera, an IR camera, a MMW radar, a scanning equipment, a controller, a data processor and a display. Yamamoto and Yamada demonstrated that a high resolution IR sensor is suitable to detect a long and thin object as power lines4,5. Then, we introduced the IR camera that can detect 3-5µm IR with 500 x 800 IR sensing elements. The color camera is also used to collect reference images. The MMW radar is highly expected to detect power lines for military applications. However, there was very little work about the development of the MMW radar for civil helicopters. So, we developed the MMW radar by ourselves. We chose 94GHz as the radar frequency because as the frequency becomes higher, the resolution of the radar tends to be high and the radar size tends to be small. FMCW (Frequency Modulated Continuous Wave) technique was employed as the radar modulation because this technique is more efficient than other type radars. Then we can expect to extend the range by comparatively low power generator. Details of the radar were described in some papers6-8. And the MMW radar is also expected to reduce the size of the sensor in comparison of the other frequency range. We are also studying to develop the new antennas or feeding systems9-12. A PC is used as data processor. It gathers color and IR images and radar data, and fuses images and radar data to provide enhanced obstacle images on the display. It predicts the threat of obstacles and gives the alarm to the pilot. Another PC controls the MMW radar and its scanning equipment. However, we are considering the algorithm to fulfill the above all tasks only by one PC to downsize the system. Fig. 2 shows a basic concept of the image fusion from these sensors. The CCD camera and the IR camera provide the moving color image and IR image respectively. The radar outputs intermediate frequency (IF) signal by mixing transmission and reception signals. The PC collects the moving images from CCD and IR cameras and the IF signals
Figure 2:
Concept of image fusion technique
of the radar. It converts the IF signals into spectrum and tries to search the nearest target among many peaks in the spectrum. An image processing algorithm has been developed to enhance target to background contrast and to derive long and thin targets that are running horizontally in the IR image. This IR image is combined with the color image (reference image) to create moving fused images. Information about radar to target distance is merged into the fused images as the color variation or as the color density. This technique enables to simplify the total system because it does not need sophisticated scanning antenna and time consuming image synthesis algorithm in the radar. This technique is designed not only to detect power lines and display them in real-time but also to record total data to conduct post processing for radar range and accuracy estimation. Detailed description of this technique is described in other papers5,13,14.
Radar CCD Camera
IR Camera
(a) Overview of the helicopter and on-board system
(b) Sensors on gimbal ring
(c) Control and Recording system
Figure 3: Onboard system for flight test Fig. 3 (a) shows an overview of a helicopter and the on-board system. The helicopter used for test is Kawasaki BK-117. Fig. 3 (b) shows the sensors on the gimbal ring to manipulate the direction of the axis of the sensor or to eliminate the vibration from the helicopter. The sensors always points at the right side from the inside of the helicopter. Fig. 3 (c) shows the control and recording system. The system just needs a few basic elements but the onboard system consists of many components to record the all of the data for the post processing. Weight of the total system is 210 kg. And the total power consumption is 1.5 kVA.
3. FLIGHT TEST AND RESULTS The flight tests were held in September of 2004, February of 2005 and March of 2006 to collect the data with different environmental conditions. The scenes of the test site are shown in Fig. 4. Fig. 4 (a) shows the typical mountainous area in winter. The mountains were partly covered with the snow. Such condition causes some difficulties to find or recognize the power lines. All of them were conducted in the mountainous area of Gifu Prefecture (close to Nagoya, central part of Japan) with the cooperation of Kawasaki Heavy Industries.
(a) In the snowy mountain (Feb. 2005) Figure 4: Scenes of test site
(b) In autumn (Sep. 2004)
Every measurement was conducted by flying along the power lines. The GPS navigation keeps the separation between the helicopter and the power lines such as 200m, 400m, 600m, and 800m. Fig. 5 (a) and (b) represent some examples of the display in real-time detection.
(a) Before the target detection (b) After the target detection Figure 5: Displays of the system
Both figures show a typical view of the display. The upper left figure shows fusion images with all calculated results. The lower left figure shows radar IF spectrum displayed in time domain, frequency domain or as spectrogram. The upper right figure is the control window. And the lower right figure represents the distance to target after the nearest target detection. Fig. 5 (a) shows a typical the image in the snowy mountain. The color images didn’t show the power lines because it
was too difficult to see them under such environmental conditions although the visibility is satisfactory for flight. This figure also shows that the radar failed to detect target. Vibration and fluctuation of the helicopter often cause target missing because the radar beam width is very sharp (about 1.5 degree). And, after investigating the radar output, we found that the magnitude of radar reflection from a power line is dependent on the incident angle to the target. This can explain the fact that the Radar Cross Section (RCS) of power lines varies sharply with observation angles. [K. Sarabandi, “Millimeter-Wave Radar Phenomenology of Power Lines and a Polarimetric Detection Algorithm”, IEEE Trans. AP Vol.47, No.12 December 1999] Thus, we have to say that the power lines can not always detect even if the radar beam hits the target. Scanning function must then be added to our present radar. Fig. 5 (b) shows the fusion image when the radar has succeeded in detecting targets in the same environment. Our system has successfully enhanced the image of power lines in green and the shape of the obstacle has been distinguished. In our data processing procedure, IR image is employed to derive target and the target is superposed on the color image in different transparent factor. The transparent factor is set to be the function of radar to target distance. By this procedure, the system achieved a refresh rate of over 8 frames per second without any degradation of the total system performance. Fig.6 (a) and (b) show other examples in which the system could detect targets in different radar to target distances.
(a) at 200m separation (b) at 600m separation Figure 6: Display with the pseudo-colored images
The target has been displayed in pseudo color. The figure shows that the difference of distances between Fig.6 (a) and (b) can be expressed in the density of each pseudo color. However, we cannot say that Figs. (a) and (b) have sufficient contrast difference to recognize the distance in real-time warning. Then, we tried to improve the method to represent radar to target distance. Fig. 7 (a) and (b) show solid-colored images at the same separation to the target as Fig.6. Different colors are assigned to overlaid images to express the distance to the target. The system uses the color table corresponding to the distance from red to blue (red – yellow – green - cyan - blue). This technique looks better to recognize the range to the target instinctively, and it is much easier than the pseudo-colored images, especially during the flight test in real-time.
(a) at 200m separation (b) at 600m separation Figure 7: Displays with the solid-colored images
4. DISCUSSION The entire radar signal was stored as frequency domain data. We could not only trace the radar data in real time but also did the post processing. The spectrograms of the IF signals of the radar are shown in Fig.8 (a) and (b).
0MHz
25MHz
0MHz
25MHz
(a) Result at 600m separation (b) Result at 800m separation Figure 8: Spectrograms of the IF signal of radar output
Vertical axis of the figure indicates the time of the measurement, the horizontal axis indicates the frequency, and the brightness or colors of the pixel indicate the intensity of the signal. The sampling interval of each scan was almost 0.1 ms depending on the processing time of the target enhancement. The distance to the target corresponds to the 37.5m per 1MHz, the maximum range from the spectrogram represents 937.5m approximately. We can see two threads of bright doted line in the certain frequency range corresponding to the distance to the target. The reflection from the power lines is not stable because of the vibration and maneuver of the helicopter. And the width of the signal from power lines got wider relating the range to the target because of the non-linearity of the transmitting wave. The two wide bright lines are unified in case of 800m separation. Two bright straight lines in these figures at 5MHz and 20MHz are harmonic noises of the AD converter. They are also including some brighter spots corresponding to the reflection from the pillar of power towers. The received signal from the power lines is about 20 dB higher than the noise level, so that the radar performs the 800m
range for the power line detection. A new printed antenna was developed by LEAT. The antenna was and installed and tested on the helicopter. Fig. 9 (a) shows the overview of the antenna. This figure compares the conventional Cassegrain antenna with the new one. Fig.9 (b) shows the spectrogram of the radar signals obtained by this new antenna.
0MHz
25MHz
(a) Comparizon of the new printed antenna (b) result with new antenna at 600m separation with Cassegrain antenna Figure 9: New antenna and a spectrogram of measurement result
The printed antenna is quite small and light in comparison with the Cassegrain antenna. Its diameter is a half of the Cassegrain antenna. Its measured gain is about 35 dBi and 8 dB lower than that of Cassegrain antenna. Theoretically, it decreases 16 dB of the received signal power. We confirmed the difference between them on the ground before the flight test. We can also trace the two bright lines in this spectrogram and find strong bright spot in these lines in the Fig. 9 (b). From this figure, we can say that the radar with this new antenna could detect power lines 600m apart. However, we could not detect power lines 800m apart although the pillar of the power tower was detected. It is due to the difference of the gain between both antennas. A 40 dBi or higher gain antenna will be expected to extend the range of the radar.
5. CONCLUSION This paper describes the experimental obstacle detection and warning system for civil helicopters and the results of flight test by this system. First, we mentioned the outline of the onboard system and the procedure of the flight test. A new reflector antenna has been developed to reduce the radar size. The performance of the antenna was also presented. The results showed that the range of our system to detect power lines exceeds 800m. Our image processing procedure as different color assignment to target distance exhibited satisfactory target enhancement. Image refresh rate of this system was over 8 frames per second. For the new reflector antenna, the detection range was 600 m for the power lines and 800m for the pillar because of the difference of the gain of the antenna. The results showed the ability of the minimization of the radar.
ACKNOWLEDGEMENT This project was partly supported by the Program for Promoting Fundamental Transport Technology Research of the Japan Railway Construction, Transport and Technology Agency（JRTT）.
REFERENCES 1. G M Hogg, “CLARA – A Coherent CO2 Multi-Mode Laser Radar,” Proceedings of IEE Radar 97,No.449, pp.678-682, 1997. 2. L. Sadovonik, V. Manasson, et al., “Helicopter obstacle detection radar system,” Proceedings of SPIE, Vol.4023, pp.2-12, 2000. 3. K. Sarabandi, M. Park, “Millimeter-Wave Radar Phenomenology of Power Lines and a Polarimetric Detection Algorithm,” IEEE Trans. on Antennas & Propag. Vol. 47, No.12, 1999, pp.1807-1813. 4. K. Yamamoto, and K. Yamada, “Image processing and fusion to detect navigation obstacles,” SPIE Proceedings Vol. 3374, pp.337-346, 1998 5. K. Yamamoto, and K. Yamada, “Obstacle detection for helicopter flights by infrared images,” Proceedings of SPIE, Vol. 4363, 2001 6. K. Yamamoto, K. Yamada, et al., “Millimeter wave radar for the obstacle detection and warning system for helicopters,” Radar 2002, No. 490 , IEE Conference Publication, pp. 94-98, 2002. 7. K. Yamamoto, K. Yamada, N. Yonemoto, et al., “94GHz FMCW radar for obstacle detection,” Proceedings of IRS 2003, No. 490 , pp. 481-486, 2003. 8. K. Yamamoto, N. Yonemoto, et al., “The Performance of Airborne 94 GHz Radar for Obstacle Detection,” Proceedings of IRS 2005, pp.147-152, 2005. 9. C. Migliaccio, B.D. Nguyen et al., “Vivaldi Antenna for Obstacle Detection and Warning System at 94GHz,” Journees Internationales de NICE sur Les Antennes (JINA 2002) , Volume 1, No. 2.22, pp.279-282, 2002. 10. B.D. Nguyen, C. Migliaccio et al, “Comparison des performances d’une antenne Vivaldi et d’un reséau de fentes à 94 GHz,” 3émes Journées Nationales Microondes, 1B2, Lille, France, 2003. 11. B.D. Nguyen, C. Migliaccio, et al, “Compact primary source for W-band reflector antenna,” Electronic Letters, 10th November 2005, Vol. 41, No.23, pp. 1262-1264 2005 12. B.D. Nguyen, C. Migliaccio, et al., “Printed reflector antenna for MMW detection radar,” Journees Internationales de NICE sur Les Antennes (JINA 2004) , No. 3.30, 2004. 13. N. Yonemoto K. Yamamoto, et al., “Obstacle Detection and Warning for Helicopter Flight by Infrared and Millimeter Wave,” Proceedings of SPIE, Vol. 5081, pp.31-38, 2003. 14. N. Yonemoto K. Yamamoto, et al., “A new color, IR, and radar data fusion for obstacle detection and collision warning,” Proceedings of SPIE, Vol. 5424, pp.31-38, 2004.
