video switching and sensor fusion for multi-camera ... - UC CEAS

VIDEO SWITCHING AND SENSOR FUSION FOR MULTI-CAMERA SENSING SYSTEMS Mayank Saxena, Vishnuvardhanaraj Selvaraj, Rahul Dhareshwar and Ernest L. Hall Center for Robotics Research, ML 72 University of Cincinnati Cincinnati, OH 45221 ABSTRACT With the low cost of solid-state camera systems, it is now possible to include many cameras on a mobile robot or other machine. However, video processing is still relatively expensive. Therefore it is often desirable to share several cameras with a single processor. The purpose of this paper is to describe the design of a video switching system that permits eight cameras to be multiplexed with a single chip. Multiples of eight could also easily be accomplished. The heart of the system is a Maxim video switch. The user simply selects using a three-bit control signal, which camera signal is selected. The output of the video switch is then the desired camera image. One application of this video switch is a four camera input system to a mobile robot being constructed at the University of Cincinnati. Other applications include surveillance and other mobile systems. The decision as to which camera to observe can be made automatically from a computer providing a great versatility. For example, supplemental motion detectors could be used to activate the camera selection for a surveillance system. Higher-level logic has been used on our mobile robot application. Still higher-level logic could be used to fuse the video information in various ways before processing. The significance of this device is that it provides a wealth of video information to be used at the discretion of either a human viewer or automatic system. KEYWORDS: computer vision, video switching, multiplexing, mobile robot.

1.INTRODUCTION The reliability of an autonomous vehicle5 navigating in an unstructured environment depends on a great extent to the information that the intelligent system can collect and comprehend about its environment. Sensors allow a system to study the present state of the world and to adjust itself to the changing environment suitably. The use of multiple sensors in an intelligent system would tend to increase the capability of the system. There are different approaches to integrating the information from multiple sensors into the operation of the system. A simple approach would consist of providing a single input line to the system. However this approach would be effective in a situation where the multiple sensors study and observe different aspects of the same environment. It would be ideal to build a certain degree of overlap between the sensors, where one sensor would influence the operation of another sensor depending on the current state of the environment. This would mean that the combined information provided by the sensors would be greater than the sum of the information provided by the individual sensors. This would provide the system with information of high quality and precision which otherwise could not be directly sensed by an individual sensor operating individually. Integrating multiple sensors into the operation of the system is a major factor in the overall design of the system. The specific capabilities of each individual system and the information they provide would

Correspondence: Email: [email protected], Telephone: (513)556-6130

influence the design of the system. The varying requirements make it increasingly difficult to define any general-purpose method for multi sensor integration. Previous attempts by researchers have led to the evolution of many different frameworks and control structures for multi sensor integration for a basic general-purpose system. Navigation of an autonomous system1 requires real time, dynamic feedback system to keep the vehicle on the planned path, avoid obstacles and return to the original planned path. This builds up a complicated situation because of the unpredictability in the state of the environment and incomplete knowledge. So preprocessing the state of the environment and using powerful computing techniques may not be able to solve the problem. On the other hand a spherical sensory data structure similar to human perception would provide an efficient and unified representation scheme for multi sensor integration3 and fusion, navigation, motion control and spatial reasoning. This paper focuses on the design of a video switching system that would permit the multi sensor integration and fusion. The Robotics team at the University of Cincinnati has developed a mobile robot incorporating a video switching unit that outputs the desired camera image from the sensors, which is shown below.

This solution addresses the four fundamental aspects of incorporating multiple sensors4: redundancy, complementarities, timeliness and cost of the information. The low cost of solid-state camera systems makes it possible to include many cameras on a mobile robot, thereby enhancing the reliability of the system to unpredictable changes in the environment. A video multiplexer forms the heart of the system. Multiplexers have served as fundamental building blocks in analog circuit design. Integrated-circuit switches have replaced most signal-switching circuits made from discrete component transistors and logiclevel shifters. Equally important, Integrated Circuit switches have continued to benefit from process and design improvements that reduce supply voltage, power consumption, on-resistance, charge injection, and switching time.


This paper is organized as follows. In Section 2, we propose the dynamic multi sensor fusion technique that is employed to navigate the robot through its environment. The constraints on the system are defined that outline the specific problem to be tackled.

2.SENSOR FUSION There are many challenges to the navigation of the mobile robot. The robot is required to stay on course as well as avoid any unexpected obstacles that it might encounter on its path. These set of constraints establish most of the requirements that a multi sensor system must incorporate, enabling some measure of “intelligence” for the mobile robot to interact with its unstructured environment without the control of a human operator. The problem is typically one of dynamic multi sensor fusion that involves a dynamic robot and a static object. The problem of dynamic multi sensor fusion refers to the temporal integration of static multi sensors at different time instants. The multi sensor system2 on the mobile robot is comprised of two video cameras that send images to the central processing system that acts accordingly and navigates the vehicle along its intended course. As the robot moves along its path, the multi sensor system constructs a local map of the surroundings, which are not completely known to the robot. This local map is constantly updated as the robot moves along its course. The use of multiple cameras on the robot supports the idea of a spherical multi sensor system that provides a complete field of view and a complete local map. This system also detects that are unknown to the global path manner. This is particularly useful in a situation when the robot is trapped in a complex haze, since the updated map allows the robot to retrieve to a position from where it could select an alternative path. The mobile robot moves along its course by evaluating alternative free pathways and then selecting a suitable one for the next period of motion

3.CAMERA SWITCHING UNIT The CCSU camera-switching unit7 could be used for a four-camera input or an eight camera input. They can be either colored or black and white. The CCSU-8/CCSU-8BW produced by FSR inc. accepts the red, green, and blue video inputs (black and white for 8BW) and switches the selected camera to the vision processor during the vertical interval. Features: • Can be used for up to eight camera with only one vision processor. • Works with color or black and white cameras. • Standard 12 pin camera connectors are used. • Remote switching by PLC or equivalent. • Indicator lights show which camera is in use. • All pluggable connections. The switching of the camera inputs on the unit is controlled by a specific code, which is a Binary code. There are three input bits, which change the output or switch from one camera to other according to the table shown below-


BCD Control Input Table

INPUT CAMERA

D0

D1

D2

1 2

L H

L L

L L

3

L

H

L

4

H

H

L

5

L

L

H

6

H

L

H

7

L

H

H

8

H

H

H

Here L (low) stands for 0 and H (high) stands for 1. As the input bits are changed the Camera Switching unit switches cameras. The control inputs are opto-isolated and will operate with any positive voltage from 5 VDC to 24 VDC.


CCSU Pinout Table

CCSU-8 COLOR CAMERA INPUT 1 GND 2 +12V 3 GND R 4 RED 5 GND HD 6 HD 7 VD 8 GND G 9 GREEN 10 GND B 11 BLUE 12 GND VD

OUTPUT 1 GND 2 NC 3 GND R 4 RED 5 GND HD 6 HD 7 VD 8 GND G 9 GREEN 10 GND B 11 BLUE 12 GND VD

CCSU-8 BW BLACK AND WHITE INPUT 1 GND 2 +12V 3 GND VIDEO 4 VIDEO 5 GND HD 6 HD 7 VD 8 GND 9 NC 10 GND 11 NC 12 GND VD

OUTPUT 1 GND 2 NC 3 GND VIDEO 4 VIDEO 5 GND HD 6 HD 7 VD 8 GND 9 NC 10 GND 11 NC 12 GND VD

The HD (horizontal drive) and VD (vertical drive) genlocking signals from the processor are buffered and continually distributed to all cameras, insuring the switching is transparent to the processor. The input and output connectors are the standard 12-pin circular connectors, which permits the use of standard camera cables.

SPECIFICATIONS The amplitude of the Video signals is 0.7V p-p positive going. Each camera receives a buffered TTL level signal. The input from the processor is terminated into 470 ohms and the output is back terminated with a 75 ohm 1% resistor. BCD type signals are used for control and the switching takes place during vertical interval. The Unit power is 115VAC to 220VAC, 60/50 Hz. and 75 watts. It accepts any outside source on the pluggable screw connector and routes it to each camera through the 12 pin input connector. The heart of the system is a Maxim video switch. The user simply selects using a three-bit control signal, which camera signal is selected. The output of the video switch is then the desired camera image. Switch and multiplexer architectures have not changed in many years, but the constant demand for lower supply voltage, better precision, and tighter specification tolerance obliges manufacturers to persevere with development-if only to achieve incremental performance improvements. The MAX442 video multiplexer/amplifier combines a 140 MHz video amplifier with a high-speed, 2-channelmultiplexer in an 8-pin package. Its 36ns switching time and low differential gain (0.07%) and phase (0.09°) errors make it ideal for broadcast-quality video applications. The MAX442 operates from ±5V supplies and consumes 300mW.The low input capacitance (around 4pF) maximizes high-speed performance. MAX442 video amplifier is compensated for unity-gain stability, and features a 140 MHz bandwidth and a 250V/µs slew rate. A ground pin separating the two input channels minimizes the cross talk, which also simplifies the board layout.


Line Power Input CAMERAS

Camera power supply Input

OUTPUT

Control Signal Input

CAMERAS

Vision Processor Pin Configuration

IN0

1

8

A0

GND

2

7

V+

IN1

3

6

Vout

V-

4

5

N-


PIN DESCRIPTION

PIN No.

NAME

FUNCTION

1 2 3 4 5 6 7

INO GND IN1 VINVout V+

Analog Input, Channel 0 Ground Analog Input, Channel 1 Negative Power Supply, -5V Amplifier Inverting Input Amplifier Output Positive power Supply

8

A0

Channel Address Input A0: Logic 0 selects Channel 0 A0: Logic 1 selects Channel 1

The bipolar construction of MAX4426 results in a typical channel input capacitance of only 4pF, whether the channel is on or off. Even with source impedances as great as 250Ω, which is a significant improvement over common mux or switch alternatives the MAX442’s low channel input capacitance allows full AC performance of the amplifier. For most of today's analog switches, the actual switching element is a pair of metal-oxidesemiconductor field-effect transistors (MOSFETs). Unlike bipolar transistors, MOSFETs can handle bidirectional drain-to-source channel currents. Moreover, a voltage-controlled MOSFET is free of the error caused by base-to-emitter currents in a bipolar transistor. MOSFET switches exhibit on-resistance, but no dc offset. In switching applications, enhancement-mode MOSFETs, offering better characteristics and easier fabrication are preferable to depletion types. Enhancement-mode types are self-isolating, with drain and source regions formed in a single diffusion step. Because all active regions are reverse-biased with respect to each other and the substrate, adjacent devices on the same substrate are electrically isolated without recourse to dielectric isolation or other special techniques. The MOSFET's insulated gate minimizes the effect of dc control voltage on the signal channel. A single n-channel or p-channel enhancement-mode MOSFET can serve as an analog switch, but its on-resistance will vary considerably with signal voltage. Connecting an n-channel and p-channel device in parallel-the almost universal configuration for CMOS analog switches-greatly reduces this variation. Complementary gate-drive signals turn the two devices on or off simultaneously.

4.APPLICATION OF THE VIDEO SWITCHING SYSTEM FOR U.C. ROBOT The motion control and navigation of mobile robots requires is an on-line, real-time sensory feedback system to guide the robots to stay on planned paths, to avoid obstacles and to return to the planned path if a detour is necessary. This is a difficult problem because of dynamic and incomplete world knowledge, which includes obstacles and unknown track to follow. Hence it is desirable to have an efficient model that captures human cognitive and sub-cognitive characteristics. Images of sonar sensing and those from the cameras may be integrated in the model to enhance image understanding and object recognition capabilities. The multisensor fusion integrates the data to yield useful information about the obstacle and the tracks.


The block diagram given below explains how this function is accomplished using the video switch for the U.C. robot.

The UC robot follows a line track and switches camera views to see if it is at the right track or not. The data flow represented by solid lines and the broken lines in the figure show the flow of control. The image from the left or right cameras is sent to the video switch and then to the ISCAN. ISCAN is the software used to see whether the robot is following the line track or not. It does that by taking the centroid of the image and matching it with a threshold value. The central logic controller then processes the signals it gets from the ISCAN. This sends out signals to control the motion of the robot accordingly. The GALIL motion controller does the motion control. If the centroid of the line track doesn’t match the threshold value specified, the central logic controller to switch the cameras generates a signal. The Video switch using the CCSU-camera switching unit does this part. The bits are changed to switch the camera view either from left to right or from right to left camera. Now the signals captured are again sent along the same path. If there is a match then the motion of the robot is set accordingly to keep the robot in track or else the robot keeps moving straight.


5.CONCLUSION This paper has described a technique for autonomous vehicle navigation in an unstructured environment. Test runs conducted on the mobile robot developed at the Robotics Center, University of Cincinnati have demonstrated the ability to accurately estimate the robot’s position and orientation with respect to its course and any obstacles that it might encounter on its path. This paper has described the use of a video switching system that permits eight cameras to be multiplexed with a single chip. Extensive refinement of this technique would enable multiple cameras to be multiplexed with a single processor thereby representing a significant advancement in autonomous vehicle navigation. Future work includes the continuous implementation of the video switch in related applications that include surveillance and mobile systems.

6.REFERENCES [1] E. T. Baumgartner, P. C. Leger, P. S. Schenker, and T. L. Huntsberger, “Sensor-Fused Navigation and Manipulation from a Planetary Rover,” Proc. SPIE 3523, pp. 58-66, November 1998. [2] A.G. O. Mutambara, “Nonlinear Sensor Fusion for a Mobile Robot,” Proc. SPIE 2905, pp. 102-113, November 1996. [3] C. C. Williams Hulls, and W. J. Wilson, “Integration of camera and range sensors for 3D pose estimation in robot visual servoing,” Proc. SPIE 3523, pp. 76-87, November 1998. [4] A. Fatemi, and H. Lecocq, “Multi-Sensor Detection Schemes for Mobile Robots,” Proc. SPIE 2905, pp. 150-160, November 1996. [5] R. C. Luo, and M. G. Kay, “Mutli-Sensor Integration and Fusion for Intelligent Machines and Systems,” Ablex Publishing Corporation, 1995. [6] http://dbserv.maxim-ic.com [7] http://www.fsrinc.com
