Vision Based Monitoring System for Passenger's Safety in ... - CiteSeerX

First page Template Secretariat uses only. Do not type in this box.

Vision Based Monitoring System for Passenger’s Safety in Railway Platform Sehchan Oh, Changmu Lee and Hanmin Lee Advanced EMU Research Team of Korea Railroad Research Institute 360-1, Woram-dong, Uiwang-si, Gyeongi-do, Korea Daeho Lee, Youngtae Park Dept. of Electronic Engineering, Kyung Hee University Seocheon-dong, Giheung-gu, Yongin-si, Gyeongi-do, Korea

Abstract The proposed system automatically detects accident in platform and analyzes level of danger using image processing technology, finally inform operators of the accident with video and alarm immediately. Especially, the system uses stereo vision technology with multi-sensors for minimizing detection error in various railway platform conditions. The information acquisition process monitors entire area of the track line in platform by using a series of stereo and thermal cameras, and infrared sensors. The decision making process analyzes the situation and level of danger with detection result from IAP. The information multicasting process transmits video information and alarm message with standard operation procedure to preset receivers, such as central control room, station and train. Keywords: Vision, monitoring, railway, platform, stereo

in unattended station environment, which can detect dangerous situation and inform real-time supervisor or operator of the accident with video data and alarm message. The paper proposes a vision based monitoring system for passenger’s safety in railway platform. The proposed system automatically detects accident in platform and analyzes level of danger using image processing technology, finally inform operators of the accident with video and alarm immediately. Especially, the system uses stereo vision technology with multi-sensors for minimizing detection error in railway platform conditions, various illumination effects of train and station lighting facilities. As shown in Figure 1, the information acquisition process(IAP) monitors entire area of the track line in platform by using a series of stereo and thermal cameras, and infrared sensors. The decision making process(DMP) analyzes the situation and level of danger with detection result from IAP. The information multicasting process(IMP) multicasts video information and alarm message with standard operation procedure(SOP) to preset receivers, such as central control room (CCR), station and train.

1 INTRODUCTION However it is very difficult to prevent actually various accidents in railway platform environment, though many safety facilities are applied, such as emergency stop button, safety fence, CCTV(Closed Circuit TV) and so on. There are many research activities for passenger’s safety in railway applications. Advancement in information technology have enabled applying vision sensor to railway, such as CCTV. CCTV has been widely used in railway application, however the CCTV is a passive system that provide limited capability to maintain safety from boarding platform. The station employee should monitor continuously CCTV monitors. Therefore immediate recognition and response to the situation is difficult in emergency situation. Currently, Korean urban transit operators have applied video transmission system using wireless communication technology, which transmits video information about platform situation to train approaching at the station. Similarly, the same problem is occurred, that the train driver should observe continuously the video monitor when approaching at every station. Recently, urban transit operators are pursuing applying an unattended station operation system for their cost reduction. An intelligent monitoring system is need for platform safety

2 SYSTEM CONFIGURATION The proposed system consists of three main process, i.e., IAP, which detects dangerous factors, such as passenger

1

Second page and after Template Secretariat uses only. Do not type in this box. object, and thermal camera processing unit offers thermal information of fallen object, and infrared sensor processing unit provides position and roughly size information of fallen object. The detailed algorithm of detection process was described in Chapter 3.

caught in between train doors and fallen on the track line and so on, and DMP, which analyzes the situation with detection result from IAP and decides level of danger, and IMP, which multicasts video and alarm message about the accident to CCR and station employees and train driver approaching in the station.

2.2 Decision Making Process DAP estimates and analyzing the accident situation and level of danger with detection results from IAP. In Figure 3, the process procedure of DMP is represented. The detection results from IAP are stacked in event queue through the network. DMP checks every event whether it is valid information or not, then classifies input events according to accident types. DMP retrieves similar accident cases from event history DB formed previous accidents, and estimates current event situation and decides level of danger. The event history DB is updated with the processing results. In case of emergency DMP send the event with both video and alarm to IMP for multicasting.

Camera sensors - Thermal cameras - Stereo video cameras

Information acquisition process - Fallen passenger detection

Fallen passenger detection

Decision making process - Situation analysis - Level of danger decision

Information multicasting process - Multicast video and alarm

Event Queue

Check validity No Central Control Room

Station Employee

Valid?

Train Driver

Yes

Figure 1. System concept of vision based railway platform monitoring system

Event Classify retrieval

2.1 Information Acquisition Process

Analyzing Situation and Level of danger

As shown in Figure 2, IAP monitors entire track line of platform with a series of stereo and thermal cameras and infrared sensors. A stereo camera covers about 40m of track, and a thermal camera is responsible for about 100m of track, and an infrared sensor has 30Cm monitoring area. However, theses detection area can be modified according to platform condition. For example, in case of curved platform, more detection sensors needed, and consequently detection area of a single sensor should be reduced.

No

Event history DB event

Emergency? Update history DB Inform the result

Figure 3. Flowchart of DMP procedure 2.3 Information Multicasting Process

Infrared sensor

Stereo Camera

IMP provides preset clients, such as such as local station employee, CCR employee and train driver with results of DMP, i.e. corresponding alarm message including SOP for employees with video information about the accident situation in order to deal promptly with emergencies.

Thermal Camera 10m

40m

3 DETECTION PROCESS 1

2

3

4

5

6

7

8

9

10

Detection process follows the procedure described in Figure 4. To make a right decision of dangerous factor for fallen object in monitoring area, it is very important to find the accurate train states in the area. The system detects four different train states i.e. approaching (IN-state),

Figure 2. System configuration of IAP Detection results from each sensor are transmitted to DMP through the network. Stereo camera processing unit provides distance information as well as position information of fallen

2

Second page and after Template Secretariat uses only. Do not type in this box. stopping(ON), pulling out(OUT), empty(OFF). Detection process for fallen passenger should be carried out in OFF train states, and detection of passenger caught in between train doors should be performed in OUT mode.

3.3 Object Detection To detect object, the system uses background subtraction technique, and renews the current background every time with residual information, that is difference between current and previous images. The renewal of background is done in OFF mode because camera can archive correct background image in that mode. The renewal of background can be defined as: (2) BGcurr = BG prev + K ( IMGcurr − BG prev )

Train Detection

OFF mode?

No

Yes OUT mode? Yes Detection of Passenger caught in between train doors Yes

Fallen Object Detection

where BG and IMG represent background and input images respectively, and K means background renewal constant. In this paper, K is given as 0.0005. To detect object more accurately, the system integrates detection result from every single sensor. For example, system finds out object in danger area with single lens(left or right) of stereo camera, and calculates distance from camera lens to the scene object by using stereo vision algorithm. The basic concept of stereo vision is exploiting the similarities along the disparity, the displacement between two separate views. The system searches the minimum displacement in target view for each block of reference view. With the minimum displacement, we can find out distance from camera focus to the target object. As shown in Figure 6, distance from object can be calculates as:

Fallen Object Recognition No Emergency? Yes Information Multicasting

Figure 4. Flowchart of detection algorithm 3.1 Train Detection To make decision of dangerous factor for fallen object in monitoring area, it is important to find the accurate train states in the area for every single camera. Train and monitoring areas should be clearly defined. Monitoring area used for monitoring passenger’s danger, and danger area used for monitoring train states. Transition of train states can be achieved with motion vectors in danger area. Motion vector in danger area is defined as : MVB(i ) currt = MB(i ) curr − MB(i ) prev (1)

f Z = b×( ) d

(3)

With distance information, the system decides whether the object is fallen danger area or not. In case of human detection, thermal information of object from the thermal camera processing unit is also used.

Left camera axis

where MVB(i ) , MB(i ) represent motion vector of i-th block and i-th macro block respectively. The transition of train states is archived as described in Figure 5. In OFF and ON states, motion vector should be grater than preset threshold to change its state. OFF

Right camera axis

No

M

Scene object point (x, y, z) P

N

Yes No MV > Tr?

No MV?

No Z

Yes

IN

OUT Left image plane

Yes No

No MV? Yes

MV > Tr?

x`l L

x`r pl

R

Cl

No

Right image plane

Cr Left camera Lens center

pr

f

Focal length

Right camera Lens center

b (Base line)

ON

Figure 6. Object distance calculation using stereo camera

Figure 5. Transition condition for each train state

3

Second page and after Template Secretariat uses only. Do not type in this box. Experimental results of detecting train states for Sung-nae and Hey-hwa Station are represented in Figure 9 and Figure 10. For reducing error rate, one state is changed another by checking five consecutive frames.

4 EXPERIMENTAL RESULTS To verify system performance, the experiments have been executed in Sung-nae, a aboveground station, and Hey-hwa, a underground station, of Seoul Metro since 2006. A frame of test sequence acquired by stereo camera is presented in Figure 7. Figure 7(a) shows a right-view frame of Sung-nae station, and Figure 7(b) shows a right-view frame of Hey-hwa station.

(a)

(b)

(a) (b) Figure 7. Test sequences: (a) Sung-nae station, (b) Hey-hwa station (c) (d) Figure 9. Experimental results of train detection in Sung-nae station: (a) OFF state, (b) IN state, (c) ON state, (d) OUT state

In Figure 8, blue-lined and red-lined area show train area, danger area and object area respectively.

(a)

(b)

(a)

(c) (d) Figure 10. Experimental results of train detection in Hey-hwa station: (a) OFF state, (b) IN state, (c) ON state, (d) OUT state According to the experimental results, we can find out the proposed train detection algorithm operates robustly in real railway platform environment. In Figure 11, the test results for disparity estimation are showed according to different type and size of subject. Figure 11(a) and (b) shows disparity map for a fallen box in danger area. Similarly, Figure 11(c), (d) and Figure 11(e), 11(f) are for a fallen doll and human respectively. To take fine disparity estimation results every time is very difficult and a challenging problem. Disparity estimation algorithm

(b) Figure 8. Train and danger areas: (a) Sung-nae station, (b) Hey-hwa station.

4

Second page and after Template Secretariat uses only. Do not type in this box. will be modified continually because fine disparity map directly affects the results of object detection rate.

(a)

(b)

(c)

(d)

5 CONCLUSIONS The paper proposes a vision based monitoring system for passenger’s safety in railway platform. The proposed system automatically detects accident in platform and analyzes level of danger using image processing technology, finally inform operators of the accident with video and alarm immediately. Especially, the system uses stereo vision technology with multi-sensors for minimizing detection error in railway platform conditions, various illumination effects of train and station lighting facilities. We verify the system performance with experimental result in real platform condition. The experimental result shows that detection of train state and object is conducted robustly by using proposed stereo-vision based object detection algorithm. Currently, we are pursuing an effective information transmission system for immediately dealing with the safety accidents. We expect the proposed monitoring system will play key role in establishing highly intelligent monitoring system for passenger’s safety in future railway environment. ACKNOWLEDGEMENT This research was supported by a grant(R&D/05Advanced Rail Tech. A01-01) from Development of Advanced Urban Transit System Program funded by Ministry of Land, Transport and Maritime Affairs of Korean government.

(e) (f) Figure 11. Test results for disparity map: (a) original image for a box, (b) disparity map for a box, (c) original image for a doll, (d) disparity map for a doll, (e) original image for a human, (f) disparity map for a human.

REFERENCES [1] Y.Sasaki, N.Hiura. “Development of Image Processing Type Fallen Passenger Detecting System,” JR-EAST Technical Review Special Edition Paper, No. 2, pp.66-72, 2003. [2] I.Yoda, K.Sakaue. “Ubiquitous Stereo Vision for Controlling Safety on Platforms in Railroad Station,” IEEJ Tr. on Electronics, Information and Systems, Vol. 124, No. 3, Mar., pp.805-811, 2004. [3] J. Vhzquez, M. Mao, "Detection of moving objects in railway using vision," IEEE Intelligent Vehicles Symposium University of Parma, Parma, Italy Jun. 1447, 2004. [4] I.Yoda, "Image processing technology for advanced safety to people in railroad transportation - For railroad crossing and station platform ," IPSJ Magazine Vol.48, No.1, pp.10-16, Jan. 2007. [5] Shigeki Sugimoto, Hayato Tateda, Hidekazu Takahashi, Masatoshi Okutomi, "Obstacle Detection Using Millimeter-Wave Radar and Its Visualization on Image Sequence," icpr, pp. 342-345, 17th International Conference on Pattern Recognition (ICPR'04) - Volume 3, 2004

The Figure 12 shows the results of human detection in danger area with stereo and thermal cameras.

(a)

(b)

(c) (d) Figure 12. Test results for object detection: (a) original image, (b) disparity map, (c) object detection with stereo camera, (d) object detection with thermal camera

5