Multi-sensors data fusion system for fall detection Damien Brulin and Estelle Courtial
Abstract— This paper presents a data fusion architecture whose goal consists in improving home support of the elderly mainly by detecting a potential fall. Among the different fusion modules composing this architecture, we focus on the human posture recognition system based on the fusion of visual features using a fuzzy logic algorithm. The posture, combined with other information (location, duration), enables to monitor the people state. Experimental results illustrate the robustness and the efficiency of the proposed approach.
I. INTRODUCTION The problem of population ageing constitutes one of the main concerns of modern societies. Combined with the increasing desire of the elderly to stay as long as possible in their own home, it is necessary to propose solutions to better manage their comfort and autonomy while providing a medical survey. In the last decades, the concept of Health Smart Home (HSH) has been developed to realize the monitoring of the person and the communication of information. For a complete overview of HSH projects, one can referred to [1]. Among these projects, we can cite the HIS project of the TIMC-IMAG laboratory [2] or the CompanionAble project [3] whose problematic are similar to ours. Several data are needed to make the decision: human presence, location, posture and health state. However, the knowledge of these data can not be obtained using a single sensor. A multi-sensor approach is usually considered which involves the processing of data of different types. To ensure the human presence detection, movement detectors, like Passive Infrared detector (PIR) are generally used offering a good performance for a reduced cost. For the posture determination, the classical approach consists in using an invasive sensor embedded on the person like an accelerometer [3][4]. Within the CAPTHOM project, framework of this work, one of the industrial constraints is not to use an invasive sensor, because people, especially the elderly, hardly bear embedded sensor. To get around this difficulty, a non invasive approach has been considered by using a visual sensor. Several features, extracted from the 2D signal delivered by the camera, are fused to determine the posture of a person. In [5], authors have developed an approach based on evidence theory, in order to deal with imprecise data, to determine four static human postures. Their approach needs the determination of a reference posture realized at a given distance to ensure the recognition. Therefore, they D. Brulin and E. Courtial are with Institut PRISME UPRES EA 4229, Polytech’Orléans, 8, rue Léonard de Vinci, 45072 Orléans Cedex 2, France
[email protected],
[email protected]
978-1-4244-6561-3/10/$26.00 ©2010 IEEE
are dependent with respect to the person-camera distance. We propose an alternative approach that enables us to be more robust considering the distance person-camera and its morphology. To take into account the imprecise aspect of the data delivered by a camera and to deal with the potential conflict situations, a symbolic method, based on fuzzy logic, is used to fuse the extracted features. We choose this method to ensure the respect of some constraints as computation time and integration on an embedded processing platform. One can argue that using a visual signal can be a breach of privacy. We want to underline that no 2D signal is recorded or visualized. The remainder of this paper is organized as follows. Section 2 presents the HSH global system in which the posture determination system is used and illustrates the application and acquisition conditions. Then section 3 illustrates the main step of the fuzzy logic theory for the posture recognition problem. Section 4 shows the obtained classification result and its purpose for fall decision. Finally, in the last section, we conclude the paper and give some prospects. II. THE MONITORING SYSTEM As we have mentioned above, many sensors are involved in an HSH project because none of them can, on its own, provides data concerning all the required information : human presence, location, posture. So, the determination of that information involves the fusion of all the data at different levels. The knowledge of that information will allow to make a decision concerning a potential warning situation (a fall for example). We propose a global fusion architecture which allows to ensure a constant acquaintance of the different information based on the data coming from three sensors : PIR detectors, thermopiles and camera. The proposed system is illustrated by the Fig. 1.
Fig. 1.
Data fusion architecture.
Each sensor constitutes one input of the system and, as output, a message or an alert is sent. The fusion architecture
is composed of four modules. These modules are : (1) human presence detection using data from the three kinds of sensor, (2) the location module based on visual measurements and PIR signals, (3) the posture recognition, topic of this paper, based on features extracted from the 2D signal of the camera and (4) a decision tool for home monitoring. In this paper, we focus on the posture recognition system which permits to determine if a person is fallen or is in a normal posture according to its location in the room. Indeed, if a person is lying in bed, the system does not have to set off an alert. Conversely, if the person is lying on the ground in the kitchen, an alert should be sent to an intervention center. The 2D signal of a camera is used to determine the posture. The monitored environnement consists in an indoor scene where there is only one person at a time. This assumption can be made insofar as, if another person is present, the system has no more reason to be active. We also make the assumption that the person is not partially occluded. The camera is placed at about 2 meters height and the 2D signal is 320 × 240 pixels.
∙ ∙
𝑎𝑥1 , component of the axis 𝑒 according to 𝑢, 𝑎𝑥2 , component of the axis 𝑒 according to 𝑣.
Fig. 3.
Representation of 𝐷1, 𝐷2 and 𝐷3.
III. POSTURE RECOGNITION MODULE A. The pre-processing step Before the posture recognition procedure, we have to detect the person in the 2D signal and to extract useful features. The visual people detection is realized thanks to the algorithm developed by [6]. This algorithm is composed of three steps : adaptative background removal, tracking of point of interest and human characterization. We only use information from the first and the last step. The segmentation of the person done by the background removal allows us to compute the first principal axis 𝑒 of the person and its gravity center 𝐶 (Fig. 2). A principal component analysis method is used to compute these two parameters [7]. If the detected
Fig. 2.
Principal axis and gravity center display.
entity is characterized as a human, the person is surrounded by a rectangular bounding box (Fig. 3). From this box, three distances are computed : 𝐷1 the height of the box (head/feet distance in standing posture), 𝐷2 the width of the box and 𝐷3 the distance between 𝐶 and the bottom of the box. From these data (𝐷1, 𝐷2, 𝐷3, 𝐶 and 𝑒), we define four parameters used as inputs of a fuzzy logic algorithm (figure 4): ∙ 𝑟1 = 𝐷3/𝐷1, ∙ 𝑟2 = 𝐷2/𝐷1,
Fig. 4.
Posture recognition flowchart.
We use distance ratios in order to improve the robustness of our approach with respect to the distance between the person and the visual system and to permit the posture recognition regardless of the person appearance. B. Fuzzy logic algorithm According to the imprecision of the parameters and the potential conflict situation, a symbolic approach is used to fuse the four parameters mentioned above [8]. We first define an output universe composed of the fuzzy subsets associated to each considered posture : lying (𝐻1 ), squatting (𝐻2 ), sitting (𝐻3 ) and standing (𝐻4 ). We also add a fifth hypothesis 𝐻0 corresponding to undetermined or unrecognized postures. The fuzzy inference system is composed of three steps: fuzzification, fuzzy rules (inference) and defuzzification. (a) Fuzzyfication Contrary to classical logic, fuzzy logic uses the concept of partial membership. It means that one element can gradually belong to different fuzzy subsets. This property is represented by membership functions which, for a fuzzy subset A of a universe U, can be defined as follows: ∀𝑥 ∈ 𝑈, 𝑓𝐴 (𝑥) ∈ [0; 1].
(1)
The first step consists in identifying, for each input and output, the different fuzzy subsets and their membership functions. From several experimentations corresponding to scenarios involving the different postures, we have defined the different membership functions. The Fig. 5 illustrates the variations of the parameter 𝑟2 for a sequence where all the postures except the lying one have been realized. From these variations, we identified two fuzzy subsets: weak and
average value. With a sequence involving the lying posture, we defined a third fuzzy subset (strong value). Then, for the parameter 𝑟2 , three trapezoidal membership functions have been used (Fig. 6). We noticed that the parameters do not have the same sensibility with regard to the posture and so, we identified between two and four fuzzy subsets for each parameter. r2=D2/D1 0.9 H2
H4
H3
H4
H3
H4
H2
𝑓𝑆 (𝑧) = max(𝑓𝑆1 (𝑧), 𝑓𝑆2 (𝑦)), . . . , 𝑓𝑆8 (𝑧)).
(5)
(c) Defuzzification The final step, called defuzzification, consists in turning the membership function solution generated by the fuzzy rules into a more comprehensive value. Two approaches can be considered: (1) the method of maxima and (2) the method of the center of gravity. We have chosen a maxima method, the Mean of Maximum (MOM), as it requires less computing power which is significant for real time application.
1
0.8
For an input vector 𝑉 = (𝑟1 , 𝑟2 , 𝑎𝑥1 , 𝑎𝑥2 )𝑇 , we compute the membership degree for each of the eight rules. To obtain the membership function solution, we finally use the aggregation operator max to combine the membership degrees of all rules:
H4
0.7 0.6 0.5 0.4 0.3 0.2
IV. APPLICATIONS AND RESULTS
0.1 0
0
100
200
300
400
500
600
700
A. Evaluation of posture recognition
800
time
𝑟2 variations according to one scenario.
Fig. 5.
Degree of membership
strong
average
weak
1
Let 𝑉 = (0.5, 2.9, 0.9, 0.23)𝑇 the input vector. Only the rule (3) is verified. 𝑎𝑥1 is strong with a degree of 1 (Fig. 7(a)) and 𝑎𝑥2 is weak with a degree of 0.33 (Fig. 7(b)). So, according to the Mandani implication and after aggregation, we obtain the membership function solution illustrated by the Fig. 7(c). After defuzzification, the system decides that the person posture is lying. A set of 15 video sequences (≈2-3min) has been recorded in a real HSH in order to evaluate the posture recognition system. Each sequence, realized by two subjects, illustrates situations of everyday life or of emergency. The four considered postures have been realized several times, some of them are illustrated below (Fig. 8). The recognition rates obtained for each posture are given by the Table I. Columns show the real posture and lines represent the posture recognized by the system.
0.8
0.6
0.4
0.2
0 0
0.5
1
1.5
2
2.5
3
3.5
4
r2=D2/D1
Fig. 6.
Membership functions of the 𝑟2 input.
Comment: 𝑟2 has rarely a value between 1 and 2.5 which explains the absence of membership function for these values. (b) Fuzzy rules and inference system As in classical logic, it is possible to apply reasoning, called fuzzy inference rule, on the fuzzy subsets structured in an IF-THEN format: 𝐼𝐹 (𝑥 ∈ 𝐴) 𝐴𝑁 𝐷 (𝑦 ∈ 𝐵) 𝑇 𝐻𝐸𝑁 (𝑧 ∈ 𝐶),
(2)
TABLE I T EST CONFUSION MATRIX . Syst/Réel 𝐻0 𝐻1 𝐻2 𝐻3 𝐻4
𝐻1 1,2% 83,1% 6,5% 9,1% 0,0%
𝐻2 0,0% 0,0% 83,0% 14,9% 2,1%
𝐻3 0,0% 0,3% 4,9% 94,8% 0,0%
𝐻4 1,3% 0,4% 4,5% 1,8% 92,0%
𝐻0 8,2% 13,7% 25,9% 32,8% 19,5%
where 𝐴, 𝐵 and 𝐶 are fuzzy subsets. For our application, eight rules have been defined taking into consideration all the inputs and outputs. For example, one of the rules is defined as follows: 𝐼𝐹 (𝑎𝑥1 𝑖𝑠 𝑠𝑡𝑟𝑜𝑛𝑔) 𝐴𝑁 𝐷 (𝑎𝑥2 𝑖𝑠 𝑤𝑒𝑎𝑘) 𝑇 𝐻𝐸𝑁 (𝑝𝑜𝑠𝑡𝑢𝑟𝑒 𝑖𝑠 𝑙𝑦𝑖𝑛𝑔)
(3)
It exists different rules to defined the implication (THEN) and conjunction (AND) operators. We have chosen the Mamdani rules [9] which uses the operator min to realize these two operations. So, for the rule (2), the membership degree is defined by: 𝑓𝑆𝑖 (𝑧) = 𝑚𝑖𝑛(𝑚𝑖𝑛(𝑓𝐴 (𝑥), 𝑓𝐵 (𝑦)), 𝑓𝐶 (𝑧)),
(4)
Fig. 8.
Examples of posture in video sequences.
The average recognition rate of the system is 88,2%. In general, the different postures are correctly recognized; some conflicts exist between the postures squatting and
(a)
(c)
(b) lying
weak 1
Degree of membership
X: 0.9 Y: 1
0.8 average
weak
Degree of membership
0.8 Degree of membership
squatting
sitting
standing
1
strong
1
strong
0.6
0.4
0.2
0.6
0.33 0.2
0.2
0
0
0 0
0.2
0.4
0.6
0.8
1
0
0.2
0.4
ax1
0.6
0.8
1
0
2
4
6
8
10
12
14
16
18
posture
ax2
Fig. 7.
Illustrations of the fuzzy logic algorithm.
sitting. The different detection errors can be explain either by lighting variations or by shadow and reflection areas that can affect the segmentation result and the size of the bounding box. To reduce these errors, some fittings on membership functions or inference rules could be considered. Concerning the lying posture, the recognition rate is 83,1% which is correct considering the fact that if the person is lying in the camera axis, the bounding box size does not always permit to discriminate the posture. B. Fall detection Once the posture has been determined, the global fusion system is able to decide if the person is fallen or if the posture is correct depending on its location in the observed scene. The decision also depends on the posture duration. If the person stays lying less than 1s, the posture is not considered. On contrary, the system considers that the person may have fallen and sends the message : alertness. If the lying posture lasts more than 10s then the fusion system decides to activate an alert. To illustrate this procedure, we have developed a graphical user interface (GUI) on Matlab (Fig. 9) which shows the different information and decisions depending on the video sequence (several sequences have been realized but only one example is presented here due to lack of space).
Fig. 9.
0.6
0.4
X: 0.2333 Y: 0.3333
0.4
0.8
Decision system GUI - Fall example.
V. CONCLUSION A method based on fuzzy logic for the posture recognition of a human from 2D signal of a camera has been proposed. This approach offers good performances and is robust regardless to the person appearance and the distance from
the visual system. The posture recognition module belongs to a global fusion system designed for home support of the elderly. According to the posture, the location and the duration, the decision system determines if there is an alert situation or not. A GUI has been created to illustrate the system response for different video sequences. To improve the global result, we plan to refine the membership functions and the inference rules by recording video in real conditions, ie. with aged people in the HSH of the "Bellevue" rest-home in Bourges. VI. ACKNOWLEDGMENTS We especially thank all our partners involved in the CAPTHOM project. This work was realized with the financial help of the French Industry Ministry and local collectivities, within the framework of the CAPTHOM project of the Competitiveness Pole 𝑆 2 𝐸 2 , www.s2e2.fr. R EFERENCES [1] Chan M., Estève D., Escriba C.,Campo E., A review of smart homesPresent state and future challenges. Computer Methods and Programs in Biomedicine, vol. 91(1), 2008, pp. 55–81. [2] Fleury A., Vacher M., Glasson H., Serignat J.F., Noury N., "Data Fusion in Health Smart Home: Preliminary Individual Evaluation of Two Families of Sensors", in proc. of the 6th International Conference of the International Society for Gerontechnology, Pisa, Italy, May 2008. [3] Medjahed H., Istrate D., Boudy J., Dorizzi B., "A fuzzy logic system for home elderly people monitoring (EMUTEM)", in proc. of the 10th WSEAS International Conference on Fuzzy Systems, Prague, Czech Republic, March 2009, pp. 69–75. [4] Demongeot J., Virone G., Duchêne F., Benchetrit G., Hervé T., Noury N., Rialle V., Multi-sensors acquisition, data fusion, knowledge mining and alarm triggering in health smart homes for elderly people, Comptes Rendus Biologies, vol. 325(6), 2002, pp. 673–682. [5] Girondel V., Caplier A., Bonnaud L., "A Belief Theory-Based Static Posture Recognition System for Real-Time Video Surveillance Applications", in proc. of the IEEE International Conference on Advanced Video and Signal based Surveillance, Como, Italy, September 2005. [6] Benezeth Y., Emile B., Laurent H., Rosenberger C., "A Real Time Human Detection System Based on Far Infrared Vision", in proc. of the 3rd International Conference on Image and Signal Processing, Cherbourg-Octeville, France, July 2008, pp. 76–84. [7] Duda R.O., Hart P.E., Stork D.G., Pattern Classification, WileyInterscience, 2nd Edition, 2001. [8] Klir G.J., Yuan B., Fuzzy sets and fuzzy logic : theory and applications, Prentice Hall, 1st Edition, 1995. [9] Mamdani E.H. Applications of fuzzy set theory to control systems: a survey, in Fuzzy Automata and Decision Processes, Gupta M.M., Saridis G.N. and Gaines B.R. eds, 1977, pp. 1–13.
