A Fuzzy Multi-Sensor Architecture for Indoor Navigation A. Amanatiadis, D. Chrysostomou, D. Koulouriotis, and A. Gasteratos Department of Production and Management Engineering, Democritus University of Thrace, University Campus Kimmeria, GR-67100, Xanthi, Greece Email:
[email protected], {dchrisos, jimk, agaster} @pme.duth.gr
Abstract—This paper presents an indoor navigation system based on sensor data from first responder wearable modules. The proposed system integrates data from an inertial sensor, a digital camera and a radio frequency identification device using a sophisticated fuzzy algorithm. To improve the navigation accuracy, different types of first responder activities and operational conditions were examined and classified according to extracted qualitative attributes. The vertical acceleration data, which indicates the periodic vibration during gait cycle, is used to evaluate the accuracy of the inertial based navigation subsystem. The amount of strong feature correspondences assess the quality of the three-dimensional scene knowledge from digital camera feedback. Finally, the qualitative attribute, in order to evaluate the efficiency of the radio frequency identification subsystem, is the degree of probability of each location estimate. Fuzzy ifthen rules are then applied to these three attributes in order to carry out the fusion task. Simulation results based on the proposed architecture have shown better navigation effectiveness and lower positioning error compared with the used stand alone navigation systems. Index Terms—Indoor navigation, pedestrian localization, multi-sensor fusion, first responder navigation system.
I. I NTRODUCTION Inertial navigation systems (INS) are one of the most widely used dead-reckoning systems. They provide continuous position, velocity, and also orientation estimates, which are accurate for a short term, but are subject to drift due to noise of the sensor [1]. Conventional Pedestrian Dead Reckoning (PRD) systems involve the indirect estimation of step length and direction of walking. However, these approaches have severe limitations. The models must be trained for a specific users and inaccuracies can arise if the walking surface or shoes are modified. Furthermore, tracking a person for more than a few seconds using inertial sensing alone causes a slowly accumulative tilt error. For these reasons, foot-inertial approaches to pedestrian navigation have been proposed. These approaches measure the distance between footfalls by 3D acceleration and orientation measurements sensed directly at the foot [2]. This approach corrects the velocity error after each stride, breaking the cubic-in-time error growth and replacing it with an error accumulation that is linear in the number of steps. Camera based indoor navigation approaches use the threedimensional knowledge of a scene from images [3]. More precisely, they use Structure from Motion (SfM) algorithms.
These algorithms refer to the problem of recovering the structure of the scene using multiple 2D images taken by a moving camera and motion information of the camera. The motion information is the position, orientation and intrinsic parameters of the camera at the captured views. Given feature correspondences, the geometric constraints among the different views can be established. Thus, the projection matrices that represent the motion information can be recovered. Existing algorithms can be classified in two families, namely batch algorithms [4], [5], which recover all pose and structure parameters in one step, and sequential algorithms where the parameters are recovered progressively as new views become available. Besides, significant effort has been put into so-called auto-calibration [6], where the initially unknown intrinsic parameters of the camera are recovered together with the pose. Location systems based on estimations from Received Signal Strength Indication (RSSI) of wireless signals are rapidly gaining popularity and are becoming more and more precise and sophisticated [7]. RSSI based location estimation techniques are broadly divided into deterministic and probabilistic techniques. In the deterministic techniques, the physical area making up the environment is first divided into cells. Next, training phases are executed in which readings are taken from several fixed, known access points or beacons. Finally, localization is performed in a determination phase, in which the most likely cell is selected by determining which cell fits the new measurement best [8]. On the other hand, probabilistic techniques construct a probability distribution over the targets location for the physical area making up the environment. In order to estimate the location of the target, different parameters like the mode of the distribution or the area with highest probability density may be used. While probabilistic techniques provide more precision, a trade-off between computation overhead and precision is introduced. The main problem in such techniques is that the signal measurements are inherently noisy due the multipath problem and other factors, that will be discussed in the next section. If one or more of the distance estimates are of poor quality, then the signals around the base stations may not intersect at all, in which case the methods does not produce a location estimate at all. The approach also breaks down completely if signals from less than three base stations are observable. In this paper we use these three heterogenous but com-
plementary technologies along with a sophisticated fusion algorithm for an enhanced navigation system in terms of positioning performance. The system is adapted to first responder needs, thus different types of first responder activities and operational conditions were examined and classified according to extracted qualitative attributes. The vertical acceleration data, which indicates the periodic vibration during gait cycle, is used to evaluate the accuracy of the inertial based navigation subsystem. The amount of strong feature correspondences assess the quality of the three-dimensional scene knowledge from digital camera feedback. Finally, the qualitative attribute, in order to evaluate the efficiency of the radio frequency identification subsystem, is the degree of probability of each location estimate. Fuzzy if-then rules are then applied to these three attributes in order to carry out the fusion task.
accuracy of an RFID system. Some of the radio attenuation factors that can affect the transmission of radio signals inside a building are humidity, smoke, high temperature and thermal layers that could reflect or refract radio waves. What is more, when the building suffers from inside collapses the accuracy of the RFID system is even more deteriorated. All the aforementioned first responder activities and operational conditions make their indoor navigation a very demanding and challenging task. In our implementation we have first examined all these conditions separately trying to estimate and evaluate qualitative characteristics for optimal subsystem operation. Subsequently, the multi-sensor architecture will exploit these attributes in order to carry out the fusion task. III. P ROPOSED I NDOOR NAVIGATION S YSTEM A. Inertial Sensor Subsystem
II. T HE F IRST R ESPONDER C ASE The typical operation and activity of a first responder can not be characterized by routine occurrences and scenarios. A first responder must respond to incidents and events, the cause, severity, and consequences of which are not a priori defined. Furthermore, these incidents rarely occur at predetermined places and structures. Operations inside critical infrastructures can conceal high risks with potential life threatening situations. Indoor navigation data could significantly improve the safety of the responders operating in such critical infrastructures [9]. However, indoor navigation of a first responder is very demanding and challenging task since the conditions and movement types are described by high complexity and high variety. Thus, first responder indoor navigation systems must be more sophisticated than conventional pedestrian navigation systems [10]. The first challenge corresponds to the variety of the movements that the first responder executes. Apart from the typical walking movement other types of movements are executed from first responders, such going up or down stairs, crawling, running and climbing. The accuracy of conventional pedestrian navigation systems degrades gracefully with such extreme modes of legged locomotion. Real-time simultaneous localization and mapping using a single camera has been extensively used in robot navigation [11]. Several approaches have been also proposed for pedestrian localization using mounted digital cameras [12], [13]. The challenging task using this approach is the environmental conditions which are different of those where robots operate. Smoke, fire or low lightning conditions are very often in operational scenarios of first responders. Thus, stand alone camera navigation systems have been proved to be inefficient in such cases. The number of RFID tags in critical infrastructures constitutes the trade-off between practicality and desired localization resolution. Furthermore, their placement has been identified as one of the most critical issues for the functionality survival of the localization system in such environments [14]. However, even in an ideal RFID placement and resolution many factors that are met in first responder events can deteriorate the
In the proposed system the inertial sensor is placed on a foot in order to exploit the zero velocity updates. When the foot comes to rest, zero velocity updates reduce the time window of INS predictions to less than a second, ultimately leading to significantly improved navigation performance. However, the accuracy of the PDR system degrades gracefully with extreme modes of legged locomotion, such as running, jumping, and climbing, since the foot rest cannot be determined efficiently. For automatic detection of the foot rest we analyze the signature of the accelerometer signals. Since the sensor is placed on a foot, the accelerometer signature demonstrates specific repeatability corresponding to each phase of the gait cycle. When the magnitude of the 3D acceleration sensed by the accelerometer is close to the earth’s gravity we identify a foot rest pattern. The qualitative criteria of the system is the accuracy of the detected walking pattern. If the detected pattern meets the pre-defined walking pattern we have a high accuracy in the INS location estimates [15]. In the other modes of legged locomotion the accuracy of the subsystem is deteriorated affecting also the qualitative features. B. Structure from Motion Subsystem The method used in our system belongs to sequential algorithms where the parameters are recovered progressively as new views become available and consists of the following steps: • Extract and track feature points through the image sequence. A fast, invariant and robust feature description framework that consists of a detector and a descriptor [16] is used together with a correlation-based tracker, similar to the KLT tracker [17]. • Eliminate outliers using the constraints imposed by the epipolar geometry. For sequences where the matching of the individual frames is difficult, for instance due to a wide baseline stereo or excessive noise, the RANSAC paradigm is used to eliminate the outliers. • Recover the structure and motion parameters using the factorization scheme [4], achieving an initial estimation of Euclidean structure and motion parameters.
TABLE I E XTRACTED RULES FOR THE PROPORTIONAL PERFORMANCE FOR EACH SUBSYTEM SEPARATELY AND FOR THE OVERALL QUALITY MEASURE . T HE THREE INPUTS , RIN S : INERTIAL SUBSYSTEM , RSf M : STRUCTURE FROM MOTION SUBSYSTEM , RRF ID : RADIO - FREQUENCY IDENTIFICATION SUBSYSTEM . T HE FOUR OUTPUTS , WIN S : INERTIAL SUBSYSTEM WEIGHT, WSf M : STRUCTURE FROM MOTION SUBSYSTEM WEIGHT, WRF ID : RADIO - FREQUENCY IDENTIFICATION SUBSYSTEM WEIGHT AND W : OVERALL LOCALIZATION TECHNIQUE WEIGHT.
If RIN S is
and RSf M is
and RRF ID is
then WIN S is,
then WSf M is,
then WRF ID is,
then W is
Low Low Low Low Low Low Low Low Low ... High High High
Low Low Low Medium Medium Medium High High High ... High High High
Low Medium High Low Medium High Low Medium High ... Low Medium High
Low Low Low Low Low Low Low Low Low ... High High High
Low Low Low Medium Medium Medium High High High ... High High High
Low Medium High Low Medium High Low Medium High ... Low Medium High
Very Low Very Low Low Very Low Medium Medium Medium Medium High ... High Very High Very High
Refinement of Euclidean structure and motion using bundle adjustment. The qualitative criteria, which will determine the efficiency of this susbsystem, is the number of extracted strong descriptors used in the image sequence. Since the geometric constraints among the different views are tightly coupled with the number of extracted feature correspondences, this metric can be defined as a fine quality measure. •
C. Radio-Frequency Identification Subsystem In order to overcome the RFID aforementioned problems, we used a probabilistic positioning framework, where the world is taken to be probabilistic, not deterministic, accepting the fact that the measured signals are inherently noisy. The formula used is an example of an application of a mathematical theorem known as the Bayes rule. Based on probability theory, the theorem gives a formal way to quantify uncertainty, and it defines a rule for refining a hypothesis by factoring in additional evidence and background information, and leads to a number representing the degree of probability that the hypothesis of the location estimate is true. The probabilistic model which assigns the probability for each possible location L given observations O consisting of the RSSI of each channel is the following: P (L|O) =
P (O|L) × P (L) P (O)
(1)
where P (O|L) is the conditional probability of obtaining observations O at location L, P (L) is the prior probability of location L and P (O) is is a normalizing constant. The final location in such techniques is strictly defined by the highest calculated probability. However, even in cases with poor quality signals and calculations, the location system must choose a final position based on the overall highest probability. The fact that the chosen location is defined by
the highest probability among possible low overall estimations is transparent to the user, however it identifies a higher uncertainty. This leads to the conclusion that the measure of the calculated probability can be used as a metric to the overall accuracy of the positioning RFID method. IV. F UZZY M ULTI -S ENSOR A RCHITECTURE We first define as RIN S , RSf M and RRF ID , the qualitative measures of inertial sensor, structure from motion and radiofrequency identification subsystems, respectively. In order to derive the weight which will define the proportional contribution of current location estimate, we present the qualitative components as three separate inputs to a fuzzy (Mamdani type) inference system. The system consists of three Triangular Membership Functions (TMFs) for all the input components, three for the three subsystem output weights and five for the overall system weight output. The first input variable is the RIN S which indicates the similarity of the pre-defined walking pattern from the estimated one and ranges from 0 to 100. The input variable RSf M denotes the number of used features used in the image sequences, and its value ranges between 0 and 128. The input variable RRF ID is the highest probability extracted by the probabilistic positioning technique of the RFID subsystem and ranges from 0 and 1. The parameterization was developed in a way that only two membership functions will overlap for any input variable. The overlapping TMFs are tightly coupled with the overall timing performance since a three function overlap would make the design more complicated and time demanding. The defuzzification method utilized is that of the centroid. The three inputs are cross connected to the output through a set of 27 if-then rules as presented in Table I. The linguistic rule premises were attained after extensive comparisons with results from straight forward calculations.
The current localization estimation in P DR(t) is given by P P DRsubsystem (t) × Wsubsystem (t) P P DR(t) = (2) Wsubsystem (t) for all subsystem ∈ {‘IN S’,‘Sf M ’,‘RF ID’}. The final localization technique weight W represents the efficiency of the overall location estimates and determines the accuracy of the estimated location in each time state t. A proportional weight of the previous location estimation is used in order to maintain the consistency of the position, when the current state estimation suffers from high uncertainty. The formula used is the following: F P DR(t) = W × P DR(t) + (1 − W ) × F DP R(t − 1) (3) where t represents the current time state and 0 < W < 1. V. E XPERIMENTAL R ESULTS Preliminary experimental results have shown a better navigation effectiveness and lower positioning errors. Fusing INS information, SfM information and RFID measurements has several advantages. First, it improves the performance of the whole system in terms of positioning, and secondly it allows accurate estimation when one or two subsystems are unavailable due to special conditions and first responder activities. VI. C ONCLUSION In this paper we used three heterogenous but complementary technologies along with a sophisticated fusion algorithm which led to an enhanced system in terms of positioning performance. The system is adapted to first responder needs, thus different types of first responder activities and operational conditions were examined and classified according to extracted qualitative attributes. In order to derive a proportional contribution of each navigation subsystem, we present the calculated qualitative components to a fuzzy inference system. The final location estimate is calculated by applying different weights to each subsystem coordinates depending on their current time state performance quality. Preliminary experiments have shown a better navigation effectiveness and lower positioning error compared with the used stand alone navigation systems. ACKNOWLEDGMENT This work was supported by the E.C. under the FP7 research project for Innovative and Novel First Responders Applications, “INFRA”, ICT-SEC-2007-1.0-04. R EFERENCES [1] E. Foxlin, “Pedestrian tracking with shoe-mounted inertial sensors,” IEEE Computer Graphics and Applications, vol. 25, no. 6, pp. 38–46, 2005. [2] S. Beauregard, “Omnidirectional Pedestrian Navigation for First Responders,” in 4th Workshop on Positioning, Navigation and Communication, 2007, pp. 33–36. [3] R. Hartley and A. Zisserman, Multiple view geometry in computer vision. Cambridge Univ Pr, 2003. [4] C. Poelman and T. Kanade, “A paraperspective factorization method for shape and motion recovery,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 3, pp. 206–218, 1997.
[5] H. Aanaes, R. Fisker, K. Astrom, and J. Carstensen, “Robust factorization,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1215–1225, 2002. [6] L. VanGool, M. Pollefeys, and M. Proesmans, “Self calibration and metric reconstruction in spite of varying and unknown internal camera parameters,” in Proc. International Conference on Computer Vision, 1998, pp. 90–95. [7] C. Alippi and G. Vanini, “A RSSI-based and calibrated centralized localization technique for Wireless Sensor Networks,” in Proc. IEEE Int. Conference on Pervasive Computing and Communications Workshops, 2006, pp. 301–306. [8] T. Roos, P. Myllymaki, H. Tirri, P. Misikangas, and J. Sievanen, “A probabilistic approach to WLAN user location estimation,” International Journal of Wireless Information Networks, vol. 9, no. 3, pp. 155–164, 2002. [9] V. Renaudin, O. Yalak, P. Tom´e, and B. Merminod, “Indoor navigation of emergency agents,” European Journal of Navigation, vol. 5, no. 3, pp. 36–45, 2007. [10] S. Sawyer, A. Tapia, L. Pesheck, and J. Davenport, “Mobility and the first responder,” Communications of the ACM, vol. 47, no. 3, pp. 62–65, 2004. [11] A. Davison, “Real-time simultaneous localisation and mapping with a single camera,” in Proc. International Conference on Computer Vision, vol. 2, 2003, pp. 1403–1410. [12] M. Kourogi and T. Kurata, “A method of personal positioning based on sensor data fusion of wearable camera and self-contained sensors,” in Proc. IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, 2003, pp. 287–292. [13] R. Jirawimut, S. Prakoonwit, F. Cecelja, and W. Balachandran, “Visual odometer for pedestrian navigation,” IEEE Transactions on Instrumentation and Measurement, vol. 52, no. 4, pp. 1166–1173, 2003. [14] E. Welbourne, M. Balazinska, G. Borriello, and W. Brunette, “Challenges for pervasive RFID-based infrastructures,” in Proc. International Workshop on Pervasive RFID/NFC Technology and Applications, 2007. [15] M. Kourogi and T. Kurata, “Personal positioning based on walking locomotion analysis with self-contained sensors and a wearable camera,” in Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality, 2003, p. 103. [16] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (SURF),” Computer Vision and Image Understanding, vol. 110, no. 3, pp. 346–359, 2008. [17] S. Baker and I. Matthews, “Lucas-kanade 20 years on: A unifying framework,” International Journal of Computer Vision, vol. 56, no. 3, pp. 221–255, 2004.
