Mixed Initiative Wheelchair Control using 3D sensors. Jesse Hoey† James Tung Alex Mihailidis Intelligent Assistive Technology and Systems Laboratory Department of Occupational Therapy, University of Toronto 500 University Ave, Toronto, CANADA M5G1V7 †
[email protected] Abstract We present a collision avoidance system for powered wheelchairs used by people with cognitive disabilities. Such systems increase mobility and feelings of independence, thereby enabling reversal of some symptoms of depression and cognitive impairment and improvement of quality of life. Three dimensional imaging sensors will allow the wheelchair to model oncoming obstacles and predict imminent collisions. The collision avoidance system will use this obstacle detection to take actions that optimise over the desires of the user (to get somewhere) and the need for safety. We present obstacle detection using two 3D sensors, and discuss a method for performing the control optimisation.
1
Introduction
High quality of life is of the utmost importance and mobility is a key component of a positive quality of life. Unfortunately, many older adults face various impairments and disabilities that result in their mobility being compromised. Furthermore, many of these people lack the strength to manually propel themselves, and require powered wheelchairs. However, powered wheelchairs are not appropriate for older adults with a cognitive impairment, such as dementia or Alzheimer’s disease, as they do not have the cognitive capacity required to effectively and safely maneuver the wheelchair. In addition, their sometimes aggressive and unpredictable behaviour makes wheelchair use unsafe for both themselves and others sharing the environment. Currently there are an estimated 15 to 18 million people worldwide who have been diagnosed with dementia with this number expected to reach 34 million by 2025 [BBB04]. The safe control of a powered wheelchair requires a significant skill level, attention, judgment, and appropriate behaviour. It not only requires the driver to have the ability to control the wheelchair, but also to be constantly aware of his/her surroundings in order to not crash into surrounding people, objects, and walls. There have been many reported incidents of older adults being injured as a result of a powered wheelchair being driven into them, or into their cane or walker, resulting in a fall to occur. As a result, many care institutions (e.g. hospitals, long-term care facilities, etc.) have implemented policies that restrict driving for reasons of safety, especially for residents with cognitive impairment. 1
(a) Bumblebee
(b) Canesta
Figure 1: 3D imaging sensors (a) stereo camera (b) time-of-flight laser
Reduced mobility results in reduced quality of life. The combination of social isolation, limited life space and choice, learned dependence (e.g. requiring someone to push a manual wheelchair), frustration, and limited autonomy likely contributes to symptoms of depression and exacerbation of cognitive impairment and undesirable behaviours. It should also be noted that this chain reaction of symptoms resulting from reduced mobility are also observed in other patient groups beyond older adults (e.g. disabled children, adults with traumatic brain injury, etc.), thus broadening the scope of these problems and requirements from potential solutions. If we can provide these users with some level of independence, irrespective of ability, without placing the person or others at unreasonable risk, then it may be possible to reverse some symptoms of depression and cognitive impairment and improve quality of life. The goal of this project is the novel application of 3D imaging sensors to adapt a powered wheelchair, specifically, the Nimble RocketT M so that it can be driven safely by users with cognitive and other complex impairments. This will be accomplished through the design and development of an intelligent collision avoidance system. We will use two different 3D imaging sensors. The Bumblebee stereo vision camera from Point Grey Research 1 and the time-of-flight infrared laser sensor from Canesta 2 . These sensors can be seen in Figure 1. A sketch of the Canesta 3D sensor mounted on the wheelchair is shown in Figure 2. There is a large literature on obstacle modeling and avoidance for autonomous robots. Overviews are provided in [Thr02, Thr00]. Systems for automating wheelchairs [MS95, Yan98, Yan00, NDVB01] and othotic devices [MDK+ 03, UB01] Some examples of using robots in an elder care situation are describe in [PMP+ 02, RGT03, MPR+ 02].
2
Design of Collision Avoidance System
The role of a 3D sensor in the collision avoidance system will be to rapidly and robustly detect obstacles when they are in an intermediate range from the wheelchair. Objects that are very far away (> 3m) require less robustness (they pose no imminent danger) and less accuracy (estimates of distance can be refined as the object nears). Very close objects (within 5cm of the outer edge of the wheelchair) can be detected with contact sensors as an emergency response. 1 2
www.ptgrey.com www.canesta.com
Figure 2: Nimble RocketT M wheelchair with Canesta 3D perception sensors.
The critical range is between these two, where evasive or protective action can be taken before contact. The two sensors we have chosen provide good resolution and reliable, fast detection in this range. The following delineates our approach.
2.1
Obstacle Representation
We will use occupancy grids [ME85, ML98], a well known technique for modeling obstacles in mobile robotic applications. The idea is to have an estimate of the presence of an obstacle at each 2D location {x, y} in the reference frame of the (moving) wheelchair at each time, t. The occupancy at a location relative to the wheelchair, ~x = {x, y}, will be given by c~x , a binary variable that, if 1 means the cell at ~x is occupied, and if 0 means the cell at ~x is free space. An occupancy grid at some time, t, gives, for each cell in the grid, P (c~x = 1|I1 . . . It ), where Ii i ∈ 1 . . . t is the depth image observed at time i. This probability will be close to 1 if there is an object at ~x. A 2D location relative to the wheelchair, ~x = {x, y}, will contain an obstacle if there is some object in the column of space at {x, y} extending from the ceiling to the floor3 . To compute this, the depth image must be converted to a 2D map of the obstacles in the scene along the horizontal plane of the floor. This process is shown in Figure 3. The 3D sensors view a scene, as in Figure 3(a), and output a depth image, as shown in Figure 3(b), which gives the distance (or some parameter from which distance can be estimated) from the camera to each location in the scene (or, equivalently, pixel in the image). The sensors will typically only output distances for pixels at which it has valid depth information. Other pixels will be marked as invalid. For example, in the stereo depth map in Figure 3(b), the invalid 3
Occupancy grids can be generalized to 3D occupancy. However, this would be necessary only in the case of objects that are located vertically in such a way as to pose no impediment to a wheelchair. We don’t expect these kinds of obstacles, and proceed to discuss the 2D case only.
(a)
(b)
(c)
(d)
Figure 3: Constructing a radial map from a depth image. An image (a), its depth image, (b) is projected onto the ground to give a radial map (c). Rays are then computed from the sensor position to give the radial map (d).
pixels are marked in black. Figure 3(c) then shows the projection of Figure 3(b) onto the ground plane. The floor pixels are removed before this projection using known geometry of the horizontal floor plane from the height of the camera from the floor. The final step is to relate the image column in Figure 3(c) to an actual 2D position in the real world using ray tracing from the camera center, producing the radial map shown in Figure 3(d). The axes of the radial map correspond to the actual 2D floor plane, and the values in the radial map can be interpreted as the (binary) occupancy given the current image, if the location ~x is occupied (black) 1 P (c~x |It ) = 0.5 if the location ~x is unknown(gray) 0 if the location ~x is clear (white) The occupancy at c~x is then computed by integrating these estimates over time. However, since the wheelchair is moving, we must use some estimates of its movement, from odometry sensors in the wheels, or from the control signal from the joystick controller. Then, we can temporally integrate the occupancy grid using the following formula: P (c~x = 1|I1 . . . It ) = (1 − α)P (c~x |It ) + αP (c~x−δx ~ = 1|I1 . . . It−1 )
(1)
~ = {δx, δy} is the change in the sensor’s position as given by the wheelchair odomewhere δx try, P (c~x−δx ~ = 1|I1 . . . It−1 ) is the old occupancy grid map, and α ∈ [0, 1] is the rate at which
older readings are discarded. This learning rate is necessary as the odometry will eventually cause drift in the cell’s position relative to the wheelchair, and so older measurements must be discarded. Typically, a α will be close to 0, as we want to rely more heavily on recent measurements. The method used in [ML98], and which is currently implemented in our system, is computationally simpler, and does not make use of odometry readings. It maintains a set of grid values, G(i), for each cell i in the occupancy map. It then updates each cell by a constant +K if the cell is in the occupied region of a radial map, and by −K if its in the clear region.
2.2 Obstacle Avoidance The collision avoidance system needs a control strategy for responding to obstacles. We will ˚ use a partially observable Markov decision process, or POMDP, to encode this strategy [A65, SS73, KLC96]. POMDPs (and their fully observable counterparts, MDPs) have become the semantic model of choice for decision-theoretic planning in the AI planning community. An MDP models the domain of interest with a finite set of (fully observable) states S. Actions of the system (wheelchair in this case), are drawn from a finite set A, and induce stochastic state transitions, with P (St |At , St−1 ) denoting the probability with which state St ∈ S is reached when action At ∈ A is executed at state St−1 ∈ S. The state is not fully observable, but can be inferred from some observations, O, through some observation function P (O|s). A real-valued reward function R, associates with each state, s, and action, a, its immediate utility R(s, a). Efficient algorithms have been developed to solve such models, generating a control policy π(S) which gives the optimal action to take in each possible state [ZZ01, PGT03, HP05]. The wheelchair application will be modeled as an POMDP by making the set of states be the possible configurations of obstacles in the path of the wheelchair, as given by the occupancy grid (Equation 1). The actions will be those the wheelchair can take (e.g. go forward, stop, or turn right). The reward function will assign a large penalty for the wheelchair coming into contact with any occupied cell, but a positive reward for the wheelchair responding to the user’s command (e.g. from a joystick). The control policy will therefore optimise satisfaction of user’s needs for mobility (their control commands) with the requirements for safety in the wheelchair’s location. As shown in Figure 4, the model can also take into account other task variables, describing other aspects of the patient’s current state, and attitude variables, describing their current mental or cognitive state (e.g. frustrated, happy, independent, etc.). The continuous nature of parts of the state space may call for a continuous state POMDP, or perhaps we can use some ideas about visibility envelopes for control [KvdP04].
3 Results Figure 5 shows an example collision detection using the Bumblebee stereo vision camera mounted on one side of the wheelchair. The first column (a) shows the view from the camera, the second (b) shows a view from an independent camera, the third (c) shows the stereo image, and the last shows the occupancy grid. The control policy here simply issues a stop command when there is an obstacle detected within some threshold (here set to 60cm).
SYSTEM CONTROL user
user control
control
attitude
attitude
R
WHEEL CHAIR
task
task
map
map
3−D SENSOR t−1
3−D SENSOR
WHEEL CHAIR t
Figure 4: POMDP for mapping and control
References ˚ ˚ om. Optimal control of Markov decision processes with incomplete state estimation. Journal of [A65] K. J. Astr¨ Mathematical Analysis and Applications, 10:174–205, 1965. [BBB04] J. Bates, J. Boote, and C. Beverly. Psychosocial interventions for people with a dementing illness: A systematic review. Journal of Advanced Nursing, 45(6):644–658, 2004. [HP05] Jesse Hoey and Pascal Poupart. Solving POMDPs with continuous or large discrete observation spaces. In Proc. Intl. Joint Conf. on Artificial Intelligence, Edinburgh, July 2005. to appear. [KLC96] Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Technical Report CS-96-08, Dept. of Computer Science, Brown University, February 1996. [KvdP04] Maciej Kalisiak and Michiel van de Panne. Approximate safety enforcement using computed viability envelopes. In IEEE Int. Conference on Robotics and Automation, volume 5, pages 4289–4294, 2004. [MDK+ 03] A. Morris, R. Donamukkala, A. Kapuria, A. Steinfeld, J. Matthews, J. Dunbar-Jacobs, and S. Thrun. A robotic walker that provides guidance. In IEEE International Conference on Robotics and Automation (ICRA), Taipei, Taiwan, 2003. [ME85] H. Moravec and A. Elfes. High-resolution maps from wide-angle sonar. In Proc. IEEE Int’l Conf. on Robotics and Automation, St. Louis, Missouri, March 1985. [ML98] D. Murray and J. Little. Using real-time stereo vision for mobile robot navigation. In Proceedings of the IEEE Workshop on Perception for Mobile Agents, Santa Barbara, CA, June 1998. [MPR+ 02] Michael Montemerlo, Joelle Pineau, Nicholas Roy, Sebastian Thrun, and Vandi Verma. Experiences with a mobile robotic guide for the elderly. In Proceedings of the AAAI National Conference on Artificial Intelligence, Edmonton, Canada, 2002. AAAI.
(a)
(b)
(c)
(d)
Figure 5: Example frames 100,170,200 and 240 (a) Sensor view (b) view of scene from another camera (c) disparity map (d) occupancy grid
[MS95] David P. Miller and M.G. Slack. Design and testing of a low-cost robotic wheelchair prototype. Autonomous Robots, 2:77–88, 1995. [NDVB01] Marnix Nuttin, Eric Demeester, Dirk Vanhooydonck, and Hendrik Van Brussel. Shared autonomy for wheel chair control: Attempts to assess the user’s autonomy. In Autonome Mobile Systeme 2001, 17. Fachgespr¨ach, pages 127–133, London, UK, 2001. Springer-Verlag. [PGT03] Joelle Pineau, Geoff Gordon, and Sebastian Thrun. Point-based value iteration: An anytime algorithm for POMDPs. In Proc. of International Joint Conference on Artificial Intelligence, pages 1025–1032, Acapulco, Mexico, August 2003. [PMP+ 02] J. Pineau, M. Montemerlo, M. Pollack, N. Roy, and S. Thrun. Towards robotic assistants in nursing homes: challenges and results. In T. Fong and I. Nourbakhsh, editors, Workshop notes (WS8: Workshop on Robot as Partner: An Exploration of Social Robots), IEEE International Conference on Robots and Systems, Lausanne, Switzerland, 2002. IEEE. [RGT03] N. Roy, G. Gordon, and S. Thrun. Planning under uncertainty for reliable health care robotics. In Proceedings of the International Conference on Field and Service Robotics, Lake Yamanaka, Japan, 2003. [SS73] Richard D. Smallwood and Edward J. Sondik. The optimal control of partially observable markov processes over a finite horizon. Operations Research, 21:1071–1088, 1973. [Thr00] S. Thrun. Probabilistic algorithms in robotics. AI Magazine, 21(4):93–109, 2000. [Thr02] S. Thrun. Robotic mapping: A survey. In G. Lakemeyer and B. Nebel, editors, Exploring Artificial Intelligence in the New Millenium. Morgan Kaufmann, 2002. to appear. [UB01] Iwan Ulrich and Johann Borenstein. The guidecane - applying mobile robot technologies to assist the visually impaired. IEEE Transactions on Systems, Man and Cybernetics - Part A: Systems and Humans, 31(2):131–136, March 2001. [Yan98] Holly A. Yanco. Wheelesley, a robotic wheelchair system: Indoor navigation and user interface. In V.O. Mittal, H.A. Yanco, J. Aronis, and R. Simspon, editors, Lecture Notes in Artificial Intelligence: Assistive Technology and Artificial Intelligence, pages 256–268, 1998. [Yan00] Holly A. Yanco. Shared User-Computer Control of a Robotic Wheelchair System. PhD thesis, Massachusetts Intstitute of Technology, September 2000. [ZZ01] N. Zhang and W. Zhang. Speeding up the convergence of value-iteration in partially observable Markov decision processes. Journal of Artificial Intelligence Research, 14:29–51, 2001.
