Intelligent Robotic Systems in Service of the Disabled - CiteSeerX

IEEE TRANSACTIONS ON REHABILITATION ENGINEERING, VOL. 3, NO. 1, 1995 (IN PRESS)

1

Intelligent Robotic Systems in Service of the Disabled Kazuhiko Kawamura, Sugato Bagchi, Moenes Iskarous, and Magued Bishay

Abstract |We argue that intelligence is necessary in robots used for rehabilitation in order to reduce the amount of mental activity needed by the user of these robots. With this in mind, the areas of research relevant to imparting robotic systems with the capability of assuming a more intelligent role are identi ed. We describe our implementation of functionalities such as fuzzy command interpretation, object recognition, face tracking, and task planning and learning, which are part of the ISAC, an intelligent system designed to feed individuals with physical disabilities.

I. Introduction

Intelligence in a rehabilitation robotic system is not usually stressed because of the presence of an intelligent person (the user) who is expected to make all the decisions requiring intelligence. This expectation has resulted in the development of many rehabilitation robots that provide an interface for the user to issue robot motion commands by means of teleoperation [1]. Although teleoperation may relieve the user of the physical burden of a manipulative task, this is replaced with the mental burden of controlling the robot. Reducing the mental load on the user during the performance of manipulative tasks will help in increasing the acceptance of robotic aids for rehabilitation [2], [3], [4]. \Intelligence" is hard to de ne but easy to perceive. For our purposes, we consider a robot to be intelligent if, in any way, it assumes some of the mental activities that a human would otherwise perform. Consider the simple action of moving a robot's gripper to a desired place. With teleoperation, the user enters the control loop, sensing the distance to the target, the velocity of the gripper, and providing the appropriate control signal to the input device. A great deal of mental load can be removed if a motion controller is present. Currently the task for the user is to compute a numerical distance for movement. This is often hard to do, especially when considering three axes of translational motion and three axes of rotations. Since humans often think in fuzzy terms, additional mental load could be removed if the robotic system could accept fuzzy commands like \move closer" or \move faster." These commands have to be translated into a quantitative distance or velocity. In addition, the target to move towards has to be inferred. To simplify the mental task even further, the user could specify the action instead of having to describe it. Instead of issuing fuzzy or crisp motion commands to move the gripper to a spoon and then commanding the robot to grasp it, the user could directly say \pick up spoon." This re-

quires the robot to use sensors to determine the location of the spoon. Rather than having to be blindly led by the user, the presence of vision and other sensors greatly enhances the robot's autonomous capabilities, with a corresponding reduction of the mental activity on the part of the user. There is scope for further reduction: Instead of specifying actions, the user could specify his intent. Rather than having to specify actions like \pick up spoon," \dip into bowl," \bring to my mouth," the user could simply say \feed me soup." The necessary actions that need to be performed will be generated by the system through a planning process. For planning, the causes and eects of actions must be known to the system. This can either be known or taught. The ability to learn imparts additional intelligence to the system, allowing the user to extend the activities for which the robot was designed. This paper looks at how far rehabilitative robots have come in possessing abilities that relieve the user from the mental burden of controlling the robot. In the context of man-machine interfaces, this is our de nition of intelligence. The next section describes various rehabilitative robots and the tasks they perform. Section III describes our implementation of intelligent capabilities, such as fuzzy command interpretation, object recognition, face tracking, task planning and learning. These are part of the ISAC (Intelligent Soft Arm Control) system | a robotic aid for feeding individuals with physical disabilities [5], [6], [7], [8]. II. Existing Systems

Leifer et al. identi ed capabilities necessary for various rehabilitative tasks. These capabilities include command, control, dialog, planning, grasp, manipulation, re ex and mobility. The authors also identi ed dierent tasks such as food preparation, personal hygiene and reading and the necessary and desirable capabilities for each task. Several robotic systems have been developed to include some of these capabilities. In the area of food service, HANDY 1, developed at the University of Keele, UK [1], [4], helps severely disabled people feed themselves. Other features are now being added to provide functions such as drinking, shaving and teeth cleaning. In Japan, a Meal Assistance Robot System was developed at SECOM's Intelligent Systems Laboratory [9]. This system uses a specially designed arm that is simple, light and small. The hand consists of a spoon and spatula which increases its ability to grasp the food and serve it. The user manipulates the arm pointing an optical pointer mounted on his head towards a sensor The authors are with the Center for Intelligent Systems, Box 1804, panel. Station B, Vanderbilt University, Nashville, TN 37235, U.S.A. E-mail: fkawamura,bagchi,moenes,[email protected]. In the area of computer usage and oce assistance, De-

2

IEEE TRANSACTIONS ON REHABILITATION ENGINEERING, VOL. 3, NO. 1, 1995 (IN PRESS)

VAR (Desktop Vocational Assistant Robot) is a commercial system that emerged from the work at Stanford University and the Palo Alto VA Medical Center [10], [11]. The system uses a PUMA-260 robot which runs inverted on a four-inch motorized overhead transverse track. It uses a modi ed Otto-Bock Greifer prosthetic hand with a nger position sensor for servo control. It can be used to handle paper, oppy disks, pick up and use the telephone, and retrieve medication. A voice recognition system is used to command the arm. For safety, the robot stops and shuts itself o if the user says \stop", shouts, presses a panic switch, or if it encounters a resistance of 2.3 Kg or more [12]. RAID (Robot for Assisting the Integration of the Disabled) is a system that is developed under the European Community TIDE program [1], [13]. It uses a modi ed RTX robot in the SCARA-con guration. The system consists of a PC, a fax machine, a telephone, a printer, and a robot manipulator operated from a joy stick mounted on the user's wheelchair. Teleoperation, dexterity and safety are important requirements in such systems. Wheelchair-mounted systems are also used in rehabilitative tasks. MANUS is most well known among this class [1]. It is an electrically powered arm with its motors and motor controllers mounted in the base of the arm. Individually selectable controls and recon gurable microcomputerassisted procedures are used to control both gripper and wheelchair [14], [15], [3]. A therapist can con gure the MANUS to suit individual needs by means of a personal computer. It is intended to operate in an unstructured environment of which it has little or no prior knowledge. Thus, much emphasis is placed on the interactive procedures in which the user directly controls the gripper. The MANUS arm is made to fold away next to the armrest of the wheelchair to be able to pass through the normal doorways [12]. For a robotic aid system to be useful to the disabled or the elderly, the user must be able to communicate with the system in a natural way. Lees and Leifer [16] developed an environment for programming robots using graphical symbols called RoboGlyph. A program to perform a speci c task is assembled by dragging a sequence of symbols that represent the robot positions and motions onto a storyboard. This system has been developed to be used with DeVAR. RoboGlyph is used to position the robot in the vicinity of an object, then force feedback is used by RoboGlyph to determine the object orientation and facilitate its manipulation. Figure 1 shows two motion storyboards with fragments of RoboGlyph programs. The top row shows how to open a door by positioning the robot and using the exploration primitive to locate the door. The second row shows an example of a straight line motion with the constant end-eector orientation [16]. Ultimately, the system should be able to understand the user's intentions in order to successfully perform the required task. Sato et al. [17] proposed a system for the active understanding of human intentions by monitoring human behavior. The authors constructed a micro-

Fig. 1. Fragments of RoboGlyph programs.

teleoperation robot which can understand an operator's intention through the unconscious behavior of touching a desk with the hand while writing. The system was used to change the control mode of a master-slave manipulation system from ne to rough motion depending on the position of the operator's hand with respect to the desk. Bell at al. [18] developed a shared control method for the NavChair assistive navigation system. The NavChair allows the user to move the wheelchair while overriding unsafe maneuvers. Adaptive shared control methods were also investigated for autonomous mode selection based on observations of user behavior. The authors hypothesized that corrections in response to applied disturbances can be used to qualitatively model the user. They designed an experiment to demonstrate how reactive human models can be used to detect human adaptation in real time. The experiment used a simulated wheelchair viewed from above in a world of walls and users were asked to drive them through a series of hallways and rooms as quickly as possible. Joystick input from the user and disturbances applied to the path were recorded and an autoregressive model was identi ed to relate both of them together. This model was then used to predict the behavior of the user [18]. III. Research Areas for Intelligent Rehabilitation Robots

This section describes areas of research that we consider to be important for intelligent robots, using the ISAC system to illustrate how these areas can be applied to a rehabilitation robot. We have implemented modules for fuzzy command interface, object recognition, 3D face tracking, and task planning and learning, which are all part of the ISAC system shown in Figure 2. Figure 3 shows the software and hardware architecture of ISAC. A. Fuzzy Command Interface

Humans often communicate with fuzzy terms. When a person asks another person to \increase the temperature by a little bit," this command is understood within the context and interpreted based on the person's previous experience to determine the action needed. In an intelligent user interface, the same kind of communication is needed to provide a natural environment for the user. Within this framework, two main issues need to be addressed. The rst issue is the context, which is used to determine what action is to be taken. In the previous example, which temperature should be increased? Is it the room temperature, the oven

KAWAMURA ET AL.: INTELLIGENT ROBOTIC SYSTEMS IN SERVICE OF THE DISABLED

3

Fig. 2. ISAC: An intelligent robotic aid for feeding.

Blackboard BB

BB

BB

Macro Action Builder

BB

Arm

Arm

Emergency stop

BB

Object Recognition

3-D Face Tracking

Motion Execution Arm

Reflex action

Voice System

Arm Server

Arm

Hardware

Audio Output

BB

User Interface

CCD

BB

Task Learning System

BB

Software

BB

Fuzzy Command Interpreter

CCD CCD

Parallel Controller

Soft Arm

Fuzzy Processor

Fig. 3. Software and hardware modules in the ISAC system

4

IEEE TRANSACTIONS ON REHABILITATION ENGINEERING, VOL. 3, NO. 1, 1995 (IN PRESS)

Medium

Small

Very small

Very Large

Large

Medium

Small

Very Small

Very Large

Output

Inputs

1

µ (offset)

µ (position)

1

10

20

30

40

50

60

70 position

mm

20

40

60

80

100

offset mm

Fig. 4. Input Fuzzy Membership Functions

temperature, or something else? In this case, the current context determines the action type. The second issue to be addressed is how to translate a fuzzy term into a numerical quantity. In the previous example, the expression \a little bit" is fuzzy and actual amount by which to increase the temperature depends on the experience of the user. Fuzzy sets oer a framework to use human knowledge in understanding fuzzy commands. In the ISAC system, a fuzzy command interpreter is implemented to provide the user with a natural way to command the system [19]. A user can issue a command such as move closer. To translate that command into a numerical motion command for the robot controller, the interpreter has to determine the context (e.g., move closer to what?) and the distance (move closer by how much?). The blackboard keeps track of the current context. This is done from the commands posted into the blackboard. For example, if the previous command was \track face," then the context is the face and the destination is towards the face. Once the context is known, the distance between the robot's gripper and the face is determined. This information is used to calculate the distance by which the gripper must travel. The fuzzy command interpreter has two inputs. The rst input is the command issued by the user such as \move closer," \move a little closer," or \move away." The second input is the context-dependent oset between the gripper and the object. As a result of fuzzy command interpretation, the gripper's motion depends on the task context and its position for the same command issued by the user. This allows a more natural interaction. Figure 4 shows the membership functions de ned for this oset. A rule base is used to infer the output membership functions which are then defuzzi ed to generate the next position of the gripper. Figure 5 shows the output position membership functions. The following are examples of the rules used by the fuzzy command interpreter: RULE 1: IF (command is closer) AND (oset is small) THEN (position is small) RULE 2: IF (command is little closer) AND (oset is small) THEN (position is very small)

Fig. 5. Output Fuzzy Membership Functions

B. 2-D Object Recognition

One of the requirements of the ISAC system is to take a snapshot of the table and return the location of all recognizable utensils. This information is then used to generate a plan for a command such as \feed me soup." The requirements for the recognition algorithm are as follows: Size independence: Recognition should be based only upon the shape of the object and not its size. Robustness: Slight variations in shape, as is common between various instances of the same type of utensil, should not aect the recognition. Short processing time: Since the user is waiting for a task to be performed, the algorithm should not take a long time. Orientation independence: The object should be recognized at any orientation. The orientation must also be calculated along with location. Extensibility: The recognition mechanism should be able to accommodate new object models without changing the source code. These requirements were ful lled by making some assumptions about the environment. First, 2D recognition is performed from an image taken from a monochrome CCD camera above the table. Second, it is assumed that the objects can be segmented out of the table background after binarizing the image using a common threshold value. To avoid the eects of shadows, we are using a dark tabletop with light colored utensils. Using the above assumption, the recognition algorithm works on segmented binary images. For each segmented object, its centroid is rst computed. Next, a distance histogram of the object is obtained. This is done by counting the number of object pixels along the circumference of circles of increasing radius, all centered on the centroid. This is shown in Figure 6. This histogram is independent of the orientation of the object.1 It is then normalized to make it size independent as well. The normalized histogram can then be stored as a model of the object (during the model building phase) or compared with existing models (during the recognition phase). The comparison of an object's histogram with model histograms are performed using the least mean square ap1 The original image is corrected for the 5:4 aspect ratio of the CCD elements.

KAWAMURA ET AL.: INTELLIGENT ROBOTIC SYSTEMS IN SERVICE OF THE DISABLED X right

5 D

X left Ymax

Number of pixels

Distance Histogram

X max

X min Yleft Yright

+

Distance from centroid

Fig. 6. Computation of the Distance Histogram

proach. We have 5 models: fork, spoon, cup, bowl, and napkin (of a xed shape). On the average, it takes 1 15 seconds for one object on the table, 1 33 seconds for two, and 1 54 seconds for three. All tests were performed on a Sun SPARCstation 2 with the Androx imaging board. A potential problem with the least mean square approach is that recognition time is proportional to the number of models in the database. To avoid this problem we tried multivariate discriminant analysis. This suered from the fact that this is a linear approach. The \distance" between a fork and a spoon was too small compared to that between a fork and a bowl. A nonlinear method such as a neural network is more suitable. We have trained a neural network which uses the distance histogram as input and has one output node for each recognizable object. Once the network is trained, this is a faster method as it is independent of the number of models. However, the addition of a new model requires retraining. Once the objects in the image are recognized, their 2D position and orientation on the table are returned. These are used by the planner to generate motion commands for the robot. :

:

:

C. Face Tracking

Face tracking is necessary in order to position the food accurately in front of the user's mouth. In addition, it is also used for detecting sudden head motion that may result in a collision with the arm. The requirements for the face tracking algorithm are as follows: 3-Dimensional: The face must be tracked along all three axes in order to position the food in front of the user. Tracking along the axis perpendicular to the image plane is also necessary for detecting forward head movements. Speed: Speed is necessary for two reasons: For detecting sudden motion well before a collision occurs, and for reducing the search space. Assuming an upper limit on the user's velocity, the slower the sampling time, the larger the area to be searched, further slowing down the tracking. Accuracy: The accuracy need not be too high because the user is expected to be able to move his head (hence the need for tracking) as long as the food is reasonably

Fig. 7. Two One-Dimensional Tracking

close to his mouth.

Background sensitivity: The algorithm should work for

any background. Some of these requirements have been ful lled at the cost of others. Initially, we implemented a correlation-based tracking mechanism. This required the acquisition of an initial model of the tracked area, which was then searched for in subsequent images. The correlation-based method suered from two major drawbacks. The tracking speed was very slow, requiring search to be limited to a small area. If the user moved out of the area very quickly, the tracking would fail. Second, a change in the head orientation would cause the tracking to fail. Among its main advantages was that it was background invariant. In order to circumvent some of the problems of correlation based tracking, a two one-dimensional face tracking algorithm has been developed [20]. This makes the assumption of a white background against which the user's head can be easily segmented using intensity thresholding. Figure 7 shows the image used during tracking. The tracking point is initially selected manually. A point on the forehead is tracked because it will be unaected even when the gripper comes close to the mouth. Unlike the correlation based approach, tracking is not performed by matching the pixellevel properties. Instead, global properties like the position of the segmented head are used. The position along the axis in the previous image is used to nd the position along the axis in the current image by searching a 1-dimensional vertical line. Similarly, the position along the axis of the current image is used to search for the position along the axis in the next image by another 1-dimensional search, this time along a horizontal line. Refer to [20] for complete details. This algorithm is simple enough for fast sampling, even when performed independently on more than one tracking point per image for eliminating noise. In addition, this is x

y

y

x

6

IEEE TRANSACTIONS ON REHABILITATION ENGINEERING, VOL. 3, NO. 1, 1995 (IN PRESS)

performed on images from a stereo camera pair. The disparity between the points in the two images is converted into the position along the dimension perpendicular to the image planes. The current sampling time on a Sun SPARCStation 2 using an Androx imaging board is 12 Hz, even under the loading of other ISAC modules. In order to achieve the desired speed, the following assumptions were made: 1. There should be no other dark object in contact with the contour of the user's head. This is the reason for choosing the hair. The region near the mouth is affected by the presence of the gripper. The tracking succeeds in the presence of background and foreground moving objects, as long as they do not touch the contour of the head in the stereo images. 2. The background has to have a higher intensity (preferably white) than the user (preferably having dark hair). The current tracking system has limitations that requires a background that is separable in the intensity domain from the user's face. At least, if the background contains objects that are segmented to be foreground they should not touch the user's head. Also such a system can not be implemented on a moving camera since the eld of view would contain, at some position of the camera, objects that would be thresholded with the foreground. Currently, we are building our own grabbing board that would perform model-matching-based tracking. The tracking problem has been previously studied by other researchers. Roach and Aggarwal [21] have presented a scheme for tracking rigid convex polyhedra. Their scheme was based on image segmentation which is time{ consuming. Hunt and Sanderson [22] have presented algorithms for visual tracking based on mathematical prediction of the position of the object's centroid. Such algorithms require the computation of the centroid, hence can only track very slow moving objects. Lee and Wohn [23] have used image dierencing techniques. We have experimented with successive image dierencing followed by computation of the center of gravity of the absolute of the difference, but the results were sensitive to head tilts. It was also time consuming to compute the center of gravity. Luo, Mullen, and Wessel [24] have presented a robot conveyer tracking system that uses visual and acoustic sensing. The algorithm does 1-D tracking and assumes the knowledge of the conveyer speed. Sensors, other than vision, would not be appropriate to track the user, since the user is not \emitting" some signal that could be received by acoustical or ultrasonic tracking devices. A compromise has to be decided between the tracking speed and accuracy. Goldenberg et al. [25] used the PIPE real-time machine to perform visual tracking. Allen [26] presented a method based on spatio-temporal ltering [27] which would not satisfy our timing constraints. For accurate tracking, model-based techniques [28] also called sum-of-squared dierences (SSD) optical ow are used. In our system we have implemented a dierent version called sum-of-absolute dierences but it was slower than the user's motion which leads to losing the

tracked area of the user's face. Sudden motion is detected by monitoring for an increase in velocity along a line between the face and the arm. Unfortunately, this is the axis perpendicular to the image plane. The resolution along this axis is not very good. It can be increased by moving the cameras closer to the user, but at the cost of reducing the visible area covered by both cameras. A more accurate method for tracking the user's position along this axis is by the use of a ranging device. We have mounted an ultrasonic sensor on the gripper. As the arm is directed towards the face of the user using the information from the cameras, the sonar returns the distance between the arm and the face. This has a much higher accuracy. The presence of the sonar also allows us to use only a single camera for tracking. Face tracking by fusion of information from the camera and the sonar is currently under implementation. D. Task Planning and Learning

Using a robot as a teleoperated device can soon become very exhausting, especially if it has to perform repeated tasks such as feeding. ISAC provides the ability to combine a set of commands into an action macro. User-de ned actions can be used for forming more complex actions. These macros describe how to perform a certain action. It still remains the responsibility of the user to specify when to perform it, or what sequence of actions to take in order to achieve a task. This section describes how these can be learned in order for the system to generate a plan of actions for a given task. The learning environment in ISAC makes use of the permanent presence of the user. It is currently implemented in a simulated environment where the task learning and planning algorithms are being tested. Initially, the system only knows how to perform an action (this can be taught as a macro) but does not know under what conditions it can be performed or what its eects are. As the user instructs ISAC to perform an action, the conditions in the environment, both before and after its execution are monitored. The correlations between environment conditions and the action performed are used to form the action's preconditions and eects. When this is learned, the system has the ability to plan. To describe this knowledge acquisition process in detail it is necessary to describe how the knowledge is represented. D.1 Representation The domain can be represented as a set of propositions C=[1 2 N ]. Each proposition i 2 C is Boolean in nature, i.e., it can be true or false. The agent has a set of actions [ 1 2 M ] that it can perform. Figure 8 shows the representation of an action and its associated propositions as nodes of a network. The strength of a link ij , represents the correlation between the connected nodes and . For the link, between an action j and one of its preconditions i , the correlation is de ned c ; c ; : : :; c

c

a ; a ; : : :; a

w

i

j

a

c

KAWAMURA ET AL.: INTELLIGENT ROBOTIC SYSTEMS IN SERVICE OF THE DISABLED s

a

either by obeying instructions from the user or as a result of internal planning, the following statistics can be kept:

s

P(a j c i =true) + P(a j c i =false)

s aj

ci

From action nodes

s

aj was successful c aj

c

ck

Action node c

i = true i = false

success

j

n

o

p

c

P a

P a

c

P a

success j ci = true) ? P (asuccess j ci = false) j j success j ci = true) + P (asuccess j ci = false) : a

( ij = (

nq

P a

w

P

j

(1) This correlation is 1 (?1) if the precondition is required to be true (false) for the action to execute. Intermediate values represent soft constraints , which are preconditions that alter the probability of an action's success, but not to 0 or 1, as the essential preconditions or hard constraints do. They represent preconditions that are not directly related to an action, but are derived from or related to an essential precondition of the action that cannot be sensed. An example of a soft constraint for the spoon pickup action is the boolean proposition table-disturbed. When this is true, the probability of the action's success is diminished, but not down to zero, because in spite of the disturbance, the spoon may not have moved. The link strength jk, between an action j and one of its eect propositions k is de ned as follows:  ( k = true j executed ) if j ! ( k = true) j = jk ? ( k = false j executed ) if j ! ( k = false) j (2) These are the prior probabilities of the action j 's success, unconditioned on the state of the environment. A negative strength denotes that the expected state of the proposition after the action executes, is false. Based on this representation, the set of actions, their preconditions and eects can be represented as a plan net. The Markov assumption applies in determining the execution of an action: The probability of an action succeeding depends only on the state in which it executes, and not on the sequence of actions that led to this state. D.2 Learning The strengths of the links from preconditions to actions and those from actions to eects represent the knowledge necessary for planning [29], [30]. We now look at how the strengths are learned and adapted for the domain. a

c

w

P c

P c

a

a

a

c

;

a

c

:

a

j

n

no

np

nq

n

no

np

op

p

qp

np

op

p

qp

np

nq

:

:

op

c

j

w

failure

q

P a

aj was executed

Fig. 8. Representation of an action and its preconditions and eects.

as:

a

n o p q , , , and represent the number of times the action was performed under the dierent circumstances mentioned in the table. These numbers provide an estimate of the following likelihood factors used while planning: ( success j i = true) 2 + + + j = 2+ + + (3) ( success j ) ( success j i = false) 2 + + + j = 2+ + + (4) ( success j ) For propositions that do not aect the success or failure of an action, ' , resulting in a likelihood factor of 1. The correlation between ( i = true) and the success of j , given by ij in Equation 1, can be obtained by subtracting Equation 4 from Equation 3. The link strength between an action j and its eect k is de ned in Equation 2. This can be implemented by using the following algorithm, similar to Hebbian learning: 1. Maintain the strength of the link between j and k as jk = correlations executions . 2. Increment executions every time the action j executes. 3. Increment correlations whenever k changes to true from false after j executes. 4. Decrement correlations whenever k changes to false from true after j executes. 5. Keep correlations unchanged when the proposition remains unchanged after j executes. This makes the assumption that all the eects of an action can be sensed between the time the action executes and the next. This is true for most robotic actions involving manipulating objects. Figure 9 shows the convergence curve for learning the preconditions and eects of the action use-spoon. The error axis represents the dierence between the actual and ideal strengths of the links between the action and its preconditions and eects. The initial drop in the error signi es acquisition of qualitative knowledge, i.e., what are the preconditions and eects. The subsequent portion of the curve signi es acquisition of quantitative knowledge, i.e., convergence to the ideal link strengths. c

Proposition nodes x x (preconditions) wjk = P(c k=true a j ) or -P(c k=false a j ) x

a

c

Proposition nodes (effects)

aj

Consider a proposition i as a precondition for the action j . As the agent performs this action a number of times, c

s

P(a j c i =true) - P(a j c i =false)

To action nodes

wij =

7

a

w

a

c

a

w

N

c

=N

N

a

N

c

a

N

c

a

N

a

IV. Conclusions

We have presented some research areas that need to be addressed in order to add intelligence to robotic systems for rehabilitation. Our implementation of some of these functionalities have been described. The integration of these functional modules into one working system has been an important issue. We have adopted a distributed architecture where the modules communicate and coordinate with

8

IEEE TRANSACTIONS ON REHABILITATION ENGINEERING, VOL. 3, NO. 1, 1995 (IN PRESS)

V. Acknowledgements

The authors would like to thank all the people who provided reviews and material for this paper, especially the members of the Intelligent Robotics Laboratory for their fruitful discussions and comments. The ISAC project is partially sponsored by Bridgestone Corporation, Japan.

12

10

Error

8

6

[1] [2]

4

2

[3] 0 0

10

20

30

40 50 60 Number of executions

70

80

90

100

Fig. 9. Error curve for the action use-spoon.

[4] [5]

each other through a \blackboard" (see [31] for details). This allows modules to be added or removed without af- [6] fecting the operation of rest of the system. The ISAC modules have been implemented in C and C++, using TCP/IP as the underlying communication mechanism. [7]

A. Limitations and Challenges

The major limitation of applying the advances made in the research areas described in the previous section is the number and nature of the assumptions made. For example, in ISAC, the color of the utensils on the table must be light and the table top must be dark to help the segmentation process for object recognition. The hair of the user must appear darker than the background for face tracking to work. These may be dicult to ensure in a practical setting, especially when the users are not aware of these limitations. Another limitation is our assumption of reasonable speech capabilities on the part of the user. When this is not the case, many of the functionalities, such as fuzzy commands, will not be available. Research on other means of understanding human intentions [17] are at a preliminary stage. We are currently working on enhancements to the modules described above. The task learning ability has been tested under simulation, but is yet to be integrated with ISAC. The face tracking system is being moved to an active vision system where the two cameras can pan and tilt in order to keep the user in view. The same cameras can also be used for looking at the table for object recognition. The main challenge facing the incorporation of AI into rehabilitation robotics is the relatively modest progress made by AI research when compared to the general belief about the capabilities of an \intelligent" system. However, we believe that providing practical applications in which to ground AI research will help the eld. Rehabilitation robotics is such an application.

[8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18]

References R. D. Jackson, \Robotics and its role in helpingdisabled people", Engineering Science and Educational Journal, Dec. 1993. Lary J. Leifer, \RUI: Factoring the robot user interface", in RESNA International, Toronto, Ontario, Canada, June 1992, pp. 580 { 583. G. Verburg, M. Milner, S. Naumann, J. Bishop, and O. Sas, \An evaluation of the wheelchair-mounted manipulator", in RESNA International, Toronto, Ontario, Canada, June 1992, pp. 602 { 604. Marian Whittaker, \HANDY 1 robotic aid to eating: A study in social impact", in RESNA International, Toronto, Ontario, Canada, June 1992, pp. 589 { 594. Sugato Bagchi and Kazuhiko Kawamura, \An architecture for a distributed object-oriented robotic system", in 1992 IEEE/RSJ International Conference on Intelligent Robots and Systems, Raleigh, NC, USA, July 1992, vol. 2, pp. 711{716. Mourad El-Gamal, Atsushi Kara, Kazuhiko Kawamura, and Mobolaji Fashoro, \Re ex control for an intelligent robotic system", in 1992 IEEE/RSJ International Conference on Intelligent Robots and Systems, Raleigh, NC, USA, July 1992, vol. 2, pp. 1347{1354. Kazuhiko Kawamura, Sugato Bagchi, Moenes Iskarous, Robert Todd Pack, and Ashraf Saad, \An intelligent robotic aid system for human services", in Proceedings of AIAA/NASA Conference on Intelligent Robotics in Field, Factory, Service, and Space, Houston, TX, USA, Mar. 1994, AIAA, vol. 2, pp. 413 { 420. Kazuhiko Kawamura and Moenes Iskarous, \Trends in service robots for the disabled and the elderly", in IEEE/RSJ/GI International Conference on Intelligent Robotics and Systems, 1994, In press. Sumio Ishii, Fumiaki Hiramatsu, Shinji Tanaka, Yasufumi Amari, and Isao Masuda, \A meal assistance robot system for handicapped people's welfare", in Conference on Robots and Mechatronics. Japan Society of Mechanical Engineers, 1991. Joy Hammel, Karyl Hall, David Lees, Larry Leifer, Machiel Van der Loos, Inder Perkash, and Robert Crigler, \Clinical evaluation of a desktop assistant", Journal of Rehabilitation Research and Development, vol. 26, no. 3, pp. 1{16, 1989. Rehabilitation Robotics Newsletter, number 1, Wilmington, DE, USA, 1994. The RehabilitationRobotics Research Program, Applied Science and Engineering Laboratories. Michael Kassler, \Robotics for health care: A review of the literature", Robotica, vol. 11, pp. 495 { 516, 1993. \TIDE: Technology initiative for disabled and elderly people, pilot action synopses", Commission of the European Communities, Brussels, Mar. 1993. H. H. Kwee, M. M. M. Thonnissen, G. B. Cremers, J. J. Duimel, and R. Westgeest, \Con guring the MANUS system", in RESNA International, Toronto, Onatrio, Canada, June 1992, pp. 584 { 587. John W. Bishop, Geb Verburg, Morris Milner, and Martin Mifsud, \A command monitoring system for the MANUS robotic arm", in RESNA International, Toronto, Ontario, Canada, June 1992, pp. 328 { 330. David S. Lees and Larry J. Leifer, \A graphical programming language for robots operating in lightly structured environments", in Proceedings of the 1993 IEEE International Conference on Robotics and Automation, Atlanta, GA, USA, May 1993, vol. 1, pp. 648 { 653. Tomomasa Sato, Yoshifumi Nishida, Junri Ichikawa, and Yotaro Hatamura, \Active understandingof human intention by a robot through monitoring of human behavior", in Proceedings of the 1994 IEEE/RSJ International Conference on Intelligent Robots and Systems, Munich, Germany, Sept. 1994, In Press. David Bell, Simon P. Levine, Yoram Koren, Lincoln Jaros, and Johann Borenstein, \An identi cation technique for adaptive

KAWAMURA ET AL.: INTELLIGENT ROBOTIC SYSTEMS IN SERVICE OF THE DISABLED

[19]

[20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31]

9

Sugato Bagchi is expecting to receive his shared control in human-machine systems", in Proceedings of Ph.D. in electrical and computer engineeringin the Annual International Conference of the IEEE Engineering spring, 1995 from Vanderbilt University where in Medicine and Biology Society, San Diego, California, USA, he is the manager of the Intelligent Robotics Oct. 1993, pp. 1229 { 1300. Laboratory, pursuing his research interests in Kazuhiko Kawamura, Magued Bishay, Sugato Bagchi, Ashraf AI planning and learning systems, robotic sysSaad, Moenes Iskarous, and Masashi Fumoto, \Intelligent user tem integration, and intelligentmanufacturing, interface for a rehabilitation robot", in Fourth International while hoping that this research experience and Conference on Rehabilitation Robotics, Wilmington, DE, 1994, upcoming degree, as well as his previous M.E. pp. 31{35, ASEL, University of Delaware/A.I. duPont Institute. degree in systems science and automation from Rudi Ernst, Mourad El-Gamal, Richard Alan Peters II, and the Indian Institute of Science, Bangalore and Kazuhiko Kawamura, \3-D face tracking in ISAC", Tech. Rep. B.E. degree in electrical engineering from Jadavpur University, CalCIS-93-06, Center for Intelligent Systems, Vanderbilt University, cutta will help him nd a challenging, research-orientedemployment. Nashville, TN, 1993. J.W. Roach and J.K. Aggarwal, \Computer tracking of objects moving is space", IEEE Trasaction PAMI, vol. 1, no. 2, 1979. A.E.Hunt and A.C.Sanderson, \Vision-based predictive tracking of a moving target", Technical Report CMU-RI-TR-82-15, Moenes Iskarous is currently a Ph.D. candiCarnegie Mellon University, The Robotics Institute, 1982. date in the Department of Electrical and ComS.W. Lee and K. Wohn, \Tracking moving objects by a moputer Engineeringat Vanderbilt University. He bile camera", Technical Report MS-CIS-88-97, Department of received the B.S. degree in electrical engineerComputer and Information Science, University of Pennsylvaing from Ain Shams University, Cairo, Egypt nia, November 1988. in 1987 and the M.S. degree in electrical enR.C.Luo, R.E.Mullen Jr., and D.E.Wessel, \An adaptive robotic gineering from Vanderbilt University in 1992. His research interests are in robot control and tracking system using optical ow", in Proceedings of the IEEE International Conference on Robotics and Automation, 1988, neuro-fuzzy control for complex nonlinear systems. He is a student member of the IEEE pp. 568{573. Robotics and Automation Society, the Control R. Goldenberg, W.C.Lau, A. She, and A.M. Waxman, \Progress on the prototypepipe", in Proceedings of the IEEE International Systems Society and the IEEE Computer Society. Conference on Robotics and Automation, 1987, pp. 1267{1274. P.K. Allen, \Real-time motion tracking using spatio-temporal lters", in Proceedings of DARPA Image Understanding Workshop, 1989, pp. 695{701. Magued Bishay received the B.Sc. degree D.J. Heeger, \Depth and ow from motion energy", Science, in Electrical and Computer Engineering from pp. 657{663, 1986. Ain Shams University in Cairo, Egypt, in 1988. P. Anandan, \Measuring visual motion from image sequences", He received the M.S. degree in Electrical EngiTechnical Report COINS-TR-87-21, COINS Department, Unineering from Vanderbilt University in 1992. He versity of Massachusetts, 1987. is currently completing the Ph.D. in computer Sugato Bagchi, Gautam Biswas, and Kazuhiko Kawamura, \A vision. He worked for Schlumberger Overseas, spreadingactivationmechanismfor decision-theoreticplanning", S.A. as a eld engineer from 1988 till 1990. His in AAAI Spring Symposium on Decision-Theoretic Planning, current research interests are in computer viMar. 1994. sion applications that involve control theory. Sugato Bagchi, Gautam Biswas, and Kazuhiko Kawamura, \Generating plans to succeed in uncertain domains", in Proceedings of the Second International Conference on AI Planning Systems, Chicago, IL, June 1994, pp. 1{6, AAAI Press. Sugato Bagchi and Kazuhiko Kawamura, \An architecture of a distributedobject-orientedroboticsystem", in IEEE/RSJ International Conference on Intelligent Robotics and Systems (IROS '92), Raleigh, NC, July 1992, pp. 711{716.

Kazuhiko Kawamura is a Professor of Elec-

trical and Computer Engineering and Management of Technology and Director of the Center for Intelligent Systems at Vanderbilt University. He is the chair of the Technical Committee on Service Robots for the IEEE Robotics and Automation Society. He teaches courses in intelligent robotics, machine intelligence, and technologymanagement. His research interests are in intelligent systems, intelligent robotics, and robotic aid systems. Previously, he was a Lecturer at the University of Michigan, a systems planner at Battelle Memorial Institute, and an Invited Professor at Kyoto University. He holds the B.E. degree from Waseda University in Tokyo, the M.S. degree from the University of California, Berkeley, and the Ph.D. degree from the University of Michigan, Ann Arbor, all in electrical engineering.