I. INTRODUCTION. During the past decade, the interest in safe human robot collaboration (HRC) for applications in industrial production has developed from a ...
Collaborative Behavior Design of Industrial Robots for Multiple Human-Robot Collaboration 2 ! ! Hao Ding , Malte Schipper , Bjorn Matthias
IABB Corporate Research, Germany, {hao.ding, bjoem.matthias}@de.abb.com 2Hochschule Darmstadt, University of Applied Sciences, Germany I(Tel: +49(0)6203 716028, Fax: +49(0)6203 716412)
Abstract-- Industrial robots
are being introduced to assist
human workers in performing assembly tasks such as small parts assembly. A mixed environment is the best choice for certain
assembly
operations,
of which
some
are
better
executed by robots and others are better handled by human workers.
However,
it
is
a
challenge
to
design
the
collaborative behavior of robots to maximize productivity respecting
to
safety
constraints
while
interacting
with
human workers. This becomes more complex for multiple human-robot collaboration. An approach has been proposed
for structuring the collaborative behavior using finite automata
(FSA).
state
In this paper, the approach is extended to
deal with multiple human-robot collaboration, by applying the composition rules of
FSA.
The approach is
finally
demonstrated in an ABB Dual-Arm Concept Robot working with multiple human workers in an industrial assembly scenario.
Index
Terms-Collaborative
behavior,
Finite
state
automata, Multiple human-robot collaboration, Small parts assembly.
I.
INTRODUCTION
During the past decade, the interest in safe human robot collaboration (HRC) for applications in industrial production has developed from a niche research topic [1] to a broad effort [2] encompassing activities from basic research to application profiles and from standardization [3], [4] to biomechanics and ergonomics [5]. The driving force behind the entire effort is the vision that practically relevant industrial scenarios will include both human workers with their specific expertise and robotic production assistants with their characteristic strengths, combining forces to empower a production facility with superior productivity and flexibility [6]. A drawback of present prototypical HRC implementations, be they particularly in industrial environments, is that the basic reaction in the event of an impending risk to a human worker is to stop robot motion, thus interrupting the application and curtailing productivity. In this paper, we address this issue and present an approach that considers productivity of the collaborative application as an objective in its own right, subordinate only to concerns of safety. The paradigm is to avoid unnecessary stops whenever this is possible. Depending on the properties of the robot being used, this may comprise different alternatives. To avoid productivity loss due to such down-times, the situations causing them must be avoided. Solving this by requiring workers to adhere to very rigid behavior
requirements would result in serious difficulties in the acceptance of HRC technology in the production workplace. If, on the other hand, it is the collaborative robot that adapts its behavior appropriately, the acceptance of the technology may actually be furthered. Such adaptivity must be designed to ensure worker safety while simultaneously upholding productivity to the highest extent possible. Advances in safety-rated control of industrial robots have been studied for decades, see some recent work in [8][9][lO][11]. For guaranteeing safety, four collaborative modes are defined so far in the standard [3]. The advent of robotic systems with advanced safety concepts makes scenarios with direct contact between robots and humans possible (physical human-robot interaction, pHRI) [2][12]. Additionally, the balancing between productivity and safety has been studied, e.g., using the architecture proposed in [l3][14], optimal/optimized motion planning with safety constraints [15][16][17], role/task assignment of human and robots [18][19]. However, the systematical way of structuring the collaborative behavior of industrial robots in the collaborative environment has not been subjected to sufficiently rigorous discussion so far. We proposed an approach in [20] using .finite state automata (FSA) to structure the collaborative robot application. In such a way, the handling of situations requiring adaptive behavior proceeds in a way that is roughly analogous to exception handling in modem software applications. A situation in a collaborative operation is analyzed to detennine if either worker safety or productivity can be compromised if the application proceeds as programmed. The approach has also been tested in different scenarios [21][22]. In this paper, the approach is extended for dealing with collaboration with multiple humans and multiple robots. The rules are provided for composing the different FSA. Finally, the approach is demonstrated on an ABB Dual-Arm Concept Robot, also referred to as FRIDA [23][24], working with multiple human workers in an industrial assembly scenario. II.
DESIGN OF COLLABORATIVE ROBOTIC BEHAVIOR
To analyze the collaborative application with regard to safety and productivity, this section introduces terminology and a descriptive formalism. Whenever specificity is required, the application type that we consider is sequential additive assembly in a collaborative setting.
The basic event at the core of our considerations is a safety-related situation in a collaborative robot application that demands a safety-related reaction by the collaborative robot system. This might be due to, for example, inadvertent approach of the worker too close to moving parts of the robot. Present practice in technology demonstrations is to execute an emergency stop or a protective stop [3]. Resuming production from this situation requires a manual intervention to reset and restart the application. Our aim is to develop a formalism by which to systematize the various options that are actually available in most applications, such as reducing speed, changing robot pose, or selecting an alternative motion path, all while continuing with the assigned assembly step(s). Based on this structure, we will describe how unnecessary application downtime in actual assembly applications can be avoided.
event set containing all points in time where a transition is taken, j EN. A feasible run of CB is the sequence of states rps' = {S(to), S(tl), . . . , S(tj), ... }, where S'(tj) C S of the activated states between event tj and tj+l. An example of a finite state automaton for exception reaction and exception recovery is shown in Fig. 1.
A. Structure and Terminology
Three states are defined in the finite state automaton. • Sl E S: the production behavior state (like normal operation), which is the initial state Sl E So. • S2 E S: the advanced safety state (like speed reduction, or standstill when speed is zero). • S3 E S: the basic safety state (like a protective stop). E = {e], e2, e3, e4, es, ed are the events for transitions from one state to another, where el, e2, e3 and e4 represent the corresponding exception reaction, and es and e6 for exception recovery. Events for certain transitions are dependent on the input(s) (measurements), like the distance between the robot and human denoted by iI, i2, and is. The output, e.g, the defined velocity, might be a function of the input like 02 = y(i2)' For collaboration of multiple human workers and multiple robots, the above approach is extended. Each collaborative behavior of a single HRC can be modeled as a CBk with k EN. The overall collaborative behavior CBtatal = CBI II CB2 II ... II CBq can be composed by all sub-CBs by, e.g., the standard composition method of automata [7]. Definition 3: The parallel composition of CBs is defined as CBtatal = CBj II CB2'" II CBq = (Sj x S2 " ' X Sq , EJ U E2 "·U Eq, So , IJ U 12"'u Iq, 0, y). The execution of CBtotal complies to the following definition of the semantics: Definition 4: Let T = {t o , tl, . . . , tk , ... } denote the event set containing all points in time where a transition is taken, kEN. A feasible run of CBtotal is the sequence of composed states rp, = {S(to), S(tl), . . . , S(tk), ... }, where S(tk) E = Sl X S2 X ... X Sq of the activated states between events at tk and at tk+I' The composition for multiple HRC has been applied to an assembly scenario, which will be presented in the next section. Additionally, the common events for HRC in industrial assembly are summarized and presented in [20] with possible exception reactions and the corresponding recovery strategies. The list can be used as a reference for structuring the collaborative behavior of industrial robots.
The structure of the collaborative behavior with respect to interruptions is defined and used throughout the paper with the following terms [20]. • Production Behavior: Maintain productivity and avoid safety-critical situations • Safety and Productivity (S&P) Exception: Irregular situation which either interferes with productivity or compromises workers safety o Exception Reaction: execution of S&P functions to uphold productivity and to ensure worker safety o Exception Recovery: dedicated set of commands to restore regular productive operation after Exception Reaction In the normal operation, collaborative robots are operated with production behavior. Safety and productivity exceptions which either interfere with productivity or compromise workers safety are caught by the corresponding exception reaction. A dedicated set of commands in exception recovery restores the regular productive operation after exception reaction.
B. Formulation o/Collaborative Robotic Behavior The collaborative behavior of robots is formulated as an event-driven finite state automaton, as presented in [20]. The following definition is given: Definition 1: The robotic collaborative behavior is defined as a tuple CB= (S, E, S o, 1, 0, y) consisting of: • the finite set of states: S= {Sj,s}" " ,sn } of states with n EN; • the finite set of transitions (or named event): E \ ',,;. �>-'/ " Hmmlll
...:!L
Robot Controller
Robal
J
Camera
(b) Fig. 2: Experimental setup for HRC in industrial assembly.
,
16 3115
Joint 1: Jojnll:
Two interaction zones in which direct contact between the robot and the human might occur are designated in this example scheme. One zone is located between the robot and the human standing next to each other, the other is located between the robot and a human standing across the workbench. These two interaction zones are the most interesting regions in the setup, as human workers are in close proximity and even contact to the robot here. The sensors are installed in orientations that allow them a good view on the interaction zones and the human workers present in these regions. The sensors are mounted so that each one is facing one of the human workers. In our setup the Microsoft Kinect sensor is used because of its 3D detection capability without any marker and its ease-of-use. The skeleton detection library is used for checking the nearest point of the human worker next to the robot arm . A generic collaborative system has three key components: robot, sensor, and controller. The full structure of the application is shown in Fig. 2(b). Humans are observed by the sensor(s). Data acquired by these
�1.0134 -6S,OW 69 , 476
-$5.1625
Join13:
-",2340
Joint 4:
06116
"-"'" -)9)540
-38,0093
Joint 5, Joint 6:
-11".2370
In9125
Joint 7:
·2375,0353
Tool X:
.c68)670
Tool Y:
-396,4910
Tool Z:
2S6)900 -119,8280 -]2)006
TooIRX: TooIRY: TooIRZ:
I RoboIVeloc.ty;
65,4911 522._ 1194510
-40,.. , 1
2&5,4560 ,,, ..
491 5
·1061860
1 100
AudK> Ordft:
100
Start!
���.���F_�'���'����8! :"�:: �t D«isOn Sode::
1691lm172891.3
fiMI DecIHln:
CAL 2 StQ)fOf RlQhtAim
All
HR.C5cfver,t"".
I
(&2 StDl:lfofLl:ftArm
ESTOP
Color indicatorfor side distance to Robot 2 Color indicator for front distance to Robot 2 Color indicotorfor final store a/Robot 2
J
Fig. 3: Graphical user interface for the HRC assembly station.
C. Collaborative Behavior{or Multiple HRC
The experiment is based on the HRC assembly station, shown in Fig. 4. The station consists of the Dual-Arm Concept Robot (FRIDA), and two human workers w {w], W2} with three thresholds b {b], b2, b3} for W2 and two thresholds b {b", b"d for WI, respectively The dashed lines indicate the possibility of operating two robotic arms interacting with two human workers in front. It has been described in [20] which is but not the focus in this paper. =
=
=
03 O2 01
1-y>� � J JJQ _
_
.. ...... .... :.
A
1 1 1 1 'Vorkiug area 1 _____________ 1
Tbresbolds: Stand-still
=
Spe.ed-reductioD Elbow-down
-- am --on
...... -.. ; lV3 : ..........
eII, iIII
eI, iIi
(2)
.
·
SIll E S: the reduced speed state where the production is to be upheld as far as possible • S'V E S: the standstill state to ensure a safe situation E(CB 2) { e], e", em, elV } are the events for transitions from one state to another, where el and e" represent corresponding exception reaction, and eIII and e,v for exception recovery. •
FIg. 4: Schematic overvIew of the HRC assembly statIOn.
The inputs generated by the movement of the workers {W I , W2} are listed in Tab. l. The distance of the workers to the robot is defined with d1(t) and d2(t). Tab I· Inputs generated by movement of workers (w, w,)
Input
io i, i, i3 iT ill itu
Notation
W2 behind 6,: w2crossing 6,: W2 crossing 6,: W2 crossing 63: W, behind 6JJ: WI crossing 611: WI crossing 6111:
d,(t) < 6, d2(t) 2: 61 d2(t) 2: 62 d2(t) 2: 6, d,(t) < 6n d, (t) 2: 6n d, (t) 2: 6rrr
The resulting finite state automaton which expresses the robotic behavior dependent of the movement of the workers is shown in Fig. 5 and Fig. 6 for side-by-side and front scenarios, respectively. The finite set of states for side-by-side CB] is defined as S {S1' S2, S3., S4} , where • S] E S: the normal production behavior state, which is the initial state S] E So • S2 E S: the elbow down safety state where the robot changes its elbow position to avoid collision using the redundant DOF to keep production • S3 E S: the speed reduction state where the production is to be upheld as far as possible • S4 E S: the standstill state for the most safety-critical situation E(CE1) {e], e2, e3, e4, es, ed are the events for transitions from one state to another, where e], e2, e3 and represent corresponding exception reaction, and e4, es and e6 for exception recovery.
em, if
elV, il
Fig. 6: Finite state automaton eE2: robotic behavior interacting with WI.
According to Definition 3, the parallel composition of both automata is defined with CE1 II CE2 (S1 X S2 , E1 U E2 , So , 11 U 12, 0, y), shown in Fig. 7. There are additional transitions indicated with: * (e6,io) and ** *** (elV,il), which are not displayed. =
=
=
=
=
=
Fig. 5: Finite state automaton eE,: robotic behavior interacting with w,.
The finite set of states of CE2 for front interacting with W1 is defined as S {s" SIll, S'V } , where • s, E S: the normal production behavior state, which is the initial state s, E So =
Fig. 7: Composition of eE, and eE,.
The composed state is defined as S {S], S2, . . . , sd (S1,SIII), . . . , Ss where S1 (S1,S,), S2 (S2,S,), . . . , Ss (S4,S,V). In order to guarantee a safe situation, the worst case state of the sub-automata obtained highest priority and defines the resulting state of the composed automaton, except S3, S6 and S7 where the resulting state is a combination of both sub states: • S1 E S: the normal production behavior state, which is the initial state S1 E So • S2 E S: the elbow down safety state where the robot changes its elbow position to avoid collision • Ss E S: the reduced speed state • S3, S6, S7 E S: the reduced speed state with elbow down • S4, ss, . . . , S12 E S: the standstill state to ensure a safe situation The set of events is E {e], e2, . . . , e6, e], . . . , eIV } . The inputs are defined as I {io, iJ, i4, i3, h, i". im } . The outputs 0 {oJ, 02, 03, 04} are given in Tab. 2. =
=
=
=
=
=
=
=
Tab. 2: Outputs generated by the composed automaton. Notation Output
0, 02 03
Robot execution with normal speed v(t) Robot execution with elbow down Robot execution with reduced speed vet)
the productivity is the same as the status shown in Fig. 9. It is the state
86 in the composed model.
�
f(d(t» 04
Robot execution with standstill v(t)
�
0
The output functions Y : I x S � 0 is given here with: YI(iO, 54)=01, Y2(iO, 58)=03, Y3(iO, 5u)=04' Y4(iJ, 51)=02, .... The output behavior of the other states is determined by state description above. To establish the behavior of the composed automaton, it is applied to the robot in the HRC assembly station, shown in the following figures. In this experiment the right arm of the robot is tested by interaction with the worker in the front. The figures only provide the functionality of the composed automaton for the left arm which both workers interact with. The actual state of the robot is determined by the decision engine with the given state of the composed automaton.
I RobotV�locrty: I
14�,5300726744
I
:����:r.ity: I :
Fig. 8: Both workers are behind the thresholds (d" dn). Both decision engines for front and for side acknowledge the nOlmal production behavior state (s), sD. It is the state.5, in the composed model. 1691.27711728913
1�lxltVelocity:
I R;lbotVelocitr
1
1
Fig. 9: lVI (front) is crossing the threshold dl! and W2 (side) is still final�ision: behind threshold 61. Decision engine front indicates the state Sm and the side one still acknowledges the state Sl. The final decision is the reduced speed state due to worst case consideration. It is the state.55 in the composed model. D&"o" >�,
539841246505452
'252
66452277432633�
13"
m�896I50262962
1
Fig. 12: lVl crosses the threshold dn and W2 crosses the threshold d3. >,�, D&"o. Decision engine front indicates the state Slll and the side one acknowledges the state S4. The final decision is the standstill state due to worst-case consideration. It is the state.58 in the composed model. ,1.wi536422407
0«"0">m"'
1818
I 72,'>061513739973
Fig. 11: WI crosses the threshold dl! and W2 crosses the threshold d2. Decision engine front indicates still the state Slll and the side one acknowledges the state s,. The [mal decision is the composed state "reduced speed with elbow down". The reduced speed is computed based on the smallest distance of the distance from the side worker or from the front worker. This status is equivalent with state 87 in the composed model. 14,45036697S!l736
70.5083581571604
""
1
Fig. 13: WI is behind the threshold dl! and W2 crosses the threshold d3. Oeci�iQn Side: Decision engine front indicates the state Sl! and the side one acknowledges the state S4. The final decision is the standstill state. It is the state 84 in the composed model. 1
l�tVelocity: 1 1
1100
Fig. 14: WI is still behind the threshold dn and W2 is now behind the threshold d1. Decision engine front indicates still the state Sl! and the side one acknowledges the state St. The final decision is now in nonnal production as all workers are behind all thresholds again. It is the state 81 in the composed model. 1703,693198151'"
I ::�:�:ity I ::
I
Fig. 10: WI crosses the threshold dn and lV, crosses the threshold d,. Decision engine front indicates >,,.. "".". the state Slll and the side one acknowledges the state S2. The final decision is the composed state "reduced speed with elbow down". But WJII9795296
D&",">�'
13"
n,514H4:>47B52
Experimentation with this setup and the finite state automata described above have shown that the uptime of the collaborative assembly application can be improved, particularly in this example by moving down the elbow in case that the worker's arm approaches. The robot does not have to stop all the time with human interference. The productivity can be upheld as far as possible while respecting safety constraints. IV.
[10]
[11]
CONCLUSIONS
In this paper, the structured collaborative behavior in mixed environment has been extended with multiple human workers and multiple robots. The collaborative behavior in an HRC assembly station has been modeled using the finite state automata with the proposed composition rules. The concept has been validated in this application. Experimentation with this setup has shown that the robot does not have to stop as frequently due to human intervention. Speed reduction or standstill is mostly sufficient, which can be automatically recovered to normal operation from exception. As a result, this reduces the frequency of unintended contacts between worker and robot, and upholds the productivity
[12]
[13]
[14]
[15] ACKNOWLEDGMENT
The authors thank Jakob Heyn and Junjie Zhang for their contributions to the setup of the assembly station for human-robot collaboration.
[16]
REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
M. Hagele, W. Schaaf, E. Helms: Robot Assistants at Manual Workplaces: Effective Co-operation and Safety Aspects. In: International Symposium on Robotics ISR 2002 / CD-ROM: Proceedings. October 7-11, 2002, Stockholm, Sweden. Stockholm, 2002. A De Santis, B. Siciliano, A De Luca, A Bicchi: Atlas of Physical Human-Robot Interaction, Mechanism and Machine Theory, Vol. 43, No. 3, March 2008, p. 253-270. ISO 10218-1:2011, Robots for industrial environments Safety requirements - Part 1: Robot; ISO 10218-2:2011, Robots for industrial environments - Safety requirements Part 2: Industrial Robot System and Integration, ISO, Geneva (2011). ISO/PDTS 15066, Robots and robotic devices - Safety requirements for industrial robots - Collaborative operation, ISO, Geneva (2012). A M. Zanchettin, L. Bascetta, P. Rocco: Acceptability of robotic manipulators in shared working environments through human-like redundancy resolution. To appear in Applied Ergonomics; DOT: 10.1016/j.apergo.2013.03.028. 1 Kriiger, T. K. Lien, A Veri: Cooperation of human and machines in assembly lines. In: CIRP Annals Manufacturing Technology. 58 (2009), No. 2, p. 628-646. J. Hopcroft, R. Motwani, and J. Ullman: Introduction to Automata Theory, Languages, and Computation, 3rd Ed. Addison-Wesley, 2006. Haddadin, S.; Albu-Schaffer, A; Hirzinger, G.: Requirements for safe robots: measurements, analysis and new insights. International Journal of Robotic Research, vol. 28, pp. 1507-1527, 2008 Haddadin, S.; Suppa, M.; Fuchs, S.; Bodenmiiller, T.; Albu-Schaffer, A; Hirzinger, G.: Towards the robotic co-
[17]
[18]
[19]
[20]
[21]
[22]
[23] [24]
worker. Robotics Research, ser. Springer Tracts III Advanced Robotics, C. Pradalier, R. Siegwart, and G. Hirzinger, Eds. Springer, 2011, vol. 70, pp. 261-282. Mekki-Mokhtar, A; Blanquart, 1; Guiochet, 1; Powell, D.; Roy, M.: Safety trigger conditions for critical autonomous systems. IEEE 18th Pacific Rim International Symposium on Dependable Computing, 2012, pp. 61-69. Woodman, R.; Winfield, A F.; Harper, C.; Fraser, M.: Building safer robots: Safety driven control. International Journal of Robotic Research, vol. 31, no. 13, pp. 16031626, 2012. Matthias, B.; Kock, S.; Jerregard, H.; Kallman, M.; Lundberg, I.; Mellander, R.: Safety of collaborative industrial robots: Certification possibilities for a collaborative assembly robot concept. IEEE International Symposium on Assembly and Manufacturing, 2011. Kain, S.; Ding, H.; Schiller, F.; Stursberg, 0.: Controller architecture for safe cognitive technical systems. Computer Safety, Reliability, and Security (SAFECOMP), ser. LNCS, F. Saglietti and N. Oster, Eds., vol. 4680. Springer, 2007, pp. 518-531. Stengel, D.; Ostermann, B.; Ding, H.; Bortot, D.; Schiller, F.; Stursberg, 0.; Bengler, K.; Huelke, M.; Som, F.; Strunz, U.: An approach for safe and efficient human robot-collaboration. 6th International Conference on Safety of Industrial Automated System, 2011. Ding, H.; ReiBig, G.; Stnrsberg, 0.: Increasing efficiency of optimization-based path planning for robotic manipulators. IEEE Conf on Decision and Control (CDC) and European Control Conf (ECC), 2011, pp. 1399-1404. Ding, H.; Wijaya, K.; ReiBig, G.; Stnrsberg, 0.: Optimizing motion of robotic manipulators in interaction with human operators. International Conference on Intelligent Robotics and Applications, ser. LNCS, S. 1 et aI., Ed. Springer, 2011, pp. 520-531. Schraft:, R.D.; Schmid, S.; Thiemermann, S.: Man-robot cooperation in a flexible assembly cell. Assembly Automation, vol. 22, no. 2, pp. 136-138, 2002. Mortl, A; Lawitzky, M.; Kucukyilmaz, A; Sezgin, M.; Basdogan, C.; Hirche, S.: The role of role: Physical cooperation between humans and robots. International Journal of Robotic Research, vol. 31, p. 16561674, 2012. Wilcox, R.; Nikolaidis, S.; Shah, 1: Optimization of temporal dynamics for adaptive human-robot interaction in assembly manufacturing. Robotics, Science and Systems (RSS), 2012. H. Ding, J. Heyn, B. Matthias, and H. Staab: Structnred Collaborative Behavior of Industrial Robots in Mixed International Environments. IEEE Human-Robot Conference on Automation science and Engineering, 2013. B. Matthias, H. Ding, 1 Heyn: Betriebsstrategien fUr Mensch-Roboter-Kooperationsarbeitsplatze - Produktivitat und Sicherheit. VDT Berichte 2209, AUTOMATION 2013, p. 105-108. H. Ding, B. Matthias: Safe Human-Robot Collaboration Combines Expertise and Precision in Manufacturing - A Paradigm for Industrial Assembly in Mixed Environments. To appear in atp edition, 2013. ABB, Dual-arm concept robot, www.abb.com. Kock, S.; Vittor, T.; Matthias, B.; Jerregard, H.; Kallman, M.; Lundberg, I.; Mellander, R.; Hedelind, M.: Robot concept for scalable, flexible assembly automation: A technology study on a harmless dual-armed robot. IEEE International Symposium on Assembly and Manufactnring (lSAM), May 2011, pp. 1 -5.
