Machine Operation Skill and Visual Perception

第 1巻

第 1号

平成

平成 20 年 1 月 31 日発行（隔月発行)

年 月

日発行（隔月発行）第

巻 第 号

-

…………………………………………………………………………………………………

■ …………………………………………………………………… ……………………………………………… ―

―

………………………………………………………………… ︶

…………………………………………… ………………………………………………………………………………………… …………………………………………………… ………………………………………………………………………………………… ……………………………………………………………………… ……………………………………………… ―

―

…………………………

編集兼発行人 〒 113-0033 東京都文京区本郷 1-35-28-303 島貫 洋 発 行 所 〒 113-0033 東京都文京区本郷 1-35-28-303 社団法人計測自動制御学会 Tel.03-3814-4121 Fax.03-3814-4699 (振替 00160-9-127863) 発 売 所 〒 112-0011 東京都文京区千石 4-46-10 株式会社コロナ社 Tel.03-3941-3131 印 刷 所 東京都荒川区西日暮里 5-9-8 三美印刷株式会社

- - 雑誌

―

定価 ,

円（本体 ,

円)

-

//

-

/

/

SICE Journal of Control, Measurement, and System Integration, Vol. 1, No. 1, pp. 018–025, January 2008

Future of Mechatronics and Human — Machine Operation Skill and Visual Perception — Fumio HARASHIMA ∗ and Satoshi SUZUKI ∗ Abstract : This paper mentions circumstance of mechatronics that sustain our human society, and introduces HAM(Human Adaptive Mechatronics)-project as one of research projects to create new human-machine system. The key point of HAM is skill, and analysis of skill and establishment of assist method to enhance total performance of human-machine system are main research concerns. As study of skill is an elucidation of human itself, analyses of human higher function are significant. In this paper, after surveying researches of human brain functions, an experimental analysis of human characteristic in machine operation is shown as one example of our research activities. We used hovercraft simulator as verification system including observation, voluntary motion control and machine operation that are needed to general machine operation. Process and factors to become skilled were investigated by identification of human control characteristics with measurement of the operator’s line-of sight. It was confirmed that early switching of sub-controllers / reference signals in human and enhancement of space perception are significant. Key Words : Human Adaptive Mechatronics, skill, machine manipulation, line-of-sight, voluntary motion, brain, human control.

1. Introduction Mechatronics is known as the discipline integrated by mechanical, electrical and information technology and has been used to produce advanced artifacts used in modern society, which has been developed through the interdisciplinary studies of diverse engineering fields. Our modern life is surrounded and enhanced by gadgets and gimmicks of mechatronics products. Origin of machine in mechatronics is a 18-19th century factory industry in the industrial revolution. After electrical and electronics technologies were developed, fusion of mechanism and electronics gave us mechatronics. At that time, main issues for mechatronics are how to control the machine efficiently and how to make many products having uniform quality [1]. As we understand from the automatization of the manufacturing, science and technology in 20th century were for liberation of people from physically painful labor. For this aim, human had been inventing various kinds of machine slaves, such as air conditioners, cars, heaters, telephones, PCs and so on. These slaves, however, consume much more energy and resources than human itself. One person itself generates 1 kg CO2 a day, and whole of human society generates ten times CO2 than human itself does. This means one human has ten mechanical slaves. According to one estimate, one Japanese has 25 slaves, and one American has 55 slaves [2]. How many do each of you have? (Fig. 1) Machines that were invented to help human are wrecking geoenvironmental assessment. This is sad contradiction. Best way to preserve the earth environment may be the declining of our modern society’s quality. But it is truly difficult to forsake convenience that human once got. That is neither more ∗

School of Science and Technology for Future Life, Department of Robots and Mechatronics, Tokyo Denki University, 2-2 Kanda-Nishiki-cho, Chiyoda-ku, Tokyo 101-8457, Japan E-mail: {harashima, ssuzuki} @fr.dendai.ac.jp (Received September 11, 2007)

nor less than selfish request, but it is demanded that the earth’s environment is improved and that human society is kept comfortable. As other issue, our society structure changes with few equals in history. That is a problem of aging or low birthrate problem. In case of Japan, peak of the population came in 2004, and the number is decreasing from then [3] (By linear forecast, only two Japanese may remain in the middle of 22nd century. If the survivors are both female, or both male, Japanese would perish!).

Fig. 1

How many slaves do you use?

Above discussions are summarized into the following as our society’s goal. • clean environment • happy aging society • intelligent human life We think that one of effective solution for the three goals is mechatronics. Since aged person’s physical ability is disrupted

c 2007 SICE JCMSI 0001/08/0101–0018 

SICE JCMSI, Vol. 1, No. 1, January 2008

certainly, machines should be alternative methods of their abilities. However, such usage of machines produces contradiction. Even though machines were developed to make human comfortable, much time and effort are needed for human to become skilled, because of the high-complexity from high-function and multi-functions. Hence, human is often demanded to learn an operation of the machine implicitly. Relatively young persons can master the operation easily, but to aged persons the learning process may become pain. Human can adapt to the machine, but existing machines cannot adapt to human. This unbalance in an interaction between human and machine sometimes make trouble in human-machine systems. Therefore, we believe that science and technology in 21st century must liberate people from mentally painful labor. In these background, new system structure called ‘Human Adaptive Mechatronics (HAM)’ was proposed in the COE (Center of Excellence) research project from Tokyo Denki University [4],[5]. The HAM differs from the existing manmachine system whose main purposes are an wrong operation avoidance and the hazard notice. Not only support to avoid such accident but also positive enhancement of human operation and the machine’s performance are main theme of the HAM. There are many systems hoped to be applied as HAM. Actually, in our research projects, the surgery support system [6], the haptic assist device [7],[8] and the teleoperated legged robot [9] have been studied as application of HAM. As theories for the HAM, the safe manual control [10] and the rate saturation of actuators [11] were proposed. Skill analyses from human brain functions have been studied in the mirror drawing test [12] and the virtual pendulum stabilization task [13]. As one research of them, experimental skill analysis of vehicle operation [14] is introduced in this paper. The vehicle operation is a good example for HAM study because that includes observation and voluntary motion control. In the study, a virtual hovercraft manipulation task is used to investigate the operational skill. The remained sections are organized as follows. Section 2 discusses vehicle operation and skill with brain research. In Section 3, an experimental set-up of the hovercraft task and a line-of-sight (LOS) measurement to detect the switching of the operator’s targets are explained. In Section 4, skill analysis using the LOS is mentioned. Section 5 shows an identification analysis to reveal human’s control characteristics. Section 6 presents the conclusion, and an epilogue is in Section 7.

2. Vehicle Manipulation, Skill, and Brain Locomotive type machines such as a car, a motorcycle, an airplane and a train are representative human-in-the-loopmachine systems, and have been very useful in order to enlarge our human activity. But, their manipulation is normally difficult, because a machine has unknown dynamics to the operator. When we tries to move our body parts, our brain controls the musculoskeletal system by using the musculoskeletal internal models that have been learned already in cerebellum [15]. Human can realize even new motion relatively soon by utilizing the internal models/controllers that have been learned [16]. The internal models to move our body is, however, not directly effective for the vehicle manipulation since machine’s dynamics and human body’s dynamics differ. Hence, long time training is demanded to learn other internal model of the vehicle. In 1980’s, Kawato proposed a ‘feedback error learning’

19

model, that explains human has forward and inverse models of dynamics of body’s motion and that the learning is done by minimizing motion error [17]. As a functional-anatomical studies, Tamada et al. utilized a fact such that relations of the right and left-side are opposite in a cerebrum and a cerebellum, and discovered a functional relation between a part of cerebellum and a ventral premotor cortex in brain cortex of when human uses tools [18]. Imamizu et al. showed that different areas in cerebellum are activated according to kind of tools. This means that some control neural circuits are formed in different part in brain according to kind of tasks [16]. Wolpert and Kawato extended the ‘feedback error learning’ model into a MOSAIC(MOdule Selection And Identification Control) by expansion of the forward-model into a predictor and of the inverse-model into controller [19]. The concept of MOSAIC model is that learning is done by switching acquisition of internal models of unknown dynamics and utilization of previouslylearned models. As shown by these studies, the idea, such that models of controlled objects (i.e., dynamics of human body, and ways to use tools) are formed in cerebellum, is one of dogmas in brain science. From these studies, it is certain that so-called internal model is needed for controlling something. And we think that both setting of targets and execution of each motion control are needed to enhance total performance of the manipulation as shown in Fig. 2. Because both adequate reference signal and good controller are needed for better performance from the view point of the control engineering. In case of vehicle manipulation, it can be guessed that the operator chooses references by watching something in the environment. Hence, we considered that operation skill is related to active observation, and analyzed skill in the vehicle operation by combining with line-of-sight measurements.

Fig. 2

Conceptual block diagram of human brain control model. (The block diagrams except the ‘machine’ and the ‘environment’ indicate control functions in human’s brain.)

3. Virtual Hovercraft Manipulation Task Among various types of vehicle manipulations, teleoperation system requires efforts to human. The first reason is paucity of information that enables the operator to estimate the vehicle’s status. The second reason is that an image captured by a camera differs from unaided image. By the axiom of a camera and distortion of the camera’s optical system, original 3D visual information is degenerated into 2D image. This degeneration of transmission of visual information affects operator’s 3D-space recognition, and may relate to the operation skill. 3.1 Hovercraft Task Based on the dynamics equation, a 3D-graphic hover-craft game was constructed by using OpenGL. A model of a hover-

20

SICE JCMSI, Vol. 1, No. 1, January 2008

craft is shown in left of Fig. 3. The hover’s status is the position (G x, G y) and its orientation G φ. Assuming that the operator controls the hovercraft’s thruster forces fr , fl by manipulating a joystick in a forward-backward direction α ∈ [−1, 1] (in normalized value) and in rotational angle γ ∈ [−15, 15] [degree], the relation from joystick status α and γ to the force fl and fr is given by     fl k · α(1 − γ) = , (1) fr k · α(1 + γ) where k is a gain parameter. Then, the dynamics of the hovercraft is obtained as ⎤ ⎡ G ⎤ ⎡ bt G ⎤ ⎡⎢ cos G φ˙ ⎢⎢⎢ x¨ ⎥⎥⎥ ⎢⎢⎢ − m x˙ ⎥⎥⎥ ⎢⎢⎢ m · 2kα⎥⎥⎥⎥ ⎢⎢⎢ G ⎥⎥⎥ ⎢⎢⎢ bt G ⎥⎥⎥ ⎢⎢⎢ sin G φ˙ ⎥ (2) ⎢⎢⎣ y¨ ⎥⎥⎦ = ⎢⎢⎢ − m y˙ ⎥⎥⎥ + ⎢⎢⎢ m · 2kα ⎥⎥⎥⎥⎥ , ⎦ ⎣ ⎦ ⎣ l G¨ φ − bIr G φ˙ · 2kαγ I where m, I, bt , br , l are the mass, the rotational inertia, a translational/rotational viscosities and an offset length between the center of the mass.

Fig. 3

eyes are closed are removed from the measured data by checking the eye condition. Participant’s head was fixed on a chin support to avoid head moving against the monitor. The measured LOS position raw data was compensated by checking positions of markers attached on corners of the monitor by vision processing. Next step is projection of the gaze’s coordinate value on the monitor into the virtual 3D game space. This computation is done based on geometrical transformation among the actual monitor coordinate system, virtual game screen existing in the virtual 3D game space, the local coordinate system attached to the hovercraft (virtual game space), and the global space G S shown in Fig. 4 [14]. The last step is a judgment of gaze against flags. By preexperiments, we determined an angle of view of eye fixation as 5 degrees in this experimental set-up. We judge that an operator is looking at the flag when a relative angle between the flag and his eye fixation point’s location in the monitor plane is less than 5 degrees. In later explanation, as a terminology, nearest flag of all flags that are seen in the game screen is called the ‘1st-flag’. Similarly, the next closer flag is called ‘2nd-flag’.

Modeling of a hovercraft (left) and command by joystick (right).

As an adequate task that is easy to detect the change of reference-target, we designed a game course like a giant slalom field of ski as shown in Fig. 4. We names areas that are separated by flags as ‘zones’. Participants were demanded to operate a hovercraft to the goal, request-1) as fast as possible by passing through 15 flags on the way, request-2)

Fig. 5 Experiment scene (left), and camera image captured by the LOS measurement system (right).

Figure 6 shows an example that shows trajectory of the hovercraft and the computed gaze trajectory. It can be confirmed that operator is looking circumferences of flags ahead of the manipulated hovercraft.

as close to the flags as possible.

Result of success passing is informed to the operator by changing color of the flags and sound at the moment when the manipulated hovercraft enters the circle area around the flags.

Fig. 6 Trajectories of the hovercraft (bold solid line) and gaze position (dotted thin line). Filled circles are flags’ position.

Fig. 4

Course layout of a hovercraft task.

3.2 LOS Computation to Detect Switching of Operator’s Targets The LOS data was measured by using an eye motion recorder system as shown in Fig. 5 (left). Data of when blink occurs or

3.3 Evaluation of Task Performance Because we asked participants to consider requests-1 and 2, the following can be considered as performance indexes. • total task time; T t • sum of minimum approach distances to flags; E f

21

SICE JCMSI, Vol. 1, No. 1, January 2008

• sum of hovercraft’s trace error against ideal path; E p T t is time that the hovercraft reaches to the goal-line after passing through the start-line. E f is computed as Ef =

15

 min

( G x − x f,i )2 + ( G y − y f,i )2 ,

i=1

where (x f,i , y f,i ) is position of i-th flag. Computation of E p is done by Ep =

15

f (G y ) − G x Δ , y i i

relation between operational skill and the LOS. The data was obtained from beginner who became a better operator docilely. Figures 8–10 show change of the evaluation indexes E f , E p and T t . Their x-axes are the number of trial, and the 1–10, 11–15, 16–20 and 21–25 trials were done in first, second, third and fourth day, respectively. Experiments were done continuously in weekday (total 4 days). The data shown at the 26 trial is other result of an expert for comparison. Color coding by bars in the

(3)

i=1

where (G xi , G yi ) are linear interpolated coordinate value around hovercraft’s position (G xi , G yi ), and f (G yi ) is an ideal trajectory function approximated by using 17th order polynomial function. Δy is a step-distance of G Y direction in order to accumulate the error between the actual trajectory of the hovercraft and the ideal path f . G yi is chosen as integral multiple of Δy . Coefficients of the polynomial were obtained by using constraints such that (a) the curve passes through all flags and that (b) the bisectors of an angle made by adjacent three flags correspond to normal-line to the curve.

4.

LOS Behavior and Skill

Observation is firstly executed and is significant in the human process of the machine operation. In our other study, we have been studying relation between skill and observation by utilizing balancing task [20]. In that study, checking the activated regions by referring the homunculus map (Fig. 7) of a high-skill person and a low-skill led the following [13].

Fig. 8

Change of the number of flags passing successfully (bold line) and of accumulation of the minimum of approach distance E f lag (dashed line).

• High skilled operator has high activity at sensory wide region and at eyeball area in motor region, but a motor control region is not so rather strong. • Low skilled operator has a tendency of activation mainly on sensor region in the primary somatosensory cortex and motor region in the primary motor cortex, i.e., non-expert tends to concentrate only on control of arm. In short, it was confirmed that, an expert (a) reinforces the sensory function, and (b) decreases ratio of paying attention to motor control at cortex. Fig. 9

Fig. 7

Change of trace errors E p .

Lateral surface of the cerebral hemisphere (left) and motor homunculus (right).

In the giant slalom task, the flags appear one after another, so the operator has to control the hovercraft to approach each flag. Hence, for skilled operation, an adequate self-switching of targets on the manipulation is important. We thought that measuring of operator’s LOS enables to detect switching of the references. We would like to show one typical case to explain the

Fig. 10 Change of total-time T total .

22

SICE JCMSI, Vol. 1, No. 1, January 2008

identification based on the switching timing [21]. As a terminology, a part of whole trajectory from a moment the operator looks at the 2nd-flag on i-zone to next moment he looks at next 2nd-flag on (i + 1)-zone is named ‘path’. ARX identification was done against each path. 5.1 Decision of Operator’s Controller Structure

Fig. 11 Ratio of gaze toward flags.

Fig. 9 means ratio of errors of each 15 zones. Both trace error E p (Fig. 9) and total time T t (Fig. 10) are decreasing roughly to the same level of the expert after 4 days trials. Bold line of Fig. 8 indicates the number of flags that the hovercraft passed through. During 1-16 trial, the hovercraft can only pass through less than 9 flags of 15 flags, so we can say that the participant was not used to the manipulation. After 17 trials, the participant could pass through more than 9 flags, and the number is increasing almost monotonically after the 17th trial. Mean of minimum value of approach distance to each flag, that is shown by dashed line in same figure, is also decreasing almost monotonically at the skill-upgrading phase except 14th and 16th trial. This outlier phenomenon can be found by checking the trajectory difference against ideal path. Seeing Fig. 9, it can be confirmed that error of 9th zone in 16th trial is larger than the others. In short, because of one large fail of passing through a flag, total performance got worse. This figure shows also that an error of difference is decreasing after 17th trial. Further, as shown in Fig. 10, the total-task-time is decreasing almost monotonically after 14th trial and is decreasing after 17th trial without outlier. Hence, we can say that the beginner was getting used to the manipulation step by step after 17th trial. In later discussion, a period from 17th to 25th is called ‘skill-upgrading phase’, and the period is called ‘trial phase’. Next, we investigated ratio of gaze-times against the 1st-flag and 2nd flag. The result is summarized in Fig. 11. Two bars are drawn in each trial, and their left and right bars means the gaze ratios against the 1st-flag and 2nd-flag respectively. In consideration of the beginner’s status analyzed in above, it can be found that ratio against the 1st-flag is monotonically decreasing and that the other ratio against the 2nd-flag is increasing instead in ‘skill-upgrading phase’. This tendency means that early action of glancing to the 2nd flag is needed to upgrading of the hovercraft manipulation. In other words, we can say that early setting of the reference target and the adequate are important to the skill.

5. Identification of Operator’s Control Characteristics We assumed that the switching of controllers occurs at the time that the operator looked at the 2nd-flag first. Then, we sectioned the operation sequence into segments for the ARX

Human tries to estimate the dynamics of the controlled object and decides its manipulation based on the model of the dynamics. Hence, considering of the hovercraft’s dynamics is important to investigate characteristics of the human manipulation. For giant-slalom task treated in our experiments, adequate rotational control is significant. From the input term in hovercraft’s dynamics equation (2), we can find that the rotation φ depends on two inputs α and γ of a joystick and γ has no relation to position x and y. Therefore, it is better to assume that an output from the human controller is a scalar variable α · γ =: v, because analysis of a single-output system is easier than multioutput system. Now, we will examine the input information that is utilized in human controller. In the hovercraft task, the operator has to execute basically steering control to go close to the flag. This means a stabilization control that makes ’relative angle’ between the hovercraft and the 1st-flag, say ξ, converge to zero. Hence, ξ is adopted as one of input variables to the human controller. Distance between the 1st-flag and the hovercraft is also important, and is described as d. If there is only one flag, only stabilization control to converge ξ to zero may be sufficient because it may not be necessary to control speed. In giant slalom task, flags appear in sequence, so the operator has to control the hovercraft according to the distances against flags. However, d is virtual geographical variable in the virtual game space, and the operator cannot obtain the physical variable directly. It can be guessed that the operator reconstructs the 3D virtual space inside one’s brain from the 2D image projected to the monitor. For the ARX identification, d is adopted as second factor with consideration of an effect of the spatial perception. To evaluate a break timing of the car driver, ’optic flow’ is known as effective index [22]. The optic flow is change of elevation angle against the fore object. This index is based on the concept that human utilizes a change of elevation angle to estimate change of depth distance. Hence, we guessed that the elevation angle, say θ, plays some role on the hovercraft manipulation and adopted as third input variable. We do not adhere to accuracy of the computation because of relative comparison between participants. Hence, instead of exact computation by projection from the virtual space to the monitor screen, θ is computed simply by using the distance d(t) as θ(t) = tan−1 (6.0/d(t)), where 6.0 means a height of the flag. On the visual processing of a car driver, it is said that the driver recognizes not instant image in real time but special image accumulated in terms of time [23]. As is known as common sense in control engineering, derivative information is needed to control. To include derivative structure in the ARX model, time shift is considered. Summarizing the above discussion, the following discrete ARX model is assumed.

23

SICE JCMSI, Vol. 1, No. 1, January 2008

v(t ) = a1 ξ(t ) + a2 ξ(t − 1) + a3 ξ(t − 2)

against abscissa of the number of trials j (= 1, · · · , 25)

b1 θ(t ) + b2 θ(t − 1) + b3 θ(t − 2) c1 d(t ) + c2 d(t − 1) + c3 d(t − 2),

T t = pt · j + qt (4)

where t is local time just after time when the operator see the 2nd flag first at i-th zone, and ai , bi , ci (i = 1, 2, 3) are gains to be identified. 5.2 Statistic Analysis of Skill Participants are ten male beginners (age=22 or 23) with no history of neurological deficits. Written consent and ethical approval were obtained before the examinations. Ten trials on the first day and five trials on other days were imposed continuously in weekday (total 4 days). Eventually, valid and reliable data could be obtained from total five case, and we call each case as participants A, B, C, D, and E. It was confirmed that some beginners tried to control the hovercraft by repeating short time acceleration and free-wheeling by utilizing the inertia. Since assumption of application of the ARX model is that target system is a linear continuous system, the ARXidentification should not be applied to above-mentioned nonnormal cases. From this reason, the paths including the inching operation are eliminated. And, inadequate data are eliminated by checking low correlation between actual raw data and the reproduced data. In later statistic analysis, only valid identified ARX parameters that have more than 0.9 correlation coefficients were used. Figure 12 shows transitions of sum of minimum approach distances to flags E f (dashed line) and total time T t (solid line) vs the number of trials. The regression lines of each transition

T f = pf · j + qf are drawn by their bold lines. Table 1 summarizes the coefficients p∗ and the correlations r∗ . We can find that pt , pd < 0 of all five participants, hence, fact-1) all participants became skilled in sense of time T t and approach distance E f because these indexes were decreasing as the trials increased. Table 1

Gradient and correlation coefficients concerning regression lines of T t and E f . case A B C D E

pt -0.34 -0.69 -0.30 -1.52 -1.47

rt 0.63 0.73 0.44 0.88 0.79

pf -2.42 -1.17 -1.59 -1.08 -0.59

rf 0.71 0.45 0.48 0.24 0.16

The formation of the assumed controller is expressed as the linear combinations of ξ, θ and d, and we can think that strength of gains to the factors reflects the resultant output value from the human controller. After we eliminated data of non-normal running, percentage of the sum of absolute value of gains, described by ρ∗ , were calculated against data which gives high reconstruction result, and correlation between the percentages and the task performance index E f are investigated. The gains’ ratio ρ∗ are computed as follows. ρξ = 100ave[˜a/(˜a + b˜ + c˜ )] ρθ = 100ave[˜a/(˜a + b˜ + c˜ )]

(5)

(6) ˜ (7) ρd = 100ave[˜a/(˜a + b + c˜ )],

3 where a˜ = 1 |ai |/3, b˜ = 31 |bi |/3, c˜ = 31 |ci |/3, and ave[∗] means average operation. Figures 13 and 14 are examples of the analysis in case of participants A and B. The graphs show transients of the averages of ρξ , ρθ and ρd and each half of the standard deviations.

Fig. 12 Transition of total time T t (solid lines) and accumulation of minimum approach distance E f (dashed lines) of all participants. Numeric characters attached to dashed lines lines are the number of fails in passing through flags.

Fig. 13 Transition of gains’ ratio in case of participant-A.

24

SICE JCMSI, Vol. 1, No. 1, January 2008

high gain to the relative angle ξ and the distance d to the 1stflag. However, remind that ξ and d are variables in virtual 3Dspace. And, note that these values are not observable directly to the operator by their eyes. Taking these facts into consideration, skilled operator can reconstruct and recognize the virtual 3Dspace from the 2D image information, and then can utilize them to the skilled operation. Combining the construal A and B, it can be concluded that the human in telemanipulation with monitoring the environment weakens dependency on the surface visual information on the 2D monitor, and reinforces recognition of virtual 3D work space, then becomes an expert.

6.

Fig. 14 Transition of gains’ ratio in case of participant-B.

Other gain ratios ρ∗ of other participants are computed similarly, further correlation r∗,ρ∗ between ρ∗ and E f or T t are computed and are summarized in Table 2. Table 2 Correlation coefficients concerning T t and E f . case A B C D E

rE f ,ρξ -0.18 -0.71 -0.32 -0.16 -0.36

rE f ,ρθ 0.55 0.81 0.44 0.77 0.72

rE f ,ρd -0.37 -0.15 -0.08 -0.75 -0.44

rTt ,ρξ -0.07 -0.31 0.29 -0.29 0.12

rTt ,ρθ 0.38 0.23 -0.07 0.33 0.33

rTt ,ρd -0.34 0.20 -0.61 -0.08 -0.64

This table gives us the following facts. fact-2) Most large (in absolute value) coefficient among correlations is rE f ,ρθ in all cases of participants, and the signs are all positive(In the table, those numbers are written by bold fonts). fact-3) Combinations of signs in correlation concerning E f are same in all participants. In short, (a) rE f ,ρξ , rE f ,ρd < 0 and (b) rE f ,ρθ > 0. fact-4) There is no common pattern in all participants of correlation concerning T t . Now, let us analyze comprehensively results shown in Tables 1 and 2. Construal-A: The fact-1 indicates that all participants became skilled in senses of index E f , and the fact-2 shows correlation of E f and ρθ is positive, hence the gain ratio of θ is decreasing as the participant becomes skilled. Speaking concretely, weakening of utilization of elevation angle information has relation with the upskilling. In other words, on the beginner phase, participants depended too much on change of optical flow in the monitor screen, and then were changing to utilize other information as the trial was increasing. Construal-B: The fact-3 means that common factor for the skill in all participants can be found. Especially, the fact-3(a) indicates that it is necessary to form own controller that has

Conclusion

In this paper, we mentioned past and future of science and technology, and explained new mechatronics’ concept of Human Adaptive Mechatronics (HAM), whose purpose is an enhancement of human performance. As one of the HAMresearch activities, an experimental skill analysis using a virtual vehicle manipulation, that is a hovercraft test task of giant slalom, was introduced. Line-of-sight of the operator was measured, and we investigated the switching of the reference targets that is needed to execute the hovercraft manipulation. From analysis of ratio of gaze-time, it was proved that adequate switching of sub-controllers by adequate early target switching was significant for skilled operation. Moreover, other analysis on identification of human control characteristics by ARX model based on information of the switching was introduced. And transitions of the characteristics were analyzed by statistics. As the results, in the case of telemanipulation with monitoring of the environment, it was confirmed as human is becoming skilled that human weakened dependency to the 2D-visual information and reinforced recognition of the virtual 3D work space. Due to limitations of space, we only introduced briefly our other researches concerning brain analysis, but it had been confirmed that an area of ocular motion control is more active than voluntary motion control at brain cortex level in case of dynamic task such as machine manipulation. Summarizing these facts, we can say that an observation (that includes adequate observation of targets for manipulation, and an visual immersion into (virtual) work space) plays significant roles to the skill.

7.

Epilogue

A word of ‘mechatronics’ was devised by a Japanese engineer T. Mori in 1969, so this word is ‘made in Japan’. And Japan is also a machine kingdom that owns most robots in the world, and is in ahead in many areas of the researches and products. As the academic, big research projects, the ‘super-mechano system’ COE-project in Tokyo Institute of Technology [24](leaders are K. Furuta [1997-2000] and S. Hirose [2000-2002]), ‘interaction and intelligence’ sakigakeproject [25](leader F. Harashima [2001-2006]), and our HAMCOE project had been executed. The name of our Tokyo Denki University includes meaning of ‘electrical’ and ‘mechanical’, that is mechatronics, and the university is a foster parent of the electric-IT market Akihabara. Considering these historical backgrounds and our circumstances, we would like to contribute to development of mechatronics.

SICE JCMSI, Vol. 1, No. 1, January 2008

Acknowledgments Research topics mentioned in this paper is supported by the Grant-in-Aid for 21st Century COE(Center of Excellence) Program in Ministry of Education, Culture, Sports, Science and Technology. The authors are grateful to the Ministry for supporting this research. The data could be obtained thanks to many participants who embraced our requests kindly, H. Igarashi and Y. Watanabe. Authors appreciate their cooperation. References [1] S. Bennett: A history of control engineering 1800-1930, Peter Peregrinus Ltd., U.K., 1979. [2] The Energy Data and Modeling Center: EDMC handbook of energy and economic statistics in Japan, The Institute of Energy Economics, Japan, 2007. [3] Ministry of Health, Labour and Welfare: http://www.mhlw.go.jp/english/index.html. [4] K. Furuta: Control of pendulum: from super mechano-system to human adaptive mechatronics, IEEE Conf. on Decision and Control, Maui, Hawaii, USA, plenary, pp. 1498–1507, 2003. [5] F. Harahima: Human Adaptive Mechatronics, IEEE Industrial Electronics Society, Raleigh, North Carolina, USA, keynote speech, 2005. [6] F. Miyawaki, K. Masamune, S. Suzuki, K. Yoshimitsu, and J. Vain: Scrub nurse robot system — intraoperative motion analysis of a scrub nurse and timed-automata-based model for surgery -, IEEE Trans. on Industrial Electronics, Vol. 52, No. 5, pp. 1227–1235, 2005. [7] S. Suzuki, K. Kurihara, K. Furuta, and F. Harashima: Assistance control on haptic systems for human adaptive mechatronics, Journal of Advanced Robotics, Vol. 20, No. 3, pp. 323–348, 2006. [8] S. Suzuki, K. Kurihara, K. Furuta, F. Harashima, and Y. Pan: Variable dynamic assist control on haptic systems for human adaptive mechatronics, IEEE Conf. on Decision and Control and European Control Conference, Seville, Spain, pp. 4596– 4601, 2005. [9] H. Igarashi, A. Takeya, S. Shirasaka, S. Suzuki, and M. Kakikura: Adaptive teleoperation system with HAM, IEEE International Workshop on Robots and Human Interactive Communication, Nashville, USA, pp. 484–489, 2005. [10] J. Åkesson and K. J. Åstr¨om: Safe manual control of the Furuta pendulum : IEEE International Conference on Control Applications, Mexico City, Mexico, pp. 890–895, 2001. [11] K. Furuta, M. Iwase, and K. J. Åstr¨om: Control of pendulum from human adaptive mechatronics, Chinese Conf. in Decision and Control, Harbin, China, plenary, pp. 15–21, 2005. [12] H. Kobayashi, T. Yasuda, S. Suzuki, and H. Takase: Nearinfrared spectroscopy and an example of HAM study: brain activation in the development of drawing skills, Int. Conf. on Control, Automation and Systems, Gyeonggi-Do, Korea, pp. 1745–1748, 2005. [13] S. Suzuki, Y. Pan, F. Harashima, and K. Furuta: Skill analysis of human in machine operation, IEEE Int. Conf. on Neural Networks & Brain, Beijing, China, pp. 1556–1561, 2005. [14] S. Suzuki, Y. Watanabe, H. Igarashi, and K. Hidaka: Human skill elucidation based on gaze analysis for dynamic manipulation, IEEE Int. Conf. on Systems, Man and Cybernetics, Montreal, Canada, 2007. [15] M. Ito: Internal model visualized, Nature, No. 403, pp. 153– 154, 2000. [16] H. Imamizu, T. Kuroda, S. Miyauchi, T. Yoshida, and M. Kawato: Modular organization of internal models of tools in the human cerebellum, National Academic Science, Vol. 100, pp. 5461–5466, 2003. [17] M. Kawato and H. Gomi: A computational model of 4 regions

[18]

[19]

[20]

[21]

[22]

[23]

[24]

[25]

25

of the cerebellum based on feedback-error learning, Biological Cybernetics, Vol. 68, No. 2, pp. 95–103, 1992. T. Tamada, S. Miyauchi, H. Imamizu, Y. Yoshida, and M. Kawato: Cerebro-cerebellar functional connectivity revealed by the laterality index in tool-use learning, NeuroReport, No. 10, pp. 325–331, 1999. D. M. Wolpert and M. Kawato: Multiple paired forward and inverse models for motor control, Neural Networks, Vol. 11, pp. 1317–1329, 1998. J. L. Cabrera and J. G. Milton: On-off intermittency in a human balancing task, Physical Review Letters, Vol. 89, No. 15, 158702/1-4, 2002. S. Suzuki and Y. Watanabe: Experimental analysis of human control characteristics on driving tasks based on gaze measurement data, Int. Conf. on Control, Automation and Systems, Seoul, Korea, 2007. M. A. Brackstone, B. Sultan, and M. McDonald: Findings on the approach process between Vehicles, Transportation Research Record, Vol. 1724, paper No.00-0831, pp. 21–28, 2000. L. Itti, C. Koch, and E. Niebur: A model of saliency-based visual attention for rapid scene analysis, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 20, No. 11, pp. 1254– 1259, 1998. K. Furuta: The fusion of control and mechanism: supermechano systems, Journal of Computing and Control Engineering, Vol. 14, No. 3, p. 5, 2003. F. Harashima: Interaction and intelligence, IEEE Int. Workshop on Robot and Human Interactive Communication, invited talk, Berlin, Germany, 2002.

Fumio HARASHIMA (Member, Fellow) Received B.S., and M.S. and Ph.D. degrees all in Electrical Engineering from University of Tokyo in 1962, 1964 and 1967, respectively. He was employed as Associate Professor at Institute of Industrial Science, University of Tokyo in 1967, and was Professor from 1980 to 1998. He was President of Tokyo Metropolitan Institute of Technology in 1998-2002, and is President of Tokyo Denki University now. He is Professor Emeritus of University of Tokyo. His research interests are in power electronics, mechatronics and robotics. He served as President of IEEE Industrial Electronics Society in 1986-1987 and 1990 IEEE Secretary. He was a member of IEEE Executive Committee and Board of Directors in 1990. In 1995 he served as Founding Editor-in-Chief of IEEE/ASME Transactions on Mechatronics, and served as President of IEE of Japan from 2001-2002. He has received a number of awards including 1978 SICE Best Paper Award, 1983 IEE of Japan Best Paper Award, 1984 IEEE/IES Anthony J. Hornfeck Award, 1988 IEEE/IES Eugene Mittelmann Award and IEEE Millennium Medal in 2000. He is a Fellow of IEEE and a Fellow of SICE.

Satoshi SUZUKI (Member) Received his B.S. degree in Control Engineering, M.S. degree in Department of Systems Science, and Ph.D. in Department of Mechanical and Control Engineering from Tokyo Institute of Technology in 1993, 1995 and 2004, respectively. From 1995 to 1999 he was a Development Engineer working for TOSHIBA Corporation at the Heavy Electrical Laboratory, Power & Industrial Systems R&D Center and the Elevator Development Center. In 1999, he moved to Tokyo Denki University, and was employed as a research associate and an assistant professor in Frontier R & D Center and the 21st Century COE (Center Of Excellence) Project Office, respectively. He is now an associate professor in Dept. of Robotics and Mechatronics. His major research interests are human-machine system, control theory and robotics. He is a member of SICE, JSME and IEEE.