Collection of Visual Data in Climbing Experiments for ...

Collection of Visual Data in Climbing Experiments for Addressing the Role of Multi-modal Exploration in Motor Learning Efficiency Adam Schmidt(B) , Dominic Orth, and Ludovic Seifert University of Rouen, Mont-Saint-Aignan, France [email protected]

Abstract. Understanding how skilled performance in human endeavor is acquired through practice has benefited markedly from technologies that can track movements of the limb, body and eyes with reference to the environment. A significant challenge within this context is to develop time efficient methods for observing multiple levels of motor system activity throughout practice. Whilst, activity can be recorded using video based systems, crossing multiple levels of analysis is a substantive problematic within the computer vision and human movement domains. The goal of this work is to develop a registration system to collect movement activity in an environment typical to those that individuals normally seek to participate (sports and physical activities). Detailed are the registration system and procedure to collect data necessary for studying skill acquisition processes during difficult indoor climbing tasks, practiced by skilled climbers. Of particular interest are the problems addressed in trajectory reconstruction when faced with limitations of the registration process and equipment in such unconstrained setups. These include: abrupt movements that violate the common assumption of the smoothness of the camera trajectory; significant motion blur and rolling shutter effects; highly repetitive environment consisting of many similar objects. Keywords: Visual scene understanding · Data collection · Registration system · Motor learning · Gaze tracking · Rock climbing

1

Introduction

Understanding of skilled performance in human endeavor, such as sport, has benefited markedly from technologies that can track movements of the limb, body and eyes with reference to key surfaces, objects, events and significant others [5]. Indeed, ongoing technological advances continues to increase capacities of researchers to test their assumptions in the actual conditions, under which the relevant expertise is expressed [11]. In conjunction, image processing approaches for automatic detection of movement relative to environmental reference frames have also extended the timescales over which the evolution of c Springer International Publishing AG 2016 ⃝ J. Blanc-Talon et al. (Eds.): ACIVS 2016, LNCS 10016, pp. 674–684, 2016. DOI: 10.1007/978-3-319-48680-2 59

Multi-modal Exploration in Motor Learning Efficiency

675

expertise can be observed (such as learning and development) [1]. Within this context, a remaining challenge to overcome is to develop time efficient methods for behavioral analysis at multiple levels relative to the environmental reference (e.g., body-gaze-environment coordination). Thus, the current work presents automatic image processing methods for combining limb, body and gaze movements as they evolve through practice in a sport performance task relative to a three dimensional environmental reference. A general purpose of observing practice dynamics is to address the efficiency of motor learning (the stabilization of new actions) under a given set of constraints (i.e., the combined task, environmental and individual boundary condition) [12]. Indeed, the efficiency of learning is highly influenced by the individuals level of experience under similar task and environmental constraint [4]. For example, different sub-systems (such as the visual or muscular systems), limited in their rate of adaptation to practice, strongly influence the learning efficiency [3,23]. Whilst efficiency provides an estimate of skilled performance [12], criteria for quantifying the novel (or exploration) component in motor learning is an ongoing problematic [15]. One approach, taken in this study, is to establish the degree of novelty in performer-environment interactions across trials and determine co-related levels of efficiency [10]. For example, in climbing tasks, the purpose of exploratory activity is to assist the individual to adapt to a more or less vertical and ever changing structure of a climbing surface to complete the route [14]. In climbing, exploration is functional (supports the individuals goals) across multiple levels as related to constraints that emerge though learning [8], such as determining postural stability to avoid falling or efficient progress to further optimize performance [14]. For example, it has been shown that visual search (non-physical exploration of a route), is used by experienced climbers to identify rest locations [18]. Haptic exploration of holds reduces uncertainty in reach-ability and grasp-ability of supports [13,17] and a key skill is to support exploration whilst maintaining an efficient climbing trajectory [22]. Finally, emergence of new climbing actions, through exploration, may also be observed over practice [20]. For example, from one trial to the next, different visual search patterns, route pathways, body orientations or grasping patterns can be developed and potentially stabilized [20]. Thus, during practice the nature of motor learning may be better understood by evaluating the levels at which exploration emerges. The challenge, therefore, in this study was to assess exploration across gaze, limb and postural levels of analysis. Through observing the evolution of exploration over practice, we aimed to determine its relative importance in supporting performance efficiency. The additional interest in recording exploration across multiple levels, was to determine in what ways gaze, limb and hip levels of exploration might evolve at different rates.

676

2 2.1

A. Schmidt et al.

Methods Variables of Interest

The goal of this work was to develop a registration system and procedure to collect data necessary for studies of the skill acquisition and exploration of the climbing wall by skilled climbers. As there are different ways of exploring the following data needed to be recorded: – the gaze direction and head pose required for the analysis of the visual exploration; – the positions of the limbs on the climbing wall for the analysis of the haptic exploration; – the position of the hips relative to the limbs for the analysis of the postural exploration. Traditionally, determining exploratory behavior in climbing requires that a given level of analysis is assessed with respect to motion detected at the hip (i.e., hip displacement being the overall objective of the climber). For example, visual fixations and limb actions in climbing are considered exploratory’ when they are not associated to hip motion and performatory when they are associated to hip motion [13,17]. The main limitation in current work for determining hip mobility is operator involvement. For example, criteria for mobility have included manual frame-by-frame video notation using criteria statements like: “progress of the hips was observed” [2] whereas, criteria for immobility have included: “no discernible movement in pelvic girdle” [24]. Since hip mobility is determined as an acceptable level of displacement over time, one solution is directly using the hip velocity and applying an appropriate threshold (for example [21]). Finally, qualitative assessment of kinks or knots in hip trajectories over trials of practice, are suggestive of exploration at the hip level, often referred to as ‘route finding’ [4]. In this case, assessment of the hip path, from one trial to the next, provides the area explored by the hip. 2.2

Constraints

The conditions present at the climbing wall, the characteristics of the climbing task, and the required data imposed several constraints on the selection of the equipment used for the experiment. Wall climbing at advanced levels of difficulty is an activity requiring performing at the limits of strength, agility and endurance. Therefore, any additional equipment worn by the climbers increases the difficulty of the approach and alters their behavior due to limited field of view, restricted movement or even subtle shifts in the center of balance. Therefore, the amount, size and weight of the wearable equipment needed to be kept light and unobtrusive. The artificial climbing wall is a very demanding environment for a motion tracking system due to the amount of dust in the air and high presence of metal components in the walls (e.g. hold screws and mounting holes). Moreover, due

Multi-modal Exploration in Motor Learning Efficiency

677

to the organizational and safety reasons, the cameras could not be installed permanently, their possible mounting locations were limited to certain areas of the opposing wall, and preferably no wired connection were to be used. Finally, the deployment, calibration and dissembling of the registration system had to be performed repeatedly and thus the procedure had to be as efficient and effortless as possible. 2.3

Instrumentation

After considering the requirements and constraints it was decided, that the system comprise of static external cameras for the reconstruction of the climbing wall and tracking of the LED markers attached to the climbers body. For detecting gaze positioning, the climber would wear eye-tracking glasses with a portable recording unit. The GoPro Hero cameras were used in the registration system due to their robustness and reliability. Moreover, they can be controlled remotely, operate for prolonged periods on battery power supply and can record high-resolution (1920 × 1080) videos at 25 fps. The cameras were mounted on the opposing wall (i.e., 5 m away from the climbing surface) (Fig. 1) in a way making sure, that regardless of the climber’s position, the markers could be visible from at least two cameras at the same time. The biggest limitations of the GoPro cameras is that the manufacturer does not provide an SDK allowing users to directly control the cameras through

Fig. 1. The external cameras placement on the opposing wall.

678

A. Schmidt et al.

Fig. 2. The equipment of the climbers: the LED markers - left, the SMI-ETG - right.

custom software. As a result, a wireless controller had to be used, which caused slight de-synchronization of the cameras. To solve that issue an additional procedure for synchronization of the video streams was proposed (Sect. 3.2). The SensoMotoric Instruments Eye-Tracking Glasses (SMI-ETG) were used to record the video sequences of the scene observed by the climbers during their ascent approaches as well as to track the gaze direction during the experiments. The scene camera of the glasses records videos at 1280 × 960 pixels with the frequency of 24 fps. The binocular gaze direction data is recorded with a frequency of 60 Hz. The glasses provide automatic parallax compensation and track the gaze with the accuracy of 0.5◦ . The glasses weigh 68 g and are used with the recording unit weighing additional 246 g. Their main drawback is that the scene camera uses a rolling shutter which unfortunately results in presence of distortions such as wobble, skew and smear during fast head movements. Additionally, the climbers were equipped with LED stripes attached to their forearms and ankles and a LED lamp placed on their back at the hip level. All the LED elements can be easily detected on the video frames and used to triangulate the positions of limbs and hips. Figure 2 shows the placement of the equipment on the climber’s body. The system also supports 3-point, non-colinear, calibration prior to data collections and can be updated in post processing if needed. 2.4

Participants

A group of 20 experienced climbers were recruited on the basis of self-reported after practice ability levels between 6b–7a on the French rating scale of difficulty [6]. All participants provided informed consent and the study was conducted with local university ethical committee approval and in accordance with the Geneva Convention. Additionally, age, anthropometric characteristics (standing height, arm-span, hip-leg distance, neck-hip distance) and climbing histories (climbing age, best on-sight and after practice ability level on boulder routes)

Multi-modal Exploration in Motor Learning Efficiency

679

were recorded. Finally, body weight was recorded using a portable electronic scale whilst fully equipped (climbing gear and instrumentation). 2.5

Routes and Experimental Design

Participants were required to climb two different routes repeatedly and in succession over six trials. Each route was designed at a 6b F-RSD level. The level was confirmed by consensus between two qualified route-setters [6]. The routes were designed to be relatively challenging for participants in order to promote a learning effect [7] (Fig. 3). Participants were assigned into one of two different conditions of treatment (i.e., 10 participants per condition) by random allocation. That is, the list of 20 participants were randomly sorted to an order of position in a list from 1–20 and every odd numbered participant climbed under condition 1 and every even ordered participant climbed in condition 2. In the first condition participants were afforded a two minutes period prior to each climb to non-physically practice the route from the ground (referred to as ‘route preview’ and is a typical feature of a climbers pre-ascent preparation [16]). The purpose of this manipulation was to allow the group to visually explore the route prior to attempting to climb it. It was anticipated that this would allow the ‘with-preview’ group of participants to plan their ascent before climbing, thus reducing the degree of haptic, postural and visual exploration required during the climb and throughout practice. The group of climbers assigned to condition 2, were not afforded a preview prior to climbing. In this case it was anticipated that a greater degree of exploration would be evident in the, ‘no-preview’, group during climbing and throughout practice. The route that was practiced first was counterbalanced across participants to control for any order of treatment effects. That is, five participants from each group carried out their first six trials of practice on

Fig. 3. The climbing routes used in the experiments

680

A. Schmidt et al.

Route 1, before carrying out six trials of practice on Route 2, and, the remaining participants started on Route 2 and ended on Route 1. 2.6

Procedure

Across all testing sessions participants upon arrival were immediately fitted with recording apparatus and materials (gaze recording unit and glasses and LED markers). All participants used their personal climbing shoes. They were then required to undertake a 10-min climbing specific warm-up in order to habituate to the equipment and warm-up their hands, feet and body. Their anthropemtric details, climbing histories and informed consent were then recorded. Prior to undertaking experimental climbs they were instructed as to the general procedures and given the global task goal to: climb the route to end and through practice attempt to climb as fluently as possible. In the condition where participants were given a preview, they were instructed that they would have up to two minutes to visually inspect the route from the ground. They were also informed they were not allowed to physically touch the holds during preview, but were free to move around as they wished. All participants, prior to the commencement of recording, were shown the location of each hold and volume in the route. During this they were explicitly requested not to simulate the climb. Prior to each trial the calibration of the glasses were updated using 3-, nonco-linear, points. Accuracy was verified using the portable recording unit interface. The recording unit was then activated to begin data collection. The four GoPro cameras were then initiated to begin recording using the wi-fi remote. The participants, who standing 3 m from the wall, were then required to look directly at a red LED that was flashed 5 times to act as a synchronization for the GoPros and SMI scene camera. In conditions ‘with-preview’, to help locate the end of preview in post-processing the participants, after completing their preview looked at a single LED flash. Participants then whilst touching with both hands the first hold of the route, fixated on a cross, marked on the wall for this purpose, for five seconds (this allowed for a final fail safe to make any offset corrections to the gaze tracking during post-processing). They were then instructed to begin to climb when they liked. The beginning of the climb was marked as when both feet had left the ground and the end was marked as when both hands made contact with the last hold. In cases where the climbers fell, the moment of last contact with either hand with the route was taken as the end of the trial. All recording equipment was then stopped. Between each climb, a seated 4-min rest was enforced to minimize effects of fatigue on performance. Participants were also required to solve a hand-held physics game during each rest. The purpose of this activity was to prevent them mentally simulating/practicing the route during the rest periods. They were also explicitly told not to mentally practice during rest periods and especially just after finishing each climb (since this is when climbers also tend to visually inspect).

Multi-modal Exploration in Motor Learning Efficiency

3 3.1

681

Data Processing Calibration

All the cameras used in the system (the GoPros and the scene camera of the SMI-ETG) were calibrated according to the Heikkila and Silv´en camera model [9] using the freely available OpenCV library. As the external cameras were reassembled before each experiment their pose regarding the climbing wall had to be reestablished every time. In order to do that the multi-camera calibration method proposed by Schmidt et al. [19] was adapted. The images used for the calibration contain both the calibration marker of known size held in different pose while being observed by all the cameras and the empty scene containing flat surfaces of known structure (the climbing wall with regularly spaced mounting points). Moreover, the images of the climbing wall itself, with the mounting holes defining the coordinate system were used for the calibration (Fig. 4).

Fig. 4. Exemplary image used for the calibration of the camera poses

682

A. Schmidt et al.

3.2

Synchronization

Despite being triggered with the same remote controller the GoPro cameras recorded video sequences are de-synchronized. To synchronize all the videos (including the scene camera of the SMI-ETG) each trial started with a red

Fig. 5. The detection of the blinks - two consecutive frames and the thresholded differential image

Fig. 6. Exemplary, synchronized images from the GoPro cameras (top) and the scene camera of the SMI-ETG (bottom) (Color figure online)

Multi-modal Exploration in Motor Learning Efficiency

683

LED light being blinked 5 times. Such a signal is easily detectable on all the videos through the thresholding of the differential images. The time-shifts for the GoPro recordings relative to the SMI-ETG glasses recording are found by maximizing the correlation between the detected time series of blinking. The exemplary, synchronized images from all five cameras are presented in Fig. 6.

4

Conclusions

The paper presents the experimental setup and the procedure for registration of the visual data during climbing on an artificial, indoor climbing wall. The further work will focus on developing methods for estimating the trajectories of the climbers’ heads, limbs and hips as well as tracking the gaze fixations projected onto the climbing wall (and particular holds) (Fig. 5). The results of this will be used in further research on human skill acquisition including the understanding of the exploration across tactile, visual and postural forms. However, the visual data may be also of interest to the wider computer vision community (especially in the field of SLAM and visual odometry). The data recorded using the scene camera comprise images observed by humans involved in a physical activity. The main difficulty of the trajectory reconstruction lie in the limitations of the registration process and equipment in such unconstrained setups: – abrupt movement of the head clearly violating the common assumption of the smoothness of the camera trajectory; – significant motion blur and rolling shutter effect caused by the limitations of the SMI-ETG; – highly repetitive environment consisting of many similar objects. In order to facilitate research in both computer vision and human movement science the data will be made publicly available according to the open science paradigm.

References 1. Barris, S., Button, C.: A review of vision-based motion analysis in sport. Sports Med. 38(12), 1025–1043 (2008) 2. Billat, V., Palleja, P., Charlaix, T., Rizzardo, P., Janel, N.: Energy specificity of rock climbing and aerobic capacity in competitive sport rock climbers. J. Sports Med. Phys. Fitness 35(1), 20–24 (1995) 3. Cordier, P., Dietrich, G., Pailhous, J.: Harmonic analysis of a complex motor behavior. Hum. Mov. Sci. 15(6), 789–807 (1996) 4. Cordier, P., France, M.M., Pailhous, J., Bolon, P.: Entropy as a global variable of the learning process. Hum. Mov. Sci. 13(6), 745–763 (1994) 5. Davids, K., Button, C., Ara´ ujo, D., Renshaw, I., Hristovski, R.: Movement models from sports provide representative task constraints for studying adaptive behavior in human movement systems. Adapt. behav. 14(1), 73–95 (2006)

684

A. Schmidt et al.

6. Draper, N., Dickson, T., Blackwell, G., Fryer, S., Priestley, S., Winter, D., Ellis, G.: Self-reported ability assessment in rock climbing. J. Sports Sci. 29(8), 851–858 (2011) 7. Guadagnoli, M.A., Lee, T.D.: Challenge point: a framework for conceptualizing the effects of various practice conditions in motor learning. J. Motor Behav. 36(2), 212–224 (2004) 8. Guerin, S., Kunkle, D.: Emergence of constraint in self-organizing systems. Nonlinear Dyn. Psychol. Life Sci. 8(2), 131–146 (2004) 9. Heikkila, J., Silv´en, O.: A four-step camera calibration procedure with implicit image correction. In: 1997 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1997. Proceedings, pp. 1106–1112. IEEE (1997) 10. Hristovski, R., Davids, K., Araujo, D., Passos, P.: Constraints-induced emergence of functional novelty in complex neurobiological systems: a basis for creativity in sport. Nonlinear Dyn.-Psychol. Life Sci. 15(2), 175–206 (2011) 11. Mann, D.T., Williams, A.M., Ward, P., Janelle, C.M., et al.: Perceptual-cognitive expertise in sport: a meta-analysis. J. Sport Exerc. Psychol. 29(4), 457–478 (2007) 12. Newell, K.M., Latash, M., Turvey, M.: Change in movement and skill: learning, retention, and transfer. In: Dexterity and Its Development, pp. 393–429 (1996) 13. Nieuwenhuys, A., Pijpers, J.R., Oudejans, R.R., Bakker, F.C., et al.: The influence of anxiety on visual attention in climbing. J. Sport Exerc. Psychol. 30(2), 171–185 (2008) 14. Orth, D., Davids, K., Seifert, L.: Coordination in climbing: effect of skill, practice and constraints manipulation. Sports Med. 24(2), 255–268 (2015) 15. Pacheco, M.M., Newell, K.M.: Transfer as a function of exploration and stabilization in original practice. Hum. Mov. Sci. 44, 258–269 (2015) 16. Pezzulo, G., Barca, L., Bocconi, A.L., Borghi, A.M.: When affordances climb into your mind: advantages of motor simulation in a memory task performed by novice and expert rock climbers. Brain Cogn. 73(1), 68–73 (2010) 17. Pijpers, J., Oudejans, R.R., Bakker, F.C., Beek, P.J.: The role of anxiety in perceiving and realizing affordances. Ecol. Psychol. 18(3), 131–161 (2006) 18. Sanchez, X., Boschker, M., Llewellyn, D.: Pre-performance psychological states and performance in an elite climbing competition. Scand. J. Med. Sci. Sports 20(2), 356–363 (2010) 19. Schmidt, A., Kasinski, A., Kraft, M., Fularz, M., Domagala, Z.: Calibration of the multi-camera registration system for visual navigation benchmarking. Int. J. Adv. Robot. Syst. 11, 1–12 (2014) 20. Seifert, L., Orth, D., H´erault, R., Davids, K., Davis, T., Passos, P., Dicks, M., Weast-Knapp, J.: Affordances and graspingpatterns variability during rock climbing. In: Davis, T., Passos, P., Dicks, M., Weast-Knapp, J. (eds.) Studies in Perceptionand Action XII: Seventeenth International Conference on Perceptionand Action, pp. 114–118. Psychology Press, Taylor & Francis, Estori (2013) 21. Seifert, L., Wattebled, L., Herault, R., Poizat, G., Ad´e, D., Gal-Petitfaux, N., Davids, K.: Neurobiological degeneracy and affordance perception support functional intra-individual variability of inter-limb coordination during ice climbing. PloS ONE 9(2), e89865 (2014) 22. Seifert, L., Wattebled, L., Orth, D., LHermette, M., Boulanger, J., Davids, K.: Skill transfer specificity shapes perception and action under varying environmental constraints. Hum. Mov. Sci. 48, 132–141 (2016) 23. Thelen, E.: Motor development: a new synthesis. Am. Psychol. 50(2), 79–95 (1995) 24. White, D.J., Olsen, P.D.: A time motion analysis of bouldering style competitive rock climbing. J. Strength Conditioning Res. 24(5), 1356–1360 (2010)