www.nature.com/scientificreports
OPEN
Received: 24 October 2017 Accepted: 1 June 2018 Published: xx xx xxxx
The relationship between reinforcement and explicit control during visuomotor adaptation Olivier Codol , Peter J. Holland & Joseph M. Galea The motor system’s ability to adapt to environmental changes is essential for maintaining accurate movements. Such adaptation recruits several distinct systems: cerebellar sensory-prediction error learning, success-based reinforcement, and explicit control. Although much work has focused on the relationship between cerebellar learning and explicit control, there is little research regarding how reinforcement and explicit control interact. To address this, participants first learnt a 20° visuomotor displacement. After reaching asymptotic performance, binary, hit-or-miss feedback (BF) was introduced either with or without visual feedback, the latter promoting reinforcement. Subsequently, retention was assessed using no-feedback trials, with half of the participants in each group being instructed to stop aiming off target. Although BF led to an increase in retention of the visuomotor displacement, instructing participants to stop re-aiming nullified this effect, suggesting explicit control is critical to BF-based reinforcement. In a second experiment, we prevented the expression or development of explicit control during BF performance, by either constraining participants to a short preparation time (expression) or by introducing the displacement gradually (development). Both manipulations strongly impaired BF performance, suggesting reinforcement requires both recruitment and expression of an explicit component. These results emphasise the pivotal role explicit control plays in reinforcementbased motor learning. In a constantly changing environment, our ability to adjust motor commands in response to novel perturbations is a critical feature for maintaining accurate performance1. These adaptive processes have often been studied in the laboratory through the introduction of a visual displacement during reaching movements2. The observed visuomotor adaptation, characterized by a reduction in performance errors, was believed to be primarily driven by a cerebellar-dependent process that gradually reduces the mismatch between the predicted and actual sensory outcome (sensory prediction error) of the reaching movement1,3,4. Cerebellar adaptation is a stereotypical, slow and implicit process and therefore does not require the individual to be aware of the perturbation to take place5,6. However, a single-process framework cannot account for the great variety of results observed during visuomotor adaptation tasks7. Specifically, it has recently been shown that several other non-cerebellar learning mechanisms also play a pivotal role in shaping behaviour during adaptation paradigms such as explicit control8,9 and reward-based reinforcement10–15. Explicit control usually consists of employing simple heuristics such as aiming off target in the direction opposite to a visual displacement, to quickly and accurately account for it5. However, this requires explicit knowledge of the perturbation, which in turn usually requires experiencing large and unexpected errors8,16–18. Explicit control contrasts with cerebellar adaptation in that it is idiosyncratic9, volitional, and can lead to fast adaptation rates19. Importantly, in this work, we consider explicit control as the contribution to performance that can be suppressed (or expressed) by participants upon request20, as opposed to the additional requirement of being able to verbalise a strategy. Critically, cerebellar adaptation takes place regardless of the presence or absence of any explicit process, even at the cost of accurate performance5. More recently, another putative mechanism contributing to motor adaptation has been proposed, through which the memory of actions that led to successful outcomes (hitting the target) is strengthened, and therefore more likely to be re-expressed14,21. Such reinforcement is considered to be an implicit process, but distinct from cerebellar adaptation in that it is not driven by sensory prediction error but task success or failure10,11. To examine this phenomenon, several studies employed a binary, hit-or-miss feedback (BF), paradigm which School of Psychology, University of Birmingham, Birmingham, UK. Correspondence and requests for materials should be addressed to O.C. (email:
[email protected])
SCientifiC REPOrTS | (2018) 8:9121 | DOI:10.1038/s41598-018-27378-1
1
www.nature.com/scientificreports/ promotes reinforcement over cerebellar processes11,12,22. For example, in one study, participants receiving only binary feedback following successful adaptation expressed stronger retention than participants who had received a combination of visual and binary feedback12. The authors argued this could be due to greater involvement of reinforcement-based process that is less susceptible to forgetting12. With the multiple processes framework of motor adaptation, the question of interaction between the distinct systems becomes central to understanding the problem as a whole, and it remains an under-investigated question for reward-based reinforcement. In decision-making literature, it has long been suggested that two distinct “model-based” and “model-free” systems interact23,24 and even require communication to be optimal25,26. Interestingly, model-based processes share many characteristics with explicit control during motor adaptation, in that they are both more explicit, rely on an internal model of the world (explicit control27,28; model-based decision-making29), and are closely related to working memory capacity (explicit control30,31; model-based decision-making32,33) and pre-frontal cortex processes (explicit control30; model-based decision-making26,34). On the other hand, the concept of reinforcement in motor adaptation comes directly from the model-free systems described in decision-making literature28, and is often labelled as such. It is considered more implicit, relies on immediate action-reward contingencies and is thought to recruit the basal ganglia in both cases (visuomotor adaptation22; decision-making23). Despite these interesting similarities, unlike model-based and model-free decision-making, the relationship between explicit control and reinforcement during visuomotor adaptation paradigms is currently unknown. Evidence of this relationship exists from a recent study which showed participants needed to experience a large reaching error in order to express a reinforcement-based memory18. In addition, there is a wealth of evidence which shows explicit control also requires experiencing large errors16,17,27. Thus, it is possible that the formation of a reinforcement-based memory requires, or at least benefits, from some form of explicit control35. To address this possibility, we first examined the contribution of explicit control to the reinforcement-based improvements in retention following binary feedback12,22. Secondly, we used a forced reaction time (forced RT) paradigm36 to investigate the importance of being able to express explicit control when encountering binary (reinforcement-based) feedback.
Results
Experiment 1: Explicit control occurs during reinforcement-based retention. We first sought to
investigate the role of explicit control in the retention of a reinforced visual displacement memory. In experiment 1, participants made fast ‘shooting’ movements towards a single target (Fig. 1a). After a baseline block involving veridical vision (60 trials) and an adaptation block (75 trials) where a 20° counter-clockwise (CCW) visuomotor displacement was learnt with online visual feedback (VF), participants experienced the same displacement for 2 blocks (asymptote blocks; 100 trials each) with either only binary feedback (BF group, Fig. 1b, top) to promote reinforcement, or BF and VF together (VF group, Fig. 1b, bottom). Following this, retention was assessed through 2 no-feedback blocks (100 trials each), during which both BF and VF were removed. Before these no-feedback blocks, half of the participants were told to “carry on” as they were (“Maintain” group) and the remaining ones were informed of the nature of the perturbation, and to stop re-aiming off target to account for it (“Remove” group). Thus, there were four groups: BF-Maintain, BF-Remove, VF-Maintain and VF-Remove (N = 20 for each group). Group performance is shown in Fig. 2a. All groups showed similar baseline performance (Fig. 2b; H(3) = 4.59 p = 0.20; see Methods for detailed information on statistical analysis), and had fully adapted to the visuomotor displacement prior to the asymptote/reinforcement blocks (average reach angle in the last 20 trials of adaptation, Fig. 2c; H(3) = 2.56 p = 0.46). Interestingly, at the start of the first asymptote block, participants in both BF groups showed a dip in performance, effectively drifting back toward baseline before adjusting back and returning to plateau performance. This “dip effect” was completely absent in the VF groups, and has previously been observed independently of our study when switching to BF after a displacement is abruptly introduced12. Therefore, success rate was compared independently across groups in the first 30 trials (Fig. 2d) and the remaining 170 trials (Fig. 2e) of the asymptote block. Both BF groups exhibited lower success rates than the VF groups in the early asymptote phase (H(3) = 46.79, p
