Grasp potentiation by visual objects
The potentiation of grasp types during visual object categorization
Mike Tucker and Rob Ellis Department of Psychology University of Plymouth Plymouth, England.
Running Head: Grasp potentiation by visual objects
Address: Department of Psychology University of Plymouth Drake Circus Plymouth PL4 7AA England Email:
[email protected]
Grasp potentiation by visual objects
Abstract
The close integration between visual and motor processes suggests that some visuomotor transformations may proceed automatically and to an extent that permits observable effects on subsequent actions. A series of experiments investigated the effects of visual objects on motor responses during a categorisation task. In Experiment 1 participants responded according to an object’s natural or manufactured category. The responses consisted in unimanual precision or power grasps that could be compatible or incompatible with the viewed object. The data indicate that object grasp compatibility significantly affected participant response times and that this did not depend upon the object being viewed within the reaching space. The time course of this effect was investigated in Experiments 2-4b by using a go-nogo paradigm with responses cued by tones and go-nogo trials cued by object category. The compatibility effect was not present under advance response cueing and rapidly diminished following object extinction. A final experiment established that the compatibility effect did not depend on a within-hand response choice, but was at least as great with bi-manual responses where a full power grasp could be used. Distributional analyses suggest that the effect is not subject to rapid decay but increases linearly with RT whilst the object remains visible. The data are consistent with the view that components of the actions an object affords are integral to its representation.
2
Grasp potentiation by visual objects
Introduction In common with many other actions, reaching out to grasp an object requires a complex set of visual to motor transformations. The pattern of stimulation on the retina must be transformed into a series of muscle commands that result in an accurate grasp. It is natural to think of this process as a combination of two quite distinct steps: first, the formation of a visual representation of the object within the environment and second, the transmission of this visual information to the motor system, following the decision to act. Such a characterisation, however, has shortcomings. For vision to guide our actions successfully, the representations we form of the environment need to be related to our behavioural capabilities. Not only must the visual system inform the motor system but also the motor system must inform the visual system - the two processes are inter-dependent. One of the first theorists to emphasise this point was, of course, Gibson (1979) whose ecological approach to perceiving and acting had at its core the idea that the central function of vision is to provide information about the possibilities for action, or affordances. The idea of an affordance essentially bridges the gap between the behavioural capabilities of the perceiver and the visual layout of the environment. How can visual information be represented in a manner that takes account of the perceiver’s motor capacities and thereby makes information about affordances available? We have argued (Tucker & Ellis, 1998; Ellis & Tucker, under review) that one way in which this could be achieved is to involve the motor system in the (visually derived) representation of the object. Borrowing from the ecological tradition we have used the term micro-affordances (Ellis and Tucker, in review; see also Jeannerod, 1994a)
Grasp potentiation by visual objects
to refer to action-relevant object properties whose representation is in part constituted by the partial activation of the motor patterns required to interact with them. Such activation would enable behaviourally relevant visual information to form an integral part of the representation of the object, and is consistent with the visuomotor integration that occurs within the parietal cortex and its projections to the motor areas. In particular, there is much evidence that the transition from visual to motor representations is indistinct, with considerable cortical space devoted to sensorimotor integration – a process that would be an ideal candidate for representing affordances. Cortical areas of visuomotor integration Within the human and primate visuomotor system, several areas contribute to the integration of visual and motor representations. These include the posterior parietal cortex, the premotor and supplementary motor area and also the primary motor area wherein only a proportion of the cells have purely motor related activation functions (Georgopoulos 1991, 1992). These areas serve as a highly integrated visuomotor network and receive projections from the primary visual cortex via the dorsal pathway. Ungerleider and Mishkin (1982) originally characterised the dorsal system as a ‘where’ system dealing principally with spatial location and navigation, in contrast to the ‘what’ system of the ventral pathway, dealing with object identification. This functional distinction has been reinterpreted, largely as result of influential work by Goodale and Milner (Goodale, Milner, Jakobson, & Carey 1991; Goodale & Milner, 1992; Goodale, 1993) as a distinction between the perception of the world for recognition and experience and the use of vision for all aspects of visually guided action (and not only those related to the spatial location of objects).
4
Grasp potentiation by visual objects
Within the dorsal areas, a distinction between visual and motor processes is not really appropriate, as many cells have complex response functions that relate both to the visual properties of objects and to their significance for action (see also Goodale, 1998). In parietal cortex cells exist which are sensitive not merely to visuospatial stimulus properties but also to the type of grip required to interact with them (Taira, Mine, Georgopoulos, Murata, & Sakata, 1990; Sakata, Taira, Mine, & Murata, 1990; Stein, 1992). In the premotor areas, as well as primary motor cortex, a similar visuomotor gradation is seen among cell populations. It is certainly not the case, for example, that the primary motor cortex is chiefly a muscle controller, as only about one third of the cells here have any direct relationship with actual muscle output (Georgopoulos, 1991, 1992; Requin, Riehle, & Seal 1993) and may, for instance, represent the direction of a movement independently of the movement required to achieve it (Alexander & Crutcher, 1990). Both parietal and motor regions thus contain populations of cells that exhibit varying combinations of sensitivity to visual and to motor information and in which multiple representations of possible and intended movements can co-exist (Kalaska & Crammond, 1995). Goodale et al. (1991) originally based their reinterpretation of the dorsalventral functions on a dissociation, in a single patient, between an impaired ability to recognise and match the shapes of objects, and a preserved ability to visually guide actions to the same objects. This pattern of impairments, which they attributed to ventral damage, is essentially the opposite to that found in optic ataxics, a condition resulting from damage to the parietal lobes (Perenin and Vighetto, 1988; Jakobson, Archibald, Carey, & Goodale 1991). Optic ataxics generally have an intact ability to recognize objects and discriminate amongst patterns and shapes but very poor visually
5
Grasp potentiation by visual objects
guided actions to the same objects. Their disrupted motor performance includes features such as poor directional control of the arm to the target and inaccurate calibration of the fingers and wrist to the size and orientation of the target. None of these impairments arises from a purely motor deficit, but instead, they reflect a quite specific inability of the motor system to use visual information. Damage to the parietal system also disrupts the planning and rehearsal of forthcoming movements. Sirigu, Duhamel, Cohen, Pillon, Dubois, & Agid, (1996), for example, found that patients with damage to the parietal lobes were unable accurately to imagine the time needed to perform a finger opposition sequence, whilst a patient with lateralised damage to motor cortex accurately predicted the asymmetric performance of the affected and unaffected hands. Thus the visuomotor transformations occurring in parietal cortex seem to be critical not only to the execution of a task but also to its planning. The dissociation between the use of vision for action and perception has also been shown to occur in normal subjects. For instance, perceptual judgements of the size of objects becomes less accurate in the visual periphery but the calibration of the grip aperture needed to grasp the same objects remains highly accurate (Goodale & Murphy, 1997). Similarly, the effect of some visual illusions is restricted to the perceptual domain and does not affect the motor system. Both illusions of apparent size (Haffendale & Goodale, 1998; Aglioti, DeSouza, & Goodale, 1995) and apparent motion (Bridgeman, Kirk, & Sperling, 1981) have been shown to be restricted to the conscious perception of object properties, but to have little, or no effect, on the terminal accuracy of motor tasks directed at the same objects. Under degraded (monocular) conditions, however, such illusions can influence grip aperture, as
6
Grasp potentiation by visual objects
pictorial cues become more important (Marotta, J.J, DeSouza, J.F.X., Haffenden, A.M. & Goodale, M.A. (1998)). Related results come from studies of normal subjects performing reaches to targets that undergo changes in position, time-locked to the peak velocity of the saccade that normally occurs at the start of the reaching movement - so called double step paradigms. Under these conditions the conscious perception of the change in target position is prevented, but the motor system nonetheless rapidly accommodates the change on-line (Prablanc & Martin, 1992; Goodale, Pelisson, & Prablanc, 1986). These neurophysiological and neuropsychological data point to the existence of a specialised vision-to-action route in humans and primates in which visual and motor representations are highly integrated, forming a visuomotor continuum which, except at the extremes, is difficult to divide into visual or motor processing components. The extent to which this integration takes place automatically, in the sense of being independent of an active goal, such as a decision to grasp an object, remains uncertain. A perceiver’s current goals obviously provide considerable constraints on the visual information that is relevant to achieving them. There is, in fact, neuropsychological evidence that certain object properties (such as handle orientation) affect responses most when those properties are task-relevant. Riddoch & Humphreys (1998), for example, report the case of a patient with ‘anarchic hand’ syndrome who showed more interference from the orientation of a cup’s handle during a grasping task than during a pointing task. Whilst our immediate goals are important in determining what visual information is relevant, and which affordances need to be exploited to achieve them, there is also good reason to suppose that the actions an object affords need to be made available independently of our goals. This is
7
Grasp potentiation by visual objects
because goals are often specified at a relatively high level – we may decide to make a cup of coffee without at the same time having any detailed plan of the series of movements necessary to bring this about. How the detailed components of the highlevel goal are formed, depend, in part, on our ability to pick up information about the possible actions in a scene, that may, or may not, contribute toward the desired end state. To successfully perform such a goal requires an ability to generate representations of potential actions afforded by the scene (see Gross, Heinze, Seiler, & Stephan, (1999), for a neural model that incorporates hypothetical actions as a means of sensorimotor planning). The dorsal system would appear to be a good candidate for generating visuomotor representations based solely on the physical characteristics of objects. The system is not isolated, however, and may be influenced by other inputs, including those from the ventral system, in order for the correct transformations to occur. Semantic knowledge about an object’s function may be necessary to direct the dorsal system to appropriate parts of an object, such as its handle (see Goodale & Humphrey, 1998). Under conditions where goals are unspecified, or only loosely specified, there needs to be a process that enables the most relevant vision-to-motor transformations to take priority based, presumably, on the past history of the individuals interactions with visual objects. With respect to low-level object-action associations, individuals develop strong associations between objects and the kinds of grip required to interact with them (Klatzky, Pelligrino, McCloskey, & Lederman, 1993) and are capable of categorizing objects according to their associated grasp type. There is also neuropsychological (Riddoch, Humphreys, & Price, 1989; Pilgrim & Humphreys, 1991) and behavioural (Rummiati & Humphreys, 1998) evidence concerning higher
8
Grasp potentiation by visual objects
level object-action associations that points to the existence of a direct vision-to-action route that bypasses semantic knowledge. Such direct object-action associations, built up over the lifetime of the individual, would enable higher level (functional) affordances to be activated directly by visual inputs without the mediation of semantic knowledge. In the absence of a current goal, however, the kind of motor representations automatically generated by a visual object may be expected to remain fairly primitive, possibly restricted to visuomotor primitives that are common to many different actions. Visuomotor primitives of this sort correspond to what we have termed the micro-affordances of the object and would be more likely to be generated by a semantically-blind system such as the parietal-motor systems.
Behavioural evidence for motor involvement in visual representations In a previous study (Tucker & Ellis, 1998) we found that simply viewing an object could potentiate a particular response. When participants had to make left or right responses based on whether an object was upright or inverted, they were faster if the task-irrelevant horizontal orientation of the object was compatible with the required response. For example, if participants were instructed to make right responses to upright objects and left responses to inverted objects, these were more accurate and executed faster, if the depicted object had its graspable region to the right when upright and to the left when inverted. It should be emphasised, here, that the orientation of the object was entirely irrelevant to the task but influenced participant’s response times in the direction predicted. A recent study carried out by Craighero, Fadiga, Rizzolatti, & Umiltà (1998) showed that the presentation of a prime stimulus facilitated a subsequent reach and grasp movement when the orientation of the prime
9
Grasp potentiation by visual objects
was congruent with the to-be-grasped object. Although here there was a ‘motor set’ as participants were actually making precision grasp responses to the target objects, the orientation of the prime itself was irrelevant to the task. Simply viewing the irrelevant prime (it was presented at fixation) appeared to be capable of priming the angle of rotation of a subsequent reach and grasp. The effects of irrelevant stimulus location on spatial responses: The Simon effect A robust finding within the stimulus-response compatibility (SRC) paradigm is that the location of a target stimulus has an influence on the speed of spatially defined responses, despite being irrelevant to the task (Simon & Ruddel, 1969). Although originally interpreted as an instance of an orienting response, most recent accounts of the Simon effect are based on the notion of abstract location codes that the target position evokes automatically. In Kornblum’s (1990, 1994) dimensional overlap model of SRC effects, the target stimulus automatically generates a location code. Because the responses are also defined spatially (i.e. they share a spatial dimension with the target stimulus) the corresponding response code is activated. It is the congruence or incongruence between this automatically activated response code and the required response (defined by some other target property) that causes the effect. We have argued elsewhere (Tucker &Ellis, 1998) that abstract coding accounts of the Simon effect fail to give any reason why such location codes should automatically generate response codes (this criticism is not new, see Michaels (1988, 1993) for a similar criticism from an ecological perspective). Only recently has a
10
Grasp potentiation by visual objects
possible neurophysiological mechanism been put forward to account for it. This is the pre-motor theory of attention developed by Rizzolatti and co-workers (Rizzolatti, Riggio, Dascola, & Umiltà, 1987). The pre-motor theory bases spatial compatibility effects on the formation of motor programs for potential eye movements that are formed whenever covert attention is shifted. Thus a covert attentional movement involves the formation of the motor program for the saccade required to foveate the target. The model implies that the Simon effect depends on the current position of covert attention. Recently Rubichi, Nicoletti, Iani, &Umiltà, (1997) provided evidence for the premotor model of the Simon effect. By manipulating the position of participant’s covert attentional focus, in relation to the time at which responses had to be selected, they showed the Simon effect to be restricted to conditions where the last attentional shift was either congruent or incongruent with the required left or right response. As Craighero et al. (1998) suggest, the preparation of an eye movement command, that enables attention to be directed to a certain location, may reflect general visuomotor mechanisms that can be extended to more complex visual properties and potential responses. In all the experiments reported below we examined the influence of an actionrelevant (but task-irrelevant) object property on responses that mimicked either a compatible or incompatible action to the viewed object. Responses were always cued either by a separate object property that had no compatibility relation with the responses, or by an auditory signal. If simply viewing the object potentiates the specific actions it affords then one would expect the action relevant object properties to influence response times despite being task-irrelevant. The experiments thus follow the same logic as previous experiments in which we observed compatibility
11
Grasp potentiation by visual objects
relations between an object’s orientation and the hand executing a response (Tucker & Ellis, 1998) and between the orientation of an object’s opposition axis and the direction of a wrist rotation response (Ellis & Tucker, under review).
Experiment 1 In the first experiment participants had to make speeded responses based on the category of an object placed directly in front of their dominant hand. Real objects were used and a liquid crystal window controlled their exposure time. Participants had to signal whether an object was natural or manufactured by making precision or power grip responses. The experiment had two major aims. One was to examine a further object-response relation to those previously reported. The other was to test whether any observed compatibility effects required objects to be within the reaching space of the participants (Rizzolatti, Gentilucci, & Matelli, 1985; Previc 1990, 1998). Previc (1990, 1998) reviews a body of evidence that supports the view that the peripersonal space, particularly within the lower visual field, is specifically adapted for performing the visuomotor transformations associated with grasping and manipulation. Within this region of space the visual system appears, for example, to be biased toward global, rather than local, processing at crossed disparities and to rely extensively on depth and motion cues. These visual properties characterise the visual input received from the arm during reaching to a (more distant) fixated target.
12
Grasp potentiation by visual objects
Method Participants: Forty students from the University of Plymouth took part. All had normal or corrected to normal vision and were unaware of the hypothesis of interest. Apparatus and Materials: Participants sat at the end of a table with their dominant hand resting in front of their body mid-line. They held a response device, illustrated in Figure 1, consisting of two slim-line tactile feedback keyboard switches (they clicked when depressed). One of the switches was held between the index finger and thumb. The other switch was attached to the lower end of an aluminium cylinder 18mm across and 11cm long. A section of aluminium tubing ran down one side of this cylinder covering the second switch at the base and attached to the top of the cylinder. It thus acted as a hinge that triggered the switch when the whole device was squeezed. To prevent the device slipping it was covered in a thin layer of cotton gauze. Participants were instructed to wrap the middle, ring and little fingers of their dominant hand around the cylinder, much as they would hold a handle, but without using their index fingers and thumb. The two responses thus mimicked precision and power grips. The experimental stimuli were forty common objects listed in Appendix 1. Twenty were natural objects and twenty were manufactured. Within each category half the objects were small and would normally be grasped with a precision grip (grape, screw) and half were large and would normally be grasped with a power grip (cucumber, hammer). The objects were presented behind the LCD plastic screen, which could be made to change from opaque to transparent rapidly following the passage of a small alternating electric current. The screen measured 27 cm square and was surrounded by a black mask (60cm by 50cm) to prevent participants viewing around it. It was raised above the table enabling participants to rest their hands
13
Grasp potentiation by visual objects
underneath it. This allowed them to see both their hand and the object when the screen was made transparent. The experimental set up is illustrated in Figure 2. The objects were placed on a blue felt pad and participants wore ear protectors, both measures minimising any auditory cues about the manufactured or natural category of each object. The objects were presented either 15 cm from the participants’ hands (near condition) or 2000cm (far condition). Procedure: Half the participants were instructed to respond by pressing the switch between their index finger and thumb if the object was natural and by squeezing the cylinder if it was manufactured. The other participants were given the opposite instructions. They were instructed to respond as rapidly and accurately as possible. At the beginning of each trial two yellow LEDs vertically aligned at the top and bottom of the centre of the screen came on for 1 second and then went off. These served as warning cues that the screen was about to clear. Immediately after the LEDs were turned off the screen cleared and participants could see the object on the presentation pad. The screen remained transparent until a response had been made or three seconds had elapsed. Error responses were followed by a beep from the computer that was sufficiently loud to enable participants to hear it through their ear protectors. At the end of each trial the experimenter changed the object according to a randomised sequence displayed on a computer screen out of sight of the participants. The experiment was run in two blocks of 160 trials, one for each distance. The order of blocks was balanced across participants and they had about three minutes break between blocks whilst the experimenter changed the position of the presentation pad and realigned the angle of the screen to keep it at right angles to their line of sight.
14
Grasp potentiation by visual objects
Results Response Times: Error responses (4.7%) and reaction times more than 2 SDs from each participants average RT (3.9%) were excluded from the analysis. The means from the remaining data were entered into a mixed ANOVA with within participants factors of Object Size (Small or Large), Response (Precision or Power) and Distance (Near or Far) and the between participants factor Mapping (Natural = precision response/ Manufactured = power response or Natural = power response / Manufactured = precision response). For the effects of most relevance to the hypotheses under consideration we give both analyses by participants as well as by objects and include a calculation of Min F’(Clark, 1973) to provide a conservative test of the ability of the effect to generalise to a new sample of participants and objects1. The analysis revealed two highly significant main effects of object size and distance. Large objects were reacted to faster (M=481) than small objects (M=499), [F(1,38)=63.18, p
