The Jinmutl of General Psychology, 2004. /jy(4). 365-377
Space- and Object-Based Attention Depend on Motor Intention MARTIN H. FISCHER Department of Psychology University of Dundee, Scotland NOEM! HOELLEN Department of Psychology University of Halle/Saale. Gertnany
ABSTRACT The authors investigated the impact of diiferent motor demands on space- and object-based attention allocation. Responses to targets were either lifting a linger, or pointing to the target, or grasping a clay object placed on Ibe target location. Reaction times and movement times were recorded to assess ctwert and overt attention, respectively. Both reaction times and movement times sbowed more space-based attention for pointing tban for finger lifting and more object-based attention for grasping tban for pointing. That result supports the view that visual selectivity is tuned to specific motor intentions (H. Bekkering & F. W. Neggers. 2002) and illustrates the tight coupling of perception and action. Key words: finger lifting, grasping, perception and aclion. pointing, space-based and object-based attention
ATTENTION IS A COGNITIVE MECHANISM that helps us to select relevant information from the environment. Selection first occurs covertly, in the absence of overt movements, and is then normally expressed through eye, head, or hand movements to the selected location (e.g.. Deubel & Schneider, 1996; Deubel, Schneider. & Paprotta, 1998; Hoffman & Subramaniam. 1995). Much research is aimed at understanding how covert attending ulone reliably improves perceptual processing and what cognitive representations are accessed by that mechanism (see Pashler, 1998. for review). In an influential study. Egly, Driver, and Rafal (1994) measured the behavioral consequences of covert attention shifts to infer the cognitive representaThis research was supported by the British Academy (LRG-M696). The authors thank Nadine Kloth for her assistance with pilot work. Address correspondence to Martin Fischer. Department of Psychology. Universitx of Dundee. Dundee DDl 4HN. Scotland:
[email protected] (e-mail). 365
366
The Journal of General Psychology
lions on which visual attention operates. They presented two parallel rectangles such that their four ends were precisely the same distance apart. One end was briefly cued with a visual on.set. and participants pressed a key when they detected a target that was subsequently presented in one of the tour ends. Egly et al. compared detection speed in three conditions: (a) valid (V), in which the target appeared in the cued end; (b) invalid same-object (IS), in which the target appeared in the opposite end of the cued rectangle; and (c) invalid differentobject (ID), in which the target appeared in the end of the uncued rectangle closest to the cued end. Egly et al. replicated the standard finding of fastest detection in V trials, indicating that attention shifts across a cognitive representation of visual space from a cued location to uncued target locations, thus increasing detection latencies (e.g.. Posner. 1980). It is important to note that the authors also showed that still more time was required to detect ID targets compared with IS targets, despite their identical distance from the cued location. That result suggested that visual attention operates on cognitive representations of ohjects as well as of space. Traditionally, attention mechanisms have been investigated through perceptual manipulations, such as the aforementioned perceptual grouping of two locations through an outline rectangle, while at the same time limiting response requirements to simple key presses. But the results of a seminal study (Tipper. Lortie. & Baylis. 1992). in which manual pointing responses were used, revealed that visual selection can occur relative to the responding hand. Participants in that study pointed to and pressed a target hutton while they ignored distracter buttons that were illuminated simultaneously with the target button but in a different color. Task-irrelevant distracters interfered with pointing more when they were near the hand or between the start and target location of the movement than when they were hehind the target, regardless of movement direction. It is important to note that such conflgural effects on attention were obtained only with pointing actions and not with verbal responses (Meegan & Tipper, 1999), thus highlighting the task specificity of visual selection. Bekkering and Neggers (2002) recently compared attentional adjustments with different action intentions. Their participants pointed at or grasped a target object with a defined orientation and color from among a large set of similar distracter objects. The results showed that they made more eye fixations on distracters with the relevant orientation before they initiated grasping responses than they did for pointing responses. Thus, orientation selection (but not color selection) improved when participants intended to grasp the target. That intriguing finding is in agreement with the notion of separate visuomotor channels that process visual information as required by either the transport or manipulation components of goal-directed hand movements (e.g.. Paulignan & Jeannerod. 1996). Specifically, the grasping task required orientation information for the successful completion of the manipulation component of the action. That task requirement was reflected in participants' allocation of overt attention, as
Fischer &Huellen
367
expressed in their eye fixations. The result underlined the impact of action intention on overt visual attention. In the present study, we combined that previous work on the coupling of visual attention and action intention with the method of Egly et al. (1994) for measuring object- and space-based covert attentional effects. To see if covert attention was allocated in a task-specific manner, we compared visual selectivity related to three types of intended actions: (a) simple fmger lifting, (b) goaldirected pointing, and (c) grasping. Finger lifting was used as a baseline task to measure covert anention allocation in the absence of task-related overt attention shifts. Pointing was used to introduce an overt attention shift and to explore how that requirement would influence covert attention allocation. Finally, grasping was used to explore whether the requirement to manipulate a part of the object would further influence covert attentional strategies. By measuring response initiation times (RT) in V, IS, and ID trials for each task, we expected to fmd more space-based attention in pointing relative to finger lifting because goal-directed pointing requires attention shifts to the movement target (Deubel et al.. 1998). We also expected to fmd more object-based attention in grasping compared with pointing because the additional manipulation component of the forthcoming response should enhance attention to object attributes such as size and orientation (Bekkering & Neggers, 2{K)2; Paulignan & Jeannerod, 1996). Thus, we manipulated attention both endogenously (by task instruction) and exogenously (with peripheral cues), and we obtained measures of both covert attention (using RT) and overt attention (using movement time IMT]). Method Participants Twenty-four participants (17 women, 7 men; mean age = 26 years, range: 20-38; 1 left-handed according to self-report) with normal or corrected vision took part in the study. The authors also participated in the study. All the participants except the authors gave informed consent and received £5. Apparatus The participants sat in a quiet room on a height-adjustable chair in front of a touch monitor. The monitor (Philips 4 CM 2299 Autoscan Professional Color, 20-in. diagonal screen size) was tilted back 45° and equipped with a touch interface (ELO-Touch, controlled with ELO-Graphics MonitorMouse 2.0). The spatial resolution of that interface corresponded to the pixel resolution of the screen (1,024 X 768 pixels). An Apple 4400/200 PowerMac controlled stimulus presentation and recorded fmger lifting, pointing, and grasping contacts from pressure changes on the touch screen.
368
The Journal of General Psychology
Stimuli The stimuli were presented on a black background (see Figure I). At an average viewing distance of 60 cm. two parallel white outline rectangles subtended I.9'*x 11.3" of visual angle and were positioned 2.9" from the display center, which was occupied by a green start box (1.9° x 1,9°). The rectangles were oriented either horizontally or vertically with their four ends occupying precisely the same locations throughout the experiment. The attention cue was a white square and the target was a green square (both 1.9" x 1.9"). In the grasping condition, a small object made of blue tack (7 mm diameter. I -2-g weight) was stuck on each of the four rectangle ends without covering the target locations. The tacks were small enough to allow clear visibility of cues and targets. They were directly connected to the rectangles and hence considered as parts of the rectangle objects. Task and Procedure Each participant was individually calibrated to the touch screen. A trial began with a display containing two rectangles and the start box until the participants touched the start box with ihcir right index fmger. They were cautioned not to
Siart display Time
Lining
Puinliii"
Grasping
FIGURE 1. Schematic of the event sequence in the three experimental conditions finger lifting, pointing, and grasping. Rectangles were only outlined but are shown here in white for clarity. ISI = interstimulus interval.
Fischer & Hoeileii
369
obstruct their view on the rectangles and to maintain contact until a target appeared. Thus, hand orientation covaried with rectangle orientation. The screen contact changed the color of the start box to black and triggered a foreperiod. After 1,000 ms, the cue appeared tor 50 ms. followed after another 150 ms by the target. The target remained visible until the trial had timed out or the participant had responded in accordance with the task instruction. In separate blocks, the participants either lifted their finger, or pointed on the target, or picked up the object above the target with a precision grip. The RT was the time from presentation of the target to the moment when the participant's finger broke contact with the touch screen. The MT was the time from that lift-off to the next pressure change on the screen (touch-down or object contact). The trials were terminated and error feedback given when the RT or MT was outside 100-3,000 ms or when ihe touchdown was outside a 60-pixel tolerance around a target. Design The cue appeared equally often in all four rectangle ends and predicted the target location in 67% of the trials (V trials). In 22% of trials, the target appeared either in the uncued end of the cued rectangle (IS trials) or in the equidistant end of the other rectangle (ID trials). In the remaining 119f of trials, no target appeared (catch trials). The participants were informed about those contingencies and performed 144 trials for each task (12K experimental trials, 16 catch trials). Display orientation, target position, and trial type were randomized, and task was blocked with the order of blocks counterbalanced across participants. There were 20 practice trials at the beginning of each task block, and all faulty trials (outside the defmed temporal or spatial window) were repeated at the end of each task block. Results The exquisite sensitivity of the touch screen to pressure changes enabled the recording of object contacts during grasping, but it also meant that a number of anticipatory lift-offs from the start box were registered during the foreperiod even when the participants did not lift their fmgers but moved them slightly. The experimenter recorded all such instances but could not predict the trial lype in those cases. Thus, the resulting 4.4% error trials could be analyzed only as a function of task. The average number of errors associated with the lifting, pointing, and grasping tasks were 9, 12, and 15, respectively, F(2, 46) = 2.30,/j> .11. Because there was no main effect or interaction involving task order, all F values < 1, we averaged our data across task order before analyzing RT and MT. To evaluate our predictions, we also averaged across stimulus orientations and target locations. In a supplementary analysis, we evaluated effects of stimulus orientation (horizontal, vertical) on RT and MT. In RT, there was an interaction of orientation with trial type, f(2, 46) = 4.10, p < .05, indicating that, in the ID condition only.
370
The Journal ofGenerat Psychology
responses were initiated 12 ms faster with vertical displays than with horizontal dLsplays. In MT, only the main effect of orientation was reliable. ^(1. 23) = 5.77, /; < .05. owing to an 8-ms faster movement completion for vertical displays than for horizontal displays. We also looked at effects of target legation (Side: left vs. right; Height: upper vs. lower) on RT and MT. Left-side RT was 8 ms slower than rightside RT. F(l,23)= 17.82./? .31. Average RTs for lifting, pointing, and grasping responses were 367, 356, and 366 ms, respectively. There was a significant main effect of trial type, F(2, 46) = 27.94. /j < .001. Average RTs for V, IS, and ID trials were 338, 373, and 378 ms. respectively. The effect was owing to reliably faster responses for V compared with IS trials, K23)^ 5.16./;< .001, and V compared with ID trials./(23) = 5.92, p < .001. The difterence between IS and ID was marginal, /(23) = 1.55. p < .07. !t is important to note that the interaction between task and trial type was higbly significant. F(4. 92) - 4.51./?< .005. That result is depicted in Table I (top: Reaction Time) and was further analyzed to test our predictions about the amounts of spacebased attention (faster V compared with IS trials) and object-based attention (faster IS compared with ID trials) in the three tasks. In the lifting task, the 17-ms space-based effect was reliable. r(23) = 2.25, p = .017, but the 4-rns object-based effect did not reach significance. ?(23) = 0.73, p = .24. In the pointing task, there was a 46-ms space-based effect, /{23) = 4.97, p < .001. that was significantly larger than the space-based effect in the lifting task. /(23) = 3.70, p < .001. There was, however, again no object-based effect. /(23) = 0.06, p = .43, and thus also no difference in the size of the object-based effects for lifting and pointing, t(23) = .55, p > .29. Finally, in the grasping task, there was again a 41-ms space-based effect. /(23) = 5.09. p < .001, that did not differ statistically from the same effect in the pointing task, /(23) = .52. p > .30. The difference in the space-based effect between lifting and grasping was significant. r(23) = 2.59. p < ,02. One should note that there was then a significant 10ms object-based effect. t{2y) = 1.72. p < .05. indicating that grasping induced object-based selection, which was not present with finger lifting or pointing. Pointing and grasping trials were also analyzed for object- and space-based effects on overt attention. A two-way ANOVA evaluated the effects of task (point-
371
Fischer & Hoellcn
ing, grasping) and trial type (V, IS, ID) on mean MT Results are shown in Table 1 (middle: Movement Time). There was noreliableeffect of task, F(l.23) = .169, p > .34. The MTs were 284 and 290 ms for pointing and grasping, respectively. There was a significant main effect of trial type, F(2, 46) = 17.45, p < .001. The MT was 24 ms faster in V (268 ms) than it was in IS (292 ms) trials, r(23) = 3.42, p < .001, and 9 ms faster in IS than in ID trials. /(23) = 1.3, p = AQ. Finally, the interaction between task and trial type was not reliable, F(2, 46) = 0.542, p - .29. The space-based effects were significant for both pointing, ^(23) = 3.43, /J < .001. and grasping. 1(23) = 3.42, /J < .001. As had been the case for movement planning, there was no reliable object-based effect during the execution of pointing movements. /(23) = 1 . 0 1 , / J > .16. The object-based effect was numerically larger and approached significance for grasping. r(23) = 1.30, p = A. TABLE 1. Numerical Overview of Results (in ms)
Measure Reaction Time Lifting M SE Pointing M SE Grasping M SE Movement Time Pointing M SE Grasping M SE Total Time Pointing M SE Grasping M SE V = .05
Valid
Invalid Different Same object object
354 13
371 17
375
325
371 14
371
336 9
377 13
387 15
267 14
290 14
295 15
269 10
294 10
306 12
592 17
661 20
667
604 14
671 13
693 16
Effect Spacebased
Objectbased
17*
4
46*
0
41*
10*
23*
5
25*
12
69*
6
67*
22*
18
14
21
372
The Journal of General Psychology
80
1 1 Lifting ^ 1 Pointing
60
B
T 1
40
V
Effect
'tyi
20
I)
[ ^ Grasping
im1
-j1—'—1 T
Space -20
T Object
Effect
FIGURE 2. Effect plot showing space- and object-based effects in the three tasks (lifting, pointing, grasping), averaged across the 24 participants. Error bars represent one .standard error of the mean.
Finally, when RTand MT were added up for each trial to compute total times on each task (Tipper et al.. 1992), hoth space-based and object-based effects were more salient (see Table 1. bottom: Total), indicating that there was no compensatory relation between movement planning and execution. One should note that the object-based effect was then significantly larger for grasping than for pointing responses. r(23) = 1.95./? < .05. Discussion By comparing three tasks with respect to space-based and object-based selection (Egly et al., 1994), we found a modulation of those two components of attention by the current motor intention: The RT pattern showed that there was more space-based attention in pointing than in fmger lifting and more object-based attention in grasping than in pointing. That result signitlcantly extended recent observations on intentional effects on attention by showing that overt attention toward action-relevant object features is preceded by a similar covert selection bias. In agreement with the visuomotor channels hypothesis (e.g., Paulignan & Jeannerod. 1996), visual information is selectively gathered to specify either just tbe transport component or also the manipulation component of goal-directed manual responses, even prior to movement initiation. Therefore, both pointing and grasping tasks incurred a larger space-based effect than mere finger lifting, possibly reflecting the attentional support of visuomotor coordination before (and during) hand transport.
Fischer & Hoellen
The MT results support the aforementioned interpretation because there was a reliable space-based effect in both the pointing and grasping tasks and a trend toward more object based attention in the grasping compared with the pointing task. Those findings can be interpreted as overt attention effects and suggest that the attentional timing might extend beyond the motor planning phase and into movement execution. Such an interpretation makes sense from the point of view of two attentional systems, one for object selection and one for spatial selection, which support the two visual systems in the primate brain for perception and action, respectively (Milner & Goodale. 1993; Soto & Blanco. 2004. p. 15). Two aspects of the present data require further discussion. First, the slow RTs in the valid finger-lift condition are surprising because that condition would seem to require the least amount of movement planning. One possible account for that refers to the slightly lower number of errors in the lifting compared with the pointing and grasping tasks. Thus, the participants might have adopted slightly different response criteria across tasks. Second, we failed to replicate the object-based effect in our lifting condition, although there was a trend in the correct direction. Object-based effects in the standard Egly paradigm have been reported in several studies (see Soto & Blanco, 2004. for a recent review and methodological discussion). The use of squareshaped cues, as opposed to brackets in the original Egly paradigm, is unlikely to have been responsible for the difference between our result and the standard result (Marrara & Moore. 2003). Soto and Blanco have pointed out that perceptually more difficult tasks often yield stronger object-based effects, so one could argue that mere fmger lifting left more attentional resources available for perceptual coding than did our other tasks. However, there appeared to be only one other study that used finger lifting as a measure of attention allocation and that replicated Egty's object-based effect (Bekkering & Pratt, 2004, Experiment 1). We are only beginning to understand how the presence of body parts in the visual field might modulate attentional processes (e.g.. lvanoff & Klein. 2001; Rizzolatti. Riggio, & Sheliga, 1994). Thus, it is unclear to what extent details of our method might have diluted the object-based effect. A fmal point relates to the consistency of fmdings across participants and whether it was possible that the results could have been affected by strategies. Figure 3 addresses that concern by showing that both object- and space-based effects are rather inhomogeneous across participants. The data from Figure 2 are redisplayed in the same arrangement as before but separately for each participant. It is unclear to what extent that considerable variability reflects individual strategies, differential practice, or some combination. Nevertheless, the use of catch trials and randomized cuing conditions certainly minimized strategic biases within task blocks. Our finding extends existing support for task-dependent visual selection for action (e.g.. Bekkering & Neggers, 2002; Meegan & Tipper, 1999) to the earliest possible level of perceptual encoding. Contrary to Bekkering and Neggers'
374
The Journal of General Psychology
Fischer & Hoellen
375
l-l
S ?
III fe.E t We s
R § (siu) 3ZIS p
376
The Jourtml of General Fsyeholo^y
method, the present approach relied on a combination of exogenous attention manipulation (through cuing) and endogenous attention manipulation (through task instruction). Our tlndlngs suggest that we compute different cognitive representations for different tasks, rather than excluding irrelevant information completely. The observed pattern of effects was consistent with tbe theoretical predictions and even extended into movement execution, as one would expect for online optimisation of visuomotor control. Our finding fits well with the findings in recent brain imaging studies that suggest that there are dedicated pathways from perception to action for pointing and grasping. For example, the comparison between grasping and pointing by Faillenot, Toni. Decety. Gregoire,, and Jeannerod (1997) showed a selective cerebral blood flow increase in the anterior part of the inferior parietal cortex and part of the posterior parietal cortex in the graspitig condition. In addition. Jeannerod. Arbib. Rizzolatti. and Sakata (1995) found '"grasping" neurons in the inferior parietal lobule and the infetior premotor area (area F5) that code size, shape, and orientation of objects and the specific grip types needed to grasp them. Those specialized neural structures might have mediated the object based covert visual selection in our grasping task. REFERENCES Bekkering, H,, & Neggers. F, W, (2002), Visual search is modulated by action intentions. F.syclwhfiicul Science. Ll 370-374, Bekkering. H,, & Pratt. J, (in press). Object-based processes in the planning of goaidlrected hand movements, Deubel, H,. & Schneider. W. X. (1996). Saccade target selection and object recognition; Evidence fora common attentional mcchanistii. Vi.vioii Research. 36. 1827-1837. Detibcl, H,. Schneider. W, X,. & Paprotta. I, (1998). Selective dorsal and ventral processing: Evidence for a common attentional tnechanism in reaching and perception. Visual Cof^iuTion. 5. 8!-!O7, Egly. R.. Driver. J,, & Rafal, R. D, (1994). Shifting visual attention between objects and locations: Evidence from normal and parietal lesion subjects. Journal of Experimental P.sychology: General. I2J. 161-177, Faillenot. I.. Toni, 1,. Decety. J,. Gregoire, M.-C. & Jeannerod. M, (1997). Visual pathways for object-oriented action and object recognition: Functional anatomy wilh PET, Cerebral Cortex. 7, 77-85, Hoffman. J, K.. & Subramaniam, B, (1995), The role of visual attention in saccadic eye movements. Perception & Psychophysici, 57. 787-795. Ivanoft, J.. & Klein. R, M. (2001), The presence of a nonresponding effector increases inhibition of return, Psycliononiic Biillerin A Review. 8(2). 307-314, Jeannerod. M.. Arbib. M, A,. Riz/.olatli. G,, & Sakata. H, (I99S). Grasping objects: The cortical mechanisms of visuomotor transformation. Trends in Neurosciences, 18(1). 314-320. Marrara. M. T, & Moore. C. M, (2003), Object-based selection in the two-rectangles method is not an artifact of the three-sided directional cue. Perception & Psychophysics. 65(7). 1103-1109, Meegan. D. V,. & Tipper. S. P (19991, Visual search and target-directed action, Jminuil of Experimental Psychology: Hunuin Perception atid Performance, 25, 1347-1362.
Pischer & Hoellen
377
Milner. A. D., & Goodale, M. A. (1995). Tlie vistiul brain in m-lioii. Oxford. England: Oxford University Press. Pashler. H. E. (1998). The psychology of uneniiou. Cambridge. MA: MIT Press. Paulignan. Y., & Jeaniierod, M. (1996). Prehension movemenls: The visuomolor channels hypothesis revisited. In A. M. Wing, P. Haggard, & J. R. Flanagan (Eds.), Haml and Brain (pp. 263-286). New York: Academic Press, Posner. M. I. (1980). Orienting of attention. Quarlerlv Journal of Experimenlal Psychology 32,3-25, Rizzolatti. C . Riggio. L.. & Sheliga. B. M. (1994). Space and selective attention. In C. Umilta & M. Moscovitch (Eds.). Alleiifion and performance: XV. Conscious and nonconscious informaiion processing (pp. 231-265). Cambridge. MA: MIT Press. Soto. D.. & Blanco. M. J. (2(J
