DISTRIBUTED METACOGNITION Distributed ...

Distributed Metacognition 1 RUNNING HEAD: DISTRIBUTED METACOGNITION

Distributed Metacognition: Examining the Metacognitions Associated with Retrieval from Internal and External Stores *This manuscript has not been peer-reviewed* Timothy L. Dunn1 Connor Gaspar2 Daev McLean2 Derek J. Koehler2 Evan F. Risko2 1. Leeds School of Business, University of Colorado Boulder 2. University of Waterloo, Department of Psychology

WORD COUNT: 5165 All data, code, and pre-registration protocol are freely available via the Open Science Framework at osf.io/k6jpw This work was supported by a Discovery Grant from the Natural Sciences and Engineering Research Council of Canada (NSERC) and funding from the Canada Research Chairs program to E.F.R. Address correspondence to: TLD ([email protected]), Leeds School of Business, University of Colorado Boulder, 995 Regent Drive, Koelbel Building, 419 UCB, Boulder, CO 80309-0419.

Distributed Metacognition 2 ABSTRACT The ability to monitor our cognitive performance (i.e., metacognitive monitoring) is central to efficient functioning. Research investigating this ability has focused largely on tasks that rely exclusively on internal processes. However, our day-to-day cognitive activities often consist of mixes of internal and external processes. In the present investigation, we expand research on metacognition to this more distributed domain. Specifically, we examine participants’ ability to accurately monitor their performance in a knowledge retrieval task when they were required to rely on only their internal processes and when required to rely on both internal processes and external processes (i.e., utilizing the Internet). Results reveal a reduced ability to discriminate between correct and incorrect performance when an external aid was used. This result, among others, provides the beginnings of an understanding of metacognition in distributed cognitive contexts.

Distributed Metacognition 3 Distributed Metacognition: Examining the metacognitions associated with retrieval from Internal and External Stores Metacognitive processes (i.e., monitoring and control) provide an important tool in predicting our cognitive performance and assessing it either online or after-the-fact (Ackerman & Thompson, 2017; Fleming, Dolan & Frith, 2012; Metcalfe & Finn, 2008; Thiede, Anderson & Therriault, 2003). As an example, students’ beliefs about their memory could lead to non-optimal decisions about how best to go about studying (Karpicke & Roediger, 2008; Karpicke, 2009; Tullis, Finley & Benjamin, 2013). While there is a long and rich history of research on metacognition, much of it has focused on cognitive acts that rely largely on internal processes. However, cognitive acts often include an “external” aid or aids (Clark, 1997; 2010; Gray et al., 2006; Heersmink, 2013; Hutchins, 1995; 2010; Michaelian & Sutton, 2013; Sterelny, 2010; Sutton, 2010; Wilson, 2002; Wilson & Clark, 2008). For example, individuals offload a tremendous amount of information retrieval onto the Internet with various consequences (Ferguson, McLean & Risko, 2016; Fisher, Goddu, & Keil, 2015; Sparrow, Liu & Wegner, 2011; Ward, 2013). How then does thinking in this more distributed manner influence our ability to predict and monitor our cognitive performance? In the present investigation, we begin to examine this question by having individuals predict and assess their cognitive performance in a general knowledge fact retrieval task when relying on internal processes or relying on both internal processes and an external aid. Metacognition and Cognitive Offloading Metacognition has come to play a central role in research on cognitive offloading – the trading off of internal processing through external means (Risko & Gilbert, 2017). The fusing of these lines of research specifically looks to examine how individuals think about their thinking when including the use of the body (e.g., gesture) or physical environment (e.g., paper and pen),

Distributed Metacognition 4 and how this thinking influences individuals’ decisions to attempt to offload cognitive processing (Gilbert, 2015; Dunn & Risko, 2016; Risko & Dunn, 2015). For example, Gilbert (2015) demonstrated that participants’ memory confidence predicted the likelihood of offloading, where individuals lower in confidence were more likely to offload memory demands when given the opportunity. This persisted even after controlling for actual memory performance. Furthermore, engaging in cognitive offloading can influence our metacognitions. For example, searching for information on the Internet can influence individuals’ subsequent judgments about their own knowledge (Fisher, Goddu, & Keil, 2015). Ferguson et al. (2015) demonstrated that individuals were more likely to report not knowing the answer to a question when they had the Internet available. This suggests that metacognitive errors in evaluating our knowledge could arise by (or be partially induced by) thinking in more distributed contexts. While interesting theoretically, such results also point to the practical importance of better understanding our metacognitions in these more distributed contexts. Metacognitive Judgments in More or Less Distributed Cognitive Tasks Metacognitive judgments can best be viewed as a kind of cue-based inference (e.g., Dunlosky, Mueller, & Tauber, 2014; Koriat, 1997; Mueller, Dunlosky, & Tauber, 2016). From this cue-utilization perspective, a metacognitive judgment consists of the application, implicitly or explicitly, of some rule/heuristic over available cues. For instance, according to the analytic processing theory of judgments of learning (JOL; Dunlosky, Mueller, & Tauber, 2014; Mueller, Dunlosky, & Tauber, 2016) individuals search for variability across studied items (i.e., cues) that can be plausibly related to memory. As an example, studied items may consist of either words or non-words. Here, the cue is lexicality which varies across the items. Individuals may then apply the heuristic that words are easier to remember than non-words over the lexicality cue to infer

Distributed Metacognition 5 what their memory performance will be. Metacognitive judgments are often further classified into theory- or mnemonic-based judgments. The former relies on an explicit inference based on pre-existing beliefs. The latter relies on the online experience of internal cues such as fluency (i.e., an experience of the ease of information processing; Whittlesea, 1993), and are used more automatically (Koriat, 1997; Koriat & Bjork, 2006). Within a cue-utilization framework, the choice between performing a cognitive task when relying on internal processes or those integrating some external aid can be seen as potentially differing in the types of cues used and the nature of individuals’ pre-existing theories brought to bear. First, with respect to the types of cues, integrating an external aid could be viewed as reducing the contribution of internal cues to the metacognitive judgement. For example, when retrieving information from an external store, the feeling of fluency typically part-and-parcel of internal retrieval (e.g., Kelley & Lindsay, 1993) could be less salient (i.e., due to a lack of directed internal search), ignored, or non-informative, if that “processing” is external to the cognitive system. Assuming that such internal cues are at least to some extent reliable predictors of performance (Koriat & Adiv, 2012; Michaelian, 2012; Proust, 2008; Reber & Unkelbach, 2010), this might predict a reduction in metacognitive accuracy when retrieving from an external source. In a similar vein, it is likely that individuals have pre-existing beliefs that reflect the fact that external aids are very reliable and will be associated with reliable outcomes, or at least more reliable than if that function were performed internally. This likely has metacognitive consequences. For instance, this could lead to an insensitivity to errors and reduced metacognitive accuracy. This could also lead individuals to ignore internal cues. That said, it is important to also note that the use of external aids could be associated with cues not available internally (e.g., cues available in a search results page). If these cues are reliable, then they could

Distributed Metacognition 6 compensate for any lack of internal cues or even improve metacognitive performance relative to relying only on internal processes. One can also assess the relation between metacognitive judgments across distributed and non-distributed contexts. If one tends to be metacognitively accurate in less-distributed contexts, then do they also tend to be more metacognitively accurate in more-distributed contexts? This question bears directly on ongoing debates about the extent to which a domain general metacognitive skill exists (Kelemen, Frost & Weaver, 2000; Maki, Shields, Wheeler, & Zacchilli, 2005; Mengelkamp & Bannert, 2010). For example, Kelemen et al. (2000) found significant correlations in bias scores (i.e., extent of over- or under-confidence), but not G or discrimination scores (i.e., ability to discriminate correct from incorrect responses), within tasks across time and across different tasks (for a similar result see Mengelkamp & Bannert, 2010). It is worth noting that the tasks used in this work relied completely on internal processes. The present investigation provides a novel contrast to this work as the general task is the same (i.e., knowledge retrieval), but differ in the resources that participants are allowed to utilize while completing them (i.e., internal only vs. a mix of internal and external). Present Investigation In the present investigation, we examined participants’ accuracy and their metacognitive judgments of their accuracy in a knowledge retrieval task. Individuals had to rely either on their internal memory alone, or had the option to use their internal memory and the Internet. Participants answered fact-based general knowledge questions (e.g., “What is the capital of South Dakota?”). Individuals either had to complete the task relying solely on their internal knowledge or used the Internet to retrieve the answer (Ferguson et al., 2015; Risko, Ferguson & McLean, 2016). Risko et al. (2016) provided some evidence that participants can make accurate

Distributed Metacognition 7 judgments about their own speed of retrieval of unknown answers to general knowledge questions using the Internet. This suggests that individuals have some ability to predict their performance in retrieving information from the Internet – what the authors called a “feeling-offindability” (Risko et al., 2016). In the present investigation, participants made judgments about their future or past performance. That is, on each trial participants either made a prospective judgment about their cognitive performance before completing the trial (but after being given information about the upcoming trial), or a retrospective judgment of their performance following the trial. The use of both types of judgments provides an opportunity to examine the extent to which experience performing the cognitive act (i.e., retrieving the answers to the questions) contributes to metacognitive performance. For example, retrospective judgments might be expected to be superior provided the direct experience of performing the trial (Fleming, Massoni, Gajdos, & Vergnaud, 2016). Two item sets were used across the internal and external conditions based on careful item selection (based on pilot testing and previous research; Ferguson et al., 2015; Risko et al., 2016). The items were selected to yield similar accuracy across conditions. In other words, the internal and external conditions were made equivalently “easy”. We also selected items in the external condition that would be difficult for most people to answer while relying on their own knowledge. Moreover, we asked participants whether they could have answered the question without the use of the Internet following each trial. This was done in an attempt to limit the contribution of internal knowledge. When comparing an internal condition to an external condition there is typically, and possibly always, an asymmetry in the sense that the external condition does not preclude an internal contribution (i.e., internal + external sources). Though the internal condition does preclude the contribution of the external artifact (i.e., internal alone).

Distributed Metacognition 8 In the present context, this could complicate the interpretation of the results, for example, when an individual knows the answer to a question in the external condition. In this case, their metacognitive judgment may not reflect a judgment about their performance together with the Internet. By selecting difficult items and removing items for which individuals reported having known the answer prior to searching the Internet we can reduce, though potentially not eliminate, this issue. Methods Participants One hundred and ninety-four participants were recruited through Amazon’s Mechanical Turk to take part in the study in exchange for financial compensation. Each individual was randomly assigned to either a prospective or retrospective judgment group. Design A 2 (Knowledge Location: Internal vs. External) x 2 (Confidence Judgment: Prospective vs. Retrospective) mixed design was utilized. Judgment type was assigned randomly to create 2 balanced groups and order of presentation for knowledge location was counterbalanced. Stimuli and apparatus Participants were presented with two lists of 30 general knowledge questions selected from Tauber et al.’s (2013) updated list of Nelson and Narens (1980) normed general knowledge questions. All participants completed the study online via Amazon Mechanical Turk where the task was presented through Qualtrics online survey software. Procedure Following instructions, participants performed two blocks of trials spanning both knowledge location conditions. On each trial, participants were first presented with a general

Distributed Metacognition 9 knowledge question. Participants in the prospective judgment condition were then asked to provide an estimate of how likely they were to provide the correct answer using a slider with end points of 0 and 100 in increments of 1. This screen did not appear for participants in the retrospective condition. In the internal location condition, a screen was then presented with an empty box where participants were to type their answer. In the external location condition participants were presented with the general knowledge question and instructed to “Click next when you are ready to begin your search.” On the next screen the question remained displayed and participants were instructed to “Please search the Internet now. Click “NEXT” when you have found the answer.” Finally, they were presented with the box to input their answer. In the prospective judgment condition, participants in the internal location condition moved on to the next trial and participants in the external condition were asked if they knew the answer to the question before they searched the Internet for it following the entry of their estimate. In the retrospective judgment condition participants in both the internal and external conditions entered their estimate after entering their answer (and before the question pertaining to prior knowledge in the external condition). At the end of the experiment participants were asked to provide an estimate as to how many of the questions that they believed that they had answered correctly in each block out of thirty. Because we only analyze questions in the external condition that participants did not know before looking up the answer, we do not report an analysis of these questions as their estimates would refer to all questions. The entire experiment took approximately 30 minutes to complete. Results Data analysis and visualization was completed in the R programming language (R Core Team, 2016) with the assistance of add-on packages (boot, Canty & Ripley, 2016; effsize,

Distributed Metacognition 10 Torchiano, 2016; ez, Lawrence, 2016; ggplot2, Wickham, 2009; hmisc, Harrell & Dupont, 2016; psych, Revelle, 2016; reshape2, Wickham, 2007). Five individual trials were removed due to not following instructions. We removed 36.4% of trials because participants said they knew the answer prior to searching in the external condition. Importantly, an analysis including these trials yielded the same overall pattern of results.1 Greenhouse-Geisser corrections (Greenhouse & Geisser, 1959) are used to adjust significance levels when Sphericity assumptions are violated. Welch’s t-tests are used where applicable. Effect sizes are reported using generalized eta squared ( hG2 ; Bakeman, 2005) and/or Cohen’s d (Cohen, 1988). As noted above, half of the participants started the experiment in the internal condition and the other half in the external condition. When we included “Order” as a between subject factor in the reported analyzes we found no main effect or any interactions with the variable. To aid in interpretation of the results, we first report analyzes for the internal and external conditions separately, followed by comparisons across memory location. While the latter contrast is our main interest, provided the lack of research in this general domain it thought it valuable to report separate analyzes within the Internal and External conditions as well (see Fig. 1).

Distributed Metacognition 11 Fig. 1. Actual and judged accuracy as a function of location and judgment type.

Note: error bars are 95% Bias Corrected and accelerated (BCa) confidence intervals.

Knowledge Location: Internal Condition A 2 (Judgment Type: Prospective vs. Retrospective) ANOVA was performed on actual and judged accuracy. There was no effect of judgment type on actual accuracy, F(1, 192) = 2.91, p = .09, hG2 = .015. This effect was significant in judged accuracy, F(1, 192) = 6.84, p = .01, hG2 = .03. Participants judged themselves to be less accurate in the retrospective condition than in the prospective condition. A 2 (Measure: Actual Accuracy vs. Judged Accuracy) x 2 (Judgment Type: Prospective vs. Retrospective) mixed ANOVA revealed a main effect of measure, F(1, 192) = 15.5, p < .001, hG2 = .02, such that participants’ actual accuracy was higher than their judged accuracy

Distributed Metacognition 12 (i.e., underconfidence). There was no interaction between measure and judgment type, F(1, 192) = 1.96, p = .16, hG2 = .002 (see left panel of Fig. 1). Knowledge Location: External Condition A 2 (Judgment Type: Prospective vs. Retrospective) ANOVA was performed on actual accuracy and judged accuracy. There was a significant effect of judgment type on actual accuracy, F(1, 192) = 8.57, p = .016, hG2 = .03, such that participants were less accurate in the retrospective condition than in the prospective condition. This effect was not significant in participants’ judged accuracy, F(1, 192) = .42, p = .52, hG2 = .002. A 2 (Measure: Actual Accuracy vs. Judged Accuracy) x 2 (Judgment Type: Prospective vs. Retrospective) mixed ANOVA revealed no main effect of measure, F(1, 192) = .005, p = .82,

hG2 = .0001, but there was a significant interaction between measure and judgment type, F(1, 192) = 4.33, p = .039, hG2 = .009. In the prospective condition participants were numerically underconfident but not significantly so, F(1, 96) = 1.66, p = .20, hG2 = .008. They were significantly overconfident in the retrospective condition, F(1, 96) = 4. 44, p = .038, hG2 = .013 (see right panel of Fig. 1). Comparisons Across Knowledge Location A 2 (Knowledge Location: Internal vs. External) x 2 (Judgment Type: Prospective vs. Retrospective) mixed ANOVA was conducted on actual accuracy and judged accuracy. There was a main effect of location on actual accuracy, F(1, 192) = 49.38, p < .001, hG2 = .09. Participants answered more questions correctly in the internal condition than the external condition. There was no interaction between location and judgment type, F(1, 192) = 1.02, p = .31, hG2 = .002. With respect to judged accuracy there was a main effect of location, F(1, 192) =

Distributed Metacognition 13 9.73, p = .002, hG2 = .02, such that participants’ judged accuracy was greater in the internal condition than the external condition. There was also an interaction between location and judgment type, F(1, 192) = 4.39, p = .037, hG2 = .009, such that the higher confidence in the internal condition was significant in the prospective condition, F(1, 96) = 8.49, p = .004, hG2 = .04, but not the retrospective condition, F(1, 96) = 1.31, p = .25, hG2 = .004. Last, a 2 (Measure: Absolute Actual vs. Absolute Judged) x 2 (Memory Location: Internal vs. External) x 2 (Judgment Type: Prospective vs. Retrospective) mixed-design ANOVA was performed assessing the complete crossing of all factors. There was a significant three-way interaction between measure, location and judgment type, F(1, 192) = 7.2, p = .008, hG2 = .005. This reflects the different forms taken by the interaction between measure and judgment type discussed above. Namely, in the internal condition individuals were underconfident both in making prospective and retrospective judgments but in the external condition individuals were (non-significantly) underconfident in the prospective condition but overconfident in the retrospective condition. Metacognitive Accuracy We first indexed metacognitive accuracy by correlating participants’ trial-level accuracy and confidence ratings into Gamma coefficients (Nelson, 1984). Gamma coefficients can range from -1 (a perfect negative association) to +1 (a perfect positive association), with 0 representing an absence of association (i.e., confidence ratings at random). In the following, we present average Gamma correlations as a function of conditions. Given a lack of variability in accuracy (e.g., always correct) or confidence ratings (e.g., an individual never changed their rating over trials), 27.3% (n = 53) of participants that had no Gamma coefficient and were removed from the

Distributed Metacognition 14 following analyses (n = 35 for the prospective condition, n = 18 for the retrospective condition). As a final note, we additionally observed 24 occurrences where participants achieved a perfectlyinverse Gamma of -1, and 45 occurrences where there was perfectly-positive Gamma of +1. For the purpose of main analysis below we included these participants. When these ‘perfect’ awareness scores were excluded in a secondary analyses the results were not qualitatively different2. First, for the internal condition, a 2 (Judgment Type: Prospective vs. Retrospective) between-subject ANOVA demonstrated no effect of judgement type, F(1, 139) = 3.2, p = .08,

hG2 = .02. For the external condition, a 2 (Judgment Type: Prospective vs. Retrospective) between-subject ANOVA demonstrated an effect of judgement type, F(1, 139) = 27.96, p < .001,

hG2 = .17. Here, participants were more metacognitively accurate when judging retrospectively relative to prospectively. Next, following from above, we compared across the location conditions. A 2 (Knowledge Location: Internal vs. External) x 2 (Judgment Type: Prospective vs. Retrospective) mixed ANOVA demonstrated a main effect of knowledge location, F(1, 139) = 49.84, p < .001,

hG2 = .15, and judgement type, F(1, 139) = 24.72, p < .001, hG2 = .08, such that metacognitive accuracy was highest in the internal location and retrospective conditions. Interestingly, there was also an interaction between location and judgement type, F(1, 139) = 6.46, p = .012, hG2 = .02. For the internal location, metacognitive accuracy in the prospective condition, M = .59, SD = .62, did not differ from the retrospective condition, M = .74, SD = .42, Welch’s t(102.9) = 1.71, p = .09, d = .29, d 95% CI = [-.05, .63]. However, for the external location, metacognitive accuracy was much lower in the prospective condition, M = -.03, SD = .63, relative to the

Distributed Metacognition 15 retrospective condition, M = .45, SD = .43, Welch’s t(103.3) = 5.06, p < .001, d = .86, d 95% CI = [.51, 1.21] (see Fig. 2).

Fig 2. Average Gamma correlations as a function of location and judgment type.

Note: error bars are 95% Bias Corrected and accelerated (BCa) confidence intervals.

As a complimentary analysis, we next assessed individuals’ ability to discriminate correct from incorrect answers by analyzing accuracy estimates as a function of actual accuracy within and across locations, and between judgment types. Forty-three of the 194 participants performed every trial correctly in either the internal (16.0%) or external condition (8.8%), and therefore had

Distributed Metacognition 16 no estimates of accuracy when incorrect. These participants were removed for the purposes of this analysis. A 2 (Actual Accuracy: Correct vs. Incorrect) x 2 (Judgment Type: Prospective vs. Retrospective) mixed ANOVA was conducted for the internal and external conditions separately. In the internal condition, there was a main effect of actual accuracy, F(1, 149) = 301.79, p < .001, hG2 = .45, such that participants’ judged accuracy was higher when individuals were correct than incorrect. There was no interaction between actual accuracy and judgment type, F(1, 149) = 1.95, p = .16, hG2 = .005. In the external condition there was also a main effect of actual accuracy, F(1, 149) = 69.87, p < .001, hG2 = .058. Participants’ judged accuracy was higher when correct than incorrect. There was also a significant interaction between actual accuracy and judgment type, F(1, 149) = 27.87, p < .001, hG2 = .024, such that discrimination was greater in the retrospective condition than in the prospective condition but significant in both nonetheless, F(1, 78) = 66.85, p < .001, hG2 = .23; F(1, 71) = 6.88, p = .011, hG2 = .005, for the retrospective and prospective conditions respectfully. A 2 (Actual Accuracy: Correct vs. Incorrect) x 2 (Knowledge Location: Internal vs. External) x 2 (Judgment Type: Prospective vs. Retrospective) ANOVA was performed to compare the ability to discriminate across the two memory locations. There was an interaction between actual accuracy and memory location, F(1, 149) = 113.44, p < .001, hG2 = .096. The difference in judged accuracy when correct and incorrect was smaller in the external condition than the internal condition. There was no interaction between actual accuracy, memory location, and judgment type, F(1, 149) = 1.91, p = .17, hG2 = .002 (see Fig. 3).

Distributed Metacognition 17 Fig 3. Judged accuracy as a function of actual accuracy, location, and judgment type.

Note: error bars are 95% Bias Corrected and accelerated (BCa) confidence intervals.

Correlations We computed bias scores (i.e., participants’ average judged accuracy minus their actual accuracy) and discrimination scores (i.e., participants’ judged accuracy when correct minus incorrect) for each location. There was a significant positive correlation between bias in the internal and external knowledge conditions, rs (192) = .17, p = .018. There was no correlation between discrimination ability across memory locations, rs (149) = -.01, p = .935. Furthermore, we correlated participants’ Gamma coefficients in the internal and external memory conditions with one another. Overall there was no correlation between metacognitive accuracy between the internal and external conditions, r(139) = .08, p = .343.

Distributed Metacognition 18 Discussion Participants in the external condition were less able to discriminate correct from incorrect responses than in the internal condition. This was observed despite overall performance not being superior in the external condition. Recall that items were selected in order to have approximately equal accuracy across the two tasks. Overall, metacognitive accuracy was higher for the internal location and retrospective judgment condition. Interestingly, the observed interaction between location and judgement type suggests a large cost in metacognitive accuracy when making prospective judgements when using external aids. This difference across prospective and retrospective judgments was not observed when individuals relied on their internal memory alone. We return to this finding in the following General Discussion. Second, there was a significant positive correlation in bias scores (i.e., the extent of over/underconfidence) but not when discrimination was considered. That is, the extent to which participants’ actual accuracy differed from their judged accuracy was related across the internal and external conditions, though their ability to discriminate correct from incorrect answers across the two conditions was not. With respect to bias scores, participants were not overconfident across the board. Rather, in the internal condition participants were underconfident. In the external condition participants were slightly underconfident in the prospective condition and significantly overconfident in the retrospective condition. Thus, at least in the retrospective condition, we see divergence between bias across the internal and external conditions when accuracy across condition was approximately equal. General Discussion Our day-to-day lives are filled with cognitive acts that can best be considered distributed in the sense of including an external contribution. Here, we examined individuals’

Distributed Metacognition 19 metacognitions in similar tasks across distributed (internal memory and the Internet) and nondistributed (internal memory only) conditions. Notably, metacognitive accuracy was lower when using the Internet, and was practically zero when making prospective judgments while using the Internet. Discrimination was not correlated across the internal or external conditions, but bias scores were. Participants were not generally over- or under-confident. Rather, whether individuals were over or underconfident changed across tasks, conditions, and judgment type. That said, when individuals assessed their performance after-the-fact they were overconfident in the external location condition. In the following, we further discuss these patterns and situate them within the existing literature on distributed metacognition. Distributed Metacognition We have suggested two general reasons for why metacognitive accuracy might be lower in a distributed cognitive context: (1) a lack of internally generated mnemonic signals in the external condition and/or, (2) a general bias to assume external cognitive aids are highly reliable. With respect to metacognitive accuracy as measured using discrimination, this general prediction seems to have been borne out. Metacognitive accuracy was poorer in the external condition relative to the internal condition. This appears to reflect a relative insensitivity to errors. Namely, when individuals in the external condition were incorrect they tended to think they were correct. For instance, when individuals searched and found an incorrect answer to the general knowledge question, their average estimate of the likelihood that their answer was correct was 75% (when correct it was 85%). That said, individuals relying on the Internet were able to tell correct from incorrect responses to some extent. In addition, at least here, metacognitive accuracy was better in the retrospective than the prospective condition. This is important as it suggests that the experience of retrieving information from this particular external store provides some useful

Distributed Metacognition 20 information with respect to individual’s judgements about the veracity of their answers. What this information is or what these cues might be represents an important question for future research. Why are individuals better at discriminating correct from incorrect answers in the internal condition? Individuals have accurate (to some extent) knowledge of what they do and do not know (Koriat, 1997). Internal cues, for example the speed of retrieval, provide useful information with respect to predicting what one knows (Kelley & Lindsay, 1993). When retrieving knowledge from an external source the contribution of these internal cues might be reduced. This could happen for a number of reasons (or combinations of those reasons). For example, the items selected for use in the external condition were ones that most people would not know and, in addition, we removed items from the analysis that participants said they did know. Thus, any between-item differences in feeling-of-knowing or confidence in the external condition would likely be much less salient than between-item differences in feeling-of-knowing or confidence in the internal condition where some items could be known and others unknown. With these limited internal cues, individuals would need to rely on knowledge about an item’s likelihood of being located in the external store (i.e., feeling-of-findability; Risko, Ferguson & McLean, 2016) and/or cues that become available during retrieval from the external store (i.e., here searching the Internet). Provided the present results it seems these potential sources, and others we may not have identified, are not sufficient to support a high level of discrimination between correct and incorrect information. That said, these sources of information are sufficient to support some amount of discrimination, and in comparing the prospective to retrospective conditions, it seems clear that cues made available during the act of searching can support discrimination (i.e., discrimination was superior in the retrospective than prospective condition).

Distributed Metacognition 21 In the absence of any information that would discriminate items from one another, individuals likely rely on pre-existing beliefs about the reliability of a given source (i.e., internal knowledge; the Internet). As noted above, in the external condition it seems clear that individuals generally believed information was correct when retrieved from the Internet. For the kind of general knowledge questions asked here this seems a reasonable assumption. Interestingly, our selection of items in the internal and external conditions as being approximately equal in difficulty here makes it likely that any bias to believe a given source was accurate or not comes from preexisting experience. That is, it is not simply the participant’s experience within the experiment that led them, in the external condition, to assume incorrect answers were correct, because participants were actually more accurate in the internal condition. This suggests that a high level of accuracy (locally defined) is not sufficient to explain the poorer discrimination in the external condition. Though there are substantial differences between the internal and external resources available in the conditions used presently, there was a correlation across the internal and external conditions in terms of metacognitive performance. Thus, metacognitive judgments made across the distributed and non-distributed cognitive tasks are not completely independent and may signal a common contribution to both. Specifically, bias (i.e., the difference between actual and judged accuracy) correlated significantly across the internal and external conditions. The more overconfident (or underconfident) individuals were in the internal condition the more overconfident (or underconfident) they were in the external condition, though the effect was rather small. Nonetheless, this finding provides a novel extension of previous research demonstrating correlations in bias across different internal tasks (e.g., Kelemen et al., 2000) and

Distributed Metacognition 22 suggests a common contribution to metacognitive judgments elicited across distributed and nondistributed cognitive tasks. For instance, this correlation might reflect an individual’s general confidence. Interestingly, while bias correlated across internal and external conditions, the ability to discriminate did not. Whether an individual was good or bad at discriminating correct from incorrect responses in the internal condition did not predict whether they would be good or bad in the external condition. This result also replicates and extends previous research (e.g., Kelemen et al., 2000) demonstrating little in the way of correlations between discrimination on different tasks within individuals. The extension here is interesting as the general tasks are ostensibly similar, but carried out using a different set of procedures rather than completely different tasks. These results are consistent with skepticism regarding a domain general metacognitive ability (e.g., Kelemen et al., 2000; c.f. Gilbert, 2015). Conclusions The present investigation represents one of the first systematic analyzes of metacognition across a task completed with and without the aid of an external aid. Future work examining these types of tasks, as well as expanding to other domains, will further inform our understanding of metacognition in distributed contexts. With increases in integration between cognition and various cognitive technologies the need to understand the similarities and differences is of great interest.

Distributed Metacognition 23 References Ackerman, R., & Thompson, V. A. (2017). Meta-Reasoning: Monitoring and Control of Thinking and Reasoning. Trends in Cognitive Sciences. 21, 607-617. Anderson, J. R., Bothell, D., Lebiere, C., & Matessa, M. (1998). An integrated theory of list memory. Journal of Memory and Language, 38, 341-380. Bakeman, R. (2005). Recommended effect size statistics for repeated measures designs. Behavior Research Methods, 37, 379-384. Castel, A. D. (2008). Metacognition and learning about primacy and recency effects in free recall: The utilization of intrinsic and extrinsic cues when making judgments of learning. Memory & Cognition, 36, 429-437. Canty, A. & Ripley, B. (2016). boot: Bootstrap R (S-Plus) Functions. R package version 1.3-18. Retrieved from https://CRAN.R-project.org/package=boot Clark, A. (1997). Being there. Cambridge, MA: MIT Press Clark, A. (2008). Supersizing the mind: embodiment, action, and cognitive extension. Oxford: Oxford University Press Cohen, J. (1988). Statistical power analyses for the social sciences. Hillsdale, NJ, Lawrence Erlbauni Associates. Dunn, T. L. Lutes, D. J. C. & Risko, E. F. (2016). Metacognitive evaluation in the avoidance of demand. Journal of Experimental Psychology: Human Perception and Performance, 42, 1372-1387. Dunn T., & Risko, E. F. (2016). Toward a metacognitive account of cognitive offloading. Cognitive Science, 40, 1080-1127.

Distributed Metacognition 24 Ferguson, A. M., McLean, D., & Risko, E. F. (2015). Answers at your fingertips: Access to the Internet influences willingness to answer questions. Consciousness & Cognition, 37, 91102. Fisher, M., Goddu, M. K., & Keil, F. C. (2015). Searching for explanations: How the Internet inflates estimates of internal knowledge. Journal of Experimental Psychology: General, 144, 674-687. Fleming, S. M., Dolan, R. J., & Frith, C. D. (2012). Metacognition: Computation, biology and function. Philosophical Transactions of the Royal Society, 367, 1280-1286. Fleming, S. M., Massoni, S., Gajdos, T., & Vergnaud, J. C. (2016). Metacognition about the past and future: quantifying common and distinct influences on prospective and retrospective judgments of self-performance. Neuroscience of Consciousness, 1, niw018. Gilbert, S. J. (2015). Strategic offloading of delayed intentions into the external environment. The Quarterly Journal of Experimental Psychology, 68, 971-992. Gray, W. D., Sims, C. R., Fu, W. T., & Schoelles, M. J. (2006). The soft constraints hypothesis: a rational analysis approach to resource allocation for interactive behavior. Psychological Review, 113(3), 461-482. Harrell Jr, F. E. (2016). Hmisc: Harrell Miscellaneous. R package version 4.0-1. Retrieved from https://CRAN.R-project.org/package=Hmisc Hurlstone, M. J., Hitch, G. J., & Baddeley, A. D. (2014). Memory for serial order across domains: An overview of the literature and directions for future research. Psychological Bulletin, 140, 339-373. Heersmink, R. (2013). A taxonomy of cognitive artifacts: function, information, and categories. Review of Philosophy and Psychology, 4, 465-481.

Distributed Metacognition 25 Hutchins, E. (1995) Cognition in the wild. Cambridge: The MIT Press Hutchins, E. (2010). Cognitive ecology. Topics in Cognitive Science, 2(4), 705-715. Karpicke, J. D. (2009). Metacognitive control and strategy selection: deciding to practice retrieval during learning. Journal of Experimental Psychology: General, 138, 469-486. Karpicke, J. D., & Roediger, H. L. (2008). The critical importance of retrieval for learning. Science, 319, 966-968. Kelemen, W. L., Frost, P. J., & Weaver, C. A. (2000). Individual differences in metacognition: Evidence against a general metacognitive ability. Memory & Cognition, 28, 92-107. Koriat, A. (1997). Monitoring one's own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126, 349-370. Koriat, A., & Adiv, S. (2012). Confidence in one’s social beliefs: Implications for belief justification. Consciousness & Cognition, 21, 1599-1616. Koriat, A., & Bjork, R. A. (2006). Mending metacognitive illusions: a comparison of mnemonicbased and theory-based procedures. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32, 1133-1145. Lawrence, M. A. (2015). ez: Easy Analysis and Visualization of Factorial Experiments. R package version 4.3. Retrieved from https://CRAN.R-project.org/package=ez. Maki, R. H., Shields, M., Wheeler, A. E., & Zacchilli, T. L. (2005). Individual Differences in Absolute and Relative Metacomprehension Accuracy. Journal of Educational Psychology, 97, 723. Mengelkamp, C., & Bannert, M. (2010). Accuracy of confidence judgments: Stability and generality in the learning process and predictive validity for learning outcome. Memory & Cognition, 38, 441-451.

Distributed Metacognition 26 Metcalfe, J., & Finn, B. (2008). Evidence that judgments of learning are causally related to study choice. Psychonomic Bulletin & Review, 15, 174-179. Michaelian, K. (2012). Is external memory memory? Biological memory and extended mind. Consciousness & Cognition, 21, 1154-1165. Michaelian, K., & Sutton, J. (2013). Distributed cognition and memory research: history and current directions. Review of Philosophy and Psychology, 4(1), 1–24. Mueller, M. L., Dunlosky, J., & Tauber, S. K. (2016). The effect of identical word pairs on people's metamemory judgments: What are the contributions of processing fluency and beliefs about memory? The Quarterly Journal of Experimental Psychology, 69, 781-799. Mueller, M. L., Dunlosky, J., Tauber, S. K., & Rhodes, M. G. (2014). The font-size effect on judgments of learning: Does it exemplify fluency effects or reflect people’s beliefs about memory? Journal of Memory and Language, 70, 1-12. Peirce, J. W. (2007). PsychoPy—psychophysics software in Python. Journal of Neuroscience Methods, 162, 8-13. Proust, J. (2008). Epistemic agency and metacognition: An externalist view. Proceedings of the Aristotelian Society, 108, 241–268. R Core Team (2016). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from https://www.R-project.org/. Reber, R., & Unkelbach, C. (2010). The epistemic status of processing fluency as source for judgments of truth. Review of Philosophy and Psychology, 1, 563-581. Revelle, W. (2016) psych: Procedures for personality and psychological research. R package version 1.6.9. Retrieved from https://CRAN.R-project.org/package=psych

Distributed Metacognition 27 Risko, E. F., & Dunn, T. L. (2015). Storing information in-the-world: Metacognition and cognitive offloading in a short-term memory task. Consciousness & Cognition, 36, 61-74. Risko, E. F. & Gilbert, S. (2016). Cognitive offloading: Emerging trends and future directions. Trends in Cognitive Sciences, 20, 676-688. Risko, E. F., Ferguson, A., & McLean, D. (2016). On retrieving information from external knowledge stores: Feeling-of-findability, feeling-of-knowing and Internet search. Computers and Human Behavior, 65, 534- 543. Schönbrodt, F. D., & Perugini, M. (2013). At what sample size do correlations stabilize? Journal of Research in Personality, 47, 609-612. Siegler, R. S., & Lemaire, P. (1997). Older and younger adults' strategy choices in multiplication: testing predictions of ASCM using the choice/no-choice method. Journal of Experimental Psychology: General, 126, 71-92. Sparrow, B., Liu, J., & Wegner, D. M. (2011). Google effects on memory: Cognitive consequences of having information at our fingertips. Science, 333, 776-778. Sterelny, K. (2010). Minds: extended or scaffolded? Phenomenology and the Cognitive Sciences, 9(4), 465-481. Storm, B. C., & Stone, S. M. (2015). Saving-enhanced memory: the benefits of saving on the learning and remembering of new information. Psychological Science, 26, 182-188. Storm, B. C., Stone, S. M., & Benjamin, A. S. (2017). Using the Internet to access information inflates future use of the Internet to access other information. Memory, 25, 717-723. Sutton, J. (2010). Exograms and interdisciplinarity: History, the extended mind and the civilizing process. In R. Menary (Ed.), The extended mind (pp. 189–225). Cambridge: MIT Press

Distributed Metacognition 28 Thiede, K. W., Anderson, M., & Therriault, D. (2003). Accuracy of metacognitive monitoring affects learning of texts. Journal of Educational Psychology, 95, 66-73. Torchiano, M. (2017). effsize: Efficient Effect Size Computation. R package version 0.7.1. Retrieved from https://CRAN.R-project.org/package=effsize Tullis, J. G., Finley, J. R., & Benjamin, A. S. (2013). Metacognition of the testing effect: Guiding learners to predict the benefits of retrieval. Memory & Cognition, 41, 429-442. Wegner, D. M. (1987). Transactive memory: A contemporary analysis of the group mind. In Theories of group behavior (pp. 185-208). Springer New York. Whittlesea, B. W. (1993). Illusions of familiarity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 19, 1235. Wickham, H. (2007). Reshaping data with the reshape package. Journal of Statistical Software, 21, 1-20. Wickham, H. (2009). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York, NY. Wilson, M. (2002). Six views of embodied cognition. Psychonomic Bulletin & Review, 9(4), 625-636. Wilson, R. A., & Clark, A. (2009). How to situate cognition: letting nature take its course. In P. Robbins & M. Aydede (Eds.), The Cambridge handbook of situated cognition (pp. 55 77). Cambridge: Cambridge University Press.

Distributed Metacognition 29 Footnotes 1. In the main analysis of the external condition trials on which individuals reported that they would have known the answer to the question without searching the Internet were removed. We conducted a reanalysis of the external condition with those trials included. The results were qualitatively the same with the following exception. With respect to judged accuracy unlike the main analysis, there was no main effect of location, F(1, 192) = .08, p = .78, hG2 < .001. There was an interaction between location and judgment type, F(1, 192) = 6.69, p = .01, hG2 = .01, such that confidence was not significantly different across locations when given prospectively, F(1, 96) = 2.83, p = .095, hG2 = .01, but confidence was higher in the external condition when given retrospectively, F(1, 96) = 4.8, p = .03, hG2 = .01.

2. In the metacognitive accuracy section we chose to include those participants who produced perfect Gamma correlation coefficients (either +1 or -1). While not statistically improbable, the likelihood a priori of any individual being perfectly metacognitively accurate in a real-world setting is very unlikely. Specifically, the perfect Gamma coefficients we observed here are likely a function of accuracy overall being relatively high in the experiment. Thus, we further elected to re-analyze Gamma correlations that excluded: (1) those who produced gamma coefficients of -1, and (2) those who produced gamma coefficients of -1 and/or +1. In the former scenario, the sample size was reduced by 14.8% (N = 120) relative to the main analysis. However, the pattern of results is similar. Participants were more metacognitively accurate retrospectively than prospectively in the external condition, t(118) = 3.49, p = .001, d = .62, d 95% CI = [.24, 1.00], but not in the internal condition, t(118) = .88, p = .38, d = .16, d 95% CI = [-.21, .53], F(1, 118)

Distributed Metacognition 30 = 6.37, p = .012, hG2 = .025, judgement type by location interaction. In the latter scenario, where the sample size was reduced by 43.9% (N = 79), the originally observed main effects are conserved, but the interaction is not signficant. Participants in the external condition were more metacognitively aware retrospectively than prospectively, t(77) = 2.27, p = .025, d = .49, d 95% CI = [.028, .954], but not in the internal condition, t(77) = .68, p = .496, d = .15, d 95% CI = [.30, .61], F(1, 77) = 2.08, p = .154, hG2 = .013, judgement type by location interaction. Critically, in each case the overall pattern of results stays the same: individuals’ metacognitive accuracy was worse prospectively relative to retrospectively in the external condition, with no difference across judgement type in the internal condition. Furthermore, it is likely that the loss of the significant between-subjects interaction effect in the second scenario is a result of a loss of power, not from a change in the pattern of results due to exclusions.

3. We additionally performed two correlational analyses on Gamma coefficients across locations based on the two scenarios outlined in Footnote 2. In the first scenario where those who produced gamma coefficients of -1 were excluded, we observed a non-significant correlation across locations, r(118) = .07, p = .423. In the second scenario where those who produced gamma coefficients of -1 and/or +1 were excluded, we again observed a non-significant correlation, r(77) = .04, p = .748. Thus, these results did not differ from those observed originally.