John Benjamins Publishing Company
This is a contribution from Implicit and Explicit Learning of Languages. Edited by Patrick Rebuschat. © 2015. John Benjamins Publishing Company This electronic file may not be altered in any way. The author(s) of this article is/are permitted to use this PDF file to generate printed copies to be used by way of offprints, for their personal use only. Permission is granted by the publishers to post this file on a closed server which is accessible to members (students and staff) only of the author’s/s’ institute, it is not permitted to post this PDF on the open internet. For any other use of this material prior written permission should be obtained from the publishers or through the Copyright Clearance Center (for USA: www.copyright.com). Please contact
[email protected] or consult our website: www.benjamins.com Tables of Contents, abstracts and guidelines are available at www.benjamins.com
Challenges in implicit learning research Validating a novel artificial language* John Rogers, Andrea Révész, & Patrick Rebuschat
Qatar University / UCL Institute of Education, University College London / Lancaster University This chapter documents some of the methodological challenges in the use of artificial grammars in second language research. In the three experiments reported here, participants were exposed to an artificial system based on Czech morphology under incidental learning conditions. After several modifications to the design of Experiments 1 and 2, Experiment 3 provided evidence that learners can acquire knowledge of L2 case marking incidentally. Taken together, these three experiments illustrate the challenges that researchers can face when carrying out incidental learning research, in particular the often unreported difficulty in establishing an initial learning effect when piloting a novel, semiartificial language system.
Introduction A number of recent experiments have investigated the effects of incidental exposure on the acquisition of implicit and explicit knowledge of second language (L2) grammar. Many of these studies (e.g. Grey, 2013; Grey, Williams, & Rebuschat, 2014; Morgan-Short, 2007; Morgan-Short, Faretta-Stutenberg, Brill-Schuetz, Carpenter, & Wong, 2014; Rebuschat, 2008; Rebuschat & Williams, 2006, 2009, 2012; Rogers, Révész, & Rebuschat, in press; Serafini, 2013; Tagarelli, Borges Mota, & Rebuschat, 2011, 2015) have adopted an artificial language as part of their experimental design. Despite the established tradition of using artificial, semi-artificial or simplified natural language systems in the field of Second Language Acquisition (SLA) (e.g. Alanen, 1995; Andringa & Ćurčić, 2015; Brooks & Kempe, 2013; DeKeyser, 1994, 1995, 1997; de Graaf, 1997; Gullberg et al., 2010; Gullberg, Roberts, & Dimroth, 2012; Hulstijn, 1989; Jackson, 2014; Kempe, Brooks, & Kharkhurin, 2010; Leung, 2007; Leung & Williams, 2011,
* We would like to thank Daniel Jackson for his insightful comments on an earlier version of this chapter.
doi 10.1075/sibil.48.12rog © 2015 John Benjamins Publishing Company
John Rogers, Andrea Révész, & Patrick Rebuschat
2012, 2014; MacWhinney, 1983; Paciorek, 2012; Paciorek & Williams, this volume; Rebuschat, Hamrick, Sachs, Riestenberg, & Ziegler, 2013; Rebuschat, Hamrick, Riestenberg, Sachs, & Ziegler, 2015; Robinson, 2002, 2005, 2010; Tagarelli, 2014; Tagarelli, Jiang, Laka, Barbey, Morgan-Short, & Ullman, 2014; Tellier & Roehr-Brackin, 2013; Williams, 2004, 2005, 2010), one important, but rarely discussed aspect of this increasingly popular line of research is the important step of establishing an initial learning effect in the experiments. That is, in order to work with artificial systems, researchers need to first ensure that the systems are actually learnable by subjects. Once the learning effect has been observed, it is then possible to observe, in subsequent experiments with the same artificial language, how different manipulations impact on learning.1 This chapter reports on a series of three experiments that set out to validate a novel semi-artificial language based on Czech with the view of utilizing the system in future studies (see e.g. Rogers, in prep; Rogers et al., in press).
Artificial language research and SLA In an early review, McLaughlin (1980) pointed out that artificial language research has focused on two broad areas of inquiry. The first of these is concerned with how learning occurs and the general learning mechanisms underlying this process. The second lies with the intrapersonal and situational variables which mediate the process of learning. Given the focus and scope of this research agenda, it might appear that the results from artificial language studies would be relevant for all fields related to education and learning, including SLA. However, the validity and generalizability of the results of artificial language experiments have been challenged in SLA, in particular due to the differences in complexity between artificial languages and natural language systems (e.g. DeKeyser, 1994; Ellis, 1999; McLaughlin, 1980; Schmidt, 1994; Winter & Reber, 1994). These differences in complexity have not resulted in an abandonment of artificial language research within SLA, and recent years have witnessed an increasing interest in the methodology, presumably because of the significant advantages provided by using artificial systems. Firstly, the artificial language paradigm has generated very robust, and easily replicable, findings since the 1960s (e.g. Braine, 1963, 1966;
. Obtaining the initial learning effect is also important from a publication perspective. This is due to the well-documented phenomenon of publication bias (e.g. Oswald & Plonsky, 2010) in which journals are generally less interested in manuscripts reporting failures to learn a given linguistic feature. Of course, observing a failure to learn can be revealing as well since it informs us about potential limitations of the mechanisms underlying implicit and explicit learning. In addition, it also serves to inform researchers as to the effectiveness of various aspects of different experimental designs.
© 2015. John Benjamins Publishing Company All rights reserved
Validating a novel artificial language
Moeser & Bregman, 1972; Segal & Halwes, 1965, 1966; Smith, 1966). Although this robustness is undoubtedly hinged on a number of different variables (e.g. the nature of the stimuli, the complexity of the underlying rule system, the exact nature of the training task, the total amount of exposure during the training phase), the general methodology has been utilized across a wide range of studies, and has proven to be flexible and adaptable to different experimental contexts (see Gómez & Gerken, 2000, and Ziori & Pothos, this volume, for reviews of artificial language and artificial grammar research, respectively). Secondly, potentially confounding variables, such as prior knowledge, can effectively be accounted for, and input variables such as frequency of the target structure can be readily manipulated. In short, artificial systems provide the experimenter with “complete control of the consistency and purity of input” (Cook, 1988, p. 509). As such, a number of SLA studies have taken early criticisms of artificial languages (e.g. Schmidt, 1994) into account and have utilized miniature language systems or semi-artificial languages in an attempt to bridge the gap between construct validity and experimental control. These studies have contributed to the body of knowledge on a wide range of issues related to psycholinguistics and SLA, including the degree to which various aspects of L2 grammar can be acquired under incidental learning conditions (e.g. Alanen, 1995; DeKeyser, 1994, 1995; Hama & Leow, 2010; Leung & Williams, 2011, 2012; Rebuschat, 2008; Rebuschat et al., 2013, 2015; Rebuschat & Williams, 2006, 2009, 2012; Robinson, 2002, 2005, 2010; Williams, 2004, 2005; Williams & Kuribara, 2008). But what exactly are miniature and semi-artificial language systems? Miniature language systems, simply put, represent simplified versions of a single natural language. Typically, these include a restricted number of lexical items, as well as controlled morphosyntactic features. For example, a study by Robinson (2002) utilized a miniature Samoan system, which consisted of one article, 11 verbs, and 15 nouns, to examine the acquisition of L2 morphology, specifically locative, ergative and noun incorporation case markers. Other studies have successful utilized miniature systems across a wide range of artificial languages (e.g. Andringa & Ćurčić, 2015; de Graaf, 1997; DeKeyser, 1995, 1997) as well as natural languages, which include Finnish (Alanen, 1995), Russian (e.g. Brooks & Kempe, 2013), Samoan (Robinson, 2002, 2005, 2010) and Basque (Tagarelli, 2014; Tagarelli et al., 2015). In contrast to miniature systems, semi-artificial systems combine aspects of two or more natural languages. Typically, the lexis in semi-artificial languages is presented in the L1, or a well-known L2, of the participants of the study. This ensures that participants can pay attention to the meaning of the stimuli, and it also greatly reduces the need for vocabulary pre-training. While the vocabulary of semi-artificial languages is presented in a familiar language, this lexis is also often combined with morphosyntactic features of a different language, which represent the learning target of the experiment. For example, Hulstijn (1989), in an experiment with native-speakers of Dutch, used a semi-artificial
© 2015. John Benjamins Publishing Company All rights reserved
John Rogers, Andrea Révész, & Patrick Rebuschat
language consisting of Dutch vocabulary with the addition of artificial morphemes and artificial word order. Williams and Kuribara (2008; see also Grey, Williams, & Rebuschat, 2014) employed a semi-artificial language consisting of English lexis with Japanese word order and Japanese case-markers (e.g. That sandwich-o John-ga ate.). Rebuschat (2008; see also Rebuschat & Williams, 2006, 2009, 2012) developed a language system consisting of English words with German syntax (e.g. Brian defended usually many shots during his matches.) to investigate the incidental learning of L2 word order. In summary, the use of miniature systems and semi-artificial languages has grown out of the cognitive tradition which is willing to sacrifice some degree of external validity in exchange for reliability and experimental control (Hulstijn, this volume; Hulstijn, Young, Ortega, et al., 2014). Artificial systems allow researchers to focus their experiments towards the acquisition of specific linguistic features (e.g. noun- determiner systems in Williams, 2004, 2005) as well as effectively control for prior knowledge on the part of the participants. As such, miniature systems and semiartificial languages appear particularly well-suited to investigate the initial stages of language learning, including the effects of first-exposure towards the acquisition of various aspects of L2 grammar (e.g. Grey et al. 2014). Looking critically at miniature systems and semi-artificial languages, miniature systems could be argued to be the more valid alternative in that they more closely reflect the complexity of a natural language system. However, from a methodological point of view, miniature systems come with several drawbacks. In particular, as the lexis in miniature systems is entirely in an unfamiliar L2, these studies either require extensive vocabulary pre-training phases (e.g. Robinson, 2002) or prolonged exposure phases (e.g. Brooks & Kempe, 2013) to allow participants’ enough time to learn the vocabulary which is used in the experiment. In contrast, semi-artificial languages, as noted above, typically utilize lexis from the participants L1, thus obviating the need for any vocabulary pre-training.2 In this regard, semi-artificial languages have a clear methodological advantage over miniature systems in that they provide more ease of experimentation, and allow for the possibility of an entire experiment being completed in a short, single session. Although the possibility of completing an entire experiment in one session is appealing, the use of semi-artificial languages in this regard is not without its methodological challenges. As noted above, establishing the initial learning effect is of great importance in order to demonstrate that the artificial system is, in fact, learnable. However, establishing even a slight learning effect after a minimal amount of exposure
. A number of recent studies in neuroscience have provided evidence that the presentation of lexis in an L2 results in the unconscious activation of their L1 counterparts (e.g. Martin, Dering, Thomas, & Thierry, 2009; Thierry & Wu, 2007; Wu & Thierry, 2010). This lends support to the use of semi-artificial languages as a valid alternative to miniature linguistic systems: Presenting lexis entirely in the L2, as is the case with miniature systems, would also result in the activation of the corresponding lexical item in the L1.
© 2015. John Benjamins Publishing Company All rights reserved
Validating a novel artificial language
is not always a simple matter. A number of studies have reported, in detail, what stages where necessary to ensure that the artificial system was learnable via very brief exposure. A study where this can be clearly seen is Rebuschat’s (2008) investigation into the implicit and explicit learning of natural language syntax. Experiment 1 of this study utilized a semi-artificial language consisting of English lexis with German syntax (see example above) to examine the learning of four verb placement patterns. In the training phase, which lasted approximately 20–30 minutes, participants were exposed to 128 sentences auditorily. For each training trial, they first listened to the sentence, then judged whether it was semantically plausible or not. Following the training phase, participants completed a surprise grammaticality judgment test, where they had to classify new sentences as being grammatical or ungrammatical based on the sentences in the training set. The results of this test indicated that participants were not able to perform at levels above chance, thus providing no evidence of learning of the target patterns. Experiment 2 in this series built upon the results of Experiment 1 in order to investigate whether changes to the stimulus materials might impact on the overall learning effect. Whereas in Experiment 1 all sentence elements had fixed positions within the sentence, the stimuli in Experiment 2 were diversified so that only the verbs had fixed positions. Other elements, such as adverbs, had relatively free word order within the utterance. The rationale behind this modification was that the fixed position of the verbs, in contrast to the other sentence elements, might make their position more prominent, thus leading to more noticing (Schmidt, 1990) during exposure. With the exception of this change to the stimulus materials, Experiment 2 was identical in material and procedure to Experiment 1. Like the results of Experiment 1, Experiment 2 did not produce a significant overall performance on the grammaticality judgment task. However, further analyses indicated that there was learning, but that this was restricted to participants who had developed explicit knowledge. In Experiment 3 of Rebuschat’s (2008) study, the author set out to see if further changes to the stimulus material and training procedure would influence the overall learning effect. To this end, the stimulus material for Experiment 3 was simplified to include only three verb placement rules, as opposed to four in Experiments 1 and 2. A further modification was that elicited repetitions were added to the procedure of the training phase. As such, the training phase of Experiment 3 required participants to listen to the sentence, then repeat the entire sentence aloud from memory after a delayed prompt, then finally judge the semantic plausibility of the sentence. In contrast to the results of Experiments 1 and 2, the results of the testing phase of Experiment 3 revealed an overall significant learning effect in both aware und unaware subjects, suggesting that the modifications to the experimental procedure helped lead to learning of the target syntactic system. Experiments 4, 5 and 6 of Rebuschat (2008) represent variations of Experiment 3. © 2015. John Benjamins Publishing Company All rights reserved
John Rogers, Andrea Révész, & Patrick Rebuschat
Although it is without question that a great number of variables play on the learning demonstrated in any given experiment, Rebuschat (2008; see also R ebuschat & Williams, 2012) serves as an illustration of the challenge in striking a balance between the training task(s) and training materials, in particular in developing a system which is complex enough for implicit learning to take place (see Reber, 1993; Rebuschat, 2013) yet simple enough to be learned following a minimal amount of exposure. In the case of this study (Rebuschat, 2008), it was only after simplifying the complexity of the stimuli and modifying the training task to include more in-depth, comprehension-based processing of the target structures that a significant learning effect was established after a very brief exposure period.
Motivation and research questions It is the position of this chapter that there is much to be gained from reporting the entire process of piloting and validating novel methodological designs. For instance, such a description might help cast light on the limitations on what can be learned incidentally as well as the degree that different tasks and experimental conditions might lead to implicit or explicit knowledge. Furthermore, such descriptions might inform future researchers when faced with similar methodological difficulties. Much like the series of experiments referenced above (Rebuschat, 2008, Experiments 1–3), it is hoped that the experiments reported here will also serve as an illustration of some of the challenges in using artificial languages to investigate the acquisition of implicit and explicit L2 knowledge. Below we document the process of validating a novel, semiartificial linguistic system, based on Czech morphology. The purpose of the project was to investigate the extent to which incidental learning conditions can promote the acquisition of L2 morphology (case marking). Specifically, we were interested in investigating (i) to what extent L2 case markings can be acquired under incidental learning conditions and (ii) what type of knowledge is acquired as a result of this exposure, implicit or explicit. For a comprehensive description of procedures and results, see Rogers (in prep; Rogers et al., in press).
Experiment 1 Experiment 1 focused on the incidental learning of Czech morphology, specifically the nominative marker -a, the accusative marker -u, and the instrumental marker -ou, by means of an artificial language paradigm. This experiment entailed three stages: a training phase, a testing phase, and debriefing (oral interview).
© 2015. John Benjamins Publishing Company All rights reserved
Validating a novel artificial language
Methods Participants. Fifty-two native speakers of English with no background in any Slavic (or other morphologically-rich) languages took part in Experiment 1. Participants were randomly assigned to an experimental (n = 28) and a control (n = 24) group. All participants were undergraduate students at a university in the United Kingdom, but none were majoring in linguistics or foreign languages. The ages of participants ranged from 18 to 24 years (M = 19.7, SD = 1.8). The data set for one participant was discarded due to a disruption in the experimental environment; another set was lost due to experimenter error.
Stimulus material. Training set. A semi-artificial morphological system, based on Czech case marking, was used to generate the stimulus material for this experiment. As shown in Table 1, the system consisted of English phrases and a Czech noun, which was inflected according to one of three cases (nominative -a, accusative -u, instrumental -ou), depending on its function in the sentence (subject, object, instrumental). Table 1. Descriptions and examples of the three morphological categories Morphological Syntactic category category in English
Examples
Nominative
Subject
The britva cut David’s face at the sink last night.
Accusative
Direct Object
Peter used a britvu in the bathroom today.
Instrumental
Adverbial (meaning “to do/ Anne cut her leg with a britvou in the morning. make something with an X”)
A total of 48 Czech nouns, foreign words to the participants, were used in the training set. All of the Czech words were regular, feminine nouns that end with the inflection -a in their nominative form. All of the nouns followed the same pattern of declension. Only nouns with relatively “concrete” meanings were chosen in order to ensure that the nouns could be easily represented visually through images in the training phase. For the training phase, 96 clip-art images were collected. Forty eight of these images corresponded to the foreign Czech words used in the training set, and 48 images were distractor images that did not correspond to any of the foreign words. Each of the 96 images was used three times in the training phase of the study. The distribution of the distractor images was balanced throughout the training phase so that they did not occur more than once with any particular Czech noun.
© 2015. John Benjamins Publishing Company All rights reserved
John Rogers, Andrea Révész, & Patrick Rebuschat
Three stimulus sentences were created for each of the 48 Czech nouns in the training set. Of these three sentences, one sentence was written so that the Czech noun occurred in the nominative case, one sentence included the noun in the accusative case, and one sentence had the noun marked for the instrumental case. In sum, each Czech noun occurred three times in the training set, each time in a different sentence and each time with a different function and case marking. This resulted in a total of 144 sentences – 48 sentences for each of the three morphological categories. In addition to the inflected Czech noun, the word order in the sentence was arranged in accordance with four syntactic patterns (see Table 2 for templates of syntactic patterns and example sentences). There were a number of reasons for scrambling the word order in these sentences. Firstly, scrambled word order is more reflective of natural languages with rich morphologies. Like Czech, such languages rely on caseendings rather than word order to mark functions within the sentence. Secondly, the syntactic patterns allowed for controlling the position of the foreign or Czech word within the sentence. This ensured that the position of the foreign word could not serve as a reliable indicator of its function within the sentence. All sentences in the training phase were written so that an approximate meaning of the foreign word could be inferred by the participants from the rest of the sentence (see Appendix A for sample sentences and corresponding pictures from the training set). In the construction of the training and testing sets, care was taken to control for the length of the sentences. Each sentence had exactly 12 syllables, and a frequency analysis confirmed that number of words per sentence was not a reliable indicator of morphological category during the training phase, F (2, 141) = 1.322, p = .27, η2 = .02. In addition, all English words in the stimulus materials were among the 2000 most frequent English words as determined by Lextutor’s vocabulary profile program (Cobb, 2012). The sentences were also comparable in terms of lexical diversity (types per tokens, M = .042, SD = .01), lexical density (content words per total number of words, M = .59, SD = .01), and average word length (number of syllables per word, M = 1.30, SD = .04). Testing Set. The testing set of this experiment consisted of 48 new sentences. Half of these sentences (24) were generalization items, i.e. they consisted of novel Czech words and sentences which had not occurred in the training set. The other half of the sentences (24) were partially trained items, i.e. the Czech word in these sentences had occurred in the training set but in a different sentence context. All items in the testing set were designed with the same considerations as the ones in the training set, controlling for the total number of syllables per sentence (12), as well as lexical and syntactic complexity. A frequency analysis of the testing set indicated that the average stimulus length was the same for the grammatical and ungrammatical items (9.04 words per sentence). There was also no significant difference between the sentence length of the stimulus materials used in the training phase (M = 9.15 words) and testing phase (M = 9.04 words), t (66.901) = 1.068, p = .29, d = .18. This indicates that sentence length could not serve as a reliable predictor of grammaticality during the testing phase. © 2015. John Benjamins Publishing Company All rights reserved
Validating a novel artificial language
Table 2. Templates, sample sentences, and frequencies for the four syntactic patterns Pattern
Template
Frequency in training set
Pattern 1
[[AP]TEMP > [NP]OBJ > [VP] > [NP]SUBJ > [PP]]
Nominative
Last summer the grass ate the koza in the field.
(12)
Accusative
Last month the kasu opened Patrick with the key.
(12)
Instrumental
Some time ago John scared the child with a zrudou.
(12)
Pattern 2
[[AP]TEMP > [NP]SUBJ > [VP] > [NP]OBJ > [PP]]
Nominative
Last year the prodejna shipped goods to the shoppers.
(12)
Accusative
All week the builder took his vrtacku to work.
(12)
Instrumental
Today the wooden board cut he with a pilkou.
(12)
Pattern 3
[[NP]OBJ >[VP] > [PP] >[AP]TEMP > [NP]SUBJ]
Nominative
David’s face cut at the sink at night the britva.
(12)
Accusative
The zahradu planted with fruits weeks ago Andrea.
(12)
Instrumental
The dishes washed with a myckou last night Peter.
(12)
Pattern 4
[[NP]SUBJ >[VP] > [PP] >[AP]TEMP > [NP]OBJ]
Nominative
The kocka killed with its teeth this morning the bird.
(12)
Accusative
The cat chased in the house in summer the mysku.
(12)
Instrumental
Sarah shot with a flintou weeks ago a bird.
(12)
The ungrammatical items in the testing set were generated by replacing the correct case marking with one of the other two case markers which had also been present in the training set. The incorrect case markings were balanced across the testing phase. Out of 48 total items in testing phase, 16 were nominative (-a), 16 accusative (-u), and 16 instrumental (-ou). Eight of the nominative items were grammatical and eight were ungrammatical. Of the eight ungrammatical nominative case-items, four were created by replacing the nominative marker (-a) with the accusative marker (-u) and four by replacing the nominative marker with the instrumental marker (-ou). The same procedure was followed in creating the accusative and instrumental items. In sum, care was taken to ensure that participants could only make correct judgments in the testing phase if they were able to identify instances of correct and incorrect case marking. Examples of the testing set can be found in Appendix B. Procedure. As noted above, the experiment consisted of three phases: a training phase, a testing phase, and a debriefing session during which participants provided retrospective verbal reports. The experimental group completed all three of these phases; the control group took part in the testing phase and debriefing session. The training and testing phases of the experiment were delivered via the stimulus presentation software Superlab 4.5 (Cedrus Corp, San Pedro, CA). Following the testing phase, participants completed a short debriefing questionnaire, followed by an oral interview. © 2015. John Benjamins Publishing Company All rights reserved
John Rogers, Andrea Révész, & Patrick Rebuschat
Training phase. Participants were exposed to the stimulus material under incidental learning conditions, i.e. participants did not know they were going to be tested. Furthermore, following common practice in recent studies on incidental learning (e.g. Hamrick, 2013; Rebuschat & Williams, 2012), the training phase was deliberately designed to disguise the real purpose of the training task. Participants were told that they were going to take part in a study on learning foreign language vocabulary. Their task was to listen to a sentence, then match the meaning of the foreign (Czech) word (e.g. žehličkou) to one of two pictures (see description of images above) displayed on the monitor (e.g. an iron or a broom). Participants were given no feedback on the accuracy of their decision. Figure 1 below illustrates the training procedure. Fixation Cross
Match foreign word to picture
Listen to next sentence Fixation Cross
Listen to sentence
Figure 1. Training procedure in Experiment 1. Each trial consisted of listening to a sentence and matching the foreign word to a picture
At no point during the training phase were participants informed that the foreign nouns were inflected for case nor that they would be tested afterwards. Participants listened to all 144 sentences without a break. These sentences were presented in a different, randomized order for each participant. The entire training phase, on average, took about 25 minutes to complete. Testing Phase. The testing phase for this experiment consisted of a 48-item grammaticality judgment task. Following the training phase, participants in the experimental group were informed that the sentences in the previous section were not arbitrary but part of a complex system. They were then told that they would listen to 48 new sentences, half of which belonged to the same system, half of which did not. For each test sentence, participants had to decide as quickly as possible if the sentence belonged to the same system. No feedback was provided on the accuracy of participants’ decisions. See Figure 2 below for an illustration of the testing procedure. Judge Sentence
Fixation Cross Listen to Sentence
Report Basis of Judgment Report Confidence
Figure 2. Testing procedure in Experiments 1–3. Each trial consisted of listening to a s entence, judging its grammaticality, reporting the confidence level and the basis for the g rammaticality decision
© 2015. John Benjamins Publishing Company All rights reserved
Validating a novel artificial language
At the end of the experiment, participants were also prompted to describe, on both a written questionnaire and as part of a follow-up oral interview, any rules or patterns they might have noticed. In the oral interview, the researcher asked the candidates to elaborate on their responses in the written questionnaire. Finally, the researcher explained the underlying rule system and asked participants again if they had figured this out or had any intuition about it at any point during the experiment. Statistical analyses. Performance on the grammaticality judgment task was analyzed using both d-prime scores and mean accuracy rates. D-prime scores are based on signal detection theory and considered a more accurate measure given that response bias is taken into account (see Macmillan & Creelman, 2005, for an overview). In the case of the present experiment, a d-prime score of 0 can be interpreted as chance performance. A d-prime score significantly higher than 0 indicates that participants are able to discriminate between grammatical and ungrammatical items, with higher d-prime scores indicating superior performance. Before carrying out each analysis, the distribution of scores were plotted and examined for skewness and kurtosis and, additionally, Shapiro-Wilk tests of normality were run via SPSS. Unless otherwise noted, the results revealed the data to be normally distributed. Means, standard deviations, exact p-values, 95% confidence intervals, and effect sizes are reported for all tests of significance.
Results The analysis of participants’ performance on the picture-matching task in the training phase of the experiment indicated that they were able to correctly match the foreign word with its corresponding picture with great accuracy (M = 95.93, SD = 3.50). The analysis of the grammaticality judgment test revealed that the mean accuracy of the experimental group (M = 53.13% SD = 3.79%) did not significantly differ from that of the control group (M = 50.60%, SD = 5.92%), t (48) = 1.822, p = .075, 95% CI [–0.26, 5.30], d = 0.51. The results for the d-prime values confirmed a non-significant difference between the performance of the experimental (M = .230, SD = .411) and control group (M = .033, SD = .303) on the grammaticality judgment task, t (48) = 1.916, p = .061, 95% CI [–0.010, .404], d = .91. Although a non-significant difference between the d-prime scores of the control and experimental groups could be argued to obviate the need for additional analyses, the data were subjected to further analyses in order to have a more refined view of participants’ performance and to ascertain if there were any patterns in the data which might inform future experiments. One pattern that emerged was that participants displayed a clear bias in their responses towards foreign words which they had previously encountered in the training phase. On new grammatical items, i.e. test sentences with Czech words which were © 2015. John Benjamins Publishing Company All rights reserved
John Rogers, Andrea Révész, & Patrick Rebuschat
not included in the training set, participants classified only 34.58% (SD = 11.71) of the sentences correctly. In contrast, on old grammatical items, i.e. sentences with Czech words that were part of the training phase, participants classified 76.28% (SD = 15.75%) of the sentences accurately. These results suggest that participants were more likely to judge a sentence as grammatical if the Czech word was familiar, and ungrammatical if the Czech word was unfamiliar. The same pattern held for ungrammatical items, where participants classified 66.99% (SD = 16.24) of old and 34.60% (SD = 15.40) of new ungrammatical items correctly, suggesting again that participants were basing their judgments on the familiarity of the foreign word, rather than the grammaticality of the case marker. In short, the results for the target case-markers seem to have been skewed due to a shortcoming in the experimental design. Retrospective verbal reports. An analysis of the retrospective verbal reports (both written and oral) revealed several important points.3 First, none of the participants (0/28) in the experimental group were able to verbalize the target morphological rules at the end of the experiment, even when prompted to guess by the experimenter. In addition, after the experimenter explained the rules to participants and asked if they had thought of this rule previously, none of the participants stated that they had done so. Also, only around half (13/28) of the participants reported noticing the endings of the foreign words and the fact that the endings were changing during the experiment.
Discussion The performance of the experimental participants did not reveal an overall learning effect. It would appear that the training task might not have been sufficient to promote the learning of the target rules, potentially as a result of the limited amount of exposure in the training phase. Also, taking into account the fact that the participants were able to complete the picture-matching task in the training phase with high degrees of accuracy, it also appears that the training conditions did not necessitate that the participants attend to the morphological markers in order to complete the given task. In previous studies of incidental learning (e.g. Rebuschat, 2008; Rebuschat & Williams, 2012), where no learning was detected in initial experiments, researchers were able to trigger development by modifying the training task to include elicited repetitions (asking participants to repeat the stimulus sentence aloud) in addition to making a judgment based on the content of the sentence. It would also seem likely that reducing
. As the focus of this chapter is on establishing a significant learning effect, the results of the subjective measures of awareness are not included here. For a full discussion of the results of the measures of awareness, please see Rogers (in prep) or Rogers et al. (in press)
© 2015. John Benjamins Publishing Company All rights reserved
Validating a novel artificial language
the overall complexity of the syntactic patterns and/or increasing the total amount of input might promote learning among participants.
Experiment 2 Like Experiment 1, Experiment 2 set out to investigate the extent to which Czech morphology can be acquired under incidental learning conditions. Experiment 2 was identical to Experiment 1, with the exception of two alterations which are outlined below.
Methods Participants. Fourteen native speakers of English participated in Experiment 2. The demographics of the participants were similar to those of the participants in Experiment 1. The ages of the participants (English L1) ranged from 18 to 23 years (M = 19.93, SD = 1.59). Stimuli. The first alteration from Experiment 1 to 2 is in regards to the stimulus materials for the training and testing sets. In order to reduce the overall level of complexity of the stimulus material, the number of syntactic patterns, both for the training and testing set, was reduced from four to two patterns to only include patterns 1 and 2 from Table 2. Procedure. The second alteration in Experiment 2 was that elicited repetitions were added to the training phase, i.e. participants were asked to repeat the sentence aloud, prior to judging which of the two pictures more closely matched the meaning of the foreign word. Figure 3 gives an illustration of the training procedure of E xperiment 2. Listen to sentence
Match foreign word to picture Repeat sentence aloud
Listen to next sentence
Figure 3. Training procedure in Experiment 2. Each trial consisted of listening to a sentence, repeating the sentence aloud and matching the foreign word to a picture.
Results Like in Experiment 1, participants demonstrated a near perfect performance on the picture-matching task in the training phase of the experiment (M = 98.30, SD = 1.14). However, in regards to the testing phase of the experiment, participants did © 2015. John Benjamins Publishing Company All rights reserved
John Rogers, Andrea Révész, & Patrick Rebuschat
not perform significantly above chance (50%) on the grammaticality judgment task (M = 51.34%, SD = 5.52%): t (13) = .908, p = .38, 95% CI [–1.85, 4.53], d = .24. This non-significant result was confirmed by calculating the d-prime values (M = .07, SD = .28), which were non-significant against chance levels: t (13) = .927, p = .37, 95% CI [–.09,.23], d = .51. Retrospective Verbal Reports. An analysis of the debriefing questionnaire revealed that none of the participants had become aware of the underlying rule system during the experiment, but 11/14 participants noticed that the endings of the foreign words were changing during the training phase of the study. When pushed to explain the rule underlying the changing case-endings, most of the participants (7/11) said that they assumed it was related to the gender of the noun. When participants were asked what prompted them to notice the case-markers, several of them stated that they only noticed the changing case-endings when the foreign word repeated across different sentences in close proximity to each other, and several mentioned that they noticed the case markers on the shorter foreign words during the experiment.
Discussion The changes made to the stimulus materials and procedure of the training phase did not result in a clear learning effect in Experiment 2. Also consistent with the results of Experiment 1 was the fact that the participants were highly accurate on the picture-matching task during the exposure phase. Taken together, the disparity between the results suggests that the training conditions were not sufficient to trigger learning of the target grammatical system. As noted above, during the post-experimental debriefing session, several of the participants commented on the fact that they noticed the endings of the shorter foreign words, such as kocka and myska, during the training phase. The length of the foreign word was not a variable that was controlled for in either Experiments 1 or 2, but these comments indicate that word length would be worth considering in future experiments. In addition, some participants explained that they only noticed the endings when the foreign words repeated in a different case. It seems then, as per our original assumption, that the repetition of the foreign word across the different morphological categories might promote noticing of the changing case-ending.
Experiment 3 Experiment 3 had the same objectives as Experiments 1 and 2. Experiment 3 was identical to Experiment 2, with the exception of three alterations which are outlined below.
© 2015. John Benjamins Publishing Company All rights reserved
Validating a novel artificial language
Methods Participants. Forty-two participants were recruited and assigned to either an experimental or a control group (each n = 21). The demographics of the participants were similar to those in Experiments 1 and 2. All participants were native speakers of English and had no background in Slavic or other languages with rich inflectional systems. Stimuli. The first two alterations to Experiment 3 were made to further reduce the overall complexity of the training and testing sets. First, the number of morphological categories was reduced from three (nominative, accusative, instrumental) to only two (nominative, accusative) in both the training and testing sets. In addition, the training set was cut from 48 nouns to only 24 nouns. Only nouns with two syllables were kept. These 24 nouns were repeated three times for both nominative and accusative cases across three training blocks for a total of 144 total stimulus sentences (72 nominative and 72 instrumental). This maintained the same total amount of input as in Experiments 1 and 2, but represented an increase in exposure for each morphological category. Procedure. The final alteration was in regards to the training procedure, which was modified from Experiment 2 in that the participants had to repeat the foreign word in isolation, in addition to repeating the entire sentence. Figure 4 illustrates the modified training procedure for Experiment 3. Listen to sentence
Repeat foreign word aloud Repeat sentence aloud
Fixation Cross Judge Picture
Listen to next sentence
Figure 4. Training procedure in Experiment 3. Each trial consisted of listening to a sentence, repeating the sentence aloud, repeating the foreign word aloud and matching the foreign word to a picture.
Results Like in Experiments 1 and 2, participants were able to complete the picture matching task in the training phase with a near perfect level of accuracy (M = 99.08, SD = 0.82). For the testing phase, the results revealed that the experimental group (M = 55.44%, SD = 7.00%) significantly outperformed the untrained controls (M = 49.71%, SD = 5.80%) on the grammaticality judgment task, t (40) = 3.166, p = .003, 95% CI [2.2%, 10.2%], d = .89. The average d-prime score on the grammaticality judgment task for the experimental group was .314 (SD = .369), which was significantly higher than the average d-prime score of the control group, –.035 (SD = .494), t(40) = 2.587, p = .013,
© 2015. John Benjamins Publishing Company All rights reserved
John Rogers, Andrea Révész, & Patrick Rebuschat
95% CI [.08, .62], d = .80. This significant difference in d-prime values indicates that participants were able to successfully discriminate between grammatical and ungrammatical case-endings in the testing set. Retrospective verbal reports. The analysis of retrospective verbal reports indicated that all participants (21/21) reported noticing the morphological inflections at the end of the foreign words during the training phase of the experiment. However, like in Experiments 1 and 2, none of the participants were able to verbalize the underlying morphological rules. When prompted to guess, 13 participants stated that the affixes might have represented noun class, such as gender. Two other subjects mentioned that they thought that the inflection might be connected in some manner with the position of the foreign word in the sentence, but could not explain the nature of this connection. At the very end of the experiment, when the rules were explained to participants, none claimed that they had thought of the rules at any point during the experiment.
Discussion The results of Experiment 3 demonstrated that participants can develop knowledge of L2 case markings after a minimal amount of exposure, without feedback, and under incidental learning conditions. Given that a number of changes have been made to the training conditions and stimulus materials between experiments, it is not possible to pinpoint which of the alterations, or combination thereof, are responsible for the incidental learning that was observed in Experiment 3. One variable in the present study which undoubtedly impacted on learning was the functional redundancy of the case-markers in the training set. For instance, the example “Last summer the grass ate the koza in the field”, contains the foreign word “koza” with the nominative marker (-a). However, the verb in this sentence, “ate”, implies that the agent of this action is animate, thus making it clear that “koza” is the subject of the sentence, and rendering the inflectional marker “-a” redundant. When we take into account participants’ extreme degree of accuracy on the training task across all three experiments (app. 98%), then it would appear that the training conditions allowed for both the semantic meaning of the foreign word as well as its function (subject or object) to be inferred from the context of the sentence. The availability of multiple cues in the training set would then make it unnecessary for participants to attend to the inflectional markers in order to complete the training task. Given the argument that conscious registration of the input is necessary for learning to take place (e.g. Schmidt, 1990), the redundancy of the case-markers would appear a plausible explanation for the lack of learning in Experiments 1 and 2. In regards to Experiment 3, which resulted in a significant learning effect, it seems likely that both the increase in exposure to the individual morphemes and the repetition of the exemplars (both as part of the sentence © 2015. John Benjamins Publishing Company All rights reserved
Validating a novel artificial language
and individually) were likely to have played a role in achieving a learning effect, given the role that frequency arguably plays in second language acquisition. The procedure of repeating the sentence aloud has also been used successfully in previous experiments (e.g. Rebuschat, 2008; Rebuschat & Williams, 2012; Williams, 2005). It could be argued that repetition led to increased amount of rehearsal within short-term memory, creating additional opportunities for noticing (Schmidt, 1990) to occur at the time of encoding (e.g. Ellis, 2002, 2005, 2008, see also Brooks, Kempe, & Sionov, 2006). One criticism which could be levied against the results of Experiment 3 is that the performance of the experimental group was only slightly above chance-levels (approx. 56%). It is worth reiterating an earlier point that finding even a slight learning effect is often difficult in studies operating in this arena, in particular after such a minimal exposure period (20–30 minutes). Once the initial learning effect has been demonstrated, however, the experimental procedure can be further manipulated to examine how these changes impact on learning. It is also important to compare the results of this experiment to those of previous research. Artificial language learning studies typically result in a 55% to 80% performance on grammaticality judgment tests ( DeKeyser, 2003; Ziori & Pothos, this volume). Although caution needs to be exercised when making comparisons across studies due to differences in designs, participants, and contexts, these trends indicate that the findings of the present study are not atypical within the artificial language paradigm and within SLA research that investigates the acquisition of L2 case marking following incidental exposure. Slightly more encouraging is that the effect size from Experiment 3 (d = .8) is reasonable by the preliminary interpretation (d = .40 small, d = .70 medium, and d = 1.00 large) suggested by Oswald and Plonsky (2010).
General discussion and conclusion This chapter sought to demonstrate the value of describing the process of validating a new artificial language. The larger project of which Experiments 1–3 are part ( Rogers, in prep; Rogers et al., in press) investigates the degree to which L2 case markings can be acquired as a result of incidental exposure. After several modifications to the experimental design of Experiments 1 and 2, including changes to both the training procedure as well as to the stimuli of the training and testing set, Experiment 3 provided evidence that participants can develop some knowledge of L2 case marking under incidental learning conditions, without feedback, and after a limited amount of exposure. Despite the positive results reported here, there are number of limitations to this series of experiments. One such limitation stems from the use of grammaticality judgments as a measure of learning (see Sanz & Grey, this volume, for discussion). The rationale for employing grammaticality judgments was that this instrument has a long © 2015. John Benjamins Publishing Company All rights reserved
John Rogers, Andrea Révész, & Patrick Rebuschat
use in implicit learning research both within cognitive psychology and SLA. However, utilizing grammaticality judgments as a sole measure of learning has been criticized based on the fact that these judgments only assess participants’ ability to recognize the target construction, and provide no information about their ability to produce it (Hama & Leow, 2010; Leow & Hama, 2013). Given the slight learning effect observed only in Experiment 3, it would seem unlikely that any evidence of learning would have emerged on measures requiring the participants to produce the target morphological case-endings. Future research would benefit from employing alternate or multiple measures in order to gain a fuller picture of the quality and quantity of learning that has taken place (see Godfroid & Winke, this volume; Morgan-Short, Faretta-Stutenberg, & Bartlett-Hsu, this volume; Sanz & Grey, this volume, for suggestions). Another limitation of this research lies in the lack of information as regards the strategies adopted by participants when dealing with the stimuli in the training phase. Incidental learning conditions as operationalized here provide no guarantee that learners actually learn incidentally, that is, without intention. For example, Robinson (2002, 2005) reported that his participants engaged in rule-search behavior under an experimental condition which was designed to create opportunities for incidental learning. To address this shortcoming, future research, like many of Robinson’s studies in this area (1997, 2002, 2005), could include retrospective questions asking participants if they were actively involved in searching for rules. The answers to these questions could uncover, for example, if there was variation amongst participants in how they approached the experimental tasks, or if the participants overall demonstrated different behavior than expected in the training phase of the experiment. This information could therefore prove valuable to researchers when deciding how to manipulate the experimental procedure, in particular when confronted with a lack of an overall learning effect, or small effect sizes. In conclusion, despite the limitations of the present study, it is hoped that the series of experiments described here, and the lessons gleaned from these experiments, illustrate some of the challenges of carrying out research with artificial languages and under incidental learning conditions, and serve to inform and encourage future studies in this area.
References Alanen, R. (1995). Input enhancement and rule presentation in second language acquisition. In R. Schmidt (Ed.), Attention and awareness in foreign language learning and teaching (pp. 259–302). Honolulu, HI: University of Hawai’i Press. Andringa, S., & Ćurčić, M. (2015). How explicit knowledge affects online L2 processing: Evidence from differential object marking acquisition. Studies in Second Language Acqusition, 35(2), 237–268.
© 2015. John Benjamins Publishing Company All rights reserved
Validating a novel artificial language
Braine, M.D.S. (1963). On learning the grammatical order of words. Psychological Review, 70(4), 323–348. DOI: 10.1037/h0047696 Braine, M.D.S. (1966). Learning the positions of words relative to a marker element. Journal of Experimental Psychology, 72(4), 532–540. DOI: 10.1037/h0023763 Brooks, P.J. & Kempe, V. (2013). Individual differences in adult foreign language learning: The mediating effect of metalinguistic awareness. Memory and Cognition, 41, 281–296. DOI: 10.3758/s13421-012-0262-9 Brooks, P.J., Kempe, V., & Sionov, A. (2006). The role of learner and input variables in learning inflectional morphology. Applied Psycholinguistics, 27, 185–209. DOI: 10.1017/S0142716406060243 Cedrus Corporation. (2012). SuperLab (Version 4.5). San Pedro, CA. Cobb, T. (2012). Web Vocabprofile, accessed 14 November 2012 from 〈http://www.lextutor.ca/ vp/〉, an adaptation of Heatley & Nation’s (1994) Range. Cook, V.J. (1988). Language learners’ extrapolation of word order in micro-artificial languages. Language Learning, 38, 497–529. DOI: 10.1111/j.1467-1770.1988.tb00165.x de Graaf, R. (1997). The eXperanto experiment: Effects of explicit instruction on second language acquisition. Studies in Second Language Acquisition, 19, 249–297. DOI: 10.1017/S0272263197002064 DeKeyser, R.M. (1994). Implicit and explicit learning of L2 grammar: A pilot study. TESOL Quarterly, 28(1), 188–94. DOI: 10.2307/3587210 DeKeyser, R.M. (1995). Learning second language grammar rules. Studies in Second Language Acquisition, 17(3), 379–410. DOI: 10.1017/S027226310001425X DeKeyser, R.M. (1997). Beyond explicit rule learning: Automatizing second language morphosyntax. Studies in Second Language Acquisition, 19, 195–221. DOI: 10.1017/S0272263197002040 DeKeyser, R.M. (2003). Implicit and explicit learning. In C. Doughty & M. Long (Eds.), The handbook of second language acquisition (pp. 313–348). Oxford: Blackwell. Ellis, N.C. (1999). Cognitive approaches to SLA. Annual Review of Applied Linguistics, 19, 22–42. Ellis, N.C. (2002). Frequency effects in language acquisition: A review with implications for theories of implicit and explicit language acquisition. Studies in Second Language Acquisition, 24, 143–188. Ellis, N.C. (2005). At the interface: Dynamic interactions of explicit and implicit language knowledge. Studies in Second Language Acquisition, 27, 305–352. Ellis, N.C. (2008). The dynamics of second language emergence: Cycles of language use, language change, and language acquisition. Modern Language Journal, 92(2), 232–49. DOI: 10.1111/j.1540-4781.2008.00716.x Gómez, R.L., & Gerken, L.A. (2000). Infant artificial language learning and language acquisition. Trends in Cognitive Sciences, 4(5), 178–186. DOI: 10.1016/S1364-6613(00)01467-4 Gullberg, M., Roberts, L., & Dimroth, C. (2012). What word-level knowledge can adult learners acquire after minimal exposure to a new language? International Review of Applied Linguistics, 50, 239–276. Gullberg, M., Roberts, L., Dimroth, C., Veroude, K. & Indefrey, P. (2010). Adult language learning after minimal exposure to an unknown natural language. Language Learning, 60, 5–24. DOI: 10.1111/j.1467-9922.2010.00598.x Grey, S. (2013). A neurocognitive investigation of bilingual advantages at additional language learning. Unpublished doctoral dissertation. Georgetown University.
© 2015. John Benjamins Publishing Company All rights reserved
John Rogers, Andrea Révész, & Patrick Rebuschat Grey, S., Williams, J.N., & Rebuschat, P. (2014). Incidental exposure and L3 learning of morphosyntax. Studies in Second Language Acquisition, 36, 1–34. DOI: 10.1017/S0272263113000727 Hama, M., & Leow, R.P. (2010). Learning without awareness revisited. Studies in Second Language Acquisition, 32(03), 465–491. DOI: 10.1017/S0272263110000045 Hamrick, P. (2013). Development of conscious knowledge during early incidental learning of L2 syntax. Unpublished dissertation. Georgetown University. Hulstijn, J.H. (1989). Implicit and incidental second language learning: Experiments in the processing of natural and partly artificial input. In H. W. Dechert & M. Raupach (Eds.), Interlingual processing (pp. 49–73). Tubingen: Gunter Narr. Hulstijn, J.H. (1997). Second language acquisition research in the laboratory. Studies in Second Language Acquisition, 19, 131–43. DOI: 10.1017/S0272263197002015 Hulstijn, J.H., Young, R.F., Ortega, L., Bigelow, M., DeKeyser, R., Ellis, N.C., Lantolf, J.P., Mackey, A., & Talmy, S. (2014). Bridging the gap: Cognitive and social approaches to research in second language learning and teaching. Studies in Second Language Acquisition, 36, 361–421. DOI: 10.1017/S0272263114000035 Jackson, D.O. (2014). The relative contribution of input modification, learner awareness, and individual differences to second language construction learning. Unpublished doctoral dissertation. University of Hawai’i at Manoa. Kempe, V., Brooks, P.J., & Kharkhurin, A. (2010). Cognitive predictors of generalization of Russian grammatical gender categories. Language Learning, 60, 127–153. DOI: 10.1111/j.1467-9922.2009.00553.x Leow, R.P., & Hama, M.K. (2013). Implicit learning in SLA and the issue of internal validity: A response to Leung and Williams’ ‘The implicit learning of mappings between forms and contextually derived meanings’. Studies in Second Language Acquisition, 35(3), 545–557. DOI: 10.1017/S027226311300003X Leung, J. (2007). Implicit learning of form-meaning connections. Unpublished doctoral dissertation. University of Cambridge. Leung, J.H.C., & Williams, J.N. (2011). The implicit learning of mappings between forms and contextually derived meanings. Studies in Second Language Acquisition, 33(1), 33–55. DOI: 10.1017/S0272263110000525 Leung, J.H.C., & Williams, J.N. (2012). Constraints on implicit learning of grammatical formmeaning connections. Language Learning, 62(2), 634–662. DOI: 10.1111/j.1467-9922.2011.00637.x Leung, J.H.C., & Williams, J.N. (2014). Crosslinguistic differences in implicit language learning. Studies in Second Language Acquisition, ,36, 733–755. Macmillan, N., & Creelman, C. (2005). Detection theory: A user’s guide. Mahwah, N.J: Lawrence Erlbaum Associates. Martin, C.D., Dering, B., Thomas, E.M., & Thierry, G. (2009). Brain potentials reveal semantic priming in both the ‘active’ and the ‘non-attended’ language of early bilinguals. Neuroimage, 47, 326–333. DOI: 10.1016/j.neuroimage.2009.04.025 MacWhinney, B. (1983). Miniature linguistic systems as tests of the use of universal operating principles in second-language learning by children and adults. Journal of Psycholinguistic Research, 12, 467–478. DOI: 10.1007/BF01068027 McLaughlin, B. (1980). On the use of miniature artificial languages in second-language research. Applied Psycholinguistics, 1(4), 357–69. DOI: 10.1017/S0142716400001004
© 2015. John Benjamins Publishing Company All rights reserved
Validating a novel artificial language
Moeser, S.D., & Bregman, A.S. (1972). The role of reference in the acquisition of a miniature artificial language. Journal of Verbal Learning and Verbal Behavior, 11(6), 759–769. DOI: 10.1016/S0022-5371(72)80010-0 Morgan-Short, K. (2007). A neurolinguistic investigation of late-learned second language: Effects of explicit and implicit conditions. Unpublished doctoral dissertation. Georgetown University. Morgan-Short, K., Faretta-Stutenberg, M., Brill-Schuetz, K.A., Carpenter, H., & Wong, P. (2014). Declarative and procedural memory as individual differences in second language acquisition. Bilingualism: Language and Cognition, 17, 56–72. DOI: 10.1017/S1366728912000715 Oswald, F.L., & Plonsky, L. (2010). Meta-analysis in second language research: Choices and challenges. Annual Review of Applied Linguistics, 30, 85–110. DOI: 10.1017/S0267190510000115 Paciorek, A. (2012). Implicit learning of semantic preferences. Unpublished doctoral dissertation. University of Cambridge. Reber, A.S. (1993). Implicit learning and tacit knowledge: An essay on the cognitive unconscious. London: Clarendon Press. Rebuschat, P. (2008). Implicit learning of natural language syntax. Unpublished doctoral dissertation. University of Cambridge. Rebuschat, P. (2013). Measuring implicit and explicit knowledge in second language research. Language Learning, 63(3), 595–626. DOI: 10.1111/lang.12010 Rebuschat, P., Hamrick, P., Sachs, R., Riestenberg, K., & Ziegler, N. (2013). Implicit and explicit knowledge of form-meaning connections: Evidence from subjective measures of awareness. In J. Bergsleithner, S. Frota, & J. K. Yoshioka (Eds.), Noticing: L2 studies and essays in honor of Dick Schmidt (pp. 255–275). Honolulu, HI: University of Hawaii Press. Rebuschat, P., Hamrick, P., Riestenberg, K., Sachs, R., & Ziegler, N. (2015). Triangulating measures of awareness: A contribution to the debate on learning without awareness. Studies in Second Language Acquisition, 37(2), 299 – 334. Rebuschat, P., & Williams, J.N. (2006). Dissociating implicit and explicit learning of natural language syntax. In R. Sun. & N. Miyake. (Eds.), Proceedings of the annual meeting of the Cognitive Science Society (p. 2594). Mahwah, N.J.: Lawrence Erlbaum. Rebuschat, P., & Williams, J.N. (2009). Implicit learning of word order. In N. Taatgen & H. van Rijn (Eds.), Proceedings of the 31th annual conference of the Cognitive Science Society (pp. 425–430). Austin, TX: Cognitive Science Society. Rebuschat, P., & Williams, J.N. (2012). Implicit and explicit knowledge in second language acquisition. Applied Psycholinguistics, 33(4), 829–856. DOI: 10.1017/S0142716411000580 Robinson, P. (1997). Generalizability and automaticity of second language learning under implicit, incidental, enhanced, and instructed conditions. Studies in Second Language Acquisition, 19(2), 223–247. DOI: 10.1017/S0272263197002052 Robinson, P. (2002). Effects of individual differences in intelligence, aptitude and working memory on adult incidental SLA: A replication and extension of Reber, Walkenfeld and Hernstadt, 1991. In P. Robinson (Ed.), Individual differences and instructed language learning (pp. 211–266). Amsterdam: John Benjamins. DOI: 10.1075/lllt.2.13rob Robinson, P. (2005). Cognitive abilities, chunk-strength, and frequency effects in implicit artificial grammar and incidental L2 learning: Replications of Reber, Walkenfeld, and Hernstadt (1991) and Knowlton and Squire (1996) and their relevance for SLA. Studies in Second Language Acquisition, 27(2), 235–268. DOI: 10.1017/S0272263105050126
© 2015. John Benjamins Publishing Company All rights reserved
John Rogers, Andrea Révész, & Patrick Rebuschat Robinson, P. (2010). Implicit artificial grammar and incidental natural second language learning: How comparable are they?. Language Learning, 60 (supp. 2), 245–63. DOI: 10.1111/j.1467-9922.2010.00608.x Rogers, J. (in prep). Developing implicit and explicit knowledge of L2 case-marking under incidental learning conditions. Unpublished doctoral dissertation. UCL Institute of Education, University College London. Rogers, J., Révész, A., & Rebuschat, P. (in press). Implicit and explicit knowledge of L2 inflectional morphology. Applied Psycholinguistics Schmidt, R. (1990). The role of consciousness in second language learning. Applied Linguistics, 11, 129–158. DOI: 10.1093/applin/11.2.129 Schmidt, R. (1994). Implicit learning and the cognitive unconscious: Of artificial grammars and SLA. In N. Ellis (Ed.), Implicit and explicit learning of languages (pp. 165–209). London: Academic Press. Segal, E.M., & Halwes, T.G. (1965). Learning of letter pairs as a prototype of first language learning. Psychonomic Science, 3(1), 451–452. DOI: 10.3758/BF03343227 Segal, E.M., & Halwes, T.G. (1966). The influence of frequency of exposure on the learning of a phrase structural grammar. Psychonomic Science, 4(1), 157–158. DOI: 10.3758/BF03342226 Serafini, E. (2013). Cognitive and psychological factors in the long-term development of implicit and explicit second language knowledge in adult learners of Spanish at increasing proficiency. Unpublished doctoral dissertation. Georgetown University. Smith, K.H. (1966). Grammatical intrusions in the recall of structured letter pairs: Mediated transfer or position learning? Journal of Experimental Psychology, 72, 580–588. DOI: 10.1037/h0023768 Tagarelli, K.M. (2014). The neurocognition of adult second language acquisition: An fMRI study. Unpublished doctoral dissertation. Georgetown University. Tagarelli, K., Borges Mota, M., & Rebuschat, P. (2011). The role of working memory in the implicit and explicit learning of languages. In L. Carlson, C. Hölscher, & T. Shipley (Eds.), Proceedings of the 33rd annual conference of the Cognitive Science Society (pp. 2061–2066). Austin, TX: Cognitive Science Society. Tagarelli, K., Borges Mota, M., & Rebuschat, P. (2015). Working memory, learning context, and the acquisition of L2 syntax. In W. Zhisheng., M. Borges Mota., & A. McNeill. (Eds.), Working memory in second language acquisition and processing: Theory, research and commentary (pp. 224–247). Bristol: Multilingual Matters. Tagarelli, K.M., Jiang, X., Laka, I., Barbey, A.K., Morgan-Short, K., Ullman, M.T. (2014, March) Examining the trajectory of language acquisition with a mini-language model. Paper presented at the annual meeting of the AAAL Annual Conference. Portland, Oregon. Tellier, A., & Roehr-Brackin, K. (2013). Metalinguistic awareness in children with differing language learning experience. In L. Roberts, A. Ewert, M. Pawlak, & M. Wrembel (Eds.), EuroSLA Yearbook, 13, 81–108. Amsterdam: John Benjamins. DOI: 10.1075/eurosla.13.06tel Thierry, G., & Wu., Y.J. (2007). Brain potentials reveal unconscous translation during foreignlanguage comprehension. Proceedings of the National Academy of Sciences of the USA. 104(30), 12530–12535. DOI: 10.1073/pnas.0609927104
© 2015. John Benjamins Publishing Company All rights reserved
Validating a novel artificial language
Williams, J.N. (2004). Implicit learning of form-meaning connections. In B. VanPatten, J. Williams, S. Rott, & M. Overstreet (Eds.), Form-meaning connections in second language acquisition (pp. 203–218). Mahwah, NJ: Lawrence Erlbaum Associates. Williams, J.N. (2005). Learning without awareness. Studies in Second Language Acquisition, 27(2), 269–304. DOI: 10.1017/S0272263105050138 William, J.N. (2010). Initial incidental acquisition of word order regularities: Is it just sequence learning?. Language Learning, 60(supp. 2), 221–244. DOI: 10.1111/j.1467-9922.2010.00607.x Williams, J.N., & Kuribara, C. (2008). Comparing a nativist and emergentist approach to the initial stage of SLA: An investigation of Japanese scrambling. Lingua, 118(4), 522–553. DOI: 10.1016/j.lingua.2007.03.003 Winter, B., & Reber, A.S. (1994). Implicit learning and the acquisition of natural languages. In N. Ellis (Ed.), Implicit and explicit learning of languages (pp. 115–145). London: Academic Press. Wu, Y.J., & Thierry, G. (2010). Chinese-English bilinguals reading English hear Chinese. The Journal of Neuroscience, 30(22), 7646–7651. DOI: 10.1523/JNEUROSCI.1602-10.2010
Appendix A Sample items and pictures presented during the exposure phase for nominative, accusative, and instrumental items. Morphological category
Example sentence and pictures
Nominative
Last summer the grass ate the koza in the field
All night the lednicka cooled the food in the store
(Continued)
© 2015. John Benjamins Publishing Company All rights reserved
John Rogers, Andrea Révész, & Patrick Rebuschat
Appendix A. (Continued) Morphological category
Example sentence and pictures
Accusative
Last week a zkousku gave the teacher to the class
Today Peter used a britvu in the bathroom
Instrumental
All morning a hole dug James with the lopatou
Last night David started a fire with a sirkou
© 2015. John Benjamins Publishing Company All rights reserved
Validating a novel artificial language
Appendix B Examples of Grammatical and Ungrammatical Patterns Used in the Grammaticality Judgment Testing Sets. Pattern
Grammatical
Nominative
In the evening the bunda warmed him in the park.
Accusative
This morning she changed the plenu in the bathroom.
Instrumental
Today the cut cleaned the doctor with a lupou. Ungrammatical
*Nominative (-u)
In summer the lisku ate the eggs in the field.
*Nominative (-ou)
Last weekend a dress bought the zenou at the shop.
*Accusative (-a)
Last weekend he cooked the kachna in the oven.
*Accusative (-ou)
Last year a paprikou grew Beth in the garden.
*Instrumental (-a)
Last week his wife surprised he with a kvetina.
*Instrumental (-u)
In the winter James cleared the ice with a skrabku.
© 2015. John Benjamins Publishing Company All rights reserved
