Elements of Research in CALL Software Development Projects

Marina Dodigovic

Elements of Research in CALL Software Development Projects Marina Dodigovic Macquarie University ABSTRACT CALL is a relatively young discipline and is greatly affected by the consequences of a predominant focus on program development at the expense of basic research. The central issue of this paper involves the integration of research components into the CALL development process for the purpose of not only improving the quality of software but also demonstrating how empirical observations that emerge during the course of development can have an important impact on underlying theory.

KEYWORDS Research and Development, Scientific Method, Culture, Natural Language Processing, Artificial Intelligence, Student Modeling

INTRODUCTION It is difficult to imagine a good CALL software development project completely detached from research. Effective CALL development requires a multidisciplinary approach to software design and implementation that draws consistently on fundamental research in disciplines such as linguistics, sociolinguistics, phonology, language acquisition, psychology, learning theory, instructional methodology, and software engineering. The integration of these disciplines frequently poses new research questions that are answered over the course of the development process. CALL programs are very often intended to address specific learning problems which require initial scientific explanations, or they are based on teaching methods which can only be confirmed when the programs are ultimately implemented. In most CALL projects, research forms an important part of the development process (Levy, 1997). However, this research frequently goes unnoticed by the academic community, and developers often fail to receive academic recognition for the research component imbedded in their development projects. Even some publications in the area of CALL laVolume 15 Number 4

25

Elements of Research in CALL Development Projects ment the apparent lack of documented research (Cameron, 1997; Brett, 1998). Whether this controversy is a mere coincidence or an integral part of a recognizable pattern of CALL program development is a question which cannot be answered until the academic convention of distinguishing between research and development is critically examined.

RESEARCH VERSUS DEVELOPMENT The reasons that research is not uniformly recognized as a constituent part of CALL software development projects can be sought both within and without the CALL profession. Outside the profession, in the sciences, research and development tend to be viewed as separate entities, development following upon research as a simple application process (Valter, 1988). For example, the Macquarie Dictionary (1996) defines research and development as “that part of industry concerned with scientific research and the technological development of the results.” Perhaps it is this very distinction that polarizes CALL developers themselves. In his extremely thoughtful essay on theory driven CALL, Levy (1997) finds that CALL development is frequently divided between those who make a particular theory their point of departure (formalists) and those who make discoveries while writing programs (proceduralists). The former appear to believe that completed research is a prerequisite for successful development, whereas the latter allow for the integration of research into the development process. Levy acknowledges that revision of original theory is a recurrent factor when development is undertaken in a multidisciplinary area such as CALL. Knowledge gained in the course of development may confirm or disconfirm various aspects of an underlying theory, thereby altering and improving the original theory itself. Clearly, the research process is far from complete when a theory is selected to serve as a basis for a CALL development project. Levy’s insight coincides with a modern circular approach to scientific method in which research activities are presented as circular activities; a theory is followed by an hypothesis which leads to empirical observations which in turn lead to generalizations which finally have an effect on the theory itself (see Lewins, 1990). According to this model, theories are not the only valid starting point for research programs; research can legitimately start at any of the four stages in the circle described in Figure 1.

26

CALICO Journal

Marina Dodigovic Figure 1 Research Activities in the Circular Scientific Method

theory theories

empirical empirical generalizations generalizations

hypothesis hypothesis

observations observation

The model above differs from the previously promulgated linear approach to research which holds that researchers should start with a theory or an hypothesis, collect and classify data, and conclude with generalizations. Although potentially perplexing to some scholars, the circular view of the scientific method seems to be gaining acceptance in the academy (Lewins, 1990; Levy, 1997). This growing acceptance again poses the crucial question of why CALL development is not likely to be treated as a research activity, despite the fact that it can support systematic inquiry into the subject, result in new knowledge, and contribute to basic theories. The only plausible answer to this question seems to reside in the difference between the view of research held by the academy as an institution and the approach to scientific method as a set of beliefs held by individual faculty members (see Dodigovic, 1993). Theories and beliefs of individual faculty members are the first to change because they suffer less from academic inertia than institutions. In fact, institutions only change to accommodate a new model when that model has been comfortably accepted by the majority of its members. Since the academy still greatly relies on the linear approach to scientific research, it may well take some time for it to adapt to the changes dictated by the new insights in epistemology. The above discussion is based on the Russian school of semiotics, according to which culture is seen as a finite set of conventions with a ‘cenVolume 15 Number 4

27

Elements of Research in CALL Development Projects ter’ (with standard—or canonical—forms), a ‘periphery’ (with substandard or subculture forms) and a ‘border’ (dividing culture from non-culture or one culture from another) (see Petkovic, 1984). The changes in a culture, coming either from within (evolution) or without (genesis), always occur at the periphery, which means that every innovation at first has the stigma of being a substandard form. The academy as an institution is a culture in its own right, and the status of the circular scientific method still occupies somewhat of a peripheral position. Given the dynamic evolution of culture, substandard forms tend to move from the periphery towards the center, if adequately supported, and eventually become standard forms themselves. Consequently, as acceptance of the circular view of the scientific method grows among the members of the academy, it will likely move towards the center of that culture. Until then, we will continue to witness a constant need, as is always the case within a culture, to bridge the gap between the no longer adequate standard forms and growing substandard forms. The traditional linear model pervades the whole infrastructure of CALL funding, with the result that support is frequently provided by financial agencies other than research agencies. Requests for funding often become complicated because faculty who seek funds for CALL development projects that incorporate explicitly formulated research agendas do not usually submit two separate applications—one for research and one for development. They are more likely to apply for funding from a single source not only to minimize the administrative effort involved in the grant application but also to reduce the risk of obtaining development funds without adequate research funds. Economic rationalism seems to be another barrier that keeps CALL development officially separate from research. For example, anecdotal evidence suggests that the European LINGUA agency, which specializes in funding language learning development programs, initially looked with disfavor on CALL development applications that included research components. The agency apparently feared that research activities would lead to research papers rather than to practical software solutions. In addition to the development rationalization prevalent in the academy, CALL development projects conducted by commercial institutions probably face a similar, if not even more restrictive, set of regulations. Levy (1997) has also pointed out that some CALL development projects are mainly technology driven. Some developers do not pay sufficient attention to the linguistic underpinnings or methodological features of the program under development but, rather, appear to be motivated mainly by the computer’s technological capabilities. Even though research can potentially play an important role in these projects, developers seem to take no notice of such considerations with the result that development projects often fail to deliver new theoretical insights. This situation can 28

CALICO Journal

Marina Dodigovic only be explained by the fact that, from the semiotic point of view, CALL itself is still a substandard or peripheral discipline and does not necessarily conform to the canonical procedures of standard cultural forms. The preceding paragraphs have proposed several reasons for the general unwillingness to categorize CALL development activities as activities which offer research contributions. The most compelling of these reasons seems to be the principle of the linear approach to research which is still strongly entrenched in the central cultural position in academic institutions. This principle strictly differentiates between research and development in terms of funding and frequently relegates development projects to the periphery of funding priorities.

CONSEQUENCES OF CALL DEVELOPMENT PROJECTS FOR PROFESSIONAL ADVANCEMENT At many universities, research grants tend to be more highly valued than development grants. Younger faculty members often refrain from applying for development grants because they feel the need to concentrate their energies on what is considered to be pure research in order to advance their careers. Embarking on a CALL development project requires a considerable degree of professional courage when one considers the possible ramifications of unwise career choices. In this context, being able to prove that CALL development projects entail at least some research is a crucial step towards the emancipation of CALL as an academic discipline. Developers would do well to identify the research components in potential CALL development projects, to organize research projects around these components (with or without separate funding), and, most important, to present the results of research projects in separate research publications. Such a strategy will not only bolster one’s chances for professional advancement but also will yield valuable contributions to CALL as an area of legitimate research. The effective use of this strategy will also strengthen the position of CALL in the academic world and sustain the reputation of CALL researchers.

IDENTIFICATION OF RESEARCH COMPONENTS AND THE IMPORTANCE OF EARLY PLANNING The identification of research components in CALL development projects should ideally start as early as possible, in the initial phase of needs analysis and project planning. This approach is usually proposed by the supporters of the linear method of scientific research and, although not always feasible, does present certain advantages (see Manning, 1991). DeVolume 15 Number 4

29

Elements of Research in CALL Development Projects velopers who identify the research potential of the project at an early stage can then take appropriate steps to organize the research and development components in an optimal way. (It should be noted this approach also has some disadvantages; some research questions may not be identifiable until an advanced stage of the project.) Institutions, which generally favor the linear scientific method, show a marked preference for early identification of research components which may well limit the developer’s choices in research planning. In addition, institutionally supported research programs are often viewed in terms of being financially justifiable. Projects are often seen as enterprises which will hopefully meet their intended objectives; the objectives in this case being hypotheses to be proved by the end of the project. In this context, late identification of research hypotheses is likely to be interpreted as a side effect rather than a success of the project. Numerous opportunities for good research exist in CALL development projects. However, scholars who undertake these projects may not be aware of the implications that early research planning can have for the project as a whole or the cluster of other research projects derived from the original project. Because institutions favor projects designed to address problems anticipated in advance, it is in developers’ best interest to identify primary and secondary research questions. In the case that unexpected outcomes occur, by virtue of the very nature of the circular scientific method, even the most exciting discovery that requires some further action will most probably be regarded as a failure in financial planning. Further, a new project drafted around this discovery may well yield results conflicting with those of the original project with the consequence that the previous research might be viewed as unsuccessful or cost ineffective.

THE PASSIV PROGRAM The observations offered in this article are based on the author’s own experience in articulating her research needs within the context of a CALL development project. She developed her first serious CALL package, PASSIV, with considerable enthusiasm. After an extensive review of current CALL literature, she turned to various learning theories and other authorities for help in lesson design (Dodigovic, 1991). The explicitly stated research goal of the project was a post-development goal—to measure the effectiveness of a specific electronic tool to combat a clearly identifiable learning problem of a well defined group of learners. The investigation proved that certain aspects of the software package were beneficial to the learners. Although success in the project was achieved partly because of the inherent flexibility, the diagnostic feedback, and the self-paced learning features unique to the software, students’ actual improvement could 30

CALICO Journal

Marina Dodigovic be traced back to an underlying linguistic assumption. At the time, this assumption was merely a suspicion on the part of the author, based on her experience with a given type of academic discourse. This suspicion led to a systematic compilation of a corpus and a linguistic study. In effect, the author’s original hunch became a research hypothesis that the development project proved to be correct (Dodigovic, 1995). If the hypothesis had turned out to be incorrect, it would have thrown a completely different light on the whole cluster of individual research projects. The linguistic, or maybe rather the epistemological, merit of the PASSIV program was that it established a pragmatic link between the use of the passive voice in scientific discourse and the application of a model of formal logical reasoning to academic discourse (Dodigovic, 1993). Linguistics had so far treated academic language as a linguistic code only, but the research inspired by the success of the PASSIV program demonstrated that academic language is subject to multiple encoding and decoding procedures: linguistic, logical, and subject specific processes (Dodigovic, 1995). This view has now been accepted and discussed by other authors (see, for example, Hwu, 1997). The research component included in the software project had other implications for the research cluster derived from the original project. Had the ex post facto investigation into the linguistic foundations of the successful software package found that these foundations were in fact wrong, this situation (apart from being highly embarrassing to the researcher) would have posed several new questions. Questions would have arisen as to why the learning outcomes were positive, even though the software was based on a faulty linguistic assumption, or even whether the software package should be abandoned because its linguistic foundations were not sound. Of course, negative conclusions could also have meant that the pedagogical variables had not been sufficiently well controlled during the experiment. In summary, failure to identify the research component early in the project’s planning phase could have proven to be a great disadvantage to the whole research cluster that evolved during the project. On the other hand, the process of developing and implementing the program made a linguistic discovery possible, which led to an empirical generalization, which, in turn, led to an improvement in the underlying theory. The project clearly demonstrated that the scientific method does not have to be linear; successful projects do not have to start with a theory or hypothesis and blindly follow the consequences of that theory or hypothesis to their ultimate conclusions. Another factor which argues against the linear scientific method and early identification of research questions is the interdisciplinary nature of CALL. The complex nature of CALL shifts and changes the context of CALL development projects. The meanings of terms borrowed from other disciplines such as linguistics, psychology, and learning theory change in Volume 15 Number 4

31

Elements of Research in CALL Development Projects ways which may not be easy to predict (Petkovic, 1984). What can be predicted, however, are the areas in which these disciplines overlap. The issues and research questions common to these disciplines support the formulation of a set of general hypotheses which can then be used to emulate the linear scientific method. Due to the marked preference for linear and sequential projects for which support is sought, institutions and agencies expect the specification of possible research components in a research cluster early on. Because the circular nature of the development process itself will inevitably raise questions for further research, it is important to provide at least some rough approximation of the areas in which research questions are most likely to occur.

EXAMPLE CLUSTER: TEACHING ACADEMIC ENGLISH ON THE WEB This section describes an example of the early identification of research questions in a CALL development project. This project, called Teaching Academic English on the Web, has recently been funded by the Macquarie University Flagship Scheme. The project will allow the English for Academic Purposes program (EAP) at Macquarie University to be extended in flexible ways via the Web to improve student learning. Since the EAP program is the first contact for many students with Macquarie University, successful implementation of the CALL program will enhance the university’s reputation for instructional innovation and the use of best teaching practices. The EAP program provides a range of services for up to 500 national and international students at Macquarie University every year. The CALL development project will cover three courses that carry university credit as well as a number of non-credit workshops. Making an interactive EAP learning package available to these students will improve the quality of their learning by giving them much more feedback and individual guidance than they previously received in traditional instructional formats. The main modules of the package will consist of 1) hypertext course notes, 2) interactive tasks, 3) grammar and vocabulary tests, 4) an essay marking aid, and 5) a corpus explorer.

1.Hypertext Course Notes The hypertext course notes are essentially lecture notes with hypertext links to external resources such as CD-ROM multimedia dictionaries, concordancing software, pronunciation aids, and other Web sites. 32

CALICO Journal

Marina Dodigovic 2.Interactive Tasks The interactive tasks are directly related to concepts explained in the course notes and provide individualized feedback within a learner centered approach. This module will feature an intelligent parser capable of analyzing student sentences for grammatical correctness. 3.Grammar and Vocabulary Tests The grammar and vocabulary tests include automatic assessment tools designed to free instructors to focus on more creative tasks such as more individualized approaches to teaching. 4.An Essay Marking Aid The essay marking aid is intended to provide a means for delivering feedback to students in more efficient ways. Based on earlier research of students’ errors in writing essays, typical errors will be classified into distinct groups. Students will be able to examine various types of errors by choosing relevant sentences from a database. 5.A Corpus Explorer The corpus explorer permits students to search for examples of usage of particular words and expressions in academic writing. Students will have the choice of various commercially available corpora or Macquarie University research based corpora. All modules are to be interconnected via hypertext links. The goals of the program are a) to make the courses available to students both on campus and at a distance, b) to provide flexible, student centered, and self-paced learning, c) to introduce innovative teaching techniques for language exploration and self-assessment, and d) to capitalize on students’ interest in using computers (see Dodigovic, 1995, 1998). The organized hypertext links to other Internet resources will also enable students to use the Web as a library of up-to-date, authentic language texts (Borchardt, 1998). The program is expected to lead to substantial improvements in the quality of course delivery for students by offering learning strategies that meet individual students’ needs (see Bull, 1997; Liou, 1997). The system will give students a choice of learning techniques and help them to find the technique that works best for them. Finally, students will be free to work with the package on their own and explore the aspects of the program of special interest to them. The teaching methods to be implemented in the package will be ‘tutorial’ and ‘non-tutorial’ (Higgins, 1988). The tutorial method will include Volume 15 Number 4

33

Elements of Research in CALL Development Projects the provision of explicit guidance, as though students were working under the close supervision of a tutor. The non-tutorial method will consist of an exploratory mode of learning in which students will be able to browse through the materials they wish to explore or engage in a learning activity which requires integrative competency in the language (Higgins, 1988). The intelligent sentence parser is the truly innovative part of the software package. The parser will evaluate the grammar of sentences students enter and provide quick feedback and correction. Although parsers have been developed at a number of research institutions, their full potential in education has not been exploited on a wider scale. In fact, astonishingly few language teaching programs make use of parsers mainly because of the traditionally wide gap between research in Natural Language Processing (NLP) and courseware development. One of the few exceptions to this tendency is the program at University of Exeter, which has contributed pioneering efforts to the integration of NLP into CALL (Pollard & Yazdani, 1995). Normally, parsers developed for research purposes can handle only correct language usage because they are based on a set of perfectly correct grammar rules. In order to be able to deal with language expressions produced by language learners, parsers must have a grammar component based on a model of students’ hypotheses about grammar rules (O’Brien, 1995). Such grammars can be best developed by analyzing a corpus with representative language samples of specific types of users. Evaluation of the project will be conducted in several steps. Formative evaluation is planned as a series of user and peer testing sessions which will take place at critical points in the prototyping and full scale development phases. Summative evaluation is likely to become a research project in its own right. This broader form of evaluation will involve program testing in controlled experimental conditions and overall program evaluation by experienced teachers and CALL specialists. The variables to be measured in the evaluation include a) attitude towards the training program, b) usability of the instructional program, c) comprehension, retention, and transfer of knowledge gained, and d) attitude towards subject matter. For this purpose, a combination of think aloud protocols, interviews, questionnaires, and error protocols will be used (Gordon, 1994). RESEARCH COMPONENTS IN THE EAP CLUSTER The intelligent sentence parser and the summative evaluation part of the project offer considerable research potential. Investigation of questions in these two areas can be expected to provide new information to address academic questions such as what constitutes typical student linguistic behavior in response to certain linguistic tasks or how hypertext resources facilitate the non-linear reading of texts. Figure 2 lists estimates of the amount of time needed to complete each stage of the project. 34

CALICO Journal

Marina Dodigovic Figure 2 Project Timeline 1998 1999 JFMAMJJASONDJFMAMJJASOND Needs Analysis Beginning Prototyping Style Guidelines Implementation User Testing Peer Testing Revision Full Style Manual Imaging Student Model Description Full Scale Development Evaluation End

It is anticipated that the NLP based student model design and summative evaluation will together account for approximately one half of the total time to be spent on the project. Because of the importance of these activities and the time required to complete each of them, it is reasonable to organize each one around a separate research project. This organization will assure scientific validity of the content on the one hand and the procedural validity of the evaluation process on the other.

PROJECT MANAGEMENT IMPLICATIONS Once a CALL development project has been started, the identification of research questions necessarily raises the need for adjustments in the project. What may have originally seemed to be one project may well become three or more projects. In order to avoid neglecting any of the ancilVolume 15 Number 4

35

Elements of Research in CALL Development Projects lary projects in the research cluster, each will need to be managed separately. For the faculty who are obliged to present evidence of effective research, separate project management can help them focus more clearly on specific research issues and tailor their publications accordingly. In this case, the management of the whole research cluster will inevitably become more complex. Funding for separate research projects is yet another matter to consider. Isolating separate research objectives which go beyond the original development goals will help these projects gain recognition from funding agencies because research funding agencies are not likely to fund an activity aimed primarily at improving a given set of teaching materials. For example, the EAP project described here has three potential research components: student grammar, learning strategies modeling, and evaluation of instructional effectiveness. If these three activities were conducted mainly to satisfy the criteria of the current development project, they would not likely be supported as separate research projects. If, however, they are extended in an effort to answer questions such as how certain types of learners respond to academic writing tasks or how hypertext resources encourage non-linear reading, then these projects will stand on their own merits regardless of their potential contributions to the current development project. Even if separate funding is not granted for research activities, it is still important to define general research objectives because the definition of such objectives will enable developers to find clearer answers to problems and increase their awareness of possible discrepancies in their developmental model. CONCLUSION It is of utmost importance to produce effective and innovative software in CALL development projects. In order to meet this criterion, development projects will inevitably address new problems and provide answers to existing questions. These conditions are ideal for research either internal or external to the development project. It is also important to state the research objectives unequivocally because academic institutions normally attach greater importance to research activities than to teaching aids development. While some institutions foster and recognize the research contributions of development projects, others tend to overlook their scientific value because they view development projects as the linear sequels to research findings. Strengthening the role of research in CALL development projects by identifying research components in the projects and promoting the research components in their own right will substantially help resolve the problem. The use of both strategies will have a beneficial effect on the quality of CALL software as well as the standing of CALL as an academic discipline. 36

CALICO Journal

Marina Dodigovic REFERENCES Borchardt, F. (1998). On the history and aesthetics of screen design (or why do most screens put learners to sleep?). In K. Cameron (Ed.), Multimedia CALL: Theory and practice (pp. 3-10). Exeter, UK: Elm Bank Publications. Brett, P. (1998). An intuitive, theoretical and empirical perspective on the effectiveness question for multimedia. In K. Cameron (Ed.), Multimedia CALL: Theory and practice. Exeter, UK: Elm Bank Publications. Bull, S. (1997). Promoting effective learning strategy use in CALL. Computer Assisted Language Learning, 10 (1), 3-39. Cameron, K. (1997). Editorial. Computer Assisted Language Learning, 10 (1), 1-2. Delbridge, A., Bernard, J. R. L., Blair, D., Peters, P., & Butler, S. (1996). Macquarie dictionary. Sydney: Macquarie University. Dodigovic, M. (1991). Efficacy of pragmalinguistic CALLware in language for special purpose (LSP) university courses. Applied Computer Translation, 1 (1), 29–36. Dodigovic, M. (1993). Interdisciplinary: Computer assisted linguistic research and the development of LSP courseware. CALICO Journal, 10 (4), 5-15. Dodigovic, M. (1995). Computergestuetztes Lernen und Lehren einer Fachsprache. Bremen: Hochschule Bremen/Universität Bremen. Dodigovic, M. (1998). Multimedia in natural language processing. In K. Cameron (Ed.), Multimedia CALL: Theory and practice (pp. 57-64). Exeter, UK: Elm Bank Publications. Gordon, S. (1994). Computer interface design guidelines. In F. Borchardt and E. Johnson (Eds.), Proceedings of the CALICO Annual Symposium: Human Factors (pp.106-113). Durham, NC: CALICO. Higgins, J. (1988). Language, learners and computers. London: Longman. Hwu, F. (1997). Providing an effective and affective learning environment for Spanish phonetics with a hypermedia application. CALICO Journal, 14 (2-4), 115-134. Levy, M. (1997). Theory-driven CALL and the development process. Computer Assisted Language Learning, 10 (1), 41-56. Lewins, F. (1990). Writing a thesis. Canberra: Australian National University. Liou, H. -C. (1997). Research of on-line help as learner strategies for multimedia CALL evaluation. CALICO Journal, 14 (2-4), 81-96 Manning, P. (1991). Methodological considerations for the design of CALL programs. In A. Hall and P. Baumgartner (Eds.), Language learning with computers (pp. 76-101). Klagenfurt: WISL. O’Brien, P. (1995). eL: Using AI in CALL. In Yazdani, M. (Ed.), Multilingual multimedia (pp. 85-139). Oxford: Intellect. Petkovic, N. (1984). Od formalizma ka semiotici. Beograd: BIGZ/Jedinstvo.

Volume 15 Number 4

37

Elements of Research in CALL Development Projects Pollard, D. & Yazdani, M. (1995). A multilingual multimedia restaurant scenario. In Yazdani, M. (Ed.), Multilingual multimedia (pp. 1-13). Oxford: Intellect. Valter, Z. (1988). Main features of research work in Croatia. In K. Potthast (Ed.), Proceedings of the 4th Scientific Colloquium: Science for Practice (pp. 65-80). Bremen: HSB.

AUTHOR’S BIODATA Marina Dodigovic coordinates English for Academic Purposes in the Department of Linguistics at Macquarie University. She has been actively involved in the theory and practice of CALL since the early 1980s. Apart from authoring several CALL packages, she carried out one of the first effectiveness studies in CALL and continues to contribute to international CALL publications. Her Ph.D. thesis was based on the research and development cluster involving computer assisted learning of language for specific purposes.

AUTHOR’S ADDRESS Department of Linguistics Macquarie University Department of Linguistics 2109 NSW Australia Phone: +61-2-9850-9936 Fax: +61-2-9850-8240 E-mail: [email protected]

38

CALICO Journal