Indeed, sometimes knowledge does not help: A replication study

Instructional Science 26: 391–407, 1998. c 1998 Kluwer Academic Publishers. Printed in the Netherlands.

391

Indeed, sometimes knowledge does not help: A replication study ROBIN STARK1, ALEXANDER RENKL2, HANS GRUBER1 & HEINZ MANDL1; 1

University of Munich, Germany; 2 School of Education, Schw¨abisch Gm¨und, Germany ( Correspondence address: University of Munich, Institute for Educational Psychology, Leopoldstraße 13, D-80802 Munich, Germany; Email: [email protected] 

Abstract. In an earlier study we found that intermediate experts in the domain of economics did not surpass novices in complex learning and knowledge application with a computer-based business simulation. In the present study, it was investigated whether these contra-intuitive findings can be replicated. In order to scrutinize the reasons which led to these findings, some parameters of the learning environment were changed. The duration of the exploration phase and of the problem-solving phase as well as the complexity of the situations were increased, motivation and acquired declarative knowledge were assessed. In view of mastering recurring demands and the functionality of mental models, no differences were found between a group of novices (15 students of humanities with a supplementary training in economics) and a group of intermediate experts (13 advanced students of economics). The findings of the original study were replicated, motivation had no effect on the result. In terms of declarative knowledge, the novices turned out to be even better. Key words: action competence, knowledge application, mastering recurring demands, functionality of mental models, declarative knowledge, computer-based learning environment, expertise

Difficulties in knowledge application have already been empirically shown in different domains (Renkl et al., 1996). In many types of instructional settings, this phenomenon manifests in similar shape. It is not declarative knowledge that is missing but knowledge that can be flexibly applied in contexts out of classroom. We define applicable knowledge as action competence and interpret it as the ability to successfully master the requirements of a domain. In our action competence model which is based on different research domains (theories of complex learning with computer-based learning environments; theories concerning the inert knowledge problem and its instructional remedies; theories of different knowledge types) we thereby differentiate between three aspects. Whoever wants to be successful in his or her domain should first of all be able to effectively master recurring demands which make up a large part of a regular working day. In early stages of learning processes, so-called close analogies can be used to master recurring demands economically. As a rule, experience-based episodic knowledge is

392 used here. However, using close analogies when solving tasks can impair transfer performance. As analogies are tied to specific situational conditions, difficulties in transferring them to novel situations can occur. Especially in complex and less structured knowledge domains, such routine expertise based on the use of close analogies is not a sufficient condition for successful action, at least in the long run. In order to overcome mere routine expertise and gain “adaptive expertise” (Hatano & Inagaki, 1986), mental models, which constitute the second aspect of our model of action competence, can be constructed. Knowledge represented in the form of mental models is less dependent on the situation. The contextual ties which can impair transfer are reduced. Running mental models in “mental simulations” (De Kleer & Brown, 1983) allows consequences of actions to be predicted. Effective actions can then be chosen in situations in which no previous experience was gathered. This envisioning function of mental models which enables dynamic representations not only requires knowledge about the structure and the relevant states and features of the situational components represented by the model, but also requires knowledge about causal relations of different component states. In order to be functional, the represented features have to be relevant with respect to structure and function but not only analogous in a superficial manner. The differentiation between superficial and structural features which can be regarded as constitutive for expertise in a domain (Chi et al., 1981) necessitates much domain-specific declarative knowledge, which is the third aspect in our model of action competence. In our understanding, mental models and declarative knowledge reciprocally influence each other. On the one hand, functional mental models enable the construction of dynamic representations having a comprehensive declarative knowledge base at one’s disposal. On the other hand, when mental models are constructed and used during learning and knowledge application, declarative knowledge can also be influenced. For example, knowledge about component features can be enriched, causal relations between component states can be modified or new relations can be created. Renkl et al. (1994) showed that students of business management, who were going to take their final exam, had great difficulties in using their extensive knowledge in managing a computer-simulated company. Henceforth, these students are called “intermediate experts” to characterize people who have already acquired well-founded theoretical domain-specific knowledge but lack “real work experience”. In achieving business profits which is an indicator for mastering recurring demands, intermediate experts did even worse than novices. In terms of the quality of the mental models of the system assessed after an exploration but prior to a problem-solving phase

393 there were no significant differences between the two groups. The business students were not any better than the novices at making use of the 30-minute exploration phase in order to generate sophisticated and coherent mental models of the economic system. After the problem-solving phase, the quality of mental models in both groups was better than before. However, the quality increase was higher in the group of intermediate experts. They did also better regarding the quality of reflections during the process of problem-solving. The knowledge aspects were not significantly correlated. To a certain amount, the findings could be explained through analysis of verbal protocols. As expected, the intermediate experts were able to reflect sophisticatedly on domain-specific aspects. But when constructing theories they often took into account more aspects than they could structure and integrate in order to find a functional decision (Mandl et al., 1995). Furthermore, the content aspects expressed by the business students proved to be veridical regarding economic theories but they were not relevant to the simulated problem situations. Additionally, it was obvious that the decisions reached by the intermediate experts were in principle suitable to achieve the defined goals. However, the deduced actions were often too cautious. The intermediate experts were not able to come to quantitative decisions suitable for achieving the intended effects. The novices’ reflections, on the contrary, were rather simple and one-dimensional. They did not take into account complex relations between variables and ignored potential side-effects of actions. However, the reflections and the corresponding resulting actions were functional, at least in the short term, because they led to relatively high business profits. Adelson (1984) also showed that novices with rather concrete and superficial knowledge representations outperformed experts with abstract representations in various tasks. Other authors reported about costs of expertise in the sense of temporary decrease of performance (Lesgold, 1984). The performance costs so arisen can exceed the advantages of possessing higher expertise, at least in the short run. This may happen when effective novice problem-solving strategies are given up and replaced by expert-like strategies provided that an expert-like knowledge base has not been previously constructed. The development of expertise seems not to be a monotonous process in the sense of additive accumulations of skills. Lesgold et al. (1988) postulate a curvilinear relation between level of expertise and performance. In spite of these plausible attempts to interpret unexpected results in expertise research, the findings of Renkl et al. (1994) remain surprising and thus need to be replicated. In particular, it has to be investigated whether they remain stable when some of the variables of the learning environment are varied and potential contributing factors are taken into account. To realize these demands, we increased the duration as well as the complexity of the

394 simulation. Moreover, we changed the operationalization of action competence in some respects. Finally, we assessed the subjects’ motivational states. (1) Increase of the duration of the simulation. The difference in achieving profits between the two groups found in the original study can possibly be explained by the short phase of problem-solving. There are some hints that the intermediate experts would have surpassed the novices if the problemsolving phase had been longer. Therefore we prolonged the problem-solving phase significantly in our study; we also allowed more time for the subjects to explore the simulation. (2) Increase of the simulation’s complexity. The finding that novices were successful in achieving business profits with rather simple concepts can also be explained by the fact that the prior setting of the simulation was rather simple (only two relevant input variables; only one market situation in the exploration as well as in the problem-solving phase). Therefore we increased the complexity of the simulation in the replication study by raising the number of input variables and by confronting the subjects with varying market situations. (3) Assessment of action competence (I): the functionality of mental models. In the original study, mental models were assessed by a teachingback procedure. This operationalization aimed at the estimation of the quality of mental models and mainly focused on theoretical, application-distant knowledge aspects. As the unexpected results of the original study are more associated with knowledge application, we focused more on near-to-action knowledge; therefore we assessed the functionality of mental models by employing standardized problem-solving and prediction tasks. (4) Assessment of action competence (II): acquired declarative knowledge. In order to support the validity of the more application-specific knowledge aspects assessed in the following study, we investigated the quality of the declarative knowledge acquired by working on the simulation. It is plausible to assume that intermediate experts outperform novices in acquiring and applying declarative knowledge when learning with the simulation. Especially in the declarative knowledge test, the intermediate experts should have advantages as they can refer to a more sophisticated domain-specific knowledge base. (5) Assessment of motivational states. A possible post-hoc explanation of the contra-intuitive findings of the original study is that the two groups were in different motivational states when learning with the simulation. The simulation possibly was less interesting and had less incentives for the business students. Negative effects concerning the employed learning strategies and the learning outcomes are possible (Hidi & Anderson, 1992). Therefore, motivation variables were recorded in the replication study, comprising the

395 acceptance of the learning environment and the intrinsic motivation to learn with the computer simulation employed in the study. Specifically, the replication study investigated the following questions: (1) Do intermediate experts in business and novice counterparts differ in mastering recurring demands? (2) Do the two groups differ in the functionality of mental models constructed during the work on the simulation? (3) Do the two groups differ in the declarative knowledge acquired during the work on the simulation? (4) Do the two groups differ in their motivational states during exploration of the simulation? Method Subjects Twenty-eight students with an average age of 26 years participated in the study. Fifteen students of humanities who had been given an introduction about basic business concepts by visiting the course “Student und Arbeitsmarkt” were defined as novices. The course “Student und Arbeitsmarkt” at the Ludwig-Maximilian University of Munich offers educational training for students of humanities to facilitate their entry into working life. Thirteen business students took part as intermediate experts. Despite their lack of experience that sets them apart from real domain experts, intermediate experts’ domain-specific knowledge base should enable them both to successfully control the simulation described below and to acquire applicable knowledge when working on the simulation. A short instructional text was provided to ensure that not only the intermediate experts but also the novices had the knowledge of basic economic concepts necessary for dealing with the simulation. The simulation “JEANS MANUFACTURING” as the learning environment As in the original study, the computer-based simulation “JEANS MANUFACTURING” was used as the learning environment (Preiß, 1994). JEANS MANUFACTURING has proved appropriate as the learning environment in practice in vocational schools for years. It is a near-to-reality learning environment programmed by domain experts for such a practical use. Nevertheless, it has been used in various investigations of learning subject-matters in business (e.g., F¨urstenau, 1994). The company and market models on which JEANS MANUFACTURING is based reflect the fundamental goods

396 and economic structures of performance processes found within a company. The dynamic nature of the simulation in combination with the provided input modus and the given feedback information makes JEANS MANUFACTURING an explorative learning environment well suited to investigate processes of knowledge acquisition and application. As in the original study, a duopolistic market was implemented. Subjects had to defend their company’s place in the market against the challenge posed by a competitor. We implemented three different market situations in contrast to the original study in which only one market situation was simulated. The first market situation was presented in the exploration phase. Production capacities and stock levels of the two competing companies were exactly the same, the demand on the jeans market was a bit recessive. The two market situations of the problem-solving phase were constructed in such a way that various market variables and functions appeared in the foreground. In each market situation, different actions are appropriate to successfully manage the simulation. The superficial feature of different market situations were identical. Varying the market situations should exclude the possibility of mastering the simulation by using only simple, one-dimensional concepts and by applying mechanical actions which have proved functional in one specific simulated situation. The subjects’ task consisted in managing their company as Managing Directors. During the exploration phase, subjects had to get acquainted with economic concepts and relations in order to be able to maximize the company profit in the following problem-solving phase. In contrast to the original study in which only two input variables (production quantity, quotation price) could directly be manipulated, the subjects in the replication study could directly influence four input variables (production quantity, quotation price, production capacity, advertising expenditures). In each simulation period, subjects had to make quantitative decisions determining the four input variables and to key in the corresponding numbers. In order to adjust their decisions they could use a calculation-tool provided by the learning environment. When the participants had definitely determined their decisions, the market simulation of the just planned period was run. Thereby, subjects were provided with graphics and tables with relevant information concerning the development of their company and of the (computer-simulated) competitor’s company (e.g., profits and losses, market shares, advertising effects, consumption behavior, stock level). Decisions for the next period had to be made based on these pieces of market information (and so on). While managing the simulation, subjects were systematically supported by a multi-staged problem-solving scheme. They were guided to explain their decisions, to predict action results and to draw final conclusions. Stark et al.

397 (1995) showed that the intensity of the learners’ exploration of the simulation can be increased when the learners operate according to the problem-solving scheme. Furthermore the construction of mental models can be fostered. The novices could possibly be put into a disadvantageous position by the increase of complexity realized in the present study. They may get overburdened (Sweller, 1994). However, such an effect would be of conservative nature as it would render a replication of the original study’s findings more difficult. An eventual replication of the results would be the more convincing. Experimental procedure Each subject was assessed individually in a session of approximately four hours. After reading the instructional text about basic economic concepts, a detailed introduction to the simulation was given. Then all subjects explored the same market situation within the simulation for 45 minutes. While exploring the simulation, they were supported by the problem-solving scheme. In the problem-solving phase, subjects had to cope with two different market situations. For each situation, they had to work for 40 minutes. The number of periods (simulation months played) dealt with in the market situations of the exploration and problem-solving phase was free, the time-on-task was controlled. The order of market situations presented was identical for all subjects. In the problem-solving phase, mastering recurring demands (the first aspect of action competence) was recorded. Subsequently, motivation was assessed (the acceptance of the learning environment, intrinsic motivation). Finally, the functionality of mental models and declarative knowledge (the second and third aspects of action competence) were assessed. Instruments As in the original study, control performance in the simulation was used as indicator for managing recurring demands. The duration of the problemsolving phase, which comprised two market situations instead of one market situation in the original study, was increased from 20 to 80 minutes. The differences between these two market situations did not change the structure of demands that the subjects had to fulfill in order to control the simulation. The average periodic profit in the problem-solving phase was defined as the measure for control performance. The reliability of control performance was computed by correlating the period gains made in odd periods with those in even periods. The reliability amounted to 0.78 after Spearman-Brown correction. The functionality of mental models was recorded by two indicators. The first indicator was control performance in novel problem situations within

398 the simulation. Two standardized simulation situations were presented on the computer screen. They were superficially similar but not equivalent to any of the situations encountered during the exploration and problem-solving phases. In the first situation, company profit had to be maximized and in the second situation stock level had to simultaneously be reduced. This dual goal increased the task complexity. The internal validity of the indicator was improved as well because the reduction of stock level is an important condition for successful management of JEANS MANUFACTURING. Company profit and stock level were aggregated to one factor. The second indicator for the functionality of mental models was prediction performance in novel problem situations. Quantitative predictions regarding the development of important market variables had to be made in standardized simulation situations. Prediction accuracy was computed using squared differences (according to Hasselhorn & Hager, 1989). The internal consistency of prediction performance, aggregated over different simulation situations and market variables, was 0.64 (Cronbach’s alpha). In order to allow predictions, relevant market and company data were presented in written and graphical form. So the development of the market could be followed through several periods prior to the situation for which the prediction had to be made. We assume that the complex prediction task cannot be mastered routinely. Instead, mental models have to be used which are constructed while working on the simulation. Declarative knowledge was assessed by an 11-item paper-and-pencil test aiming at knowledge about simulation-specific situations and action possibilities (example: “Expense by unit decreases in case the capacity increases and production output stays the same”). One of the possible answers “right”, “wrong”, or “do not know” had to be chosen. The economic concepts and relations assessed in this test were estimated as action-relevant by a domain expert; they could be learned by working on the simulation. The internal consistency of the declarative knowledge test amounted to 0.63 (Cronbach’s alpha). For assessing motivation, rating scales were developed for “acceptance of the learning environment” and for “intrinsic motivation”. The acceptance scale comprised five items (example: “If it were up to me, JEANS MANUFACTURING would be used in vocational schools.”). The reliability of this scale was 0.72 (Cronbach’s alpha). The intrinsic motivation scale comprised 11 items (example: “I was so fascinated by the simulation that I forgot everything around me.”). The reliability of this scale was 0.88 (Cronbach’s alpha). Table 1 summarizes the most important differences between the original study and the replication study.

399 Table 1. Differences between the original study and the replication study. Original study

Replication study

Simulation Number of input variables Number of market situations

2 1

4 3

Procedure Exploration Problem-solving

30 min. 1 20 min.

45 min. 2 40 min.

Operationalization Mastering recurring demands Mental models

Control performance Teaching back procedure

Acquired declarative knowledge Acceptance of learning environment Intrinsic motivation

– – –

Control performance Standardized tasks of problem solving and prediction Paper-pencil-test Questionnaire Questionnaire





Results Results concerning the three aspects of action competence are presented in the same order in which they were introduced in the theory section. Mastering recurring demands There was no significant difference between intermediate experts and novices in the number of simulation periods dealt with. This was true for the exploration phase as well as for the problem-solving phase. As mentioned, time-ontask was kept constant. Regarding mastering recurring demands, the results in both market situations of the problem-solving phase converged. Therefore, the data were aggregated. The business intermediate experts were not better at controlling the simulation than the novices. They did not achieve higher company profits (t(26) = ,1.33, n.s.); descriptively, they were even worse. The difference could not be supported statistically, even though the effect was of medium size (see Table 2). In view of mastering recurring demands (the first aspect of action competence), no significant difference between the two groups occurred. Functionality of mental models In the first standardized simulation situation, the intermediate experts were not able to make higher company profits. Neither could they surpass the novices in the dual-goal situation (situation 2) in which profit had to be

400 Table 2. Mean values (z-scores; standard deviations in parentheses) of both experimental groups with regard to the mastering of recurring demands, functionality of mental models and declarative knowledge. Intermediate experts Recurring demands Mental models (situation 1) Mental models (situation 2) Mental models (predictions) Declarative knowledge

–0.27 (0.85) 0.20 (0.72) 0.31 (1.16) –0.01 (0.75) –0.40 (1.25)

Novices 0.23 (1.09) –0.17 (1.19) –0.26 (0.78) 0.00 (1.20) 0.35 (0.55)

maximized and stock level needed be reduced simultaneously. Descriptively, the intermediate experts showed superior performance in both situations (see Table 2). However, the differences in both situations were not significant (situation 1: t(26) = 0.98, n.s.; situation 2: t(26) = 1.54, n.s.). In view of the first indicator of the functionality of mental models, no significant difference between the two groups occurred. Prediction performance, the second indicator for the functionality of mental models, yielded identical results in both groups (t(26) = ,0.03, n.s.). The business students were not able to give more accurate quantitative predictions of relevant market variables than the students of humanities. To sum up, in terms of the functionality of mental models (the second aspect of action competence), intermediate experts were not better than novices. Declarative knowledge In the declarative knowledge test, the novices fared even better than the intermediate experts (see Table 2: t(26) = ,2.12, p < 0.05); the effect size was substantial. The business novices with the additional training in “Student und Arbeitsmarkt” were able to acquire more simulation-relevant declarative knowledge or had more success in adapting already existing knowledge to the specific conditions of the learning environment when exploring and controlling the simulation. The different aspects of action competence were not significantly associated (see Table 3). Acceptance of the learning environment and intrinsic motivation The scale “acceptance of JEANS MANUFACTURING as the learning environment” yielded no significant difference between the two groups (see Table 4: t(26) = ,0.28, n.s.). Concerning intrinsic motivation, the intermediate experts descriptively had even higher scores than novices. However,

401 Table 3. Correlation of aspects of action competence, acceptance of learning environment and intrinsic motivation. (2)

(3)

(4)

(5)

(6)

(7)

(1) Mastering recurring demands 0.22 –0.03 –0.08 0.06 0.14 –0.03 (2) Mental models (situation 1) –0.14 –0.30 –0.23 0.12 0.31 (3) Mental models (situation 2) 0.30 –0.16 0.14 –0.09 (4) Mental models (predictions) 0.05 –0.26 –0.13 (5) Declarative knowledge 0.19 0.31 (6) Acceptance of learning environment 0.50  (7) Intrinsic motivation Note:  p

< 0.01.

Table 4. Mean values (z-scores; standard deviations in parentheses) of both experimental groups regarding the acceptance of the learning environment and intrinsic motivation. Intermediate experts Acceptance of learning environment Intrinsic motivation

–0.06 (1.25) 0.32 (0.92)

Novices 0.05 (0.76) –0.28 (1.02)

the difference between the groups could not be statistically supported (t(26) = 1.63, p < 0.12). The intermediate experts and novices did not differ in their motivational states when learning with JEANS MANUFACTURING, neither in their acceptance of the learning environment nor in their intrinsic motivation to learn with the simulation. The motivation variables significantly correlated with each other, but were not associated with the three aspects of action competence (see Table 3).

Discussion In the original study, the learning outcomes of the business intermediate experts were unexpectedly low in comparison to the novices. These contraintuitive findings could be replicated in the present study. Intermediate experts did not show more action competence than novices. However, as the size of the two experimental groups was not very large, the results of the replication study have to be interpreted with caution. Mastering recurring demands The groups did not differ in mastering recurring demands. In the original study, the novices had even outperformed the intermediate experts in control-

402 ling the simulation. The difference became smaller at the end of the problemsolving phase. However, the post-hoc hypothesis that the intermediate experts might have surpassed the novices if the problem-solving phase had comprised more periods cannot be confirmed by the new findings. Even with the significantly extended duration of the simulation, the business students were not better than the students of humanities. This was true despite the simultaneous enhancement of complexity. More information had to be processed, and successful management of the simulation required more flexibility. It was not confirmed that the increased complexity would create a decrease of performance mainly in the group of the novices. The novices in the replication study obviously have not been thwarted to a larger extent than the intermediate experts by the increase of complexity. Like the business students in the original study, intermediate experts were not able to profit from their domain-specific knowledge acquired during university education. This addresses severe problems of knowledge application. Functionality of mental models The intermediate experts were not able to construct mental models of higher functionality than the novices were. In standardized problem situations, the business students surpassed the students of humanities neither in control performance nor in prediction performance. This parallels the findings concerning the mastering of recurring demands. Such convergent results are by no means trivial, taking into account the methodological problem of standardizing quality indicators. This problem regularly occurs when simulations are used as an instrument for learning or research (M¨uller, 1994). In order to produce effective decisions and precise predictions, the subjects had to select and integrate much information concerning the development of the market. It is plausible to assume that the subjects ran the simulations mentally, probably using mental models constructed while working on JEANS MANUFACTURING. By no means the prediction tasks could be mastered automatically or through simply maintaining input patterns that proved successful during the problem-solving phase. The intermediate experts should have been able to profit from their domain-specific knowledge base. However, this was not the case. Our results consistently show once more that the presence of abstract domain-specific knowledge by no means guarantees successful knowledge application and transfer. In the original study, the intermediate experts were not able to construct mental models of higher quality than the novices were. They were only superior in improving the quality of their mental models by the experiences gained during the exploration phase. The results of both studies cannot be directly compared, because in the original study, the quality (in the sense of veridicality) of mental models was

403 estimated, but in the replication study, it was the functionality of mental models that was evaluated. On an abstract level, however, the results converge: in both studies, business students showed severe deficits in knowledge application and transfer. Declarative knowledge Least to expect was the result in the third aspect of action competence, simulation-relevant declarative knowledge. Declarative knowledge had not been recorded in the original study. In the declarative knowledge test, the novices were more successful than the intermediate experts. The knowledge test was based on simulation-specific situations and action possibilities. Concrete simulation-relevant concepts were evaluated instead of general, abstract economic concepts. Probably the intermediate experts would be more successful than the novices if a test on abstract economic knowledge had been administered. Nevertheless, the third aspect of action competence – declarative knowledge – was conceptualized as less application-specific than the two other aspects. The business students should have surpassed the students of humanities especially in the declarative knowledge test, because they should profit more directly from the knowledge acquired at university. Even if the business students in the replication study did only have a restricted degree of expertise (NB: “real” experts instead of the intermediate experts should have outperformed the novices), our finding appears to be rather surprising. Taking into account the qualitative findings of the original study, the failure of the intermediate experts is less surprising. The intermediate experts in the replication study, too, were probably impeded by the complexity of their concepts. Such impediment would show up when subjects try to assimilate their concepts with the specific conditions of the learning environment or when they try to extract specific concepts from the simulation. Assimilation of existing knowledge with the learning situation might be easier for less complex concepts like those created by the novices. Sternberg and Frensch (1992) pointed out that modifications of prototype materials, tasks, and rules can have more adverse affects on the performance of experts than on novices’ performance. Such modifications probably have been arisen as the learning situation given in university education distinctly differs from complex learning with a computer-based simulation. In this context, the results are relevant as they underline the difficulty of effectively mastering new information when it interferes with the already existing knowledge. Funke (1992) found that the acquisition of declarative knowledge can be significantly impeded when the existing knowledge on the one hand and the concepts underlying a simulation on the other hand interfere with each other. The intermediate experts in the original study had

404 veridical concepts which were not relevant to solve simulation-specific problems. Negative interferences have to be expected if concepts acquired in some context are brought into another one (e.g., a problem-solving situation), when the specific situational conditions have not been previously considered. Such interferences can be considered as negative transfer effects. In this sense, deficits in knowledge application not only appeared in the near-to-action tasks employed to assess the first two aspects of action competence (mastering recurring demands, the functionality of mental models), but deficits in terms of declarative knowledge could also be detected. The failure of the intermediate experts concerning declarative knowledge test reveals insufficient context sensitivity, too. As in the original study, the different aspects of action competence were not significantly associated. This result also replicates the findings of a former experimental study of our research group (Stark et al., 1995). It supports the necessity to simultaneously consider different aspects of action competence. To speak of action competence in the sense of a one-dimensional construct would be too simplified. Acceptance of the learning environment and intrinsic motivation to learn with the simulation The learning environment was accepted in a similar manner by the intermediate experts and novices. Neither did the two groups differ with respect to intrinsic motivation to learn with the simulation. Both motivation variables were associated significantly. However, they were statistically independent from the three aspects of action competence. Stark et al. (1996) found similar results among students of vocational school. Failures of the intermediate experts cannot be attributed to differences in motivational states. As the samples investigated in the original study and in the replication study are comparable, the motivation argument as a post-hoc explanation of the unexpected results is losing its ground. The variations which were realized in the replication study as compared to the original study (increase of the duration and the complexity of the simulation, consideration of declarative knowledge as an additional aspect of action competence) were conservative ones. They should rather hamper novices than the intermediate experts and thereby reduce the probability to replicate the results of the original study. However, neither in mastering recurring demands, nor in constructing functional mental models, nor in acquiring and applying declarative knowledge, the business students were able to outperform the students of humanities with an additional training in the course “Student und Arbeitsmarkt”.

405 The lack of differences between the two groups cannot be attributed to floor effects or ceiling effects. The mean performance in both groups was clearly superior as compared to the performance of some subjects in the pilot study who attempted the problem-solving phase of the simulation without first exploring the simulation. Besides, in both groups the accuracy of predictions substantially increased while working on the simulation. As the tasks employed were rather difficult, no ceiling effects occurred. Both the novices and the intermediate experts performed far below an optimal level. Thus, the results of the original study could be clearly replicated. They uncover severe problems university-trained learners have in knowledge application and transfer. We do not question that in university courses business students learn to talk competently about and reflect upon the domain. However, they apparently have problems in knowledge application and transfer. The question of the validity concerning the learning environment or the employed knowledge tasks may arise. It is impossible to reach the complete complexity of “real” problem situations even by employing well-constructed simulations like JEANS MANUFACTURING. However, this inevitable restriction in authenticity does not endanger the validity of our study. Learners who fail to master the simplified model of reality represented by the simulation will probably also have tremendous difficulties when confronted with the complexity of “real” problem situations in working life. We assume that learning and testing situations supplied by this kind of learning environment are at least far more authentic than those provided in traditional school contexts or university contexts. The problems of knowledge application and transfer demonstrated by the university-trained learners point squarely to the issues of university education. Acquisition of applicable knowledge seems to be neglected in the university education program. Development of action competence should be an essential component of university education which cannot exclusively rely on later training-on-the-job. With respect to the three aspects of adaptive expertise that we discussed in the introduction, the present results suggest that our graduate students are far from having “real” expertise. They do not effectively employ experienced-based episodic knowledge for mastering recurrent demands and they are not able to construct functional mental model. More surprisingly, the graduate business students who definitely had more declarative domain knowledge as compared to the humanities students had substantial difficulties in adapting their declarative knowledge for the problem at hand. In contrast, real experts should have substantial competencies with respect to all three aspects. Given these deficits, university education should no longer primarily focus on the provision of abstract declarative knowledge. In order to foster real

406 expertise, it is necessary to offer learning opportunities which require the students to use their declarative knowledge, to gain episodic knowledge on successful problem solutions and unsuccessful attempts, and to construct mental models on business problem situations. This can be accomplished by supplementing traditional forms of learning by problem-based learning units. Problem-based learning can be implemented, for example, by using case studies, units of practical training in companies, and by employing simulations such as the JEANS MANUFACTURING. The latter types of learning environments have a “bridging function” between typical abstract university instruction and learning in real working contexts. In order to investigate the educational potential of simulations for fostering expertise, our research group has conducted experimental studies. It has been shown that different aspects of action competence could be fostered by instructional measures (e.g., multiple learning contexts and guided problemsolving) inspired by situated cognition approaches (Stark et al., 1995). Presently, we are working to further improve and develop learning environments that support the development of expertise and action competence. We hope that our research will make a significant contribution for the improvement of current forms of instruction.

References Adelson, B. (1984). When novices surpass experts: The difficulty of a task may increase with expertise. Journal of Experimental Psychology: Learning, Memory, and Cognition 10: 483–495. Chi, M.T.H., Feltovich, P.J. & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science 5: 121–152. DeKleer, J. & Brown, J.S. (1983). Assumptions and ambiguities in mechanistic mental models. In D. Gentner & A.L. Stevens, eds, Mental Models (pp. 155–190). Hillsdale, NJ: Erlbaum. Funke, J. (1992). Solving complex problems: Exploration and control of complex systems. In R.J. Sternberg & P.A. Frensch, eds, Complex Problem-Solving: Principles and Mechanisms (pp. 185–222). Hillsdale, NJ: Erlbaum. F¨urstenau, B. (1994). Komplexes Problemlo¨ sen im betriebswirtschaftlichen Unterricht [Complex Problem-Solving in Business Management Instruction]. Wiesbaden: Gabler. Hasselhorn, M. & Hager, W. (1989). Prediction accuracy and memory performance: Correlational and experimental tests of a metamemory hypothesis. Psychological Research 51: 147–152. Hatano, G. & Inagaki, K. (1986). Two courses of expertise. In H. Stevenson, H. Azuma & K. Hakuta, eds, Child Development and Education in Japan (pp. 262–272). New York: Freeman. Hidi, S. & Anderson, V. (1992). Situational interest and its impact on reading and expository writing. In K.A. Renninger, S. Hidi & A. Krapp, eds, The Role of Interest in Learning and Development (pp. 215–238). Hillsdale, NJ: Erlbaum. Lesgold, A.M. (1984). Acquiring expertise. In J.R. Anderson & M. Kosslyn, eds, Tutorials in Learning and Memory (pp. 31–60). San Francisco: Freeman.

407 Lesgold, A., Rubinson, H., Feltovich, P.J., Glaser, R., Klopfer, D. & Wang, Y. (1988). Expertise in a complex skill: Diagnosing X-ray pictures. In M.T.H. Chi, R. Glaser & M.J. Farr, eds, The Nature of Expertise (pp. 311–342). Hillsdale, NJ: Erlbaum. Mandl, H., Gruber, H. & Renkl, A. (1995). Mental models of complex systems: When veridicality decreases functionality. In C. Zucchermaglio, S. Bagnara & S.U. Stucky, eds, Organizational Learning and Technological Change (pp. 102–111). Berlin: Springer. M¨uller, A. (1994). Die M¨aßigkeit der Maße. Zur Schwierigkeit der Beurteilung von Steuerungsleistung beim L¨osen komplexer Probleme [The Mediocrity of Measures. The Difficulty of Measuring Control Performance When Solving Complex Problems]. Unpublished Diploma Thesis, Westf¨alische Wilhelms-Universit¨at, M¨unster. Preiß, P. (1994). Planspiel JEANSFABRIK. Betriebliche Leistungsprozesse [Computer Simulation Game JEANS MANUFACTURING]. Wiesbaden: Gabler. Renkl, A., Gruber, H., Mandl, H. & Hinkofer, L. (1994). Hilft Wissen bei der Identifikation und Steuerung eines komplexen o¨ konomischen Systems? [Does knowledge help with the identification and control of a complex economics system?] Unterrichtswissenschaft 22: 195–202. Renkl, A., Mandl, H. & Gruber, H. (1996). Inert knowledge: Analyses and remedies. Educational Psychologist 31: 115–121. Stark, R., Graf, M., Renkl, A., Gruber, H. & Mandl, H. (1995). F¨orderung von Handlungskompetenz durch geleitetes Probleml¨osen und multiple Lernkontexte [Fostering action competence through guided problem-solving and multiple learning contexts]. Zeitschrift f¨ur Entwicklungspsychologie und P¨adagogische Psychologie 27: 289–312. Stark, R., Gruber, H., Graf, M., Renkl, A. & Mandl, H. (1996). Komplexes Lernen in der kaufm¨annischen Erstausbildung: Kognitive und motivationale Aspekte [Complex learning in basic business training: Cognitive and motivational aspects]. Zeitschrift f¨ur Berufs- und Wirtschaftsp¨adagogik, Beiheft 13: 23–36. Sternberg, R.J. & Frensch, P.A. (1992). On being an expert: A cost-benefit analysis. In R.R. Hoffman, ed., The Psychology of Expertise. Cognitive Research and Empirical AI (pp. 191– 203). New York: Springer. Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction 4: 295–312.