Designing Capstone Courses in Management Education: Knowledge ...

DESIGNING CAPSTONE COURSES IN MANAGEMENT EDUCATION: KNOWLEDGE ACTIVATION AND INTEGRATION USING AN ERP-BASED SIMULATION GAME Completed Research Paper

Christine Legner Faculty of Business and Economics (HEC), University of Lausanne CH-1015 Lausanne, Switzerland [email protected]

Thibault Estier Faculty of Business and Economics (HEC), University of Lausanne CH-1015 Lausanne, Switzerland [email protected]

Hazbi Avdiji Faculty of Business and Economics (HEC), University of Lausanne CH-1015 Lausanne, Switzerland [email protected]

Thomas Boillat Faculty of Business and Economics (HEC), University of Lausanne CH-1015 Lausanne, Switzerland [email protected]

Abstract Serious games are becoming increasingly popular as learning tools in management education. Owing to their engaging nature, they enhance students’ intrinsic motivation for learning and provide challenging environments for problem-based and experiential learning. However, they also require changes to instructional and pedagogical strategies in order to leverage the full potential of problem-based learning. This paper suggests using simulation games based on enterprise resource planning systems (ERP) to activate and integrate knowledge from different management disciplines. Our research contributions are threefold: First, we suggest a set of learning goals and a curriculum design for a capstone course that leverages ERP-based simulation games. Second, we present empirical insights based on a systematic evaluation of the learning outcomes with two cohorts of 32 students. Third, we derive a set of recommendations related to the role of ERP-based simulation games and course design in general management curricula. Keywords: business simulation, capstone course, enterprise resource planning systems (ERP), problem-based learning, serious games

Thirty Fourth International Conference on Information Systems, Milan 2013

1

Curriculum and Education in IS

Introduction Serious games are becoming increasingly popular as learning tools in management education. Owing to their engaging nature, they enhance students’ intrinsic motivation and provide challenging environments for problem-based learning. An increasing number of publications report positive learning experiences through applying simulation games, including the beer distribution game (Coakley et al. 1998; Haines et al. 2011; Riemer 2008) for supply chain management and ERPsim (Cronan and Douglas 2012; Cronan et al. 2012; Seethamraju 2011) for information systems. Existing research has mostly focused on describing the educational process and the impact of simulation games on affective or behavioral learning (Anderson and Lawton 2008). To date, cognitive learning outcomes have hardly been assessed, and it has been difficult to prove the effectiveness of game-based learning compared to alternative learning concepts (Randel et al. 1992). However, the changes to instructional and pedagogical strategies that are required to leverage the full potential of problem-based learning and simulation approaches (Léger et al. 2011) have received very little attention. We therefore still lack research on systematic course curriculum design that takes full advantage of a game-based learning approach. This article addresses this research gap and investigates simulation games in management education. Specifically, it suggests using simulation games based on enterprise resource planning systems (ERP) to activate and integrate management-related knowledge in the context of a capstone course. Such courses prepare students for their professional careers and typically take place in the last semester of their studies (Durel 1993). We address the following two research questions: RQ1. How does one design a course curriculum that leverages ERP-based simulation games in capstone courses in order to activate and integrate knowledge from different management disciplines? (curriculum design) RQ2. How effective is the aforementioned curriculum in terms of cognitive learning outcomes? (curriculum evaluation) Our research context is the Faculty of Business and Economics (HEC) at the University of Lausanne. The HEC is a renowned business school that has been accredited by EQUIS (EFMD Quality Improvement System) and AMBA (Association of MBAs). The study was conducted in 2012–3, when the HEC introduced a capstone course for undergraduate students in its Bachelor in Management program. Here, we present the pedagogical strategies applied in the course curriculum design as well as the empirical results from evaluating the learning achievements. Our research contributions are threefold: First, we suggest a set of learning goals and a curriculum design for a capstone course that leverages ERP-based simulation games for activating and integrating knowledge from different management disciplines. Second, we present empirical insights based on a systematic evaluation of the learning outcomes with two cohorts of 32 students. Third, we derive a set of recommendations and propositions for the role of ERPbased simulation games in general management curricula. The remainder of this article is structured as follows: In the next section, we will review prior literature related to serious games and analyze the role and the learning outcomes of business simulation games in management education. We will then present our research setting and approach. The subsequent sections will introduce the learning objectives and principles applied in the curriculum design for a capstone course. We will systematically evaluate the learning achievements in relation to the suggested learning objectives. This paper concludes by interpreting and discussing our findings in the context of existing work on serious games.

Prior Literature Business Simulation Games in Management Education Serious games are associated with “games for purposes other than entertainment,” encompassing all aspects of education – teaching, training, and informing (Michael and Chen 2005). Game-based learning is of particular interest for management education because of the interactive nature of digital games and

2 Thirty Fourth International Conference on Information Systems, Milan 2013

Legner et al. / Knowledge Activation and Integration Using ERP-based Business Simulations

their possibility to simulate complex business environments (Faria 2001). Among the management disciplines that apply simulation games most often for educational purposes are supply chain management, project management, and information systems (see Table 1). In supply chain management, the beer distribution game (see Forrester 1958) exposes students to the importance of coordination in supply chain management and the so-called “bull-whip effect.” Project management curricula (Bonazzi et al. 2012; Pfahl et al. 2004) make use of the complexity that can be created by simulation games to assess and stress different methods of managing projects. Serious games in the form of ERP-based simulations such as ERPsim (Léger 2006) also prove useful in the area of information systems, mainly to teach students the dimensions of enterprise systems and business process integration through a learning-bydoing approach (Cronan and Douglas 2012; Cronan et al. 2012; Foster and Hopkins 2011; Seethamraju 2011). With the purpose of enriching the learning of strategic decision-making and critical thinking, general management curricula also integrate serious games, such as the ISM simulation (McKone and Bozewicz 2003) or the computer-based International Operations Simulation Mark/2000 (Thorelli and Graves 1964). The latter incorporates a rich simulator that assigns a company that produces PCs to the participants, generates markets with different preferences and behaviors, and simulates the passing of time.

Table 1: Overview of Management Curricula Based on Simulation Games Management discipline

Learning objectives

Simulation game

Related studies

Supply chain management

Hands-on experience of supply chain management problems, most importantly the “bull-whip” effect

Beer distribution game developed at MIT (Forrester 1958)

(Coakley et al. 1998)

Grasp the complexity of projects by assessing different project management methods

Albasim (www.albasim.com)

(Bonazzi et al. 2012)

Support learning related to software project management

System Dynamics simulation model

(Pfahl et al. 2004)

Project management

Information systems

Learning-by-doing approach for teaching enterprise systems and business process integration

(Riemer 2008) (Haines et al. 2011)

(Foster and Hopkins 2011) ERPsim developed by HEC Montréal (Léger 2006)

(Seethamraju 2011) (Cronan and Douglas 2012) (Cronan et al. 2012)

General management

Strategic decision-making and general management concepts (financial forecasting, marketing, and operations management)

ISM simulation

(McKone and Bozewicz 2003)

International Operations Simulation Mark/2000 (Thorelli and Graves 1964)

(Ben-Zvi and Carton 2008)

Business Simulation Games and their Educational Outcomes Instructors use computer-based business simulation games to achieve a wide set of objectives, which can be classified into three main learning categories (Anderson and Lawton 2009): • Learning objectives: teaching students the terminology, concepts, and principles of general management or a specific discipline. A business simulation would help students to understand interrelationships of various business functions and to experience the difficulty of executing relatively simple business concepts in a complex environment.


3


•

Attitudinal objectives: changing students’ attitudes towards a discipline by involving and engaging students in the learning process.

•

Behavioral objectives: teaching students to apply and implement concepts and to improve their decision-making skills.

These three categories can also be associated with Bloom’s taxonomy of learning objectives (Bloom et al. 1956), which contains the three domains of cognitive (knowing), affective (feeling), and psychomotor (doing) learning. Unsurprisingly, most existing studies build on Bloom’s taxonomy for assessing learning outcomes in business simulations. Simulation games are associated with the pedagogical approaches of problem-based and experiential learning (Kiili 2005). Compared to the traditional subject-based learning model, they typically reverse the order of presenting course concepts (Anderson and Lawton 2008). Instead of presenting the primary concepts before applying them to a particular problem, instructors let students “discover” the course concepts during the simulation. As a result, many studies have been performed to understand the consequences of game-based learning on students’ attitudes, thereby emphasizing the affective learning outcomes. These studies have identified a positive impact and reveal that students find the game-based pedagogical approach generally more enjoyable and stimulating. Concerning behavioral learning effects and the psychomotor domain, researchers have been interested in understanding whether students’ performance in a simulation game is able to predict performance in real-world business environments. However, they only found weak evidence of correlations between simulation performance and career success regarding salaries and promotions (Wolfe and Roberts 1986; Wolfe and Roberts 1993). Despite the fact that simulation games have been used in management education for more than 30 years, there is continuous debate about their effectiveness in terms of cognitive learning outcomes (Anderson and Lawton 2008). Given the substantial time commitment required by most simulation games, they are considered a relatively inefficient means to teach the basics of a course, such as terminology, factual knowledge, or basic concepts. Whereas business simulations are not effective for addressing lower levels of learning in Bloom’s taxonomy of learning goals – knowledge and comprehension – they are found to support the higher-order intellectual activities – application, synthesis, and analysis (Bloom et al. 1956).

ERPsim ERPsim is a simulation game developed at HEC Montréal and used by more than 180 universities worldwide for academic purposes. It is unique in that the simulator runs on SAP ERP, a commercial enterprise system, to simulate a market of manufactured goods such as bottled water, dairy products, cereals, or muesli. ERP systems represent the core of enterprise information systems and have been a standard for large companies since 1990s (Parr and Shanks 2000). As standard application software, ERP systems allow enterprises to support their primary processes and functions through a central database (Davenport 1998; Shank and Seddon 2002). Relying on an ERP system as a simulator allows for the creation of business simulations that are very close to complex real-world environments. It also helps students experience the integration of business functions and processes in a firm. Léger (2006) identifies three main learning objectives for ERPsim: “(i) to develop a hands-on understanding of the concepts underlying enterprise systems, (ii) to experience the benefits of enterprise integration first hand, and (iii) to develop technical skills at using ERP software.” In an ERPsim course, teams of three to four students operate their own company and compete against one another in a given simulated market. Each team must manage the whole business cycle – from planning, to procuring raw materials, scheduling production, to marketing and selling finished goods – with the goal of maximizing profits. To perform these tasks, students use SAP transactions. Students rely on standard reports and business intelligence applications to analyze transactional data from the ERP system and prepare their business decisions. During the game, three tasks are performed by the simulator. First, the software simulates a market divided into different areas and different distribution channels, ranging from mass retailers to small independent grocers. Each area and each distribution channel have predefined preferences in terms of goods and the layout of these goods. The latter are not communicated to the students, who must – among others – analyze the market by testing the goods they deliver to customers. Second, the game includes a

4



temporal dimension by simulating game days in terms of actual seconds. Thus, the purchase of raw materials and the production of goods, among others, are completed within certain intervals, which adds a temporal dimension to the game. Last, the software automates administrative tasks so that the participants can focus on the key business decisions of a make-to-sell cycle. Several studies report learning experiences with ERPsim: From their evaluation of a one-day interinstitutional competition between universities with 24 participants, Foster and Hopkins (2011) confirm that ERPsim increases ERP-related skills and learning about a real-world supply chain. Cronan and Douglas (2012) analyze 280 students over six semesters and find that the ERP simulation game was instrumental in their learning of ERP concepts and business processes. In addition, participation in a simulation game significantly increased students’ acceptance of IT. Based on their experiences with ERPsim, Cronan et al. (2012) investigate the effectiveness of cognitive learning in ERP simulation games. They also find that self-assessed perceptions of learning results did not differ from objective measures. (Seethamraju 2011) integrated ERPsim into a business process integration course with 52 students, and found that students significantly improved their abilities, particularly concerning process concepts and SAP skills, but relatively less for process management and analysis as well as process significance and awareness factors.

Research Gap Existing research shows that simulation games provide positive learning experience in many academic disciplines. They raise the awareness of students regarding specific aspects such as the importance of coordination in supply chain management (Forrester 1958) or the assessment of different project management methods (Bonazzi et al. 2012; Pfahl et al. 2004). These related studies primarily focus on the opportunities provided by simulation games, as well as their benefits compared to traditional learning strategies. However, some studies reveal that simulation game adoption must involve changes to instructional and pedagogical strategies in order to leverage the full potential of problem-based learning and simulation approaches (Moreno-Ger et al. 2008; Léger et al. 2011). Though some studies aim to describe the educational process and analyze the impact of simulation games on affective or behavioral learning (Anderson and Lawton 2008), there is a lack of research on systematic course curriculum design that takes full advantage of a game-based learning approach.

Research Context The Faculty of Business and Economics (HEC) of the University of Lausanne is a renowned business school with 2,500 students enrolled in its undergraduate and graduate programs. In 2011, it introduced a capstone course for Management Bachelor program students, to prepare them for their professional career. The course comprises 6 credits according to the European Credit Transfer System (ECTS), which defines that one credit generally corresponds to 25 to 30 hours of student workload. The capstone course takes place in the last semester of undergraduate programs. The core team responsible for designing and establishing the capstone course comprises two professors, and one doctoral student. In 2013, a scientific collaborator joined the team to support further course development and to prepare rollout to all Bachelor students as a mandatory course. To date, the course has been run twice: in spring term 2012 (with 48 students) and then in spring term 2013 (with 64 students). In the second iteration, the course was set up with two parallel groups of 32 students to integrate more students into the course. The course is offered as an optional course outside the traditional semester. This paper reports on the experiences with designing this capstone course, that is, its learning objectives and the detailed curriculum design, and will evaluate its learning effectiveness.


5


Learning Objectives and Curriculum Design Learning Objectives Capstone courses fulfill a specific role in a management curriculum: they are a method of summative evaluation and of measuring curricular outcomes attainment while preparing students for their professional careers (Dunlap 2005; Gilinsky and Robison 2008). The capstone course expectations are a display of the ability to integrate knowledge from different management disciplines and to apply it to solve practical problems in complex, close-to-reality situations (Davis and Comeau 2004). While other courses emphasize the recall and comprehension of discipline-specific knowledge, the capstone course focuses on their integration and application. In our context, the capstone course had to be designed so as to complement the existing curriculum of a Management Bachelor program. HEC Lausanne’s curriculum comprises mostly discipline-specific courses such as marketing, operations, finance, or accounting. It thus faces some difficulties in preparing students for the cross-functional and multidisciplinary challenges of their professional careers. These challenges have recently been emphasized by the school’s business advisory board. The role of the capstone course is therefore to draw together the learnings and discipline-specific skills from different courses in the existing curriculum and to focus on a management curriculum’s multidisciplinary learning achievements. It should allow students to demonstrate their accomplishment of the full spectrum of learning and to provide a functionally integrative curriculum experience. Besides the affective and psychomotor objectives pursued with the capstone course, HEC Lausanne pursued the goal of enhancing cognitive processes through students’ application of general management and discipline-specific theories and concepts. The learning scenario is built around a typical management cycle, as exemplified by the Management System approach by Kaplan and Norton (1996), who define a management system as “the integrated set of processes and tools that a company uses to develop its strategy, translate it into operational action, and monitor and improve the effectiveness of both.” HEC Lausanne deduced four specific objectives: •

define a company strategy

•

operationalize the defined strategy in the different business functions and processes, most importantly marketing and sales, production, procurement, finance, and controlling

•

defend the company strategy in a dynamic market, in competition with other teams

•

set goals and analyze performance based on a balanced scorecard.

Learning objective 1 relates to students’ ability to identify suitable strategic positions, for instance, building on Porter’s generic strategies (cost leadership strategy, differentiation strategy, and segmentation strategy). The strategic positioning translates into team decisions about the design of their products, their target markets, or the investments they will make. These choices are of utmost importance for companies to differentiate from each other, since they play in the same market (2 markets; 8 teams). Learning objective 2 requires students to demonstrate their ability to apply concepts from different management disciplines in order to implement their strategy in the different business functions and processes. For instance, students will create their sales forecast, decide on inventory levels, or fix their cash reserve. Learning objective 2 also emphasizes the coherence between the overarching company strategy and the related functional strategies. Learning objective 3 relates to students’ capacity to defend the defined strategy in a dynamic environment and assesses their responsiveness to unforeseen situations. Learning objective 4 relates to the students’ analytical capacities and their ability to assess business performance.

Curriculum Design The Principles of Curriculum Design In terms of curriculum design, a capstone course is characterized by a student-centered approach and seldom resembles traditional classroom courses. For this type of course, a problem-based approach is

6



recommended, since it should center on learning, not on teaching (Dunlap 2005). Hence, a capstone course curriculum design is driven by problem analysis, information sharing, and creative solutions and projects. In designing our curriculum, we followed the general recommendations for capstone courses (Dunlap 2005; Gilinsky and Robison 2008) and guidelines for designing and building effective business simulation models identified by Faria (2001) as well as Stainton et al. (2010). Specifically, we identified several principles to guide our curriculum design: •

Provide a realistic representation of the real-world business environment by means of an ERP-based simulation game environment: Prior studies found that ERP-based simulation games create an understanding of how organizations must operate in a crossfunctional and process-oriented manner (Deng and Chandler 2010). We argue that ERP-based simulation games provide a promising environment for simulating the complexities of strategy implementation and business operations, since they provide complete transactions for strategic, tactical, and operational decision-making and reporting in all main business functions.

•

Problem-based learning to improve managerial thinking (Smith 2005) and create learning outcomes in the higher levels of the cognitive domain, that is, application, synthesis, and analysis: By participating in a simulation game, students are immediately confronted with practical problems and are then naturally forced to solve them.

•

Foster activation and integration of knowledge from prior courses instead of acquisition of new knowledge: In contrast to traditional “ex cathedra” training, students will not be exposed to acquiring new concepts or knowledge. Instead, short modules (30 to 60 minutes) will be intermingled with game phases and debriefings. By addressing difficulties encountered by students during the game, these modules trigger activation of knowledge previously studied and acquired during prior courses.

•

Enable learning through reflection on theory: Since simulation games are perceived as stressful by participants, it is recommended to allow sufficient time for results analysis, discussion, and reflection (Faria 2001; Stainton et al. 2010). Whereas many other curricula focus on gaming, we decided to combine game phases with self-study elements and reflection phases.

ERP-based Simulation Game Environment As noted, an ERP-based simulation game environment can create close-to-reality situations and simulate complex business environments. This particularly applies to ERPsim, which provides a “rich and realistic demand model” (with a large population of customers attached to regional distribution channels) and leaves strategic and tactical decisions in the primary business functions to students, while automating some of the more administrative tasks to make the game more fluid (Cronan et al. 2012). Our capstone course relies on the predefined scenarios for ERPsim, but emphasizes their integration into the broader strategic context: In teams of three to four, students must run a muesli manufacturing company and compete against 8 to 10 other teams on a single market. During the game, students must make a number of key business decisions by using SAP transactions: 1.

Product portfolio: definition of the type of muesli (wheat, nuts, blueberries, oats, raisins, strawberries), packaging size (1kg, 0,5kg), and recipe.

2. Target customers and markets: three distribution channels (wholesale, grocery retail, or convenience retail markets) and three geographical regions (North, East, and West). 3. Pricing: determining the per unit price per product. 4. Marketing: advertising expenditures for each product (per region and period). 5.

Sales forecasting: evaluating market needs and predicting sales demand (volumes per product and period).

6. Material requirements planning: execution of planning runs to determine orders for raw materials, packaging, and finished goods.


7


7.

Materials management: definition of re-order strategy and execution of purchase orders.

8. Production scheduling: determining production schedules based on stock levels and production resource utilization. 9. Investment decisions: investment in production improvements (capacity, set-up time). 10. Liquidity and treasury management: managing cash flows and liquidity as well as deciding on investments and debt reimbursement. Games are repeated several times during the overall course in order to allow students to understand the basic game manipulation and context. Each game can be individually configured to comprise between one and eight quarters. Each quarter corresponds to 30 days and lasts approximately 20 to 30 minutes. Every team can rely on a business intelligence system to analyze real-time data about its company and to access market reports. Teams can manipulate the information and can create their own management dashboards through pivot tables in Microsoft Excel. During the game, teams cannot see other teams’ data; nonetheless, we provide it at the end of every game, to foster their understanding and allow them to improve their strategy. Overview of the Curriculum The capstone course’s curriculum comprises three phases: introduction, reflection, and synthesis. In total, the three phases last 7 weeks, within which students work either on the school’s premises or autonomously (see Table 2). The last row in Table 2 relies on Bloom’s taxonomy of learning goals. In Phase 1 (introduction to the business simulation game), students spent one week (five full days) at the university (1) to understand the simulation environment and the basic ERP manipulations, (2) to master the tools with which to analyze their company’s performance, and (3) to get guidance on how to elaborate their strategy. The main emphasis of week 1 is the introduction to the game and activation of prior knowledge. By means of short modules of 30 to 60 minutes along the week, concepts from prior courses are reviewed and demonstrated with the simulation game. This allows students to activate knowledge from prior courses and strengthens their integration and application of the game. Having five consecutive days allow instructors to efficiently present the game environment and analysis tools, while students can focus wholly on the course. On day 1, to facilitate the manipulation of the ERP-based simulation tool and provide a quick hands-on experience, teams rely on the introductory version of ERPsim. In this version, teams start with finished goods and raw materials in their inventory. Thus, they can directly market and sell from the inventory, while learning to manipulate the SAP transactions step-by-step. They play three rounds, starting with the sales process in round 1, including the production process in round 2, and adding procurement and a simplified planning process in round 3. On day 2, teams switch to the extended version of the simulation game. Compared to the introductory game, teams must perform pre-games transaction; for instance, they must decide on their investments in production capacities / set-up time and must define their product portfolio and recipes. The extended game requires them to forecast and plan their sales, then order raw materials and run the production accordingly, before they can begin to sell. The same day, teams are introduced to the design of new products as well as the different investment decisions they can realize along the game. On day 3, students master the game manipulation and are introduced to in-depth business data analytics and business intelligence tools (BI tools) to analyze the performance of their company. Every team can access real-time data from the SAP system using a standard Microsoft Excel file with predefined ODBC connections and pivot tables. In the afternoon, students play two to three rounds to incorporate the analytical tools in their decision-making. On day 4, students first review the basics of formulating and operationalizing a strategy. They create a balanced scorecard that will help them to define and monitor the strategy they will then apply. Along a two-round game, students have the opportunity to test these tools and implement their strategy in the game. Day 5 is dedicated completely to a game of four to five rounds, to allow the teams to defend their strategy and apply the concepts and tools that will support its implementation. In Phase 2 (reflection on company strategy), teams work autonomously during six weeks in order to define their company’s strategy and to work on operationalization in the different business functions for the final game. Based on the experience and concepts acquired during Phase 1 and their prior

8



knowledge, teams must elaborate a strategy that will allow them to become the most profitable company. They must document their strategy in a report and provide justifications for their decisions by analyzing data from the previous games and by using concepts taught in their Bachelor program. Teams are also requested to explain how they will operationalize their strategy with the ERP-based simulation game in the final game. In addition, teams must define their strategic objectives using a balanced scorecard and must prepare analytical tools to measure and monitor their performance during the game by means of key performance indicators (KPIs). Since teams do not have the opportunity to play during the six weeks of the current phase, two training sessions are organized. In case of questions, teams rely on an online forum shared with the course’s professors and assistants. At the end of this phase, each team hands in a report explaining their strategy, its operationalization, and their complete balanced scorecard as performance management tool. Phase 3 (synthesis) comprises three days at the university. Teams play the final game, analyze their performance, and present their synthesis to the other teams. Day 1 is dedicated to the final game, which takes between 5 and 6 hours. At the end of day 1, we provide teams with data of all companies so that they may use this to underpin their analysis. On day 2, teams work autonomously to critically analyze their performance in the final game. They must explain how they followed their strategy, explain their performance, and derive the success factors of a winning strategy. On day 3, each team presents its strategy and performance to the other teams. This allows students to understand the market dynamics during the game and to compare themselves to their competitors. In line with our learning objectives, emphasis is placed on discussing the strategic positioning of the different teams (in terms of market positioning, investments, and internal operations management) and analyzing the differences in performance.


9


Table 2: Overview of the Curriculum 1. Introduction

2. Reflection

3. Synthesis

Duration

5 days (on-site)

6 weeks (autonomous work, in groups)

3 days (on-site)

Course content

Phase

Day

Day 1.1: Introduction to ERPsim

Day 1.2: Introduction to ERPsim

Day 1.3: Analysis tools

Game rules and operations

Basic game manipulations: Sales, pricing, production, planning, procurement

Basic game manipulations: Financials, product design, investments

Business intelligence

Gaming and debriefing

Introductory game

Extended game (new)

Stepwise introduction in 3 rounds: 1) sales 2) production 3) procurement / simplified planning

Knowledge activation

Day 1.4: Strategy

Day 1.5: Test game

In parallel to semester

Day 3.1: Final game

Extended game (new)

Extended game (new)

Extended game (new)

Training sessions: Ext. game

Extended game (new)

4 rounds

2 to 3 rounds

2 to 3 rounds

4 to 5 rounds

2 x 2 rounds

5 to 8 rounds

Cash-tocash cycle

Performance analysis / KPIs

Strategy formulation

Analysis of company performance

Preparation for test game

Self-study

Application

Application

Application, analysis

Day 3.3: Final presentation

Balanced scorecard Preparation of strategy

Analysis of company and market performance

Operationalization Balanced scorecard

Learning goals (Bloom et al. 1956)

Day 3.2: Analysis




Application, analysis, synthesis

Application Analysis

Analysis, synthesis

Synthesis


Evaluation Evaluation Approach Our evaluation approach consisted of two types of evaluations, (1) an assessment of the learning effectiveness based on our predefined cognitive learning goals, and (2) an evaluation of the overall learning experience. We gathered data of a total of 64 students who completed the capstone business course during spring term 2013. For evaluation type 1, we applied a pretest and posttest design study to determine whether the capstone course has the intended effect of cognitive learning on the participants. For this purpose, students were asked to self-assess their performance along different factors, most importantly our four cognitive learning objectives. We included additional factors that were not our primary objective but were often assessed in other studies related to ERPsim (Foster and Hopkins 2011; Seethamraju 2011; Tyran and Springer 2012). These studies evaluated participants’ ERP skills improvement, their ability to analyze and monitor their performance, effectiveness of team work and organization, understanding of business processes, and their integration. The questionnaires were handed out and gathered at the end of week 1 (i.e. day 1.5 in Phase 1) as well as after the final game (i.e. day 3.1 in Phase 3). We decided to exclude the early phase of the course from our evaluation, because participation in business simulation games requires some preparation and introduction to the basic game manipulation. This particularly applies to our students, who did not have prior ERP knowledge, and had to be trained to use the SAP transactions in ERPsim. For evaluating the learning effectiveness of our capstone course, we deliberately emphasized the learning achievements between the end of Phase 1 and Phase 3, which we consider the most critical phases concerning the achievement of the defined learning objectives. In total, we received complete questionnaires for both the introductory week and the final game from 36 students. Evaluation type 2 consisted of a standard questionnaire used at the end of every curriculum in HEC Lausanne. It sought primarily to support the development of teaching skills, as well as to assess the quality of teaching delivered by the professors (Berthiaume et al. 2011). This questionnaire was built on the work of Marsh (1992) on the multidimensional perspective of students’ evaluation and on the research carried out by Centra (1993), and Perry and Smart (1997) on the teaching effectiveness. Thereby, questions relating to students’ overall satisfaction regarding the course and the surrounding pragmatic aspects (availability of the professors, number of assistants, workload required from students, etc.). Even if all aspects would not fit in the current analysis, several points addressed by the questionnaire proved relevant to assessing the overall learning experience and the appropriateness of the capstone course’s curriculum design. Such aspects include reflection stimulation, integration into the Bachelor program, and fit with prior knowledge. A total of 56 students answered the standard questionnaire.

Evaluation Results Regarding Learning Effectiveness As outlined in our literature review, a central issue in applying simulation games for educational purposes is the demonstration of learning effectiveness. For this purpose, we included the four learning objectives (Questions 1 to 4) into the evaluation. We also decided to evaluate the importance of other factors, such as team organization and the capacity of mastering ERP and analyzing performance (Questions 5 to 7). These questions are measured in other studies related to ERPsim. Finally, we added a control question against the learning objectives (Question 8) and asked students to rate their performance in each of these dimensions on a 5-point Likert scale. To evaluate the overall learning achievements, we computed the means and the percentage of agreement for each question in the two different stages (after Phase 1 and after Phase 3). We used the Wilcoxon signed rank test (significance value set to 5%) to test for significance. The motivation for using this test instead of a t-test was that our results are not normally distributed. The results are displayed on Table 3. For assessing learning effectiveness and comparing progress between the two measurements, we particularly focused on the percentage of students with high agreement levels, that is, those indicating 4 or 5 on the 5-point Likert scale. The difference between the number of students attending the course and the number of answers for the questionnaires can be explained by the fact that participation in the questionnaires was not mandatory.


11


Our results demonstrate that students have progressed with regard to the predefined four primary learning objectives, although with differences. Regarding the definition of a strategy, students indicated that their ability to formulate a clear, consistent, and flexible strategy had significantly improved. Overall, 83.3% of the students were of the view that they were able to define adequate strategies at the end of the curriculum, while the proportion after the introductory phase represented only half the respondents. We explain this progress by means of the fact that students invested most of the time between the introduction week and the final game in defining their strategy and documenting it in the report before the final game.



Table 3: Results from the Questionnaires on Learning Effectiveness1 After Phase 1

After Phase 3

Wilcoxon test Asymp. Z significance

Mean

Agreement2

Mean

Agreement2

Means difference3

1.1 Our team was able to define an adequate strategy.

3.46

51.4%

4.14

83.3%

0.68

-2.821a

0.005

1.2 Our strategy was based on a clear vision of how we are going to do business.

3.78

73.0%

4.24

94.6%

0.46

-2.812

a

0.005

1.3 Our strategy had a high level of coherence across different business areas.

3.51

51.4%

4.08

80.6%

0.57

-2.808a

0.005

1.4 Our strategy met the trade-offs and constraints in the game.

3.38

48.6%

3.84

67.6%

1.5 We used concepts from our prior studies to define our strategy.

3.39

51.5%

3.76

73.5%

2. Operationalizing your strategy 2.1 We were able to operationalize our strategy through well-specified decisions and actions. 2.2 We defined clear actions regarding marketing and sales.

3.61

58.3%

3.77

3.22

43.2%

2.3 We defined clear actions regarding production.

4.31

2.4 We defined clear actions regarding purchasing.

4.00

2.5 We defined clear actions regarding finance.

Learning objectives of the curriculum and related questions 1. Defining your strategy

0.46

-2.307

a

0.021

0.37

-2.235a

0.025

71.4%

0.16

-.692a

0.489

3.95

75.7%

0.73

-3.172a

0.002

94.4%

4.30

94.6%

81.1%

4.30

94.6%

3.76

64.9%

4.30

3.1 Our team was able to set suitable objectives for managing our business.

3.62

55.9%

3.2 We defined clear objectives relating to our overall business performance.

3.58

3.3 We defined clear objectives relating to customer-related aspects.

3.33

3.4 We defined clear objectives relating to financials.

-0.01

-.024

b

0.981

0.30

-1.669a

0.095

83.8%

0.54

-2.330a

0.020

3.89

72.2%

0.27

-1.950a

0.051

55.6%

4.03

75.7%

0.44

-2.497

0.013

41.7%

3.91

74.3%

0.58

-2.553a

0.011

3.59

56.8%

4.00

78.4%

0.41

-1.610a

0.107

3.5 We defined clear objectives relating to internal business.

3.86

75.0%

4.00

73.0%

0.14

-.991a

0.321

3.6 We defined clear objectives relating to innovation and learning. 3.7 Our objectives were refined by suitable indicators (KPI) to measure the achievements. 3.8 We formulated our objectives and goals in the form of (complete) balanced scorecard. 4. Defending your strategy

3.57

56.8%

3.92

73.0%

0.35

-1.986a

0.047

3.38

48.6%

4.11

80.6%

0.73

-3.337a

0.001

3.54

45.9%

4.19

80.6%

0.65

-3.540a

0.000

4.1 Overall, we were able to successfully defend our strategy in the market.

3.03

27.8%

3.42

61.1%

0.39

-1.418a

0.156

4.2 During the game, we were able to quickly react to the market dynamics.

3.05

29.7%

3.65

54.1%

0.59

-2.750a

0.006

4.3 During the game, we had to make major adjustments of our strategy.

3.46

59.5%

3.41

51.4%

-0.05

-.094b

0.925

4.4 During the game, we were able to achieve the predefined objectives. 3.24 Legend: 1 Questionnaires used a 5-point Likert scale 2 Agreement = percentage of answers on points 4 and 5 of the Likert scale 3 Means difference = Mean (after Phase 3) – Mean (after Phase 1) Wilcoxon signed rank text: a based on positive ranks; b based on negative ranks

40.5%

3.51

51.4%

0.27

-1.218a

0.223

3. Setting goals a

Thirty Fourth International Conference on Information Systems, Milan 2013 13


Table 3 (cont.): Results from the Questionnaires on Learning Effectiveness1 After Phase 1

After Phase 3

Wilcoxon test Asymp. Z significance

Mean

Agreement2

Mean

Agreement2

Means difference3

5.1 Overall, my team was well organized.

4.59

97.3%

4.59

91.9%

0.00

.000c

5.2 Each team member assumed a well-defined role.

4.73

100.0%

4.70

91.9%

-0.03

-.188

5.3 Each team member was able to fulfill his or her role.

4.57

94.6%

4.70

94.6%

0.14

-1.698a

0.090

5.4 In our team, we established a sound work structure.

4.57

94.6%

4.65

94.6%

0.08

-.832a

0.405

5.5 In our team, we took decisions quickly and efficiently.

4.14

75.7%

4.27

83.8%

0.14

-.729a

0.466

5.6 In our team, we were able to resolve conflicts.

4.30

89.2%

4.46

91.9%

0.16

-1.163a

0.245

5.7 In our team, we enjoyed working in the team.

4.73

100.0%

4.73

94.6%

0.00

-.028

b

0.978

6.1 Each team member was able to execute the relevant SAP transactions.

4.29

85.7%

4.29

85.3%

0.01

-.206a

0.837

6.2 I was able to execute the SAP transactions relating to marketing/sales.

4.41

86.5%

4.19

80.6%

6.3 I was able to execute the SAP transactions relating to planning.

4.06

66.7%

3.86

66.7%

6.4 I was able to execute the SAP transactions relating to production.

4.22

83.3%

4.14

6.5 I was able to execute the SAP transactions relating to purchasing.

4.08

75.0%

6.6 I was able to execute the SAP transactions relating to financials.

4.06

72.2%

Other factors mentioned in studies on ERPsim and related questions 5. Organizing your team

a

1.000 0.851

6. Mastering the ERP -0.21

-.716

b

0.474

-0.19

-.790b

0.430

72.2%

-0.08

-.508b

0.612

3.89

66.7%

-0.19

-.630b

0.529

3.89

66.7%

-0.17

-.370b

0.712

7. Analyzing your performance 7.1 At any time during the game, we had a good overview of our performance.

3.88

76.5%

4.11

88.9%

0.23

-2.502a

0.012

7.2 Our tools were sufficiently ready to analyze performance.

3.58

63.9%

4.19

86.1%

0.61

-2.571a

0.010

7.3 Our tools were sufficiently ready for us to draw conclusion from them.

3.58

55.6%

4.06

77.8%

0.47

-1.541a

0.123

8.1 Overall, our team achieved excellent performance. 2.81 8.2 We could have achieved better performance if we had defined a better or other 3.81 strategy. 8.3 We could have achieved better performance if we had been able to better 3.53 operationalize our strategy. 8.4 We could have achieved better performance if we had set clearer objectives and 3.47 goals. 8.5 We could have achieved better performance if we had been able to do a better data 3.61 analysis. 8.6 We could have achieved better performance if we had been luckier. 4.00 Legend: 1 Questionnaires used a 5-point Likert scale 2 Agreement = percentage of answers on points 4 and 5 of the Likert scale 3 Means difference = Mean (after Phase 3) – Mean (after Phase 1) Wilcoxon signed rank text: a based on positive ranks; b based on negative ranks

24.3%

3.05

41.7%

0.24

-1.166a

0.244

63.9%

2.76

38.9%

-1.05

3.632

b

0.000

63.9%

2.68

27.8%

-0.85

2.769b

0.006

58.3%

2.27

8.3%

-1.20

3.793b

0.000

61.1%

2.35

16.7%

-1.26

3.826b

0.000

78.1%

3.30

58.3%

-0.70

2.085b

0.037

8. Overall performance in the final game



With respect to the operationalization of students’ strategy, our results show overall good performance. Students indicated that their aptitude for defining clear actions in finance, marketing, and sales improved significantly over the course of the curriculum. However, the definition of a set of actions regarding production and purchasing does not show clear progression. We explain this by the high rates for students’ self-assessed performance for these factors after the introduction phase (4.31 and 4.00 respectively). Also, 94.4% of the respondents indicated that they had a good grasp of their production action plan, while 81.1% indicated a good grasp of the actions related to purchasing. This is a result of our emphasizing the importance of having a clear purchasing and production schedule so as to never fall short in inventory in the introduction phase. Overall, at the end of the curriculum, students felt more confident in operationalizing the strategy they had defined (71.4% vs. 58.3% after Phase 1), but we cannot assume that this progress is significant. The outcomes of the questionnaires for learning objective 3 – setting goals – require thorough evaluation. The Wilcoxon test does not show significant improvement in setting suitable objectives, especially objectives related to financial, internal business, innovation, and learning aspects. However, the asymptotic significances of Questions 3.1, 3.4, and 3.6 are close to the significance value of 5% that we had predetermined. Additionally, at the end of the curriculum, more than 70% of the respondents considered themselves able to set suitable goals for each field they had to assess (Questions 3.1 to 3.6), while all proportions except those relating to internal business were less than or equal to 56.8% after the introduction phase. This points to significant problems in defining realistic goals for the final game. On the positive side, the definition of performance indicators in the form of KPIs and the formulation of the goals and objectives in the form of balanced scorecard show clear improvement between these two phases. One may conclude that the setting of objectives is easier in form than in substance. Finally, defending the strategy did not significantly improve when looking at the mean value. However, the level of agreement tells a different story: While during the first phase, only 27.8% of students considered themselves able to defend their strategy, this value increased to more than 60% in the final phase. This a priori contradictory result can be explained by the distribution of the responses (see Table 4). The major difference is found in the mode shifting from point 3 on the Likert scale during Phase 1 to point 4 for the final phase. Thus, the mean is not subject to significant change, but the students clearly indicate higher confidence in the way they defended their strategy the second time. Nevertheless, even if their conduct of the strategy may have improved, one may think that students were not clearly able to achieve the predefined objectives. We infer this from the significant market dynamics, which are hard for students to predict, and the unrealistically high objectives students had defined in the presentations and reports they delivered in Phase 2. Table 4: Distribution of Answers to Question 4.1 Not fully

Fully N

Phase 1

3

7

16

6

4

36

Phase 3

3

8

3

15

7

36

Furthermore, our results demonstrate that there was no significant change regarding ERP-related, BIrelated, and organizational skills (factors 5 to 7). These results confirmed that our curriculum design did not foster skills relating to system usage. Since the students were new to the ERP system, we attribute the stability of their ERP-related skills to the nature of the curriculum. In fact, students already spent considerable time using the ERP system in the introduction phase. As the first questionnaire was filled out only after Phase 1, we assume that the main learning progress took place during our introduction phase. Regarding BI-related aspects, we infer the lack of analytical skills improvement from the already high percentage of students who indicated having a good overview of their performance (76.5%). Interpersonal and organizational skills were highly valuated in both phases (between 75.7% and 94.6%), thus leaving little room for improvement. In conclusion, students have improved their abilities to define a strategy and to formulate goals and objectives in the form of KPIs and a balanced scorecard. For these learning objectives, we can safely affirm the learning effectiveness of the curriculum we designed. Students also indicated higher confidence in operationalizing the strategy, but had difficulties defending their strategy in a dynamic market.



Based on the responses to open questions, we found that the current curriculum also activates knowledge, since students declared using the following notions that had been covered in the Bachelor program: (i) management, (ii) operations management (set-up time, capacity, bottlenecks, etc.), (iii) decision analysis, (iv) analytical accounting (ROE, working capital, etc.), (v) pricing policies (market-based, cost-based, etc.), (vi) macro-economics and micro-economics, and (vii) marketing.

Evaluation Results Regarding the Capstone Course’s Appropriateness We also evaluated the capstone course’s appropriateness by means of a standard questionnaire handed to every student at the end of every curriculum in the faculty. This questionnaire contains questions about several aspects that increase the satisfaction level with the course, which students must rate on a 4-point Likert scale. Of relevance for this paper are the questions regarding the integration into the curriculum, stimulation of reflection, and fit with prior knowledge. First, the current curriculum appears to be relevant and fit well with the student’s overall curriculum, since 87.3% of the students indicated that this course curriculum is well integrated into their Bachelor studies (rate 4 on the Likert-scale). Second, 89.3% considered themselves mentally stimulated. Third, 74.5% posited that the knowledge and skills required by the course had previously been covered in their curriculum. In general, the business game proved valuable according to 74.1% of the respondents. In fact, a great deal of comments from students supporting the course’s value mentioned the opportunity to practice the theories they had covered in their education and the opportunity to elaborate and defend a multidimensional strategy that requires them to use a large skills set specific to managers. In addition, 85.6% greatly appreciated the curriculum. In line with prior studies on business simulations, the main aspects contributing to the positive experiences, according to the students, relate to the engaging and stimulating nature of simulation games, the autonomy that problem-based courses require from the students, and the strong analogy to real-world situations, which allowed every student to feel like a “real manager” for a few weeks.

Interpretation and Discussion Based on our empirical data, we conclude that our three-phased curriculum design fosters knowledge activation and integration among Bachelor students. In contrast to the standard curriculum applied for business simulation games, our course design comprises extensive reflection and synthesis phases, which allow students to integrate knowledge from their prior discipline-specific courses in order to formulate and operationalize their company’s strategy. We find this aspect to be particularly important in order to achieve the cognitive learning goals, because other curriculum designs often focus only on the gaming experience. Since they are also shorter (for ERPsim, for instance, between one full day and 6 sessions of 3 hours are reported), such courses risk not giving students enough time for reflection and analysis. Our empirical evaluation confirms learning outcomes related to cognitive learning, most importantly, strategy definition and goal setting. In contrast to other studies (Dunlap 2005), our study does not find that team efficiency is increased. This may be due to our evaluation approach, which does not cover the early phases of introduction to the game. A similar observation relates to ERP-related skills, which in our case were not found to change significantly over time. Our study demonstrates that ERP-based simulation games can fulfill several purposes: While they are already widely used for teaching IS, they can also enrich general management curricula. In contrast to more specialized business games, they allow one to integrate knowledge from different management disciplines, notably marketing, operations management, controlling, and accounting. They are particularly useful for demonstrating the complexity of real-world environments, since the ERP system supports decision-making and operational transactions within and across different business functions.

Conclusion In this article, we suggest a curriculum design for a capstone course that leverages ERP-based simulation games to create an experiential learning experience for students and a close-to-reality business environment. Our study thereby complements prior work on ERP-based simulation games that mostly



focus on reporting learning experiences and pay little attention to the instructional and pedagogical strategies that are required to leverage the full potential of problem-based learning and simulation approaches (Léger et al. 2011). Our empirical results demonstrate that this curriculum design is effective with regard to cognitive learning and fulfills the goal of activating and integrating management-related knowledge from prior discipline-specific courses. To our best knowledge, the suggested curriculum is the only one to leverage general management knowledge and skills through the use of an ERP-based simulation, that is, ERPsim. Our research thereby underlines the versatility of ERP-based business simulations and confirms their usefulness in creating learning achievements in learning contexts other than information systems.

Implications for Research and Education As an implication of our research, we suggest that more studies should investigate learning objectives and curriculum design in the context of business simulation games. We also suggest studying the versatility of ERP-based business simulations and identifying the different roles they may play in management education. While prior research has emphasized their role in teaching information systems and business processes, we find that they are also valuable in teaching general management concepts.

Limitations One of the study limitations is that the evaluation of learning achievement relies on students’ perceptions in the specific capstone course. To complement this subjective evidence, it would be interesting to assess the actual, direct evidence of learning (Anderson and Lawton 2008). In our case, a direct evaluation would be possible, for instance, by analyzing the ERP system logs. Since our approach assesses cognitive learning outcomes only within our student cohort, it does not prove the effectiveness of game-based learning compared to alternative learning strategies. In this regard, we are confronted with the same methodological difficulties in assessing learning effectiveness as the prior literature (Randel et al. 1992)

Reference List Anderson, P. H., and Lawton, L. 2008. “Business Simulations and Cognitive Learning: Developments, Desires, and Future directions,” Simulation & Gaming. Ben-Zvi, T., and Carton, T. 2008. “Simulation Gaming in Technology Management,” in Proceedings of the 14th America Conference on Information Systems, Toronto, ON, Canada. Berthiaume, D., Lanarès, J., Jacqmot, C., Winer, L., Rochat, and J.-M., 2011. “L’Evaluation des Enseignements par les Etudiants (EEE): une Stratégie de Soutien au Développement Pédagogique des Enseignants,” Recherche & Formation, pp. 53–72. Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., and Krathwohl, D. R. 1956. “Taxonomy of Educational Objectives: Handbook I: Cognitive Domain,” New York: David McKay (19), p. 56. Bonazzi, R., Missonier, S., Jaccard, D., Bienz, P., Fritscher, B., and Fernandes, E. 2012. “Analysis of Serious Games Implementation for Project Management Courses,” in Information Systems: Crossroads for Organization, Management, Accounting and Engineering, Springer, pp. 491–498. Centra, J.A., 1993. “Reflective Faculty Evaluation: Enhancing Teaching and Determining Faculty Effectivenes, ” The Jossey-Bass Higher and Adult Education Series. Coakley, J. R., Drexler, J. A., Larson, E. W., and Kircher, A. E. 1998. “Using a Computer-based Version of the Beer Game: Lessons Learned,” Journal of Management Education (22:3), pp. 416–424. Cronan, T. P., and Douglas, D. E. 2012. “A Student ERP Simulation Game: A Longitudinal Study,” Journal of Computer Information Systems . Cronan, T. P., Léger, P.-M., Robert, J., Babin, G., and Charland, P. 2012. “Comparing Objective Measures and Perceptions of Cognitive Learning in an ERP Simulation Game A Research Note,” Simulation & Gaming (43:4), pp. 461–480.



Davis, C. H., and Comeau, J. 2004. “Enterprise Integration in Business Education: Design and Outcomes of a Capstone ERP-based Undergraduate e-Business Management Course,” Journal of Information Systems Education (15:3), pp. 287–299. Deng, X. (Nancy), and Chandler, J. 2010. “Learning in Enterprise System Support: Specialization, Task Type and Network Characteristics,” in Proceedings of the International Conference on Information Systems . Dunlap, J. C. 2005. “Problem-Based Learning and Self-Efficacy: How a Capstone Course Prepares Students for a Profession,” Educational Technology Research and Development (53:1), pp. 65–85. Davenport, T.H., 1998. “Putting the Enterprise into the Enterprise System,” Harvard Business Review (76:4), pp.121–131. Durel, R. J. 1993. “The Capstone Course: A Rite of Passage,” Teaching Sociology (21:3), pp. 223–225. Faria, A. J. 2001. “The Changing Nature of Business Simulation/Gaming Research: A Brief History,” Simulation & Gaming (32:1), pp. 97–110. Forrester, J. W. 1958. “Industrial Dynamics,” Harvard Business Review (36:4), pp. 37–66. Foster, S., and Hopkins, J. 2011. “ERP Simulation Game: Establishng Engagement, Collaboration and Learning,” in Proceedings of the Pacific Asia Conference on Information Systems, Australia. Gilinsky, A., and Robison, R. 2008. “A Proposed Design for the Business Capstone Course with Emphasis on Improving Students’ Information Competency,” Journal of Management Education (32:4), pp. 400– 419. Haines, R., Hough, J., Haines, D., and Metlen, S. 2011. “Using a Web-Based Supply Chain Management Simulation as an Experiential Learning Tool Across the Business Curriculum,” Journal of College Teaching & Learning (TLC) (2:12). Kaplan, R. S., and Norton, D. P. 1996. "Using the Balanced Scorecard as a Strategic Management System," Harvard Business Review (74:1), pp. 75-85. Kiili, K. 2005. “Digital game-based learning: Towards an experiential gaming model,” The Internet and higher education (8:1), pp. 13–24. Léger, P.-M. 2006. “Using a Simulation Game Approach to Teach Enterprise Resource Planning Concepts,” Journal of Information Systems Education (17:4), pp. 441–447. Léger, P.-M., Charland, P., Feldstein, H. D., Robert, J., Babin, G., and Lyle, D. 2011. “Business Simulation Training in Information Technology Education: Guidelines for New Approaches in IT Training,” Journal of Information Technology Education (10), pp. 39–53. Marsh, H.W., 2007. “Students’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases and usefulness, in: The Scholarship of Teaching and Learning in Higher Education: An Evidence-based Perspective, ” Springer, pp. 319–383. McKone, K., and Bozewicz, J. 2003. “The ISM Simulation: Teaching Integrated Management Concepts,” Journal of Management Education (27:4), pp. 497–515. Michael, D. R., and Chen, S. L. 2005. Serious Games: Games that Educate, Train, and Inform, Muska & Lipman/Premier-Trade. Moreno-Ger, P., Burgos, D., Martínez-Ortiz, I., Sierra, J. L., and Fernández-Manjón, B. 2008. “Educational game design for online education,” Computers in Human Behavior (24:6), pp. 2530–2540. Parr, A.N. & Shanks, G., 2000. “A Taxonomy of ERP Implementation Approaches," System Sciences, in Proceedings of the 33rd Annual Hawaii International Conference, USA, p. 10. Perry, R.P., Smart, J.C., 1997. “Effective Teaching in Higher Education: Research and Practice, ” Agathon Press.



Pfahl, D., Laitenberger, O., Ruhe, G., Dorsch, J., and Krivobokova, T. 2004. “Evaluating the Learning Effectiveness of Using Simulations in Software Project Management Education: Results from a Twice Replicated Experiment,” Information and Software Technology (46:2), pp. 127–147. Randel, J. M., Morris, B. A., Wetzel, C. D., and Whitehill, B. V. 1992. “The Effectiveness of Games for Educational Purposes: A Review of Recent Research,” Simulation & Gaming (23:3), pp. 261–276. Riemer, K. 2008. “The Beergame in Business-to-Business eCommerce Courses – A Teaching Report,” In Proceedings of the 21st Bled eConference, Slovenia. Roberts, C. R., and Wolfe, J. 1986. “The External Validity of a Business Management Game: A Five-Year Longitudinal Study,” Simulation and Games (17:1), p. 45. Seethamraju, R. 2011. “Enhancing Student Learning of Enterprise Integration and Business Process Orientation through an ERP Business Simulation Game.,” Journal of Information Systems Education (22:1), pp. 19–29. Shang, S. & Seddon, P.B., 2002. “Assessing and managing the benefits of enterprise systems: the business manager’s perspective, ” Information Systems Journal (12:4), pp.271–299. Smith, G. F. 2005. “Problem-based Learning: Can it Improve Managerial Thinking?,” Journal of Management Education (29:2), pp. 357–378. Stainton, A. J., Johnson, J. E., and Borodzicz, E. P. 2010. “Educational Validity of Business Gaming Simulation: A Research Methodology Framework,” Simulation & Gaming (41:5), pp. 705–723. Thorelli, H. B., and Graves, R. L. 1964. International Operations Simulation: With Comments on Design and Use of Management Games, (Vol. 8) Free Press. Tyran, C., and Springer, M. 2012. "Learning and Team Attributes in an Enteprise Systems Simulation," in Americas Conference on Information Systems 2012 Proceedings (28). Wolfe, J., and Roberts, C. R. 1993. “A Further Study of the External Validity of Business Games: Five-Year Peer Group Indicators,” Simulation and Gaming (24:1), p. 21.
