AN EXPERIMENT IN COLLABORATIVE LEARNING ... - CiteSeerX

Proceedingsofthe 28thAnnualHawaiilnternational Conferenceon SystemSciences- 1995

AN EXPERIMENT IN COLLABORATIVE LEARNING USING A GAME AND A COMPUTERMEDIATED CONFERENCEIN ACCOUNTING COURSES Malcolm Worrell Director, M S in Management School of Industrial Management 201-596-5262 email: W [email protected]

Starr Roxanne Hiltz Distinguished Professor Computer Science Department 201-596-3388 email: [email protected]

Murray Turoff Distinguished Professor Computer Science Department 201-596-3399 email: [email protected]

Jerry Fjermestad Assistant Professor School of Industrial Management 201-596-3255 email: [email protected]

All affiliated with New Jersey Institute Heights, Newark, New Jersey 07102.

University

particularly subject to criticism. most medium-sized Also, and large commercial firms have their own internal audit departments whose function is entirely audit. Closely related to internal auditing is the design of accounting information systems, an area that has also been subject to much criticism. One of the main areas of concern to and auditors accounting information system designers is that of internal controls. The AICPA Statement on Auditing Standards (SAS) No. 55 provides guidance on the independent auditor's consideration of an entity's internal control structure in an audit of financial statements in accordance with generally accepted auditing standards. Auditors are required to be familiar with three elements: the control environment, the accounting system, and control procedures. SAS No. 30 describes procedures an accountant should apply to report on systems of internal accounting controls. SAS No. 47 defines the three types of risk that auditors must be aware of: inherent risk, control risk, and detection risk. SAS No. 53 concerns errors, irregularities and fraud. The need for internal controls, as to fraud prevention and they relate is a concern of auditors and detection, must be a concern of educators.

Abstract The accounting profession has criticized schools for failing to provide graduates who know how businesses function. For over 50 fifty years educators have tried new methods to improve delivery of this education. This paper describes a field experiments combining the use of a game and the availability of a Computer Mediated Conference (CMC) to improve learning in accounting courses. Results suggest a significant increase in learning with participation in the game and to a lesser extent suggest that the game in combination with the CMC may have a synergistic effect leading to collaborative learning.

Introduction In 1989 the Big eight Accounting firms the emphasized need to abandon the current lecture/problem style of teaching and to seek new methods. This led to the formation of the Accounting Change which Education Commission provided funding to a number of universities to promote innovations in accounting teaching. Since many of the accounting graduates enter the work force as auditors, this field was

1060-3425/95$4.00O1995IEEE

of Technology,

63

Proceedings of the 28th Hawaii International Conference on System Sciences (HICSS '95) 1060-3425/95 $10.00 © 1995 IEEE

Proceedings of the 28th Annual Hawaii International Conference on System Sciences -

Attempts courses

to

improve

accounting

In 1990, Hilts reported results of tests of collaborative learning using a "Virtual Classroom," a concept pioneered at New Jersey Institute of Technology (NJIT) and copyrighted. One objective of the virtual classroom is: "The improvement of the quality and effectiveness of education through using the computer to support a collaborative learning process.'* (Hiltz, 1990, p. 59) Some of the student perceptions of online courses included: increased convenience (65 percent), less work (63 percent), ability to participate on an irregular basis (49 percent), increased interest, involvement, and motivation for those who participated regularly, and increased interaction and better access to the professor (58 percent). Two recent books, summarizing with experiences distance learning throughout the United States and Europe (Mason 1990, and Kaye, and Harasim, 1990) indicate that the use of CMC for distance education is increasing and that the results cited previously are consistent with results reported in the books. Both books generally support a conclusion that learning, especially distance learning, is enhanced through the use of CMC. Currently over 80 programs worldwide are known to be offering courses partially or completely via computer-mediated communication (See, for example, Harasim, 1989; Harasim, Hiltz, Teles & Turoff, 1993; Hilts, 1986; Hsu & Hiltz, 1991; Mason & Kaye, 1989; McCreary & VanDoren, 1987; Paulsen & Rekkedal, 1990; Rice & Case, 1983; Weedman, Wells, Welsch, 1991; 1990; 1982). The sophistication and flexibility of software structures for distance vary supporting education simple widely, from electronic mail systems to conferencing systems that have been specially enhanced to support classroom-likeexperiences, particularly group discussions and joint projects. (Hiltz, 1993).

The academic community has responded to pressures over the years by coming up with novel methods of increasing realism in accounting courses. Many of these approaches dealt directly with some aspect of internal controls. They can be categorized as: Exposing students to business transactions and their supporting documents (see Van Voorhis, 1953 and Tyson, 1986). Using techniques that are thought of as the MIS or computer science domain, such as: decision tables (Malcolm, 1966) ; computer processing, test decks and flow charts (Myers, 1972), and flow charts (Natulich, 1977). Involving students in business games (such as DeCoster, 1973). Using essentially simulation to techniques evaluate internal controls. (Weber, 1978 and Wiggins, 1986) Using simulations to model various parts of the accounting cycle (G-y, 1983 and Benjamin,l983; and Sale, 1972). Helping students increase their skills in auditing planning and decisions (Felix, 1985).

Attempts learning

to

use

CMCs to

I995

improve

Computer conferencing has been used in accounting instruction for a number of years (Stern, 1965). He describes the use of this method as a pedagogical teaching an device in accounting information systems course. He extols the value of this technique in improving the communications between students and the instructor when they He are unable to meet face-to-face. this concept is of suggests that greatest value when it fills a need that cannot be easily met by other methods, such as communication in a course that meets only once each week. 64

Proceedings of the 28th Hawaii International Conference on System Sciences (HICSS '95) 1060-3425/95 $10.00 © 1995 IEEE

Proceedings of the 28th Annual Hawaii International Conference on System Sciences -

Use of Gaming Improve Learning HSU'S 1991 thesis that supported four

and

a

cm

to

1995

realistic communication constraints appropriate to the game environment. Furthermore, players can reflect upon their views and choose a time to engage game and the communication in the process that is consistent with the satisfaction they have with their ideas and contributions. probably It is the communication have limited requirements for games that the use of games as a "real" management analysis and decision tool. Games are a standard approach to planning in the but they are rarely used in military, situations. real management Only in limited experiments where managers could escape for days at a time to engage in gaming exercises by total immersion has there been some use of gaming by real managers. The incorporation of CMC technology into the gaming environment provides the for the use of games as a management support tool.

reported experiment5 hypotheses:

There will be a favorable difference in learning interpersonal skills by participating in an extended organization simulation augmented by a Computer Mediated Conference (CMC) compared with same simulation without the aid of a CMC. There will be increased student interest/satisfaction when a CMC is provided for organization simulation or role play. be improved There will game performance when a CMC is provided for practicing managerial skills, particularly decision-making skills and information-processing skills.

Concept of collaborative with a CMC

There will be an increase in student written communication effectiveness when a CMC is provided for practicing managerial skills, particularly decision making skills and information processing skills.

learning

Collaborative learning is defined as a learning process that emphasizes group or cooperative efforts among faculty and students. It stresses active participation and interaction on the part of both students and instructors. Knowledge is viewed as a social construct, and therefore the educational facilitated by process is social interaction in an environment that facilitates peer interaction, evaluation and cooperation (Bruffee, 1986; Johnson, 1981; Johnson & Johnson, 1975). In the CMC environment one can do far more to collaborative encourage learning and utilize it as a learning mechanism than is possible in the normal face-to-face classroom environment. An example of a collaborative learning that is strategy included in most courses is the "seminar" type of interchange in which the students become Individuals or the teachers. small groups of students select a topic; read material not assigned to the rest of the class; prepare a written summary for the ideas in the class of the most important

His research was based on 327 students in an introductory management course, approximately half of which used a CMC. Hsu included a The experiment5 of comprehensive analysis of gaming as an educational tool. The work by Hsu, Hiltz, and Turoff (1992) was based on the hypothesis that limitation on the current the effectiveness of learning through gaming was imposed by a lack of communication Even facilities among the players. students, let alone real managers, have severe constraints on the amount of time they can devote to face-to-face group face for game playing, meetings. Also, interactions are often to face unrealistic in terms of reflecting the true communications structure of a game. In the CMC environment, both private messages can be and conferences utilized to reflect structured and 65

Proceedings of the 28th Hawaii International Conference on System Sciences (HICSS '95) 1060-3425/95 $10.00 © 1995 IEEE

Proceedings of the 28th Annual Hawaii International Conference on System Sciences -

material; and lead a discussion on the topic or material for which they are responsible. The use of CMC technology changes considerably the way in which course material is delivered when compared to face-to-face classes.

Theories

conference (CMC) available for their use. The game and CMC environments were tested under four conditions. The groups number of is given with the number of students in parentheses. Game Only CMC Only

to be tested

Both Neither

In considering the potential for the improve use of a game and a CMC to learning we contemplated the following. Design a game which would serve to teach the and to reinforce concepts of internal controls and fraud prevention or detection in a real business Make a CMC available for situation. students to use as a tool, but not make it a required part of the course. This seemed to be necessary since most of the students in the experiment were at offand we could campus locations not be guarantee that a modem would always communication. We available for theorized that the two concepts would work together to increase learning and also improve student satisfaction with the course. We then developed hypotheses to obtain data on the following theories involving the combination of a game and a CMC: Students design, fraud, Course Group

would learn and auditing.

experience decision

would making

Students would have attitudes about computers.

Development

more

of field

more

11

(37)

17 19 14

(45) (59) (50)

Students were given pre and post course tests to measure their learning in the courses and end of course questionnaires to obtain reactions in areas such as the effectiveness of the simulation, perception of learning, and attitudes toward computers. The primary research design is a two by two factorial. The first factor is participation in the game. The second factor is the availability of a computer mediated conference to develop and submit input to the game and to other course projects. All subjects were also members of groups in their courses. The hypotheses were tested at a significance -05. level of All tests in this research were based on SPSSPC+ software. The hypotheses were evaluated using a two way analysis of variance. Research subjects were selected from students in the Master Of Science in Management program at the New Jersey Institute of Technology and the Rutgers MBA. The research was conducted over a four semester period, beginning in the spring 1992 semester and ending in the spring

about

be improved.

would

1995

improve. favorable

1993

experiments

semester.

Discussion CMC

The field experiments involved the and CMC to introduction of a game improve accounting information systems and auditing education. In the game the students would function as a member of a team to design controls, to attempt to or to audit the system. commit fraud, of In addition to the game, about half the students had a computer mediated

How the

of game

is

the

game and

the

played

As stated previously, students were Members teams. assigned to one of three of the design team were given $100,000 to spend to buy internal controls for an They had to select system. inventory the controls that they considered most 66

Proceedings of the 28th Hawaii International Conference on System Sciences (HICSS '95) 1060-3425/95 $10.00 © 1995 IEEE

Proceedings of the 2Nth Annual Hawaii International Conference on System Sciences - 1995

valuable from a list of controls with an aggregate total of about $400,000. The learning value came from the discussions of the various controls. Members of the fraud team were given $100,000 to spend to try to commit fraud against the system. They were required to gather as much information about the control5 in the system in order to develop means to circumvent them. They had alternative choices valued at about $400,000 available to them. Their learning was to come from discussing controls with a view toward circumventing them. Finally, members of the audit team were given $100,000 to spend and had to learn enough about the system to write a report on the adequacy of the system of controls. They had to internal determine the data they needed and then design a set of audit procedures to They did not know if obtain this data. a fraud had been committed. As with the other teams, the audit team could select valued at about from alternatives $400,000. How the

CMC is

the

Search

to

Set up their own conferences members of the group.

in having

and if

all

to

messages their the

Submit instructor time.

via

Learning

design,

suggestions desired.

using

to

facilitate

key

words.

choices to the CMC, thereby saving

fraud,

and audit

Students were given tests in the beginning of the course to test their knowledge of the inventory area in general and of the three components of the game - design, fraud, and audit. Students were given the same test at the end of the course. ANOVAS were then run to on the end of course scores using the course as a beginning of scores covariate. The results of these tests show:

DF F-Ratio Total of all questions Game Effect 1 83.628 CMC Effect 1 20.688 Total for design questions Game Effect 1 40.166 4.194 CMC Effect 1 Total for fraud questions Game Effect 1 53.684 CMC Effect 1 18.493 Total for audit questions 24.684 Game Effect 1 CMC effect 1 5.295

Students were entered as members of a computer mediated conference for the specific class they were taken. They were given literature on the CMC, but did not have any formal training on it. Some of the regular class assignments were given out through the CMC, but hard Students copies were also given out. were encouraged to use the CMC but not Therefore, all required to do so. hypotheses refer to the availability of Actual a CMC and not its actual use. As embers usage data was not obtained. of the CMC, students would be able to do the following: meetings without meetings.

messages anonymously,

Use "pen-names" communication.

used

all Conduct asynchronous mode, arrange face-to-face

Submit group

Significance of F -000 .ooo . 000 .023 -000 . 000 . 000 .013

For all four areas of learning, the difference in scores for participation in the game were significant at the p < -001 level. This is a very strong indication that participation in the game improved learning. These results also indicate that the CMC can improve The fact that learning. this not stronger relationship is may be attributed to the fact that students had to learn how to use a CMC, which may have detracted from their learning of inventory areas. the various (The

an to

limited

67

Proceedings of the 28th Hawaii International Conference on System Sciences (HICSS '95) 1060-3425/95 $10.00 © 1995 IEEE

Proceedings of the 28th Annual Hawaii International Conference on System Sciences -

hypotheses called for a one tailed analysis since they pertained to an increase in learning. Accordingly, the effect of the CMC in total learning and in fraud learning were ignored as they reflected a decrease in learning.) Course experiences Three hypotheses game from an end of course questionnaires and related to students reactions in the area of course experiences. Specifically they concerned: how they felt about the group impact on learning, increased awareness of business problems and increased understanding of internal controls. These latter two are of particular interest since they relate to the concerns of the accounting profession. Significance DF F-Ratio of F Group impact on learning Game Effect 1 .555 -250 CMC Effect 1 .773 -386 Combined Effect 1 4.098 -024 Neither the game nor the availability of a CMC had an impact individually, but the the combination of two was significant.

Significance DF F-Ratio of F Awareness of business problems 1 3.510 -033 Game Effect 1 1.193 . 140 CMC Effect The game produced significant results, which is consistent with its nature. Significance DF F-Ratio of F Understanding internal controls Game Effect 1 2.370 .065 1 2.359 .065 CMC Effect gave results Neither treatment significant at the p = .005 level, however, they were at the p = .lO level. Internal controls was a key subject each course, so it is not of great concern understanding of internal that the controls was not significantly increased as a result of either treatment.

1995

The results of this series of tests were mixed. It is possible that the game and the availability of a CMC combined to improve results, but this is not conclusive.

Group decision

making

Four hypotheses came from the end of course questionnaire and related to the area of group decision making. Since students in all courses were assigned to groups for projects, these hypotheses were appropriate regardless of the game or the CMC. These hypotheses related to contributing to decisions, group behavioral problems experiences by the the expression of negative groups, and comfort in contributing to opinions, the The group. joint effect was considered for each of the hypotheses. Significance DF F-Ratio Contributing to Group Decisions Game Effect 1 4.575 CMC Effect 1 .024 Combined Effect 1 .039 Students felt they contributed group decisions with the game but a CMC available made no difference.

of

F

.018 .439 -422 to having

Significance DF F-Ratio of F Group behavioral problems Game Effect 1 2.768 -050 1 6.959 CMC Effect -005 Combined Effect 1 3.741 .029 All of the relationships are strong. Both the game and the availability of a CMC improved results individually and in total. Significance DF F-Ratio of F Expressing negative opinions Game Effect 1 3.496 -034 CMC Effect 1 6.063 -009 Combined Effect 1 3.645 -031 As with the area of group behavioral both the game problems, and the availability of a CMC produced favorable combination of the results as did the two.

Proceedings of the 28th Hawaii International Conference on System Sciences (HICSS '95) 1060-3425/95 $10.00 © 1995 IEEE

Proceedings of the 28th Annual Hawaii International Conference on System Sciences -

Significance DF F-Ratio of F Comfort in contributing to group Game Effect 1 2.221 .071 CMC Effect 1 .026 .437 Combined Effect 1 1.376 . 130 The results in all instances were positive but not significant.

Course

Experience GAME

Group

Impact

Awareness

Improved computers

attitudes

toward

Group Cont

of

Business

Behavioral

Significance DF F-Ratio of F Game Effect 1 * 094 . 381 CMC Effect 1 4.561 .018 the Only availability of a CMC produced significant differences, although the responses were heavily skewed for all combinations and were skewed to indicate a favorable attitude. A summary of the results of each set of hypotheses is presented below. GAME Learning Total

Measured Learning

Design

CMC BOTH

by Test

Learning

.OOO .OOO

N/A

.OOO .023

N/A

Fraud

Learning

.OOO .OOO* N/A

Audit

Learning

,000

Perception Inventory Audit Design

of

N/A

Learning

Learning

.009

Learning Learning

-013

N/A

Comfort

Opinions

Computer

.024 Problems .033 .065

N/A .065

N/A

Contributing

,018 .050

005

.029

.034

.009

.031

.018

N/A

-071

Attitudes

*The item direction

evaluated is of the hypothesis.

in

opposite

Conclusions The results discussed here support that participating in a game would improve learning and have favorable results in the remaining areas. The availability of a CMC produced a number of favorable, although not always significant, results. The combined effect of the two was significant in three of the five areas where it was tested. We conclude that there is likely to be a synergistic effect of combining participation in a game and having a CMC available. For students participating in the game, there will be a benefit from having the CMC available as a tool of communication and discussion. For the student who has a CMC available, the game gives him or a reason and a place to use. In essence, the game provides a structure for the CMC.

N/A .095

Decisions

Problems

in

BOTH

making

Group

Negative Ten questions in the end of course tests that measures students attitudes toward computers. The results of the analysis of these questions showed:

Controls

Decision to

CMC

on Learning

Understanding There is a clear indication that the combination of the game and the availability of a CMC improved results related to group decision making.

1995

N/A

Proceedings of the 28th Hawaii International Conference on System Sciences (HICSS '95) 1060-3425/95 $10.00 © 1995 IEEE

Proceedings of the 28th Annual Hawaii International Conference on System Sciences - 1995

REFERENCES F. Hansen, James V., and William Messier, "A Relational Approach to Decision Support for EDP Auditing," Communications of November the ACM, 1984, pp. 1129-133.

American Institute of certified Public Accountants, Auditinq Statement on Standards No. 30: Reportins on Internal Accountina Control, New York, 1980.

Harasim, Linda, (Ed.), Online Perspectives on a New EnvironntantPraeger, New Your,

American Institute of certified Public Accountants, Statement on Auditinq Standards No. 47: Audit and Risk Materialitvin Conductins the Audit, New York, 1983.

Hiltz, Starr Learning: The Approach," T.H.E. pp. 59-65.

American Institute of certified Public Auditinq Accountants, Statement on Standards No. 55: Consideration of the control Structure in a Internal New York, Financial Statement Audit, 1988.

"Expert Vs Novice Marinus J., Accounting: A Making in Accountins. Oraanizations and 1984, pp. 325-327.

Bruffee, History American (7), pp.

K. A. (1984). Background to Collaborative Learning Collese Enalish, Colleges. 635-652.

(1993). The Virtual Hiltz, S. R. Classroom: Learnino Without Limits via Computer Networks. Norwood NJ: Ablex Corp., Human-computer Publishing Interaction Series. Starr R, Kenneth Johnson, Hiltz, and "Experiments in Group Murray Turoff, Decision Making: Communication Process Outcome in Face-to-Face Versus and Computer Conferences," Human Communication Research, Winter 1986, pp.

and in 46

225-252.

Hsu, Enrico, "Management Games for Management Education: A Case Study,' Unpublished Ph.D. Dissertation, Rutgers University, 1991.

"An and George Prater, DeCoster, Don, Study of the Use of a Experimental Business Game in Elementary Accounting," The Accountino Review, January 1973, pp. 137-142. Donna A ., Hall, Software: Spreadsheet Financial Planning Journal of Accountinq, 159-169.

R., "Collaborative Virtual Classroom Journal, June 1990,

Hiltz, S. R. (1986). The virtual classroom: Using computer-mediated communication for university teaching. Journal of Communication, 36 (21, 95-104.

"A the The

Bouwman, Decision Summary"' Society,

1990.

Harasim, Linda, S. R. Hiltz, L. Teles, and M. Turoff, Learnina Networks: a Field Guide. MIT Pross, 1993.

American Institute of certified Public Accountants, Statement on Auditinq 53: 'The Auditor's Standards No. Responsibilitv to Detect and Report Errors and Irresularities, New York, 1988.

Benjamin, James, and Donald Ricketts, Planning Project in Profit Accounting Course, Management Accountinq Review, October 1973, PP. 794-797.

Education:

"Role-Event Gaming-Simulation Hsu, E., in Management Education: A Conceptual Framework and Review," Simulation and December 1989, 409-438. Games, (20:4),

"Microcomputer Powerful A Georqia Tool," Spring 1986, pp. 70

Proceedings of the 28th Hawaii International Conference on System Sciences (HICSS '95) 1060-3425/95 $10.00 © 1995 IEEE

Proceedings of the 28th Annual Hawaii International Conference on System Sciences - 1995

Hsu, E., & Hiltz, S. R. (1991). Management gaming on a computer-mediated conferencing system: A case of collaborative learning through computer Proceedinos of the Twentv conferencing. Fourth Annual Meetins of the Hawaii Conference on International Svstem IPP . 367-3711. Sciences, Vol. IV. Washington, DC: IEEE Computer Society..

"Computer Stern, Myles, Instruction, Accounting Accountina Education, 69-79.

for of pp.

Turoff, "Computer Murray, Mediated Requirements Communication for Group Support," Journal of Oraanizational Comoutinq, Volume 1, Number 1, pp. 85-113, 1991.

Hsu, E., Hiltz, S. R., and Turoff, M., Computer-Mediated Conferencing System as Applied to a Business Curriculum: A Research Update. In V. S. Jacob and H. Pirkul, eds., The Impact of Information Business Technology on Schools: Research, Teaching and Administration, Proceedinss of the 20th Annual North American Conference of the International Business School Computer Users Group, po . 214- 227, 1993.

and Starr R. Turoff, Murray, Hiltz, "Remote Learning: Technologies and Proceedinos of the World Opportunities," Conference on Continuino Enoineerinq Education, 1986. Turoff, M., U. Rao, and S. R. Hiltz, "Collaborative Hypertext and Computer Mediated Communications," Proceedinos of the 24th Hawaii International Conference on Svstems Science. Volume IV, IEEE Computer Society Press, 1991, 357-366.

Johnson, David W. (January, 1981). interaction: The Student-student variable in education. neglected Educational Research. (pp. 5-10). Knechel, Simulation Practice Education,

Conferencing Journal Spring 1965,

W.

Robert, "Using a Business Game as a Substitute for a Issues Set,", in Accounting Fall 1989, pp. 411-424.

Van Voorhis, 'Props' in Review, April

Robert H., "The Use of Teaching," The Accountinq 1953, pp. 285-286.

Weedman, J. (1991). Task functions of a computer used conference in education: A measure of International Journal of Studies, 34, 303-318.

Mason, Robin, and Anthony Kaye, (Eds.), Mindweave: Communication, Computers and Distance Education. Pergamon Press, New York, 1990. Matulich, Serge, Loren A. Nikolai, and Stevan K. Olson, "Earnings Per Share: A Flow Chart Approach to Teaching Concepts and Procedures," The Accountinq Review, January 1977, pp. 233-247.

and

non-task

professional flexibility. Man-Machine

Wells, R. A. (1990). Computer-mediated communications for distance learning and Boise, ID: training. Boise State University. (1982). Welsch, L. A. Using electronic mail as a teaching tool. Communications of the ACM, 25 (2), 105-108.

McCreary, E. K., & Van Duran, J. (1987). applications of computer Educational Canadian conferencing. Journal of Communication, 16, Educational (2)t 107-115.

Wiggins, Casper E. and L. Murphy Smith, "A Generalized Audit Simulation Tool for Evaluating the Reliability of Internal Controls," Contemporary Accountinq Research, 1986, pp 316-337.

"Using Computerized Sale, J. Timothy, Budget Simulation Models as a Teaching October Device," The Accountina Review, 1972, pp. 836-839. 71

Proceedings of the 28th Hawaii International Conference on System Sciences (HICSS '95) 1060-3425/95 $10.00 © 1995 IEEE