implementation issues with games-based assessment

IMPLEMENTATION ISSUES WITH GAMES-BASED ASSESSMENT Nafisa Awwal, Claire Scoular, Patrick Griffin Melbourne Graduate School of Education, The University of Melbourne (AUSTRALIA)

Abstract Evidence gained through assessment data informs us about what a student is ready to learn and how teachers can then scaffold their learning. Games are gaining momentum in the field of assessment for evaluating student performance and other complex 21st century skills which are often difficult to measure using traditional assessment design. Changes in organizational infrastructure have stressed the importance on enhancing critical-thinking skills through the use of collaborative learning environments; creation of which is now possible due to the advances in technology. These environments can be tailored to provide learning activities and the opportunity to collaborate, problem solve and apply critical thinking skills. With the addition of networks, they become multi-user environment, allowing multiple perspectives, argumentation and collaborative decision making TM1 behaviours. The Assessment and Teaching of 21st Century Skills (ATC21S ) project designed digital game-like collaborative problem solving assessments for measuring cognitive and non-cognitive 21st century skills. Construction of such complex and interactive game-based assessment tasks can instigate logistical and pedagogical challenges for both the developers and the potential users of such technology. Development of games is not only challenging, but also expensive, time-consuming, resource-demanding process. Issues in game construction are manifold and can arise anytime during the period of conception to implementation – specifically games intended for learning or assessments. This is partly attributed to the specialised design and technical skills required for such development undertakings. Given the pitfalls associated with the development of games, appropriate awareness needs to be created among individuals involved in such effort. This paper will highlight the experiences and reflect recommendations of the development team as observed during the deployment of the collaborative problem solving assessment design and delivery endeavour. Keywords: Game-based, implementation.

1

assessment,

problem

solving,

technology,

game

design,

issues,

INTRODUCTION

Evidence collected during assessment notifies us on what a student is ready to learn and how teachers can then scaffold their learning. Changes in organizational infrastructure have emphasised the importance on enhancing critical-thinking, problem solving, creativity and collaborative skills through learning environments – a primary goal of technology education. The advances in technology can potentially support such creations. Following the new technological opportunities created by the new digital era, the popularity of games has risen. Game is one avenue that can combine simulation, pedagogy to create active and engaging learning platform [1]. These environments can be tailored to potentially track assessment and learning activities that promotes collaboration, problem solving and critical thinking skills. With the addition of networks, they become multi-user environment, allowing multiple perspectives, argumentation and collaborative decision making behaviours. Games and other immersive virtual environments are gaining momentum in the field of education and training. Games are acknowledged by educators as assessment tool for evaluating student performance on complex 21st century skills which are often difficult to measure using traditional assessments. The Assessment and Teaching of 21st Century Skills (ATC21S) project proposed ways of assessing 21st century skills to prepare individuals to be successful in the workforce an as global citizens. During this study digital game-like collaborative problem solving assessments were designed for measuring some of cognitive and non-cognitive 21st century skills. Development of games is challenging, expensive, timeconsuming and resource-demanding process. Issues in game construction are manifold and can arise anytime during the period of conception to implementation – specifically games intended for learning or assessments. This is partly attributed to the specialised design and technical skills required for such development undertakings. Given the difficulties associated with games development, individuals involved in such effort need to be more aware of the associated pitfalls related to such endeavour. 1

The acronym ATC21S paper as ATC21S.

TM

has been globally trademarked. For purposes of simplicity the acronym is presented throughout this

Proceedings of EDULEARN16 Conference 4th-6th July 2016, Barcelona, Spain

0905

ISBN: 978-84-608-8860-4

This paper will discuss the obstacles and reflect recommendations of the ATC21S development team experienced during the implementation of the collaborative problem solving assessment design and delivery. The descriptions are measures that are suggested as good practice to improve the design and deployment of such assessments delivered online.

2

DATA DESIGN CONCERNS

A major advantage of gamed based assessment is the amount of rich information they are able to generate from the participants while engaged in the game environment. The process data (processes by which students reach their answers) derived through such assessments are considered richer than traditional data as they can describe the type, order and quantity of interactions with a task [2, 3]. This form of data is essentially collected through the capture of discrete mouse and keyboard events in a process discussed in more detail by Adams, Vista, Scoular, Awwal, & Griffin [4]. Design of the session logs (systematic records of user–game interactions) is thus crucial, can provide salient solution processes in an unobtrusive way [2, 5]. Many researchers have highlighted the importance of leveraging complex games not only to implement assessment delivery but also to establish automated scoring [6, 7]. The format in which data are captured ensures efficient interpretation of participant responses for establishing reliable scoring rules. To validate the scoring rules, it is vital that session log files are structured to allow human interpretation without obscuring their understanding of the activity that occurred during gameplay. For example, each participant’s response should be recorded as separate attributes as single instances with corresponding user identification, present state, timestamp, record index and other data as deemed necessary by researchers for the task analysis. In addition, it is imperative to ensure the optimum level of detail is captured for both analysis and processing of data for automating the scoring process. Timestamping is essential for logging response data from the assessment games. Timed data, along with a form of record indexing, can be useful in sequencing user interactions with the game environment. In the mentioned study, database design allowed to capture responses only in corresponding seconds. From the accumulated data logs, it was observed that more precise times (i.e. milliseconds) of user responses are often required to differentiate sequences of events that occur almost simultaneously. Multiple actions can be recorded as occurring at the same time when recorded in seconds, but in games activities do occur consecutively and this should be more accurately reflected in the way responses are captured and arranged to ensure that more accurate interpretations of events occurrence. Regarding the structure of the data capture the team acknowledged that milliseconds are required to be as intricate and accurate as possible in identifying behaviours and the sequences in which they occur [4]. It is a good practice to capture all possible activities that is allowed within a game environment. If all user interactions are captured and analysed, it can provide meaningful understanding of participant’s salient response processes and allow measuring of any underlying 21st century skillset that are likely to emerge through analysis of the session logs. In short, any clicks or activities on screen by users should be captured regardless of their effectiveness, because invalid, ineffective and tentative actions may prove to be more informative in later analysis [8]. The session logs generated from games can become very large if most information are recorded – which is why often it can be seen that developers choose in during design phase what to capture and how. But it is important that to information if left unrecorded as it may not be apparent to either the developer or the researcher what patterns of activity will elicit certain behaviour or confirm if tasks within games can elicit certain behaviours among participants. One way to deal with such large datasets is by visualising behavioural paths and patterns as described by Vista, Awwal & Care [9] to link it to cognitive processes involved while working with complex games. It may be impractical instead of trying to detect patterns using conventional analysis method. Event types described across different games should be defined in a uniform method. Consistency in event definition is important for future developers and researchers for understanding the events they represent. In the study, the assessment games were initially designed by multiple developers. As a result, the language and format used to define the same event – for instance ‘chat’ – were quite different and had different naming conventions across the various games (e.g. ‘Send message’, ‘Type message’, ‘Enter text’ etc.). Development of games and the capacity to automatically score responses collected for assessment purposes is a resource intensive undertaking, even in traditional and welldefined educational domains [10]. Due consideration should be given to future analysis needs and intended use of the data to support inferences while designing complex assessments of this nature.

0906

3

GAME & GAME ENGINE ISSUES

Although in its infancy, games are now used as assessments to capture information-rich student performances. The technological requirements of such assessments are those for the capture, storage and processing of time-stamped click events. Various forms of client and server configurations can be used for this purpose. For the study, the graphical components and other elements of the games were 2 3 4 designed using Flash with ActionScript 3 . In addition, SmartFoxServer 2X and Flash Media 5 Interactive Server , considered a standard choice for streaming media and shared object synchronisation for swf-based clients, were used as the socket server technology to support sharing of communications and synchronisation of shared objects between the multiple users. The use of Flash limited the range of devices, such as tablets with iOS that could be used for the games. Moreover, to access the tasks at a client browser with relatively up-to-date Flash Player plug-in ws required. It is important to consider the appropriate choice of technology, as it may limit the reach of such development only to a niche audience rather than a larger population, which in the context of education can have important consequences. Games need to be designed wisely. Not all actions or click in a game environment triggers changes at a given time. For example, participant responses including actions that trigger a change to the game on screen are clicks on buttons, text inputs etc. and events that don’t trigger any change are clicking on buttons which are not enabled. Developers of such game need to plan out stages on when to make which activities possible or active and what are possible resultant outcomes per user action to enable certain user behaviour to produce. Game rules need to be devised with assessment consideration in mind. In addition, the games for the study were designed to be collaborative i.e. not to be played as single player. The games being collaborative, no user is allowed to move forward without corresponding partner. In these games, if at any point one partner’s session stops, the other partner are not allowed to proceed to a new stage. It is essential to preserve reasoning coherence in the game design for maintaining the game flow and its intended assessment purpose [11]. The use of internet have become quite widespread, but often educational institutions do not experience its true benefits due to connectivity issues and bandwidth. Considering the fact that schools are the potential users of such development, the games for the study were designed to deal with unexpected interruptions due to internet connection failures or other technical issues. In the cause of such events, it allowed students to continue access from their previous responses that is last state of activities. During a collaborative session, if one student of the pair encounters a technical problem or closes their window (or hits refresh/back button on their browser) by mistake, the games allowed both users to re-enter to the page on which they were last collaborating. In addition, the partner is provided with a system notification of partner loss if their session stops at any time. If a partner closes or refreshes their browser window, the games will detect this activity and inform the other partner of it, as long as the network connection remains active. This is an important consideration to be noted during any computer based assessment development. It can become confusing for students (as students collaborating in a session may not be in the same physical location) if they ae unable to know what is the status of their progress in such unknown environment and frustrating if they have to keep coming back to do the same thing repeatedly, thus perhaps losing motivation and engagement with the overall task. Responses received in this form can be considered contaminated as it will not give accurate information about the student’s performance of what they can do. The aim of any assessment development is to reduce student frustration and anxiety and other type of noise when collecting data to evaluate student performances accurately.

2

Flash is an Adobe’s multimedia product used for creating vector graphics, animation, games (online, desktop, mobile) and various other rich applications (online, desktop, mobile).

3

ActionScript is an object-oriented programming language, used primarily for the development of applications developed using Adobe Flash product.

4

SmartFoxServer is a middleware for creating multi-user applications, communities and games with Adobe Flash/Flex/Air, HTML5, Android, Unity3D, Apple iOS, Java etc. 2X refers to its community edition.

5

Flash Media Interactive Server (FMIS) is an Adobe product that works with the Flash Player runtime to create media driven, multiuser RIAs (Rich Internet Applications). It uses Real Time Messaging Protocol (RTMP) to connect to Flash Media Server Hub to synchronize complicated data structures and call remote methods on multiple clients in one go by having clients subscribe to a shared object.

0907

4

PLATFORM & DELIVERY ISSUES

This section will highlight some of the issues identified with the platforms designed for this study and provide recommendations for design choices on technologies to improve on the lessons learned through previous deployment. Multiple platforms co-existed for delivering the range of assessment games that were developed for the project. These platforms with differing technological requirements often made it difficult to deliver the example games to a wide range of schools and students. It was realised that existing know-how needs to be integrated with emerging technologies to consolidate multiple portals, automated scoring of responses and feedback mechanism to teachers. Schools are often limited in IT support and knowledge, which is all the more reason to create and deliver systems in the most accessible form to schools. Key considerations for such development should be an integrated system that is able to cater for large numbers of simultaneous user access, dispersed location of users and devices in use, the multi-lingual content capability (if required) and results of user attempts to feedback in real-time. The aim of such feedback mechanism is to allow teachers to monitor student progress over time and link that progress to successful teaching strategies [12]. Cloud based technologies are relatively new for assessments, but other e-business (e.g. EBay, Amazon etc.) have commercialised their development efforts into services. This form of technology can be used as a good alternative source for games delivery to alleviate some of the issues that may 6 have been imposed for previously exercising Flash technology. The use of open standards (e.g. W3C standards) can be adhered easily, making it a cost effective solution for such deployment. In addition, such technologies offer a variety of practices, to ensure that system design can be scaled to support a large number of users generating 10-40 time more transaction data compared to traditional online test items Games in general produces more operation between client and server due to the nature of the interaction from answering these collaborative games. In such environments when an action is 7 completed by a user, the game send a message through web socket to the server, and the server then sends that message to the other client to inform them of the action that has been attempted [13]. Hence, the platform architecture and game engine warrants thorough considerations during the design phase with the aim to be able to handle heavy loads and traffics. This ensures that the real time synchrony be as fast and fluid as possible. A key criterion for the study was to develop interactive assessment games that use synchronous 8 communications to provide support a collaborative process. HTML5 has been gaining momentum over Flash in creating such applications. Unlike Flash, HTML5 is supported by all devices (including iOS devices). In addition, it is not constrained by licensing limits for the use of additional server based software which is required for the scalability of accommodating synchronous communication among large number of simultaneous users. Such technological options are viable to be supported financially for deployment in schools later. Within the standards of HTML5, web sockets also provide a means for building collaborative services to support synchronous communication among collaborating users by the use of only a standard web browser. It has the capacity to track and record data in a form that allows later analysis and reporting effective. Employment of these technologies is likely to provide a consistent experience for users across all browsers and platforms. However, a disadvantage of these design choices is that older browser versions do not adequately support the HTML5 standards. While some schools may still be using older browsers, the freely accessible latest browser versions will improve this problem and it is expected that schools will upgrade to current HTML5 browsers over the coming years.

5

ASSESSMENT DESIGN CONCERNS

To gather data from games for assessment and evaluation, clear understanding of the design characteristics is imperative, both in the characteristics of the games and the structure of the data capture. Perhaps the most efficient way of designing games for assessment is to have a generic set of events that can be captured and a specific set of characteristics from which game design can be based. Lacking these specifications leads to inefficiencies in relation to both cost and time.

6

W3C stands for World Wide Web Consortium, which is the main international standards organization for the web.

7

Web Socket is a protocol providing bi-directional communication channels over a single TCP connection between a browser and the server, facilitating the real-time data transfer from and to the server.

8

HTML5 is a markup language used for structuring and presenting content on the web, including detailed processing models to encourage more interoperable implementations.

0908

The components of ATC21S assessments, including the log stream data structure and scoring procedures, are game dependent. This dependency made initial game design laborious and intricate and limits its capacity to incorporate new games easily and efficiently. Presentation of the log stream for capturing data is crucial, particularly in the streamlining of scoring procedures. As mentioned earlier, most of the games developed have similar data events but the naming convention under which these are recorded varies. For example, in the Olive Oil game communicating in the chat box is labelled ‘chat’ but in the Plant Growth game it is labelled ‘send message’, the reason for this in part was due to different designers developing different games [14]. While both events are in essence the same, naming them differently has led to complications and additional work during the scoring procedures. This may inflict difficulty for future designers and developers in creating new assessment games. Additional considerations for scoring procedures arose in the study due to the variability of the games. The games range from one single page to eight pages in length, with time varying on average between 5-45 minutes for completion. No one game is expected to capture all of the behaviours and greater variability between games is generally better for capturing a wide range of behaviours. However, in this scenario shorter, fewer page games were in some cases inefficient at capturing much due to the data events associated with them. In some cases, a game was too short to include many game characteristics and therefore, little or no opportunity was presented to students in which they could demonstrate particular skills or behaviours. The construction of games intended for assessment and learning purposes can be difficult. This is often because the goals of the games do not always align with the goals of assessment or learning, thus distracting individuals with in the game play process with dramatic game features. For games to be effective evaluation tools, they need to offer ample opportunities to participants to create, acquire, rehearse and apply skills and knowledge through active exploration within the collaborative virtual space [1]. Poor game design may involve tasks which are centred on simple “drill and practice” models and less on increasing students' progress [15]. Appropriate game design is likely to promote engagement including exploration and individual’s abilities to reflect on their experience to exhibit their cognitive performance when playing games collaboratively [16]. Together with good game characteristics and assessment principles games are able to engage and motivate individuals to actively and critically experience, practice and reflect on their ideas in a problem based context – providing educators a means to evaluate student performance on complex cognitive and non-cognitive 21st century skills. However, game design can often be challenged to successfully balance the contribution and engagement from players in such environments [17]. In terms of development, it is difficult to design challenging but impartial games that require collaboration, while at the same time preventing all single player solutions.

6

CONCLUSION

Most games that have been developed were not designed for educational purposes as their primary goal or their most important design feature. Games are in their early stage of evolution. With experience our implementation and adaptation of games will become increasingly more sophisticated. Using games in education will require “unlearning” many conventional beliefs, assumptions, and values about teaching and learning, as well as the structure of education [18]. Construction of complex and interactive game-based assessment can instigate logistical and pedagogical challenges for both the developers and the potential users of such technology. Schools are the potential users of such development work. But unfortunately many schools are still restricted in their access on state of the art technology including software, hardware, internet access and bandwidth. This gap is more profound in rural areas and locations where students are not functioning in a technology-rich environment. Schools are likely become equipped with better technology in the coming years as the need for teaching the 21st century skills are realised. As new technologies and standards continue to evolve, and with evolving educational needs and opportunities the requirement to explore to extend such undertakings is likely to expand and unfold in the coming decades. Given the pitfalls associated with the development of games for educational purposes, appropriate awareness needs to be created among individuals involved in such effort. Research and development of games needs to be aligned primarily with the educational goals, with special attention to issues that can surface during such development endeavour. Any such undertakings should also focus on the potential users of this form of implementation - therefore design and technology choices for deployment may need to consider implications of its users and usage.

0909

REFERENCES [1]

Awwal N, Alom M, Care E. Game Design for Learning to Solve Problems in a Collaborative Environment. Proceedings of the European Conference on Game Based Learning 2016: 25-33.

[2]

Bennett RE, Jenkins F, Persky H, Weiss A. Assessing complex problem solving performances. Assessment in Education 2003; 10: 347-59.

[3]

Greiff S, Wüstenberg S, Funke J. Dynamic problem solving : A new assessment perspective. Applied Psychological Measurement 2012; 36: 189-213.

[4]

Adams RJ, Vista A, Awwal N, et al. Automatic Coding Procedures. In: Care E, Griffin P, Wilson M, eds. Assessment and Teaching of 21st Century Skills Volume 2 - Methods & Approaches. Dordrecht: Springer, 2015: in press.

[5]

Zoanetti NP. Interactive computer based assessment tasks: How problem-solving process data can inform instruction. Australasian Journal of Educational Technology 2010; 26: 585-606.

[6]

Mills CN, Potenza MT, Fremer JJ, Ward WC. Computer-based testing: Building the foundation for future assessments. New Jersey: Laurence Erlbaum Associates, Inc., 2002.

[7]

Williamson DM, Mislevy RJ, Bejar II. Automated scoring of complex tasks in computer-based testing. New Jersey: Lauwrence Erlbaum Associates, Inc., 2006.

[8]

Awwal N, Griffin, P. & Scalise, S. Platforms for delivery of collaborative tasks. In: Griffin PC, E., ed. Assessment and teaching of 21st century skills: Methods & approach. Dordrecht: Springer, 2015: 105-13.

[9]

Vista A, Awwal N, Care E. Sequential actions as markers of behavioural and cognitive processes: Extracting empirical pathways from data streams of complex tasks. Computers & Education 2016.

[10]

Masters J. Automated scoring of an interactive geometry item: A proof-of-concept. The Journal of Technology, Learning, and Assessment 2010; 8: 4-28.

[11]

Csapó B, Ainley J, Bennett RE, et al. Technological Issues for Computer-Based Assessment. Assessment & Teaching of 21st Century Skills 2012: 143.

[12]

Griffin P. ARC Learning Profiles: Users Manual. Melbourne2000.

[13]

Alom M, Awwal N, Scoular C. Technology Integration in Multiplayer Game Design. Proceedings of the European Conference on Game Based Learning 2016: 10-4.

[14]

Care E, Griffin P, Scoular C, et al. Collaborative problem solving tasks. In: Griffin P, Care E, eds. Assessment and teaching of 21st century skills: Methods and approach. Dordrecht: Springer, 2015: 85-104.

[15]

Squire K, Jenkins H, Holland W, et al. Design Principles of Next-Generation Digital Gaming for Education. Educational Technology 2003; 43: 17-23.

[16]

Moreno-Ger P, Burgos D, Martínez-Ortiz I, et al. Educational game design for online education. Computers in Human Behavior 2008; 24: 2530-40.

[17]

Manninen T. Towards Communicative, Collaborative and Constructive Multi-Player Games. In: Mayra F, ed.: Tampere, Tampere University Press, 2002: 155-70.

[18]

Tsekleves E, Cosmas J, Aggoun A. Benefits, Barriers and Guideline Recommendations for the Implementation of Serious Games in Education for Stakeholders and Policymakers. British Journal of Educational Technology 2016; 47: 164-83.

0910