A design science research methodology for developing a computer ...

A design science research methodology for developing a computer-aided assessment approach using method marking concept Hussein Genemo, Shah Jahan Miah & Alasdair McAndrew

Education and Information Technologies The Official Journal of the IFIP Technical Committee on Education ISSN 1360-2357 Educ Inf Technol DOI 10.1007/s10639-015-9417-1

1 23

Your article is protected by copyright and all rights are held exclusively by Springer Science +Business Media New York. This e-offprint is for personal use only and shall not be selfarchived in electronic repositories. If you wish to self-archive your article, please use the accepted manuscript version for posting on your own website. You may further deposit the accepted manuscript version in any repository, provided it is only made publicly available 12 months after official publication or later and provided acknowledgement is given to the original source of publication and a link is inserted to the published article on Springer's website. The link must be accompanied by the following text: "The final publication is available at link.springer.com”.

1 23

Author's personal copy Educ Inf Technol DOI 10.1007/s10639-015-9417-1

A design science research methodology for developing a computer-aided assessment approach using method marking concept Hussein Genemo 1 & Shah Jahan Miah 2 & Alasdair McAndrew 1

# Springer Science+Business Media New York 2015

Abstract Assessment has been defined as an authentic method that plays an important role in evaluating students’ learning attitude in acquiring lifelong knowledge. Traditional methods of assessment including the Computer-Aided Assessment (CAA) for mathematics show limited ability to assess students’ full work unless multi-step questions are sub-divided into sub questions. This issue persisted significant drawback especially within the notion of method marking approach. To address this issue, the aim of the study is to develop a methodological framework that will create an information and communications technology (ICT) artefact prototype. The prototype (termed as method marking assessment (MMA) artefact) implements a method-marking assessment concept to assess through multi-step questions. Extensive literature reviews have revealed that there are features in common between complex-problem solution characteristics and multi-steps questions assessment using ICT; therefore complex problems paradigm is used in the study for developing the MMA prototype. Keywords Computer aided assessment methods . Design science research . Expert systems . Method marking

* Shah Jahan Miah shah.miah@vu.edu.au Hussein Genemo hussein.genemo@students.vu.edu.au Alasdair McAndrew Alasdair.McAndrew@vu.edu.au 1

College of Engineering and Sciences, Victoria University, Footscray Park Campus, Melbourne, Australia

2

College of Business, Victoria University, Footscray Park Campus, Melbourne, Australia

Author's personal copy Educ Inf Technol

1 Introduction Assessment is used to examine individuals’ knowledge in a particular domain so actions could be taken based on the outcome of the assessment. An effective assessment tool is required to implement various concepts such as method marking in assessing the question type such as multi-step questions (MSQ). Implementation of concepts plays significant role in guiding learners towards their learning which results in acquiring lifelong knowledge (Bloxham and Boyd 2007). The concept of method marking is the notion that provides awarding learners work, based on the techniques that are used in solving MSQ, which required more than one step to produce an answer (Lawson 2012). Using traditional way of awarding learners’ works, the method marking performs through a manual assessment process operated by human assessors. The key question is how this manual process can be automated using ICT. This type of problems has been solved through breaking MSQ into sub questions that require only one step to produce a step answer. However this action interferes with the authenticity of the assessment that may produce low quality of awarding (Lawson 2012). Previous studies suggest that the approaches are determined by the nature of assessment that accompanies the learning process. Although various ICT approaches are used to deliver teaching with flexibility and ease of use, implementing ICT assessment tool for method marking to effectively assess MSQ is still with substantial design challenges such as on how to conceptualise the solution artefact specially to meet the assessors’ work process and required complexity of the issue. This is the key concern of the paper. This paper proposes a research approach that will guide to develop an artefact prototype. The conceptual paper is structured as follows: Section 2 describes background details of the study. The section defines aspects such as educational assessment, assessment authenticity, purpose of assessment and assessment tools in producing quality assessment outcome. The background focuses on the problem of ICT tools awarding learners for their full works and methods applied to produce final answers, especially in assessing MSQs that are considered complex when ICT tool is assigned to assess. The section also identifies characters shared between complex problems and complex problems solutions on one hand and MSQs assessment using ICT assessment tool on the other hand. The section also sheds some lights on expert systems and design research paradigm. Section 3 deals with the method marking assessment (MMA) conceptual framework that will be used as the basis in order to develop MMA artefact. This section also provides the structure of the framework components and the each tasks of the component. Section 4 discusses about the paradigm, methodology and methods to be used to study examination solutions papers (ESP), develop, design, and build MMA artefact. Finally the Section 5 concludes this conceptual paper by summarising the study.

2 Background Assessment can be viewed as the main element in teaching and learning process. For the learners the assessment is the most significant activities that influence their approach to learning (Price et al. 2011; Sangwin 2012). There are three well known types of assessment (namely summative, formative and diagnostic) in education;


however the focus of the study is limited to formative assessment that is used to check current learners’ status; where they aim to head and how to reach their targets (Wiliam 2011). Assessment also directs the teaching activities (Gears 2005) and contributes to the learner’ knowledge (Shepard 2005). Learners who are given successful formative assessment tasks have also shown high achievement in summative assessment results (Balcombe et al. 2011; Black and Wiliam 2005; Gears 2005). Among many other issues, assessment designers should emphasise on assessment authenticity and issues associated to make an assessment as authentic as possible due to its purpose (Palm 2008). Some of the issues are related to curriculum alignment that is about measure important skills which are considered Bintellectual quality^. However, the issue of what to assess differs from a discipline to a discipline. For example the definition of assessment authenticity in mathematics is based on the characteristics of the specific discipline associated with mathematics (Palm 2008). The assessments method hold potential to automate that could reduce the use of pencil and paper. Computer aided assessment for mathematics is limited only to checking a final answer for a MSQ (Beevers et al. 1999; Livne et al. 2007; Sangwin et al. 2010). The only solution to this problem was applied for partial credit method which can award marks for a partially correct portion of the solution even though the final answer may be wrong (Ashton et al. 2006; Beevers et al. 1999; Lawson 2012). However, the software packages for mathematical assessment, have been developed are to implement the partial credit concept. These software are used to design through questions, collected answers and assessed them in the same way (Beevers et al. 1999; Jones 2008; Livne et al. 2007; Passmore et al. 2011; Sangwin 2012). In these cases, to implement partial credit concept, MSQs have been redesigned by breaking them into sub questions that require only one step to solve thereby enabling partial credit concept implementation. The idea of splitting multi-step questions raises problems with the assessment authenticity of these type of questions due to the following facts:(1) the choice and use of algorithms to solve the question are not in themselves tested, because methods are shown to the students and (2) a particular step is forced upon the students to follow, which might be new or hard to use (Lawson 2012). In method marking, what assessed are not only the final and correct answer of a question, rather, methods and works shown to produce answers. As indicated above, since using ICT tools in implementing MMC in assessing MSQ is a challenging process, it is vital to investigate complex problems’ natures and approaches to solve them in order to determine the relationships between method marking assessment using ICT tool and complex problems features. Accoridng to (Fischer et al. 2012) Btrying to make a modern computer do what it is supposed to turn out to be a complex problem as well as changing certain settings of an unknown mobile phone device^ (pp.36). Wüstenberg et al. (2012) specify the attributes of complex problem as challenging problem for the problem solvers. The authors describe the nature of problem as: lacking information to start solving the problem, requiring appropriate approaches to create information, demanding for procedural ability to be in charge of the system. Difficult problems require better understanding of problems to deal with them. According to the study (Fischer et al. 2012), what makes a problem complex is, generally, its Bstructure of the external problem representation^ regardless of the problem solver’s capability to solve them.


Fischer et al. (2012) denote that the complexity of a problem can be measured by the number of elements or variables in the problem or the problem’s solution and the interdependency between the elements. Hevner et al. (2004) also identify the relationships between elements in the solution of complex problem as one of the features of wicked problem characteristics that are similar to the complex problem features. The wicked problem are defined through the following properties: definitive formulation of the problem does not exist; complex interactions occur between the problem and solution domains; there exists an inherent flexibility to change design processes as well as design artefacts; solution is critically dependent upon human cognitive ability and social abilities (Hevner et al. 2004). In method marking, the difficult aspect is to uncover intentions of problem solvers. For example, during assessment using ICT, it is hard to detect the number of steps, order of steps, or steps mentally computed or steps skipped in solving MSQs. Components of complex problem solution possess characteristics such as difficulty, interdependency, vigour, unspecified goal structure and demands for weighing and coordinating goals components or facets (Funke 2010). The actions that involve MSQ assessment by ICT tool are comparable to the complex problem or wicked problem solution procedures. Complex problem solving (CPS) is a process where individuals approach, manage and address ongoing issues that are highly unpredictable (Yeo and Marquardt 2012). According to Sonnleitner et al. (2013) CPS paradigm is composed of: rule identification, rule knowledge and rule application. The authors claim that knowledge amassing plays the most important part to master complex problems by learners. Funke (2010) classifies a problem solving question as the ability to produce a target state by conquering impediments that stop in the way; complex problem solving demands processes that require more thinking efforts and actions to solve a problem. Similar view is expressed by the view of Wüstenberg et al. (2012). The authors refer to the problem solvers’ skills as observation of problems’ and solutions’ characters; they also add knowledge acquisition to the characters observation as one of the means to achieve a target in a problem solving process. They explain CPS tasks as the activities that involve uncovering new information by applying successive interrelated steps. 2.1 Design science research Design science research (DSR) has been an effective methodology for developing ways of solving complex issues through innovating solution artefact. Fischer et al. (2012) and Hevner et al. (2004) express the nature of solving a complex problem as searching for an activity or set of activities to achieve the desired goal. These authors agree on identifying, acquiring and utilizing knowledge to find acceptable solution for a problem. Studying the phenomena of complex problem where method marking assessment practiced in, to identify the assessment process manually, and observing the results of assessment process help to understand the phenomena. Knowledge is acquired from new data and information that are extracted from findings in the phenomena that is investigated, domain knowledge, previous literatures and experts in fields. The acquired knowledge will be used in understanding the problem more and presenting multi solutions that can be utilized in building MMA artefact prototypes. DSR has now better positioned as a research paradigm in information systems in order to provide new way of thinking on what makes information systems design


research relevant to practitioners. Research in design has a history in many field including education, engineering and other disciplines (Cross 1984). Bayazit (2004) suggests that DSR provides a systematic inquiry whose goal is knowledge of or in the embodiment of configuration, composition, structure, purpose, value and meaning in developing artificial objects and systems. Faste and Faste (2012) identify four basic categories of DSR to clarify the differing objectives of designers across the disciplines such as: (a) design through research, wherein researchers perform activities that would conventionally be considered research—regardless of their awareness that their activities are design (b) design of research, the activities routinely performed by researchers to plan and evaluate their experimental designs (c) research on design, wherein researchers interested in improving design practice by examining it, often by studying design practitioners at work or manipulating experimental variables to influence design outcomes, thereby revealing relevant design process; and (d) research through design, wherein designers design things Bas usual^ but consider their results research because, in addition to shaping tangible outcomes, they have learned something new about their practice. The aim of the study meets the category (d) in that we consider the result of research that will provide a solution prototype as tangible outcome from which we gather new knowledge that will improve practices of doing activities such as marking. 2.2 Artefact details The intended artifact to address the assessment issues will be based on an expert system approach. According to the theory of artificial intelligence, knowledge is divided into declarative and procedural types (Jaques et al. 2013). The knowledge that is required in the research is the procedural knowledge which is considered as an act of performing tasks implemented by the production rule represented in the form of IF-THEN statement (Jaques et al. 2013; Kamel Boulos 2012; Angeli 2010). The act of representing knowledge in this form fits the description of the rule-based system that will be used to implement the proof-of-concept prototype solution. It is important to understand that mathematical content is procedural in nature; therefore, it is easy to formalize it into computer algorithms (Jaques et al. 2013). Method marking concept will be implemented in an expert system as a form of an artefact instantiation prototype. An expert system is a system that acts as a specialist in a domain to resolve real-world problems which typically need a human skill (Grosan and Abraham 2011; Kamel Boulos 2012). Expert systems are utilized in predicting consequences from given raw data and diagnosing fault using observed information. This kind of systems can play very important roles in many fields as decision support system to aid medical practitioners in taking proper decisions (Kamel Boulos 2012; Kaidar et al. 2013), as advisors to educators (Hwang et al. 2011) and farmers (Khan et al. 2008), in mathematics knowledge assessment, especially in tutoring (Jaques et al. 2013), as diagnosing plants disease (Khan et al. 2008). In this research, DSR approach will be used in developing the artefact. DSR involves many steps to produce artefacts in the form of construct, model, method and instantiation (Hevner et al. 2004). Hevner’s seven design science research guidelines as well as (Vaishnavi and Kuechler 2007) design process approach will be used in directing the artefact construction part of the research. According to (Hevner et al. 2004), the direction and recommendations to follow their guidelines are important for reasons:


Innovations, such as database management systems, high-level languages, personal computers, software components, intelligent agents, object technology, the Internet, and the World Wide Web, have had dramatic impacts on the way in which information systems are conceived, designed, implemented, and managed. In summary, method marking assessment using ICT is a complex problem that requires complex problem solving and design science research approaches to produce artefacts that are purposeful in resolving the issue of MSQs assessment to implement method marking concept.

3 Proposed method marking assessment concept framework It is anticipated that Method Marking Assessment (MMA) concept framework will provide a basis to begin research in the development and construction of the artefact. Previous literature reveals the importance of assessment, in education, and the use of ICT as the assessment tool in advancing learners’ educational life and beyond. As discussed in the background section, assessing MSQ using ICT tools is considered as complex problem. Properly structured framework, to initiate design science research, plays an important role in directing the development, building, evaluating and concluding of purposeful artefacts. A framework structure is part of a good research planning, that leads to selecting suitable research methods and language instruments (Österle et al. 2011). Visual representation of actions or concepts also play important roles in understanding and solving problems (de Jong 2014). Kirsh (2010) emphasise this by saying: In a closed world, consisting of a person and an external representation—a diagram, illustration, spoken instruction, or written problem statement—why do people do more than just think in their heads? The framework shown in Fig. 1 is composed of environment, design science research and knowledge base components. The descriptions of the components are explained in the sub-section below. 3.1 Environment components An environment is the source of research questions that the research investigates to provide acceptable and relevant solutions (Hevner et al. 2004) and it is made up of components. In this research the components are constituted from people who are represented as assessors and problem solvers, assessment processes, examination solutions papers (ESP), evaluation criteria and proposed method marking artefact prototype. Vaishnavi and Kuechler (2007) also explain environments as the sources of research problems suppliers and they call this Bawareness of problem^. Stakeholders are the sources of IS research in terms of providing research questions and other necessary resources (Österle et al. 2011) and it is impossible to separate a phenomena to be studied from its environment (Pereira et al. 2013).

Author's personal copy Educ Inf Technol Knowledge Base Environment Assessment process Educators/ Assessor Problem Solver/Learner Examination Paper Evaluation Criteria MMA Artefacts

Foundation Method Marking Assessment Research (Design Science Research) Develop/Build Phenomena Study and Analysis MMA Concept Justify/Evaluate Analytical Testing

Pragmatist Complex problem paradigm Design Science Paradigm Mathematical Knowledge Experts and experiences Assessment theories Expert Systems (existing artefact) Expert System Shells (existing artefact) MMC artefact Knowledge Methodologies Experimentation Statistical Analysis Mixed Methods

Fig. 1 Proposed MMA research framework (adapted from Hevner et al. 2004)

According to Hevner (2007) the episodes in the design science research are linked by the relevance cycle to the research scheme that is produced by the contextual environment. In this research, ICT implementation problem in assessing MSQs provides opportunities to DSR researchers to investigate the examination papers’ solutions and use the result as the base for data to develop and build method marking assessment artefact. Hevner states Bthe output from the design science research must be returned into the environment for study and evaluation in the application domain^ (2007, pp 89). In our research, the MMC assessment artefact will be tested in the environment which is assessing method marking algebraic questions. The criteria for the MMA artefact evaluation will be established by the environment. 3.2 Design science research component This block is made up of develop/build and justify/evaluation parts. It is linked to the knowledge base part of the MMC conceptual framework through the rigor cycle. Hevner et al. (2004) identify that IS research requires two complementary phases to represent the business needs. They specify that behavioural science is required to communicate the truth and it is used to develop and justify theories that detail or predict the phenomena related to the identified business needs. In MMA concept research, the purpose is to produce an artefact that assesses MSQs. However the job of the behavioural science is to explore the ESP to uncover errors and solutions strategies. In design science research, data collection and empirical analysis methods are guided by methodologies that are originated in the methodology part of the knowledge base block (Hevner et al. 2004). In the MMA concept framework illustrated in Fig. 1, the phenomena study and analysis element represents space where the


research problem is investigated to produce the data required to design the MMA artefacts. Mixed methods methodology supplied by the knowledge base part will be used to study and analyse the phenomena. The initial MMA concept artefact will be the construct artefact followed by the framework of March and Smith (1995) in that classes of artefacts are model, method and instantiation. The artefacts will be made in sequence as the MMA concept research progresses and challenges are overcome through the DSR methodology. The process of building artefacts involves repetition actions to solve research questions till the desired artefacts are built; these are challenging tasks that are designed to solve previously unresolved problems or to improve it significantly to claim the originality of the research. One of the features of the design science is repetitiveness and it is difficult to find out the ideal design for a particular IS problem solution (Hevner et al. 2004). The two methodologies, analytical and testing, which are driven from the elements of the methodology part of the knowledge base block, have been selected as the appropriate methodologies for evaluating the MMA artefacts prototypes. To achieve proper evaluation of a designed IT artefact, acceptable methods of measuring or weighing the artefact, collecting proper evaluation data and analysing it, is a fundamental evaluation process (Hevner et al. 2004). There are two types of analytical methods: optimization that is used to discover the optimal characteristics of the artefact and dynamic analysis that is utilized to explore the dynamic qualities of the artefact such as performance. In testing method, there are two testing methods which are used for functional that tests the artefact’s interfaces and structural that investigates the implementation of the method marking concept. These analysis and testing methods ensure the acceptability of the evaluation methods and finally the artefacts themselves as the proper design science research product. Since an artefact is one of the design science research products, its acceptance, in IS, is measured by the value it adds to the knowledge base and to the solution of the known problem and in this research it is implementing method marking concept using ICT tool. 3.3 Knowledge base This part of the MMC assessment framework that is composed of two main parts, which are foundations and methodologies. The two parts encompass elements that make each part. A proper selection and effective usage of parts of knowledge base components’ elements has influence on the design science research outcome, which could be published to give access to interesting parties. Basic and initial knowledge, to begin a design science research, come from knowledge base. It is important to choose suitable existing knowledge that is implemented efficiently to fulfil research rigor required and to conduct proper research. However in the absence of existing knowledge, the alternative and valuable sources or methods are imagination, ingenuity and trial-and-error. Once the suitable knowledge is applied to a design science research, it is necessary to perform the analysis of the chosen knowledge to find out its impacts in/on the research and the research outcome (Hevner et al. 2004). Publishing the research outcome empowers other researchers to reconstruct and/or implement the MMC assessment artefacts. It also enables researchers to extend the artefacts features and evaluate them. Reproducibility combines the concepts of replication, repeatability, validation and verification (Borgman 2012). The knowledge


gained from a design science research in IS can be added to the appropriate parts of the knowledge base as the contribution from the research. In current research, the noticeable contribution from MMC assessment research will be the MMC assessment artefact that will add knowledge to the knowledge base to be used as a source of using ICT in assessing questions with multi-step or complex problems using ICT in general. In summary, the importance of the interactions between the rigor and relevance cycles in producing purposeful artefact is stressed by Hevner (2007) as follow: BHowever, practical utility alone does not define good design science research. It is the synergy between relevance and rigor and the contributions along both the relevance cycle and the rigor cycle that define good design science research^ (pp. 91).

4 Adapted research view Research paradigms Binfluence the practice of research^ (Creswell 2003, pp3). BThe practical research needs combination of philosophical ideas, general procedures and detailed methods^. A suitable paradigm is required during all phases of the research project; these include planning, instruments development, data collection, data analysing and interpretation, data validation and reporting. Multiple paradigms can be applied to one research, due for the possibility of research components coming from different backgrounds or environments that need explanations and/or guidance. For example, this research attempts to use pragmatism, design science research, formative assessment theory and complex problem solving paradigms in order to drive the research practice. Pragmatism paradigm is suitable for this research for the reasons detailed next. In regard to Pragmatism paradigm, it gives freedom to researchers to select methods, techniques, and procedures of research Bthat best meet their needs and purposes^ (Creswell 2003) and it enables Bresearchers to look beyond the Bwhat^ and Bhow^ to research based on its intended consequences-where they want to go with it^ (Creswell 2003). Methodologies or strategies of inquiries (also known as traditions of inquiries) are general procedures to Bprovide specific direction for procedures in a research design^ (Creswell 2003) and they reflect the perspective of the paradigms they are allied with (Creswell 2003) although there are opinions that disagree with this view. The linkage between research paradigm and research methods is neither sacrosanct nor necessary (Johnson and Onwuegbuzie 2004). For example, qualitative researchers should be free to use quantitative methods, and quantitative researchers should be free to use qualitative methods. (Johnson and Onwuegbuzie 2004). This research will use mixed method methodology since this methodology is generally associated with pragmatism paradigm. In the mixed methods approach, the qualitative and quantitative strategies are used to collect and analyse data, using both qualitative and quantitative methods, in a single study (Johnson and Onwuegbuzie 2004; Creswell 2003).


In terms of method we have mainly two parts. First part of the method is to explain steps and methods of investigating student’s examinations solutions papers, analysing and validating the investigation results; in Fig. 1, the phenomena study and analysis element represent these facts. The second details the process of extracting information from the examination paper study output, which will be used as the requirements inputs to build and evaluate MMA concept constructs, models, methods and instantiations. This process is represented by the MMC artefacts element in Fig. 1. 4.1 Phenomena study and analysis The background where the research problem is originated is considered as a phenomena that Bcan be many things: a program, an event, an activity, a problem, or an individual (Pereira et al. 2013, pp 152). In the views of (Creswell 2003), specific methods are linked to collecting and analysing data. According to Creswell (2003), A research problem is an issue or concern that needs to be addressed, if the problem is identifying factors that influence an outcome, the utility of an intervention , or understanding the best predictors of outcome, then a quantitative approach is best^. The identified errors and question solving methods influence the way rules and algorithms are built to implement method marking concept in MMA artefacts. In studying and analysing the papers, the priority will be given to the quantitative approach for the reasons that have been viewed by Creswell (2003). These papers are formed from two similar first-year undergraduate mathematics units; therefore the outputs from the two examination papers solutions investigations are expected to be the same or similar. The details of studying and analysing approaches will be detailed in two phases: phenomena exploration and data analysis in phenomena. In the first phase, studying the phenomena and measuring its impact are commencing activities of research (Chatterjee 2010). It is very important to study the phenomena to find out problems associated with the phenomena before examining reasons for the existence of problems (Davis 2007). In our research, the EPSs are the main research material that provides opportunity for investigating the phenomena. The papers will be divided based on the semesters and subjects. One subject will be studied using qualitative method and the second subject will be studied using the quantitative method. Initially, all the solved questions will be scanned. Then appropriate questions with the rich information will be concentrated on. Since the research method chosen is concurrent triangulation, the two subjects will be studied at the same time; the questions that will be selected from both subjects will have common characteristics. The criteria that determine the importance of the questions to be studied will be guided by the previous findings in common algebraic errors research and current researcher’s domain knowledge. This will be achieved based on the rigorous research of the previous research. Once the criteria are set up, a rubric will be formed to collect the two main information which are errors and strategies that are used to solve the questions that will be studied. An act of data management prevents or limits number of errors and manipulations leading to better data integrity (Van den Broeck et al. 2013). Properly constructed rubric and usage of it assist in achieving these aims, because observations and measurements must be precisely recorded as possible (Chatterjee 2010).


In the second phase according to (Creswell 2003, pp.3), Bdata analysis in mixed methods research relates to the type of research strategy chosen for the procedures^. Data from the two subjects will be analysed using quantitative and qualitative approaches. Text and/or statistical analysis will be used to analyse the qualitative data. As for the quantitative data, statistical analysis will be used to determine the similarity or difference between years, to isolate causes and effects, and to eliminate outliers. At the interpretation phase, the collected qualitative data will be transformed into the quantitative data and compared with the quantitative data to establish criteria that will be used to build rules and algorithms for framework that will be implemented in expert systems using design science method (Miah et al. 2014, 2009). 4.2 MMC assessment artefacts development The MMA artefacts development starts from the problem investigation stage. At this stage the development will concentrate on building constructs and model artefacts. Once the development, building and evaluation of the two artefacts are established, the research will progress to the next artefacts - methods and instantiations- development and building. The validated data from the phenomena is used as the base to establish the requirements to develop, build and evaluate MMA artefacts. The first step in this process is to compile constructs. According to Hevner and Chatterjee (2010), using the recorded observation and measurement, researchers’ structure constructs. This section contains steps and approaches that will be followed to develop, build and evaluate the MMA artefacts. 4.2.1 MMC assessment constructs The task of constructs is to describe characteristics of a problem domain and allow the development of the research’s terminologies (Pereira et al. 2013). According to (March and Smith 1995), Constructs or concepts form the vocabulary of a domain. They constitute a conceptualization used to describe problems within the domain and to specify their solutions. They form the specialized language and shared knowledge of a discipline or sub-discipline. Constructs Bprovide the vocabulary and symbols used to define problems and solutions^ (Hevner et al. 2004, pp. 83). In method marking, the terminologies that will be used to come from literature review (published common errors), current researchers’ expertise and ESP investigation findings. Domain vocabularies are also used in developing software solution for a difficult business questions (Boyd 2010). The research findings will be used to form categories (Chatterjee 2010). Regarding method marking assessment, the extracted errors and solution strategies will be classified into related groups and the attributes—will be tabulated—in the phenomena determine the structure of the classification. Hevner and Chatterjee (2010) specify that at the conceptual level, research findings and classifying essences those exist in the research field and their associations are the tasks of the concept and conceptual frameworks. In this research, one of the information that can be extracted from the phenomena study will be problem solvers strategies. In method marking assessment concept, knowing problem solvers’ working strategies or intentions in respect to steps they take or actions they perform would help in tracking


their works and assessing their works. The outcome of the ESP study could reveal the strategies that are used by problem solvers to solve questions. Different solutions could be tried to implement method marking assessment concept using ICT tool. One of the possible solutions is shown in Fig. 2. The constructs in the figure demonstrate tracking actions for the problem solvers’ works assessment in each step. StpAss Construct: This construct makes a decision to start assessing a particular step’s work. AssCntStp construct: This construct assess current step’s work and the assessment criteria is based on the criteria that is inferred from findings of common errors, for example misreading the original question, sign errors, incomplete work, etc. AwdCntStp Construct: The construct awards mark to the current step’s work based again on the assessment criteria formed from common errors findings. AcmAwd Constructor: This construct updates the mark awarded to the problem solvers’ works. PasToNxtStp Construct: In this construct the value of current step answer is sent to the next step if the condition of previous answer availability is true. The condition is checked by the PrvAnsHld construct. fedBack Construct: If all the step’s works have finished or the problem solvers opt to exit from continuing on solving current question, provide an appropriate feedback such as displaying current question’s final answer. endAss Construct: This construct task is to end the step works assessing processes.

MMC assessment model The categories structure (errors and solution strategies classification formats), formed from the phenomena (examination papers investigation), shows the relationship between the categories (errors and solution strategies) and the intended outputs which are rules and algorithm that will be utilized to build production rules that will be stored in the expert shell working memory. The relationship between the category-defining attributes (not identified yet in method marking) and the intended outputs (rules and algorithms to build production rules) will be explored. Hevner and Chatterjee (2010) state that exploring the relationship between attributes in the phenomena and the intended outcomes and recognizing types and magnitudes of association each attribute has with the patterns in the envisioned outcomes results in the output of studies producing models. In current research, the model that will be produced will be method marking technique/model. It has been explained above why the quantitative method in studding the papers is given the priority in the selection process of the research methods; it is obvious to see StpAss If any PrvAnsHld If notDon fnlAns

PasToNxtStp AssCntStp

FedBck AwdCntStp EndAss

Fig. 2 Steps tracking constructs

AcmAwd


the variables in the outcome being influenced by the variables in the phenomena and the model support in explaining this fact. MMC assessment method Methods are instructions or guidelines detailing steps or algorithms to accomplish a task. In DSR methods are made from constructs and models (Hevner et al. 2004; March and Smith 1995; Offermann et al. 2010). The details of methods artefact construction will be dealt with in future articles. MMC assessment instantiation The prospect of constructs, models, or methods realization is demonstrated by the artefact instantiation (Hevner et al. 2004; March and Smith 1995; Offermann et al. 2010). MMC assessment instantiation artefact will be established using expert system prototype to demonstrate method marking assessment concept. The methods artefact that is represented in the form of algorithms and rules will be used to build the selected expert systems’ knowledge base data. In the research, Java programming language, Netbeans 7 or eclipse IDE—to develop rule building and user interface -, and Java expert system shell (JESS)—to implement the MMC assessment instantiation prototype- will be used to answer the research questions. There is no particular reason for selecting Jess to implement the method marking assessment concept, except for the familiarity of the researcher with Java programming language. MMC assessment evaluation The chosen evaluation methods will be used in executing evaluation actions that are guided by the evaluation criteria. Since the evaluation activities are iterative, building and evaluating processes starts from the research problem investigation stage and continues until the required design product is/are realized. At this phase there could be possibilities to use different, additional or modified evaluation criteria and if this happen, any changes will be recorded and clearly explained. The evaluation results will be fed back to the suggestion phase (construction and/or model building phases) and reused (Vaishnavi and Kuechler 2007). The artefacts will be evaluated Bin terms of functionality, completeness, consistency, accuracy, performance, reliability, usability, fit with the organization, and other relevant quality attributes^ (Hevner et al. 2004). As mentioned above, in the design science research section, the selected evaluation methods are analytical, testing and descriptive. These methods are applied to different stage of the evaluation development process and to evaluate MMAC artefacts. The construct artefact evaluation will use descriptive method. Each construct will be tested using detailed scenarios. For example, ASSCntStp construct can be tested with many normal and abnormal scenarios. For example, in solving simultaneous linear equations to find the values of x and y in equation (1), 2x+y=3 and equation (2), x+y= 5, using the AssCntStp construct, a scenario of changing sign in one of the equation can be assessed. The assumption is the problem solver has changed the sign in the process of finding the values of the two variables and the assessment criteria for awarding mark is based on showing the method regardless of the answer produced. The act of changing the sign does not diminish the problem solver’s domain knowledge. Therefore, the AssCntStep construct should continue assessing the work and notify the AwdCntStep construct to award a mark based on the demonstrated work irrespective of the sign changes in the original equations.


Descriptive method also seems appropriate for evaluating MMA model artefact while for method and instantiation artefacts, testing and analytical approaches will be utilized. The strategy in this research is, if during the research time any method that is more suitable is required or discovered and if it is possible to use it, the artefacts will be evaluated using this new method. In summary the iteration nature of the evaluation process entails the application of this type of approach and others make the adaptation of the design science research output, both in the environment to be used in and the knowledge base that it is associated with, easy and smooth.

5 Overall conclusion Method marking assessment requires checking all the learners’ works to award them for the used their detailed understanding of problem solving method. It has been a challenging issue to use ICT to award marks for the learners’ works when method marking concept is applied. To address this issue current CAA for mathematics resorted to breaking MSQ into sub questions. This improves marking practices to award marks to the parts of the answer that is correct. This helped overcome limitation of the partial credit assessment by enhancing method based marking. The problem of using ICT tool to implement method marking when assessing MSQ is the same as solving complex problem, due for the feature they share. This paper proposes a research approach that will produce an artefact prototype in order to implement method marking concept. Pragmatism paradigm and mixed method methodology is suggested to carry out the research. The target phenomena will be selected from previous years’ students’ solutions to examination papers. The investigation through the papers will provide sample errors made by students when solving questions in the examination papers. Qualitative and quantitative methods will be used to study the papers while our proposed research approach will be used in designing and evaluating the MMA artefact prototype. The expectation is that the MMA artefact implementation will play very important role in advancing the process of performing or achieving authentic assessment—using ICT assessment tool—that advances education delivery, accessibility, quality improvement and contribution to long-term knowledge that will reach to wider community. Authentic assessment also will have impact on giving authentic feedback to all stakeholders involved in education sectors. Although algebraic questions solutions assessment is targeted in the study for designing the MMA artefact, there are possibilities for applying the MMA artefact in other fields of mathematics.

References Angeli, C. (2010). Diagnostic expert systems: from expert’s knowledge to real—time systems. Advanced Knowledge Based Systems: Model, Applications & Search, 1, 50–73. Ashton, H. S., Beevers, C. E., Korabinski, A. A., & Youngson, M. A. (2006). Incorporating partial credit in computer-aided assessment of mathematics in secondary education. British Journal of Educational Technology, 37(1), 93–119. doi:10.1111/j.1467-8535.2005.00512.x.

Author's personal copy Educ Inf Technol Balcombe, A., Brennan, M. & Everiss, L. (2011). Formative Learning: A Description of an Assessment Framework in Initial Teacher Education. In: M. Hodis & S. Kaiser (Eds.), Proceedings of the Symposium on Assessment and Learner Outcomes, Victoria University, (pp. 25–40). Wellington, New Zealand. Bayazit, N. (2004). Investigating design: a review of forty years of design research. Design Issues, 20(1), 16–29. Beevers, C., Wild, D., McGuire, G., Fiddes, D., & Youngson, M. (1999). Issues of partial credit in mathematical assessment by computer. Research in Learning Technology, 7(1), 26–32. Black, P., & Wiliam, D. (2005). Lessons from around the world: how policies, politics and cultures constrain and afford assessment practices. Curriculum Journal, 16(2), 249–261. doi:10.1080/09585170500136218. Bloxham, S., & Boyd, P. (2007). Developing Effective Assessment In Higher Education: A Practical Guide: A Practical Guide, McGraw-Hill Education, UK. Borgman, C. L. (2012). The conundrum of sharing research data. Journal of the American Society for Information Science and Technology, 63(6), 1059–1078. Boyd, N. S. (2010). Domain vocabulary. Retrieved 2013/12/19, from http://educery.com/educe/patterns/ domain-vocabulary.html. Chatterjee, S. (2010). Design research in information systems: theory and practice (Vol. 22). Springer. Creswell, J. (2003). Research design: Qualitative, quantitative, and mixed methods approaches. Boston: SAGE Publications. Cross, N. (1984). Developments in design methodology. Chichester: John Wiley & Sons Ltd. Davis, D. (2007). Complex software problem solving by means of abstractive techniques. Paper presented at the Proceedings of the 11th international conference on Computer aided systems theory, Las Palmas de Gran Canaria, Spain. de Jong, T. (2014). Emerging representation technologies for problem solving. In: Spector, J. M., Merrill, M. D., Elen, J. & Bishop, M. J. (Eds.), Handbook of Research on Educational Communications and Technology (pp. 809–816). New York: Springer. Faste, T., & Faste, H. (2012). Demystifying Bdesign research^: design is not research, research is design, IDSA Education Symposium. Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. The Journal of Problem Solving, 4(1), 19–42. doi:10.7771/1932-6246.1118. Funke, J. (2010). Complex problem solving: a case for complex cognition? Gov’t review. Cognitive Processing, 11(2), 133–142. Gears, C. (2005). Classroom assessment: minute by minute, day by day. Assessment, 63(3), 19–24. Grosan, C., & Abraham, A. (2011). Rule-based expert systems. Intelligent Systems. Springer. Hevner, A. R. (2007). A three cycle view of design science research. Scandinavian journal of information systems, 19(2), 87–92. Hevner, A., & Chatterjee, S. (2010). Design research in information systems: Theory and practice. New York: Springer Science & Business Media. Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75–105. Hwang, G.-J., Chen, C.-Y., Tsai, P.-S., & Tsai, C.-C. (2011). An expert system for improving web-based problem-solving ability of students. Expert Systems with Applications, 38, 8664–8672. Jaques, P. A., Seffrin, H., Rubi, G., Morais, F. D., Guilardi, C., Bittencourt, I. I., & Isotani, S. (2013). Rulebased expert systems to support step-by-step guidance in algebraic problem solving: the case of the tutor PAT2Math. Expert Systems with Applications, 40(14), 5456–5465. Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: a research paradigm whose time has come. Educational Researcher, 33(7), 14–26. Jones, I. S. (2008). Computer-aided assessment questions in engineering mathematics usingMapleTA®. International Journal of Mathematical Education in Science and Technology, 39(3), 341–356. doi:10. 1080/00207390701734523. Kaidar, S. M., Hussain, R. I., Bohani, F. A., Sahran, S., Binti Zainuddin, N., Ismail, F., Thanabalan, J., Kalimuthu, G., & Abdullah, S. N. H. S. (2013). Brain tumor treatment advisory system. Soft Computing Applications and Intelligent Systems. Springer. Kamel Boulos, M. N. (2012). Expert system shells for rapid clinical decision support module development: an ESTA demonstration of a simple rule-based system for the diagnosis of vaginal discharge. Healthcare Information Research, 18, 252–258. Khan, F. S., Razzaq, S., Irfan, K., Maqbool, F., Farid, A., Illahi, I., & Ul Amin, T. (2008). Wheat: A Web-based expert system for diagnosis of diseases and pests in Pakistani wheat. Proceedings of the World Congress on Engineering, Oct 2008, 2–4. Kirsh, D. (2010). Thinking with external representations. AI & SOCIETY, 25(4), 441–454. doi:10.1007/ s00146-010-0272-8.

Author's personal copy Educ Inf Technol Lawson, D. (2012). Computer-aided assessment in mathematics: panacea or propaganda? International Journal of Innovation in Science and Mathematics Education (formerly CAL-laborate International), 9(1), 6–12. Livne, N. L., Livne, O. E., & Wight, C. A. (2007). Can automated scoring surpass hand grading of students’ constructed responses and error patterns in mathematics. MERLOT Journal of Online Learning and Teaching, 3(3), 295–306. March, S. T., & Smith, G. F. (1995). Design and natural science research on information technology. Decision Support Systems, 15(4), 251–266. Miah, S. J., Kerr, D., & Gammack, J. (2009). A methodology to allow rural extension professionals to build target specific expert systems for Australian rural business operators. Expert Systems with Applications, 36(1), 735–744. Miah, S. J., Kerr, D., & Von-Hellens, L. (2014). A collective artefact design of decision support systems: design science research perspective. Information Technology & People, 27(3), 259–279. Offermann, P., Blom, S., Schönherr, M., & Bub, U. (2010). Artifact types in information systems design science–a literature review. In: Winter, R., Zhao, J. L. & Aier, S. (Eds.), Global perspectives on design science research (pp. 77–92). Springer Berlin Heidelberg. Österle, H., Becker, J., Frank, U., Hess, T., Karagiannis, D., Krcmar, H., et al. (2011). Memorandum on design-oriented information systems research. European Journal of Information Systems, 20(1), 7–10. Palm, T. (2008). Performance assessment and authentic assessment: a conceptual analysis of the literature. Practical Assessment, Research & Evaluation, 13(4), 1–11. Passmore, T., Brookshaw, L., & Butler, H. (2011). A flexible, extensible online testing system for mathematics. Australasian Journal of Educational Technology, 27(6), 896–906. Pereira, R., Almeida, R., & da Silva, M. M. (2013). How to generalize an information technology case study. In: Vom Brocke, J., Hekkala, R., Ram, S. & Rossi, M. (Eds.), Design science at the intersection of physical and virtual design (pp. 150–164). Springer Berlin Heidelberg. Price, M., Carroll, J., O’Donovan, B., & Rust, C. (2011). If I was going there I wouldn’t start from here: a critical commentary on current assessment practice. Assessment & Evaluation in Higher Education, 36(4), 479–492. Sangwin, C. (2012). Computer aided assessment of mathematics using STACK. Paper presented at the Proceedings of ICME. Sangwin, C., Cazes, C., Lee, A., & Wong, K. L. (2010). Micro-level automatic assessment supported by digital technologies. Hoyles, C. & Lagrange, J.-B. (Eds.), Mathematics Education and TechnologyRethinking the Terrain, (pp. 227-250). Springer USA. 13, 227–250. Shepard, L. A. (2005). Linking formative assessment to scaffolding. Educational Leadership, 63(3), 66–70. Sonnleitner, P., Keller, U., Martin, R., & Brunner, M. (2013). Students’ complex problem-solving abilities: their structure and relations to reasoning ability and educational success. Intelligence, 41(5), 289–305. doi:10.1016/j.intell.2013.05.002. Vaishnavi, V. K., & Kuechler Jr, W. (2007). Design science research methods and patterns: innovating information and communication technology. CRC Press. Van den Broeck, J., Brestoff, J. R., & Chhagan, M. (2013). Maintaining data integrity. In: Van den Broeck, J. & Brestoff, J. R. (Eds.), Epidemiology: principles and practical guidelines (pp. 379–388). UK: Springer. Wiliam, D. (2011). What is assessment for learning? Studies in Educational Evaluation, 37(1), 3–14. doi:10. 1016/j.stueduc.2011.03.001. Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex problem solving—more than reasoning? Intelligence, 40(1), 1–14. doi:10.1016/j.intell.2011.11.003. Yeo, R. K., & Marquardt, M. J. (2012). Complex problem solving through action learning: implications for human resource development. International Journal of Human Resources Development and Management, 12(4), 258–273.