Jan 15, 2018 - Figure 18: Process to select the Calliope Mini (Jørn Alraun, 2017) ...... In addition, some words such as âpdfâ or âorâ were added to the set achieve more .... or a simulator to test code on the computer before testing it on a robot (Barry S. Fagin et al., ... worlds they built in their free timeâ (Rodger et al., 2009).
FAKULTÄT FÜR INFORMATIK
DER TECHNISCHEN UNIVERSITÄT MÜNCHEN
Bachelor’s Thesis in Information Systems
Teaching Programming by Using a Robot: Execution and Evaluation of an Individualized in-class Exercise
Anton Coqui
FAKULTÄT FÜR INFORMATIK
DER TECHNISCHEN UNIVERSITÄT MÜNCHEN
Bachelor’s Thesis in Wirtschaftsinformatik
Programmieren lernen mithilfe eines Roboters: Erstellung und Evaluation einer individualisierten Unterrichtseinheit
Teaching Programming by Using a Robot: Execution and Evaluation of an Individualized in-class Exercise
Bearbeiter: Aufgabensteller: Betreuer: Abgabedatum:
Anton Coqui Prof. Dr. Helmut Krcmar Prof. Dr. Matthias Utesch, Robert Heininger 15.1.2018
I assure the single handed composition of this bachelors’ thesis only supported by declared resources.
Garching near Munich, 15.1.2018 Ort, Datum
________________________ Signature
Abstract The use of robots in CS education makes programming more joyful, better to understand, and shows its value in the real world. Despite that, the amount of research on programming education using affordable robots cheaper than 50$ is very limited. This paper identifies the success factors of programming lessons using robots and presents an affordable lesson artefact evaluated in a realistic environment that was built based on these success factors. The success factors were established through a literature review of previous papers in the field of programming education using robots. Four distinct success factors were identified: applicability of programming to another field, confidence of students, ease of learning and efficiency in learning. Based on these success factors, a lesson artefact was created for teaching introductory programming procedures such as the if-loop and the for-loop. The artefact consists of an educational software working with an affordable robot, an install guide, a lesson guide, and the lesson materials. Two lessons using the lesson artefact were conducted in a school in Bavaria with 8 students in the age of 11-13 each. The lesson was evaluated through a questionnaire handed out to students before and after the lesson where they estimated their skills in applying the programming procedures on a Likert scale from 1 to 6. The lesson experience increased the average student’s rating of her abilities by 1.4, 2.2 and 2.4 scale points for the procedures if-loop, for-loop and for-loop with iterator, respectively. On the same scale, the students’ ratings of confidence in their programming abilities also increased by 0.9 points on average. The lesson had no significant effect on students’ short-term and long-term motivation for future activities in the area of programming. Keywords: Programming Education, Robots, Programming Environments, Middle School, Calliope Mini, Python, cross-platform
III
Table of Contents Abstract ............................................................................................................................ III Table of Contents .............................................................................................................. IV List of Figures .................................................................................................................... VI List of Tables .................................................................................................................... VII List of Abbreviations ........................................................................................................ VIII 1
2
3
4
5
Introduction ................................................................................................................ 1 1.1 Why Programming Education? ........................................................................................ 1 1.2 Why Programming Education Using Robots? ................................................................... 1 1.3 Open Areas in Research .................................................................................................. 2 1.4 Aim of This Paper ............................................................................................................ 4 Approach .................................................................................................................... 5 Contribution of This Paper .............................................................................................. 5 The Process to Achieve the Contribution ......................................................................... 7 The Structure of This Paper ............................................................................................. 9
2.1 2.2 2.3
Literature Review ..................................................................................................... 11 3.1 Methodology ................................................................................................................ 11 3.1.1 Why Methodology Documentation? ............................................................................... 11 3.1.2 Review Scope ................................................................................................................... 11 3.1.3 Conceptualization of Topic .............................................................................................. 12 3.1.4 Literature Search ............................................................................................................. 13 3.2 Results .......................................................................................................................... 15 3.3 Success Factors ............................................................................................................. 16 3.3.1 Applicability ..................................................................................................................... 16 3.3.2 Confidence ....................................................................................................................... 18 3.3.3 Ease .................................................................................................................................. 19 3.3.4 Efficiency .......................................................................................................................... 21 3.4 Summary ...................................................................................................................... 22 3.5 Artefact research .......................................................................................................... 24 Design Process .......................................................................................................... 26 Design Basis .................................................................................................................. 27 Learning Objectives ....................................................................................................... 28 Stage 1: Lesson Prototype ............................................................................................. 29 Stage 2: Improved Software .......................................................................................... 30 Stage 3: Improved Lesson .............................................................................................. 32 Stage 4: Final Lesson and Software ................................................................................ 36 Stage 5: Lesson Preparation .......................................................................................... 36
4.1 4.2 4.3 4.4 4.5 4.6 4.7
Artefact Description .................................................................................................. 38
5.1 Educational Software .................................................................................................... 38 5.1.1 Use Guide ........................................................................................................................ 38 5.1.2 Design Principles – User Experience ................................................................................ 40 5.1.3 Improvements Suggestions – User Experience ............................................................... 41 5.1.4 Design Principles – Software Architecture ...................................................................... 42 5.2 Install Guide .................................................................................................................. 44
IV
5.2.1 Use Guide ........................................................................................................................ 44 5.2.2 Improvement Suggestions ............................................................................................... 46 5.3 Lesson Materials ........................................................................................................... 46 5.3.1 Use Guide ........................................................................................................................ 47 5.3.2 Design Principles .............................................................................................................. 48 5.3.3 Improvement Suggestions ............................................................................................... 49 5.4 Lesson Guide ................................................................................................................. 50 5.4.1 Use Guide ........................................................................................................................ 50 5.4.2 Design Principles .............................................................................................................. 52 5.4.3 Improvement Suggestions ............................................................................................... 52
6
7
8
Evaluation ................................................................................................................. 54 6.1 Aim and Methods ......................................................................................................... 54 6.2 Measurement ............................................................................................................... 55 6.3 Representation ............................................................................................................. 57 6.4 Quantitative Analysis and Results ................................................................................. 57 6.5 Qualitative Observations .............................................................................................. 61 6.6 Improvement Suggestions ............................................................................................. 62 Discussion ................................................................................................................. 63
7.1 Boundaries and Limitations ........................................................................................... 63 7.2 Discussion of Learning Objectives ................................................................................. 64 7.3 Discussion of Other Constructs ...................................................................................... 65 7.3.1 Motivation for Programming ........................................................................................... 65 7.3.2 Success Factor Confidence .............................................................................................. 66 7.3.3 Examples and Explanations ............................................................................................. 67 7.4 Value of the Lesson Artifact .......................................................................................... 67 7.4.1 Novelty ............................................................................................................................ 67 7.4.2 Value ................................................................................................................................ 68
Conclusion ................................................................................................................ 71
Bibliography ..................................................................................................................... 73 Appendix .......................................................................................................................... 81 Appendix A Literature Review Search Process .................................................................. 82 Appendix A.1 Third Set of Literature Review Search Keywords ......................................... 82 Appendix A.2 Fourth Set of Literature Review Search Keywords ....................................... 82 Appendix A.3 Lists of all Search Terms .............................................................................. 82 Appendix A.4 Backward Searches ..................................................................................... 85 Appendix A.5 Concept Matrix ........................................................................................... 86 Appendix B Design Process ............................................................................................... 89 Appendix B.1 Feedback Items for the Lesson Prototype.................................................... 89 Appendix B.2 Feedback Items for the Improved Lesson .................................................... 89
V
List of Figures Figure 1: DSR Knowledge Contribution Framework .................................................................6 Figure 2: Types of Useful Knowledge in Information Systems .................................................6 Figure 3: Prescriptive Knowledge Contribution Types ..............................................................7 Figure 4: Work Process...............................................................................................................8 Figure 5: Structure of This Paper ..............................................................................................10 Figure 6: Amount of Lesson Changes by Success Factor.........................................................16 Figure 7: Calliope Mini Robot ..................................................................................................24 Figure 8: Mu Programming Environment.................................................................................25 Figure 9: Iterative Search Process.............................................................................................27 Figure 10: Example Block ........................................................................................................30 Figure 11: For-Loop to show line of lights .............................................................................31 Figure 12: For-Loop to show line of lights ..............................................................................31 Figure 13: User Interface of Educational Software ..................................................................31 Figure 14: Example Level .........................................................................................................33 Figure 15: Example Block Description.....................................................................................33 Figure 16: Example Block ........................................................................................................38 Figure 17: Software Editor........................................................................................................38 Figure 18: Process to select the Calliope Mini (Jørn Alraun, 2017) robot as a target device 40 Figure 19: Simplified UML Class Diagram of the Educational Software ................................43 Figure 20: Example task in the “Robot Workshop” .................................................................47 Figure 21: Example block description in “The Blocks” ...........................................................48 Figure 22: Lesson Guide – Part 1 .............................................................................................51 Figure 23: Lesson Guide – Part 2 .............................................................................................52 Figure 24: Construct Measurement in Questionnaire ...............................................................55 Figure 25: Total Survey Error Framework ...............................................................................56
VI
List of Tables Table 1: Open Areas in Research................................................................................................3 Table 2: Boundaries of This LR................................................................................................11 Table 3: Taxonomy of Literature Reviews ...............................................................................12 Table 4: Key Concepts as First Set of Search Keywords .........................................................13 Table 5: All Changes Sorted by Success Factor (Number of Occurrences) .............................23 Table 6: Artefact Design Criteria and Activities to Achieve Them..........................................26 Table 7: The Revised Taxonomy of Learning Objectives, Left Column: Knowledge Type, Top Column: Type of Cognitive Process. ........................................................................................29 Table 8: Table with all blocks ...................................................................................................39 Table 9: Install Guide – Detailed Instructions ..........................................................................46 Table 10: Front Pages of the Four Booklets of the Lesson Guide ............................................47 Table 11: Question Items to Measure Constructs and Demographical Data ............................59 Table 12: Aggregated Constructs..............................................................................................59 Table 13: Statistical Parameters of All Constructs (in all except for sLO1, SF1 and SF2 the change between lessons was considered) .................................................................................60 Table 14: Statistical Parameters Concerning the Change of Construct Values Between Lessons ...................................................................................................................................................60 Table 15: Interrelationships Between Selected Constructs .......................................................60 Table 16: Task Times of Two-Student Teams in Group 1 .......................................................60 Table 17: Task Times of Two-Student Teams in Group 2 ......................................................60 Table 18: Specific Keywords and Ankles of View as Third Set of Search Keywords .............82 Table 19: Keywords of Particular Interest as Fourth Set of Search Keywords ........................82 Table 20: Lists of all 167 Search Terms ...................................................................................85 Table 21: Backward Searches ...................................................................................................86 Table 22: Concept Matrix – Part 1 ...........................................................................................87 Table 23: Concept Matrix – Part 2 ...........................................................................................88 Table 24: Feedback Items for the Lesson Prototype .................................................................89 Table 25: Feedback Items for the Improved Lesson .................................................................90
VII
List of Abbreviations CS SF OAR LR UI
computer science success factors open areas in research literature review user interface
VIII
1 Introduction 1.1 Why Programming Education? Looking at the U.S. workforce, there is one group of employees whose share has grown the fastest both in the periods 1940-1980 and 1980-2010: professionals and technicians (Autor, 2015). In particular, the fraction of computer scientists of all U.S. college graduate workers increased from slightly above 3% to more than 5% over the period of 1990-2012 (Bound, Braga, Golden, & Khanna, 2015). Looking forward, the Bureau of Labour Statistics projects a 12% growth in employment opportunities in the U.S. for computer and information technologies for the years 2014-2024 (Bureau of Labour Statistics, 2015). Also, the median annual wage of these occupations is more than twice as high as the median annual wage of all other occupations in the U.S (Bureau of Labour Statistics, 2015). This indicates a high value of these jobs for the economy. Despite that, the total amount of computer science (CS) bachelor’s degrees awarded in the U.S. declined over the period of 2004-2015 (Bizot, 2016; Zweben, 2005). One can infer that if this development continues, the amount of trained computer scientists cannot satisfy the growing need for professionals in this area. Programming, defined as “the activity or job of writing computer programs”, is an integral part of many technology- and CS-oriented job activities (Cambridge University Press, 2017). Thus it should also be taught in a CS education, which prepares students for a career in a highly demanded employment area. Learning the skill of programming also provides invaluable benefits applicable beyond the classroom or workplace. Firstly, it improves metacognitive skills like problem-solving or developing an exact way of thinking and expressing ones thoughts (Hromkovič, 2006; Papert, 1980; Szlávi & Zsakó, 2006). Secondly, it develops better cognitive abilities. A meta-analysis summarizing the results of 65 studies unveiled that “students having computer programming experiences scored about sixteen percentile points higher on various cognitive-ability tests than students who did not have programming experiences” (Liao & Bright, 1991). These two points show that all students profit from programming education no matter if they see it as a career opportunity or a way to expand their knowledge and understanding of the world.
1.2 Why Programming Education Using Robots? In order to finish a CS degree students need to get interested in the program, stay motivated throughout the program and be successful during the program. The use of robots in class can help solving all three of these challenges. Students struggle to relate the skill of programming to their everyday life. More than 80% of students in high school have no idea what you learn in a CS major (Carter, 2006). Yet, the application to another field is one of the most important reasons for males and the most important reason for females to study CS (Carter, 2006). This problem continues after enrolment in a CS program: Many students lose motivation or even drop out because they cannot relate the course contents to their everyday life (Miller, Nourbakhsh, & Siegwart, 2008). Using robots to translate programming commands to real-world movements is one way to take away the barrier between the physical and virtual world. 1
Even when students already understand the value of programming in the real world, keeping them engaged throughout a tough curriculum is still a challenge. Tangibility is a strong motivational factor: When having the choice between a tangible (building blocks) and a graphical programming interface, students in all three surveyed age groups (5–6, 7–8 and 11– 12 years) reported enjoying the use of a tangible interface more (Sapounidis & Demetriadis, 2013). In two studies comparing a robot group with a non-robot one, students showed increased fun and interest during the courses (Álvarez & Larrañaga, 2016; Pásztor, Pap-Szigeti, & Lakatos Török, 2010). Interestingly, this did not translate to better learning performance in both cases (Álvarez & Larrañaga, 2016; Pásztor et al., 2010). Yet, because of the robots being a great tool for visualization of learning outcomes, students had more confidence in their abilities compared to control groups (Kolberg & Orlev, 2001; Nagchaudhuri, Singh, Kaur, & George, 2002). A feeling of competence is one of the main factors influencing motivation (Ryan & Deci, 2000). As a result, the use of a robot decreased the amount of failed lessons as well as the perceived difficulty of tasks (Pásztor et al., 2010). Data shows that the amount of students failing an introductory programming course in university is similar to the amount of students aborting a course (Bennedsen & Caspersen, 2007). Many students struggle to translate real-world problems into code problems. They have more problems with designing programs and dividing real-world problems than understanding syntax and programming structures (Lahtinen, Ala-Mutka, & Järvinen, 2005). In particular, they are challenged by creating mental models of how computers work and developing efficient problem-solving strategies (Robins, Rountree, & Rountree, 2003) (Perkins, Schwartz, & Simmons, 1988). Based on the idea of constructivism, robots provide ways to model programming concepts like loops or conditionals in the real world that students know and can relate to (Barry S. Fagin, Merkle, & Eggers, 2001). They also help students to solve real-world problems by enabling them to see their current program in the real world and think from the robots perspective. (Flowers & Gossett, 2002). Showing programming in real-world use does also affect pass rates: putting a programming class in a context (e.g. robotics) or introducing media computation (using computing to create and edit media) both on average increase the pass rates in programming courses by 48% and 40%, respectively (Vihavainen, Airaksinen, & Watson, 2014).
1.3 Open Areas in Research It has been illustrated how the use of robots helps in solving the problem of engaging more students into obtaining an education in programming. The open question remains how to contribute to research in that area. In order to investigate that, a wide-spread review of the field of programming education using robots will be conducted. The directions for future research in this field can be categorized into four areas: learning outcomes, teaching methodology, research design and operational support of the field. In the following, some selected directions for future research in each of these areas will be presented in more detail. The propositions were selected based on how well they can be realized in the setting of this bachelors’ thesis.
2
Learning Outcomes
Teaching Methods
Research Focus
Operational Support
Effective Learning Strategies
Pair Programming
Use of Cheap Robots
Reuse of Elements
Quantitative Evaluation
Common Robotic Platform
Rigorous Research Design Table 1: Open Areas in Research Source: Own Elaboration
Based on the definition of the CEDEFOP, a learning outcome is a “statement of what a learner is expected to know, be able to do and understand after having completed a learning process” (Bjornavold, 2015). One learning outcome to investigate proposed by research is to put more emphasis on students understanding of how to use computers as a tool for other purposes (Schulte & Knobelsdorf, 2007). Many students, especially those not affiliated to CS, have a defensive attitude towards computers. In particular, they only learn about computers when they have to, do not enjoy doing that and often also fail on this task (Schulte & Knobelsdorf, 2007). Even when students started learning programming, many students exhibit ineffective learning behavior (Robins et al., 2003). Ineffective novices to programming are “those that do not learn, or do so only after inordinate effort and personal attention” (Robins et al., 2003). Thus Robins et al. recommend for future research to focus on the learning outcome of students being able to program effectively (Robins et al., 2003). When the expected learning outcomes are set, the question remains what the best teaching methods are to meet these outcomes. One recent trend is the introduction of pair programming. It was introduced on a wide scale in industry during the 90s as a part of the extreme programming methodology (Beck, 1999). Tests of this concept in programming education were successful, pass rates were improved by on average 27% through pair programming (Vihavainen et al., 2014). Still, the question does remain whether pair programming is an appropriate approach for an in-class setting (Werner et al., 2013). Programming lessons using robots in classrooms are rare, one big reason being the robot price. Yet, researchers mostly incorporate a more expensive alternative like LEGO Mindstorm robots in their research design (Benitti, 2012). This results in a lack of empirical evidence demonstrating the effectiveness of low cost robots (Benitti, 2012). The lack of evidence is not only the case for low cost robots: there is also a call for more quantitative evaluations and a more rigorous research design in the overall field (Benitti, 2012). One step to achieve that would be the introduction of systematic assessment procedures (Lee et al., 2011) (Benitti, 2012). Beyond similar assessment procedures, also the reuse of software code and findings can help the research field to grow together. One way to facilitate this synthesis is the development 3
towards a common robotic platform (Martyn Cooper & William Harwin, 1999). This way all researchers on that platform have the possibility to integrate code of other researchers with ease.
1.4 Aim of This Paper The merits of and demand for programming education, as well as why a robot is a promising tool to conduct a lesson with, have been shown. Because of that, this paper will focus on contributing to the field of programming education using a robot. In order to make a contribution of high quality, the contribution will be based on the current knowledge in the field. The aim of this paper will be to advance the field by addressing some of the open questions in the field specified in the last section Open Areas in Research (OAR). In order to gain precise insights, a lesson encompassing specific characteristics from OAR and focused on answering the open questions in OAR will be build.
4
2 Approach 2.1 Contribution of This Paper How to make a good contribution to information systems (IS) research? According to Hevner et al., there are two kinds of contributions that expand the knowledge of IS research: behavioral science and design science contributions. (Von Alan, March, Park, & Ram, 2004) Behavioral science “seeks to develop and justify theories (i.e., principles and laws) that explain or predict organizational and human phenomena surrounding the analysis, design, implementation, management, and use of information systems” (Von Alan et al., 2004). These theories only have value when they have an influence in how IS are built in the future (Benbasat & Zmud, 1999). Design science aims at the creation of innovative artefacts based on the knowledge of behavioral science that make IS in practice more efficient and effective (Von Alan et al., 2004). The role that lessons using robots play in programming education is very slim and the results concerning educational efficiency are ambiguous (Major, Kyriacou, & Brereton, 2012) (Vihavainen et al., 2014). However, 1.2 shows that the use of robots has great potential. Beyond that, prices of robots are getting lower and there are several recent studies where robots outperform a non-robot approach in a direct comparison (Summet et al., 2009) (Lauwers, Nourbakhsh, & Hamner, 2009). This led me to the conclusion to create and evaluate an artefact myself in a practical environment. By conducting design science research, the aim of this paper is to improve a known problem (efficient programming education) by a new solution (an education environment using a robot as a tool). The solution is new because it will combine several successful characteristics from previous studies in the artefact design. The contribution thus can be categorized in the improvement quadrant in the DSR Knowledge Contribution Framework presented by Gregor and Hevner, as depicted in Figure 1 (Gregor & Hevner, 2013). Based on Gregor and Hevner, improvement contributions should have a strong theoretical grounding because the problems have yet been approached numerous times (Gregor & Hevner, 2013). As usual for improvement knowledge contributions, the contribution to knowledge is twofold: Contribution of descriptive knowledge (knowledge describing information systems and their interaction with people) and prescriptive knowledge (knowledge about the creation and value of artefacts) as described in Figure 2 (Von Alan et al., 2004). One contribution to descriptive knowledge will be the confirmation or disconfirmation of theories connected with programming education using robots through the results of this paper (Gregor & Hevner, 2013). Also, new behavioral knowledge can be collected about the behavior of people interacting with the artefacts created (Gregor & Hevner, 2013). The knowledge contribution to prescriptive knowledge can be categorized into three levels of abstraction as depicted in Figure 3. (Gregor & Hevner, 2013) The more abstract the knowledge contribution, the more widely applicable and powerful it is. In this instance, a situated instantiation in form of the lesson artefacts will be created, which is a level 1 knowledge contribution. The creation of the artefact will also involve more abstract design choices that can influence the construction of future design science artefacts and are a level 2 knowledge contribution. The evaluation of the artefact will provide insights about why the lesson did achieve their objectives or not and how the lesson success 5
relates to other variables measured. This knowledge contributes to general design theory of programming lessons and is a level 3 knowledge contribution. Looking beyond the research community, my goal is to also provide an artefact useful for practitioners. The primary focus is here to make an artefact that teachers can easily implement in their lessons.
Figure 1: DSR Knowledge Contribution Framework Source: (Gregor & Hevner, 2013)
Figure 2: Types of Useful Knowledge in Information Systems Source: (Gregor & Hevner, 2013)
6
Figure 3: Prescriptive Knowledge Contribution Types Source: (Gregor & Hevner, 2013)
2.2 The Process to Achieve the Contribution Figure 4 depicts the process which achieves the contributions to the knowledge base of IS research as specified in the previous section. The work process involves three stages. First, existing research will be examined to build a quality lesson based on previous experiences and address open areas put forward by other researchers. Secondly, a lesson teaching programming including a software learning environment and robot will be created. Thirdly, the lesson will be conducted in a school environment and evaluated. Each stage answers one of three research questions that will lead the research conducted in this paper. In the following, the stages answering each of the three research questions will be explained in more detail. All descriptions in the research question paragraphs refer to elements depicted in Figure 4. Research Question 1: What are success factors for programming lessons with a robot as a tool? The aim of this research is to create a lesson that reaches its learning objectives (LOs). “Learning objectives … describe intended learning outcomes”, which are defined in 1.3 (Krathwohl, 2002). Previous researchers spent lots of time trying out and evaluating different lessons on whether they reached their LOs. A literature review (LR) of the field “should include any prior design theory/knowledge relating to the class of problems to be addressed” (Gregor & Hevner, 2013). In this instance, it will help in establishing the factors that influence whether a lesson reached its LOs, which is synonymous for lesson success. These success factors (SFs) will be the basis for the design of the lesson artefact later. Beyond that, previously created lessons, software and hardware artefacts will also be examined, as usual in design science research (Gregor & Hevner, 2013). A suitable selection will be synthesized in the design of the lesson artefact later. How can this paper make a knowledge contribution relevant to the field? The chosen approach is to focus on open areas proposed by other researchers. That’s why, beyond SFs, the OAR proposed in 1.3 will be integrated in the artefact design. Evaluating their effectiveness will generate insights on relevant questions in the field.
7
Figure 4: Work Process Source: Own Depiction
Research Question 2: What would be an approach to create a successful and measurable lesson with the help of a robot as a tool that would also contribute to the research field? Building on Research Question 1, the aim here is to construct an artefact based on previously generated artefacts and knowledge (SF, existing artefacts) whose use gives insights on new and interesting topics in the field (OAR). The SFs and OAR will form the requirements for the design of the artefact. The artefact will be built to reach certain LOs of students. This means that the aim of the whole construction process is to create an artefact that assists in reaching the LOs in the best way possible. The planning process for the artefact as well as a log of ongoing work progress is written down in a design logbook and was continuously improved in an iterative search process. (Von Alan 8
et al., 2004) In addition to detailing the work process of the artefact creation, the logbook also documents the design principles used here, which could be useful for future research (Contribution to Descriptive Knowledge – Level 2) (Gregor & Hevner, 2013). The work process will yield a lesson ready for use in a school environment consisting of software, a use guide and lesson instructions. This lesson can not only be immediately used by practitioners, but also provides an instantiation of an artefact ready for future research (Contribution to Descriptive Knowledge – Level 1) (Gregor & Hevner, 2013). Research Question 3: How did the lesson affect students’ performance in the learning objectives and motivation for future programming activities? To gain insights on the efficiency of the implemented SFs and OAR in the artefact design, the success of the lesson will be measured in terms of how well the LOs specified in Research Question 2 have been reached. The test of the artefact in a real environment might not only provide observations about student behavior when using technology, but also about the relationships of different behaviors and the technologies used (Contribution to Prescriptive Knowledge) (Gregor & Hevner, 2013). In addition, the evaluation might unveil why a specific approach (e.g. from OAR or SFs) worked (Contribution to Descriptive Knowledge – Level 3). The discussion and conclusion chapters will show the boundaries of the applicability of the research results, interpret the findings as well as connect them to the body of knowledge in the field (Von Alan et al., 2004).
2.3 The Structure of This Paper The structure and work process of this publication are largely based on the publication schema for design science research put forward by Gregor and Hevner (Gregor & Hevner, 2013). Yet, some distinctions have been made to adjust to the specific requirements of the research of this paper. All descriptions in the following paragraph refer to the depiction in Figure 5. The paper first wants to establish the relevance of the research area (1.1, 1.2) as well as the goals of this paper in an informal way (1.3, 1.4), which is the reason why 1.3 OAR is part of the introduction and not the LR as it would be defined by Gregor and Hevner (Gregor & Hevner, 2013). Beyond that, the approach and contributions in this paper were moved to a separate chapter as they are a split logical unit from the introduction and formulated in a more structured way (Gregor & Hevner, 2013). As many independent chapters (3,4 and 6) will have their own methods, the choice was made to explain the methods applied at the beginning of each chapter instead of describing the design science methods in an own chapter (Gregor & Hevner, 2013). Due to the inherently complex nature of educational lessons using a robot, the design process will be described in its own chapter and not in “artefact description” (Gregor & Hevner, 2013). It will also elaborate on the methods used during the process. The chapters artefact description, evaluation, discussion and conclusion are adopted unchanged from the publication schema (Gregor & Hevner, 2013).
9
5 Artefact Description
Figure 5: Structure of This Paper Source: Based on (Gregor & Hevner, 2013)
10
3 Literature Review The aim of this chapter is to create a foundation for the creation of an artefact in form of existing knowledge and artefacts. The existing knowledge relevant to this scenario is defined as SFs of programming lessons using robots, answering the first research question defined in 2.2. The existing artefacts relevant to this scenario at this stage are a hardware robot useful for lesson instruction and a software environment for educational use compatible to this robot. The next four sections describe the SFs, the methodology to retrieve them and how they will contribute to the artefact design. The last section will describe the existing artefacts, the methodology to retrieve them and how they will contribute to the artefact design.
3.1 Methodology 3.1.1
Why Methodology Documentation?
As IS research is an emerging research field, the amount of publications that can be included in a LR is heavily growing (vom Brocke et al., 2009). In particular, “in 2003 Peffers and Ya identified 326 journals that publish IS research, while the Index of Information Systems Journals lists … [699] active IS journals today” (Lamp, 2013; Peffers & Ya, 2003). In addition, the amount of papers relevant for a LR is especially big in this case, as the research conducted is at the intersection of numerous fields (information systems, education, CS, robotics). This leads to the conclusion that bigger amounts of relevant literature must be omitted in the LR in order to focus on the aspects most relevant for this paper ((1) in Table 3). Because bigger parts of relevant literature will be excluded, “the process of excluding sources (and including respectively) has to be made as transparent as possible in order for the review to proof credibility” (vom Brocke et al., 2009). This paper will document the process of literature reviewing based on the framework put forward by vom Brocke specifically for IS research (vom Brocke et al., 2009). The first three parts of the framework that concern the literature search process will be described in the following subsections of this section. The last two parts of the framework concerning analysis and interpretation will be elaborated in the following section. 3.1.2
Review Scope
This LR will focus on identifying the SFs of programming lessons using a robot as a tool in secondary school. This focus defines the boundaries of the literature reviewed depicted in Table 2. Research Field
Application Field
Subject
Methodology
Location
education
secondary school
programming
robot as a tool
Germany
Table 2: Boundaries of This LR Source: Own Depiction
11
Because the aim of this paper is to create an artefact based on this LR, the audience of this LR will be primarily scholars building design science artefacts, yet the findings are also understandable by practitioners and useful for specialized scholars ((2) in Table 3) (Cooper, 1988). Consecutively, the LR will only include papers where a change in lesson design made the lesson more successful by the definition in 2.2 Research Question 1. The only relevant parts of research papers are research outcomes (Is the lesson successful?) and theories (What is the design theory underlying the artefact design?) ((3) in Table 3) (Cooper, 1988). Then the changes will be categorized by what type of positive influence they had into few distinct SFs ((4) in Table 3) ((5) in Table 3) (Cooper, 1988). As the aim is to create a knowledge base for artefact creation instead of advancing how this knowledge is produced, the author will document the past research instead of criticizing it ((6) in Table 3) (Cooper, 1988). (1)
Coverage
representative
(2)
Audience
scholars building design science artefacts, practitioners / politicians, specialized scholars
(3)
Focus
research outcomes
(4)
Goal
synthesis, central issues
(5)
Organization
conceptual
(6)
Perspective
neutral
Table 3: Taxonomy of Literature Reviews Source: Based on (Cooper, 1988; vom Brocke et al., 2009)
3.1.3
Conceptualization of Topic
The main basic concepts inside this research scope can be directly derived from the boundaries specified in Table 2. After a research of the synonyms and related terms of these words, the main keywords to be combined into search queries were established, as depicted in Table 4. Inside the previously defined research scope, the aim is to have “a broad conception of what is known about the topic and potential areas where knowledge may be needed” before selecting search terms (Torraco, 2005) For that, “those sources most likely to contain a summary or overview of the key issues relevant to a subject” should be examined. (Baker, 2000) The field examined in this LR is very young and successful results when using a robot, with examples given 2.1, are recent even in this field (Lauwers et al., 2009; Summet et al., 2009). Hence the main sources examined were the latest literature reviews of the field. In order to identify these reviews, the search keywords from Table 4 were combined with a second set of review keywords. The keywords are synonyms of the word review, in particular “review”, “survey”, “summary”, “report”, and “analysis”.
12
Looking at the literature reviews, more specific concepts and ankles of view relevant to LR were identified and formed a third set of search keywords. Example concepts are “gui”, “tool” or “motivation”. Example ankles of view are “benefits”, “development” or “promising approaches”. In addition, some words such as “pdf” or “or” were added to the set achieve more precise search queries. A list of all keywords of the third set is attached in the appendix A.1.
Research Field
Application Field
Subject
Methodology
Location
education
secondary school
programming
robots
Germany
teaching
middle school
informatics
robotics
learning
high school
computer science
robot
didaktik
teenagers
informatik
age groups Table 4: Key Concepts as First Set of Search Keywords Source: Own Depiction
In a fourth step, keywords of particular interest for this LR were extracted during the LR search conducted with the three previous sets of keywords combined. These keywords include theories, tools, researchers, robots, application areas and methods. Example keywords are “alice”, “self-determination theory” or “calliope mini”. The keywords in this fourth set were combined with the three other sets and used only in unique keyword combinations. A list of all keywords of this set is attached in appendix A.2. Two lists containing all search terms combined from the four sets of keywords are included in appendix A.3. 3.1.4
Literature Search
The literature search process begins with the question where to search. Vom Brocke recommends to first identify top journals of the field and then begin the search in databases where these journals are included (vom Brocke et al., 2009). Yet, due to applied nature of the field, the relevance of journals over conferences is not that big in this LR. Also, the higher maturity of research papers published in journals has drawbacks, as the most positive findings in programming education using robots are very recent (Major et al., 2012). In addition, the field of programming education using robots is very small, only one of 61 papers in one of the big programming education literature reviews contains the word “robot” 13
(Vihavainen et al., 2014). The field does not have its own journals and conducting the search based on a selection of few journals of related fields involves the danger of omitting parts of the field. As a result, this paper decided to take a different approach and center the LR search process around the database of Google Scholar. The database of Google Scholar contains “approximately … 87% of the total number of scholarly documents found on the web” (Khabsa & Giles, 2014). Due to time concerns, the use of a single database allows for a more elaborate search process. In distinction to a single keyword search as recommended by vom Brocke, an iterative search strategy was used where new, more specific search terms are added during the process of reviewing of papers yielded by previous searches (vom Brocke et al., 2009). The aim of this strategy is to counteract the scarcity of research papers in the field of programming education using robots while still obtaining a good overview over adjacent fields. The search strategy has three stages and uses the four sets of keywords from the last subsection. First, literature reviews of the fields formed by the boundaries of our research depicted in Table 2 are examined. In this case, the main emphasis was on literature reviews of the fields programming education, programming education using robots and education using robots. Secondly, as pointed out in the last subsection, these keywords are combined with a second set of keywords representing specific ankles of view and more specific concepts. The aim here is to find papers in promising subfields as well as identify papers that remained hidden in the first stage. After learning more about which kinds of papers are relevant for this LR, areas of particular interest are investigated in a third stage using keywords noted during the second stage. This targeted search allows to discover small relevant niches of remote fields (such as intrinsic motivation) as well as a further investigation of specific, yet very important concepts to our field (such as the LEGO mindstorm robot). In a next step, vom Brocke recommends to conduct a forward / backward search (vom Brocke et al., 2009). Backward searches were conducted mainly to extract rare successful programming lessons using robots from literature reviews. A list of all backward searches is attached in Appendix A.4. As vom Brocke states, “Evaluation … means limiting the amount of literature identified by keyword search as well as backward and forward search to only those articles relevant to the topic at hand“ (vom Brocke et al., 2009). In order to do that, one has to “analyse their titles, abstracts or even full texts” (vom Brocke et al., 2009). In this LR, papers were included when a change in lesson design influenced lesson success positively or negatively. In few instances, a paper was also included when this relationship could be strongly anticipated. If the occurrence of this relationship could not be precluded after reading the title of the paper, the abstract of the paper was read. If the abstract contained the relationship, the paper was noted as relevant, otherwise the feature was subsequently sought for in the conclusion and results sections. In addition to this layer of quality assurance (as an analysis of reasons for lesson success infers an analytical approach of the paper), papers were rigorously selected by citation count. Only very few papers with unique value for this LR were selected despite one-digit or low two-digit citation counts in Google Scholar. In the end, 39 papers matched these criteria. 14
3.2 Results After all relevant papers were selected, concepts of all papers were summarized in a concept matrix, which forms a basis for an analysis of the field (Salipante, Notz, & Bigelow, 1982; vom Brocke et al., 2009; Webster & Watson, 2002). The content matrix then was augmented by some interesting properties of papers such as year of publication. The context matrix is attached in Appendix A.5. In the following, a short research agenda will point out appealing opportunities for future research as well as outline opportunities to complement underrepresented areas of this LR in future literature reviews. (vom Brocke et al., 2009) First, it can be noted that very few papers compare an approach using a robot to a control group. In a review of programming education using robots (mostly physical, but also simulated), of 34 papers already selected based on quality criteria, only 6 included a control group (Major et al., 2012). In addition, more than 50% of studies using physical robots incorporated expensive LEGO® Mindstorm robots (Major et al., 2012). Looking at a review of robots in education, 85% of all papers identified did not include a quantitative evaluation (Benitti, 2012). Finally, nearly half of the papers analyzed in this LR focused on CS education in college, as depicted in Table 22 and 23. Of those focusing on middle and high school students, many used robots only in extracurricular activities (Denner, Werner, & Ortiz, 2012; Williams, Ma, Prejean, Ford, & Lai, 2007). In order to pave the way for robots to a wide-scale use in the regular programming curriculum, we need quantitative research where cheap robots outperform a non-robot approach in teaching a regular high school curriculum. Regarding the limitations of this LR, due to the low amount of papers in programming education using robots, bigger parts of the papers reviewed did either not include a robot as a tool or programming as a LO. While this lowers the relevance of the findings to our specific case, it gives the possibility to discover strong theory on broader topics like education, programming education or education using robots. Still, there is a need for future “pure” reviews of the field of programming education using robots. Besides, the LR process did yield only small amounts of latest research. Of all 39 papers included, only four papers are younger than 2013 (Table 22, 23). The fact that the field learns fast from previous experiences, as in the instance of Lauwers and Fagin, gives even more cause to focus on recent research (Barry S. Fagin et al., 2001; Lauwers et al., 2009). The fast learning curve of the field is especially surprising that the field in the light of the variety of age groups, approaches and environments in which robots are used in programming education (Table 22, 23). This goes hand in hand with another limitation: Because of this diversity in success measures (grades, retention rate, pass rate, specific tests, questionnaires on students’ emotions) and research settings overall, the changes presented in this LR may have a more moderate or even negative impact in other research settings with different success measures. This indicates for future research to consider focusing on only one success measure or research setting to get more precise results. A drawback of this approach may be an insufficient amount of research in programming education using robots for only one success measure. 15
In contrast to the variety of success measures, the changes leading to these different kinds of lesson success are not that different from each other. In fact, all changes could be categorized into only four SFs without problems (Table 22 and 23, Figure 6). This indicates that when tests measure high scores for all four SFs, there is a very high probability that the overall lesson is successful.
34 24
21 16
Applicability
Confidence
Ease
Efficiency
Figure 6: Amount of Lesson Changes by Success Factor Source: Own Depiction
Looking at Figure 6, the first SF is applicability. This means that in a lesson programming concepts were applied to another context. Example contexts include movies, games and robots (Denner et al., 2012; Kelleher, Pausch, & Kiesler, 2007). Confidence in this case means how confident students feel in their abilities to finish a task that involves programming and robots. The SF ease is defined as how easy it is for a student with specific cognitive abilities to learn programming concepts through a specific task. All changes making it easier for a student to solve this task without learning less are included into this category. Examples are students getting a sheet of paper with the most important methods or example programs handed out to students (Lahtinen et al., 2005; Lauwers et al., 2009). Lastly, the changes included in the SF efficiency are changes that eliminate activities that have no or only a small contribution to achieving the LOs. Example are the use of finished robots instead of assembling robots in class or a simulator to test code on the computer before testing it on a robot (Barry S. Fagin et al., 2001; Wu, Tseng, & Huang, 2008). In the following section, we will elaborate on the changes of each of these four SFs in more detail.
3.3 Success Factors 3.3.1
Applicability
In a lesson, the application of programming concepts to another field makes a difference. As pointed out in 1.2, teaching programming in a context or in conjunction with media increases pass rates by 48% and 40%, respectively (Vihavainen et al., 2014). How does this happen?
16
One the one side, beginners have more ease applying their knowledge, for example by constructing a robot, than understanding concepts at a deeper level (Sartatzemi, Dagdilelis, & Kagani, 2005). In addition, applications like a robot help students to create mental models of concepts, for example loops or arrays, by visualizing them. These models lead to a deeper understanding of the concepts (Ben-Ari, 2001). On the other side, 1.2 states that the main reason for girls and the third most important reason for boys to study CS is its application to another field (Carter, 2006). Beyond that, also in college, a relevance of tasks to real life correlated with interest of students in a task (Lauwers et al., 2009). A more positive image of programming is one of three factors that make a student have an offensive approach towards programming, solving programming problems proactively instead of avoiding them (Schulte & Knobelsdorf, 2007). An obvious application in this case are robots. They have shown to improve various skills such as math, physics or programming (Benitti, 2012). As 1.2 points out, the tangibility of robots have shown to increase the interest of students up to the age of 12 (Shim, Kwon, & Lee, 2017). Beyond a curriculum focused around tangibility, a robot-based approach where students could play around with many different functions (sensors, actuators) of the robot lead to an increase in retention rate by 38% compared to a traditional, web-based approach (Wellman, Anderson, & Vrbsky, 2009). Still, there was one other application that lead to even higher retention rates than robots. When using the Alice visual programming environment, retention rates increased by 56% compared to the web-based approach (Wellman et al., 2009). In the Alice visual programming environment, students can create movies out of animals and objects such as pirate ships, and make them move with programming logic. By opinion of the author, Alice has an innate “ease of use” and provides a “positive programming experience” (Wellman et al., 2009). Another study reported that “after five to six hours of Alice each day, we still had difficulty getting them to stop, turn off their Alice worlds and logout at the end of the day” (Rodger et al., 2009). Also, they “found that the students used a large variety of ... computer science concepts in the [Alice] worlds they built in their free time” (Rodger et al., 2009). Building an Alice lesson around the concept of storytelling makes students not only spend more on their project overall, but also to focus more on programming that design (Rodger et al., 2009). “[Students] spent 42% … more time editing their programs and 54% … less time laying out their scenes” (Rodger et al., 2009). As percentages of girls in CS vary between 10% and 35%, targeting them in particular could be an efficient approach to have more students enroll in programming (Vegso, 2008). One way could also here be the Alice environment which attracted 50% females to a CS course (Rodger et al., 2009). Looking at the use of the Alice environment, several differences of interest between boys and girls can be established: While girls preferred to use natural elements, people and animals, boys preferred technical structures and special effects (Rodger et al., 2009). Apart from Alice, other approaches such as textile programming reach female rates up to 75% (Qiu, Buechley, Baafi, & Dubow, 2013). Finally, one way to connect programming to something relevant for students is a connection to other subjects in school. Based on their experience, Lindh & Holgersson state that “It is very important that the pupils can relate the material to their ordinary school work and their different 17
subjects” (Lindh & Holgersson, 2007). In addition to the benefits to both subjects by making them relevant in a different field, “Dijkstra proposes a … curriculum based on mathematical foundations” so students have a theoretical basis for CS (Dijkstra, 1989; Robins et al., 2003). In turn, programming with robots can also be a basis for successful education of other subjects, such as physics (Williams et al., 2007). 3.3.2
Confidence
A lesson’s LOs can only be achieved by students who are motivated. One of the three main factors influencing intrinsic motivation is a feeling of competence in students (Ryan & Deci, 2000). Students can only feel competent when they are confident in their abilities. This relationship between confidence and performance is confirmed by empirical data: Positive, reinforcing feedback that students received had a correlation of 30-50% with the creativity, feasibility and relevance of students’ solution after receiving the feedback (Wu et al., 2008). Still, this effect could be also caused by the fact that students receiving positive feedback perform better in general. One way to make students more confident is to create a comfortable environment. The first step is to not make the robot a burden: Students should have either enough time to build a robot, or work with robots ready for use in order to prevent frustration before even one line of code is written (Cliburn, 2006). A strategy ensuring enough time for all students to fix their robots problems is to make them have an own robot which they can work on outside of class (Barry S. Fagin et al., 2001). As a result, curricula that applied the one-student-one-robot strategy often reported robots outperforming a non-robot approach, in contrast to many other studies where a robot was used (Lauwers et al., 2009; Summet et al., 2009). This strategy in conjunction with lots of features of the robot (diverse sensors, speakers, LEDs, video camera) led to students even showing off their skills learned in class to friends (Summet et al., 2009). A basis for that is that students have enough space to play around, tinker and test different solutions when using the robots (Lindh & Holgersson, 2007). Yet, the playfulness of students can also be a burden: Students chose to implement their own designs instead of following the lesson instructions (Williams et al., 2007). A way to prevent this is to give students enough exploration time with the robots before starting a structured lesson curriculum (Williams et al., 2007). In another study, comfort level was determined as the most influential factor on success in CS on college compared to 11 others (B. C. Wilson & Shrock, 2001). In distinction to the aspects of comfort we focused on in the last paragraph, “comfort level” is defined here as a combination of anxiety level, perceived difficulty and perceived understanding of students (B. C. Wilson & Shrock, 2001). Computer anxiety has three roots: an operational root, a knowledge root and a psychological root (Howard, 1986). The operational root can be counteracted by giving students hands-on experience (Chien, 2008). The knowledge root is solved by acquiring computer knowledge, a general understanding of computers for everybody is also advised by other sources (Chien, 2008; Schulte & Knobelsdorf, 2007). Lastly, the psychological root refers to beliefs and attitudes of students (Chien, 2008).
18
Attitudes towards CS are a product of the self-image, world-image and habits of students (Schulte & Knobelsdorf, 2007). The self-image is heavily influenced by confidence, one way to support this is the use of robots for visualizing learning achievements of students (Mubin, Stevens, Shahid, Al Mahmud, & Dong, 2013; Schulte & Knobelsdorf, 2007). The world-image can be influenced by giving students a basic understanding, as suggested in the previous paragraph, or even an advanced understanding so students can become designers of CS programs instead of users (Schulte & Knobelsdorf, 2007). The differences in world image can be illustrated with the following two citations: “On my own I tried to install a new program or modem, but I always felt very unconfident. Apart from that, in 80% of the cases, it did not work as described in the manual. Then my prejudices concerning computers, which complicate matters instead of facilitating them, were confirmed” (Schulte & Knobelsdorf, 2007) “A one-week java class … raised the feeling in me that the computer would do anything I tell him” (Schulte & Knobelsdorf, 2007) Lastly, habits of programming are influenced by learning strategy, performance and problem handling (Schulte & Knobelsdorf, 2007). Regarding these factors, a distinction can be made between effective and ineffective novices or defensive vs. expansive learning (Robins et al., 2003; Schulte & Knobelsdorf, 2007). An ineffective novice fears learning new things, does not improve his problem-solving skills and avoids complex problems (Denner et al., 2012; Robins et al., 2003; Schulte & Knobelsdorf, 2007). These characteristics lead to problematic behaviors: students fear complexity and stop working, either try only one strategy when fighting bugs or use a brute-force approach without thinking, and perform poorly at designing programs and problem-solving (Denner et al., 2012; Robins et al., 2003; Schulte & Knobelsdorf, 2007). Over longer periods of time, these behaviors not only have a negative influence on students’ attitudes, but also on their speed of acquiring knowledge and overall performance, which in turn makes students even more desperate and inefficient in dealing with programming problems. Several strategies are put forward to solve this problem: First, students can be taught problem solving by seeing effective strategies of code creation in videos (Robins et al., 2003). Secondly, tools can be applied to check students coding strategies in the software editor and give feedback if needed (Denner et al., 2012). Thirdly, a teacher is advised to observe the problem-solving behavior of students to identify models, plans and goals the students are using (Robins et al., 2003). Finally, Robins et al. advise to aim at developing habits over goals, as “rather than focusing exclusively on the difficult end product of programming knowledge, it may be useful to focus at least in part on the enabling step of functioning as an effective novice.” (Robins et al., 2003). 3.3.3
Ease
As pointed out in 1.2, many students struggle with the abstract nature of programming. This subsection is all about making the introduction to programming as easy and soft as possible. We will present a plethora of changes, starting from easy ones like adjusting the software environment and working our way forward towards more challenging ones like group work or even teacher education. 19
Regarding the choice and design of the used programming language, the key is simplicity. As the aim is not to teach a language, but rather how to program in general, the lesson should be designed that students spend as much time designing programs rather than setting up the environment. In practice, this means using a mini-language with few programming commands that students have ease to deal with (Brusilovsky, 1994). The commands students used most when programming freely are “user interaction, loops and conditionals” (Maloney, Resnick, Rusk, Silverman, & Eastmond, 2010). Command types that produced even less bugs than loops and conditionals are applied ones like I/O, initialization or update (Robins et al., 2003). Beyond that, in a study using few powerful commands to describe complex robot behaviours, robots outperformed a non-robot approach (Summet et al., 2009) (Brusilovsky, 1994). Another study edited method names in a way that they were intuitive, for example “saySomething” instead of “System.out.println” as used in the programming language Java (Lauwers et al., 2009). Taking this approach, not only did students spend much more time designing programs than setting up the robots and understanding the language, but also achieved a significantly better mean grade than in the previous years (Lauwers et al., 2009). In order to make an easy programming language even more attractive, the next step is to provide an attractive programming environment. Especially because the use of a robot already adds an additional layer of complication, the interface should be as straight-forward as possible (Cliburn, 2006). The use of a high-level environment called Scratch gave students the feel that they are not programming (Maloney et al., 2010). Scratch has several functionalities that normal programming languages don’t have, such as “eliminating syntax errors, providing feedback about placement of command blocks, and giving immediate feedback” (Maloney et al., 2010). A combination of a high-level version of Java in conjunction with the graphical environment Alice also lead to a “dramatic jump of at least one letter grade over test scores in more traditional sections of the same course” (Dann, Cosgrove, Slater, Culyba, & Cooper, 2012). Also, Mitnik et al. attributed part of the success in teaching with robots to a high-level environment ignoring specific robot issues (Benitti, 2012; Mitnik, Nussbaum, & Soto, 2008). A high-level graphical environment can also help in visualizing models of programming concepts or application domains (as the robot). Often “students do not have an effective model of a computer” (BenAri, 2001; Robins et al., 2003). Not only should a programming visualization display models, but also be adapted to the educational content and teaching methodology used (Sorva, Karavirta, & Malmi, 2013). Beyond an intuitive programming environment, everything should be made to ensure students understand the concepts taught. Numerous lesson helps have proven to achieve that. Studies where robots outperformed a non-robot approach in motivation or performance used different paper sheets with the most important code commands, all code commands, system settings and declarations as a visual help for students (Lauwers et al., 2009; Wu et al., 2008). In one of these studies, the teacher also solved example problems on the board to give students a deeper understanding of applied coding (Wu et al., 2008). Beyond that, in a transitional stage between teacher instruction and autonomous work of students, example programs that students can easily apply to their programming tasks can be handed out to students (Wu et al., 2008). Example programs were rated as the by far most helpful material compared to exercise questions, programming books and interactive visualizations by both students and teachers (Lahtinen et al., 2005). 20
To further increase the chance of student success, students can join their forces and work together in groups. Working together in pairs did lead to nearly double the programming speed with a significant increase in code quality (McDowell, Hanks, & Werner, 2003). Another study noted working in groups as one of the three main reasons for curriculum success (Cliburn, 2006). Still, working groups should not be too big and not more than 2-3 students should share one robot (Cliburn, 2006; Lindh & Holgersson, 2007). It also had an influence of the students’ attitude towards programming, as one girl stated: “I never really used to like computers and now I don’t like really hit the computer when I’m mad. I sort of try and find out what’s wrong.” (McDowell et al., 2003) Going one step further in teamwork, more structured review systems can ensure quality code even if one group fails their task. Peer tutoring among students does not only gives many more students the possibility to have a personal teacher, but also helps the peer tutor to understand the content better while explaining (Hattie, 2008). Therefore, it has also a positive influence on lesson achievement across many educational disciplines (Hattie, 2008). Peer code review of other students’ code on the computer is another successful model to improve the learning outcomes (Wang, Li, Feng, Jiang, & Liu, 2012). Students reported a satisfaction rate of over 80% (Wang et al., 2012). Another very intensive and successful approach is extreme apprenticeship, where an instructor first explains models of the problem domain and then lets a student solve the problem autonomously while giving hints to the student (Vihavainen et al., 2014). Using this approach, pass rates on average increased by around 37% (Vihavainen et al., 2014). The basis for a good execution of all these lesson helps is a qualified and motivated teacher. Especially when using robots and software environments, this kind of teacher is not always the norm. The amount of teachers trained in CS is not sufficient, as well as the amount of opportunities teachers have to learn computational thinking and programming (Benitti, 2012; Carter, 2006; Lee et al., 2011; Vollstedt, Robinson, & Wang, 2007). As a result, many teachers feel uncomfortable in teaching programming (Benitti, 2012; Vollstedt et al., 2007). In a study with negative results of a robot group compared to a control group, instructor effectiveness was the criterion where students saw the biggest gap between the robot and the non-robot group (B. Fagin & Merkle, 2003). Also, students’ feedback improved with instructor experience (B. Fagin & Merkle, 2003). Beyond the qualifications, for teachers to be motivated they also need to be convinced of the usefulness of a lesson tool to use it (Sorva et al., 2013). Reasons for teachers to not adopt visualization tools are issues with learning the tools, integration into the course, and doubts about tool efficiency (Sorva et al., 2013). One example concept supported by tools teachers enjoy working with is blended learning (Boyle, Bradley, Chalk, Jones, & Pickard, 2003). 3.3.4
Efficiency
During the last subsections we advocated for using media and tools in education and outlined numerous advantages. Yet, an inclusion of many different tools holds the potential danger that students spend more time handling tools than on actual programming. How to make a curriculum where students are not only interested, but also fast learners? 21
Different visual approaches like games or robots were less effective in teaching programming compared to a traditional approach (Barry S. Fagin et al., 2001; B. C. Wilson & Shrock, 2001). The biggest problem with robots is the long feedback loop: students need much longer to test their code with a robot than in a pure software environment (Barry S. Fagin et al., 2001; Flowers & Gossett, 2002). All code commands have to be transmitted to the robot and then the students have to wait while the robot is driving or playing sounds. There are two ways to use a robot efficiently despite this drawback: either increase the speed of testing the code or include tasks where students spend a small percentage of time on testing compared to other activities. We will describe changes of these two approaches in the following paragraphs. The main activity to increase testing speed is to allow testing without a robot: students should first test their code in a simulator and, when they are sure their solution is working right, test it on the robot (Barry S Fagin & Merkle, 2002). Inside the software testing environment, a language that does not need to be compiled or the use of a dynamic programming language are ways to increase the testing speed (Sullivan, 2008; Summet et al., 2009). In addition, the abundance of complex mathematical calculations speeds up the testing loop even further, especially when testing on a robot (B. Fagin & Merkle, 2003). Students spend less time testing when they participate in project-oriented activities where they think on their own (Flowers & Gossett, 2002). With robots being an excellent practical basis for project-based learning, the combination of both allows for students to set the methods and pace they want to solve the problem with (Sullivan, 2008). This way, each group of students can choose the method they work most efficiently with (Sullivan, 2008). Besides, allowing students to learn in their own pace does have a positive influence on achievement (Hattie, 2008). Different learning paces can also be implemented in curriculum that is not project-based. Different groups of students (e.g. novice, advanced and retake students) can use for example programming languages with different levels of abstraction (e.g. Alice, Scratch, Python and C) (Álvarez & Larrañaga, 2016; Sorva et al., 2013). No matter how robots are used, there are some basic rules to cut out unnecessary lesson time that apply in any case. In order to prevent the robots from being an overhead, robots should be thoroughly tested and constructed before class by the instructor (Lauwers et al., 2009; Wu et al., 2008). One example is to check whether the robots interfere with the local wireless network (Lauwers et al., 2009). Also, when learning programming concepts, students should be only expected to learn few code commands to prevent wasting time on learning many commands specific to only one programming language (Wu et al., 2008). Finally, the best lesson in theory can fail in-class. With clear LOs, a lesson can be constantly evaluated to be structured more efficiently. A formative evaluation is the third most influential positive factor on lesson achievement in overall education (Hattie, 2008).
3.4 Summary Table 5 summarizes all changes from the last section with the amount of how often they were cited, sorted by SF. These changes have been the basis for the design process specified in the next chapter. 22
Applicability
Confidence
Ease
Efficiency
many different sensors
robots ready to use or
teachers knowledgeable
different programming
& actuators on a robot
enough time for robot
about computational
languages for different
construction
thinking
groups (2)
application of
videos visualizing
high-level environment
no complex
programming to
strategies of code
ignoring robot issues (2)
mathematical
another field
creation
special effects,
teacher analyzing
technical structures for behavior of students
calculations on robot teaching models with high-
use of robots in projects-
level environment (2)
based learning
boys curriculum related to
advanced understanding
sheet with most important
checking network
school topics
of programming
code commands
reliability of robot
natural, social
learning objective:
teachers comfortable with
software simulator for
elements for girls
effective novices
programming
code testing
positive image of
hands-on experience with user interaction, loops,
non-compiled
programming
programming
conditionals
programming language
programming in a
general understanding of
visualization suitable for
free choice of learning
context
programming
content
method
relevance of
code quality evaluation
teacher solving problems
dynamic programming
programming
by students
on board
language
curriculum based on
feeling of competence in
teacher convinced of lesson
project-oriented
math
students
tools
activities
storytelling with alice
space for playing with
sheet with all code
different learning paces
robots
commands (2)
programming with
own robot for each
groups of max. 2-3 students pre-constructed robots
robots
student
(2)
teaching mental
many features of the
I/O, initialization, update
short feedback loop (2)
models
robot
practical programming
time for exploring robots
sheet with system settings
few code commands
curricula for girls
comfort level of students
instructor effectiveness (2)
lesson evaluation
textile programming
achievement visualization
intuitive method names
media computation
reinforcing feedback
extreme apprenticeship (2)
use of robots
few powerful commands
tangibility
graphical environment
alice (6)
use of mini-language
lack of CS teachers
blended learning
easy interface
peer tutoring
working in groups example programs (2) pair programming peer code review
Table 5: All Changes Sorted by Success Factor (Number of Occurrences) Source: Own Depiction
23
3.5 Artefact research The artefact research had two objectives: first, to find a robot that is cheap, handy for both users and programmers, reliable, and usable on different platforms. Second, to find a software system matching the same requirements as well as working seamlessly with the robot. I started conduction the search for a robot only based on the criterion of price. Only few robots were cheaper than 50$, what would allow a small amount of students to work with one robot in class in an ordinary school. Among these robots, there was really one robot available to buy that had a somewhat appealing design, which was the AERobot ® (Rubenstein, Cimino, Nagpal, & Werfel, 2015). After a month of work with the robot, the robot was classified as unfit. The only software to use with it was in a low-level language, barely commented and not cross-platform. The robots’ movements and calibration did not work once reliably despite numerous attempts. In addition, there was no support on the code. After one month of work, my supervisor told me about a new educational robot to be launched in Germany. The robot Calliope Mini® depicted in Figure 7 fulfilled all the requirements specified at the beginning of this section (Jørn Alraun, 2017).
Figure 7: Calliope Mini Robot Source: Own Depiction
The search for a software environment was considerably shorter and easier. My language of choice was Python. After seeing which environments were programmed in Python and worked with the Calliope Mini cross-platform, my choice was to build my editor based on the Mu Programming Editor depicted in Figure 8 (H.Tollervey, 2017). 24
Figure 8: Mu Programming Environment Source: Own Depiction
25
4 Design Process The second research question reads: What would be an approach to create a successful and measurable lesson with the help of a robot as a tool that would also contribute to the research field? By the definition of this thesis, the lesson uses a robot as a tool. This leaves three criteria that the created lesson has to fulfil: it has to be successful, measurable in terms of how successful it is, and contribute to the field of research. To ensure these criteria being met, several activities are undertaken, as depicted in Table 6.
Criterion
Activities
Successful
Building of SF changes in artefact design, pre-testing
Measurable
LOs
Contributing to the Field
Building and evaluating SF and OAR in artefact design
Table 6: Artefact Design Criteria and Activities to Achieve Them Source: Own Depiction
The lesson success chance will be heightened by implementing many SF changes in all stages of the artefact design and iteratively evaluating and improving the artefact design based on how well it reaches the LOs. These LOs are clearly stated and can be measured in terms of how well they have been met. To contribute to the field of research, implemented SF changes and OAR in all stages of the artefact design will be evaluated for their usefulness. The whole design process started with fixed design boundaries specified by the thesis topic, the existing artefacts, and the environment in which the robot will be tested. Based on this basis, suitable LOs were selected. Following that, an artefact to reach these LOs was designed in an iterative search process (Von Alan et al., 2004). In this process, the artefact including lesson and software was developed, tested and consecutively improved(Von Alan et al., 2004). In distinction to the process specified by Hevner et al, laws, means and ends were not analyzed in a mathematical way but rather implicitly due to the scope of this project and unpredictable nature of educational settings (Von Alan et al., 2004).
26
The iterative search process is depicted in Figure 9. In a first step, a lesson plan was constructed and executed based solely on existing artefacts. Through testing it, the most efficient improvements in software to influence lesson success were established (SF “lesson evaluation”). After improving the software, a second lesson plan was constructed tied to the capabilities of the improved software. A pilot test of this improved software-lesson-plan combination led to some minor final changes. The product of these changes was the final lesson artefact that was tested in a real environment.
Figure 9: Iterative Search Process Source: Own Depiction
In the following, we will first describe the design basis and the choice of LOs forming the basis of the iterative search process. Then, the iterative search process will be illustrated in four stages, one stage for each created artefact depicted in Figure 9. Finally, a last stage describing the immediate lesson preparations and planned instructor actions during the lesson will be appended.
4.1 Design Basis The topic of this thesis sets the boundaries that the lesson has to teach programming and use a robot as a tool (SFs “programming with robots”, “use of robots”, “media computation”). The primary trade-off when selecting a robot was between usability, cost and functionality. In our case, we selected a robot that was ready-to-use, costed 35 Euros and had still enough functionality both in input (buttons, temperature) and output (pins displaying text and figures, sound, light color) (SFs “robots ready to use”, “tangibility”, “many different sensors”, “many features on the robot”, OAR “use of cheap robots”, “reuse of elements”). The fact that the robot is used in many other educational programs makes the reuse of the software environment probable (OAR “common robotic platform”). The feedback loop is also fast compared to other robots: The time to load an average program on the robot was ~5 seconds and not much longer for complex programs (SFs “no complex mathematical calculations on robot”, “short feedback loop”). The whole lesson builds on a ready-to-use robot programming environment, which gives the possibility to trigger some, but not all sensors and actuators of the robot. For creating an offline environment independent of the internet connection, the Mu® environment was the simplest 27
pre-configured solution using the widespread language Python (OAR “reuse of elements”, “dynamic programming language”, “common robotic platform”). As the aim was to create an easy programming environment for beginners, it was reasonable to start with an environment that is already easy-to-use. Due to financial reasons, only two robots could be afforded. As pair programming and group work were highly advocated in the SFs, two pairs of students sharing a robot allowed for a planned number of 8 students in the course (SFs “pair programming”, “groups of max. 2-3 students", “working in groups”, OAR “pair programming”). To exclude teaching and explanation overhead, the developer of the lesson unit was planned to teach it himself (SFs “teacher comfortable with programming”, “teacher convinced of lesson tools”). When selecting the target audience, students should not know about programming (as it would be hard to support various levels of skill) and be easy to engage. Because the lesson was tested in Bavaria, where programming is already taught in middle school and kindergarten / primary students are hard to teach in programming, students aged 10-12 were chosen for the program. The curriculum length was planned to be short in order to evaluate the whole lesson in sufficient quality. As the students were planned to be taught in school lessons, the length of a lesson was set to 90-180 minutes.
4.2 Learning Objectives Having a short curriculum for inexperienced students, it makes more sense to transport a positive image of programming rather than focusing on indoctrinating as much knowledge as possible. Thus, the main objective was that students gain a sense of achievement, see the impact of programming in the real world, and feel confident through gaining hands-on experience in programming (SFs “achievement visualization”, “relevance of programming”, “feeling of competence in students”, “hands-on experience with programming”). To achieve all three aims, the LOs were chosen to be application-focused. In programming, there are three very popular procedures that are widely applied and can create value even in very simple programs. The first is the if-loop, the second the for-loop, and the third the for-loop with an iterator. It was decided that learning the application of these three procedures is our primary LO. To support that, the necessary knowledge and understanding for applying the procedures were taught as subordinate LOs. Knowing how to apply these procedures, students should learn how to create programs on their own. The creation of programs based on a concept selected by students is also a subordinate LO. The techniques how to program make a big difference in student performance (SFs “effective learning strategies”, “LO: effective novices”). That’s why, when students did a sufficient
28
amount of tasks, they should evaluate the efficiency of the process they use when writing programs. The evaluation of their programming procedure thus also is a subordinate LO. All of these LOs can be categorized by the revised taxonomy of learning objectives by Bloom (Krathwohl, 2002). Main LOs are tainted in dark grey, subordinate LOs in light grey: Remember Understand
Apply
Analyze
Evaluate
Create
Factual Conceptual Procedural Metacognitive Table 7: The Revised Taxonomy of Learning Objectives, Left Column: Knowledge Type, Top Column: Type of Cognitive Process. Source: Based on (Krathwohl, 2002)
Beyond LOs tied to knowledge, the lesson also aims at students being motivated for future projects in the field of programming and confident in their abilities (SF “feeling of competence in students”, “positive image of programming”).
4.3 Stage 1: Lesson Prototype The decisions made to construct the first lesson prototype can be categorized into three areas: lesson structure, teaching methods and curriculum. The lesson structure was designed so that students can learn at their own pace (SFs “comfort level of students”, “different learning paces”, “feeling of competence in students”, “time for exploring robots”). In particular, each pair of students got their own task sheet with all tasks and explanations of the concepts in the lesson. After a short introduction, they were able to immediately start working on the robot on their own (SF “practical programming”). The teacher took the role as a facilitator and assisted the students in learning. Based on this structure, a main objective becomes reducing the teacher assistance time per student to make the lesson scalable to a large number of students. The basis for this are highquality self-explanatory lesson instructions. Each pair of students received a set of paper sheets called the task sheet. Each task sheet contained four tasks split into two sections: “level”, which explains the necessary prerequisites for “challenge”, which described the tasks to be done by students. In “level”, the process to run the robot as well as all necessary procedures, commands and concepts are described (SF “sheet with system settings”, “sheet with most important code commands”, “sheet with all code commands”). They are described using photos of the robot, visual explanations of software commands with input variables, and examples of the programming commands (SF “example programs”). 29
The curriculum revolved around teaching the three programming procedures described in 4.2. Despite the aim of not using too many commands, some robot-related commands were necessary to use the procedures in a relevant way with the robot (SFs “few code commands”, “relevance of programming”). Learning robot-related commands before the procedures made students see results of their work early in the curriculum (SFs “positive image of programming”, “achievement visualization”). Students were first taught how to put code on the robot, then how to display lights on the robot, and finally the if-loop and the for-loop (SFs “user interaction, loops, conditionals”, “I/O, initialization, update”). The first lesson prototype was tested with a student aged 11 in an informal environment. The student completed all tasks except the last one teaching the for-loop procedure due to time reasons. During the lesson, feedback from the students as well as observations of the instructor were collected in a list of feedback items which is appended in Appendix B.1. Most of the feedback items were small issues regarding the curriculum. Although the student understood most of the curriculum, he stated that “its hard to understand the for-loop section, but I would not know how to explain it better myself. Its just a complicated concept”. Still, the structure and task descriptions of the curriculum were partly unclear. Beyond that, there were two major problem areas that can be solved through building software: syntax and debugging. One the one hand, an estimated over 50% of lesson time was spent solely on debugging syntax errors. On the other hand, a major portion of time was used on debugging conceptual errors as there was no easy-to-understand debug help. Based on these two big problems accounting for a majority of lesson time and frustration of the student, the two corresponding solutions are a nearly syntax-free block environment as Scratch ® (Maloney et al., 2010) and a software simulator of the robot to test code in addition with debugging tools. An additional advantage of the software simulator would also be a faster testing loop (as the code needs to be no longer loaded on the robot). Finally, it was decided to focus on building a block environment as it removes most of the debugging work through drastically reducing the main error source, syntax.
4.4 Stage 2: Improved Software
Figure 10: Example Block Source: Own Depiction
Based on our aim of removing syntax, several design principles were applied. To remove unnecessary clutter like import statements, semicolons and unused commands, only commands used in the lesson were represented through blocks as in Figure 10 (SF “few powerful commands”, “use of mini-language”). To further reduce writing time and students having problem with the keyboard, students were required to only write the values that the command is executed with in the command block instead of writing the whole command (SF “graphical environment”). For example, in Figure 10 the light-red area was to be filled out, while the command “Schreibe” was already written on the block. Beyond that, testing the prototype 30
showed that students had a different understanding of for-loop indices that programmers. Usually in programming, the loop starts with the index 0 and ignores the last index specified. In contrast, our loop logic included the last index specified and all indices (e.g. the indices to show lights in specific columns on the robot) started at 1 instead of 0. For example, the Figure 11 block would display the lights in all five columns. The next objective was to make the command blocks easy to understand. First, all commands were translated into German, as the curriculum is in German. In addition, the command names were application focused, which means that they describe what the result of the command is instead of what happens to the robot at a lower level (SFs “high-level environment ignoring robot issues”, “intuitive method names”). As an example, the command “display.scroll” was replaced by “write” which means “Schreibe” in German, as depicted in Figure 10. Only in instances where students benefitted from a more precise control over the robot, commands were more specific. For example, Figure 11 shows how five lights on the robot can be displayed through a for-loop, while Figure 12 shows the same command in another programming environment. Although the command in Figure 12 is easier to understand, the command in Figure 11 is shorter and teaches students how to effectively use a for-loop.
Figure 11: For-Loop to show
Figure 12: For-Loop to show line of
line of lights
lights
Source: Own Depiction
Source: Based on (Fraunhofer IAIS, 2014)
Figure 13: User Interface of Educational Software Source: Own Depiction
31
When students understand what the commands do, the next challenge is that they are able to build a program. The design was largely inspired by the Scratch® programming environment and is depicted in Figure 13 (SF “graphical environment”). There is an area at the left of the editor where all available blocks are listed. In the middle, blocks can be appended to a stack (In Figure 13 3 blocks are appended). When clicking on the button on the right, all commands on the stack are executed. When moving a block back to the area on the left, the block is deleted. Through listing these three elements from left to right, students intuitively see in where to start and where to finish building a program. Blocks can be moved by drag-and-drop and appended by dragging them under another block. When a block is moved under a for- or if-loop, it is indented after appending (lowest block on stack in Figure 13). If-conditions can be dragged inside if blocks and are embedded in the ifblock if they are dragged on the if-block condition space (as seen on the grey if-block on the left side on Figure 13). Through intuitive drop areas generous in size students should have no problem connecting blocks with each other. All fields in blocks that can be filled with text, numbers or conditions are approximately in the size of the content they are filled with. This is not only esthetical, but also give students clues about whether they filled in the right content. Lastly, all command blocks were colored based on their command type (SF “visualization suitable for content”). For example, all logical command blocks were dyed in neutral colors like grey or black. Also, command blocks interacting with the same sensor/actuator of the robot were given the same color.
4.5 Stage 3: Improved Lesson After designing the software, an improved lesson was build based on the improved software environment, feedback from the first lesson, and some more SFs. All design decisions based on feedback of the lesson prototype (Appendix B.1) are denoted with F1 and the number of the feedback item in Appendix B.1. An example would be “F1 - 2” for item no. 2 in Appendix B.1. Looking at the feedback from the prototype lesson, most problems related to the curriculum were regarding unclear tasks. The challenge here is to describe all tasks and concepts extensively without overloading students with educational content. The approach taken in this case is a slim task sheet booklet with all explanations of concepts, procedures, commands outsourced to two other small booklets. The advantage of this approach is the ability for extensive descriptions as the main task sheet booklet only shows the basic tasks and refers to all details to the other two booklets. These extensive descriptions lead to the teacher answering fewer questions (SF “instructor effectiveness”). On the flip side, students need some time at the beginning of the lesson to understand the three-booklet strategy. This disadvantage was also confirmed in future feedback evaluations. In the following, the design decisions of all three booklets will be explained in more detail. 32
Figure 14: Example Level Source: Own Depiction
The first booklet contains all tasks of the workshop and their solutions, which makes it the most important booklet. To show clearer that “levels” used in the prototype lesson are tasks students have to do, the word “level” was used as a headline with bigger text size (Figure 14). Each level has a task description of the left side underneath and a solution description on the right side underneath (F1 - 21) (Figure 14). Because students were confused by page breaks in the middle of a level, each level was located on one page (F1 - 15). Each task description had exactly the same structure (F1 - 7). First it specified what section to read in the other booklets to gain the knowledge needed to do the task. After that, it defined the goal of the task and sometimes which procedures have to be used to do it. The solution description contained a picture of how the robot looked when the right commands were executed (F1 - 9) (Figure 14). When the solution contained actions (e.g. clicking a button on the robot) needed to generate the output, these actions were described in addition.
Figure 15: Example Block Description Source: Own Depiction
33
The second booklet contains explanations of how all needed command blocks are used (SF “sheet with all code commands”). All commands were explained in the order they appear in the curriculum so students know where to search for a command (SF “sheet with most important code commands”). Each command was explained in the same structure as extensively as possible, but not redundantly (F1 - 7): In the beginning, a short sentence described what the command does (Figure 15). Then, the command block with all input values is described based on a picture (Figure 15) (F1 – 17). Sometimes it was explained in more detail how the values entered relate to the output (F1 – 10, 17, 18). In the end, one or two examples are shown with command blocks on the left, an blue arrow, and robot pictures on the right (Figure 15) (F1 – 9). When the example needed actions to generate the output, these actions were visualized in addition. The third booklet contains explanations on how to create a program out of blocks and how to run this program on the robot (F1 – 6). This was not included only in the first level in the first booklet since it might confuse students that the instructions to run a program only apply to the first level (F1 - 19). As the instructions to run a robot are not completely intuitive, many pictures and arrows were used to show where students should click. The instructions are also held simple: All errors that may occur are rare enough to be explained by the instructor and would complicate this booklet unnecessarily. In all booklets, the quality of pictures was improved. The biggest correction was making the position of lights on the robot clearly recognizable by photographing them in a way that they do not flicker (F1 - 16). To create a relevant curriculum (SF “relevance of programming”), all tasks were given more meaning if possible (e.g. displaying the check sign instead of a random sequence of points). To complement that, each level had a name referring to the symbols and structures displayed in its solution (e.g. “two stripes” when two rows of lights were displayed). To make the curriculum more appealing, the tasks were no longer displayed in a table format. The tasks were now structured enough so that elements could be clearly distinguished from each other without the use of a table structure. In addition to all these changes in structure and format, there were also changes to the curriculum itself. Each of the following changes that also adds content to the curriculum also serves the purpose to fill 90 minutes of lesson time and prevent that students run out of tasks. One problem of the curriculum is the for-loop. Students have a hard time understanding this procedure compared to the if-loop or commands displaying output on the robot. To solve this problem, an additional task with an easier for-loop was introduced before the previous task forloop task. The easier for-loop has only the capability to execute an action a given number of times, but does not contain an iterator. It also enriches the curriculum as it is a new procedure to learn. In a similar fashion, also the concept of light strength (the strength of the lights on the robot display) was added to enrich the curriculum. In contrast to the lesson prototype, where it was anticipated to be complicated, it has shown to be easy to grasp. To make more use of the time 34
students spend learning the for-loop procedure, an additional, more advanced task based on the for-loop was added at the end of the curriculum. During the last tasks, students should evaluate their performance based on their experiences to be better prepared for the final, hardest tasks. On the one hand, a session was introduced where both students working at each computer should give each other exclusively positive feedback to heighten their confidence (SF “reinforcing feedback”). On the other hand, they should evaluate mostly their own performance, but also the performance of their partner later in the curriculum to evaluate their work process (LO procedural knowledge through evaluating, OAR “effective learning”). At the end of the curriculum and after all basic procedures and concepts were taught, two tasks with a rising degree of creativity were appended (LO conceptual knowledge through creating, SFs “project-oriented activities”, “use of robots in project-based learning”, “free choice of learning method”). The first one allowed students to build a device that had to measure and display temperature in the way they want to. This allowed for an exciting task while also making use of the temperature sensor. The second one allows them to build anything they want. Each task includes an example of what to do for less creative students (SF “example programs”). The improved lesson was tested with a student aged 9 in an informal environment. The student completed all tasks up to the last for-loop task, thus learning all procedures and concepts. During the lesson, feedback from the students as well as observations of the instructor were collected in a list of feedback items which is appended in Appendix B.2. The results of testing the improved lesson were as expected. As in the prototype lesson test, the student struggled to get along with the task format and running programs on a robot in the beginning. Then, he progressed even faster through the middle part than the student in the prototype lesson test. Lastly, he struggled with the for-loop concepts similarly to the student in the prototype lesson test. The big difference is that the student in the second test spent most time thinking about concepts instead of wasting time on syntax errors. In distinction to the previous test, the issues observed were nearly only details about software and hardware problems or minor issues with understanding the curriculum. Looking at the tasks, there is a tradeoff between teaching many concepts and teaching problem-solving. In harder tasks with much longer thinking time (e.g. the second for-loop task), the student discovered a different approach to solve the problem. On the contrary, he solved some of the first tasks in approximately 30 seconds after understanding the concept. In the improved lesson, easy tasks were given first to make students more confident, followed by more complex tasks when students were familiar with the concepts and procedures. This balance between learning concepts and problem-solving was retained. Beyond the introduction of a robot simulator on-screen as well as a tool for debugging, one main open issue are even better understandable concept and procedure explanations. To increase the quality of the explanations even further, student behavior during the tasks has to be analyzed in great detail which is out of scope of this thesis.
35
Lastly, time especially in the beginning of the lesson can also be saved by making the interaction between all three booklets even easier. The easiest way is to explain how they work together at the beginning of the lesson. An advanced approach is displaying all booklet content on the computer screen in a more integrated way.
4.6 Stage 4: Final Lesson and Software In a last step, additional teacher instructions were added on how to guide the lesson. In addition, minor changes in both software and curriculum were undertaken to address the detailed problems discovered in the test of the improved lesson. All changes based on the feedback of testing the improved lesson (Appendix B.2) are denoted with F2 and the number of the feedback item in Appendix B.2. An example would be “F2 - 3” for item no. 3 in Appendix B.2. Looking at the lesson structure, the lesson was decided to start with a short introduction of the instructor to take away the distance between students and the instructor (SF “comfort level”). After that, the purpose of each booklet was to be explained to prevent unnecessary questions and confusion of the students in the beginning of the lesson (F2 - 1). To free up teacher time for answering more sophisticated questions, the teacher was to advise the students to read the texts in the booklets thoroughly. During the lesson, the task of the teacher was to answer questions of the students. Beyond that, the behavior of students was to be analyzed so that the teacher can adjust his help to the individual needs of the group (e.g. asking weaker groups more often if they need help) (SF “teacher analyzing behavior of students”). In some cases when students failed repeatedly at a hard task, the teacher actively assisted in finding the solution, similar to the practice of extreme apprenticeship (SF “extreme apprenticeship”). To heighten the confidence of students, students were praised after successfully completing a task (SF “feeling of competence in students”). To make sure that no students run out of tasks, an additional booklet with three additional tasks was designed to be given to students that finished all other tasks. Some minor changes were also done in the other three booklets. As the button B on one robot was broken, this button was replaced by the button A on the task, also a picture of button A was included to show where the button is on the robot (F2 – 9, 10, 11). To help with an intuitive start, the example block in the explanations “how to run a program” was changed to the block used in the first task (F2 - 3). Also, each task was placed on one page to prevent students from overlooking parts of the task (F2 - 5). Looking at the software, a bug was fixed where a field could be dragged out of a condition block (F2 - 7). Also, an error message that was more disturbing than helpful was removed (F2 - 4).
4.7 Stage 5: Lesson Preparation Directly before the lesson, some preparations were made to create an appealing learning environment. 36
Computer mice were used so that students have no problems with dragging blocks in the software environment (F2 - 2). The mice with the computer and the robot were arranged in a way that students had enough space to spread all the booklets and move around the robot (SFs “space for playing with robots”, “comfort level of students”).
37
5 Artefact Description This section will explain all artefacts created: The educational software as well as the install guide, lesson materials and lesson guide. The lesson guide explains how a lesson with the lesson materials and using the educational software will be conducted. The install guide contains information about how to install the educational software on computers. Each artefact will be described briefly for users of the artefacts. Then, more detailed information about the design principles and suggestions for future improvements will be put forward for people who intend to improve the artefacts.
5.1 Educational Software 5.1.1
Use Guide
The concept of the educational software is based on a web-based software editor for children, Scratch ® (Maloney et al., 2010). It enables students to execute a series of commands on a robot.
Figure 16: Example Block Source: Own Depiction
Figure 17: Software Editor Source: Own Depiction
The editor contains three areas: the command box, the command stack and the background. Students can drag blocks, as depicted in Figure 16, representing commands from a command box at the left of the editor. On the right of the editor, blocks can be appended and arranged on a command stack in the order in which the commands should be executed on the robot. The rest of the editor is a background area where blocks can be dropped off to edit the values of the command the block stands for. The background area also contains a flash button on the right,
38
that, when clicked, executes all commands from the commands stack on a robot that is connected to the computer. The command box depicted on the left in Figure 17 contains 8 command blocks. Each block can be dragged an infinite amount of times to be dropped on the two other areas. When a block is dropped on the command box, it disappears. There are three types of blocks: non-logical blocks, logical blocks and condition elements. Nonlogical blocks are blocks that execute a simple command. Logical blocks are blocks that execute all commands appended to it with a specific logic. Condition elements return true when a condition is met and can be inserted into the If-Loop-Block. Detailed descriptions of all blocks are depicted in Table 8. Non-logical Blocks
Logical Blocks
Condition Elements
The Display-Text-Block displays
The If-Loop-Block executes the
The Temperature-
text on the robot.
commands of the appended blocks
Condition-Element
when the condition element inside it
returns true if the
is true.
temperature is bigger, equal or smaller than a given value.
The Light-Up-Pin-Block lights up a
The For-Loop-Block-Without-
The Display-Condition-
pin in a specific row and column with
Variable executes the commands of
Element returns true if a
a specific strength.
the appended blocks a specific
pin with a specific column
amount of times.
and row on the display lights up in a strength bigger, equal or smaller to a given value.
The For-Loop-Block-With-
The Button-Condition-
Variable executes the commands of
Element returns true if a
the appended blocks from a range of
button on the robot is
indices.
clicked.
Table 8: Table with all blocks Source: Own Elaboration
The values with which each command block is executed can be entered in the lighter fields on the block. In the case of the If-Loop-Block, a condition element has to be filled out and dragged into the lighter field inside the If-Loop-Block.
39
The command stack depicted on the middle right in Figure 17 starts with an immobile purple block (at the top of the series of blocks on the middle right). Blocks can be appended by being dragged directly beneath another block in the command stack. All elements that are appended to a logical block are indented. When a block is dragged under a logical block or a block appended to a logical block, it is also indented and appended to the corresponding logical block. By dragging a block to the left of under an indented block it is appended as an non-indented block and will be executed as a single command after all indented blocks were executed. Blocks can also be dragged out of the command stack to be removed from the command stack.
The Background serves as a working area, blocks can be dropped and edited on it. Before executing the series of commands by clicking the flash button (which is the button at the far right in Figure 17), all blocks on the command stack have to be filled out. Also, a Calliope Mini (Jørn Alraun, 2017) robot has to be attached via USB. When the flash button on the background is clicked, the software tries to detect the robot automatically by scanning all USB ports. If the robot is not detected, a window appears on which the robot as a target device has to be selected manually. An example selection process on a Macintosh computer is depicted in Figure 18. After that, the yellow light on the robot should blink repeatedly until the series of commands is executed after some seconds.
2
1
3
Figure 18: Process to select the Calliope Mini (Jørn Alraun, 2017) robot as a target device Source: Own Depiction
5.1.2
Design Principles – User Experience
The main design decision when building this editor was to represent commands with blocks instead of text pieces. This approach has two main benefits: students need less time filling out the blocks than writing a command and have no problems with programming syntax. Beyond that, the design is guided by four principles making the editor as intuitive and easy as possible. The principles concern the block movement on the areas of the screen, the append logic of blocks on the command stack, the connection to a robot, and the block design. 40
Looking at the areas on the screen, the construction of a program happens from left to right. When blocks are moved to the right, in particular from the command box to the background area or from the background area on the command stack, the program is developed. When blocks are moved to the left, the development of a program is reversed. The rightest element on the editor, the flash button, marks the last step, the execution of a program. The structure of all blocks appended to the command stack was designed to resemble the welldesigned code structure of programs written in textual programming languages. This means that command blocks that are appended directly to a logical command block are indented as in the programming language Python. Also, command blocks can be appended under a logical block at the left to not be indented to and unaffected by it. To make appending easy and intuitive, blocks are appended by being dragged directly under the block they are to be appended to and can be dropped in a bigger area under the block so it requires no precision to append them. As in the code structure of real programming languages, several levels of indentation are allowed. The logic of connecting with a robot was nearly completely adopted from the Mu programming environment. It tries to make the connection to a robot as easy as possible by trying to automatically connect to the robot, and if this is not working, opening a window where only the robot has to be selected from the file system. This approach hides any low-level configuration logic for the user. Lastly, to make block use intuitive, the blocks and their parts were structured by color, color density and field size. Blocks that interact with the same sensor or actuator on the robot are tainted in the same color while all logical blocks are tainted in different shades of grey. Logical blocks of the same type are also tainted in the same grey tone. All fields on the blocks where values should be entered are tainted in the same color of the block, just with lower density. All fields where values or blocks are put into have approximately the width of the expected input. This means for example that fields where a single number has to be entered are much slimmer than fields which expect words as input. 5.1.3
Improvements Suggestions – User Experience
Drawing from the experiences of the lesson, the biggest problem regarding user experience is that the robot does sporadically not behave as he should. For example, the press of a button is not recognized or the robot fails to execute commands at all. As many robot failures cannot be prevented even by thoroughly testing the robot before the lesson, replacement robots to exchange non-functioning robots should be prepared. A more resourceful approach would be to search for ways to fix the robot problems during the lesson, e.g. resetting the robot. In addition, there are some problems that occurred exclusively on some computers. Also here, the problems were not there from the beginning but occurred during the lesson. The solutions here are similar as with the robots: either replacement computers or ways to cope with problems that repeat. To prevent the loss of the students’ programs when the software crashes, a feature that periodically automatically saves the program built by students would be useful. Beyond the problems that occurred, there are three small improvements that make the lesson more enjoyable for students. The first is giving the command box background a color so 41
students can see where the area ends. The second is moving the append block more to the left inside the background area so that students can append blocks in more levels of indentation. The third is to increase the block width so that the input fields can be bigger and students are better able to see the values they enter. In addition, there are four major improvements that would have a strong impact on the lesson experience. A big change would be the addition of a robot simulator in the educational software: Students do not have to wait ~5 seconds until the commands are flashed on the robot. Also, fewer robots would be needed for the lesson as students would tests their code mostly on the simulator and not the robot. A tool that would save even more debugging time are helpful error messages, especially in the case that students only forgot to fill out an input field. To make building the blocks more convenient, it is highly recommended to improve the append logic of the if- and for-loops to a more intuitive one as the one in the Open Roberta Lab® programming editor (Fraunhofer IAIS, 2014). Lastly, as not all sensors and actuators of the robot are used, the use of more of these would bring more variety to the curriculum. This requires a major software change as the Mu software, which is used for translating commands into hex code executable by the robot, does not support these sensors and actuators. 5.1.4
Design Principles – Software Architecture
The aim of the educational software is to provide a cross-platform user interface (UI) that gives the ability to steer a robot. To build a cross-platform UI, a modern-looking framework with lots of functionality, PyQt5, was used. Functionality from the Mu programming environment was taken to steer the robot. A controller class is used as a broker between the UI and the robot. All following statements refer to a simplified version of the software architecture is depicted in Figure 19. As the development of the educational software was UI-centric, the software architecture was based on an adaptation of the architectural pattern Model-View-Controller (Deacon, 2009). When the app is started, the controller initiates all elements he manages while the app is running: the UI elements, factories to create UI elements and the interface to the robot. As the program is small, the size of the controller class is still manageable. The UI elements have a view class which inherits from a PyQt5 Widget and a model class which displays information about their design and data.
42
Figure 19: Simplified UML Class Diagram of the Educational Software Source: Own Depiction
The second design paradigm was strong object-orientation when building the UI as many UI elements share similar properties and can easily be modelled by objects. Because Python is built around duck typing and the use of abstract classes is not pythonic, UI classes of the same type share the same structure despite not inheriting from the same abstract class. This structure allows for outsourcing class-specific behavior to the classes themselves and spares differentiating behavior of each specific class in functions of the controller class. One example are the three main areas of the interface class Window, as outlined in 5.1.1, that are represented by the classes CommandBox, Background and CodeBlockStack: Each area is able to contain, catch and drop codeblocks. Another example is the the design of the classes DraggableCodeBlock and ConditionElement themselves. Both kinds of codeblocks can display 43
the commands explained in 5.1.1 by adopting different models. While the shared appearance of these different commands is stored in the DraggableCodeBlock and ConditionElement classes themselves, their individual visual design and contents are stored in several model classes of the same structure. This way, the codeblock classes can access and manipulate properties of their model indifferent of which model they adopt. To make the environment scalable to different amounts of blocks and areas, nearly all size properties of UI elements are based on the size of a DraggableCodeBlock. This means that the size of the area and the Window classes depend on a fixed block size specified when initiating the Controller class. Another example are the classes DraggableCodeBlock and ConditionElement. Each of them contains different small UI elements such as a text field, an input area or a selector. The height of each small UI element is defined as 0.8 times the height of a DraggableCodeBlock. As the ConditionElement itself is a small UI element that can be inserted into a DraggableCodeBlock, all elements inside it have 0.64 times the height of a DraggableCodeBlock. Another action taken to eliminate code duplication and make the code more expressive is the outsourcing of the generation of UI elements to factories. For example, each DraggableCodeBlock created in the environment has the same parent (Window), controller (Controller), width (specified when initiating the Controller) and height (specified when initiating the Controller). When the Controller is initiated, a factory is created that stores these four parameters and enables the programmer to create DraggableCodeBlocks by only specifying parameters that are specific to the codeblock he or she wants to create. To save time when coding, the robot interface in form of the RobotInterface class and the Contrib Package was nearly completely adopted from the Mu programming environment. This decision excluded the use of some sensors and actuators on the robot because they were not supported by the Mu programming environment. Yet, it still solved the big problem that there is no open-source code to directly steer the Calliope Mini in Python. One approach to solve the problem is to use a web editor for the Calliope Mini, download the hex code of programs which steer sensors and actuators not supported by Mu, analyze the hex code and rebuild it. Yet, only small changes of command parameters changed large portions of the hex code which prohibited recognizing larger patterns in how the hex code changes based on command values.
5.2 Install Guide 5.2.1
Use Guide
The use guide is specified for the operating systems Mac OS and Windows. The software should also be installable on the operating system Linux. All depictions are made during the install process on Mac OS, although these depictions look similar on Windows. Step 1: Download PyCharm Community Edition. For Mac OS:
44
https://www.jetbrains.com/pycharm/download/downloadthanks.html?platform=mac&code=PCC For Windows: https://www.jetbrains.com/pycharm/download/downloadthanks.html?platform=windows&code=PCC Step 2: Download Python 3.6.3. For Mac OS: https://www.python.org/ftp/python/3.6.3/python-3.6.3-macosx10.6.pkg For Windows: https://www.python.org/ftp/python/3.6.3/python-3.6.3-amd64.exe Step 3: On both downloaded files, install the software by opening the downloaded file and clicking through the install guide with default settings Step 4: Get the folder „Educational Software“ from a USB-Stick or download it from the internet and put it on the computer desktop Step 5: Open Pycharm. In the upcoming window, click „Open“ and select the folder „Educational Software“ All further instructions are contained in Table 9. In the event that the program gets closed, open the program again by doing Step 18. In the event that the program does not appear after Step 18, it might be because the last time the program failed to run. In that case, hold the key “command” and then “tab” several times until a rocket is selected. Release the “tab” key then.
45
The window should
Step 6: Click
Step 7: Click Step 8: Drop down
Step 9: Click now
Step 10: Select
now look like this
on
on “Project
on the plus in the
“Add local”
“PyCharm”
Interpreter” and click “Show All”
the interpreter bar
bottom left
and then “Preferences”
Step 11: On the left,
Step 12: Look at the System
Step 13: Confirm
Step 14: Click now
Step 15: Enter
select “System
Interpreter – if it is Python
again with “Ok”
on the Plus
“PyQt5” in the
Interpreter”
3.6, click “Ok”, else select
bottom right to
search field at the
the Python 3.6 Interpreter
add the PyQt
top
package
Step 16: Click “Install
Step 17: Close the settings
Step 18: Run the
If the program does not run, install
Package” at the
windows. Now click on
program by clicking
missing packages (Open Preferences,
bottom left
“Install requirements ” on
on the green arrow
then Step 14) specified in the error
the yellow Popup in the
at the top right of
messages. In the above example, the
main window.
the window
package “serial” is missing.
Table 9: Install Guide – Detailed Instructions Source: Own Elaboration
5.2.2
Improvement Suggestions
To make the software install process easier and less error prone, the aim would be to create an executable that is able to run the software on any computer. Unfortunately, there is no easy way to do this in Python.
5.3 Lesson Materials
46
5.3.1
Use Guide
Robot Workshop – 5 pages
Running a Program – 2 pages
The Blocks – 5 pages
Additional Levels – 1 page
Table 10: Front Pages of the Four Booklets of the Lesson Guide Source: Own Elaboration
The lesson guide is handed out to each pair of students during the lesson and contains four booklets depicted in Table 10. As the lesson consists of several tasks the students have to complete, the main booklet is the booklet “Robot Workshop” containing all tasks and solutions. Each task in “Robot Workshop” refers to other booklets for the knowledge students need to complete the task. The booklet “Running a Program” explains how to write a program and execute it on the robot. The booklet “The Blocks” explains all command blocks used (see 5.1.1) and what their commands do on the robot. Finally, the booklet “Additional Levels” contains three additional tasks for students that completed all tasks in “Robot Workshop”.
Figure 20: Example task in the “Robot Workshop”
47
Source: Own Depiction
Looking at “Robot Workshop”, it contains eight tasks and two feedback exercises. Each task called “Level” has the same structure, as depicted in in Figure 20. On the left side it contains a task description and the solution displayed by the robot on the right side. The task description also refers to “Running a Program” and “The Blocks” for additional knowledge needed to complete the task. The additional tasks in “Additional Levels” are of the same structure. “Running a Program” contains a picture-rich five-step description how to attach a robot, build a program in the educational software, and execute it on the robot. “The Blocks” contains eight description in the form as depicted in Figure 21, one for each command block (see Table 8). In each description you see a title with a short sentence that describes what the command block does at the top. Then, the command block is displayed with all its input fields. Each input field has a description which value should be entered and how the value affects what the command block does. After that, one or more examples of the use of the command are displayed. Each example shows the used command blocks on the left, a blue arrow, and the result on the robot on the right.
Figure 21: Example block description in “The Blocks” Source: Own Depiction
5.3.2
Design Principles
The structure and layout of the booklets was guided by six principles: Split of Purposes: Each booklet serves one purpose. Although it is harder for students to handle all the booklets in the beginning of the lesson, it helps them a lot later when they want to find something specific, e.g. the explanation how a command block works. 48
Simplicity: The sentences in all booklets are clear, short and use common words. Consistency: All booklets consist of several units. In “Robot Workshop” and “Additional Levels” a unit is a level, in “The Blocks” a block description, and in “Running a Program” a step. Inside one booklet, each unit has exactly the same structure and design. Beyond that, the structure and design of all units in “Robot Workshop” and “Additional Levels” is identical. Visual Focus: As theory often differs from reality in technical explanations, real pictures of the robot as well as the software environment were used in the explanation to exactly depict the desired result. For the same reason a variety of pictures was used in the explanations at the expense of text length. Clarity: Especially because the use of pictures makes the structure of the booklets ambiguous, a strong emphasis was put on a clear visual separation of each unit of a booklet and all the parts inside a unit. Understanding: Because students often have a hard time grasping abstract concepts and technical explanations, the effort was made to explain information in different ways. In all booklets, some important or complicated pieces of information are explained both in a visual and textual way. The content of the curriculum was designed acting on four principles: Fun Start: In the beginning of the curriculum, students learn the use of many commands. This heightens their interest by fast successes and exploring the different sensors and actuators of the robot. Higher-order thinking: In the end of the curriculum, students are kept interested with tasks requiring creativity and own thinking. Increased difficulty: Each task is harder than the previous task. This ensures confidence of students in the beginning and that they stay challenged till the end of the curriculum. Reflection: In two evaluation rounds later in the curriculum, students improve their problemsolving process and learn to work as a team. 5.3.3
Improvement Suggestions
In total, students enjoyed working with the booklets. As one student explained it, “it was thoroughly explained what one had to do. It was great”. Yet, there are some small improvement to be undertaken in the curriculum. The task “Introduction” should be renamed to “Level 1” and the number of all consecutive levels increased by one to prevent confusion with task indices. In the tasks “Level 2” and “Level 5”, it should be clearly specified that the button A is on the robot and not on the keyboard. To gain insights about how students work with the booklet structure in general, a detailed evaluation focusing on this aspect is needed. Despite that, there are some changes that could 49
have a bigger impact on how fast students grasp concepts. Video instructions as well as a better connection of the explanations and examples in the “The Blocks” booklet could further increase student understanding. A way to make the booklet content easier readable for both students is to display the booklet content on the computer. Lastly, the decision has to be made whether the curriculum should be focused on learning programming procedures or problem-solving in the context of programming, as explained in 6.5.
5.4 Lesson Guide 5.4.1
Use Guide
The main purpose of the lesson is to teach basic programming procedures (if-loop and for-loop) in an enjoyable way at the students’ own pace. It is recommended for students that have no previous programming experience. The lesson was designed for students with an age of 10-12 years, although it might be appropriate as well for other audiences. The lesson was tested with a class size of 8 students and is recommended for teaching up to 1216 students working in pairs by an experienced teacher. The lesson length is variable, although it is not recommended to conduct a lesson shorter than 60 minutes. The lesson instructor is required to have an understanding about the if-loop and for-loop programming procedures as well as basic knowledge of the educational software and how commands are displayed on the robot. The budget needed to buy the robots for conducting the lesson is 35 Euros per robot. Each can supply up to four students. Beyond that, one computer (Windows or Mac) with a mouse has to be available with the educational software installed.
50
As the lesson is largely based around students working with the booklets, the lesson guide is short. Because it is also self-explanatory, only the translation of it will be displayed in full in Figure 22 and 23.
Figure 22: Lesson Guide – Part 1 Source: Own Depiction
51
Figure 23: Lesson Guide – Part 2 Source: Own Depiction
5.4.2
Design Principles
The main learning objective of our lesson is to learn to “apply” rather than “remember” or “understand” the if-loop and for-loop programming procedures. This means that the curriculum is built around students working on their own tasks, with knowledge and understanding of the procedures just as a tool to solving these tasks. As a result, there are no longer periods where the instructor explains the curriculum as in similar curricula where students learn to “remember” and “understand”. Instead, students have all the knowledge including examples in their booklets and can ask the teacher if they have problems with understanding. This approach also has other advantages, one is that each group can work at their own speed, no group gets bored or left behind. Also, each pair of students takes its own responsibility for understanding the procedures instead of relying on the teacher to explain the procedures well. As every action they do is determined by themselves to solve a task, their motivation is higher than in a scenario where the explanation of the procedures is separated from the use of the procedures. In addition, the lesson is flexible in its duration. To make the work process of students more efficient and reduce the amount of asked questions, the lesson begins with explanations and advice concerning the work process. 5.4.3
Improvement Suggestions
There are three major areas than can be positively affected by improvements of the lesson guide: scalability of the lesson to more students, concentration of students, and the creation of an attractive learning environment. The biggest current obstacle to the use of the lesson in a 30-student class typical in Germany is teacher time. The teacher can only answer so many questions during the lesson. To overcome 52
this obstacle, the questions of the students during the lesson can be systematically listed and sorted by how often they occur. The answers to the most occurring questions during the lesson can be explained before the lesson to free up lesson time for answering more individual questions. Beyond that, putting the booklet contents on the computer screen instead of in paper booklets eliminates all printing costs regardless of class size. Especially in bigger classes, students also need to work efficiently on their own. A prerequisite is that students are concentrated on the tasks they are doing. When the lesson was conducted, the second group had much bigger concentration problems than the first. This indicates that concentration problems can be solved by creating a learning environment as the first group had by following the five guidelines specified in 6.5. Two other changes to increase concentration is adjusting the lesson length (90 minutes should be the maximum) and making the curriculum more entertaining, for example by including storytelling. An entertaining curriculum can also be accompanied by other options to create an attractive learning environment. One is to put students of similar strength into team of two: In the lesson conducted, one team of students had one student solving all the tasks while the other was only assisting a bit. Another phenomenon observed were instances teams successfully helping each other. Fostering this form of collaboration can significantly reduce the volume of questions the teacher gets asked. Lastly, when asked about what exactly she learned in the lesson, one student who successfully solved all tasks, could not answer the question spontaneously. Giving students more context about interesting applications and the value of programming at the beginning of the lesson can increase the motivation of students when solving the tasks.
53
6 Evaluation This chapter will first describe the design of the evaluation: the constructs to measure, the methods of measurement, how the measurement using the methods was constructed in detail, and the chosen sample of the population to be analyzed. The measurement construction and representation of the target population will be explained based on the Total Survey Error framework illustrated in Figure 25 (Groves & Lyberg, 2010; Lugtig, 2016). After that, the quantitative analysis will be explained and the results obtained will be depicted, followed by a selection of all qualitative observations made during the lesson. In distinction, the next chapter will focus on the interpretation of the results. At the end of the chapter, suggestions are made on how to improve the evaluation.
6.1 Aim and Methods The primary aim of the evaluation is to examine the effect of the lesson on the lesson aims in 4.2: The LOs, the objective motivation and the SF confidence. Beyond that, the effect of two important SF changes as well as unstructured observations during the lesson were evaluated. Three methods, among others, are prevalent in educational evaluation: questionnaires, behavioral observations and tests (Stake, 1967). As tests consume lots of time and our lesson including evaluation had a planned duration of 90-180 minutes, instead of tests, the times students needed to complete tasks were measured. Beyond that, a post-lesson questionnaire was designed. All questions in the questionnaire that needed comparability between before and after the lesson were also included in a pre-lesson questionnaire. Lastly, observations of students’ behavior during the lesson were noted down. Figure 24 explains how these methods were used to achieve the aims of the evaluation.
54
Figure 24: Construct Measurement in Questionnaire Source: Own Depiction
6.2 Measurement The next three paragraphs explain how the methods were applied. The time to complete the tasks during the lesson was measured by the instructor through taking notes after a group has shown the right solution to a task. As the instructor prioritized answering students’ questions, some time estimates are inaccurate. The observations of student behavior were noted down after the lesson based on memories of the lesson. The design of the questionnaires as the main tool of construct measurement will now be described in more detail. All descriptions of the questionnaire content refer to Figure 24. The descriptions also refer to all possible errors in the area of measurement depicted in ovals between the different stages of the measurement process in the left column in Figure 25.
55
Figure 25: Total Survey Error Framework Source: Own Depiction, based on (Groves & Lyberg, 2010; Lugtig, 2016)
The first aim is to achieve construct validity, i.e. respondents have the same understanding of the construct as in other studies. The main challenge here was that the questions regarding programming procedures (main LOs) could be easily misunderstood. That’s why precision was paramount to brevity when the questions were elaborated. The SF Confidence as well as the two SF changes questions were kept short to be precise. The questions regarding motivation were not focused directly on the construct motivation, but rather direct questions inquiring about interest in programming in the future to give insights for practitioners. Inside these boundaries, all questions were designed to be as easily understandable as possible to minimize the measurement error. In particular, it was estimated whether a student, after reaching the LOs of the lesson, was able to understand each question and answer it positively. Beyond that, all answer fields to the questions were a Likert scale if quantitative values were needed and a text field if not (Likert, 1932). All Likert scale answer fields contained six steps, because the children would be too occupied with choosing between more alternatives (Ghiselli & Brown, 1948; Symonds, 1924). Regarding construct validity, this amount of steps was sufficient for a differentiated answer (Ghiselli & Brown, 1948; Lugtig, 2016; Symonds, 1924). To weaken one well-known source of measurement error, social desirability bias, an even number of responses was chosen (Garland, 1991). After the lesson, all questionnaires were collected and all values entered into an excel spreadsheet. All Likert Scale answers were coded with values of 1-6 corresponding for the six steps of the scale. To minimize the processing error, values on the Likert scales that were between two steps were usually set to the step they were nearer. If they were near the middle of two steps, they were set to the average of both steps. Values on the left or right of the Likert 56
scales were set to the lowest or highest Likert scale, respectively. Furthermore, when the data analysis using the scripting language R was conducted, manual tests with random values were conducted on each statistical function to prevent processing errors in that area.
6.3 Representation All statements in this section refer to the representation process depicted on the right side in Figure 25. In particular, the occurrence of representation errors, depicted as ovals between the different stages of the process, in this evaluation is pointed out. The target population of the lesson artifact are students in the class grades 5-7 in German schools indifferent of gender. Looking at this target population, the lesson can be theoretically tested in many schools, and thus the coverage error is small. Yet, in our case it was only tested in one school with selected students at a specific timeframe, which implies a significant sampling error – results can vary strongly between schools and groups inside a school, as also observed in this study. To make sure the sample is inside the target population, the age and class grade of students was gathered from all respondents. All study participants are within the target population. Also, the sex of all participants was gathered to gain insights about the representation of both sexes in the sample. Female students represented 25% of all participants and around 14% of all participants included in the statistical evaluation. Thus, the results obtained in this study are much more representative for male students. After the students filled out the questionnaires, two types of nonresponse errors occurred: unit nonresponse and item nonresponse. Unit nonresponse happened in the event of two students leaving the classroom and not finishing the lesson as well as the evaluation questionnaire. As single pre-lesson questionnaires were largely useless, the units were removed from the respondents included in the quantitative analysis. Looking at item nonresponse, three qualitative answers and two quantitative answers were left unanswered. In case of the qualitative answers, these answers were ignored. In case of quantitative answers, item nonresponses were coded as NA values that were left out in all calculations. As all statistical parameters calculated summarized sets of data, the NA values had no significant influence on the calculations. No post-survey adjustments were made due to the small size of the data set.
6.4 Quantitative Analysis and Results The lesson was conducted in the school Gymnasium Trudering (Gymnasium Trudering, 2013) with two groups of 8 students. Each group consisted of four teams of two students each. In the second group, a team left the lesson earlier and thus is not considered in the analysis. The lengths of the first and second lesson are ~90 minutes and ~60 minutes, respectively. Each student filled out two questionnaires, one before and one after the lesson. All quantitative results from questions answered on a Likert scale were coded with values from 1 to 6 with 1 corresponding to the answer “not at all” and 6 to “very well” or “very”. 57
The aims of the analysis are to (a) obtain values measuring the strength of the constructs depicted in Figure 24 as well as (b) interrelationships between them and (c) reasons for the values obtained. For all constructs appearing in the questionnaires before and after the lesson, only the differences between both questionnaires were interesting and coded as construct values. The analysis contains five data tables depicted as Table 13 to Table 17. The first two data tables target analysis aims (a) and (c). For (a), the first table contains statistical parameters summarizing the values of all constructs: standard deviation, mean and median. In addition, the effect of demographical data on the constructs was measured (c). The method of choice here would be one measuring causation from the demographical data on the construct. Because the research sample was too small to discern distribution patterns such as a linear relationship, the prerequisites for a regression analysis measuring causation were not given (University of Zurich, 2016a). Instead, the correlation coefficient based on Spearman was calculated (University of Zurich, 2016b). The second data table focuses on analyzing the changes between the questionnaires before and after the lesson. For (a), the hypotheses that both a negative or positive change in the constructs occurred, were tested. Because the sample size is smaller than 30 and due to the small size, no normal distribution could be discerned, the requirements to conduct a t-test were not met (University of Zurich, 2016c). For this reason, the Wilcoxon signed-rank test was conducted (University of Zurich, 2016d). To assess the significance of the test results, the effect strength was calculated based on the correlation coefficient of Pearson (University of Zurich, 2016d) (Unknown, Unkown). In an effort to investigate whether the students’ previous performance influences the effect of the lesson on the constructs (c), the correlation coefficient based on Spearman was calculated for the reasons stated in the last paragraph (University of Zurich, 2016b). The third data table aims to investigate how selected constructs and demographical data influenced student motivation, performance and confidence (b). The construct motivation is coded as the mean of the question items Mot1, Mot2 and Mot3 in Table 11, which does not correspond to the exact scientific construct motivation, but rather an general interest of the students in future projects. The construct performance is coded as the mean of changes in the main learning objectives from before to after the lesson: the items mLO1, mLO2, mLO3 in Table 11. For the reasons cited at the end of both previous paragraphs, the correlation coefficient based on Spearman was calculated to investigate the interrelationships (University of Zurich, 2016b). The last two data tables depict how long each team in the first and second group, respectively, needed to complete all tasks in the curriculum in minutes (a). As some tasks focus on one LO, conclusions can be drawn on how well students performed in this LO.
58
Construct
Question Items
Abbreviation for Item
LO “If-loop”
How well can you
… under a certain condition?
mLO1
LO “For-Loop”
build a program
… a specific amount of times?
mLO2
LO “For-Loop with Iterator”
executed…
… for a range of numbers?
mLO3
LO “evaluating problem-
Were you better in solving the last tasks than the first
solving”
tasks?
SF Confidence
How confident are you in your programming abilities?
Conf
Motivation for Programming
How motivated are you to do a similar workshop?
Mot1
Motivation for Programming
How interested are you in school subjects concerning
Mot2
(Interest in future projects)
programming?
Motivation for Programming
Could you imagine to work as a programmer in the
(Interest in future projects)
future?
SF “example programs”
Did the examples in the booklet help you?
SF1
SF “sheet with all code
Did the descriptions in the booklet help you?
SF2
Age
How old are you?
Age
Sex
Are you male or female?
Sex
sLO1
(Interest in future projects)
Mot3
commands”
Class
Which class do you go to? Table 11: Question Items to Measure Constructs and Demographical Data Source: Own Elaboration
Class
Construct
Calculation
Abbreviation
Performance
Mean of constructs mLO1, mLO2 and mLO3
mLO
Motivation for Programming Mean of constructs Mot1, Mot2 and Mot3
Mot
(Interest in future projects) Table 12: Aggregated Constructs Source: Own Elaboration
59
mLO1
mLO2
mLO3
sLO1
Conf
Mot1
Mot2
Mot3
1.36
2.21
2.46
5.32
0.89
0.36
-0.54
1
2
2.5
6
0.5
0
1.91
2.08
1.63
1.2
1.52
Correlation with Sex
-0.18
-0.39
0.14
-0.32
Correlation with Age
0.13
0.15
0
Correlation with Class
0.36
0.81
Correlation with Group
-0.57
-0.69
mean median standard deviation
SF1
SF2
0.19
5.61
5.43
0
0
6
6
1.39
1.31
1.32
0.56
0.85
0.08
0.16
0.12
0.3
-0.09
-0.03
0.02
-0.05
-0.37
0.05
0
-0.28
-0.2
0.32
0.39
0.22
0.3
0.27
0.5
0.31
0.32
-0.54
-0.62
-0.27
-0.06
-0.45
-0.38
-0.19
-0.18
Table 13: Statistical Parameters of All Constructs (in all except for sLO1, SF1 and SF2 the change between lessons was considered) Source: Own Elaboration p-value of hypothesis "stagnation or decrease" p-value of hypothesis "stagnation or increase" Effect strength based on the Correlation Coefficient Correlation of Performance before with Change
mLO1 mLO2 mLO3 0.01 0 0 1 1 1 strong
strong
very strong
-0.72
-0.85
-0.63
Conf Mot1 Mot2 Mot3 0.02 0.2 0.94 0.57 1 0.85 0.11 0.45 moderate
-0.59
weak
weak
very weak
-0.92
-0.29
-0.22
Table 14: Statistical Parameters Concerning the Change of Construct Values Between Lessons Source: Own Elaboration mLO
Conf
Mot
Team 1
Team 2
Team 3
Team 4
Introduction
7
7
5
8
Conf
0.57
1
0.49
Level 1
3
3
5
2
Mot
0.3
0.49
1
Level 2
3
11
10
8
SF1
0.35
0.33
0.57
Level 3
4
3
3
8
SF2
0.29
-0.12
0
Level 4
8 NA
NA
NA
Sex
-0.06
-0.32
0.17
Feedback
10 NA
NA
NA
Age
-0.03
0.02
-0.1
Level 5
26
Class
0.53
0.39
0.23
Level 6
Table 15: Interrelationships Between Selected Constructs Source: Own Elaboration
NA
9 12 NA
11 NA NA
Table 16: Task Times of Two-Student Teams in Group 1 Source: Own Elaboration
Team 1
Team 2
Team 3
Team 4
Introduction
7
9
7
6
Level 1
4
4
3
5
Level 2
5 NA
11
5
Level 3
12 NA
15
5
Level 4
NA
NA
NA
NA
Feedback
NA
NA
NA
NA
Level 5
NA
NA
NA
NA
Level 6
NA
NA
NA
NA
Table 17: Task Times of Two-Student Teams in Group 2 Source: Own Elaboration
60
6.5 Qualitative Observations Looking at the motivation and performance of students, the first thing to notice is a big gap between both groups. While even the weakest students of the first group tried hard to solve nearly each task until they were successful, the students of the second group were much faster to surrender and stop thinking when approaching a more complicated task. There are five assumed causes for this disparity: -
The lesson for all students of group 1 started simultaneously while some students of group 2 started already working while others walked in
-
The other students of the class of students from group 1 had a lesson while the other students of the class of the students of group 2 had free learning time
-
The students of group two were not as fit as they had more lessons previous to the workshop
-
The instructor had not as much enthusiasm and energy in the second session compared to the first session
-
The second group had considerable more hardware problems with the robots
In future studies, an objective should be to eliminate the causes for these five phenomena. What both groups had in common was a hard start in understanding the booklets, as also observed in the pilot tests of the software and curriculum. Still, especially the second group managed to understand these basics quickly, which can be attributed to a better explanation of them before the lesson. Most importantly, this short phase of confusion had no permanent negative effect on the motivation of students during the following tasks. In contrast, the students took full advantage of all booklets. The next bigger challenge was a task where the students had to apply the for-loop with iterator in an unintuitive way and think some steps further than in the other tasks. As one can see in Table 16, team 1 needed much longer for level 5 than all the other tasks. This phenomenon unveils a tradeoff that one undertakes when designing questions: Is it more important to teach a bigger amount of concepts and procedures with fast, easy tasks or problem-solving through tougher tasks? There is no one answer, but the tasks are to be chosen carefully so that students achieve the LOs of the lesson. In distinction to the challenges students had solving the tasks, they had an easy time dealing with the educational software environment. The only major issue was the append logic of the if- and for-loop blocks when blocks were indented after they were appended. It is strongly recommended to further develop the append logic of these blocks to resemble the append logic of these blocks in the environment Open Roberta Lab®, even though this requires a major change in the educational software (Fraunhofer IAIS, 2014). An extension of the educational software could also solve another problem that occurred: The teams had problems sharing the robot and the booklets. Displaying the content of the booklets 61
on the computer is one approach to make the booklets’ content more visible to two students at a time. Improving the student-robot ratio would help in counteracting conflicts regarding the use of robots between the teams. Finally, it can be stated that the anonymous qualitative feedback of students after the lesson was uniformly positive.
6.6 Improvement Suggestions After the evaluation, some further improvements of the evaluation methods are put forward based on the Total Survey Error framework. Some measures were taken to counteract the measurement error in this evaluation, such as neutralizing the social desirability bias. Through further improving the Likert Scales, also other biases can be mitigated. Another action recommended to decrease the measurement error is to test the evaluation design alongside the prototype lesson in pilot tests to gain insights about which question items are not well understood. A further help would be written instructions how to answer the question items, in addition to the oral instructions already given. To decrease the Nonresponse error, all questionnaires collected should be tested for completeness and valid values. In the case these conditions are not met, they should be returned to the respondent for correction. In this study, this was only done sporadically. Lastly, an automated evaluation of student behavior and task time inside the educational software would provide large amounts of valuable data with a comparably small effort.
62
7 Discussion This section highlights the contributions to research. First, the boundaries and limitations of the validity of all following statements are set. Then, insights about the constructs measured in the evaluation are demonstrated and interpreted. In addition, reasons for the construct measurements as well as interrelations between constructs will be explained. All quantitative measurement are, if not stated otherwise, based on the students’ questionnaires. Finally, the value and novelty of the lesson artifact inside the research community and beyond is elaborated.
7.1 Boundaries and Limitations All the findings stated in the next sections can be generalized to other lessons that are in the fields of programming education, education using robots and middle school education. The more of these fields are covered by the lesson, the higher the validity of the findings of this paper. In addition, as the lesson developed in this paper is student-centered instead of teachercentered, the findings translate better to other lessons of this specific type. Furthermore, the evaluation was conducted in a specific setting. Thus the validity of the findings increases further for all lessons with similarity to this setting. The evaluation results apply to students of the age of 11-12 in a German school environment at the Gymnasium Trudering (Gymnasium Trudering, 2013). The lesson was conducted twice by an unskilled instructor in the German language to German-speaking students with 8 students in the morning hours. Nearly all evaluated students (~86%) were male. Looking at how the evaluation was conducted, several limitations to the evaluation results exist. The choice process of the students is unknown, it may be that students have been chosen randomly or that only students interested in the workshop were selected. The sample of students (n = 14) is much smaller than the total population, which is why the findings should be generally applied with caution. In the best case they should be used in conjunction with findings from other, similar studies. The sample size also limited the choice and precision of analysis methods. Methods with higher accuracy such as the t-test for hypothesis testing, a Bravais-Pearson correlation test or a regression analysis could not be applied, which limits the validity of the results. Another limitation is also given by the sampling process: There is no proof that all students understood each question correctly. It only can be stated that the evaluation results largely overlap with lesson observations. In addition, it can be assumed that an acquiescence bias exists: A student stated in the feedback section that “everything was great” despite claiming lower LO scores at the end of the lesson (Messick & Jackson, 1960). Lower LO scores at the end of the lesson also prompt the question whether this student had the capability to estimate his own skills, as it is hardly possible that the lesson decreased the skills of the students. It might be that students estimated their skills at the beginning of the lesson on a different scale than after the lesson.
63
Finally, as there is no exact translation from German to English, the meaning of translated constructs might slightly differ from the meaning of the original constructs evaluated in the German evaluation sheet.
7.2 Discussion of Learning Objectives Based on research question 3 in 2.2, it stands to debate how experiencing the lesson affected student performance in the LOs. First, it was tested whether it affected students’ LO performance at all. For each main LO, two hypothesis tests were conducted with the hypotheses “the student performance in this LO decreased or stagnated” and “the student performance in this LO increased or stagnated”. In the case of all three LOs, the first hypothesis was rejected and the second one accepted to the usual significance level of 0.05 (see Table 14). This means that it can be stated that all three LOs were improved by experiencing the lesson. To measure the significance of this result, the effect strength of the lesson experience was calculated. In the case of the main LOs “if-loop” and “for-loop”, the effect was strong, in the case of the main LO “for-loop with iterator” very strong (see Table 14). This is not surprising, as students were least likely to have previous knowledge using the for-loop with iterator. The absolute improvements of the main LOs from before to after the lesson are between 1 and 3 on a Likert scale from 1 to 6 (see Table 14). Since approximately half of the students applied the procedures before, this is a serious improvement. This phenomenon can also be confirmed by lesson observations: In both groups, nearly all student teams successfully completed a set of tasks in which all three procedures had to be applied. Looking at the secondary LO “evaluating problem solving”, nearly all students stated that their problem solving increased very strongly from the first tasks to the last tasks. The mean rating for this statement is 5.32 on a scale from 1 to 6 (see Table 13). Answering the question how their problem solving did improve, students listed teamwork, applied programming skills, experience, labor division, cooperation, and more thorough reading. The secondary LO “creating conceptual knowledge” was analyzed by observing student behavior in a task where they designed a device to measure temperature. Nearly all groups that got to this task in the curriculum created a useful and working temperature measuring device without much assistance. Lastly, the performance in the secondary LOs “remembering procedural knowledge” and “understanding procedural knowledge” could be only observed indirectly. In later tasks, nearly all students successfully assembled blocks with code commands to a program without rereading the sections about what the commands do. This means that the probability is high that they remembered and understood how the procedures worked. When investigating the reasons for how much the performance in the main LOs changed, the first thing to notice is how much the performance improvements of both groups differ. Table 13 gives an indication: all three main LOs have a significant negative correlation with group index. More specifically, an average student of group 1 improved his or her performance by ~8.4 Likert scale points in all three main LOs combined during 90 minutes (data not in tables). 64
An average student of group 2 achieved an improvement of ~2.4 Likert scale points in 60 minutes (data not in tables). Possible causes for this chasm are outlined in 6.5: simultaneous lesson start, the curriculum other students of the class had, lesson start time, instructor motivation, and problems with the robot. Another cause could be class affiliation: all students of group 1 belonged to class 6B, while nearly all students of group 2 belonged to class 6A. Not surprisingly, the performance improvement in all three main LOs correlates with class affiliation (see Table 13). Looking beyond the group differences, the performance improvement of the main LOs also has a moderate correlation with the constructs confidence and motivation (see Table 15). This does not automatically mean that confidence and motivation lead to the LO performance improvements. It is very well possible that students are confident and motivated because they performed well in the lesson and not the other way around. Still, due to the correlation, focusing a lesson around the constructs motivation and confidence will very likely have also a positive influence on the performance in the main LOs. In addition, students that valued especially the coding examples, but also explanations, improved more in the main LOs (see Table 15). This phenomenon can be also confirmed by lesson observations: the best-performing students patiently and carefully read the instructions. They tried a task again and again even when failing three or four time and approached realistic aims step by step. They asked questions as soon as they were stuck, but only after trying everything that was in their power. Schulte would say they practiced “offensive learning” (Schulte & Knobelsdorf, 2007). Carefully reading the instructions and always asking questions was encouraged by the instructor before the lesson, and further actions to foster the behaviors outlined in this paragraph are recommended for the design of future lessons. Lastly, it can be stated that the lesson is best suited for programming beginners, as students with few previous experience with the procedures experienced the biggest main LO improvement. The negative correlation of the improvements with previous performance ranges between 60% and 90% (see Table 14).
7.3 Discussion of Other Constructs 7.3.1
Motivation for Programming
For the same reason as in 7.2, first it was measures whether the lesson had an impact at all on motivation for three kinds of future programming-related activities. For each of the three future activities, two hypothesis tests were conducted with the hypotheses “the student motivation for this activity decreased or stagnated” and “the student motivation for this activity increased or stagnated”. In the case of all three activities, both hypotheses were rejected to the usual significance level of 0.05 (see Table 14). This means that it can be stated that the lesson had no impact on the motivation of students for future programming activities. This finding is confirmed by lesson observations and contradicted by qualitative student feedback after the lesson. During the lesson, on average, there was no big difference in motivation of students from before the lesson to after the lesson. In contrast, the feedback after 65
the lesson of all 14 students was more positive than negative. Also, only 2 students gave any negative feedback at all. Looking for reasons why the lesson had no significant positive impact on motivation for future programming activities, as with the main LOs, a small gap between both groups could be observed (see Table 13 – correlation with group). Yet, even when excluding group 2, the motivation of an average student of group 1 only increased by ~1.1 Likert scale points for all three activities combined on three Likert scales from 1 to 6 (data not in tables). Thus, it can be assumed that there are other, more significant reasons for the lack of performance improvement. Looking at the raw data, it is clearly visible that the motivation before the lesson for similar workshops and programming lessons already was very high and did not left much space for improvement. The averages were ~5.4 and ~5.5 Likert scale points for similar workshops and programming lessons respectively, on a Likert scale from 1 to 6 (data not in tables). Consequently, the motivation for the remaining future activity, the job as a programmer, was observed. It is not surprising that the workshop did not strongly affect a choice that important and in the distant future. Despite that, it can be concluded that the lesson was not able to convince students to participate in future programming activities. The motivation was positively influenced by the successful use of programming examples and correlates with students’ confidence (see Table 15). Facilitating the use of programming examples and supporting the confidence of students is expected to translate to an increase in motivation for future activities in future lessons. As anticipated, motivation also correlates with an performance increase in the main LOs (see Table 15). 7.3.2
Success Factor Confidence
As with motivation and the LOs, it was measured what kind of influence the lesson had on the confidence of students in their programming abilities. Two hypothesis tests were conducted with the hypotheses “the confidence of students in their abilities decreased or stagnated” and “the confidence of students in their abilities increased or stagnated”. Based on the usual significance level of 0.05, the first hypothesis was rejected and the second one accepted (see Table 14). This means that it can be stated that the confidence of students in their programming abilities improved by experiencing the lesson. To measure the significance of this result, the effect strength of the lesson experience was calculated. The effect the lesson had on the confidence of students in their programming abilities is moderate (see Table 14). This result is not surprising, as increasing the confidence of students was no primary objective of the lesson. Still, it was expectable that with a higher amount of practice in programming, the students’ confidence level improves. These quantitative findings regarding confidence can be confirmed by lesson observations: while many teams successfully completed even harder tasks in the curriculum, some teams were struggling in the period before they filled out the questionnaire. One additional thing to notice 66
is that the confidence of students varied strongly throughout the curriculum. It might be that, if the lesson would have been cut off at a different point in time, the results would significantly differ. Similarly as in the case of motivation and the performance in the main LOs, students of group 1 gained more confidence from the lesson (see Table 13). While the confidence of group 2 stayed the same, on average, the confidence of a student in group 1 increased by ~1.6 points on a Likert scale. The remaining question is: Is confidence a real SF improving LO performance? The data shows it is: Students, whose confidence was increased, often also showed significant improvements in motivation and the performance in the main LOs (see Table 15). Still, it cannot be said if higher confidence increases the main LO performance and motivation or whether this relationship is reversed. Although, adjusting a lesson to improve the confidence of students is expected to also positively act upon the main LOs and students’ motivation for future activities. Qualitative observations also unveiled another benefit of confidence: Confident students were more concentrated and happy in solving the tasks as they had no doubt in their abilities. 7.3.3
Examples and Explanations
The helpfulness of examples and explanations was rated with 5.61 and 5.43 points on a Likert scale from 1 to 6, respectively (see Table 13). Although the students had to use at least one of these lesson helps to solve the tasks, the high ratings indicate that the format of the lesson helps was well designed. Also, it shows that students were able to successfully apply both lesson helps in class. Furthermore, it is interesting how both SF changes affected lesson success. It can be assumed that students that rated the SF changes as more helpful did benefit more from them and used them more often. In comparison to the helpfulness of explanations, the helpfulness of examples had a small to moderate correlation with the main LOs, motivation and confidence (see Table 15). Thus, it can be recommended to increase the quality and quantity of examples in future lessons.
7.4 Value of the Lesson Artifact As the mathematician G. H. Hardy states, the quality of a research contribution should be assessed by answering three questions: Is it true? Is it new? Is it interesting? (J. R. Wilson, 2002). While the whole research paper was designed based on the paradigm of truth, this section extends on the novelty and value of the created lesson artefact. 7.4.1
Novelty
Robots have a great potential in programming education (see 1.2), yet there are few examples of educational robots that are below a price of 50$ (see 1.3), easy-to-use (see 1.5), reliable (see 1.5), and have a track record in German education. In the LR and artefact research, only one robot was identified met all of these criteria, the Calliope Mini (Calliope GmbH, 2017). 67
The website of the Calliope Mini states three programming environments that are compatible with the Robot: The Calliope Mini editor, the Pxt editor, and the Open Roberta Lab® (Calliope GmbH, 2017). The biggest difference between the educational software created for this lesson unit and the basic version of these environments is that the educational software does not require an internet connection. Although, the Pxt editor provides an easy-to-use offline version and thus, the offline programming feature cannot be considered a novelty. Beyond that, the educational software developed differentiates itself from the other environments through three unique, novel qualities. First, in distinction to all other environments being programmed with web technologies, the educational software was developed purely using the language Python. This not opens up the domain to a new set of developers, but also enables them to develop in a language that is powerful (high-level), easy-to-learn, and supporting many libraries. Secondly, the educational software natively runs on the computer instead of a web browser, which brings various benefits. As the computer’s resources can be accessed directly, the response speed of native systems is usually higher. A software running directly on the computer is also not restricted by the browser window, controls and settings, which allows for creating a richer and more comfortable user experience. Also, the robot can be directly controlled from the software. There is no need to download the hex code from the browser and drag it onto the robot each time a program has to be executed. Thirdly, all commands of the graphical educational software are translated to Python commands before being flashed on the robot. This allows for a seamless transition and integration of commands from the graphical software and the language Python. As Python is a easy-to-learn, non-graphical programming language, it provides the ideal transition from introductory graphical programming languages to more powerful and serious text-based languages. Looking at the lesson design, the booklet strategy, where students get educational resources and get the responsibility to handle the learning process themselves, is unusual for programming education using robots. During the LR and artefact research, no other lesson design was discovered with three distinct booklets containing explanations and examples, install instructions, and the lesson instructions. As the students gave very high ratings of helpfulness for both programming command explanations and examples (see Table 13), this approach appears to be not only novel, but also promising. 7.4.2
Value
There are three ways in which students benefitted from the lesson experience: Increased performance in the main LOs (see 7.2), advanced problem-solving (see 7.2), and more confidence in their programming activities (see 7.3.2). To specify in which circumstances students benefit most from the lesson, a so-called “environment of value” can be defined, in which the lesson is recommended to be used. The 68
first step to gain more value from the lesson is to adjust the environment based on the five assumptions that explain the gap in performance between group 1 and group 2 (see 6.5). Furthermore, the lesson should be taught to students agnostic to programming. The quantitative analysis has shown that students with less previous experience experienced a higher improvement in main LO performance (see 7.2 and Table 14). Another environment factor is the class size. An didactically experienced instructor that is both familiar with the curriculum and technical aspects of the software is expected to efficiently handle a student load of up to 16 students. An important prerequisite for that is that all technical equipment is well-tested before the lesson is conducted. Looking at the age of students, it can be assumed that the current age of students ranging from 11 to 12 is an appropriate age group. There were no signs that students got bored with the tasks because they were too easy or were unable to solve tasks because the tasks were beyond their abilities. Also, students had fun playing around with the educational software, yet none of them perceived it as to childish. Still, the lesson unit might be also appropriate for audiences in other ages. Lastly, the environment has shown to be robust to students with different performances in other subjects. Both well-performing and low-performing students had significant improvements in the main LOs as long as they were motivated to solve the tasks. The only difference was that some students were faster in solving the problems while others were slower. Another step to increase the value of the artefact beyond creating an optimal environment for artefact use is to improve the lesson artefact. The first recommendation here is to make the artefact available to an international audience by translating it into the English language. Furthermore, a behavioural analysis of students participating in the lesson should be conducted. This way, more insights about specific problems and behavioural patterns, which can be translated into artefact changes, can be gathered. Specific attention should be directed towards improving the examples of programming commands (see 7.3.3). Many other propositions to improve specific parts of the artefact are listed in the “Improvement Suggestions” subsections for each artefact in 5. Opposed to using the whole artefact, parts of it can be also reused in other settings. Some possible scenarios are listed. The first scenario is interesting if a more structured curriculum is desired or the students start the lesson with previous knowledge. This scenario is closest to the current lesson artefact by using another lesson guide and leaving out some of the tasks in the booklet “Robot Workshop”. The lesson guide can incorporate additional content or be adjusted to specific requirements, and tasks that are not useful, e.g. because students already understand the procedures, omitted.
69
The second scenario is a great starting point if one plans to design an own lesson, but does not want to start from scratch. Starting from the tasks in the “Robot Workshop” booklet and the Calliope Robot, one can design an own Calliope curriculum using another educational software. The third scenario is mostly interesting for advanced curricula following the “Robot Workshop” or if the teacher plans to apply an unique educational approach. By using only the software environment with the Calliope Robot, one option is to create a curriculum for students that completed the “Robot Workshop”. Furthermore, the software and Calliope Robot can also be helpful if the teacher prefers other media than the booklets, e.g. video teaching, or wants to explain the concepts herself.
70
8
Conclusion
This paper created a lesson artefact to address the lack of lessons using affordable robots in the field of programming education using robots. The artefact was created based on the state-ofthe-art in the field and its efficacy evaluated in a real school environment. A LR capturing the state-of-the-art unveiled four distinct factors that make lessons successful: applicability, confidence, ease, and efficiency. The lesson constructed in this paper demonstrates that many changes improving confidence, ease and efficacy can be implemented in one lesson artefact without impeding each other. Looking at the impact of singular changes, the two changes “example programs” and “sheet with all code commands” were evaluated. On a Likert Scale from 1 to 6, the students rated the helpfulness of example programs with 5.6 points and the helpfulness of the sheet with all commands with 5.4 points, on average. This confirms the validity of the SF changes in general and advocates for the use of these two particular SF changes in future lesson designs. Beyond that, the confidence, motivation and performance of students before and after the lesson was analysed. It can be stated that the lesson reached its aim to significantly improve the students’ ability to apply the programming procedures if-loop, for-loop, and for-loop with iterator. While the lesson also increased the confidence of students, it had no effect on the students’ motivation for future programming activities. One explanation for that is that the motivation ratings of students were already very high before the lesson. The results stand in contrast to other studies in programming education using robots that report a strong positive effect on motivation and a moderate learning performance. This difference can be attributed to a comparably disciplined curriculum and fast command execution speed of the robot. Based on the two prevalent lesson objectives in the field, learning performance and motivation, learning speed and fun during the lesson are the key metrics to improve. In the case of this lesson unit, there are two further changes that are expected to significantly improve the learning speed. First, a robot simulator software on the computer would allow students to test their code much faster. In addition, the amount of technical issues that occur as well as the time students spend waiting for the other group to test their code with the robot would be reduced. Second, the process how students understand the concepts taught and solve problems on the computer should be analysed in detail so that the curriculum and the software can be exactly adjusted to students’ needs. Inside this curriculum, the students stated that their problem-solving process improved strongly throughout the lesson. Still, all students struggled one task that required advanced problemsolving. Two changes to improve students’ fun and creative problem-solving are proposed. One promising concept are open tasks: to complete a task, students have to use a specific procedure under specific conditions in a program they can freely design themselves. In other curricula using the Alice programming environment, students, while applying a plethora of programming procedures, enjoyed free programming to the point that they did not want to stop. The other proposed change is incorporating social and emotional elements in the curriculum, so that students perceive more practical value. This can be accomplished by expanding the educational 71
software to steer additional sensors of the robot such as sound or allow for communication between robots via Bluetooth®. All these changes are not only recommended for this lesson artefact, but also other artefacts in the field of programming education using robot. Looking at the low influence robot-based curricula have in the field of programming education, one of the two main obstacles, the robot price, is soon to be overcome. To become relevant in the field, robot-based curricula have to overcome the other obstacle, slow learning speed, and capitalize on their unique capabilities in the areas of motivation and teaching problem-solving. Aiming to achieve a high impact on learning performance and motivation, these constructs should be continuously assessed with metrics and improved. One key learning and limitation of the artefact evaluation is that a quantitative analysis with bigger samples is not only more representative, but also allows for analysis methods of significantly higher value and precision. Finally, the lesson artefact opens up the domain of programming education software using an affordable robot for Python developers. It works with the Calliope Mini robot popular in Germany and provides a platform that is easy to develop, interacts seamlessly with the robot, and works on all popular operating systems.
72
Bibliography Álvarez, A., & Larrañaga, M. (2016). Experiences Incorporating Lego Mindstorms Robots in the Basic Programming Syllabus: Lessons Learned. Journal of Intelligent & Robotic Systems, 81(1), 117. Autor, D. H. (2015). Why Are There Still So Many Jobs? The History and Future of Workplace Automation. Journal of Economic Perspectives, 29(3), 3-30. Baker, M. J. (2000). Writing a literature review. The Marketing Review, 1(2), 219-247. Beck, K. (1999). Embracing change with extreme programming. Computer, 32(10), 70-77. Ben-Ari, M. (2001). Constructivism in computer science education. Journal of Computers in Mathematics and Science Teaching, 20(1), 45-73. Benbasat, I., & Zmud, R. W. (1999). Empirical research in information systems: the practice of relevance. MIS quarterly, 23(1), 3-16. Benitti, F. B. V. (2012). Exploring the educational potential of robotics in schools. Comput. Educ., 58(3), 978-988. Bennedsen, J., & Caspersen, M. E. (2007). Failure rates in introductory programming. ACM SIGCSE Bulletin, 39(2), 32-36. Bizot, S. Z. a. B. (2016). 2016 Taulbee Survey. In Computing Research Association (publ.), Computing Research News (29). Washington: Computing Research Association. Retrieved from http://cra.org/crn/wp-content/uploads/sites/7/2017/05/2016-TaulbeeSurvey.pdf Bjornavold, J. (2015). Learning outcomes in Europe. In European Centre for the Development of Vocational Training (Cedefop) (publ.), Lernergebnisorientierung im deutschen Bildungssystem – Ein bildungsbereichsübergreifender Dialog – Berlin: European Centre for the Development of Vocational Training (Cedefop). Retrieved from https://www.dqr.de/media/DQR_22.4.15_Jens_Bjornavold_Vormittag.pdf Bound, J., Braga, B., Golden, J. M., & Khanna, G. (2015). Recruitment of Foreigners in the Market for Computer Scientists in the United States. Journal of labor economics, 33(1), 187-223. Boyle, T., Bradley, C., Chalk, P., Jones, R., & Pickard, P. (2003). Using Blended Learning to Improve Student Success Rates in Learning to Program. Journal of Educational Media, 28(2-3), 165-178. Brusilovsky, P. (1994). Teaching Programming to Novices: A Review of Approaches and Tools. . Bureau of Labour Statistics. (2015). Computer and Information Technology Occupations. In US Department of Labor (publ.), Occupational Outlook Handbook (2016-2017 73
Edition). United States: US Department of Labor. Retrieved https://www.bls.gov/ooh/computer-and-information-technology/home.htm
from
Calliope GmbH (2017). Calliope Programming Environment. Accessed on 12.1.2018, retrieved from https://calliope.cc/en/los-geht-s/editor Cambridge University Press. (2017). Definition of programming. In Cambridge University Press (publ.), Cambridge Dictionary (2017 Edition). Cambridge: Cambridge University Press. Retrieved from: http://dictionary.cambridge.org/de/worterbuch/englisch/programming Carter, L. (2006). Why students with an apparent aptitude for computer science don't choose to major in computer science. ACM SIGCSE Bulletin, 38(1), 27-31. Chien, T.-C. (2008). Factors Influencing Computer Anxiety and Its Impact on E-Learning Effectiveness: A Review of Literature. Paper presented at the Academy of Human Resource Development International Research Conference, Panama City, FL. Cliburn, D. C. (2006). Experiences with the LEGO Mindstorms throughout the undergraduate computer science curriculum. Frontiers in Education Conference, 36th Annual, San Diego, California. Cooper, H. M. (1988). Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge, Technology & Policy, 1(1), 104-126. Dann, W., Cosgrove, D., Slater, D., Culyba, D., & Cooper, S. (2012). Mediated transfer: Alice 3 to java. Proceedings of the 43rd ACM technical symposium on Computer Science Education, Raleigh, NC, USA. Deacon, J. (2009). Model-view-controller (mvc) architecture. Accessed on retrieved from https://6caa0dbc-a-62cb3a1a-ssites.googlegroups.com/site/urtechfriend/Home/MVC.pdf?attachauth=ANoY7cqZGxF tbEmRr7nA1UFzV2561wwxxhGX4BTpQ63dUTQml7e_agZOhWKB_AUKHILhFB C23ohO3_MFjq74pY2Z9nYmSOBd50UyXegw_YIxOPufcXPFjx8CxBxTulnPFO1ha SWO_t3b7cTfe_q0lXsG6yA3zzmALN4WSiZHMhTDsazWvAFVnBq3qWmknWzgWKMTydONWtiJzm7tY16G6i1OPCBSVHtg%3D%3D&attredirects=0 Denner, J., Werner, L., & Ortiz, E. (2012). Computer games created by middle school girls: Can they be used to measure understanding of computer science concepts? Computers & Education, 58(1), 240-249. Dijkstra, E. W. (1989). On the Cruelty of Really Teaching Computer Science. Communications of the ACM, 32, 1398-1404. Fagin, B., & Merkle, L. (2003). Measuring the effectiveness of robots in teaching computer science. SIGCSE '03 Proceedings of the 34th SIGCSE technical symposium on Computer science education, Reno, Navada, USA.
74
Fagin, B. S., & Merkle, L. (2002). Quantitative analysis of the effects of robots on introductory Computer Science education. Journal on Educational Resources in Computing (JERIC), 2(4), 2. Fagin, B. S., Merkle, L. D., & Eggers, T. W. (2001). Teaching Computer Science With Robotics Using Ada/Mindstorms 2.0. Proceedings of the 2001 annual ACM SIGAda international conference on Ada, Bloomington, MN, USA. Flowers, T. R., & Gossett, K. A. (2002). Teaching problem solving, computing, and information technology with robots. Journal of Computing Sciences in Colleges, 17(6), 45-55. Fraunhofer IAIS, S. A. (2014). Open Roberta Lab. Accessed on 19.12.2017, retrieved from https://lab.open-roberta.org/ Garland, R. (1991). The mid-point on a rating scale: Is it desirable. Marketing bulletin, 2(1), 66-70. Ghiselli, E. E., & Brown, C. W. (1948). Personnel and industrial psychology Gregor, S., & Hevner, A. R. (2013). Positioning and presenting design science research for maximum impact. MIS quarterly, 37(2), Pages 337-356 Groves, R. M., & Lyberg, L. (2010). Total survey error: Past, present, and future. Public opinion quarterly, 74(5), 849-879. Gymnasium Trudering (2013). Gymnasium Trudering. Accessed on 12.1.2018, retrieved from http://www.gymnasium-trudering.de/ H.Tollervey, N. (2017). Code With Mu. Accessed on 28.11.2017, retrieved from https://codewith.mu/ Hattie, J. (2008). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Howard, G. (1986). Computer anxiety and management use of microcomputers Ann Arbor, Michigan, USA: Umi Research Press. Hromkovič, J. (2006). Contributing to general education by teaching informatics. In R. T. Mittermeir (Ed.), Informatics education–the bridge between using and understanding computers (Vol. 4226, pp. 25-37). Berlin/Heidelberg: Springer. Jørn Alraun, K. J. B., Prof. Franka Futterlieb, Prof. Dr. Gesche Joost, Maxim Loick, Stephan Noller (2017). Calliope Mini. Accessed on 28.11.2017, retrieved from https://calliope.cc/ Kelleher, C., Pausch, R., & Kiesler, S. (2007). Storytelling alice motivates middle school girls to learn computer programming. Proceedings of the SIGCHI conference on Human factors in computing systems, San Jose, CA, USA. Khabsa, M., & Giles, C. L. (2014). The number of scholarly documents on the public web. PloS one, 9(5), e93949. 75
Kolberg, E., & Orlev, N. (2001). Robotics learning as a tool for integrating science technology curriculum in K-12 schools. Frontiers in Education Conference, 2001. 31st Annual, Reno, NV, USA. Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory into practice, 41(4), 212-218. Lahtinen, E., Ala-Mutka, K., & Järvinen, H.-M. (2005). A study of the difficulties of novice programmers. Proceedings of the 10th annual SIGCSE conference on Innovation and technology in computer science education, Caparica, Portugal. Lamp, J. (2013). The Index of Information Systems Journals. Accessed on 15.11.2017, retrieved from http://lamp.infosys.deakin.edu.au/journals/index.php Lauwers, T., Nourbakhsh, I., & Hamner, E. (2009). CSbots: design and deployment of a robot designed for the CS1 classroom. Proceedings of the 40th ACM technical symposium on Computer science education, Chattanooga, TN, USA. Lee, I., Martin, F., Denner, J., Coulter, B., Allan, W., Erickson, J., . . . Werner, L. (2011). Computational thinking for youth in practice. ACM Inroads, 2(1), 32-37. Liao, Y.-K. C., & Bright, G. W. (1991). Effects of computer programming on cognitive outcomes: A meta-analysis. Journal of Educational Computing Research, 7(3), 251268. Likert, R. (1932). A technique for the measurement of attitudes. Archives of psychology, Lindh, J., & Holgersson, T. (2007). Does lego training stimulate pupils’ ability to solve logical problems? Computers & Education, 49(4), 1097-1111. Lugtig, P. K., Ulrich (2016). Introduction to Survey Design. Accessed on 12.1.2018, retrieved from http://www.summerschoolsineurope.eu/course/4739/introduction-to-surveydesign Major, L., Kyriacou, T., & Brereton, O. P. (2012). Systematic literature review: teaching novices programming using robots. IET software, 6(6), 502-513. Maloney, J., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010). The Scratch Programming Language and Environment. Trans. Comput. Educ., 10(4), 1-15. Martyn Cooper, D. K., & William Harwin, K. D. (1999). Robots in the classroom-tools for accessible education. Proc. AAATE Conference 1999, The 5th European Conference for the Advancement of Assistive Technology, Düsseldorf/Germany. McDowell, C., Hanks, B., & Werner, L. (2003). Experimenting with pair programming in the classroom. SIGCSE Bull., 35(3), 60-64. Messick, S., & Jackson, D. N. (1960). Acquiescence and the factorial interpretation of the MMPI. Accessed on 12.1.2018, retrieved from http://onlinelibrary.wiley.com/store/10.1002/j.2333-
76
8504.1960.tb00097.x/asset/ets200097.pdf?v=1&t=jcc155yq&s=d385774bc016455a93 a4cfdc1c1442b69997fa6b Miller, D. P., Nourbakhsh, I. R., & Siegwart, R. (2008). Robots for education. In B. S. Oussama Khatib (Ed.), Springer handbook of robotics (pp. 1283-1301). Berlin, Heidelberg: Springer. Mitnik, R., Nussbaum, M., & Soto, A. (2008). An autonomous educational mobile robot mediator. Autonomous Robots, 25(4), 367-382. Mubin, O., Stevens, C. J., Shahid, S., Al Mahmud, A., & Dong, J.-J. (2013). A review of the applicability of robots in education. Journal of Technology in Education and Learning, 1, 209. Nagchaudhuri, A., Singh, G., Kaur, M., & George, S. (2002). LEGO robotics products boost student creativity in precollege programs at UMES. Frontiers in Education, 2002. FIE 2002. 32nd Annual, Boston, Massachusetts, USA. Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas (1). New York, NY, USA: Basic Books, Inc. Pásztor, A., Pap-Szigeti, R., & Lakatos Török, E. (2010). Effects of Using Model Robots in the Education of Programming. Informatics in Education, 9(1), 133-140. Peffers, K., & Ya, T. (2003). Identifying and evaluating the universe of outlets for information systems research: Ranking the journals. Journal of Information Technology Theory and Application (JITTA), 5(1), 64-84. Perkins, D. N., Schwartz, S., & Simmons, R. (1988). Instructional strategies for the problems of novice programmers. In R. E. Mayer (Ed.), Teachilig and Learning Computer Programming (pp. 153-178). Mahwah, New Jersey: Lawrence Erlbaum Associates. Qiu, K., Buechley, L., Baafi, E., & Dubow, W. (2013). A curriculum for teaching computer science through computational textiles. Proceedings of the 12th International Conference on Interaction Design and Children, New York, NY, USA Robins, A., Rountree, J., & Rountree, N. (2003). Learning and teaching programming: A review and discussion. Computer Science Education, 13(2), 137-172. Rodger, S. H., Hayes, J., Lezin, G., Qin, H., Nelson, D., Tucker, R., . . . Slater, D. (2009). Engaging middle school teachers and students with alice in a diverse set of subjects. SIGCSE’09, Chattanooga, Tennessee, USA. Rubenstein, M., Cimino, B., Nagpal, R., & Werfel, J. (2015). AERobot: An affordable onerobot-per-student system for early robotics education. Robotics and Automation (ICRA), 2015 IEEE Washington State Convention Center in Seattle, Washington, USA. Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68.
77
Salipante, P., Notz, W., & Bigelow, J. (1982). A matrix approach to literature reviews. Research in organizational behavior, 4, 321-348. Sapounidis, T., & Demetriadis, S. (2013). Tangible versus graphical user interfaces for robot programming: exploring cross-age children’s preferences. Personal and Ubiquitous Computing, 17(8), 1775-1786. Sartatzemi, M., Dagdilelis, V., & Kagani, K. (2005). Teaching programming with robots: a case study on greek secondary education. Panhellenic Conference on Informatics, Volos, Greece Schulte, C., & Knobelsdorf, M. (2007). Attitudes towards computer science-computing experiences as a starting point and barrier to computer science. Proceedings of the third international workshop on Computing education research, Atlanta, Georgia, USA. Shim, J., Kwon, D., & Lee, W. (2017). The Effects of a Robot Game Environment on Computer Programming Education for Elementary School Students. IEEE Transactions on Education, 60(2), 164-172. Sorva, J., Karavirta, V., & Malmi, L. (2013). A Review of Generic Program Visualization Systems for Introductory Programming Education. Trans. Comput. Educ., 13(4), 1-64. Stake, R. E. (1967). The Coutenance Of Educational Evaluation. Accessed on 12.1.2018, retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.543.5561&rep=rep1&type= pdf Sullivan, F. R. (2008). Robotics and science literacy: Thinking skills, science process skills and systems understanding. Journal of Research in Science Teaching, 45(3), 373-394. Summet, J., Kumar, D., O'Hara, K., Walker, D., Ni, L., Blank, D., & Balch, T. (2009). Personalizing CS1 with robots. Proceedings of the 40th ACM technical symposium on Computer science education, Chattanooga, TN, USA. Symonds, P. M. (1924). On the Loss of Reliability in Ratings Due to Coarseness of the Scale. Journal of Experimental Psychology, 7(6), 456. Szlávi, P., & Zsakó, L. (2006). Programming versus application. Issep, Vilnius, Lithuania. Torraco, R. J. (2005). Writing integrative literature reviews: Guidelines and examples. Human resource development review, 4(3), 356-367. University of Zurich (2016a). Einfache lineare Regression. Accessed on 12.1.2018, retrieved from http://www.methodenberatung.uzh.ch/de/datenanalyse/zusammenhaenge/ereg.html University of Zurich (2016b). Rangkorrelation nach Spearman. Accessed on 13.1.2018, retrieved from http://www.methodenberatung.uzh.ch/de/datenanalyse/zusammenhaenge/rangkorrelati on.html 78
University of Zurich (2016c). t-Test für abhängige Stichproben. Accessed on 12.1.2018, retrieved from http://www.methodenberatung.uzh.ch/de/datenanalyse/unterschiede/zentral/ttestabh.ht ml University of Zurich (2016d). Wilcoxon-Test. Accessed on 12.1.2018, retrieved from http://www.methodenberatung.uzh.ch/de/datenanalyse/unterschiede/zentral/wilkoxon. html Unknown (Unkown). Pearson’s correlation. Accessed on 5.1.2018, retrieved from http://www.statstutor.ac.uk/resources/uploaded/pearsons.pdf Vegso, J. (2008). CRA Taulbee Trends: Female Students & Faculty. In Computing Research Association (publ.), Washington: Computing Research Association. Retrieved from http://archive.cra.org/info/taulbee/women.html Vihavainen, A., Airaksinen, J., & Watson, C. (2014). A systematic review of approaches for teaching introductory programming and their influence on success. Proceedings of the tenth annual conference on International computing education research, Glasgow, United Kingdom. Vollstedt, A.-M., Robinson, M., & Wang, E. (2007). Using robotics to enhance science, technology, engineering, and mathematics curricula. Proceedings of American Society for Engineering Education Pacific Southwest annual conference, Honolulu, Hawaii. vom Brocke, J., Simons, A., Niehaves, B., Riemer, K., Plattfaut, R., & Cleven, A. (2009). Reconstructing the giant: On the importance of rigour in documenting the literature search process. ECIS, Verona, Italy. Von Alan, R. H., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS quarterly, 28(1), 75-105. Wang, Y., Li, H., Feng, Y., Jiang, Y., & Liu, Y. (2012). Assessment of programming language learning based on peer code review model: Implementation and experience report. Computers & Education, 59(2), 412-422. Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS quarterly, 26(2), xiii-xxiii. Wellman, B. L., Anderson, M., & Vrbsky, S. V. (2009). PREOP as a tool to increase student retention in CS. Journal of Computing Sciences in Colleges, 25(2), 167-175. Werner, L., Denner, J., Campe, S., Ortiz, E., DeLay, D., Hartl, A. C., & Laursen, B. (2013). Pair programming for middle school students: does friendship influence academic outcomes? Proceeding of the 44th ACM technical symposium on Computer science education, Denver, Colorado, USA. Williams, D. C., Ma, Y., Prejean, L., Ford, M. J., & Lai, G. (2007). Acquisition of physics content knowledge and scientific inquiry skills in a robotics summer camp. Journal of Research on Technology in Education, 40(2), 201-216. 79
Wilson, B. C., & Shrock, S. (2001). Contributing to success in an introductory computer science course: a study of twelve factors. Proceedings of the thirty-second SIGCSE technical symposium on Computer Science Education, Charlotte, North Carolina, USA Wilson, J. R. (2002). Responsible authorship and peer review. Science and Engineering Ethics, 8(2), 155-174. Wu, C.-C., Tseng, I.-C., & Huang, S.-L. ( 2008). Visualization of program behaviors: Physical robots versus robot simulators. In M. M. S. Roland Mittermeir (Ed.), Informatics Education-Supporting Computational Thinking (pp. 53-62). Berlin/Heidelberg: Springer. Zweben, S. (2005). 2003-2004 Taulbee Survey. In Computing Research Association (publ.), Computing Research News (17). Washington: Computing Research Association. Retrieved from http://archive.cra.org/statistics/survey/04/04.pdf
80
Appendix
81
Appendix A Literature Review Search Process Appendix A.1 Third Set of Literature Review Search Keywords Concepts motivation creation gui tool tools
Ankles of View applied success benefits advantages development theory factors areas application promising approaches future use introduction methods
Other Words in pdf of
Table 18: Specific Keywords and Ankles of View as Third Set of Search Keywords Source: Own Depiction
Appendix A.2 Fourth Set of Literature Review Search Keywords Application Methods Areas textile feedback programming performance
Theory
Tools
Researchers
Robots
selfdetermination theory constructivism
alice
john hattie
affordable education robot
snap
vihavainen bennedsen
motivation theory intrinsic motivation
scratch edward deci
affordable games and education robot learning harvard Arduino
marina papastergiou louis major maja pivic brian greene arto vihavainen
calliope mini lilypad aerobot lego mindstorm
Table 19: Keywords of Particular Interest as Fourth Set of Search Keywords Source: Own Depiction
Appendix A.3 Lists of all Search Terms
82
Search Terms Composed From the First Three Sets of Keywords
Search Terms Including the Fourth Set of Keywords
tool teaching programming
selfdeterminationtheory
tool teaching programming review
self determination theory
tool programming review
constructivism
tools teaching programming
constructivism education
tool computer programming review
constructivism in education
programming education tools review
motivation theory education constructivism
programming education tools
intrinsic motivation
motivation
constructivism pdf
teaching
Constructivism in Computer Science Edu
motivation learning
education robots programming constructivism
success teaching programming
constructivism education
teaching age groups
snap computer science education j’_rg m’_nig
teaching in age groups
alice
teaching teenagers
alice programmin
motivation theory
alice programming
motivation theory education
snap computer science
motivation creation
snap computer science education
education summary
alice programming
education review
scratch computer science education
education report
alice progrmaming
education analysis
scratch languaga
programming education summary
scratch language
programming education survey
John Hattie
pdf learnging robots education computer science middle school summary
Edward L. Deci
middle school programming robots education computer science middle school review
brian greene
robot education middle school review
vihainen bennedsen
middle school computer programming review
vivhainen bennedsen
robot education middle school summary
bennedsen education
middle school education
vihainen bennedsen education
robots education middle school summary
Arto Vihavainen
middle school teaching robotics education programming middle school summary
teaching programming arduino
middle school programming review
aerobot education
teaching in middle school
programming education lilypad
robot education middle school
mindstorm programming
teaching high school
affordable education robot
Marina Papastergiou
Marina Papastergiou
aerobot
83
teaching high school germany
lego minstorm programming education
gui
aerobot harvard
gui development
affordable education robto
programming education gui
affordable education robot aerobot
robots education review
calliope mini
robots teaching review
calliope mini robot education
robots education programming middle school summary
textile programming
programming robot
feedback performance programming
robots education
Literature Review in Games and Learning
review programming robots
L. Major
computer programming review robot
L. Major robots programming
computer programming review robot education
L. Major robots programmint
informatics review robot education
Louis Major
education robots
pivic maja
education robots programming programming robot teaching teaching programming robot robots education programming robots education computer science programming education robot programming education robot tool robots programming education robotics programming review robotics programming education review robot tool programming education teaching computer science robots review teaching computer science robots education robots review teaching robots review education robot education robot review education robotics review robot programming robot programming education pdf programming robot programming applied programming applied in areas areas of applied programming programming programming applicatino programming application areas
84
promising approaches programming programming education computer programming education teaching programming review education programming education computer science teaching computer programming programming future use success factors programming education success factors informatics education informatics education informatics education success factors informatics education review informatik didaktik computer science education review programming education review programming education benefits benefits of programming education benefits programming education advantages programming education informatics education summary computer science teaching review teaching computer science review teaching programming review motivation programming introduction motivation programming introduction review review programming education methods review programming education tools review programming education computer science secondary school secondary school computer science education computer science secondary school secondary school programming programming education review secondary school programming education secondary school education review secondary school programming Table 20: Lists of all 167 Search Terms Source: Own Depiction
Appendix A.4 Backward Searches 85
Title of Paper Retrieved Experiences with the LEGO MindstormsTM throughout the Undergraduate Computer Science Curriculum Teaching Computer Science With Robotics Using Ada/Mindstorms 2.0 Measuring the effectiveness of robots in teaching computer science Teaching Problem Solving, Computing, and Information Technology with Robots Teaching Programming with Robots: A Case Study on Greek Secondary Education. Visualization of Program Behaviors: Physical Robots Versus Robot Simulators CSbots: Design and Deployment of a Robot Designed for the CS1 Classroom Personalizing CS1 with Robots PREOP AS A TOOL TO INCREASE STUDENT RETENTION IN CS* The Effect of 4-H Robotics and Geospatial Technologies on Science, Technology, Engineering, and Mathematics Learning and Attitudes An Autonomous Educational Mobile Robot Mediator
Title of Original Paper Systematic literature review: Teaching novices programming using robots Systematic literature review: Teaching novices programming using robots Systematic literature review: Teaching novices programming using robots Systematic literature review: Teaching novices programming using robots Systematic literature review: Teaching novices programming using robots Systematic literature review: Teaching novices programming using robots Systematic literature review: Teaching novices programming using robots Systematic literature review: Teaching novices programming using robots Systematic literature review: Teaching novices programming using robots Exploring the educational potential of robotics in schools: A systematic review Exploring the educational potential of robotics in schools: A systematic review
Table 21: Backward Searches Source: Own Depiction
Appendix A.5 Concept Matrix
86
Table 22: Concept Matrix – Part 1 Source: Based on (Webster & Watson, 2002)
87
Table 23: Concept Matrix – Part 2 Source: Based on (Webster & Watson, 2002)
88
Appendix B Design Process Appendix B.1 Feedback Items for the Lesson Prototype Item No.
Amount of Occurrences (default is 1)
Feedback Item
1
problems with keyboard key locations
3
2
wrong bracket syntax
3
he does not know how to go back a row
4
no row numbers in editor
5
error messages on simulator instead of robot
6
explain editor window
7
unclear tasks
5
8
students should think 3 times, then act
9
kids should look on robot after code execution
10
clear definitions of columns on robot display
11
understanding syntax is taking a long time
12
spelling mistakes prevent code compilation
13
explain error messages
14
syntax is frustrating
15
every level should be on one page
16
bad and unclear pictures
17
problems with understanding values to insert
2
18
arrows with lines and columns should be included tell that program in second level should be run the same way as in first
20
exclude useless texts in programming environment
3
21
clearer distinction between level and challenge Table 24: Feedback Items for the Lesson Prototype Source: Own Depiction
19
Appendix B.2 Feedback Items for the Improved Lesson Item No.
Amount of Occurrences (default is 1)
Feedback Item
1
not able to start without explanation of which booklet does what
2
dragging is a problem without mouse the dragged block in the booklet "how to run a program" is used instead of the one explained in the task
4
error message buggy, shows errors twice
5
write that student should flip pages
2
6
field can be dragged out of condition block
3
89
7
presenting a program in the beginning
8
delete all blocks when level is finished
2
9
tell students how to click A-button
10
include picture of button A
11
button B is broken
Table 25: Feedback Items for the Improved Lesson Source: Own Depiction
90
