Effectiveness of Integrating Program Visualizations to a Programming ...

Effectiveness of Integrating Program Visualizations to a Programming Course Essi Lahtinen

Tuukka Ahoniemi

Anniina Salo

Institute of Software Systems Tampere University of Technology, PO Box 553, Tampere, Finland {essi.lahtinen, tuukka.ahoniemi, anniina.salo}@tut.fi

Abstract To gain optimal benefits from program visualizations, their use in courses should be consistent. Instead of just covering all the topics they should also cover all the different learning situations and cognitive levels relating to the subject. We have integrated visualizations accordingly to our programming course and present some results about their usage in this paper. Our students showed a significant interest in using the visualization tool voluntarily. Keywords: Program visualization, course integration 1

Introduction

As the concepts needed in programming are abstract and thus difficult to understand and apply for many students, it is tempting to use visualizations to concretize teaching programming. However, research on the field of using visualizations in programming education has shown very inconsistent results: on the one hand visualizations have been useful for students (Nevalainen and Sajaniemi, 2006; Ben-Bassat Levy et al., 2003; Ahoniemi and Lahtinen, 2006) and on the other hand they can even distract the students’ attention from the essentials (Hundhausen, 2002). A large metastudy by Hundhausen et al. (2002) concludes that how the visualizations are used is more important than what is shown to the students. The aim of our long-term course development has been to find how to use program visualizations most beneficially in an introductory programming course. We have developed ideas on the content and usage of visualizations, technical means for implementation, and a model on how to integrate the visualizations into the teaching. We have now combined these developments and our students are using visualizations as optional learning tool in an introductory programming course. This article analyses students’ use of visualizations when guided this way, and their opinions on the visualizations. 2

Basis for Integrating Visualizations to the Programming Course

The compiler is an essential tool in a programming course. It is not used all the time, and definitely not for all purposes. For instance, when designing a program the student should not be using the compiler. Nonetheless, the students always know when c Copyright °2008, Australian Computer Society, Inc. This paper appeared at the Seventh Baltic Sea Conference on Computing Education Research (Koli Calling 2007), Koli National Park, Finland, November 15-18, 2007. Conferences in Research and Practice in Information Technology, Vol. 88. Raymond Lister and Simon, Eds. Reproduction for academic, not-forprofit purposes permitted provided this text is included.

they need the compiler and find it at the right moment for its use. If we want to make the use of visualizations beneficial for the students, they should have a similar understanding of when to use the visualization tool. This is our aim in integrating the visualization tool to the course. Naps et al. (2003) emphasize that user engagement is important in making the visualizations effective for learning. On this basis we have developed ideas on how to make students work with the visualization instead of just watching it. To make this kind of visualization technically possible we developed a mechanism called annotations for our visual interpreter tool. Further, both to improve the engagement and to make the visualizations as beneficial for students as possible, we developed a model for integrating the visualizations into the teaching. This work is summarized in the following subsections. 2.1

Supporting Higher Cognitive Skills with Visualizations

It seems that the biggest problem for novice programmers is not so much understanding basic programming concepts as learning how to apply them (Winslow, 1996). We have developed ideas on how to produce visualizations for novice programmers to enhance not only the students’ understanding of concepts but their active programming skills. The ideas for developing visualizations are divided into different levels according to Bloom’s taxonomy of the levels of cognitive development (Lahtinen and Ahoniemi, 2005). The categories are presented, with brief explanations, in Table 1. The aim of developing this categorization of visualizations was to help visualization tool developers and teachers recognize the different kinds of skill that need to be supported in learning programming. Programming courses are not only about introducing programming concepts; hence there is no point in providing only illustrative visualizations for the students. Illustrative visualizations can be used when a new programming concept is introduced, but after that we need to support the students’ learning process with the other types of visualization. This way the visualization tool becomes useful through the whole course, and integration of the visualizations and the course becomes more realistic. 2.2

Annotations for Defining Interactive Instructions

To be able to build visualizations on all the levels of cognitive development as described in the previous subsection, the visualization tool needs to interpret program code and give feedback on code written by the student. To enable this we developed a mechanism called annotations.

Level of the taxonomy 1. Knowledge & 2. Comprehension 3. Application

Visualization category

Characteristics and examples

Illustrative visualizations

4. Analysis 5. Synthesis

Problem-solving visualizations Productive visualizations

6. Evaluation

Discerning visualizations

Presenting concepts, demonstrative, used to introduce a new subject Modify some parts of the program code to make certain changes to the functionality of the program, use visualization to better understand the program and the changes Debug broken program code, analyse program behaviour, use visualization to help understanding Use the visualization tool as a visual debugger when the student independently writes program code, or visualize the assignment description Visualizing solutions to a problem and comparing or evaluating them

Utilizing visualizations

Table 1: Bloom’s taxonomy applied to visualization A program visualization tool typically shows the visualized program code in a window that can include short comments as a part of the code. However, the purpose of the visualization is to thoroughly explain the program’s execution to the student. To aid this, the visualization tool can include extra instructions in a separate window to describe the program and to draw the student’s attention to the essentials. To improve the quality of the instructions, the visualization tool developer can add annotations to define the visibility of the instructions. For example, a certain instruction could be shown only when the execution reaches the statement for the first time. When the execution returns to that statement there will be a new, more current instruction. This way the teacher can make the instructions more useful and interesting for the student. For example, when a function call statement is first encountered, the teacher would probably want to explain how parameter passing happens. When the function call finishes and the execution returns to the same line, the teacher can explain the passing of the function’s return value. In a similar way, taking the benefits of using an interpreter, it is possible to add some instructions that will be shown to the student depending on the state of the program, e.g. values of desired variables. The student can then be shown specific guiding instructions if his own program code does something wrong, or congratulated if the program state is correct. This idea is explained more in detail in an earlier article (Lahtinen and Ahoniemi, 2006). With these special annotations that can access information about the program state, we have been able to build utilizing and problem-solving visualizations for the needs of a CS1 course. These appear to the student as normal interactive exercises that they can use when deepening their knowledge on a subject. 2.3

Integration of the Visualizations and the Course

Even though there are numerous visualization tools available for use in programming courses, the visualizations are still often not integrated in the learning situations. If there are a couple of visualizations available every now and then during the course, students do not develop the habit of using them regularly and they are easily forgotten. Besides covering all the subjects handled, the visualizations should cover multiple cognitive levels within one subject so that the student can benefit from them throughout the course. To gain maximal benefit from the visualizations, the teacher needs to plan their use precisely and actively guide the students in using them. One flaw in the use of visualizations is that the

Visualization tool

Course website

Printed materials

Programming assignments

Teaching situation

Student

Figure 1: A model for integrating the visualizations into the rest of the course students do not find them or do not realize the right time to use them. This occurs when the visualization tool is in a separate learning environment, e.g. on a web page that is not connected to the course material. Other little details, such as the need to type a password for the web page, may also hinder the use of the visualization tool. To increase the use of the visualization tool it should be integrated with all the other course materials. We have used a model, (Lahtinen, 2006) illustrated in Figure 1, where the teacher includes links and references to the visualizations throughout the course material. The printed course material contains identification numbers for the visualizations, which are available on the front page of the visualization tool. The course web page where we publish the weekly exercise session materials contains hyperlinks to the related visualizations. Students can do some parts of the assignments using the visualization tool, and it is also used in some of the teaching situations. The idea of this model is to exploit the fact that students already know how to use the printed course materials in studying and they have to use the course web page to get all the practical information. With all the links and references, even the students who do not plan their learning situations and their use of the materials will benefit from the visualization tool when appropriate. 3

Target Course and Research Setting

The integration described in the previous section has been used in an introductory course for programming (CS1) in Tampere University of Technology. Prior studies examined the effects of integrating the visual-

Table 2: Subjects of pre-exercises in different weeks and the visualization categories they fall into Ex. Topic Visualization type 2 If statements Utilizing visualization 3 Loop structures Problem-solving + utilizing visualization 4 Arrays Problem-solving + utilizing visualization 5 Functions Utilizing visualization izations with the printed course materials (Lahtinen, 2006) and the use of visualizations in exercise sessions for a test group of students (Ahoniemi and Lahtinen, 2006). Now we wanted to explore whether and how the students would voluntarily use visualizations when doing small programming assignments. In addition to the normal programming tools we made a visualization tool available to the students. The weekly exercise sessions of the course, which were not compulsory, included an optional pre-exercise assignment. Students who opted to do this pre-exercise could do it with pen and paper, with the help of an ordinary compiler, or with the visualization tool. The web page where the pre-exercise assignments were given had direct links to the same assignments in the visualization tool. The visualization tool we use in our course is called VIP (Visual InterPreter) (Virtanen et al., 2005). It is based on an interpreter that uses a simple subset of C++ (namely C--). The interpreter accepts regular source code files, executes them, and automatically illustrates the program step by step. Using the annotation technique described in Subsection 2.2, we prepared pre-exercise assignments for four exercise sessions during the course. Two of these were either to modify an existing structure or to add a clearly specified new functionality to a given program, and were thus utilizing visualizations. The other two exercises had two parts: a problem-solving visualization, i.e. a task to find out what the program does, and a utilizing visualization, i.e. to make changes to the functionality of the program. The subjects of the pre-exercises are presented in Table 2. During these four weeks we monitored the number of pre-exercise assignment submissions and especially the number of students who voluntarily used the visualizations when doing the pre-exercise. After the four weeks we also asked the students, in a short survey, to reflect on the usefulness of visualizations with the pre-exercises. 4

Visualization Usage

Since students could choose each week how they did the exercise, there were numerous different combinations of behaviour. We identified some groups of student whose occurrences are presented in Figure 2. A quarter of the students seemed to appreciate the visualization tool since they had done the exercises using it every time they turned in a solution. On the other hand, almost half of the students did not use the visualization tool for any of the solutions. There were two groups of students who changed their opinion about using the visualization tool: 7% who did the early exercises without the visualization tool but moved on to using visualizations, and 2% who used the visualization tool in the beginning but stopped using it. A further 7% alternated between using the tool and not using it, and 16% used other combinations. The drop-out rate of our course is high, as is typical for programming courses. Often the first sign of failing the course is that the student stops attending

Figure 2: How students chose to do their exercises (n=308)

Figure 3: Numbers of exercises turned in by the students who always used VIP and those who always used pen and paper the exercise sessions, or at least handing in solutions. Since there were two groups of students who had a clear opinion, either for or against using the visualization tool, we observed the drop-out rates of these two groups. Figure 3 shows the numbers of exercise submissions from students in these two groups. Activity decreased among students of both groups, but more sharply among the pen and paper group (a drop of about 80%) than among the visualization group (a drop of about 63%). It seems that students who consistently use the visualization tool are more likely to continue completing the assignments. We asked the students who completed the assignments using the visualization tool to assess whether the tool was useful in the assignment. Figure 4 shows the students’ opinions by assignment. The more difficult or abstract the content of the exercise, the better evaluation the visualization gets from students. For instance, students often perceive if-statements easy and functions more difficult. There was no difference between the visualization types as presented in 2. The exercises where the visualization was seen as most useful were of different kinds: exercise 4 had both problem-solving and utilizing visualization and exercise 5 had only utilizing visualization. 5

Discussion

In our earlier research we discerned that linking the printed course materials with the visualizations increases the students’ use of the visualizations when they perform their course activities (Lahtinen, 2006). Integrating the exercise session assignments with the visualization tool was the next step towards full inte-

6

Conclusions

We guided our students so that using visualizations as extra material was easy and many of the students chose to use them voluntarily throughout the course. Another benefit we gained by using an interesting learning tool was that the students using it continued to take a more active part in the course. These results show that developing the use of visualizations in our course was worthwhile. We will keep improving their usage and their integration with the course. 7

Acknowledgments

The Nokia Foundation has contributed to the funding of this work. Figure 4: How useful students who used VIP found it in the assignments gration of the visualization tool. Overall, the integration seems to be successful. A quarter of the students always used the visualization tool in their assignments even though its use was voluntary. About half of the students tried it at least once for completing an assignment. The rates could be higher, but considering that using a new tool always requires some extra curiosity and effort, we know that some students will never do it unless it is obligatory. In fact, there is no need for all students to use visualizations. They can be a good aid for students with problems and completely useless for students who already understand programming well. We think it is best to let the students choose whether to use the visualization tool or not. If we want to use the visualization tool more extensively, the next step would be to start using it to help with the bigger programming assignments, the only place where it is not now used. However, our goal is not to develop a programming course that uses visualizations for everything but to support the students with visualization where advantageous. The students using the visualizations to solve the exercises appeared more likely to keep turning in the assignments weekly than the students who never used the visualizations. There was a clear difference in the drop-out rates of these two groups. The assignments can be a bit easier when you have the visualization to make them more concrete. However, we cannot conclude that it is the visualization tool that keeps the students doing the assignments. It might be that the most active students were also the most eager to use extra materials in their learning. The students in the other group were still active enough to do the voluntary pre-exercises, and their drop-out rate is still high. Whatever the explanation, visualizations seemed to be helpful since some students used them week after week. If the visualization can make the homework assignments more interesting for some students, by adding another perspective and thus keeping them working weekly, why would we not use it? According to the collected data, the more difficult the topic, the more useful the visualizations become. This conflicts with the fact that visualizations are most often used only in the beginning of a course to present new concepts, and are forgotten later in the course. The visualizations on the higher levels of cognitive development and the integration model are a good start for developing the use of visualizations towards supporting the students in the more difficult topics and skills.

References Ahoniemi, T. and Lahtinen, E. (2006). Visualizations in Preparing for Programming Exercise Sessions, in Proceedings of the Fourth Program Visualization Workshop, Florence, Italy, pp. 54-59. Ben-Bassat Levy, R., Ben-Ari, M. and Uronen, P. A. (2003). The Jeliot 2000 program animation system, Computers & Education 40(1), 1-15. Hundhausen, C. D. (2002). Integrating algorithm visualization technology into an undergraduate algorithms course: Ethnographic studies of a social constructivist approach, Computers & Education 39(3), 237-260. Hundhausen, C. D., Douglas, S. A. and Stasko, J. T. (2002). A meta-study of algorithm visualization effectiveness, Journal of Visual Languages & Computing 13(3), 259-290. Lahtinen, E. (2006). Integrating the Use Of Visualizations to Teaching Programming, Proceedings of the conference Methods, Materials and Tools for Programming Education pp. 7-13. Lahtinen, E. and Ahoniemi, T. (2005). Visualizations to Support Programming on Different Levels of Cognitive Development, Proceedings of The Fifth Koli Calling Conference on Computer Science Education pp. 87-94. Lahtinen, E. and Ahoniemi, T. (2006). Annotations for Defining Interactive Instructions to InterpreterBased Program Visualization Tools, in Proceedings of the Fourth Program Visualization Workshop, Florence, Italy, pp. 34-38. Naps, T., R¨ossling, G., Almstrum, V., Dann, W., Fleischer, R., Hundhausen, C., Korhonen, A., Malmi, L., McNally, M., Rodger, S. and VelazquezIturbide, J. (2003). Exploring the role of visualization and engagement in computer science education, SIGCSE Bulletin 35(2), 131-152. Nevalainen, S. and Sajaniemi, J. (2006). An experiment on short-term effects on animated versus static visualization of operations on program perception, Proceedings of the 2006 int’l workshop on Computing education research ICER ’06 . Virtanen, A. T., Lahtinen, E. and J¨arvinen, H.-M. (2005). VIP, a visual interpreter for learning introductory programming with C++, Proceedings of The Fifth Koli Calling Conference on Computer Science Education pp. 125-130. Winslow, L. E. (1996). Programming pedagogy – a psychological overview, SIGCSE Bulletin 28(3).