Can Natural Language be Utilized in the Learning of Programming ...

Can Natural Language be Utilized in the Learning of Programming Fundamentals? Osvaldo Luiz Oliveira, Ana Maria Monteiro

Norton Trevisan Roman

Computer Science Department Faculty of Campo Limpo Paulista Brazil [email protected], [email protected]

School of Arts, Sciences and Humanities University of São Paulo Brazil [email protected]

Abstract—The complexity and importance of learning programming fundamentals (i.e., sequences of sentences that express actions, conditions, and repetitions in computing) for undergraduate students has motivated the development of an intense educational research area. One frequently studied problem is the difficulty in the learning of traditional context-free grammars which are present, for example, in programming languages such as Pascal and C. This study experimentally investigates the use of natural language in the learning of programming fundamentals by two groups of undergraduate students without prior knowledge of programming and compares its use with that of a traditional grammar language. Results suggest that the use of natural language is a good alternative, despite the small differences, to the use of traditional programming languages defined by context-free grammars. This alternative is attractive and promising because the student does not need to learn a formal grammar to learn the fundamentals of programming. Keywords—programming fundamentals learning; CS1; natural language programming; human factors.

I.

INTRODUCTION

The learning of programming fundamentals (i.e., sequences of sentences that express actions, conditions, and repetitions in computing) by undergraduate students is a notoriously difficult task. Programming languages such as C and Pascal are often used for this purpose. These languages are defined by grammatical constructions typical of the context-free class in the Chomsky hierarchy [1]. The main features of these languages are: (i) the generation of nested sentences that may be imperative, conditional (e.g., “if”), or repetitive (e.g., “while”); (ii) the existence of markers for the beginning (e.g., “begin” or “{”) and the ending (e.g., “end” or “}”) of blocks of sentences; (iii) the use of balanced parentheses to change the priority of operations within expressions; and (iv) the explicit declaration of the subject (the thing about which something is said) in all clauses by means of parameters and arguments of functions and procedures. Natural language grammars, on the other hand, differ from context-free grammars in that they: (i) do not usually produce texts with nested sentences; (ii) use capital letters to mark the beginning of a sentence; (iii) use punctuation (e.g., periods and question marks) to mark the end of a sentence, and use commas to separate clauses within a sentence; (iv) allow the generation

978-1-4673-5261-1/13/$31.00 ©2013 IEEE

of expressions composed of articles, prepositions, and adverbs; and (v) allow the writing of sentences in which the subject is not explicitly stated. This study investigates the use of natural language in the learning of programming fundamentals and compares its use with that of a language with traditional grammar. A microworld composed of robots, walls, and disks was used as the source of problems, and two languages (and corresponding compilers) were developed: one following a classical contextfree grammar notation (MRt), and the other with a grammar similar to that of the Portuguese language (MRp). The learning of programming fundamentals was experimentally investigated using two groups of undergraduate students with no previous experience in programming. Group I used the MRt language, and Group II used the MRp language. One of the main contributions of this work is the way the study was planned. With the intent of studying only the effects of the type of grammar between groups, the experiment was carefully designed so that all other variables that might affect learning remained constant, with the type of grammar being the only variable to be measured. The rest of this paper is organized as follows. Section II describes the two main streams of research in the field that motivated this study. The materials and methods used in the experiments are described in detail in Section III. In particular, the languages created are presented, along with the precautions taken to isolate the type of grammar from other variables. Finally, Sections IV and V present results, discussion, and conclusions. II.

RELATED WORK

This study was motivated by the difficulties that novices face when learning the grammar of traditional programming languages. On this issue, there seems to be two main streams of research that are noteworthy. One of them is characterized by an attempt to facilitate the understanding of traditional programming language grammars, a line of research that has been extensively explored to date. Contributions along this line range from the proposal of new pedagogical components [2] aimed at the development of programming skills in general and at the learning of specific language grammars, such as Interactive Learning Objects (ILOs) [3], for example, which draw on software visualization

Robot with one disk in its bag

work from the early 1980s, to visual programming platforms that show graphical representations of the program in a flow control diagram (flowchart) (e.g., [4]). In the meantime, other alternatives have been proposed, such as new didactic methods and software tools which exemplify how the involvement of eyesight, hearing, and kinesthesia can help in the teaching and learning of a programming language grammar [5]; the use of 3D object animation, along with the development of an editor with a drag-and-drop tool to make it easier to write programs [6, 7, 8]; and the suggestion that writing the solution of a problem in English before coding reduces the cognitive load for novices, thus enabling them to better cope with the difficulties of learning of programming language grammars [9]. The second line of research investigates how languages for novices might be designed which are in line with their natural way of thinking about programming. The very idea of closing the gap between programming languages and natural language grammars has found support in some empirical studies [10, 11, 12], which show that novices writing in English exhibit fairly advanced problem-solving skills, but that programming languages pose barriers that create difficulties for them in expressing these same skills in programming. In addition, studies [13] have used knowledge from the field of humancomputer interaction (HCI) to understand how people think about programming tasks when trying to create a new program or trying to find and fix bugs in existing programs. Finally, Metaphor [14] and Pegasus [15] are examples of efforts to develop natural language programming systems (for a brief overview of the contributions and challenges of redesigning programming languages to make them more accessible to beginners, see [16]). III.

MATERIALS AND METHODS

The World of the Robots (Fig. 1) was designed for the experiments we describe here. This source of problems is an adaptation of the Karel’s World [17]. It is made up of a rectangular board representing the “world”, robots that can move within this “world”, walls, and disks. The World of the Robots creates a source of problems related to the movement and manipulation of objects (disks) on a board possibly including barriers formed by walls. A robot can perform the following elementary operations: move forward, rotate 90º degrees left, and collect or drop a disk (at the robot’s position). It can also test whether the space in front of it is clear (i.e., if there is no wall before it), if it is sitting above a disk, and if there are any disks in its bag (each robot has a bag where it keeps its collected disks). It can also check whether it is pointing north, south, east, or west.

Compiler messages

Five disks at this position

Vertical and horizontal walls

Indicator of the language selected

Fig. 1. The Integrated Programming Environment Developed.

The following components of the integrated programming environment were also developed: a graphical editor, used to design configurations for the World of the Robots; a text editor to edit programs; compilers for both languages used in the experiments (MRt and MRp); and a debugger (Fig. 1). By selecting buttons on the graphical editor toolbar, the user can create and delete robots, disks, and vertical and horizontal walls. In response to a request to execute a program, the programming environment shows an animation of the programmed movements. The program can also be run in debug mode to trace the execution of the source code, line by line, and to view the corresponding animation in the World of the Robots. A. MRt Language MRt was designed with the traditional flow-control structures of the imperative paradigm based on a context-free grammar, as found in programming languages like Pascal and C. Thus1: • Sentences such as MoveForward(r), TurnLeft(r), TakeDisk(r), and DropDisk(r) make a robot r, respectively, move one step forward, turn left 90º, and catch or drop a disk in its current position. • Boolean expressions using the operators and, or, and not can be built up from Boolean functions such as 1

The languages developed in this work use Portuguese terms. To facilitate the reading of this article, these terms have been translated into English sentences.

FrontClear(r), OnDisk(r), ThereAreDisksInBag(r), HeadsNorth(r), HeadsSouth(r), HeadsEast(r), and HeadsWest(r), which evaluate to true or false, respectively, if before the robot r there is no wall; if r is on top of a disk; if there is at least one disk in the bag of the robot; and if r points north, south, east, or west. • Conditional sentences, such as if FrontClear(r) then MoveForward(r) else TurnLeft(r), test whether the space in front of robot r is clear and, if so, make the robot move forward, or else make it turn left. • Repetitive sentences, like while FrontClear(r) and OnDisk(r) do TakeDisk(r), repeat an instruction. In this case the robot named r should collect a disk if there is one at its position and the space before it is clear. • Sets of sentences can be grouped by means of a sentence that begins with the word begin and ends with the word end. An example of a complete program in MRt language is shown in Fig. 2. The program solves the problem of making a robot find a disk within a rectangular circuit. The initial orientation (north, south, east, or west) of the robot is unknown. A possible initial problem configuration is shown in Fig. 3. while not HeadsWest (r) do TurnLeft (r); while not OnDisk (r) do begin if FrontClear (r) then MoveForward (r) else TurnLeft (r) end

programa AcharDisco; Usa r: Robô; início enquanto não ApontaParaOeste (r) faça GirarEsquerda (r); enquanto não EstaSobreDisco (r) faça se FrenteEstáLivre (r) então AndarFrente (r) senão GirarEsquerda (r) fim.

Fig. 2. A program written in the MRt language.

Fig. 3. An initial problem configuration.

B. MRp Language The MRp language was designed to allow the generation of sentences similar to those of Portuguese. The body of a program written in MRp consists of a sequence of zero or more sentences, each ending with a dot. Every sentence consists of one or more clauses, separated by commas or by the conjunction and. For example, Make the robot r turn left, move forward and drop a disk is a sentence consisting of three clauses: (1) make the robot r turn left, (2) move forward, and (4) drop a disk. A clause always has a subject, that is, the thing to which it refers. The subject may be explicitly declared as r, as in the clause make the robot r turn left, or it may be implicit, as in the clauses move forward and drop a disk. In the MRp language, implicit subjects always refer to the subject of the preceding clause. In the example, the subject of clauses move forward and drop a disk is therefore r, the subject of make the robot r turn left. Here are some examples of sentences and clauses in the MRp language and their meanings: • If there is no disk in the bag of the robot r and the robot is on top of a disk, then collect the disk, or else turn left. This sentence determines that the robot named r should collect a disk if it is currently above one and its bag is empty. Otherwise, it should turn left. • While there are disks in the bag of the robot r, drop a disk. The meaning of this sentence is to have the robot named r empty its bag, putting all the disks in its current position. • While the space in front of the robot r is clear, make it move forward and if it’s on a disk, collect the disk. This sentence makes robot r keep on collecting disks that are in front of it until it hits a wall. An example of a complete program in the MRp language, to solve the problem of making a robot find a disk within a rectangular circuit, is given in Fig. 4. A possible initial configuration of the problem is shown in Fig. 3.

While the robot r does not point to the west, make it turn left. While r is not on a disk, if the robot’s front is clear, move forward or else turn left. Enquanto o robô r não aponta para oeste, faça ele girar à esquerda. Enquanto r não está sobre um disco, se a frente dele está livre, andar para frente senão girar à esquerda.

Fig. 4. A program written in the MRp language.

C. Subjects Twenty-two volunteers participated in the study. All subjects were first-year students in undergraduate courses in a Brazilian university. Among these, 13 (59%) were male and 9 (41%) female, of whom 11 (50%) were studying exact or applied sciences and 11 (50%) were from the social sciences, with ages ranging from 17 to 28 (average 20 and median 18). Moreover, none of the participants had prior knowledge of programming, as determined by a questionnaire. D. Hypothesis The hypothesis tested was: The use of MRp language (with grammar similar to a natural language) is as favorable as the use of MRt language (with a traditional context-free grammar) to the learning of programming fundamentals. E. Experimental Setup Initially, participants were randomly divided into two groups:

In Phase 2, students took part in a test. The test took up to two hours, and consisted of four questions that described problems. They were then asked to write programs to solve these problems. Both groups received the same test, consisting of four questions: • Question 1 evaluated the use of a sequence of sentences and could be answered with a sequence of sentences expressing elementary actions. • Question 2 evaluated the use of conditional sentences and could be answered with one or more sentences expressing conditions. • Question 3 evaluated the use of sentences denoting repetitions and could be answered with sentences describing a repetition of elementary actions. • Question 4 evaluated the combined use of sentences expressing actions, conditions, and repetitions and could be answered with sentences expressing the repetition of clauses describing conditions and actions. The test is fully presented in Appendix A. The questions answered by the participants were independently reviewed by two of the researchers and were assigned a score from 0 to 5. Inconsistencies were settled by the researchers in a meeting. F. Constant Factors and Other Precautions Because only the grammar of the programming language for the two groups was planned to vary, the following factors were held constant: • The “World of the Robots” domain as the source of problems. • The semantics of the MRp and MRt programming languages.

• Group I: Participants studying programming fundamentals using the MRp language; and

• The programming environment.

• Group II: Participants studying fundamentals using the MRt language.

• The experimenter administering the experiments.

programming

The experiment was conducted in two phases, occurring in the morning (phase 1) and afternoon (phase 2) periods of the same day, with a lunch break of 1.5 hour between phases. In phase 1, each group participated in a 3.5 hour lesson, with a 15 minute break halfway through. The lessons discussed details of the World of the Robots and the concepts of sequence of sentences, and sentences that express actions, conditions and repetitions present in the programming language to be used by each group. Before the 15 minute break, participants performed an exercise in which they had to deal with these concepts. The experimenter interacted with the participants individually during the time that each tried to solve the exercise and subsequently developed and explained a possible solution. Students were given a text describing all the issues covered in the classroom, and thus, during the lesson, their only concern was taking additional notes on the subject matter.

• The dynamics of the experiment (see section E). • The problems addressed in phase 1 (lesson) and phase 2 (test) of the experiments (Appendix A). As a way to avoid bias in the resulting data set, additional measures were also taken, such as: • Restricting the experiment to the time interval comprising morning and afternoon, thereby reducing the interference of external sources, to the extent that the entire experiment took place in a single day. • Splitting up subjects into two groups of 11 students, to facilitate the interaction between them and the experimenter, thus reducing the natural difficulty in communicating with large numbers of students. • Designing the programming languages used in the experiments so that their statements could be written in Portuguese, the subjects’ native tongue.

• Choosing the World of the Robots as the source of problems, because of the small number of involved concepts and also because these concepts are fairly intuitive. IV.

RESULTS AND DISCUSSION

Table I shows the mean and standard deviation scores for both groups on a scale from 0 (insufficient) to 5 (excellent). Means and standard deviations were almost identical: 3.91 ± 0.79 for Group I and 3.98 ± 0.82 for Group II. The results showed no statistically significant differences between these averages (Student’s t = 0.20 with significance level p = 0.85). This means that the proposed hypothesis could not be rejected. TABLE I.

RESULTS (MEAN AND STANDARD DEVIATION) OF THE TEST

Group I II

Mean (0 – 5) ± SD 3.91 ± 0.79 3.98 ± 0.82

n 11 11

Separating the syntactic and semantic aspects of the solutions, i.e., isolating analyzing syntax and semantic mistakes, resulted in the mean and standard deviations shown in Table II. Still, the data show no statistically significant difference with regards to aspects of syntactic correctness (t = 0.64 with significance level p = 0.53) or semantic accuracy (t = 0.22 with significance level p = 0.83). In other words, the hypothesis is also confirmed with respect to a comparison between the syntactic aspects of the languages or between their semantic aspects. TABLE II.

MEAN AND STANDARD DEVIATION OF TEST SCORES RELATIVE TO SYNTACTIC AND SEMANTIC ASPECTS

Group I II

Syntax mean (0-5) ± SD 4.02 ± 0.93 4.25 ± 0.72

Semantic mean (0-5) ± SD 3.80 ± 0.84 3.70 ± 1.05

Appendices B and C present in detail the scores obtained by each student in Groups I and II. V.

syntactic development of programs by dragging-and-dropping blocks of sentences. The use of natural language makes solutions like these unnecessary. Signs to mark the beginning and end of blocks of sentences are used to organize the structure of a program written in a traditional context-free language, allowing also the nesting of sentences and the resolution of ambiguity. The absence of these context-free language elements in natural language forces the writing of texts which are often less structured. For example, consider the text in natural language Make the robot r move forward until it finds a wall. While it is walking forward, make it pick up a disk at each position where it passes. The two actions (move forward and pick up a disk) were structured in two sentences. Typically, in a context-free language, these actions are structured in a single block within a single repetitive sentence. Future research by the authors will investigate the relationship between the structure of programs written in natural language and in traditional context-free languages. A better understanding of this phenomenon may help in the development of better compilers for natural language and also improve programming courses based on context-free languages. Finally, educators can use in courses and researchers can use in other studies the programming languages, compilers, and integrated programming environments that have been developed for these experiments. These will be made available to the community under a General Public License. ACKNOWLEDGMENT We would like to thank all subjects that volunteered to the experiment for their invaluable contribution to this work. APPENDIX A. THE TEST QUESTIONS Question 1: Write a program to signal a path with disks. The left figure presents an initial configuration in the World of the Robots and the right figure the final configuration that must be reached after executing the program.

CONCLUSIONS

These results show that, for the population studied, the use of natural languages is an alternative, not better or worse, to the use of traditional programming languages defined by contextfree grammars. This alternative is attractive and promising because the student does not need to learn a formal grammar to learn the fundamentals of programming. Using a natural language as the first programming language enables the student to focus on the logic of the solution, thus decreasing his level of effort in writing a program. Natural language programming is also an alternative for end-user programming environments, i.e., environments that include programming and are targeted at users whose main interest may not be intrinsically computational. For example, Scratch [18] has been used for educational and entertainment purposes for children to create interactive stories, music, works of art, and games. Scratch tries to alleviate the difficulties in

Question 2: Write a program that makes the robot r point north. You do not know the initial configuration of the robot in the World, in other words, you do not know whether the robot is currently pointing north, south, east or west. Question 3: Write a program to make the robot r take all disks in a line situated before it until it reaches a wall. The figures show a possible starting (left) and final (right) condition. These figures are only an example. Your program must work with lines with an undetermined number of disks.

[4]

[5]

[6]

Question 4: Write a program to make the robot r jump over barriers such as those in the figure until it finds a disk. Again, this figure is merely illustrative. Your program must address an undetermined number of barriers at unknown positions. The barriers are always equal in size to those indicated in the figure.

[7]

[8]

APPENDIX B. SCORES OBTAINED BY STUDENTS IN GROUP I [9] Student A1 B1 C1 D1 E1 F1 G1 H1 I1 J1 K1

Syntax score (0 - 5) 4.25 5.00 4.25 4.00 2.50 4.75 2.75 4.50 4.50 2.75 5.00

Semantic score (0 - 5) 4.50 4.50 4.00 4.25 3.75 3.50 1.75 4.50 3.25 3.25 4.50

Global score (0 - 5) 4.38 4.75 4.13 4.13 3.13 4.13 2.25 4.50 3.88 3.00 4.75

APPENDIX C. SCORES OBTAINED BY STUDENTS IN GROUP II Student A2 B2 C2 D2 E2 F2 G2 H2 I2 J2 K2

Syntax score (0 - 5) 4.50 4.50 4.25 5.00 4.75 5.00 3.50 4.00 2.50 4.25 4.50

Semantic score (0 - 5) 4.25 4.00 4.25 4.75 5.00 4.00 2.50 3.75 2.00 4.25 2.00

Global score (0 - 5) 4.38 4.25 4.25 4.88 4.88 4.50 3.00 3.88 2.25 4.25 3.25

REFERENCES [1]

[2]

[3]

Hopcroft, John E., Rajeev Motwani, and Jeffrey D. Ullman. 2006. Introduction to Automata Theory, Languages, and Computation. Boston: Addison-Wesley Longman Publishing, pp. 171-224. Gomes, Anabela, and Antônio J. N. Mendes. “Studies and proposals about initial programming learning.” In In Frontiers in Education Conference. 27-30 October 2010. Proceedings of the 40th ASEE/IEEE Frontiers in Education Conference. Washington, DC: IEEE Publisher, pp. S3F1-S3F6. Villalobos, Jorge A., Nadya A. Calderon, and Camilo H. Jiménez. 2009. “Developing programming skills by using interactive learning objects.” ACM SIGCSE Bulletin. Vol. 41 (3), pp. 151-155.

[10]

[11]

[12]

[13]

[14]

[15]

[16] [17] [18]

Lavonen, Jari, Veijo Meisalo, Matti Lattu, and Erkki Sutinen. 2003. “Concretising the programming task: a case study in a secondary school.” Computers & Education. Vol. 40 (2), pp. 115-135. Kátai, Zoltán, Katalin Juhász, and Alpár Károly Adorjáni. 2008. “On the role of senses in education.” Computers & Education. Vol. 51 (4), pp. 1707-1717. Esteves, Micaela. “Contextualization of programming learning: a virtual environment study.” In Frontiers in Education Conference. 22-25 October 2008. Proceedings of the 38th ASEE/IEEE Frontiers in Education Conference. Saratoga Springs, New York: IEEE Publisher, pp. F2A17-F2A22. Klassen, Myungsook. 2006. “Visual Approach for Teaching Programming Concepts.” In International Conference on Engineering Education. 23-28 July 2006. Proceedings of the 9th International Conference on Engineering Education. San Juan, Puerto Rico, pp. TIA 1- TIA 6. Mullins, Paul M., and Michael Conlon. 2008. “Engaging students in programming fundamentals using Alice 2.0.” In SIGITE Conference. 16-18 October 2008. Proceedings of the 9th ACM SIGITE Conference on Information Technology Education. Cincinnati, Ohio: ACM Publisher, pp. 81-88. Fidge, Colin, and Donna Teague. 2009. “Losing Their Marbles: syntaxfree programming for assessing problem-solving skills.” In ACE Conference. 20-23 January 2009. Proceedings of the 11th Australasian Computing Education Conference. Darlinghurst, Australia: Australian Computer Society, Inc., pp. 75-82. Chen, Tzu-Yi, Gary L. Xavier, Robert McCartney, Kate Sanders, and Beth Simon. 2007. “Commonsense Computing: using student sorting abilities to improve instruction.” In SIGCSE Conference. 07-11 March 2007. Proceedings of the 38th SIGCSE Technical Symposium on Computer Science Education. Covington, Kentucky: ACM Publisher, pp. 276-280. Simon, B., Tzu-Yi Chen, Gary L. Xavier, Robert McCartney, and Kate Sanders. 2006. “Commonsense computing: what students know before we teach (episode 1: sorting).” In ICER Conference. 9-10 September 2006. Proceedings of the 2006 International Workshop on Computing Education Research. Canterbury, United Kingdom: ACM Publisher, pp. 29-40. Xavier, Gary L., Dennis J. Bouvier, Robert McCartney, Kate Sanders, and Beth Simon. 2007. “Commonsense computing (episode 3): concurrency and concert tickets.” In ICER Conference. 15-16 September 2007. Proceedings of the Third International Workshop on Computing Education Research. Atlanta, Georgia: ACM Publisher, pp. 133-144. Myers, Brad A., John F. Pane, and Andy Ko. 2004. “Natural Programming Languages and Environments.” Communications of the ACM. Vol. 47 (9), pp. 47-52. Liu, Hugo, and Henry Lieberman. 2005. “Metafor: visualizing stories as code.” In IUI Conference. 09-12 January 2005. Proceedings of the 10th International Conference on Intelligent User Interfaces. San Diego, California: ACM Publisher, pp. 305-307. Knöll, Roman, and Mira Mezini. 2006. “Pegasus – First Steps Toward a Naturalistic Programming Language.” In OOPSLA Conference. 22-26 October 2006. Proceedings of the 21st Annual ACM Conference on Object-Oriented Programming, Systems, Languages, and Application. Portland, Oregon: ACM Publisher, pp. 542-559. Guzdial, Mark. 2008. “Paving the Way for Computational Thinking.” Communications of the ACM. Vol. 51 (8), pp. 25-27. Pattis, Richard, E. 1995. Karel the Robot: a gentle introduction to the art of programming. New York: John Wiley & Sons, pp. 1-19. Resnick, Mitchel, John Maloney, Andrés Monroy-Hernández, Natalie Rusk, Evelyn Eastmond, Karen Brennan, Amon Millner, Eric Rosenbaum, Jay Silver, Brian Silverman, and Yasmin Kafai. “Scratch: programming for all.” Communications of the ACM. Vol 52 (11), pp. 6067.