A cognitive model for solo programming and pair programming ...

A Cognitive Model for Solo Programming and Pair Programming Kim Man Lui and Keith C.C. Chan The Hong Kong Polytechnic University {cskmlui, cskcchan}@comp.polyu.edu.hk Abstract As eXtreme Programming (XP) has grown in popularity in both academia and industry, pair programming, two programmers collaborating on design, coding and testing, has become a controversial focus of interest. Many researchers and practitioners remain skeptical of claims for the productivity of pair programming. Three controlled experiments, by Nosek, William, and Nawrocki, all reported statistical productivity measurements, but the results were inconsistent and the differences among the results could not be explained because there exists no theoretical framework to articulate pair programming. This paper proposes a cognitive model that describes the programming mechanism and this in turn can be used to explain why and when a pair may outperform two individuals. The model helps explain discrepancies in the results of the above-mentioned experiments. To show how the proposed model can be applied, a psychological experiment was conducted in which twenty-two programmers who were informed only of the experiment and the results of Nosek, and of the descriptions of Williams and Nawrocki, were able to use our model to predict the possible outcomes of the experiments of Williams and Nawrocki.

1. Introduction The idea behind pair programming is straightforward. It involves two programmers collaborating side-by-side on design, coding and testing a piece of software. One, the Controller, controls the keyboard/mouse and actively implements the program. The other, the Navigator/Observer, continuously observes the work to identify tactical defects and to provide strategic planning. Although pair programming was reported as early as 1995 [1], the practice has recently drawn much attention because of the increase in popularity of eXtreme Programming (XP) [1, 2]. A number of controlled experiments have been conducted to evaluate the effectiveness of pair programming. These experiments had a number of

similarities. First, subjects were divided into a pairprogramming group and a solo-programming group. The two groups were then asked to write the same program so that their results could be directly compared. Oddly, the experiments produced different and apparently incompatible results. Nosek reported that pairs take 42% more time than individuals [3]. Williams reported that pairs spend an insignificant 15% more time than individuals on the same task, yet the quality of pair programming was higher [4]. Nawrocki’s study indicated that pair programming could be less efficient [5]. Note that, in the statement that a pair takes y % more time than an individual, y % is defined as: ( finish_ time_ of _ pair) × 2 − ( finish_ time_ of _ individual ) × 100% finish_ time_ of _ individual

The experiments done by these different teams appear to indicate that productivity in pair programming can vary significantly but the absence of an underlying framework that addresses the mechanism of the programming process makes it hard to account for the productivity differences among the experiments. There is a need for a model of programming that allows us to comprehend pair programming experiments and to predict how and when pair programming works better than solo programming. This helps us apply pair programming in real situations. To do this, we propose a cognitive programming model (CPM) that is based on studies in the psychology of programming introduced in the 1970s, as well as on studies in a cognitive model for learning.

Proceedings of the Third IEEE International Conference on Cognitive Informatics (ICCI’04) 0-7695-2190-8/04 $20.00 © 2004 IEEE

Figure 1. A cognitive programming model (CPM): required process areas for information processing agent to write a program. We view a human programmer as an informationprocessing agent for solving programmable problems. Such a simplification allows us to ignore complicated process areas such as the unconscious and the subconscious minds. The approach intends to generalize the model applicable to both machine and human programming. Our framework differs from previous cognitive work in which human activities are modeled according to the ways of thinking or the types of knowledge possessed. In contrast, our framework, shown in Figure 1, is constructed according to the different process areas that are necessary for programming. The paper is organized as follows. Section 2 reviews the literature on pair programming. Section 3 proposes a cognitive model of programming. In Section 4, the model is used to explain our experiment illustrating the behavior of novice and experienced programmers, and the behavior from learning programming to mastering programming. Section 5 describes our psychological experiment and provides an explanation for the different experimental results obtained by Nosek, Williams and Nawrocki. We discuss how people can predict in what situations programmers collaborating in pairs on the same tasks are better. The final section summarizes our work and offers our conclusions in Cognitive Informatic.

2. Background The three controlled experiments reviewed in this section have been widely referred to in pair programming papers on education, personality, and distributed pair programming [6, 7, 8]. Critics of pair programming also refer to these three experiments [9, 10]. The three experiments are set up in similar ways: subjects are randomly divided into a pair and solo groups and are asked to write the same program. The productivity of each group is then assessed against two variables: (1) time and (2) software quality. Despite these similarities, the results as presented in terms of time contradicted each other. These discrepancies have not yet been explained in other papers.

2.1. Nosek’s Experiment In 1998, Nosek [3] reported his empirical experiment in which 15 full-time system programmers in 5 pairs and 5 singles were asked to write a UNIX

script that performed a database consistency check (DBCC) in a Sybase database. Executing the DBCC command returns the status of the database in a log file. Thus, the subjects were asked to write a program that would initiate the command, detect any error in the log and post a warning e-mail if it found any. For system programmers, these tasks should be straightforward. The only part in which they were not experienced was DBCC. The results of this experiment are shown in Table 1. Time was objectively measured. As for software quality, two independent graders evaluated the readability and functionality of the problem solutions and assigned a readability score ranging between 0 to 2: 0 for an entirely unreadable solution, 2 for an entirely readable solution, and 1 otherwise. Functionality ranged from 0 for a solution not achieving the goal at all, to 6 for achieving the goal entirely. Although the two graders examined the problem solutions with intergrader reliability of 90% [3], human judgment was involved in verifying the software quality. Singles Pairs Time (minutes) 42.60 30.20 Readability Score ( 0 – 2 ) 1.40 2.00 Functionality Score (0 – 6) 4.20 5.60 Table 1: Pairs took 41.7% more time than individuals. ((30.2 × 2) – 42.6) / 42.6 × 100% = 41.7 % [3]

2.2. Williams’s Experiment In 2000, Williams reported the first university experiment on pair programming. Forty-one junior and senior university students were randomly assigned in 14 pairs and in 13 singles. The student subjects were taught Active Server Page (ASP) and were asked to write web scripts that had dynamic content to retrieve and update a Microsoft Access database. The applications were similar to those of a typical ecommerce web site [4, 11]. The students completed four assignments over a period of six weeks. The pairs passed more of the automated post-development test cases. The students recorded the time they spent on the project in a webbased tool. A teaching assistant was responsible for executing automated testing to analyze programming quality. In the first assignment, the pairs took 60% more time than singles. After the initial adjustment period, the total programmer hours the pairs spent on the second and the third assignments decreased dramatically — 15% longer than individuals on average. Because of data entry problems in the


experiment, the completion times for the fourth assignment were not accurate. Thus, the time performance of the fourth assignment was not reported. Regarding quality, the percentage of pairs passing the test cases was 86.4~94.4% whereas the singles passed 70.4~78.1%, shown in Table 2. Prog 1 Prog 2 Prog 3 Prog 4 Singles 73.4% 78.1% 70.4% 78.1% Pairs 86.4% 88.6% 87.1% 94.4% Table 2. Percentage of Test Cases Passed [4]

2.3. Nawrocki’s Experiment In 2001, Nawrocki [5] reported 21 fourth-year students were divided into three groups, each of which used different methodologies to work on the same assignments: (i) 5 pairs adopting XP (say, pair XP), (ii) 5 singles adopting XP but without pair programming (say single XP), and (iii) 5 singles adopting Personal Software Process (PSP). Nawrocki was primarily interested in two comparisons: (i) single XP vs. PSP, and (ii) single XP vs. pair XP. In this paper, we are interested in the comparison of pairs and singles that use the same method. Thus we select single XP vs. pair XP for our review (for the sake of brevity, in this paper we shall refer to them as pairs and singles). All the subjects formally studied C and C++ for more than two years. They were asked to write four programs. These programs were about (i) finding the mean and standard deviation of samples of numerical data, (ii) finding the linear regression parameters, (iii) counting the number of lines in a program and (iv) counting the total program LOC. Based on their results [5], the singles and pairs took around the same time to complete the first three assignments: 2.4, 1.3 and 2.4 hours, respectively. This means that in total the pairs took 100% more time than individuals did. In the fourth program, the pairs completed it in 3.5 hours and the singles in 4.3. This indicates that the pairs took 63% longer than the individuals, but Nawrocki remarked that the improvement resulted from the singles misunderstanding the program requirements. The number of re-submissions was counted as the subjects had to rework the program until errors were not discovered during acceptance testing. Prog 1 Prog 2 Prog 3 Prog 4 Singles 3.4 0.1 1.6 4.6 Pairs 3.6 0.1 1 3.3 Table 3. Number of re-submission on average [5]

In Table 3, the number of re-submissions shows that the amount of rework for pairs is slightly smaller than for the singles. Nawrocki concluded that pair programming appeared less efficient than had been reported by Nosek and Williams. From a recent on-line list of pair programming publications maintained by Hanks [12], Nawrocki’s work has been the only controlled experiment that shows pair programming unproductive.

2.4. Summary Table 4 summarizes the experiments that will be of interest to us in the later sections.

Subject Pairs take y% more time than singles Conclusion

Nosek

Williams

Nawrocki

Full-time Programmer

Student

Student

y= 41.7

y = 15

y = 100

PP is productive

PP is productive

PP is unproductive

Table 4. A summary of the past experiments.

3. Cognitive Programming Model A cognitive programming model (CPM) is a framework that describes how people or intelligent agents transform a problem from natural language into a symbolic, executable program. Rather than looking at any particular tasks, the model focuses on key process areas that must be involved in programming activities. As programmers who do well at one programming task might not be equally good at other programming tasks, we seek to identify different areas that are key to performance and programming productivity. In the past, there were two perspectives on programming. From a task point of view, programmers progress on three fronts when they are involved in programming [13]. These fronts are: (i) the problem specification that defines the problem to be solved, (ii) the problem representation that models the problem, and (iii) the code that implements the problem representation using a computer language. From a psychological point of view, distinctions must be made between the surface structure of a program, its deep structure, and its execution by the machine [14]. Taking these factors into account, we propose a Cognitive Programming Model (CPM), which is composed of six key process areas: (i) Definition, (ii) Representation, (iii) Model, (iv) Schema, (v) Algorithm and (vi) Code.


These process areas are coordinated in stages, meaning that the processes in the higher-level process areas are built on the ones below. In Definition, we are dealing with elementary terms that are required to perceive problems. In Representation, we are more interested in states of variables and functions that can be used to represent different states of a problem. In Model, unknown states of variables and unstated assumptions are explored to better model the problem. For example, the problem might be given as “Polly got an ice-cream and Polly got sick.” To make sense of it, we make the unstated assumption: Polly ate the ice cream. In Schema, we formulate the sequence of high-level tasks (i.e. casual reasoning) for a solution to the problem. In Algorithm, details in relation to symbolic logic are worked out. Finally, in Code, the algorithm is converted into executable code in a specific computer language. In formal methods, we are taught to progressively break down a problem into a series of steps, and to accomplish it in a disciplined manner. Therefore, we have to complete one area before moving up to the next. When CPM strictly follows this procedure, it describes an approach similar to formal methods. This approach to problem-solving, known as machine functionalism [15], is the case for non-human intelligent systems such as Turing Machine, and the Doris system [16]. Nevertheless, the human mind does not appear to work in that way. The mind working in a networking manner moves upward to the next process area and/or downward to the previous one. The downward and upward movements provide a feedback mechanism. It should be noted that, as a result of failing memory and external stimulation in humans, the mind unconsciously surmises (missing) information. This is the significant distinction between human programmers and machine programmers. Thus a strict staged progress will actually lower the productivity of the human mind. The following describes each process area in detail.

3.1. Definition When user requirements are given or a problem is stated, the first thing we do is to perceptually recognize basic elements of the problem. These elements are used to understand the problem. For example, to writing a program to solve a Tower of Hanoi puzzle, disks and pegs are the basic elements. In order to understand what a Tower of Hanoi is, programmers need to perceive some basic terms used to define and describe the problem on the surface level (see Figure 2). Without those elements, we could not build a mental representation of the problem.

Figure 2. A set of descriptions that are defined/ understood/assumed by the mind so that a mental representation of a problem is not blank.

3.2. Representation Once the problem becomes comprehensible to us, we start exploring variables, functions, states and all their inter-relationships as provided by the problem. They can be used to represent and hence formalize the problem. For example, programmers are asked to develop a system that calculates a 10% price discount. In Definition, the mind needs to understand some basic terms such as that price cannot be negative or be more than two decimal places. In Representation, we might test some prices {100, 200, 300, 400} and the corresponding amounts {90, 180, 270, 360}. In fact, we often use this technique to make ourselves understood and to confirm our understanding, irrespective of whether the nature of the problems is well -- or ill--defined.

Figure 3. A problem can better be understood by identification of its known states. If there are only four prices to be dealt with, a finite set of state variables {(90,100), (180,200), (270,300), (360,400)} may represent the whole problem. To represent a problem in general, a generic expression can be used, e.g. f(x)=0.9x. Unfortunately, many real problems cannot be fully expressed in a set of formulas.

3.3. Model A representation merely describes the states of variables and functions that are directly given by a problem. Unfortunately, many assumptions or events are not fully addressed in the problem. To completely model the behavior of a system that solves the problem, we need to inductively reason or deductively infer


unknown states given in a problem, as depicted in Figure 4. Whatever ways we deduce or induce (or even assume) known states (or facts) must be recognized. Thus, Model is based on Representation.

Often, the logic is expressed in mathematical symbols, pseudo-code or generic logic.

Figure 6. Algorithm – the deep structure of a program. Figure 4. Model – Induction and Deduction. For example, a fast-food manger addresses a promotion idea: buying ten hamburgers and getting one free. In Representation, it is clear that a customer will get twenty-two burgers if he or she buys twenty in a single transaction. However, in Model, we always explore some situations in which we should allow the customer to buy five and another five in two transactions but within a very short time, say ten minutes.

3.4. Schema The schema is the surface structure that corresponds to a textual structure of a program, in other words, the surface units, explicit in the program, and the way in which they are arranged [14], which is causal reasoning. For example, the linear structure of a program is well known by its textual version. The structure addresses starting and end points, repeatable blocks, a block sequence and key checkpoints, as depicted in Figure 5.

Figure 5 Schema – the surface structure of a program.

3.5. Algorithm The next level of the surface structure is deep structure. The algorithm is the deep structure that corresponds to the representation of relations that succinctly states the logic and flow of control [14].

3.6. Code The algorithm must eventually be expressed in a computer language for execution. In the semiotic sense, this process area is about the transformation of syntax from an algorithm, without changing its semantics. However, from a compiler perspective, a program with correct syntax might not be executable due to constraints of hardware performance, e.g. it is out of memory. Therefore, code is very different from algorithms. It should be noted that our mind has to go through each process area regardless of whether we write a program as a way of documenting outputs of the process areas, or in a less disciplined way.

3.7. The CPM Analysis Table 5 summarizes each process area. Process Area Definition

Descriptions Processes that deal with atomic elements Representation Processes that deal with the representation of given events Model Processes that work out unstated assumptions and deduce possible events, all of which are not given in the problem, and processes that model an individual event or a set of small events Schema Processes that deal with causal reasoning among events Algorithm Processes that deal with symbolic logic and sequence Code Processes that convert symbolic logic into an executable program based on a set of language rules Table 5. A summary of process areas in the CPM.


We may use the CPM to analyze how an information-processing agent (or a person) solves a given problem. The degree of difficulty for a person to program a specific problem varies in workload and effort in the six process areas. For example, a professional COBOL programmer is asked to write a program in COBOL to calculate a possible stock market trend using a momentum method. If the programmer does not even know what the stock price and stock index are, he or she would have to expend a lot of effort to understand the problem. In short, we might measure a problem for a person by degree of familiarity and knowledge in each process area. For brevity, we may define a scale of five: 1 – 5, for different degrees of familiarity. Therefore, the COBOL programmer needs to exert effort in Definition, Representation, etc. However, once the programmer works out the generic algorithm in his or her mind, he or she can quickly finish the program. D R M S A C 1 1 1 1 1 5 Table 6. The CPM analysis: A COBOL programmer who does not know the stock market is asked to write a stock trend system using a momentum method in COBOL. Note: D denotes Definition; R, Representation; M, Model; S, Schema; A, Algorithm, C, Code.

4. Repeated Programming The initial idea of understanding programming productivity by novice and expert was first reported by us in [17]. When a programmer wrote the same program eight times, there was a significant time difference between the first and the eighth times of writing. The report however does not address the implications of the finding. Most importantly, it does not exploit a case such as that in which a pair of developers write the same program several times. This section will report the details of the experiment. Over 2002–03, we conducted an experiment to examine how programming performance varied when measured along an axis in which developers become more familiar with a programming problem. We asked each subject to write the same program eight times, each time starting from the beginning. Hence this experiment was called “Repeated Programming.” From among 63 candidates we selected three subjects whose capabilities were the most nearly identical. We did this by first testing them on several programming exercises on Transact-SQL and JSP. We

chose individuals with the same programming knowledge: they wrote programs at the same speed and quality. Whether they were good or bad was not an issue. From that group, we looked for three individuals with little knowledge of inventory operations. By choosing “close-to-identical” subjects, we had minimized a disparity in programming ability that, between the best and the worst, has been shown to vary by as much as a factor of 27.7 [18]! The subjects, the individual and the pair, were asked to write a FIFO warehouse application in our laboratory. The tasks were standard—they had to create tables in SQL 2000 and code in JSP.

Figure 7. Repeated programming: a pair vs. a single Figure 7 illustrates how long the individual and the pair took to write the same program on each occasion. The two curves are similar. The trend is far more noteworthy and meaningful than the values indicated by the curves. Programmers with different abilities can produce different sets of results. However, the trend of the curve remains consistent. The characteristic is conservative and is independent of whether they are fast or slow coders, or talented or weak programmers. Nor does it make much difference whether a value is 5.81 days or 6.81 days because what we are interested in is the shape of the trend and the holistic pattern, the look of the curves and how they cross. CPM can be used to understand the described behavior of a programmer put through “Repeated programming” tasks. We observed that more time is required when a new problem is first given to the subjects. After the first attempt, a problem-solution pattern will be constructed in the mind of the programmer. When given similar problems, this pattern will be retrieved and re-used. Time reduction of problem modeling should be quite significant for processes in Definition, Representation, and Model. In addition, we observed that the algorithms that were


used the first three times in repeated programming were quite different. Afterwards, the algorithms remained unchanged. The three subjects who participated in repeated programming and the other three programmers were invited to assess the first round and the eighth round in repeated programming using CPM. Repeated Programming and CPM were explained to them and they graded the level of familiarity in each process area. The three subjects individually evaluated themselves. The three invited programmers could then ask the three subjects some technical questions for assessment before the invited programmers did the grading. In short, the assessment combined selfevaluation and peer evaluation. The result is shown in Figure 8. Note that the first letter of each process area (i.e. D for Definition) is shown in Figure 8 and level of familiarity ranges from 1 to 5.

distributed to the subjects. In addition, we explained CPM to ensure they fully understood the model. Next, the subjects were asked to read the reports of Williams’ [4, 10] and of Nawrocki [5]. Note that the results and conclusions in the reports were taken out so that the subjects were able to learn about only the descriptions of the experiments. The subjects were asked to use CPM to analyze the arrangements of the three experiments. The result is shown below.

Figure 9. Three psychological experiments

Figure 8. Analyzing repeated programming using CPM. Figure 7 and Figure 8 illustrate that in pair programming the lower the level of familiarity, the higher the productivity.

5. Understanding Pair Programming Experiments Using CPM Three researchers reported different results on pair programming. This implies that pair programming can only be productive in some situations; otherwise, there should not be substantial deviations in their results. An experiment was conducted at the Hong Kong Polytechnic University in 2004. For this experiment, we invited twenty-two full-time programmers from different companies. They had on average 2.147 years of programming experience. Ten of them had some previous pair programming experience. First, they were asked to carefully read Nosek’s full paper [3]. Copies of Section 3 of this paper were then

Based on their CPM analysis, they were then asked to estimate the productivity in Williams’s and Nawrocki’s experiments. They estimated 20% ~ 40% for Williams’s experiment and 70 % ~ 90% for Nawrocki’s experiment. The estimations are quite close to the actual results, 15% for Williams and 100% for Nawrocki (see Section 2 in this paper). CPM explains the different results of the three major pair programming assessments. Moreover, we might use CPM to predict pair programming productivity in solo programming. Finally, although the CPM experiment provides some results that explain the discrepancies between the three experiments, it is likely that other factors are also involved. For example, one factor is pair jelling [4], which is the time taken for a conditioned solo programmer to learn how to be a pair programmer. In fact, pair jelling is similar to Tuckman’s well-known theory [19] in the field of group dynamics: all groups (in the case of pair programming, the group has two members) pass through several stages during their development: forming, storming, norming, and performing, as shown in Figure 9. Obviously, the productivity for a pair group will vary at different stages. Productivity discrepancies are expected according to Tuckman’s theory.


[4] Williams, L., Kessler, R., Cunningham, W. and Jeffries, R., “Strengthening the Case for Pair Programming”, IEEE Software, Vol. 17, No 4, 2000, pp. 19–25. [5] Nawrocki, J. and Wojciechowski, A., “Experimental Evaluation of Pair Programming”, Proceedings of the 12th European Software Control and Metrics Conference, London, April, 2001, pp. 269–276. Figure 10. Tuckman's (1965) forming, storming, norming, performing model of group development

[6] McDowell, C., Werner, L., Bullock, H. and Fernald, J., “The Impact of Pair Programming on Student Performance, Perception, and Persistence”, Proceedings of the 25th International Conference on Software Engineering, USA, May, 2003, pp. 602–607.

6. Future Work Repeated Programming, introduced in Section 4, uses three “close-to-identical” subjects selected from among 63 candidates to form one individual programmer and one pair. Obviously, more “close-toidentical” groups and results are needed to support the findings of “Repeated Programming” as shown in Figure 7 in Section 4.

7. Conclusions In this paper, we present a cognitive model that explains the differences in statistics collected in pair programming studies. Until now, none of the work had addressed how those results and experiments were related to each other. CPM provides a theoretical framework for the explanation of three programming assessments. Our work also reaches the conclusion that pair programming helps one understand an unfamiliar problem and new requirements. The primary contribution of the work, however, is that it provides an innovative approach to analyzing empirical software engineering experiments using a cognitive model.

8. References [1] Constantine, L.L., Constantine on Peopleware, Yourdon Press, Englewood Cliffs NJ. 1995. [2] Beck, K., eXtreme Programming Explained: Embrace Change, Addison-Wesley, Boston MA, 2000. [3] Nosek, J.T., “The Cast for Collaborative Programming”, Communications of the ACM, Vol. 41, No. 3, 1998, pp. 105–108.

[7] P. Baheti, E. Gehringer, and D. Stotts, “Exploring the Efficacy of Distributed Pair Programming”, Proceedings of Extreme Programming and Agile Methods — XP/Agile Universe, USA, August, 2002, pp. 208–220. [8] Heiberg, S., Puus, U., Salumaa, P. and Seeba, A. “Pair Programming Effect on Developers Productivity”, Proceedings of Extreme Programming and Agile Processes in Software Engineering, Italy, May, 2003, pp. 215–224. [9] Keefer, G., “Extreme Programming Considered Harmful for Reliable Software”, Proceedings of the 6th Conference on Quality Engineering in Software Technology, Germany, December, 2002, pp. 129–141. [10] Stephens, M. and Rosenberg, D., Extreme Programming Refactored: the Case against XP, Apress, Berkeley Calif., 2003. [11] Williams, L., The Collaborative Software Process, Ph.D. Dissertation, University of Utah, 2000. [12] B. Hanks, “Pair Programming Bibliography”, available on-line at http://www.cse.ucsc.edu/~brianh/PairProgramBib.html [13] Mattson, T., “A Cognitive Model for Programming”, available on-line at http://www.cise.ufl.edu/research/ParallelPatterns/Patter nLanguage/Background/Psychology/CognitiveModel.h tm


[14] Détienne, F., Software Design — Cognitive Aspects, Translated and Edited by F. Bott, Springer, London, 2001. [15] Putnam, H., “The nature of mental states”, in The Nature of Mind, edited by Posenthal, 1967, pp. 197– 203. [16] Bradshaw, J.A. and Young, R.M., “Evaluating Design Using Knowledge of Purpose and Knowledge of Structure”, IEEE Intelligent System, Vol. 6, No. 2, 1991, pp. 33–40. [17] Lui, K.M. and Chan, K.C.C., “When does a pair outperform two individuals”, Proceedings of the Fifth International Conference on Extreme Programming and Agile Processes in Software Engineering, Italy, May, 2003, pp. 225–233. [18] Humphrey, W. S., Managing Technical People, Addison-Wesley, Reading Mass, 1997. [19] Tuckman, B.W., “Developmental sequences in small groups”, Psychological Bulletin, Vol. 63, 1965, pp.384-399.
