A Case Library for Teaching Usability Engineering ... - CiteSeerX

A Case Library for Teaching Usability Engineering: Design Rationale, Development, and Classroom Experience JOHN M. CARROLL AND MARY BETH ROSSON The Pennsylvania State University, University Park __________________________________________________________________________________________ Case studies of professional software development practices describe how real (or realistic) projects are planned and executed. Cases provide engaging models of the activities and materials of software development to students and other novice practitioners. They vividly remind learners of the possibilities for meaningfully applying knowledge and skills in the world beyond the classroom. During the past six years, we have developed and used a collection of usability engineering case studies for teaching human-computer interaction, primarily to upper-level undergraduates in computer science and in information sciences and technology. In this article we describe the rationale for this approach, the structural schema and browser that we developed for case studies, the case-based activities we employ in courses, and the experiences of instructors and students who have used the cases. Categories and Subject Descriptors: H.5 [Information Systems]: Information Interfaces and Presentation; K.3.2 [Computers and Education]: Computer and Information Science Education General Terms: Documentation, Human Factors, Management, Measurement Additional Key Words and Phrases: Human-computer interaction, case-based learning, authentic learning, usability engineering __________________________________________________________________________________________

INTRODUCTION One thrust of contemporary educational reform is to anchor classroom learning in authentic situations and issues, that is, situations and issues that evoke, describe, or comprise real-life experiences. Case studies are descriptions of a specific activity, event, or problem, drawn from the real world of professional practice. They provide narrative models of real life to students and other novice practitioners. Cases incorporate vivid background information and personal perspectives to elicit empathy and commitment, and present contingencies, complexities, and often dilemmas intended to evoke integrative analysis and critical thinking. Cases engage the student in the drama of a real situation. Case studies are widely used in professional education — in business, medicine, law, and engineering [Williams 1992]. For example, the well-known Harvard Business School case collection includes over 7500 case studies of business decision-making [Garvin 2003]. Perhaps coinciding with contemporary recognition that all disciplines incorporate practice (and not merely knowledge), or perhaps just reflecting contemporary pedagogical concerns with active learning and critical thinking, cases have, through the past decade, become pervasive. For example, the NSF-supported National Center for __________________________________________________________________________________________ This research was supported by the US National Science Foundation, under two Course, Curriculum and Laboratory Improvement (CCLI) awards, DUE-0088396 and DUE-0231111/0354195. Authors' address: School of Information Sciences and Technology, Center for Human-Computer Interaction, IST Building, The Pennsylvania State University, University Park, PA 16802 USA. Permission to make digital/hard copy of part of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication, and its date of appear, and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Permission may be requested from the Publications Dept., ACM, Inc., 1515 Broadway, New York, NY 10036, USA, fax:+1(212) 869-0481, [email protected] © 2005 ACM 1531-4278/05/0300-ART3 $5.00 ACM Journal on Educational Resources in Computing, Vol. 5, No. 1, March 2005. Article 3.

2

•

J.M. Carroll and M.B. Rosson

Case Study Teaching in Science includes many case studies in medicine and engineering, but also environmental science, anthropology, botany, social and cognitive psychology, geology and geography, pharmacy and nutrition, and experimental design [Herreid and Schiller 2005]. Indeed, the query "case studies" in Google returns over 56 million urls; many of the first hundred or so are case study portals and Web pages about case-based learning. Prior investigations of case-based learning in the computer and information science and engineering (CISE) disciplines have been quite encouraging, primarily in the arena of professionalism and computer ethics.1 Case-based approaches have also been developed and explored in more technical CISE topic areas such as computer graphics [Shabo et al. 1996]; design [Guzdial et al. 1996]; ubiquitous computing [McCrickard and Chewar 2004]; and usability engineering [Rosson et al. 2004a; 2004b]. This research has also provided evidence that case-based learning promotes key meta-cognitive skills, including cognitive elaboration, error management, reflection, self-regulation, and transfer of knowledge [Carroll and Rosson 2005; Kolodner et al. 2004]. In this article we describe a project in which we developed a small collection of usability-engineering case studies, defined a hypertext model for the cases, and implemented a simple browser to support viewing and editing of the cases. First, we describe and motivate the structural schema we developed for these cases. Next, we describe and illustrate the browser we developed for accessing and maintaining the case collection. The cases and the browser tools are freely available through the World-Wide Web (http://ucs.ist.psu.edu), and can be used with or without an accompanying textbook [Rosson and Carroll 2002]. We next describe the case-based learning activities we have developed to employ the cases in our teaching. Then, we describe classroom experiences with the cases from the perspective of instructors, ourselves and a dozen other colleagues who have used the cases in their own teaching. Finally, we describe classroom experiences with the cases from the perspective of students we taught in two offerings of the usability-engineering course in 2004 and 2005.2 We close with discussion of future directions for this project, including enhanced support for collaborative case-based learning activities. THE STRUCTURE OF USABILITY CASE STUDIES The paradigmatic case study is a brief but provocative story, sketching a problematic situation and inviting the reader to elaborate missing or open details of the premise and to construct possible resolutions. Some of the most impressive case studies are extremely brief, comprising just a half-page of text. Cases can also be quite voluminous, involving 50 or more pages of material and requiring detailed study and analysis to characterize the problematic situation and its possible resolutions. Both brief and lengthy cases can be "good" cases, that is, both can engage learners in authentic learning. More detailed and lengthy cases evoke discussions that are more technically articulated and require more time. Stories are privileged cognitive structures. The human mind seems designed to convey, extract, and preserve meaning through narrative structures, as illustrated by dreams [Freud 1900] and myths [Levi-Strauss 1967]. And, indeed, communities of practice self-regulate through the exchange of stories: new members learn an organization's stories and established members share and consolidate lessons through stories [Orr 1996; Wenger 1998]. Thus, students who learn through analysis and discussion of case studies benefit both from authentic learning and from learning authentically. ACM Journal on Educational Resources in Computing, Vol. 5, No. 1, March 2005.

A Case Library for Teaching Usability Engineering

•

3

Because we are most interested in teaching and learning issues for science and engineering education, we have studied examples at the National Center for Case Study Teaching in Science, sponsored by the National Science Foundation, and by the Center for Case Studies in Engineering, sponsored by the American Society for Engineering Education. These two online case study repositories are quite popular and are a fair representation of the state of the art. They are among the top five sites returned by Google for the query "case studies." One of the anatomy and physiology cases from the National Center for Case Study Teaching in Science describes the story of a doctor examining a young child with a chronic cough and diarrhea (http://www.sciencecases.org/cf/cf.asp). The story describes the interaction with the parents as the clinician presents a diagnosis of cystic fibrosis and explains what this means for their child. This is accomplished in a mere 735 words. Immediately after the story, there is a list of seven questions that students are encouraged to answer as if they were clinicians interacting with the parents. Answering the questions requires going beyond the information presented in the case study; students working on this case use the Internet and physiology textbooks to develop their answers. The cases of The Center for Case Studies in Engineering are two to four times as lengthy as the National Center for Case Study Teaching in Science cases. They often include equations, as well as engineering drawings and other figures (as many as eight to ten graphics). Some of the cases have only a brief narrative sketch of a situation ("Peter is designing a two-speed coiler gearbox …"), and then launch into a procedural account of how one would carry out that particular engineering task. Obviously, this reduces the drama and vividness of the case, but such cases still model how an engineer puts together mathematics, science, and instrumentation skills to carry out an engineering job. Five years ago, with the support of an NSF curriculum development award, we developed a small collection of usability-engineering case studies. Usability-engineering is made up of the concepts, processes, and practices engaged in the development process to ensure that the software system and applications serve their intended users effectively. To a considerable extent, it mirrors and complements software engineering. Our approach to teaching these concepts and skills is called scenario-based usability engineering [Carroll 2000; Rosson and Carroll 2002]. We take scenarios of human-computer interaction as a primary context for analyzing requirements, describing specifications, envisioning designs, developing rationales, creating various sorts of prototypes, and evaluating systems and applications. Some of the core concepts in our usability-engineering course are scenarios, design trade-offs (couched as claims analysis), user interface metaphors, mental models, collaborative coordination and awareness, ubiquitous computing, graphical design, direct manipulation, Fitts' law, information architectures, and information visualization. Students learn techniques associated with the various phases in the usability-engineering process, for example, requirements-interviewing and ethnographic observation, task analysis and user-modeling, paper-based and scenario-machine prototyping, evolutionary development, and usability evaluation methods like experimental analysis of user performance, collection and interpretation of thinking-aloud protocols and field study data, and survey construction and analysis. Our case studies describe real system development projects; for example, our garden.com case study describes the development of a Web portal for gardening supplies. It is an authentic story — from a usability-engineering perspective — of one of the "dotcoms" from the late 1990s. Stories of real system development processes are neither brief nor linear. The stories can be organized by an overall timeline, but the logical ACM Journal on Educational Resources in Computing, Vol. 5, No. 1, March 2005.

4

•


dependencies among events, activities, and problems that occur in the cases are quite interconnected, and related subactivities often occur in parallel. System development case studies can also be quite voluminous. Thus, a system development project typically involves a requirements phase in which stakeholders representing those who will design, implement, buy, sell, maintain, train, and use the system are interviewed, and previous and related systems are analyzed with respect to stakeholder perceptions and/or performance. The requirements analysis activities alone can generate a huge amount of content, yet its outcome is merely a problem statement that guides subsequent system development activities. Of course, case studies can be constructed to condense or omit details. Indeed, some incompleteness is useful in cases because it invites and requires elaboration by learners. But if an instructor is using case studies to present models of professional materials and practices to students, too much abbreviation may be problematic. As an old saying goes, "design is in the details". Students cannot learn usability-engineering practices unless they can authentically experience those practices. This contrasts sharply with many other senses of "case," in which the term refers to a brief narrative document that sketches a problematic context to set the stage for an open-ended discussion. We conceived of and developed our case “library” as an approach to better leverage case-based teaching practices, which we suspected were typical in systems courses such as usability engineering. We later confirmed this suspicion about the use of cases through a Webbased survey study of computer science instructors [Rosson and Carroll 2005]. We decided to explore a hypertextual structure for the usability-engineering case studies. Rather than condensing our case materials into a linear narrative, we developed a simple information design for a collection of documents. We developed a case schema corresponding to standard phases in the system development process: (1) requirements analysis, (2) activity design, (3) information design, (4) interaction design, (5) documentation, and (6) usability testing. Each of the phases is further decomposed into a set of focal activities: requirements analysis and usability testing are decomposed into (a) planning, (b) methods and materials, (c) information gathering, and (d) synthesis; the four design phases are decomposed into (a) exploration, (b) envisionment, and (c) rationale. This results in 20 terminal categories of development activity. The terminal categories are populated with design scenarios and mock-ups, as well as a variety of other design documents. The documents include early statements of the design concept; results from requirements surveys; focus groups, and other market research (including, where possible, instrumental documents such as advertisements used to recruit participants into focus groups); notes from design brainstorming sessions and other design discussions; analyses of metaphors and technology options; user interaction design scenarios; user interface sketches, mock-ups and prototypes; use cases and pointof-view scenarios to analyze software designs; design issues and tradeoffs; usability specifications; descriptions and results from usability studies, including test materials; and design revisions and justifications for changes. A description of the content for the garden.com case study is in Table I. Across the collection of case studies we have developed and investigated, the case study schema appears to be general and effective with respect to its use by case developers and end-users (students and teachers; see sections below). Our case structures are, by design, articulated at a fine level of technical detail. Our largest case incorporates 82 design documents at the terminal category level (this does not count 26 intermediate branching nodes in the hypertext and 82 container objects for the primary design artifacts). There is also substantial interlinking among the nodes in this hypertext. For ACM Journal on Educational Resources in Computing, Vol. 5, No. 1, March 2005.


•

5

Table I. Content for the Garden.com Case Study Requirements Analysis Planning: initial brainstorming session; market research; root concept. Methods and Materials: script for interviewing nursery workers; script for interviewing customers; focus group agenda; focus group recruiting ad; screening survey questions. Information Gathering: photo of highway entrance sign; photo of nursery exterior; photo of nursery interior; photo of nursery main retail display; photo of outside covered retail display; photo of outside open retail display; photo of superstore outdoor display; photo of indoor superstore display; photo of gardening customer; garden sketch; plant hardiness map; catalog index; catalog order form; interview excerpt-nursery workers; interview excerpt-customers. Synthesis: problem claims; problem scenarios. Activity Design Exploration: summary of proposed activities; conceptual metaphors; system technology. Envisionment: activity scenarios; shopping model; sequence analysis; point-of-view analysis. Rationale: activity claims. Information Design Exploration: department information requirements; product page information requirements; summary of proposed information offerings; information metaphors; information technology options. Envisionment: sketch of map metaphor; sketch of catalog metaphor; sketch of homepage; sketch of search results; sketch of product page; information scenarios. Rationale: information claims. Interaction Design Exploration: interaction metaphors; interaction technology. Envisionment: online shopping script; functional study; design revision #1; design revision #2. Rationale: before/after main page; before/after department page; before/after product list page; before/after product information page; before/after wheelbarrow page; usability study #2 report. Usability Testing Planning: operational definition; usability specifications. Methods and Materials: user background survey; gardening quiz; attitudes toward the Internet survey; session script; usability testing scenarios; post-scenario ease of use assessment; usability report card; strategy questions; debriefing questions; equipment list Data Collection: notes on sessions; notes on participants; post-subtask comments; postsession comments; usability report card scores. Interpretation: gardening quiz scores; attitudes toward the Internet; time on task; lostness ratings; number of mouse clicks; post-subtask ratings. Synthesis: design recommendations.

example, many of the design documents are scenarios articulated at various levels of design detail, with links to other documents presenting sketches and screen-shots of prototypes implementing the scenarios, claims analysis (design rationale) of the scenarios, and other content. THE USABILITY CASE STUDIES BROWSER Our case study structures are large enough to involve significant navigation issues, even if one is only studying a single case at a time. In practice, we have found it pedagogically effective to also assign case activities involving comparisons across case studies. Thus, it is important to consider usage scenarios in which students are analyzing and navigating ACM Journal on Educational Resources in Computing, Vol. 5, No. 1, March 2005.

6

•


two or even three case study hypertextual structures at one time. Display and navigation of case study information was a major issue in our project. We designed the usability case-study browser to present the case structures in a fairly direct manner. Figure 1 displays a screen shot of a session in which the user is browsing the garden.com case study. Notice in the menu pane at the left that the "Case Studies" tab has been selected, meaning the user is accessing design documents organized by case study. The "garden.com" case study has been selected. Within that case study, "Information Design" has been selected, and within Information Design, "Envisionment" documents have been selected. The Envisionment documents are listed in the pane to the right of the menu pane. From among the Envisionment documents, one named "Hand Sketch based on Map Metaphor" has been selected, opening a container object on top of the case-study browser window. The container object provides a description of the design document it contains and a link to open that document. The primary design document, in this case a hand-sketch based on the map metaphor for the garden-com user interface, is displayed as the top window at the bottom right of Figure 1.

Fig. 1. The usability case-study browser: The map metaphor document in the lower right is drawn from the actual materials of the product development process. It is accessed through the container (above and to the left), providing mostly noncoded metadata. The container is accessed through a list of envisionment documents from the information design phase of the development process.

ACM Journal on Educational Resources in Computing, Vol. 5, No. 1, March 2005.


•

7

Users can also access design documents through a string search dialog (notice the "Search" tab in the lower left corner of Figure 1). We also provide a "Documents" tab (see the lower left corner of Figure 1) through which users can access design documents organized by type of document aggregated across case studies (e.g., they can see examples of problem scenarios from all of the case studies). CASE-BASED LEARNING ACTIVITIES A distinction can be drawn between learning activities that emphasize enactment of the types of activity that system developers carry out, and those that emphasize the content of system development concerns. An example of authentic enactment, but somewhat lessthan-authentic content, is a classroom role-play activity in which students act as clients, developers, users, marketing managers, and development managers, exploring interactions related to requirements identification, but using a "toy" situation (e.g., the widget++ from the Magic Whatnot Company). An example of authentic content, but lessthan-authentic enactment is an assignment in which students study and are tested on the functional specification for Linux FailSafe. These two kinds of learning activities might be combined in industrial co-operative education and internships or service learning. However, it is a formidable challenge to combine the two in activities that are carried out in a university education context, Table II. Six Types of Case-Based Learning Activity Modeling: Use a case study, or a part of a case study, as a paradigm for developing another instance. For example, study the problem scenarios of the garden.com case study to guide the development of a further problem scenario for that case (one that is consistent with the requirements data, stakeholder analyses, and so on). Perturbation: Use an existing case study, or part of a case study, to develop another under certain specific changed assumptions. For example, consider the new problem scenarios that might arise if the requirements for the m-Banking case study focused exclusively on international clients, and redevelop the balance of the case study under these changed assumptions. Analysis: Analyze an existing case study, or part of a case study, with respect to how particular usability engineering techniques were employed and with what consequences. For example, summarize how principles of graphical design were used in the user interface for garden.com. Tracing: Reconstruct a case study with respect downstream impacts of upstream phases or decisions. For example, identify a particular claims tradeoff in the PhoneWriter requirements analysis, and then trace through subsequent phrases of the usability engineering process to investigate whether and how the developers addressed the tradeoff, for example, whether it was addressed consistently throughout the balance of the development process. Compare and contrast: Contrast two separate case studies with respect to how they implemented particular phrases, or specific techniques within phrases, of the usability engineering process. For instance, contrast how field work was used in the Virtual Science Fair and in garden.com. Classification: Identify the most distinguishing concepts and techniques that were employed in a case study, and use them to classify the case study with respect to others. This is a blending of analysis and compare/contrast. For example, classify the primary prototyping strategy in the Tapped In case study as evolutionary development or rapid prototyping, and explain why.


8

•


particularly activities that take place within the classroom. Our objective in this project was to develop learning activities that were authentic in both the process the students enacted and in the content that they studied. We developed an analysis of case-based learning activities, as in Table II [Carroll and Rosson 2005; Rosson et al. 2004a]. The learning activities vary in the degree of cognitive effort they entail (we tried to assign the least demanding activities early in the course and to make more demanding assignments later on). This strategy can be seen more concretely in the six case-based homework assignments we used in the most recent version of the usability-engineering course (taught in Spring 2005; see Table III). For example, the first assignment used modeling, asking students to study the problem scenarios and claims developed in the requirements phase of the PhoneWriter case study, and to develop a scenario and at least one claims tradeoff of their (i.e., the students’) own. Table III. Summaries of Six Homework Assignments that Used the Case Studies from Spring 2005. (Slightly different variants of these were used in 2001-2004. Copies of the actual assignments can be downloaded from http://cscl.ist.psu.edu/public/projects/cases/activityIndex). Homework 1: Problem Scenarios and Claims. Review Requirements Analysis phase of the PhoneWriter case study as an introduction to the development of problem scenarios and claims: write (1) your own problem scenario, and (2) one example of a claim that the scenario implies. Indicate by underlining or bold-font emphasis the specific text in the scenario that is associated with the claim (like the linking in the UCS tool). Homework 2: Conceptual Metaphors in Activity Design Review the Requirements Analysis and Activity Design for the garden.com case study and identify at least two different metaphors. Explain how these metaphors helped to clarify the requirements, and to suggest starting points for activity design. Comment as to whether you agree with the garden.com designers and why. Homework 5: Interaction Design Describe two issues in the interaction design of the Tapped In case study: (1) explain what the issue is, (2) analyze the design tradeoffs pertaining to the issue, and (3) describe how the tradeoffs were resolved, or not resolved, in the design work. Finally, (4) give you own assessment of how satisfactorily (or unsatisfactorily) the designers addressed the issue and its tradeoffs in their design work. Homework 6: User Interface Design Select two claim tradeoffs from the garden.com case study, each consisting of at least one upside and at least one downside. Examine the requirements analysis to familiarize yourself with the basis for these claims. Then, study the design phases of the case study to determine whether and how the design team responded to these claims. Make a case for whether and how the design carries forward (maintaining/reinforcing) the positive consequences of the requirements claims, while addressing (removing/minimizing) the negative ones. Homework 7: Prototyping Contrast the use of prototyping during the information design and interaction design phases for two of the case studies in the UCS: the virtual science fair and Tapped In. Homework 10: Documentation Design Select two examples of how the minimalist model or the systems model was or was not effectively used in the m-Banking case study. ACM Journal on Educational Resources in Computing, Vol. 5, No. 1, March 2005.


•

9

The second homework is an analysis activity in which students identified conceptual metaphors used to clarify requirements in the garden.com project. Homework 5 is a tracing activity; students identified two interaction design issues from the “Tapped In” case study and traced how these issues were addressed by design work. Homework 6 is a more elaborate tracing activity, in which students identified issues from the garden.com requirements analysis and traced their impacts through the various phases of design. Homework 7 is a compare-and- contrast activity, asking students to contrast the prototyping techniques used in two of the case studies. Homework 10 is a classification activity: The students evaluate the documentation design in a case study with respect to how well it exemplifies two models for documentation design. In addition to the six homework assignments, we use the cases in three other ways in the usability-engineering course. First, we use case studies as background for in-class activities. For example, in one activity, groups of students assess the impact of a changed requirement for the garden.com case study (e.g., a business-to-business vision instead of direct retail). This is an example of a perturbation activity (Table II). Second, we use the case studies to exemplify principles, practices, concepts, and techniques that are described in lectures and other presentations. For example, one of the case studies is a mobile banking application that runs on a PDA, which provides a nice example for introducing user interface issues for devices that are mobile and have small displays. More generally, a case library available to all students provides a useful body of shared examples for elaborating principles throughout the semester. Third, the students use the case studies as models for their own semester projects. Each student group develops a small system project through the course of the semester, and, among other things, documents it as a case study of system development. THE INSTRUCTORS' EXPERIENCE We have used the case library and browser in teaching an undergraduate course in usability engineering at Virginia Tech (2001-2003) and at Penn State (2004-2005). In 2002, we published a textbook [Rosson and Carroll 2002] that includes one of the case studies (the Virtual Science Fair) and advertises the original url of the case browser. In April 2003, at the ACM CHI 2003 Conference, we organized a meeting and discussion for instructors who were using the online cases and/or our book. In June 2003, we organized a workshop at Virginia Tech for a similar group to discuss, in more detail, their experiences, interests, and needs with respect to case-based instruction in usability engineering. This interest group includes instructors from Drexel University, Johns Hopkins University, Pennsylvania State University, Portland State University, University of Calgary, University of Hawaii, University of Oregon, and Virginia Tech. Each of these meetings was attended by eight members from this interest group. Rosson originated the undergraduate usability-engineering course at Virginia Tech in 1995. The next year the course was required of all undergraduate computer science majors; it was normally taken in the third year. Our early experience with the course highlighted several issues: First, by their third year, the computer science majors already identified strongly with the role of software developer, and were often skeptical and sometimes hostile to the values and overall objectives of the course. Second, the few available textbooks tended to aggravate this problem by setting up a contrast between "humans" and "computers" as two foundations of human-computer interaction. Indeed, a typical organization for textbooks addressed humans and computers separately as the first two chapters (e.g., Dix et al. [1998]; Preece et al. [1994]; Shneiderman [1998]).


10

•


We decided to depart from this model and to treat usability engineering as a professional practice embedded in software engineering. Thus, our course began with the topic of requirements gathering and analysis, instead of an overview of human psychology. The curriculum we developed guided students through a semester-long walk-through of the usability engineering process: i.e., requirements gathering, user interaction design, and formative and summative evaluation. This organization was adopted to convey to students that usability engineering is about developing software systems. It also accommodated a staged semester project in which groups of students constructed a requirements study, followed by a design document, followed by an evaluation report for a software system. In effect, these three project documents constitute the students’ documentation of their project as a case of usability engineering, practicing the various concepts and skills they learned through other course activities. Rosson used the Virtual Science Fair, a collaborative community network application [Carroll et al. 2001a; Carroll et al. 2001b] as a running example throughout the course in 1998 and 1999. Over time, the Virtual Science Fair example became more codified as a case study of usability engineering. In 2000, we received an award from the National Science Foundation's CCLI (Curriculum, Course, Laboratory, and Instrumentation) program to develop an online library of cases, modeled on and extending our Virtual Science Fair case. Our experience with the case library in teaching usability engineering has been very positive. Computer science undergraduates are in general not fond of reading assignments. But we found that motivated by the homework assignments and other casebased activities described earlier, they did read the cases, and that, throughout the semester, they developed sophisticated and nuanced understandings of the challenges, techniques, and underlying values of usability engineering practices. This is not to say that the students are delighted by the reading requirements or that all of them embrace the values and concepts of what is sometimes called human-centered software design. Rather, it is to say that the discourse in class becomes more technically focused on how human-centered software design works. We still discuss the foundations of usability engineering, including argumentation about the rights of users and the responsibilities of developers. But we feel that these topics no longer have to be belabored all semester long (and in lieu of more technical discussions). Our discussions with colleagues in April and June 2003 (see the Appendix) validated our own classroom experiences, and helped to identify further pedagogical benefits of case-based teaching. For example, several instructors felt that the case studies presented prototyping at a particularly useful level of abstraction. Prototyping is a key concept and a critical technique in usability engineering, but most instructors felt their students have trouble understanding prototyping as an independent design and analysis skill. The students confused prototyping with implementation of a running prototype. The case studies helped to clarify this distinction by describing and analyzing a variety of prototypes — ranging from low-fidelity hand-drawn sketches to high-fidelity interactive simulations — that were created in the various projects, and the roles they played in the development process, but not presenting the implementations of those prototypes. The instructors described a variety of uses for the case studies, many of which we had not explicitly anticipated. For example, one instructor used one of the cases as a context for a final exam; students were told to study that case study and then to draw upon it in couching their answers. Another instructor developed an activity in which groups of students created a design brief based on the requirements analysis of a case study, but proposed an alternative design solution to the one documented in the case study. Another ACM Journal on Educational Resources in Computing, Vol. 5, No. 1, March 2005.


•

11

instructor used a role-playing activity in which students adopted either a business or technical role and then debated the decisions in the case study from those perspectives. Yet another instructor used the cases as grist for a class discussion of conflict and conflict resolution in teamwork. Several of the instructors felt that the cases were, on first encounter, overwhelming to students. Students who do not already understand and appreciate the system development process can become disoriented by all the details as they work their way through a case. Some instructors reported that they had inserted a brief evaluation activity at the beginning of their courses to give students a sense of how a usability-engineering process is measured. The instructors’ rationale was that seeing concrete evaluation outcomes is motivating to students, and helps raise questions in their minds about the prior phases of the development process. This conjecture about the importance of showing students usability evaluation early in the course suggests the approach of working backwards through a complete case, from evaluation to requirements analysis. No one in our interest group specifically suggested or reported doing this, but it would be another way to get to evaluation quickly, and thereby perhaps mitigate overwhelming the students during their first encounter with the case studies. Many suggestions for improving the case collection and the browser were raised in our meetings. For example, some instructors suggested developing a very brief "starter" case that could be discussed within one class meeting. Their rationale was that this would allow students to see the entire usability-engineering process in miniature, and thereby make interactions with the full-scale case studies less overwhelming. Some instructors suggested that we should construct deliberately incomplete case studies, consisting of requirements analysis only, or requirements and activity design only. Such partial cases would, by their form alone, engage students in thinking about subsequent phases in the development process; there would not be an option to just read ahead. It was also suggested that we should deliberately construct case studies that illustrate bad practices. Most of the instructors had presented "bad examples" of user interface design from the Web-based Internet Hall of Shame,3 and found that their students enjoyed deconstructing real design errors. The problem with this suggestion is that our project relied heavily on the cooperation of developers to create case studies, and it is not likely that that cooperation would be offered for development projects that incorporated serious flaws. A related suggestion was to highlight more strongly the conflicts and conflict resolutions that inhere in system development. Our cases emphasize conflicts in the sense of design tradeoffs; but we did not emphasize how different stakeholders in a process might evaluate tradeoffs differently, possibly leading to interpersonal conflicts. This seems more feasible, and might make the case studies seem more lively and realistic to the students. Many suggestions were made to improve the case study browser, some were straightforward extensions and refinements. For example, users can access the case study documents through the cases themselves (as in Figure 1), though document-based categories, or via a simple search tool. Some of the instructors would have liked a precompiled index that organizes the content with respect to a set of usability-engineering topics and terms. Most instructors wanted a larger and more diverse collection of cases that represented other software sectors, such as government information systems, electronic games, virtual communities, and system software. Some wanted video documents added to the case studies to model techniques involved in requirements interviewing, metaphor brainACM Journal on Educational Resources in Computing, Vol. 5, No. 1, March 2005.

12

•


storming, think-aloud protocol evaluation, and so forth. Most of the instructors wanted access to syllabi, tests, and class activities of other instructors. They also thought it would be useful to create an "instructor's guide" that describes best practices for using case studies to teach usability engineering, and to maintain a discussion board or wiki to enhance the formation of a community of users. One of the instructors was interested in exploring whether case studies could be associated with design patterns; that is, whether some or all of the cases could be summarized as exemplifying abstract and reusable patterns of software development. Another instructor was interested in including contrasting information within the cases, explaining how they might have been carried out differently if other software development methodologies had been employed. Many of the instructors were interested in building evaluation tools into the casestudy browser, for example, tools to present online surveys and to enable users to track session log-ins. These enhancements are not as challenging technically as they are socially and legally; they raise issues of informed consent and protection of student data. Some of the instructors’ suggestions would require significant redesign of the browser. For example, instructors wanted to directly reference individual documents within a case study (e.g., via url). Because our browser retrieves the cases from a database, the individual documents can only be accessed through navigation. Instructors wanted to be able to create paths through a case study, so they could more reliably guide their students to particular content, or expose them to case-study documents in a particular sequence. Instructors wanted to be able to add their own cases to the collection, to add their own metadata to cases already in the collection, and to add their own links among documents and parts of documents in the collection. They wanted case-study templates, alternative visualizations, editors, and other tools to support all this. The instructors also wanted their students to be able to use the case browser more directly as an environment in which to work together. Thus, they wanted support for student annotation and other editing, project management support for student teams, and an area for student teams to be able to publish their own case studies, analyses, and other reports. THE STUDENT EXPERIENCE As described earlier, since 2001we have made use of the cases in a variety of ways in our undergraduate usability-engineering classes. As we gradually introduced more casestudy content and case-based activities through the years 2001-2003, we noticed improvements in student attitudes and motivation and in their performance and achievement. These outcomes are difficult to defend objectively, given that the course materials (including our textbook, which students used in manuscript form, and during, 2001), the course administration, and perhaps even the instructors' skills were all developing in concert through these years. During the spring semesters in 2004 and 2005, the course was offered in the School of Information Sciences and Technology at the Pennsylvania State University. We used Penn State's course management system for electronic submissions of homework assignments and bundled an extra-credit survey that made use of our case studies with each of the assignments. During each of these two years we administered six homeworkspecific surveys. In 2004, we also administered a summarizing survey at the end of the course; in 2005, we administered a refined version of this survey at both the beginning and end of the course, so that we could explore changes in self-perceived competencies as a result of taking the course. We first discuss student feedback about the case-based ACM Journal on Educational Resources in Computing, Vol. 5, No. 1, March 2005.


•

13

homework assignments, then turn to the more general results from the course-evaluation surveys. Use of Case Studies for Homework Assignments There is considerable variety in the ways that students use cases and in the lessons that they draw from case-based learning. The preponderant use of the case studies reported by students was as a source of examples or models. In some instances this involved fairly literal resynthesis of case-study content: "I drew from the existing metaphors and scenarios for material to use as examples within the assignment." In some instances it involved analogy: "I read the entire garden.com case study online and used its scenarios and claims as a guide for mine." And in some instances, students used the case studies for general orientation to the homework activity: "I looked over the case studies in the book and the repository simply to get a feel for the type of language and tone that a problem scenario should carry." It is common to browse the case studies, focusing on specific types of content or specific pieces of content: "I mainly used the design sketches to locate examples of how problem claims were met or not met." Another reported: "I looked at the conceptual metaphors section of the garden.com case study as well as the customer interviews to support the conceptual metaphors I chose." Students sometimes browsed looking for critical decisions or features: "I just began by looking at the information and trying to find issues I may not agree with. I wanted to try and find some form of drastic change to evaluate." Another student reported: "I went through the case study and looked at the different issues that were addressed. After evaluating them I decided which ones had the biggest tradeoffs in regards to their pros and cons." The students perceived the case studies as useful. Typical general comments were that the cases helped them understand what usability engineering is about as a professional practice: "I gained some insight on the discipline of usability engineering. I realized how flexible and open-ended it was, which also makes it difficult to comprehend exactly what goes into it." Many students commented that the cases helped to relate the text and lecture material to real problems: "I was able to see how conceptual metaphors fit in with requirements analysis and activity design first hand." Students also reported fairly specific issues that the cases addressed: "I now can create problem scenarios and claims easily"; "I learned to open my mind a bit and try to think in terms of relations and metaphors. I'm not very good at it but it's something I can improve upon in future applications and uses"; and "Both the repository and the text were crucial in understanding the progression of prototypes." Some students specifically pointed to the richness of the cases as a learning resource: "I liked the abundant amount of metaphors given in the case study. It really helped me to envision the different parts one needs to consider when designing a site like garden.com." Some of the lessons learned involved the most critical concepts for this usabilityengineering course. For example 19 of 21 students in the 2005 course reported that they learned how to identify design issues and claims from scenarios through carrying out homework 1 (14 of 21 students reported that they learned how to write a problem scenario through this homework activity). Students also reported insights obtained from the case studies that are quite nuanced and typically attained through real-world experience. For example, one student reflected on how prototyping came across in the case studies: "Prototypes don't need to be pretty, they just need to represent what you're testing." This really is the essence of prototyping, but it is something many usability engineers only learn through work experience. ACM Journal on Educational Resources in Computing, Vol. 5, No. 1, March 2005.

14

•


Another student reflected on the management of tradeoffs in system development: "It was interesting to follow one or two tradeoffs from beginning to end. It showed that sometimes designers can recognize the problems associated with a design decision and still not be able to fully reach a good compromise." Again, this is a significant insight into design problem-solving. Many novice system developers fully believe that computational media are so plastic that any result is attainable, and many novice usability engineers believe that all aspects of usability should be optimized in system designs. Another example is a student who reflected on the complexity of real work: "It gave me some idea of how much research is actually put into starting a business (garden.com). The fact that there were so many interviews, surveys, and other research done shows how committed the owners were to ensuring that they [would] have a successful product." Another commented on the distinction between technical excellence and market success: "I learned how a great idea can also be a great failure." Early in the semester, students sometimes complained about using the online repository; they wanted to print on paper, as in this comment about homework 1: "I relied heavily on the online material and would have found it more helpful if the material was possible to download/print as a whole file for reading and referencing. Having to click here and there was cumbersome let alone added to the idea of having to remember where I read or saw something." But this changed quickly, as in these comment from Homework 2: " Unlike Homework 1, I had no problems finding the information I was looking for relevant to this assignment" and "A benefit to using the UCS case studies, as opposed to examples in the textbook, is I could locate relevant information in less time." Later in the semester, the students seemed to be quite comfortable with the online cases as seen in these comments about Homework 7: "I have become comfortable using the online case studies and from this find them to be of more value to me because of how I can navigate through the case studies to find information that I want or need to refer to. The reading of the information in the book does not stick in my mind as well as I mentally map out information online with the case studies, and from this I get irritated at having to search out information in the book when I can't recall exactly where it was" and "The book goes well with the information online, however I have become more dependent with information online than what is found in the book." Students did voice some complaints about the cases and case-based activities. However, in some instances, these complaints suggest that students are carrying out the learning activities as designed, but with some reluctance. A common theme in these complaints is that the cases require students to think: "I had to ‘think outside the box’ on a lot of the assignment to even come up with anything to write about it. They show too many pictures and not enough text explaining what is going on in the situation. They really made no effort to say if it was resolved or not, I had to infer it from the design and predict if the change was going to be correct or not." Another complained: "It was difficult to locate all of the prototypes because they were spread out over the case study and did not explicitly say 'prototype'." Another student reported: "The conceptual metaphors didn't come right out and show how they affect requirements or activity design, it took deeper looking." And another commented: "It made it a little difficult because when trying to conceptualize tradeoffs I had to be creative and off the cuff with possibilities." (Note that these comments are classified here as complaints because they were responses to a survey question asking about things that were difficult to do or to find in the cases.) Other complainants asked that the cases be elaborated more comprehensively: "I would liked to have seen how the positive and negative



•

15

consequences of each [information] design decision were resolved in the interaction design section." Although it is not the preponderant pattern, students do explore cases on their own initiative. In addition to carrying out the assigned activities, our students reported browsing and reading cases on their own. The overall rate of self-reported exploration was 22% (36/164), aggregating over all six assignments in 2004, and over the four assignments in 2005. About half the time, students reported that they just explored further than the assignment required; for example, by studying an entire case study when the assignment only required them to study the requirements analysis phase. The rationale for this additional exploration involved seeking further examples of a concept or principle, or comparing and contrasting other case studies to the one required for the assignment.

Percent rating (across 30 items, 21 students)

Course Surveys At the end of the semester, students could complete a survey for extra credit. A portion of the survey presented 30 self-efficacy items that addressed specific competencies that the course was designed to teach. Perceptions of self-efficacy are assessed by measuring strength of agreement to assertions about one's capacity for a specific achievement in the context of a specific obstacle [Bandura 1997]. For this usability-engineering course, an example item is "I could design a novel user interface metaphor for a personal information system, even if the system display was quite small." Here the specific competency is using metaphors in user interface design, and the obstacle is screen space limitations on visual design. People respond to such items by rating their confidence on a Likert scale. In 2004, we used a five-point Likert scale (1=Strongly Agree, 2=Agree, 3=Neutral, 4=Disagree, 5=Strongly Disagree). In general, the students agreed with the self-efficacy statements; the average rating across the 30 items ranged from 2.57 (for “I could design a ubiquitous paging and messaging system and adequately address users' concerns about privacy”) to 1.71 (for “Even if my manager had no background in visual perception, I could explain my use of the Gestalt principles to guide user interface design”). The overall average was 2.10, very close to the scale value Agree. The response distribution across the 30 items can be seen in Figure 2; over 75% of the individual responses were in the Agree or Strongly Agree category. Although this survey was not designed to be specific to 60.0

54.3 50.0

40.0

30.0

22.2 20.0

16.0

10.0

7.0 0.5

0.0

Strongly Agree

Agree

Neutral

Disagree

Strongly Disagree

Fig. 2. Percent distribution of students’ agreement to 30 self-efficacy items included in the end-of-course survey, Spring 2004.


16

•


case-based learning activities, these results from 2004 suggest that the students felt a reasonable level of self-efficacy with respect to the course learning objectives. In 2005, we refined the survey and administered it at both the beginning and end of the semester. To encourage a greater range in student responses, we asked them to use an eight-point scale (Very Strongly Agree, Strongly Agree, Agree, Leaning toward Agreeing, Leaning toward Disagreeing, Disagree, Strongly Disagree, Very Strongly Disagree). We deliberately omitted a neutral point, thus forcing a commitment to either the agreement or disagreement ends of the scale. Obviously, eight-point judgments of this sort are more difficult for people to make but also more informative, at least in principle. None of the students indicated having any difficulty making these judgments. The general pattern of results was similar to that obtained in 2004, with students indicating greater agreement with the items by the end of the course. The average response at the start of the course ranged from 4.48 (for “Even with a very diverse user population, I could use analytic evaluation methods, like heuristic evaluation and GOMS modeling, to help guide design work”) to 3.09 (for “Even if a requirements analysis was thought to be complete, I could extend it with an original scenario”). The overall average across the 30 items was 3.65. Interestingly, many of the average ratings were on the Agree side of the scale, even at the very beginning of the course! We were somewhat surprised by this, but interpreted it as a combination of generalized over-confidence (these were advanced undergraduates) and reliance on common understandings of terms like requirements, scenario, and design. As expected, the students voiced stronger agreement with the competency items by the end of the course. The item averages ranged from 3.63 (for “I could carry out a comprehensive summative evaluation comparing a proposed new release of Instant Messenger with the current software, even if I was given only 3 weeks”) to 2.06 (for “Even if my manager had no background in visual perception, I could explain my use of the Gestalt principles to guide user interface design”). Note that this is the same item that received the top score in 2004, suggesting that the course does a particularly good job in teaching the Gestalt perceptual principles. The overall average was 3.01. A paired-sample Course start

Course end

50.0 45.6 45.0 38.2

40.0

Percent ratings

35.0

32.2

30.0 25.2

25.0 20.0

14.6

15.0 10.0 5.0

5.4

6.1

13.4

5.8

2.6

5.0

2.7

1.0 0.6

0.0

Very strongly agree

Strongly agree

Agree

Learning toward agreeing

Leaning toward disagreeing

Disagree

Strongly disagree

0.0 0.0

Very strongly disagree

Fig. 3. Percent distribution of rating scale responses across the 30-item self-efficacy scale administered at the beginning and end of the course in Spring 2005. Twenty-three students completed the survey at the start of the course; sixteen completed the survey at the end.



•

17

one-tailed t-test on the responses of the 16 students who took both extra-credit surveys revealed a significant increase in their mean ratings (t(15)=5.99, p