CHI 2001 - Semantic Scholar

Papers

CHI 2001 • 31 MARCH – 5 APRIL

Better Home Shopping or New Democracy? Evaluating Community Network Outcomes John M. Carroll and Mary Beth Rosson Center for Human-Computer Interaction and Department of Computer Science Virginia Tech, Blacksburg, Virginia, 24061-0106 USA +1 540 231 8453 carroll, [email protected] ABSTRACT

This is a perspective paper on community networks — sociotechnical infrastructures supporting villages, towns, and neighborhoods. Community networking is well-established, world wide, and addresses critical societal issues, such as the “crisis of community” and the sociality of the Internet. However, community network projects have not emphasized evaluation. Relatively little is known about the economic, social, and psychological consequences of community networks for the individuals, groups, and communities served. Evaluating community networks is a momentous mutual opportunity for the development of CHI evaluation methodologies and for bringing technical CHI expertise to bear on societal issues. Keywords

Community networks, evaluation, social impact INTRODUCTION

A community network connects neighbors. The purpose of community networking is to facilitate information dissemination, discussion, and joint activity pertaining to municipal government, public schools, civic groups, local events, community issues and concerns, and regional economic development and social services. The literature on community networking projects is inspiring; the concept of local communities taking control of their own information technology resonates with democratic ideals. But the actual outcomes of these projects are unclear. Community networks are poorly funded and lie outside the science and technology mainstream. They are local infrastructures, so they attract only local resources. They tend to focus on survival and development, not on analysis and evaluation. However, community networking is an important paradigm of networked computing. It complements the global-village paradigm of the Internet. Participants in a community network are far more visible and individuated to one another. Their computer-mediated interactions are tightly coupled Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SIGCHI’01, March 31-April 4, 2001, Seattle, WA. Copyright 2001 ACM 1-58113-327-8/01/0003…$5.00.

372 Volume No. 3, Issue No. 1

with actions and events in their personal lives. Relatively anonymous interactions like shopping and entertainment play a smaller role [39]. The contribution of this paper to CHI is a perspective on a computing paradigm not yet addressed in much detail within CHI. Community networking is an important area, particularly in need of empirical study. CHI has much to contribute to understanding the use and impacts of community networking infrastructures. Reciprocally, community networking is a rich domain, different in many ways from the business computing areas on which CHI normally focuses. It provides a testbed for developing more comprehensive evaluation methodologies. COMMUNITY NETWORKS

The roots of community networking are in 1970s community activism—jobs, housing, and veterans’ issues in the Berkeley Community Memory [14], community health in the Cleveland Free Net [4], and problems of the homeless in the Santa Monica Public Electronic Network (PEN) [36,38]. Education has also been a major focus. For example, Big Sky Telegraph supported teachers in rural Montana, linking 1and 2-room schools with regional libraries, and providing computer support for the literary and artistic projects of Native Americans. The project was implemented on obsolete computer equipment refurbished by a local women’s resource center. It connected a remote and quite dispersed community to the world, for example, giving students access via electronic bulletin boards to professors at M.I.T. [46]. Community networking is now a diverse and rapidly-growing world-wide movement. There is no definitive inventory of community network projects, but it is easy to identify on the order of 500 projects through a Web-search: 1. The Association for Community Networking (http://www.afcn.net/) lists 70 members. 2. The Organization For Community Networks (http://ofcn.org/) lists 160 projects in the United States, 25 in Canada, and a handful from Finland, Germany, Ireland, New Zealand, and the Philippines. 3. Free Nets and Community Networks (http://www. lights.com/freenet/) lists 250 projects from Australia, Austria, Canada, Finland, Germany, Hungary, Israel, Italy, Kazakhstan, the Netherlands, Russia, Spain,

CHI 2001


Ukraine, the United Kingdom, the United States, and Uzbekistan. Other launchpads provide regional coverage: 4. Missouri Association of Community Information Networks (http://www.macin.org/) lists 30 projects. 5. The Telecities component of the European Digital Cities project (http://www.edc.eu.int/) lists 125 cities. 6. British Columbia Community Networks Association (http://www.bccna.bc.ca/) lists 250 projects. THE CRISIS OF COMMUNITY

One of the touchstone goals of community networking is facilitating the development local social capita—the trust, social interactions, and norms of mutual reciprocity throughout a community [12]. This is significant, since social capital appears to be declining in American society [5,34]. Bellah et al. interviewed over 200 Americans about the relation between public and private life. These interviews suggest that excesses of individualism now pervade American attitudes, that Americans think of their own society as if it is something other than them, that even civic associations are frequently conceived of in personal terms, for example, as opportunities for self-discovery. Yet they find an equallypervasive angst with respect to satisfaction and meaning in their informants. They contrast their description with Tocqueville’s account of Democracy in America in the 1830s. Tocqueville was impressed by the number and variety of associations in which Americans took part; he saw this as a validation of the Jeffersonian ideal of direct and equal participation. Bellah et al. conclude that contemporary Americans believe that each of us is fundamentally alone, and that public life cannot be fully satisfying with this basic belief. They see a need to rework and reimplement the values and mores that underpinned the original American society. Coming at the question demographically, Putnam’s work paints a similar picture of contemporary society. Between the 1960s and the 1990s, participation rates in a variety of civic activities declined: Red Cross volunteering declined by 60 percent; participation in parent-teacher organizations declined by nearly half, membership in the League of Women Voters and in the Jaycees both declined by 40 percent; the number of people reporting that they attended a public meeting on town or school affairs in the past year has declined by more than a third; volunteering of Boy Scout troop leaders declined by a quarter; voter turnout in national elections declined by nearly a quarter; churchgoing and church-related activities declined by a sixth; the proportion of Americans who socialize with neighbors more than once a year declined by nearly a sixth. Putnam’s touchstone for this broad trend is the decline in bowling leagues. Although the total number of bowlers in America increased by 10 percent between 1980 and 1993 (to nearly 80 million people), league bowling decreased by 40 percent. Americans, Putnam finds, are bowling alone. Putnam investigated various possible causes for these trends—the pressures of modern life, economic hard times,

anyone. anywhere.

Papers

residential mobility, suburbanization, two-career family stresses, weakening of marriage and family ties, the rise of franchising and the service sector, disillusion and cultural conflicts of the 1960s/1970s, growth of the welfare state, the civil rights revolution, and television and other technological developments. By eliminating other possibilities, Putnam came to a generational explanation: Civic behaviors like voting and membership in community groups, and the belief that most people can be trusted, began to decline for people born after 1930. This downward trend has continued for the past 40 years. The magnitude of these effects is substantial. On average, people born in the1920s are twice as likely to vote, belong to twice as many civic groups, and are twice as likely to trust other people as are people born in the 1960s. Putnam considered in detail the shared experiences of what he terms the “civic generation”, people born 1925-1930. These individuals grew up during the Great Depression, coming of age during World War II. Thus, they shared the sustained terror of significant crises in their youth, and the singular triumph of saving the world from fascism as young adults. These experiences may have established both sociality and optimism, particularly when the civic generation is compared to successive generations who grew up in the numbing ambiguities of the Korean War, the Cold War, and the Vietnam War, and the turmoil surrounding civil rights and women’s rights. Putnam also emphasizes that the civic generation was the last American generation to grow up without television. Television viewing correlates negatively with participation in all other social and media activities; it has increased in each successive generation since television was introduced in the 1950s [31]. Among children 9-14, television consumes as much time as all other discretionary activities combined, an average of 40 hours per week. Television has increasingly displaced other activities. People who watch less television than average and read more newspapers than average belong to 76 percent more civic organizations and are 55 percent more trusting than people who watch more television than average and read fewer newspapers than average. People who report being happy when engaged in social interaction, report being bored and unhappy when watching television [26]. Television watching has been linked to reduced physical activity and poorer physical and mental health [1,43]. The broad, negative effect of television on social engagement raises important questions about analogous effects of networked computing. Sitting in front of a computer screen is similar in many respects to viewing television. If use of computer networks consumes time that might otherwise be spent on other discretionary activities, the apparent crisis of community may worsen [13]. At the same time, it is possible that computer networks may facilitate social engagement [47]. Understanding the usage implications of emerging network infrastructures — including community networks — is a significant and urgent societal issue.

373

Papers

SOCIAL IMPACTS OF THE INTERNET

The Internet has turned out to be an extremely social technology. Interpersonal communication is the dominant use of the Internet at home [24]. But what does this mean in terms of opportunities for social engagement? Social interactions mediated by networks do not have the same character as face-to-face interactions [44]. Networks seem to be more suitable for supporting so-called weak social ties— casual, narrow-scope relationships with infrequent contact— than for supporting strong ties—deep-felt, broad-scope relationships with frequent contact. Weak ties help people access information or social resources [13]. But strong ties help people manage stress and maintain psychological health and well-being [10,22]. There has been much speculation about the possibilities for developing strong ties or community-like social systems on the Internet [35,45]. This is an interesting literature, but it is mostly anecdotal. In fact, Kraut and his colleagues [24] found that people use the Internet to maintain current relationships, but that the new relationships they form tend to be based on weak ties. The HomeNet study represents a shift toward more systematic research on social impacts of the Internet. Kraut and colleagues [25] provided 48 families (157 individuals) with networked computers and turnkey access to Internet services and applications. They used the HomeNet testbed to investigate how people get started using the Internet, and what they use it for when they have unfettered access. Almost 20% of the participants who received Internet accounts never logged on once in the first 22 weeks of the study. The modal number of logins was less than twice a week, though there were cases of people who logged in more than 20 times per week (nearly 500 logins during the 22-week trial). Twelve percent of people who logged in sent no email messages, but one sent more than 2500 messages. Use actually declined through the course of the trial: during the first few weeks, 75% of the participants logged in at least weekly, but by week 22 this had declined to about 50%. Use of various Internet services was highly correlated; participants who used any Internet service tended to use all others (the lowest correlation was 0.48). Directory and repository sites (like Yahoo) comprised half of the top ten sites accessed by participants. Thirteen of the 30 newsgroups accessed by the greatest number of participants had sexual themes, but these were generally accessed only once or twice; less than 20% of the sample accessed sexually-oriented newsgroups three or more times. The HomeNet testbed is not a community network, but the sample comprised families of teenagers with interests in journalism, from four Pittsburgh high schools. Thus, there was an antecedent social bond among the participants. It was reported that those who were relatively more sophisticated, or who had professional expertise needed by others, shared this knowledge in HomeNet newsgroups. Participants also expressed a preference for local information and



communication (movie and bus schedules, chats and forums accessed only by students in a single high school). However, these patterns were not quantified. Income and education did not predict Internet use, suggesting that removing economic and usability barriers to getting started can help to equalize access (see also [7]). However, race, gender, and generation were strong predictors. Teenagers, especially white teenage males, were the heaviest users. The median teenager sent 6 times as many email messages as the median adult; teenage males sent 25 times more messages than adult males. Most (94%) teenagers’ messages went to other teens. Several teens discovered Internet Relay Chat (IRC) and began to use it extensively; one girl began dating a boy she met on IRC. Depressed people—but also those who were innovative and sociable— used the Internet more; those with many hassles used it less. People with less free time did not use it less than people with more free time. People who watch TV were less likely to use the Internet, people who read books were more likely to use it. A subsequent report by Kraut, Lundmark, Patterson, Kiesler, Mukhopadhyay and Scherlis [23] carried out a path analysis in which changes in use of the Internet during a 12-24 month period was associated with changes in social involvement and psychological well-being, once various demographic and control variables had been statistically eliminated. In this study, greater use of the Internet was associated with subsequent declines in family communication, and in the size of both the local and distant social circle. People with greater use of the Internet subsequently reported larger increases in loneliness, a greater number of daily life stressors, and more depression. An intriguing question is the extent to which other paradigms for networked computing, including community networks, occasion similar social consequences. EVALUATING COMMUNITY NETWORKS

Studies of the crisis of community and of social impacts of the Internet raise questions about community networks. But little is known about the uses and consequences of these community efforts. Most assessments provide case study anecdotes, basic demographics, or user satisfaction surveys. Research is typically carried out in the context of a single network project, often by the project designers or managers. However, a research literature is developing. Anecdotal evidence suggests that community networking can catalyze development of local technology infra-structure. The Blacksburg Electronic Village (BEV) has been in operation for more than six years. As of January 2000, more than 90 percent of the Blacksburg population (about 36,000) have network access, compared to about 40% nationally. Over 150 community groups and more than 450 local businesses maintain Web sites (more than 75%). The BEV hosts many community-oriented initiatives (15 community newsgroups, various lists, a town chat, a senior’s nostalgia archive, publicaccess kiosks). The Town of Blacksburg uses the BEV extensively, providing on-line forms for surveys, house check

CHI 2001


requests, and e-mail to town officials, as well as dissemination of schedules and other documents. All 20 county schools have been wired for T1 Ethernet access for the past 4 years, compared to about 65% nationally. Kavanaugh has gathered BEV demographics since 1994 http://www.bev.net/project/research/AT.99.P11report.html, including use of communication software, self-assessment of computer skills, and expectations about the utility of the network. Several historical analyses of the project have been carried out (e.g. [41]), as well as a variety of technology innovation case study analyses [9,11]. Anderson [2] surveyed 320 users of the Cleveland Free-Net. Her respondents reported benefits in the areas of exchanging information, ideas, or advice, and keeping up with current events; over 80% reported they were “satisfied” with the network. Surprisingly, Anderson did not find benefits in maintaining contact with family and friends, or in meeting new people. More recently, Patrick [33] surveyed 1073 users of the National Capital FreeNet (NCF) in Ottawa, Canada. Ninety-five percent of the respondents reported a positive impact of the network on their lives: 46% reported watching less television, while only 4% reported watching more; 35% reported sleeping less, while 3% reported sleeping more; 72% reported improved computer literacy; 67% reported informal learning, and 63% reported beneficial social interactions using the network. However, people did not report increased involvement in community life, nor did they report that the network facilitated involvement in community life. The Santa Monica PEN project pushed beyond measures like time allocation and benefit ratings, looking at specific community outcomes. For example, PEN users were more active than other residents in seven types of local offline activity, ranging from attending City Council meetings to contacting city officials [38]. The network was crucial and effective in a grassroots campaign to provide showers and lockers for the homeless, because homeless citizens were able to send email directly and publicly to municipal leaders [36]. Rogers et al. also found that municipal officeholders resented the amount of time they spent answering email and felt picked on by PEN users, and that female users experienced incidents of hostility and harassment [36]. Gygi [19] argued from the work of Berry, Portney and Thomson [6] for measures of quality or depth of participation, as opposed to mere volume of participation. She noted that government information might be put on-line both to reduce dissemination costs, and to enhance opportunities for citizen participation. She urged focus on the impact of participation on political outcomes, such as agenda setting and resource allocation, and on the changes in resources committed by local government for involving citizens or community organizations. She suggests measures like “number of policy initiatives that originate from neighborhood organizations and community groups”, “degree of concurrence between citizen groups and officials on policy

anyone. anywhere.

Papers

issues”, and “number of full-time equivalent staff members dedicated to outreach per 10,000 population”. Gygi’s proposal echoes the distinction Sproull and Kiesler [44] have made between first-level and second-level effects of technology. Examples of first-level effects are increasing use, posting more information, and reducing costs or operator errors. Second-level effects are the more subtle, social consequences enabled by first-level effects. Often they involve tradeoffs: more people participate, but this makes it harder to come to consensus and entrains more risky decisions; leveling of status and increased self-disclosure creates a sense that all participants are on an equal footing, but also may lead to flaming and harassment of females. Other tradeoffs are more basic—a community might install extra phone lines dedicated for computer access at public buildings, but in fact these lines might be better used for families without phones to make telephone calls [18]! Creating an explicit analysis of impacts and tradeoffs in community network technology would facilitate linking community network goals and outcomes. QUESTIONS ABOUT COMMUNITY NETWORKS

There are many critical unanswered questions about community networks. Some of the most salient concern participation. Who in the community participates in local networks, and who does not? What are the causes and effects of unequal participation throughout the community? Little is known about why people choose not to participate in community networks, about the obstacles that prevent some from participating despite a desire to do so, or about the consequences of being excluded or self-excluded. The first author was once contacted (by phone) by a local merchant who knew he had been recently criticized in a BEV community newsgroup. He wanted us to express his side of the issue; he had a PC, but no network connection. What are the consequences of a community network for such people? Another merchant explained his non-participation in the BEV by saying that posting important information would compromise business advantages. This is interesting since it is frequently asserted that community networking will enhance trust and security [39]. Differential access to the network is also a potentially significant issue. Community networks often provide services like homework hot-lines, collaborative writing projects, chat or video conferencing, and community-based mentoring programs. Some of these services assume the high-speed networking that is common in schools, but available at home only to some students. Another set of questions relates to the actual uses of community networks. A typical community networks is a loosely integrated collection of services and applications. But providing such a network access only sets up the possibility for interesting uses and productive outcomes. Satisfaction surveys provide some insight into this, but rely on selfperception rather than actual use. How are local business

375

Papers

activities and opportunities changed, and how direct a cause is the network? In what ways does access to local government information, or to public decision-making change? It is possible that community networks are used largely as gateways for global Internet services. The HomeNet studies documented frequent access to general-purpose sites such as Yahoo. Are participants in a community network even aware of local information and innovations? Do they try the innovations? Without more detailed analyses of use we cannot distinguish between network activity that draws a community into the outside world from that which cements and reinforces local concerns. We need to understand the consequences of community networks on community life. Does a community network increase and/or diversify involvement in community activity? For example, are community members more likely to participate in local institutions like the public schools, or civic organizations like the League of Women Voters, when they are more actively involved with the network? The PEN studies suggest a correlation, but we do not yet know whether network use facilitates community involvement, or vice versa. Do community networks lead to different experiences than a general Internet service—for example do people feel safer? Do they feel better about their community and its ability to recognize and address problems? These questions touch on the issues of social capital discussed by Putnam. The early work of Sproull and Kiesler raises questions about how community networks might impact social networks. Does participation in a community network lead people to develop more extensive local social networks? Does it also (or instead) lead to more extensive distant social networks? Network use seems to shift people from a small number of relatively strong social ties to a larger number of weaker ties [13]. Does this happen within community networks, or does the interleaving of networked and face-to-face interaction indeed strengthen strong ties? Does community networking displace face-to-face interaction, or is it used to supplement and manage face-to-face meetings, as with the telephone [47]? Finally, some of the earliest questions about community networks focused on economic development. For example, many businesses in Blacksburg believe that having a presence in the BEV will increase their business, and the town believed that the network would attract new businesses. But we know little about the actual economic impacts of community networks. Some impacts may be relatively direct, for example increases in sales due to better visibility or marketing. But are there may also be indirect consequences—does a community network lead to a more attractive business climate, more confident investments? EVALUATION AS A MUTUAL OPPORTUNITY

The evaluation of community networks presents an opportunity to clarify the use and impact of an emerging computing paradigm. But it is also an opportunity to broaden



the scope and application of evaluation techniques in CHI. Community networking is unlike workplace computing: a community population is less homogeneous than a business organization; its overall organizing rubric is that people live in proximity. There is usually great diversity among members of a community. Unlike a work organization, a community network’s intended users span all phases of life (children, teenagers, adults, the elderly), personal roles (parents, siblings, friends), and occupations (teachers, shop owners, police). Groups in a community often share a high-level vision or sense of purpose, but they rarely coordinate their agendas at the operational level. Communities have pockets of relative coherence, but each pocket of activity is fairly independent of the others. Community groups do not have well-articulated dependencies and interfaces as often found among groups in a business organization. At the same time, many dependencies exist due to shared access to community resources such as buildings or other infrastructure, or to the time and expertise of community participants. It is common for public buildings such as schools or libraries to be used by many community groups with only slightly overlapping interests and goals. Community networks must run on a variety of platforms, both hardware and software. A work organization often mandates a standard platform and environment; this is impossible in a community network. It is particularly important that community systems provide acceptable user interfaces and performance on fairly low-end infrastructure, since they will often be used from home. The motivation and reward structure for community network use is also fundamentally different from workplace computing; participation comes through intrinsic motivation and individual initiative-taking. Most community residents are not paid for using the network; participation is its own reward. Innovative development within these systems usually occurs within a single, semi-autonomous pocket of the user community, initially serving local and even idiosyncratic interests and needs. A key concern is the collection and integration of these efforts such that the community at large can benefit. Finally, tasks and personal relationships in a community network are more varied, intermingled, and overloaded than in business organizations. Participating in a town chat can be recreation, political activism, and education. Another participant could be an adversary in that debate, but also one’s tennis partner, choir director, and next-door neighbor. Evaluation questions for community networking are broader than the classic issues of interface usability. But the scope of evaluation work in CHI has itself broadened throughout the past two decades. It is typical now to evaluate requirements, functions, and organizational impacts. Community networks push this further–to leisure, informal learning, civic activity, neighborliness, and the creation of social capital. The

CHI 2001


differences between the community networking paradigm and that of business computing make studies of community networks a potentially valuable testbed for CHI evaluation research. DISCUSSION: STRATEGIES AND DIRECTIONS

The mutual opportunity of evaluating community networks can be developed in three ways. First, current CHI evaluation methods can be applied to community networks to explore their feasibility and effectiveness for this paradigm. Second, new evaluation instruments and measures can be developed, or adapted from other disciplines, then generalized for other CHI evaluation needs. Finally, various types of evaluation data and data sets can be combined both to leverage specific information and to build a more comprehensive evaluation framework. Apply and extend evaluation methods

The entire range of evaluation methods used in CHI is relevant to community networking software [32]. Where necessary and possible, new variants may be developed. For example, models, walkthroughs, and laboratory experiments can be used to assess user interface interactions. Event logging can be used to track the events of a single session, but can also be extended to create an interleaved log of events originating at multiple sites. Design rationale can be used for intrinsic evaluation [40], identifying and explicitly enumerating the goals and priorities of project, the inherent tradeoffs in achieving goals, and the apparent state of evidence in the achievement of goals. The instruments and procedures employed in the HomeNet study will be particularly useful, because this is one of the first studies to collect and analyze network usage in a systematic fashion. The HomeNet methods include a social resources inventory (Cohen et al.’s Interpersonal Support Evaluation List), items from the UCLA loneliness scale, a stress scale (Kanner et al.’s Hassles Scale), and items from the Center for Epidemiologic Studies Depression Scale, as well as logging email traffic, Web-access, and newsgroup use. The HomeNet study provides an empirical baseline for understanding attitudes and social impacts measured in community networking. Ethnographic description complements quantitative survey and performance logging methods. It is used widely in CHI as an evaluation method, bringing to light subtle but critical factors in a situation that cause and support behaviors and experiences in participants. Ethnography can clarify the personal and community-oriented goals and motivations that underlie people’s use and experience of a community network. These goals and motives may be somewhat idiosyncratic to individuals, and may also be largely unarticulated; they may be unrecognized by both the individuals themselves and the BEV developers. Ethnography is particularly appropriate for gathering detailed information about community issues, activities, and events, many of which may be antecedent to or otherwise independent of the BEV.

anyone. anywhere.

Papers

A distinctive application of ethnography is the identification of social-technical archetypes. Examples of this include Gantt and Nardi’s [17] gardening patterns and Mynatt, Adler, Ito, and O’Day’s [29] dimensions of network communities. These archetypes are qualitative abstractions induced from ethnographic records. They have proven to be useful in developing vivid and succinct characterizations of the roles people assume, the activities they engage in, and the relationships they develop within various kinds of communities. Develop or import new evaluation methods

The distinctive characteristics of community networks can help to suggest and develop new approaches to evaluation. It is difficult to anticipate innovation, but two examples of evaluation methods that are novel with respect to standard practice, but particularly relevant to community networks are participatory evaluation and community efficacy. Direct user participation in design is widely recognized as effective and appropriate (e.g. [28]). Participation makes user domain expertise more accessible in guiding the development process. This helps avoid misinterpretation of user input, leading to costly prototyping and formative evaluation of inappropriate designs. Participation gives users a personal stake in new technology, increasing the chance that they will make the commitment to adopt it in their work practices. Although participatory methods are common in analysis and design, such techniques are useful throughout the development process. For instance, people using a community network might be willing to contribute directly to its evaluation, sharing and discussing usage episodes. This builds on the human factors engineering tradition of critical incidents [16]. Collection of critical incidents in a community network has been implemented by a Web-based forum [30]. This approach enhances standard critical incident reports by permitting follow-up questions and other conversational elaboration and refinement of reports. It transforms evaluation into an interactive process in which everyone can have a voice. Indeed, an evaluation forum of this sort might become a community network activity in its own right. A second novel approach is measuring the perceived efficacy of a community. Social cognitive theory provides an integrative framework for assessing the domain-specific beliefs about personal capability that influence learning and performance outcomes [3]. High perceived self-efficacy for a domain causes individuals to set more challenging goals, to work harder on difficult aspects of tasks, to master new competencies, and to achieve more. Perceived efficacy is distinct from self-esteem, which is concerned with judgments of self-worth rather than personal capability. Efficacy judgments predict goal selection and performance in a domain; self-esteem does not. Because efficacy is specific to a domain, it is also a more powerful predictor than generalpurpose measures like locus of control, perceived selfcontrol, self-concept of ability, or cognitive competence ([3], pages 47-48).

377

Papers

Bandura [3] has also shown that group members’ beliefs about collective efficacy predict group performance. Collective efficacy is a function of interrelated personal efficacy beliefs. For a student, education-related efficacy judgments derive from self-efficacy for academic topics, or for self-instruction outside of the classroom. Parents’ beliefs are tied to whether they regard education as a responsibility shared with the schools.—parents with high education selfefficacy become involved more readily, and their trust in teachers and schools is greater [42] In consequence, their children perform better academically [15]. Teachers’ education efficacy correlates highly with parental involvement in conferences, volunteering, and home tutoring [21]. Thus measuring community efficacy will require coordinated construction of many interrelated belief scales. Community networking seems quite likely to impact many collective efficacy beliefs. For example, community members (especially parents) are often frustrated when they do not have time to visit the school [37]. Community networking can improve access to school activities, and thus may enhance perceptions of public education efficacy. Similarly, it may increase participation in local politics and elections, perhaps enhancing beliefs about governmental efficacy. Participatory evaluation and measures of perceived efficacy are pertinent to community networks, but potentially also to many other usage situations of interest to CHI. The evaluation of computer-supported cooperative work setting might find variants of these methods to be of use. They are examples of how addressing evaluations issues for community networks could have a general derived benefit to the development of evaluation methodology in CHI. Coordinate and cross-leverage evaluation

As is true of many complex usage contexts, measuring a single aspect of activity within a community network may not be interpretable. For example, it is typical that session logs are somewhat indeterminate with respect to larger goals and motives that caused the session in the first place [20]. Many sources of information might have to be combined to reconstruct the causal chain of events that leads from placing a public access terminal in a library, or from providing dialup access to municipal databases, to an increase in public participation in an episode of political decision-making, and a specific impact on the decision outcome. Recruiting a range of methods make it possible to triangulate [8] or mediate [40] the interpretation of evaluation data. Different methods are differentially susceptible to issues of internal validity, statistical conclusion validity, construct validity, and external validity [27]. Integrating methods offers a strategy for minimizing general threats to validity. Because every community network is unique, developed to suit the needs and constraints of a single community, there are important sampling and validity issues even when a comprehensive set of evaluation data are obtained. For example, the networks mentioned earlier–Blacksburg

378 Volume No. 3, Issue No. 1 377


Electronic Village, Cleveland Free Net, and Santa Monica PEN—differ in myriad ways. And the evaluation studies were also widely different. Developing techniques for pooling such results (and for comparing them to other network studies such as HomeNet), is critical in building a general understanding of community networks. Again, the general challenge of integrating and developing evaluation data—both within single complex projects and among sets of related projects—is central to CHI. As applications and services become more diverse and situationspecific, it becomes increasingly difficult to raise and answer general questions about system usefulness and usability. Community networks provide a highly accessible testbed for research into such issues. Increasingly, community networks are described as standard residential communications infrastructure. The CHI community has the opportunity—and responsibility—to assess the impacts of these systems on community behavior and beliefs. At the same time, the methods used to conduct such assessments will extend the repertoire of techniques available for other evaluation efforts. ACKNOWLEDGMENTS

Our work on community networks is supported by the National Science Foundation, IIS 0080864. We thank our collaborators Anne Bishop, Albert Bandura, Lynne Dunckley, Daniel Dunlap, Philip Isenhour, Andrea Kavanaugh, Robert Kraut, and Dennis Neale. This paper was written while the authors were academic visitors at Xerox Research Center Europe in Cambridge, England. REFERENCES 1. Anderson, R.E.,Crespo, C.J., Bartlett, S.J., Cheskin, L.J. & Pratt, M. 1998. Relationship of activity and television watching with body weight and level of fatness among children. Journal of the American Medical Association, 279, 938-942. 2. Anderson, S.E. 1992. Factors associated with usage of a public telecomputing system. Ed.D. Dissertation, Curry School of Education, University of Virginia. 3. Bandura, A. 1997. Self-efficacy: The exercise of control. W.H. Freeman and Company. 4. Beamish, A. 1995. Communities On-Line: Community-Based Computer Networks. Masters Thesis, Department of Urban Studies and Planning, MIT. 5. Bellah, R., Madsen, R., Sullivan, W., Swindler, A. and Tipton, S. 1986. Habits of the heart: Individualism and commitment in American life. U. California Press. 6. Berry, J.M., Portney, K.E. & Thomson, K. 1993. The rebirth of urban democracy. Brookings Institution Press. 7. Bier, M., Gallo, M., Nuckols, E., Sherblom, S. & Pennick, M. 1997. Personal empowerment in the study of home Internet use by low-income families. Journal of Research on Computing in Education, 30(22), 106-119. 8. Campbell, D., & Fiske, D. W. (1959). Convergent and discriminate validation by the multitrait-multimethod matrix. Psychological Bulletin, 4, 297-312.

CHI 2001


Papers

9. Carroll, J.M. & Rosson, M.B. 1996. Developing the Blacksburg Electronic Village. CACM, 39(12), 69-74. 10. Cohen, S. & Wills, T.A. 1985. Stress, social support, and the buffering hypothesis. Psychological Bulletin, 98, 310-357. 11. Cohill, A.M. & Kavanaugh, A.L. 1997. Community networks: Lessons from Blacksburg, Virginia. Artech House. 12. Coleman, J.S. 1990. The foundations of social theory. Harvard University Press. 13. Constant, D., Sproull, L. & Kiesler, S. 1996. The kindness of strangers: On the usefulness of weak ties for technical advice. Organizational Science, 7, 119-135. 14. Farrington, C. & Pine, E. 1996. Community memory: A case study in community communication. In P. Agre & D. Schuler, eds., Reinventing technology, rediscovering community. Ablex. 15. Finn, J.D. 1993. School engagement and students at risk. National Center for Educational Statistics. 16. Flanagan, J.C. 1954. The critical Psychological Bulletin, 51, 28-35.

incident

technique.

17. Gantt, M. & Nardi, B. 1992. Gardeners and gurus: Patterns of cooperation among CAD users. Proceedings of CHI’92. ACM, pp. 107-117. 18. Guy, N.K. 1996. Community Networks: Building real communities in a virtual space? MA thesis, Simon Fraser University, Dept of Geography. 19. Gygi, K. 1996. Uncovering Best Practices: A Framework for Assessing Outcomes in Community Computer Networking. Community Networking ‘96. 20. Hoiem, D. E., & Sullivan, K. D. 1994. Designing and using integrated data collection and analysis tools. Behaviour & Information Technology, 13(1), 160-170. 21. Hoover-Dempsey, K.V., Bassler, O.C. & Brissie, J.S. 1992. Parent efficacy, teacher efficacy, and parent involvement: Explorations in parent-school relations. Journal of Educational Research, 85, 287-294.

28. Muller, M.J., Haslwanter, J.H. & Dayton, T. 1997. Participatory practices in the software lifecycle. Handbook of HumanComputer Interaction, 2nd edition. Elsevier, pp. 255-297. 29. Mynatt, E.D., Adler, A., Ito, M., & O’Day, V. 1997. Design for network communities. Proceedings of CHI’97: Conference on Human Factors in Computing Systems. ACM, pp. 210-217. 30. Neale, D.C., Dunlap, D.R., Isenhour, P., and Carroll, J.M. 2000. Collaborative critical incident development. Proceedings of the Human Factors and Ergonomics Society, pp. 598-601. 31. Neuman, S.B. 1991. Literacy in the television age: The myth of the TV effect. Ablex. 32. Nielsen, J. 1993. Usability Engineering. Academic Press. 33. Patrick, A.S. 1997. Personal and social impacts of going on-line: Lessons from the National Capital FreeNet. http://debra.dgbt.doc.ca/services-research/survey/impacts/ 34. Putnam, R.D. 2000. Bowling alone: the crumbling and revival of American community. Simon & Schuster. 35. Rheingold, H. 1993. The Virtual Community: Homesteading on the electronic frontier. Addison-Wesley. 36. Rogers, E.M., Collins-Jarvis, L. & Schmitz, J. 1994. The PEN Project in Santa Monica: Interactive communication, equality, and political action. Journal of the American Society for Information Science, 45(6), 401-410. 37. Scherer, M. 1998. The shelter of each other. Educational Leadership, 55(8), 6-11. 38. Schmitz, J., Rogers, E.M., Phillips, K., & Paschal, D. 1995. The public electronic network (PEN) and the homeless in Santa Monica. Journal of Applied Communication Research, 23, 2643. 39. Schuler, D. 1996. New community networks: Wired for change. Addison-Wesley. 40. Scriven, M. 1967. The methodology of evaluation. In R. Tyler, R. Gagne & M. Scriven, eds., Perspectives of curriculum evaluation. Rand McNally, pp. 39-83.

22. Krackhardt, D. 1994. The strength of strong ties. In N. Nohria & R. Eccles (Eds.), Networks and organizations: Structure, form, and action. Harvard Business School Press.

41. Sears, C. 1996. (Re)Visions of the Village: Building and participating in the Blacksburg Electronic Village. Masters Thesis in Science and Technology Studies, Virginia Tech.

23. Kraut, R., Lundmark, V., Patterson, M., Kiesler, S., Mukopadhyay, T. & Scherlis, W. 1998. Internet paradox: A social technology that reduces social involvement and psychological well-being? American Psychologist, 53 (9), 10171031.

42. Shirley, D. 1997. Laboratories of democracy: Community organizing for school reform. University of Texas Press.

24. Kraut, R., Mukhopadhyay, T., Szczpula, J., Kiesler, S. & Scherlis, W. 1998. Communication and information: Alternative uses of the internet in households. Proceedings of CHI 98. ACM, pp. 368-383. 25. Kraut, R., Scherlis, W., Mukhopadhyay, T., Manning, J. & Kiesler, S. 1996. HomeNet: A field trial of residential Internet services. Proceedings of CHI 96. ACM, pp. 284-291. 26. Kubey, R. & Csikszentmihalyi, M. 1990. Television and the quality of life: How viewing shapes everyday experience. Erlbaum. 27. McGrath, J. E. 1994. Methodology matters: Doing research in the behavioral and social sciences

anyone. anywhere.

43. Sidney, S., Sternfeld, B., Haskell, W.L., Jacobs, D.R., Chesney, M.A. & Hulley, S.B. 1998. Television viewing and cardiovascular risk factors in young adults: The CARDIA study. Annals of Epidemiology, 6(2), 154-159. 44. Sproull, L. & Kiesler, S. 1991. Connections: New ways of working in the networked organization. MIT Press. 45. Turkle, S. 1995. Life on the Screen: Identity in the Age of the Internet. Simon and Schuster. 46. Uncapher, W. 1999. New communities/new communication: Big Sky Telegraph and its community. In M. Smith & P. Kollock, eds., Communities in Cyberspace. Routledge.

47. Wellman, B. & Tindall, D. 1993. Reach out and touch some bodies: How social networks connect telephone networks. Progress in Communication Studies, 12, 63-93.

379