BY
DORIT NEVO AND MICHAEL R. WADE
HOW TO AVOID
DISAPPOINTMENT BY DESIGN
Avoid market failure by aligning system performance with stakeholder expectations.
I
n a paradox experienced by many IT managers, organizational information systems (OISs) are implemented and function exactly as designed—a success by all objective measures—yet users are still disappointed and consider them unsuccessful. Exploring this paradox here, we recommend ways to reduce the root causes of such disappointment. Researchers and practitioners alike have long focused on the success of IT projects [9]. Measures of
success include time and budget considerations, financial indicators, and usage rates. However, according to the Standish Group’s 2004 annual report of IT project resolution history (www.standishgroup. com), 29% of projects failed, and another 53% were “challenged.” Only 18% were deemed by respondents to have been successful. A failure rate as high as this or indeed a success rate as low as this would not be tolerated in any other industry. Yet, according to the same study, the rates for failed, challenged, and successful IT projects have been virtually unchanged since the mid-1990s. Could it be that IT professionals simply deliver bad products and poor services? Of course not, but what
© Scott Adams, Inc./Dist. by UFS, Inc.
COMMUNICATIONS OF THE ACM April 2007/Vol. 50, No. 4
43
accounts for the widespread disappointment with so tation is disconfirmed. If a product outperforms many systems? expectations (positive disconfirmation), the result is The answer depends to a great extent on expecta- post-purchase satisfaction. If a product falls short of tions. When OIS performance does not match OIS expectations (negative disconfirmation), the conexpectations, the system is perceived as unsuccessful. sumer is likely to be dissatisfied [7, 10], provided iniThus, even if it is elegantly designed, well built, and tial expectations were positive [8]. Disconfirmation is functions exactly to specifications, the system can still most often measured by asking respondents whether disappoint users if it falls short of (often lofty) expec- something is “better than/worse than expected.” tations (see the sidebar “Discounting Expectations”). ECT is most commonly applied to simple conConsider the example of a customer relationship sumer goods and services (such as video cameras, management system. If the purchaser expects it to Internet service providers, and Web sites). More comhelp produce better customer service, then disap- plicated is understanding expectations from largepointment is sure to follow, as much more than tech- scale OISs (such as ERP modules, CRM systems, and nology is needed to achieve this goal. Similar Voice over IP solutions). As noted earlier, these sysexamples can be drawn for knowledge management tems are complex, producing multiple forms and levsystems, enterprise systems, and other forms of OIS. els of expectations. Here, we explore insights obtained Many are so overloaded with expectations, they are through a study we conducted to investigate the almost doomed to fail in the eyes of their users. How role of expectations in a large-scale VoIP solution. can organizations avoid The findings suggest that this problem? the traditional ECT A key to satisfying cusapproach, outlined in Expectations tomers is to match their Figure 1, may need to be expectations with perforrevised to account for Disconfirmation Satisfaction mance or overdeliver. OIS complexity. They However, the antecedents offer useful guidelines to Perceived Performance of expectations for OISs organizations designing are numerous and— and implementing IT unlike simple consumer solutions that cater to the expectations of their stakeFigure 1. Expectationgoods—often not well confirmation theory, indicating holders. understood. OISs are the effect of product expectations In the study, four focus groups were conducted on stakeholder satisfaction. complex, multi-faceted over two days during the summer of 2004 in product/service offerings Toronto. The OIS was a hosted VoIP solution1 only Nevo fig 1 (4/07)introduced to the market by a large requiring coordinated efforts by many different recently telecommunications vendor. It was an appropriate groups, each with its own set of expectations. Here, we reveal the related pitfalls and provide vehicle for the study since it was a complex OIS with insight on managing expectations to support more both product and service components. Furthermore, successful OIS adoption and use. We combine insight there was a widespread lack of understanding of the from past work on expectations with data obtained features and benefits of organizational VoIP solufrom focus groups and identify several major issues tions and thus a high potential to create expectaorganizations need to focus on to manage stakeholder tions. Despite the promise of reduced costs, expectations related to OISs. The importance of increased functionality, and improved organizameeting or exceeding expectations has long been tional performance, the VoIP market in 2004 had known to affect user satisfaction. A leading schema, still not lived up to industry expectations [4]. The called expectations-confirmation theory, or ECT, disappointing uptake of VoIP solutions by organizafrom the marketing field, is often used to model the tions and consumers alike can be traced to a number relationships among expectations, performance, and of factors, including lower-than-expected cost savuser satisfaction (see Figure 1) [7, 10]. ECT consists ings, less-than-perfect reliability and quality of serof four main variables: expectations, performance, vice, and insufficient bandwidth [5]. Many of these disconfirmation, and satisfaction. Expectations are factors relate more to expectations generated by the predictive, indicating expected product or service attributes at some point in the future [10]. Expecta- 1 VoIP systems are typically one of two varieties: hosted or on-premise. Hosted solutions serve as the comparison standard in ECT—how tions are managed by a service provider; clients do not own the equipment directly, a monthly fee for a package of services. In on-premise solutions, the client buys consumers evaluate performance [6]. If performance paying the equipment and manages the VoIP system internally. The two approaches are analdeviates from this expectation, the consumer’s expec- ogous to Centrex and PBX, respectively. 44
April 2007/Vol. 50, No. 4 COMMUNICATIONS OF THE ACM
There appears to be a greater downside risk to underdelivering on expectations than an upside reward for overdelivering. proponents of the technology than to the technology itself. Participants in the focus groups were senior-level IT, systems, or telecommunications managers in midand large-size organizations. They were the primary decision makers regarding telecommunications solutions in these organizations. The sampling frame for the research was drawn from the Dun & Bradstreet directory (www.dnb.com), and participants were prescreened for appropriateness for the study. Four groups of eight or nine participants were drafted, for a total of 33 participants. The groups were screened to ensure that none of the participating firms were currently using a VoIP solution but that each was familiar with the concept. The discussion guide was organized as follows: A professional facilitator first introduced participants to the focal topic through a general discussion of existing OISs (such as CRM and ERP) within their organizations. This phase was designed to get the participants to think in terms of expectations and performance measures. The second phase aimed to generate a list of expectations specifically related to the hosted VoIP solution; the facilitator asked participants to review marketing material on the VoIP solution. The telecommunications partner provided two product brochures; the first presented a marketing view of the solution and the second fact-based FAQs. The participants then discussed the expectations that had been generated by this material, organizing them into higher-level categories (such as quality, reliability, and cost). Finally, the facilitator asked participants to consider hypothetical scenarios of negative and positive disconfirmation outcomes and discuss the effect of disconfirmation on satisfaction with the system. We then used the insights obtained from the focus groups to augment the basic ECT model, adapting it to the special characteristics of large-scale OISs. We organized our insights into three main issues—categories of expectations, relations among stakeholder groups, and effect on satisfaction—organizations must emphasize to manage OIS expectations. CATEGORIES OF EXPECTATIONS “I really feel this is a marketing piece. They paint a very good picture. It seems too flawless. Reading
between the lines, you are at their mercy. You have no control. From reading this, I would expect the install to be flawless. It’s too good to be true. My expectation would be it would be beneficial in the long run, but the install would be rocky”—Information administrator, large professional services firm. Members of each focus group collectively identified VoIP expectations, then organized them into higher-order categories (see the table here). Theoretical validation for these categories also existed, as they mapped well to the constructs identified by the model of IS success in [2, 3]. The table also includes examples of expectations generated within each category, many derived directly from the marketing material provided by the telecommunications vendor. As most ECT research focuses on discrete consumer goods, product performance has been the predominant factor on which to base expectations. DISCOUNTING EXPECTATIONS Focus-group participants did not believe everything they heard or read about VoIP, frequently “discounting” the expectations created by salespeople and marketing materials. This discount ranged from 10% to 80%, with the median value around 30%. That is, participants believed only about 70% of what they heard or read, considering the remaining 30% as overselling of product features, overpromising of service levels, or more general marketing embellishment. They dampened expectations through this intuitive discounting process in order to not be surprised, disappointed, or embarrassed. “You have to meet in the middle. They are sales people as well. You don’t want to tell your users what they say. I don’t believe everything they say. Maybe 60%–70%. I’d rather be pleasantly surprised, and my clients as well.”—IT project manager, global insurance company. “I would start at 50%, then it would rise or fall depending where I was in the sales process.”—CIO, large commercial real estate firm. “I discount depending on the vendor 30%–40%.”—Director of IT, school board. The discounting of expectations was amplified when passed through to other stakeholders within the organization. IT managers would tend to undersell some benefits and features of the solution to internal clients—typically end users or senior managers—to minimize the possibility of creating unrealistic expectations. Though not explicitly captured in the study, internal stakeholders might also discount their own expectations based on what they hear from IT managers. The most prudent course of action for IT professionals is to create realistic expectations in the minds of relevant stakeholders. However, there may be rare instances where managers might wish to overpromise, knowing that end users will discount their expectations. Conversely, IT managers might choose to underpromise in certain categories (such as reliability and implementation) where the penalty for negative disconfirmation is greatest (according to the zone of variation). c
COMMUNICATIONS OF THE ACM April 2007/Vol. 50, No. 4
45
accounts for the widespread disappointment with so tation is disconfirmed. If a product outperforms many systems? expectations (positive disconfirmation), the result is The answer depends to a great extent on expecta- post-purchase satisfaction. If a product falls short of tions. When OIS performance does not match OIS expectations (negative disconfirmation), the conexpectations, the system is perceived as unsuccessful. sumer is likely to be dissatisfied [7, 10], provided iniThus, even if it is elegantly designed, well built, and tial expectations were positive [8]. Disconfirmation is functions exactly to specifications, the system can still most often measured by asking respondents whether disappoint users if it falls short of (often lofty) expec- something is “better than/worse than expected.” tations (see the sidebar “Discounting Expectations”). ECT is most commonly applied to simple conConsider the example of a customer relationship sumer goods and services (such as video cameras, management system. If the purchaser expects it to Internet service providers, and Web sites). More comhelp produce better customer service, then disap- plicated is understanding expectations from largepointment is sure to follow, as much more than tech- scale OISs (such as ERP modules, CRM systems, and nology is needed to achieve this goal. Similar Voice over IP solutions). As noted earlier, these sysexamples can be drawn for knowledge management tems are complex, producing multiple forms and levsystems, enterprise systems, and other forms of OIS. els of expectations. Here, we explore insights obtained Many are so overloaded with expectations, they are through a study we conducted to investigate the almost doomed to fail in the eyes of their users. How role of expectations in a large-scale VoIP solution. can organizations avoid The findings suggest that this problem? the traditional ECT A key to satisfying cusapproach, outlined in Expectations tomers is to match their Figure 1, may need to be expectations with perforrevised to account for Disconfirmation Satisfaction mance or overdeliver. OIS complexity. They However, the antecedents offer useful guidelines to Perceived Performance of expectations for OISs organizations designing are numerous and— and implementing IT unlike simple consumer solutions that cater to the expectations of their stakeFigure 1. Expectationgoods—often not well confirmation theory, indicating holders. understood. OISs are the effect of product expectations In the study, four focus groups were conducted on stakeholder satisfaction. complex, multi-faceted over two days during the summer of 2004 in product/service offerings Toronto. The OIS was a hosted VoIP solution1 only Nevo fig 1 (4/07)introduced to the market by a large requiring coordinated efforts by many different recently telecommunications vendor. It was an appropriate groups, each with its own set of expectations. Here, we reveal the related pitfalls and provide vehicle for the study since it was a complex OIS with insight on managing expectations to support more both product and service components. Furthermore, successful OIS adoption and use. We combine insight there was a widespread lack of understanding of the from past work on expectations with data obtained features and benefits of organizational VoIP solufrom focus groups and identify several major issues tions and thus a high potential to create expectaorganizations need to focus on to manage stakeholder tions. Despite the promise of reduced costs, expectations related to OISs. The importance of increased functionality, and improved organizameeting or exceeding expectations has long been tional performance, the VoIP market in 2004 had known to affect user satisfaction. A leading schema, still not lived up to industry expectations [4]. The called expectations-confirmation theory, or ECT, disappointing uptake of VoIP solutions by organizafrom the marketing field, is often used to model the tions and consumers alike can be traced to a number relationships among expectations, performance, and of factors, including lower-than-expected cost savuser satisfaction (see Figure 1) [7, 10]. ECT consists ings, less-than-perfect reliability and quality of serof four main variables: expectations, performance, vice, and insufficient bandwidth [5]. Many of these disconfirmation, and satisfaction. Expectations are factors relate more to expectations generated by the predictive, indicating expected product or service attributes at some point in the future [10]. Expecta- 1 VoIP systems are typically one of two varieties: hosted or on-premise. Hosted solutions serve as the comparison standard in ECT—how tions are managed by a service provider; clients do not own the equipment directly, a monthly fee for a package of services. In on-premise solutions, the client buys consumers evaluate performance [6]. If performance paying the equipment and manages the VoIP system internally. The two approaches are analdeviates from this expectation, the consumer’s expec- ogous to Centrex and PBX, respectively. 44
April 2007/Vol. 50, No. 4 COMMUNICATIONS OF THE ACM
There appears to be a greater downside risk to underdelivering on expectations than an upside reward for overdelivering. proponents of the technology than to the technology itself. Participants in the focus groups were senior-level IT, systems, or telecommunications managers in midand large-size organizations. They were the primary decision makers regarding telecommunications solutions in these organizations. The sampling frame for the research was drawn from the Dun & Bradstreet directory (www.dnb.com), and participants were prescreened for appropriateness for the study. Four groups of eight or nine participants were drafted, for a total of 33 participants. The groups were screened to ensure that none of the participating firms were currently using a VoIP solution but that each was familiar with the concept. The discussion guide was organized as follows: A professional facilitator first introduced participants to the focal topic through a general discussion of existing OISs (such as CRM and ERP) within their organizations. This phase was designed to get the participants to think in terms of expectations and performance measures. The second phase aimed to generate a list of expectations specifically related to the hosted VoIP solution; the facilitator asked participants to review marketing material on the VoIP solution. The telecommunications partner provided two product brochures; the first presented a marketing view of the solution and the second fact-based FAQs. The participants then discussed the expectations that had been generated by this material, organizing them into higher-level categories (such as quality, reliability, and cost). Finally, the facilitator asked participants to consider hypothetical scenarios of negative and positive disconfirmation outcomes and discuss the effect of disconfirmation on satisfaction with the system. We then used the insights obtained from the focus groups to augment the basic ECT model, adapting it to the special characteristics of large-scale OISs. We organized our insights into three main issues—categories of expectations, relations among stakeholder groups, and effect on satisfaction—organizations must emphasize to manage OIS expectations. CATEGORIES OF EXPECTATIONS “I really feel this is a marketing piece. They paint a very good picture. It seems too flawless. Reading
between the lines, you are at their mercy. You have no control. From reading this, I would expect the install to be flawless. It’s too good to be true. My expectation would be it would be beneficial in the long run, but the install would be rocky”—Information administrator, large professional services firm. Members of each focus group collectively identified VoIP expectations, then organized them into higher-order categories (see the table here). Theoretical validation for these categories also existed, as they mapped well to the constructs identified by the model of IS success in [2, 3]. The table also includes examples of expectations generated within each category, many derived directly from the marketing material provided by the telecommunications vendor. As most ECT research focuses on discrete consumer goods, product performance has been the predominant factor on which to base expectations. DISCOUNTING EXPECTATIONS Focus-group participants did not believe everything they heard or read about VoIP, frequently “discounting” the expectations created by salespeople and marketing materials. This discount ranged from 10% to 80%, with the median value around 30%. That is, participants believed only about 70% of what they heard or read, considering the remaining 30% as overselling of product features, overpromising of service levels, or more general marketing embellishment. They dampened expectations through this intuitive discounting process in order to not be surprised, disappointed, or embarrassed. “You have to meet in the middle. They are sales people as well. You don’t want to tell your users what they say. I don’t believe everything they say. Maybe 60%–70%. I’d rather be pleasantly surprised, and my clients as well.”—IT project manager, global insurance company. “I would start at 50%, then it would rise or fall depending where I was in the sales process.”—CIO, large commercial real estate firm. “I discount depending on the vendor 30%–40%.”—Director of IT, school board. The discounting of expectations was amplified when passed through to other stakeholders within the organization. IT managers would tend to undersell some benefits and features of the solution to internal clients—typically end users or senior managers—to minimize the possibility of creating unrealistic expectations. Though not explicitly captured in the study, internal stakeholders might also discount their own expectations based on what they hear from IT managers. The most prudent course of action for IT professionals is to create realistic expectations in the minds of relevant stakeholders. However, there may be rare instances where managers might wish to overpromise, knowing that end users will discount their expectations. Conversely, IT managers might choose to underpromise in certain categories (such as reliability and implementation) where the penalty for negative disconfirmation is greatest (according to the zone of variation). c
COMMUNICATIONS OF THE ACM April 2007/Vol. 50, No. 4
45
Class of Expectations
However, research on expectations for informa- Business lead to more successful sysStrategic tion systems indicates that they involve more than sustainability/ tems. competitive expectations advantage just product performance [11]. Our MULTIPLE STAKEHOLDERS findings support this research, indicatImplementation “If my non-IT people ing that in the case of the hosted VoIP (such as Revenue Managerial generation expectations fit with IS knew how to use it and solution, expectations infrastructure were excited about it, I reflected a number of factors, would be extremely ranging from technical Functionality Reliability (such as number Efficiency (such Operational happy”—Director of netdetails to user-oriented fea- (such as expectations of bugs and as cost saving) working, large apparel tures, to high-level organiza- ease-of-use) errors) retailer. tional needs. This is End user IT manager Executive “We didn’t want to build up too much significant, as it ties together Stakeholder Group expectation. We were concerned in selling research on information systhe expectations to our internal customers tems and ECT to highlight the fact that disappointment Figure 2. Organizational as to whether or not the product could deliver what information system the vendor said it could.”—Managing director of may be linked to multiple expectations by class and factors, not just to product stakeholder group. communications, large accounting firm. Building on the previous section, it is relevant to performance. Nevo fig 2 (4/07) According to ECT, these categories affect not only ask to whom is the OIS important. Expectations not expectations but also attitudes about performance, dis- only can vary by type but by stakeholder as well [9]. confirmation, and satisfaction. IT managers and sys- Most ECT research has assumed a single staketem designers should identify and examine each of the holder—the end user, or person who chooses, purthree categories separately in order to pinpoint the chases, and uses the product. However, we found that source of potential disappointment. For example, a with complex OISs (such as VoIP systems) the end VoIP solution might meet expecCategory of Example expectations within each category Raised tations concerning the quality of expectations by # of (raised by focus group participants) groups the system itself but fall short for Superior to other products on the market. All business sustainability and com- Quality of product and service Vendor will provide superior service. petitive advantage. The practical Reliability of product and service No bugs in the product. All Vendor will provide timely service. implication of such a result could Cost not to exceed quoted estimate. All involve a reexamination of the Cost/value Value for money. need for the VoIP solution in the Allow us to do more with less. 2 organization. If that solution per- Efficiency of product Save us time. forms below expectations (such as Allow us to increase sales. 2 less functionality or less efficiency Revenue generation Help us to increase profits. than expected), the organization Have multiple, useful features. 2 should reexamine the specific Functionality/versatility Be able to customize the product to our specific needs. solution it purchased rather than sustainability/ Will give us a competitive edge. 2 the use of VoIP in general. Under- Business competitive advantage Will provide long-term strategic benefit. standing the source of disappointImplementation Few problems when implementing the product. 2 ment itself is valuable knowledge Will work seamlessly with existing systems. for an organization. An additional implication of Categories of user is not likely to be the same person who manages the expectation categories is their expectations from a differential effect on overall satis- hosted VoIP solution. the system or is authorized purchasing it in the first Nevo table (4/07) place. Each stakeholder is likely to have a different set faction [11]. Since satisfaction from the OIS is the sum total of the satisfaction from of expectations about the system, with some potential the individual expectation categories, IT professionals overlap. Figure 2 makes this point, using the tradimust understand the importance of each category in tional classification of IS support for strategic, mangenerating overall satisfaction. Identifying the most agerial, and operational decision making. Moreover, evidence uncovered in the study supimportant categories, perhaps by polling users in advance, may save organizations valuable resources by ports interdependence of the satisfaction among the enabling them to focus on the most important single various stakeholders. We found participants’ satisfaccontributing factor to satisfaction and subsequently tion was driven by their own disconfirmation, as well 46
April 2007/Vol. 50, No. 4 COMMUNICATIONS OF THE ACM
as by how well the technology solution worked for of expectations leads to satisfaction but not to delight; other stakeholders in the organization. For example, thus scores were lower than 5. IT managers made it clear that until other internal When asked to indicate (on a scale of 210 to 110, stakeholders were satisfied, they themselves would not with 0 being “met expectations”) how their satisfaction might change for negative or positive disconfirbe satisfied. The fact that OIS satisfaction depends on confirming mation, respondents indicated a greater shift toward the expectations of multiple stakeholders has important the negative than the positive (see Figure 3). There implications for IT professionals. It is thus critical that appears to be a greater downside risk to underdeliverthey devise a strategy for managing expectations and ing on expectations than an upside reward for overdeensuring that an OIS’s capabilities and benefits are well livering. For example, Figure 3 suggests that if the understood by multiple stakeholders prior to its introduction into the organization. Understanding Zone of Variation for Quality the expectations and interdependencies of stakeholder groups is Quality (-8.3... 5.6) also key in devising measurements Reliability (-9... 7) for OIS success [9]. Thus, IT managers and system designers must Cost (-8.2... 6.8) finely balance the expectations of Efficiency (-6.2 ... 5.3) each group. An additional benefit of studyRevenue (-7... 6.1) ing stakeholder satisfaction lies in Functionality (-8.1... 5.5) the organization’s ability to explain (and predict) OIS failure. Biz. Sustainability (-7.5... 3.9) IT managers can be dissatisfied Implementation (-8.3... 6.3) with a new system that operates according to their expectations, -10 -5 0 5 10 just because the new system does Expectations = Performance not meet end-user expectations. Greatly Somewhat Somewhat Greatly unmet unmet exceeded exceeded Understanding this alternate cause of dissatisfaction can assist organizations in achieving satisFigure 3. Zone of system greatly exceeds expectations on quality, it faction among all stakeholder indicating how would receive a rating of 15.6. But beware if these groups and improving the success variation, satisfaction changes Nevoare figgreatly 3 (4/07) unmet, as the rating drops to rates of IS projects. when performance does expectations not meet expectations. 28.6. This asymmetry was consistent for all eight catEFFECT ON SATISFACTION: THE egories of expectations studied. ZONE OF VARIATION We now introduce a general indicator measuring the “There is functionality that can be excessive. I doubt power of the effect of disconfirmation on satisfaction, it would actually increase productivity. Some of computed from the distance between the extreme negthem are difficult to put a benefit to. We need our ative and extreme positive values for each of the expecphones to be the thing that works regardless of any- tations categories. We term this difference the “zone of thing. If they go down, we might as well go variation,” as it represents the extent of variation in home.”—Director of IT, large beverage retailer. stakeholders’ satisfaction evaluations for negative and During the focus-group sessions respondents were positive disconfirmation.2 To illustrate, our findings asked to consider hypothetical scenarios of met, reveal that cost has a higher zone of variation (28.2 to unmet, or exceeded expectations and rate their level of 16.8) than revenue (27 to 16.1) or efficiency (26.2 satisfaction for each of the expectation categories in to 15.3). This difference implies that satisfaction with each scenario considered. For all eight categories of respect to the cost of the system is more sensitive to expectations, satisfaction in the case of simple confir- negative or positive disconfirmation than satisfaction mation, or met expectations, ranged from 3.6 to 4.1; resulting from revenue gains or efficiency. on a scale of 1 to 5, 1 was extremely dissatisfied, and 5 was extremely satisfied. This result is in line with the 2The proposed zone of variation complements the “zone of tolerance” and “zone of ECT literature, suggesting that simple confirmation indifference” discussed in the ECT literature; for more on these zones, see [8]. COMMUNICATIONS OF THE ACM April 2007/Vol. 50, No. 4
47
Class of Expectations
However, research on expectations for informa- Business lead to more successful sysStrategic tion systems indicates that they involve more than sustainability/ tems. competitive expectations advantage just product performance [11]. Our MULTIPLE STAKEHOLDERS findings support this research, indicatImplementation “If my non-IT people ing that in the case of the hosted VoIP (such as Revenue Managerial generation expectations fit with IS knew how to use it and solution, expectations infrastructure were excited about it, I reflected a number of factors, would be extremely ranging from technical Functionality Reliability (such as number Efficiency (such Operational happy”—Director of netdetails to user-oriented fea- (such as expectations of bugs and as cost saving) working, large apparel tures, to high-level organiza- ease-of-use) errors) retailer. tional needs. This is End user IT manager Executive “We didn’t want to build up too much significant, as it ties together Stakeholder Group expectation. We were concerned in selling research on information systhe expectations to our internal customers tems and ECT to highlight the fact that disappointment Figure 2. Organizational as to whether or not the product could deliver what information system the vendor said it could.”—Managing director of may be linked to multiple expectations by class and factors, not just to product stakeholder group. communications, large accounting firm. Building on the previous section, it is relevant to performance. Nevo fig 2 (4/07) According to ECT, these categories affect not only ask to whom is the OIS important. Expectations not expectations but also attitudes about performance, dis- only can vary by type but by stakeholder as well [9]. confirmation, and satisfaction. IT managers and sys- Most ECT research has assumed a single staketem designers should identify and examine each of the holder—the end user, or person who chooses, purthree categories separately in order to pinpoint the chases, and uses the product. However, we found that source of potential disappointment. For example, a with complex OISs (such as VoIP systems) the end VoIP solution might meet expecCategory of Example expectations within each category Raised tations concerning the quality of expectations by # of (raised by focus group participants) groups the system itself but fall short for Superior to other products on the market. All business sustainability and com- Quality of product and service Vendor will provide superior service. petitive advantage. The practical Reliability of product and service No bugs in the product. All Vendor will provide timely service. implication of such a result could Cost not to exceed quoted estimate. All involve a reexamination of the Cost/value Value for money. need for the VoIP solution in the Allow us to do more with less. 2 organization. If that solution per- Efficiency of product Save us time. forms below expectations (such as Allow us to increase sales. 2 less functionality or less efficiency Revenue generation Help us to increase profits. than expected), the organization Have multiple, useful features. 2 should reexamine the specific Functionality/versatility Be able to customize the product to our specific needs. solution it purchased rather than sustainability/ Will give us a competitive edge. 2 the use of VoIP in general. Under- Business competitive advantage Will provide long-term strategic benefit. standing the source of disappointImplementation Few problems when implementing the product. 2 ment itself is valuable knowledge Will work seamlessly with existing systems. for an organization. An additional implication of Categories of user is not likely to be the same person who manages the expectation categories is their expectations from a differential effect on overall satis- hosted VoIP solution. the system or is authorized purchasing it in the first Nevo table (4/07) place. Each stakeholder is likely to have a different set faction [11]. Since satisfaction from the OIS is the sum total of the satisfaction from of expectations about the system, with some potential the individual expectation categories, IT professionals overlap. Figure 2 makes this point, using the tradimust understand the importance of each category in tional classification of IS support for strategic, mangenerating overall satisfaction. Identifying the most agerial, and operational decision making. Moreover, evidence uncovered in the study supimportant categories, perhaps by polling users in advance, may save organizations valuable resources by ports interdependence of the satisfaction among the enabling them to focus on the most important single various stakeholders. We found participants’ satisfaccontributing factor to satisfaction and subsequently tion was driven by their own disconfirmation, as well 46
April 2007/Vol. 50, No. 4 COMMUNICATIONS OF THE ACM
as by how well the technology solution worked for of expectations leads to satisfaction but not to delight; other stakeholders in the organization. For example, thus scores were lower than 5. IT managers made it clear that until other internal When asked to indicate (on a scale of 210 to 110, stakeholders were satisfied, they themselves would not with 0 being “met expectations”) how their satisfaction might change for negative or positive disconfirbe satisfied. The fact that OIS satisfaction depends on confirming mation, respondents indicated a greater shift toward the expectations of multiple stakeholders has important the negative than the positive (see Figure 3). There implications for IT professionals. It is thus critical that appears to be a greater downside risk to underdeliverthey devise a strategy for managing expectations and ing on expectations than an upside reward for overdeensuring that an OIS’s capabilities and benefits are well livering. For example, Figure 3 suggests that if the understood by multiple stakeholders prior to its introduction into the organization. Understanding Zone of Variation for Quality the expectations and interdependencies of stakeholder groups is Quality (-8.3... 5.6) also key in devising measurements Reliability (-9... 7) for OIS success [9]. Thus, IT managers and system designers must Cost (-8.2... 6.8) finely balance the expectations of Efficiency (-6.2 ... 5.3) each group. An additional benefit of studyRevenue (-7... 6.1) ing stakeholder satisfaction lies in Functionality (-8.1... 5.5) the organization’s ability to explain (and predict) OIS failure. Biz. Sustainability (-7.5... 3.9) IT managers can be dissatisfied Implementation (-8.3... 6.3) with a new system that operates according to their expectations, -10 -5 0 5 10 just because the new system does Expectations = Performance not meet end-user expectations. Greatly Somewhat Somewhat Greatly unmet unmet exceeded exceeded Understanding this alternate cause of dissatisfaction can assist organizations in achieving satisFigure 3. Zone of system greatly exceeds expectations on quality, it faction among all stakeholder indicating how would receive a rating of 15.6. But beware if these groups and improving the success variation, satisfaction changes Nevoare figgreatly 3 (4/07) unmet, as the rating drops to rates of IS projects. when performance does expectations not meet expectations. 28.6. This asymmetry was consistent for all eight catEFFECT ON SATISFACTION: THE egories of expectations studied. ZONE OF VARIATION We now introduce a general indicator measuring the “There is functionality that can be excessive. I doubt power of the effect of disconfirmation on satisfaction, it would actually increase productivity. Some of computed from the distance between the extreme negthem are difficult to put a benefit to. We need our ative and extreme positive values for each of the expecphones to be the thing that works regardless of any- tations categories. We term this difference the “zone of thing. If they go down, we might as well go variation,” as it represents the extent of variation in home.”—Director of IT, large beverage retailer. stakeholders’ satisfaction evaluations for negative and During the focus-group sessions respondents were positive disconfirmation.2 To illustrate, our findings asked to consider hypothetical scenarios of met, reveal that cost has a higher zone of variation (28.2 to unmet, or exceeded expectations and rate their level of 16.8) than revenue (27 to 16.1) or efficiency (26.2 satisfaction for each of the expectation categories in to 15.3). This difference implies that satisfaction with each scenario considered. For all eight categories of respect to the cost of the system is more sensitive to expectations, satisfaction in the case of simple confir- negative or positive disconfirmation than satisfaction mation, or met expectations, ranged from 3.6 to 4.1; resulting from revenue gains or efficiency. on a scale of 1 to 5, 1 was extremely dissatisfied, and 5 was extremely satisfied. This result is in line with the 2The proposed zone of variation complements the “zone of tolerance” and “zone of ECT literature, suggesting that simple confirmation indifference” discussed in the ECT literature; for more on these zones, see [8]. COMMUNICATIONS OF THE ACM April 2007/Vol. 50, No. 4
47
The zone of variation represents the range of how expectations influence stakeholder satisfaction and provide a useful tool for organizations to assess the best- and worst-case scenarios for a new OIS. As with identifying the importance of expectations categories, the main contribution of identifying the zone of variation is in saving valuable organizational resources. For example, Figure 3 outlines how an organization can learn that overpromising and underdelivering on the reliability of the VoIP solution is more costly than overpromising on the system’s efficiency, since negative disconfirmation for reliability yields lower levels of satisfaction than does efficiency. By the same logic, the organization would gain more from overdelivering system reliability than by exceeding expectations for the system’s efficiency. The zone of variation highlights where most gains can be made (and most pain avoided) by managing expectations, thus providing organizations with a powerful cost-benefit analysis tool for managing expectations. While the specific ranges in Figure 3 are applicable to our example—the hosted VoIP solution—they may not hold for other solutions. IT managers may have to assess the zone of variation for each type of project. Nevertheless, the concept of the zone of variation and its implications are readily generalized. CONCLUSION “[The project] took a really long time, over three years. There had been really high expectations. We expected it to work really well, make life easier. But it ended up making things worse.”—Telecom manager, 500-employee firm. Our aim here has been to address why so many OISs are deemed failures by their users, exploring the paradox that comes from the difference between success and satisfaction; that is, systems may function as designed but still disappoint stakeholders and thus be regarded as failures. Disappointment is inextricably linked to the formation and confirmation of expectations. Thus, we’ve drawn on ECT, using focus-group data to enhance the model and increase its fit with large-scale OISs. It is not sufficient to design and produce a functioning OIS. Relevant stakeholders must have an accurate perception of the functionality of the system. System designers should focus not only on users in the analysis and design stages [1] but collaborate with project managers and marketers to ensure that expectations generated by OISs match the system’s capabilities. While some embellishment may be necessary to “make the sale,” it behooves organizations to consider the process by which expectations affect stakeholder satisfaction perception and actual project success. 48
April 2007/Vol. 50, No. 4 COMMUNICATIONS OF THE ACM
This work contributes to the general understanding and management of OIS expectations. We’ve highlighted the importance of using different categories of expectations within ECT to capture the special nature of large-scale OISs. We’ve proposed that the importance of expectations moderates the strength of their effect on satisfaction. We’ve shown how to relate expectations and satisfaction of all stakeholders involved in the adoption and use of an OIS. By adding the satisfaction levels of other stakeholders to the model, we’ve provided yet another explanation for dissatisfaction, enabling organizations to better identify problems and focus on their solution. Finally, we’ve also proposed the notion of the zone of variation, identifying the range of satisfaction gain (or loss) to be expected from managing expectations. This notion and its application enables organizations to focus resources where they have the most to gain. In a world of overhyped solutions and concepts, the IT professional must be able to determine a system’s capabilities and match stakeholder expectations with those capabilities. c References 1. Biehl, M. Success factors for implementing global information systems. Commun. ACM 50, 1 (Jan. 2007), 52–58. 2. DeLone, W. and McLean, E. The DeLone and McLean model of information systems success: A 10-year update. Journal of Management Information Systems 19, 4 (Spring 2003), 9–30. 3. DeLone, W. and McLean, E. Information systems success: The quest for the dependent variable. Information Systems Research 3, 1 (Mar. 1992), 60–95. 4. Duffy, J. SMBs driving demand for VoIP services. Network World 22, 9 (Mar. 2005); www.networkworld.com/news/2005/030705specialfocus.html. 5. Ellison, C. and Kaven, O. VoIP: Finally worth a look. PC Magazine (Aug. 2004); www.pcmag.com/article2/0,1759,1630778,00.asp. 6. Halstead, D. The use of comparison standards in customer satisfaction research and management: A review and proposed typology. Journal of Marketing Theory and Practice 7, 3 (Summer 1999), 13–26. 7. Oliver, R. A cognitive model of the antecedents and consequences of satisfaction decisions. Journal of Marketing Research 17, 4 (Nov. 1980), 460–469. 8. Santos, J. and Boote, J. A theoretical exploration and model of consumer expectations, post-purchase affective states and affective behaviour. Journal of Consumer Behaviour 3, 2 (Dec. 2003), 142–156. 9. Seddon, P., Staples, D., Patnayakuni, R., and Bowtell, M. Dimensions of information systems success. Communications of the AIS (Nov. 1999). 10. Spreng R., MacKenzie, S., and Olshavsky, R. A reexamination of the determinants of consumer satisfaction. Journal of Marketing 60, 3 (July 1996), 15–32. 11. Staples, D., Wong, I., and Seddon, P. Having expectations of information systems benefits that match received benefits: Does it really matter? Information & Management 40, 2 (Dec. 2002), 115–131.
Dorit Nevo (
[email protected]) is an assistant professor of management information systems in the Schulich School of Business of York University, Toronto, Canada. Michael R. Wade (
[email protected]) is an associate professor of management information systems in the Schulich School of Business of York University, Toronto, Canada. © 2007 ACM 0001-0782/07/0400 $5.00
