Studying Knowledge Management System Success in System ...

Proceedings of the 42nd Hawaii International Conference on System Sciences - 2009

Studying Knowledge Management System Success in System Administration Nicole F. Velasquez The University of Arizona [email protected]

Alexandra Durcikova The University of Arizona [email protected]

Abstract Significant temporal and monetary costs are associated with the development and implementation of a knowledge management system (KMS). To justify this investment, it is important for organizations to measure the success of such systems. To help achieve this objective, this paper proposes a KMS success model, which is based on prior success models for information systems (IS) and KMS. This model extends previous models by investigating two modes of KMS use – knowledge sourcing and sharing. Additionally, individual impact is extended to include both merit (perceived usefulness) and worth (productivity) metrics. By also taking into account the effects of other knowledge sources, such as Google, email, and instant messaging, we are able to explore the contribution of KMS use. This research in progress proposes testing the model in the context of system administrators, technical experts who work in complex and dynamic environments.

1. Introduction Forty-five percent of executives surveyed as part of the Foresight 2020 study stated the belief that effective knowledge management offers the most potential for increased productivity in the next decade [9]. Yet, building and implementing knowledge management systems (KMS) that truly improve employee productivity is difficult [20, 11]. Therefore, the evaluation of success or failure of these systems is of importance to both researchers and practitioners. In general, the success of an information system (IS) in prior research has been evaluated using the DeLone and McLean (DM) model of IS success [6] or its adaptations [34, 31, 7]. Given that a knowledge management system is a specialized type of IS, many models of KMS success are built upon these IS success models. Recent examples include Maier [22], Jennex and Olfman [15-17], Kulkarni et al. [19], and Bock et al. [3]. Our research builds on these existing models of KMS success, but seeks to

Rajiv Sabherwal University of Missouri, St. Louis [email protected]

advance the literature by incorporating some additional aspects: two modes of KMS use, the specifics of individual impact, and the general context in which the KMS is used. This research proposes the study of a KMS in the context of system administration. Systems that help improve the productivity of system administrators have become increasingly important with the human cost of system administration doubling over the past ten years and now exceeding the cost of hardware and software [27]. Therefore, investigating the effect of KMS use on system administrator productivity is of high interest to most companies. The purpose of this paper is to investigate how the two modes of KMS use (knowledge sourcing and knowledge sharing) influence individual impact (perceived usefulness and productivity) in the context of system administrators work. This paper contributes to both knowledge management research and practice by investigating 1) how KMS use impacts relevant outcomes of system administrators work, such as perceived usefulness and productivity, 2) how the two types of KMS use, sourcing and sharing, influence these relevant outcomes, and 3) what the fractional benefit of KMS use on individual outcomes is while controlling for other sources of knowledge.

2. Theoretical Background & Hypotheses 2.1. Measuring KMS success DeLone and McLean presented a model of IS success grounded in information and communication theories [35, 25]. This causal model includes six interrelated constructs: information quality, system quality, use, user satisfaction, individual impact, and organizational impact. This model has been validated in many contexts, with most studies focusing on the individual impact of the system, citing difficulties in capturing organizational impact [31]. In this model, system quality and information quality affect both IS use and user satisfaction, which

978-0-7695-3450-3/09 $25.00 © 2009 IEEE

1


in turn affect individual impact. Although this model does not explicitly include perceived usefulness, most studies identify perceived usefulness as an individual impact, often citing the difficulty of obtaining hard impact measures [31]. The portion of the model most commonly tested, culminating in individual impact measures, is presented below in Figure 1.

System Quality

IS Use Individual Impact

Information Quality

User Satisfaction

2.2. KMS Use

Figure 1. DM Model of IS Success (1992) One significant modification of the original DM IS success model was proposed by Seddon [34]. The main difference in these two models is the location of the IS Use construct. Seddon stated that the use of an information system is a result of the success of the system, and is therefore placed after the success measures of Perceived Usefulness and User Satisfaction. Seddons modification of DMs IS success model most commonly tested is presented below in Figure 2.

System Quality

Perceived Usefulness

Information Quality

User Satisfaction

support measures. Finally, Bock et al [3] extend the Seddon model by investigating the effects of social context on KMS success. It is clear that in these recent KMS success models, the foundation of IS success models is intact, though adapted to the nature of KMS. While these studies have contributed to the understanding of KMS success, we see room for improvement in three areas: 1) KMS use, 2) individual impact, and 3) the knowledge environment in which a KMS is implemented. We will discuss these extensions of the model in the following sections.

IS Use

Figure 2. Seddon Model of IS Success (1997) KM success models build upon either the DM IS success model or Seddons respecification. Jennex and Olfman (JO) [15] presented a KM success model based on Pitt et al.s adaptation [29] of the DM model [6], which added service quality as an antecedent to use and user satisfaction. The JO model adapted this extended model by expanding system quality, information quality, and service quality constructs specific to KMS. Qian and Bock [30] build on the JO model [15] by proposing and empirically testing a model that investigates antecedents to output quality and extends output and system quality measures to better capture KMS characteristics. On the other hand, Kulkarni et al. [19] propose a model of KMS success based on Seddons DM respecification, extending the model by controlling for organizational

When studying KM and KMS, researchers clearly define two distinct activities involved in the use of a KMS: 1) knowledge sourcing [12], and 2) knowledge sharing [38]. We define knowledge sourcing as theextenttowhichindividualsintentionallyaccesseach othersexpertise, experience, insights, andopinions [12]. Building upon this definition of knowledge sourcing and the work of others [18, 38], we define knowledge sharing as the extent to which individuals intentionally share their expertise, experience, insights, and opinions. While some users may only source knowledge from a system [23], others may dominantly share their knowledge, perhaps assigned to the role of knowledge champions in their organizations [8]. Many times, however, users accept both roles and participate in both sourcing and sharing knowledge. It is not clear, however, how these two KMS usage behaviors influence individual impact. Therefore, there is a need to model KMS use as two behaviors performed by a user. 2.2.1. Antecedents of KMS Use. Based on prior IS and KMS success models, system quality and output quality affect KMS use and user satisfaction. Additionally, system quality affects output quality. In the context of this study, KMS use can take two forms: knowledge sourcing and knowledge sharing. System quality has been represented in KMS success research studies as searchability [3], and it is also the most frequently identified issue with KMS failure [24, 4]. If an individual views a KMS as difficult to search, that KMS would be less likely to be used [32]. This is true for both sourcing and sharing activities. Why would a user spend time submitting knowledge to a system that does not appropriately retrieve it? Furthermore, the easier it is for a user to search for knowledge in the KMS, the more satisfied the user will be with the KMS [31]. Moreover, good KMS searchability allows a user to access knowledge necessary to perform tasks. If

2


two KMSs with similar knowledge content quality exist, the KMS with greater searchability will yield better output than the other [3]. Therefore, high searchability positively influences output quality. Consistent with previous KMS and IS success models [6, 34, 31, 3], if employees perceive that the system quality and searchability is high, it will positively impact KMS use, user satisfaction, and output quality. Therefore, recognizing that KMS use incorporates knowledge sourcing and knowledge sharing, we hypothesize: H1a: System quality positively affects knowledge sourcing. H1b: System quality positively affects knowledge sharing. H1c: System quality positively affects user satisfaction. H1d: System quality positively affects output quality. Information quality has been represented in KMS success research studies as output quality [3] or knowledge content quality [19]. If an individual is able to obtain good quality output from a system, he will be more likely to use the system again [6]. Also, he will be more likely to contribute his knowledge to the KMS [37]. Consistent with prior IS and KMS success models, high output quality leads to high user satisfaction. We therefore posit the following: H2a: Output quality positively affects knowledge sourcing. H2b: Output quality positively affects knowledge sharing. H2c: Output quality positively affects user satisfaction. Finally, previous research tells us that user satisfaction affects KMS use. Users who are more satisfied with a KMS are more likely to use it [6, 34]. H3a: User satisfaction knowledge sourcing. H3b: User satisfaction knowledge sharing.

positively

affects

positively

affects

2.3. Individual impact The dependant variable, individual impact, deserves further discussion. Due to the difficulty of capturing objective use metrics, most studies operationalize individual impact as perceived usefulness [31]. While perceived usefulness is widely used in models [31] , subjective measures are primarily important to the end users themselves.

However organizations are also looking for hard measures that can be more closely tied to financial metrics. Most systems are intended to improve the efficiency of end users, often with the goal of reducing labor costs [10]. Indeed, Lyytinen [21] and Massey et al. [26] suggest that different measures of success are appropriate for different stakeholder groups. Brown et al. [5] identify two types of success metrics for users and organizations: merit and worth. Merit is concerned with whether the system performs as it is supposed to and includes user experiences of the system. Perceived usefulness is a measure of system merit and is often the success metric used in IS and KMS success studies. Worth is concerned with whether the system has an organizational impact that justifies its cost. Worth metrics, such as productivity, are often less important to end users, yet ultimately important to the organizations that sponsor the development and implementation of such information systems. With the clear importance of both subjective (merit) and objective (worth) measures, we propose the inclusion of both in the measurement of individual impact. To determine the merit and worth metrics appropriate for the study of a KMS used by system administrators, we observed their work and conducted interviews. 2.3.1. The work of system administrators. System administration involves the installation, configuration, troubleshooting, and maintenance of computing infrastructures, much of which is often at the bleeding edge. Hardware and software additions to the network, firmware updates, and usage fluctuations result in a network that is constantly in flux and incredibly complex [1, 2, 14]. This continually evolving environment results in new situations and contexts every day, making even routine tasks a new challenge. The difficulty in supporting this complex infrastructure lies in simply locating the source of an error [28] and then finding up-to-date information about the error and its set of possible causes and solutions. Because so many things can go wrong, much of a system administrators time is spent defining the problem and then researching and debugging it. Although much of their work is invisible to the outside world, visible manifestations of their knowledge work lies in a variety of hardware and software tasks such as: installing patches, laying cable, installing and configuring new hardware, and creating infrastructure schematics [2]. This knowledge work is most often learned on the job and through self-learning, and mastering the field of

3


system administration may take several years [33]. For example, one system administrator we interviewed stated that while he had done the work of system administration for almost a decade, he had only been a system administrator for the past five or six years. The system administrators targeted for this study are level 2 and higher technical support operators, whose work is dictated by problem tickets and change tickets. Problem tickets are entered in the system by supported customers in response to a system error, such as a server crashing. Change requests are entered in the system by customers or by management to improve the system; these can include a firmware upgrade or new server installation. One aspect of their annual performance assessment is based on the amount of tickets they resolve and their average response time on tickets. Studying system administrators in this context allows us the unique opportunity to investigate their KMS use while providing discrete productivity measurements based on problem and change ticket completion. As described above, this study uses two measures of individual impact. Consistent with DM and JO models, we argue that KMS use positively influences perceived usefulness and productivity. Following these studies, user satisfaction positively influences perceived usefulness and productivity. H4a: Knowledge sourcing positively affects the perceived usefulness of a KMS.

H4b: Knowledge sourcing positively affects worker productivity. H4c: Knowledge sharing positively affects the perceived usefulness of a KMS. H4d: Knowledge sharing positively affects worker productivity. H5a: User satisfaction positively affects the perceived usefulness of a system. H5b: User satisfaction positively affects worker productivity.

2.4. Control variables Studies show that knowledge workers use not only knowledge repositories but also other knowledge sources to do their job [13]. These can include colleagues and manuals [13]. Studies of system administrator work practices have shown that they utilize knowledge management systems, Google, vendor documentation, and colleagues to seek and share knowledge essential in the execution of their work [2]. How then, does a knowledge management system affect the individual impact of system administrators in the presence of these other sources of knowledge? In order to answer this question, we need to control for other sources of knowledge when studying the role of KMS use in KMS success. In addition to controlling for other knowledge sources, our study will control for variables found significant in other studies. These variables include incentives (intrinsic and extrinsic [30, 3]), individual characteristics (experience and tenure [36]), organizational characteristics (time pressure [11], trust [3], and leadership [19]).

KMS Use System Quality

Knowledge Sourcing Knowledge Sharing

Output Quality

Individual Impact Perceived Usefulness

Productivity User Satisfaction

Figure 3. Proposed research model

4


3. Proposed Methodology

5. References

An international Fortune 500 company has agreed to participate in this study. A division of this company contracts out system administration support work involved in the configuration, maintenance, and management of computing infrastructures. All work activities are initiated through problem tickets or change tickets, as described above. In an effort to improve the productivity of their system administrators, the organization has developed a knowledge management system. This voluntary use system provides a list of tickets to be worked on, current and historical system state information, and an interface to search for solutions to similar tickets. The organization employs thousands of external support system administrators and has agreed to randomly select 300 to participate in this study. Survey data will be collected through online surveys and the employer will supply productivity data in Fall 2008. Except for productivity measures, all other measures will be adopted from previous research. The data will be analyzed using structural equation modeling, given that this method allows us to test complex causal models.

[1] Barrett, R., Y.Y.M. Chen, and P. Maglio, System Administrators Are Users, Too: Designing Workspaces for Managing Internet-Scale Systems (Workshop), in Conference on Human Factors in Computing Systems (CHI' 03), R. Barrett, M. Chen, and P. Maglio, Editors. 2003: Ft. Lauderdale, FL, USA.

4. Conclusion This study proposes an extension to existing KMS success models by measuring the effects of both knowledge sharing and knowledge sourcing activities. When evaluating individual impact of KMS use we will be able to evaluate the additional contribution of KMS use when other sources of knowledge are available. Furthermore, we will also be able understand whether there is a difference in the influence of the two KMS use modes on productivity and perceived usefulness of the KMS. A better understanding of the impacts of sourcing and sharing can help us answer some interesting questions: Are free-riders (those that only source) more productive than knowledge providers? As compared to free-riders, are knowledge providers adversely affected by spending time to share their knowledge? If so, are free-riders more satisfied with the KMS? In regard to the context of this study, who are the more productive system administrators, knowledge seekers or knowledge providers These important questions have not been adequately addressed in prior literature, and we hope that the proposed study will contribute by providing some initial answers.

[2] Barrett, R., E. Kandogan, P. Maglio, E. Haber, L. Takayama, and M. Prabaker. Field Studies of Computer System Administrators: Analysis of System Management Tools and Practices. in ACM Conference on Computer Supported Cooperative Work (CSCW '04). 2004. Chicago, IL, USA. [3] Bock, G., R. Sabherwal, and Z. Qian, The Effect of Social Context on the Success of Knowledge Repository Systems. IEEE Transactions on Engineering Management, forthcoming: p. 46. [4] Bowman, B.J., Building Knowledge Management Systems. Information Systems Management, 2002. 19(3): p. 32-40. [5] Brown, J., A. Massey, and E. Boling, Evaluation of Knowledge Management Systems: Insights from the Study of a Technical Support Knowledge Base. Knowledge Management Research & Practice, 2005. 3(2): p. 49-59. [6] DeLone, W. and E. McLean, Information Systems Success: The Quest for the Dependent Variable. Information Systems Research, 1992. 3(1): p. 60-95. [7] DeLone, W. and E. McLean, The Delone and Mclean Model of Information Systems Success: A Ten-Year Update. Journal of Management Information Systems, 2003. 19(4): p. 9-30. [8] Durcikova, A. and S. Brown. Influence of System, Environment, and Procedures on Knowledge Submission Frequency in Hawaii International Conference on System Sciences (HICSS '07). 2007. Big Island, HI, USA. [9] EIU, Foresight 2020: Economic, Industry and Corporate Trends. 2006, Economist Intelligence Unit: New York, NY. p. 1-96. [10] Fitzgerald, G., Evaluating Information Systems Projects: A Multidimensional Approach. Journal of Information Technology, 1998. 13: p. 15-27. [11] Gray, P. and A. Durcikova, The Role of Knowledge Repositories in Technical Support Environments: Speed Versus Learning in User Performance. Journal of Management Information Systems, 2005. 22(3): p. 159-190.

5


[12] Gray, P. and D.B. Meister, Knowledge Sourcing Effectiveness. Management Science, 2004. 50(6): p. 821834.

[24] Marwick, A.D., Knowledge Management Technology. IBM Systems Journal, 2001. 40(4): p. 814831.

[13] Gray, P.H. and A. Durcikova, The Role of Knowledge Repositories in Technical Support Environments: Speed Versus Learning in User Performance. Journal of Management Information Systems, 2006. 22(3): p. 159-190.

[25] Mason, R.O., Measuring Information Output: A Communications Systems Approach. Information and Management, 1978. 1(5): p. 219-234.

[14] Haber, E. and E. Kandogan, Security Administrators in the Wild: Ethnographic Studies of Security Administrators (Workshop), in ACM Conference on Human Factors in Computing Systems (CHI '07). 2007: San Jose, CA, USA. [15] Jennex, M. and L. Olfman. Modeling Knowledge Management Success. in Conference on Information Science and Technology Management (CISTM). 2004. Alexandria, Egypt. [16] Jennex, M. and L. Olfman, Assessing Knowledge Management Success. International Journal of Knowledge Management, 2005. 1(2): p. 33-49. [17] Jennex, M. and L. Olfman, A Model of Knowledge Management Success. International Journal of Knowledge Management, 2006. 2(3): p. 51-69. [18] Kankanhalli, A., B. Tan, and K.-K. Wei, Contributing Knowledge to Electronic Knowledge Repositories: An Empirical Investigation. MIS Quarterly, 2005. 29(1): p. 113-143. [19] Kulkarni, U., S. Ravindran, and R. Freeze, A Knowledge Management Success Model: Theoretical Development and Empirical Validation. Journal of Management Information Systems, 2006. 23(3): p. 309347. [20] Kwan, M.M. and P. Balasubramanian, Knowledgescope: Managing Knowledge in Context. Decision Support Systems, 2003. 35: p. 467-486. [21] Lyytinen, K., Expectation Failure Concept and Systems Analysts' View of Information System Failures: Results of an Exploratory Study. Information & Management, 1988. 14: p. 45-56.

[26] Massey, A.P., M. Montoya-Weiss, and S. Brown, Reaping the Benefits of Innovative It: The Long and Winding Road. IEEE Transactions on Engineering Management, 2001. 48(3): p. 348-357. [27] Patterson, D., A. Brown, P. Broadwell, G. Candea, M. Chen, J. Cutler, P. Enriquez, A. Fox, E. Kiciman, M. Merzbacker, D. Oppenheimer, N. Sastry, W. Tetzlaff, J. Traupman, and N. Treuhalf, Recovery-Oriented Computing (Roc): Motivation, Definition, Techniques, and Case Studies, in Technical report CSD-02-1175. 2002, University of California, Berkeley: Berkeley, CA. [28] Pentland, B.T., Bleeding Edge Epistemology: Practical Problem Solving in Software Support Hot Lines, in Between Craft and Science, S.R. Barley and J.E. Orr, Editors. 1997, Cornell University Press: Ithaca, NY. p. 113128. [29] Pitt, L.F., R.T. Watson, and C.B. Kavan, Service Quality: A Measure of Information Systems Effectiveness. MIS Quarterly, 1995. 19(2): p. 173-187. [30] Qian, Z. and G. Bock. An Empirical Study on Measuring the Success of Knowledge Repository Systems. in Hawaii International Conference on System Sciences (HICSS'05) 2005. Big Island, HI, USA. [31] Rai, A., S. Lang, and R. Welker, Assessing the Validity of Is Success Models: An Empirical Test and Theoretical Analysis. Information Systems Research, 2002. 13(1): p. 50-69. [32] Sabherwal, R., A. Jeyaraj, and C. Chowa, Information Systems Success: Individual and Organizational Determinants. Management Science, 2006. 52: p. 1849-1864. [33] SAGE, Sage Annual Salary Survey 2005-2006, USENIX, Editor. 2006.

[22] Maier, R., Knowledge Management Systems: Information and Communication Technologies for Knowledge Management. 2002, Berlin: Springer-Verlag.

[34] Seddon, P., A Respecification and Extension of the Delone and Mclean Model of Is Success. Information Systems Research, 1997. 8(3): p. 240-153.

[23] Markus, M.L., Toward a Theory of Knowledge Reuse: Types of Knowledge Reuse Situations and Factors in Reuse Success. Journal of Management Information Systems, 2001. 18(1): p. 57-93.

[35] Shannon, C.E. and W. Weaver, The Mathematical Theory of Communication. 1949, Urbana, IL: University of Illinois Press. [36] Sussman, S.W. and W.S. Siegal, Informational Influence in Organizations: An Integrated Approach to

6


Knowledge Adoption. Information Systems Research, 2003. 14(1): p. 47-65. [37] Wang, R.Y. and D.M. Strong, Beyond Accuracy: What Data Quality Means to Data Consumers. Journal of Management Information Systems, 1996. 12(4): p. 5-33.

[38] Wasko, M.M. and S. Faraj, Why Should I Share? Examining Social Capital and Knowledge Contribution in Electric Networks of Practice. MIS Quarterly, 2005. 29(1): p. 35-57.

7