definition for "quality" as a combination of end use r information satisfaction ... information technology to develop specialized applications fo r their own personal ...
Quality End User-Developed Applications : Some Essential Ingredient s by Donald L . Amoroso, University of Colorado, Colorado Spring s and Paul H . Cheney, University of South Florida Abstrac t This article examines the dimension of quality in end user developed applications . Quality is defined as the degree t o which an application of "high grade" accomplishes or attain s its goal from the perspective of the user . We propose a definition for "quality" as a combination of end use r information satisfaction and application utilization . We the n discuss three measurement instruments that were developed to capture the dimensions of quality and assess thei r psychometric properties .
ACM Categories : D .2 .9, H .4 .0,1 .2 .1, J .1, K .6 . 2 Keywords : End user computing, quality, applicatio n effectiveness
THE ISSU E We are well into the second decade of end user computin g (EUC) activity within most companies . The prediction o f Robert Benjamin (1982) that by 1990 EUC will absorb a s much as 90% of the total computing resources in organization s appears to be coming true . This growth has been fostered b y the proliferation of microcomputers, the availability o f powerful, user-friendly software, and by the move towar d distributed information processing . EUC continues to be a n important issue for managers of tomorrow's organizations . The increase in EUC literature provides evidence of this trend . The purpose of this study is to examine the quality dimension of end user-developed applications . We will first define the quality construct and suggest one approach for how t o measure it . We will draw on some recent research to identif y factors that contribute to the quality of end user-develope d applications . Our approach posits that the quality dimension i s very closely related to two other factors : the success and the effectiveness of a specific application . The Random House Dictionary (1972) defines fou r important terms for this research as follows : Success—The favorable or prosperous termination of attempt s or endeavors. Effectiveness—Adequate to accomplish a purpose ; producin g the intended or expected result . Quality—High grade; superiority ; excellence . An accomplishment or attainment . Quality Control—A system for verifying and maintaining a desired level of quality in a product or process by carefu l planning, use of proper equipment, continued inspection, an d corrective action as required . End user computing occurs when a computer user relies o n information technology to develop specialized applications fo r their own personal use, departmental use, or in some cases
organization-wide use . End users are typically non-dat a processing professionals . The benefits of EUC are well documented . These benefit s accrue not only to the end users themselves ; they also accrue t o the information systems group and to the organization . Tabl e 1 depicts several of the benefits of EUC that have been cite d recently in the literature . While reducing development lea d time, the major advantages of EUC are that the problem s associated with eliciting information requirements are shifte d to the insiders and at the same time ownership is immediatel y transferred to the users . Having users develop their ow n applications eliminates the problems associated with ineffective communications between the systems analyst and the en d users and results in systems that are used . To End Users • Increased decision making effectivenes s • Improved user computer literac y • Increased satisfaction with end user-developed application s • Faster response to information request s • Improved relationships with the IS staf f To IS Staff and IS Departmen t • A reduction in the backlog of IS application developmen t projects • A decrease in the proportion of IS resources spen t on application maintenance and programmin g • Improved programmer job satisfactio n • Better use of limited resources • Improved user relation s To Managemen t • Fewer user/IS conflicts • More satisfied end users and IS staf f • Direct control over departmental information, applications , and the environment in genera l • Increased end user productivit y
Table 1. The Benefits of EUC in Organization s Eliminating the systems analyst's role as the central figur e in the analysis and design process, however, carries with it a number of dangers . Davis (1982) and Amoroso, McFadden an d White (1990), discussed several dangers inherent in th e development of end user applications. First, the elimination o f the separate user and analyst roles may result in the neglect o f training, documentation, or maintenance of user-develope d applications . Second, the user may not have the ability t o correctly identify complete information requirements . Moreover, Alavi and Weiss (1986) reported that end users ma y inadvertently apply the wrong analysis technique to a situatio n or, in some cases, even attempt to solve the wrong problem .
DATA BASE
Winter
-92 1
Many researchers have noted frequent errors in end user developed applications, specifically : 1) mistakes in logic, 2 ) unreliable output, 3) unauditable applications, 4) inability t o change or modify applications, and 5) an overall lack o f comprehensibility . One of the most serious risks is embodie d in the nature of the developers : EUC developers generally lac k computer training, especially in the area of systems development techniques . Rockart and Flannery (1983) and Quillard , Rockart, Wilde, Vernon, and Mock (1983) independentl y reported that 60% of end user-developers are nontechnica l personnel, using the computer primarily as a tool to solve a problem or perform a task . Since those reports, the penetration of end user computing into the organization has increased substantially and so has the potential for poor quality applications . Despite these risks, however, user-developed application s have become the norm, accounting for 50-80% of the information technology budgets in many firms (Amoroso, 1990 ; Brancheau and Wetherbe, 1990) . Clearly, end users believe that the benefits exceed the risks otherwise they would no t engage in application development activities . The objective o f practitioners and researchers alike is to design and develo p procedures that retain the benefits and reduce the risks of EUC . In other words, quality control should be of major importanc e in the development of end user applications . We contend that quality control can be addressed b y developing and using instruments which are reliable and teste d for validity . Our premise is that there may be user and organizational variables that profoundly impact the quality of en d user-developed applications . Those variables, when properl y controlled, may lead to increased benefits and lower risks . There is a scarcity of literature that empirically examine s the quality issue of end user applications within the corporat e environment . Goodhue (1985) and Rivard and Huff (1985 ) stated that effectiveness and success are essentially the sam e construct when applied to end user computing . In a previou s article (1991), the authors proposed and tested a causal mode l which includes end user information satisfaction and application utilization as surrogates of application effectiveness . These have traditionally been used independently as measures o f information systems success and, in some cases, effectiveness . This article explores the quality construct as a combinatio n of information satisfaction and applications usage . In thi s context we define quality as the degree to which an applicatio n of high grade accomplishes or attains its goal from the perspective of the end user . We believe that end user informatio n satisfaction addresses the high grade issue, while applicatio n utilization deals with the attainment of goals . Research ha s shown that superiority and high grade issues, with respect to end user-developed applications, are mainly perceptual i n nature (Doll and Torkzadeh, 1991 ; Glorfield, Luster an d Cronan, 1991) . The research stream in end user informatio n satisfaction is useful in understanding this aspect of applicatio n quality . Measuring the usage of an application provide s evidence of its ability to accomplish a goal . This article will discuss the EUC environment in order to gain a perspective on those who will evaluate the quality o f end user-developed applications . We will then conside r previous research in the development of information satisfaction and application utilization instruments . 2
Winter
-92 DATA BASE
THE NATURE OF END USER S Early in the EUC movement, Rockart and Flannery (1983 ) took a very broad view of end user computing when they introduced six distinct classes . These end users differed significantly in terms of computer skills, method of computer use , application focus, and the amount of support needed an d obtained . • Nonprogramming end users access computerized dat a through a limited, menu-driven application program, usuall y provided by other s • Command-level end users are able to specify, access, an d manipulate data in order to generate unique reports . • End user programmers utilize both command and procedural languages directly for their own personal informatio n needs . • Functional support personnel support other end users an d themselves in the development of applications . • End user computing support personnel and D P programmers, fluent in end user languages, aid other en d users in the development process . Three additional studies utilized the Rockart and Flanner y taxonomy as a means of classifying respondents . Quillard, e t al ., (1983) added levels of programming and technical understanding to their list . In a Brancheau and Wetherbe (1985) fiel d study, end users were self-classified based upon the above use r descriptions . Experience was regarded as an important use r characteristic . Sumner and Klepper (1987) investigated use r applications with end users in the command level, end use r programmer, and functional support categories . The primary user characteristics they examined were the degree of use r involvement in application development, training, an d development of end users, and the nature of application s developed . In five studies conducted in the 1980s, less than 10% of th e end users surveyed fell into the nonprogramming end use r category . The Rockart and Flannery and Sumner and Kleppe r studies indicated that a large group of end users were found i n the functional support category . Quillard, et al ., reported onl y 20% in the functional support group, while 71% were in th e command level and end user programmer categories . Both Quillard, et al ., (1983) and Amoroso and Chene y (1987, 1991) suggest that the growth of EUC would com e from the command level end user and end user programme r categories . They found that 71% of the end users fell into thes e two categories . Davis (1990) identified the autonomous use r (mapping to the command level end user and the end use r programmer categories) as the fastest growing group of en d users in the 1990s . Perhaps the Sumner and Klepper data was skewed toward the functional support group, as they wer e investigating information systems strategy in their research . One cannot, however, deny the growth that will take place i n the functional support group over the next decade . It appears from these studies that the nonprogramming en d user does not reflect a high growth category of end user . En d user categories strongly reflect the development dimension . Each of the studies reported a random selection of end users i n the organizations chosen, so one might argue that the studie s represented in Table 2 show evidence of end users developin g
Table 2. Composite of End User Studies Rockart & Elatta.e,r..y_ 9% 22% 30% 53 % 7% 15% 140
%~es~ff End_Users Nonprogramming end user Commatad level end user End user programmers Functional support EUC support DP programmers Reported Sample Size
Quillard et al. 1% 35% 36% 20% 8% 0% 83
a p p l i c a t i o n s , r a t h e r than s i m p l y using p r e d e v e l o p e d applications.
Braneheau & ___~.¢,lll.e,r.llf,_ 4% 26% 41% 29 % 0% 0% 53
Sumner & ~ 0% 26% 13% 61% 0% 0% 31
Amoroso & ._C..ll.edl.e,y__ 3% 41% 34% 16% 4% 0% 260
Using information satisfaction and more recently end user computing satisfaction as a sun'ogate measure of success has been a continuing subject of discussion atld disagreement in the MIS literature. To a lesser extent, system usage, or in some cases information utilization, as a measure of the success of a given application system has also been hotly debated. One additional issue concerns the relationship between satisfaction and usage. One might logically argue that increased satisfaction with a system will lead to increased usage of that system. An equally convincing argument, however, is thai the more a person uses a system, the more they may like it and become comfortable and satisfied with il. We must conclude that lhese constructs should not be considered independently.
ing satisfaction is potentially both a dependent variable (when the focus of the research interest is upstream activities or factors that cause end user computing satisfaction) or an independent variable (when the domain of one's research interest is downstream behaviors affected by end user satisfaction). Most MIS research thus fat" has been upstream in nature with user information satisfaction used as a measure of systems success to evaluate vatious design at~d implementation activities. Glorfield, et al., (1991) compared the Ives, Olson and Baroudi (IOB) and D o l l / T o r k z a d e h instruments of user information satisfaction using both personal computing and mainframe applictions. They found that the two instruments measured different aspects of user information satisfaction with the exception of IOB's information product component. This finding is consistent with our contention that the user information satisfaction construct, particularly lhe information product component, focuses on the superiority and high grade c o m p o n e n t of quality. T h e r e f o r e , we c o n v e r g e d on the iuformation product using a modified Ives, Olson and Baroudi instrument for the end user computing envffonment.
END USER INFORMATION SATISFACTION
APPLICATION UTILIZATION
Bailey and Pearson (1983) defined user information satisfaction as a multidimensional attitude of the user toward different aspects of an information system. Ives, Olson and Baroudi (1983) and Iivari (1987) described user information satisfaction as the perceived effectiveness of an information system. Treacy (1985) concluded that: "Starting with the factors discovered by Ives, Olson and B a r o u d i - - a more precise, unambiguous, and complete causal model of UIS should be developed. This model would result in a diagnostic model of UIS that could have important implications for the management of end user computing." Doll and Torkzadeh (1988, 1991) argue that end user computing satisfaction (EUCS) is an important theoretical construct because of its potential for helping us discover both forward and backward links in a causal chain. Figure 1 depicts their "'system of value" chain. In their view, end user comput-
Utilization has been studied by a munber of researchers in the past two decades. Why do managers seek so diligently for a good measure of utilization? One reason involves the need to justify expenditures on information technologies which end users continue to demand. Another reason is the rapid introduction of emerging technologies in corporations every year. Also, managers have found a lack of consistency in present utilization definitions mad measures. Unlike user inforlnation satisfaction, standard utilization measures are still not present today. Srinivasan (1985) defined system utilization as a behavioral measure and states, "If the user exhibited increased evidence of system use in situations where use was not mandatory, then he must find the system useful." Ives and Olson (1984) described system utilization as a useful measure of user acceptance validating the belief that usage by end users describes the application as attaining its development goal.
THE
-,~
QUALITY
CONSTRUCTS
upstream Causal factors
EUCS Beliefs
Figure 1.
Attitudes ~
~,~ downstream Performance ~ Related Behaviors (e.g., use)
~-~----tSocial and Economic Impact
System to Value Chain DATA BASE
Winter
92
3
and application utilization . Melone (1990) commented on th e fact that performance-related operationalizations woul d enhance the value of the system usage construct . These are measures that consider the integrated context in which work i s actually accomplished and the information is actively used . We agree that additional performance-related operationalization s are needed, but we also recognize that such measures will probably be application-specific, which will make generalizatio n difficult, if not impossible .
Several problems exist with the previous studies . First , single-item measures of system utilization were employe d where a multi-dimensional construct was being assessed . Researchers often failed to report and/or account for th e variance in the type of application utilized . Differen t definitions of utilization were often employed, depending upo n the process under study . Another problem was the operationalization of the variables in a particular category . Many earl y studies did not report instrument validation or reliabilit y scores . Yet another problem was deciding which aspect o f usage to measure . Researchers often forgot to include tas k characteristics related to type of use in their variable set . The importance of examining intended and actual utilizatio n was suggested by Cheney (1980) in order to make comparisons . The "intended" dimension captures an end user's attitud e toward application usage rather than focusing solely o n behavioral characteristics . Intuitively, Cheney postulated, on e would expect to find that intended measures score higher tha n actual measures on a specific characteristic of use . From a manager' s perspective, combining intended with actual utilization yields some empirical evidence in attempting to justify th e purchase of emerging information technologies based solely o n end user intended usage requests . Actual utilization patterns , cross-referenced with intended utilization patterns, expos e areas of need and recommend areas of organizational support . We believe that there may be some correlation between th e two quality subconstructs of end user information satisfaction
ONE PREVIOUS STUDY O N EFFECTIVENES S A recent study on end user application effectiveness b y Amoroso and Cheney (1991) helps to explain the factors tha t impact end user information satisfaction and applicatio n utilization . The hypotheses that were tested in this research ar e illustrated in the model in Figure 2 . We will describe each o f the hypotheses with their justification for inclusion in th e model below : 1 . The larger the perceived application backlog, th e greater should be the level of end user information satisfaction . Cheney, Mann and Amoroso (1986) proposed that th e probability of end user application success, as measured b y user information satisfaction, should be enhanced when th e
8 End User Information Satisfactio n
3 "Perceive d Helpfulnes s of EUC Polic y
4 Perceived Org . Support for EU C Applic . Dcv . 7
Motivation t o Develo p Applications
5 Perceived Qual . , of EUC Appi .
Dev . Tools
Figure 2 . Model of End User Application Effectivenes s 4
Winter
-92 DATA BASE
9 Applicatio n Utilizatio n
application development backlog is perceived to be large . Martin (1982) stated that users will take development into thei r own hands when faced with an impossible backlog . 2. The degree of previous computer experience an d training should influence end user satisfaction and applicatio n usage . Rivard and Huff (1985) reported that a user's compute r background was a significant variable in explaining why som e viewed a tool as easy to use while others perceived the sam e tool as difficult to use . Kasper and Cerveny (1985) concluded that users with more computer experience developed a significantly greater number of applications . Yaverbaum (1988 ) reported an increase in the internal motivation to use computers as the number of years of computing experience grew . Fuerst and Cheney (1982) found a strong relationship betwee n system usage and the experience of system users . Hackathorn (1987) found training and education to b e strongly associated with the general success of the end use r computing environment . Guimaraes and Gupta (1987) teste d the impact of training on a variety of variables related to personal computing and support services, finding several positiv e relationships, including motivation . F'uerst and Cheney (1982 ) also reported a significant correlation between user trainin g and utilization . 3. The perceived helpfulness of EUC policies shoul d impact the level of end user information satisfaction . Sufficien t policies have not kept pace with the rapid growth of the EU C field . Gerrity and Rockart (1986) suggest a set of integrate d policies, standards, and guidelines to ensure the highest quality technical environment . 4. Perceived organizational support of EUC applicatio n development in the form of hardware, software, data processes, and people, has been cited as a strategy that wil l increase the likelihood of EUC success (Cheney, et al ., 1986) . Jobber and Watts (1986) and Lucas (1975) concluded that th e more positive the perception of organizational support, th e greater the degree of system utilization . 5. The perceived quality of end user applicatio n develop-ment tools can be found to positively impact both th e satisfaction and usage of end user-developed applications . Many authors have stressed the importance of the quality o f user friendly tools for successful EUC environments . Rivar d and Huff (1985) paid particular attention to the correlatio n between the quality of end-user tools and a user's motivatio n to develop new applications . A relationship exists between th e quality of EUC tools and end user satisfaction . We feel tha t there is a similar relationship that exists with applicatio n utilization . 6. The more positive a user's attitude towards end use r development, the greater the levels of both end user information satisfaction and application utilization . Robey (1979) concluded that as user attitudes toward a new system improve, th e likelihood of system success increases . Goodhue (1988 ) reported that users who held realistic expectations or attitude s toward newly implemented systems were more satisfied with the system and used it more than users whose pre-implementation expectations were unrealistic . Rivard and Huff (1985 ) found a correlation between user attitudes and overall use r satisfaction . Baroudi, Olson and Ives (1986) discovered tha t user attitudes toward an information system will influenc e behavior with respect to use of that system and its outputs .
7. The level of a user's motivation to develo p applications should impact both satisfaction and utilization . Motivation is defined as a person's internal force to behave i n a certain way . Zmud (1979) proposed a model for informatio n systems success by examining the individual differences o f system users . Trice and Treacy (1986) presented a model for structuring utilization research and suggested the inclusion o f individual differences . The data in that study were analyzed using Partial Leas t Squares (PLS), a multivariate path analysis statistical technique developed by Wold (1985) . PLS is known as a secon d generation causal modeling technique . Each step of the PL S iterative procedure involves the minimization of some residua l variation with respect to a subset of the parameters . Th e findings from this research indicated that : 1. Motivation to develop applications was positively related to application utilization . It had the strongest path coefficien t in the model (coef . = .59) . 2. Perceived quality of EUC development tools wa s positively related to end user information satisfaction (coef . = .18) . It was not as strongly related to usage (coef . = .15) . 3. Perceived organizational support was indirectly related t o improved end user information satisfaction (coef. = .53) . It also directly impacted satisfaction (coef . = .20) . 4. User attitude towards end user development was als o related to end user satisfaction (coef . = .36) . 5. Past computer experience and training was onl y indirectly related to satisfaction (coef = .17) and usage (coef . = .18) . All of these findings were tested in an effort to validate thi s model as it relates to end user application effectiveness . W e felt that additional research was needed to test the psycho metric properties of the quality instruments and to investigate utilization patterns of end users .
THE CURRENT STUDY ON QUALIT Y Forty organizations agreed to participate in this follow-u p research, out of a total of 74 randomly chosen Fortune 50 0 firms . The sample represented a wide variety of firms wit h average sales of $6 .6 billion and an average number o f employees of approximately 45,000 . End users who develope d at least one application in the past 12 months were identifie d by a corporate contact individual . Random follow-up telephone calls to corporate contacts were made within two weeks o f delivering the instrument to ensure that the respondents had n o problems understanding or intepreting the measuremen t instruments . In addition to completing multiple-item scales fo r end user information satisfaction and application utilization , respondents answered questions regarding corporat e demographics and the seven independent variables in th e causal model (Figure 2) . The unit of analysis for this study is an applicatio n developed by the end user . Table 3 indicates the functiona l areas represented by the end users in our study . Over 50% o f the end user developers in this study developed at least on e application per month . Most end users had a high level o f enthusiasm for developing future applications . DATA BASE
Winter
-92 5
Table 3 . Functional Areas Represented by End Users Functional Area Finance/Investment Accounting/Audit Marketing Customer Service Personnel General Administration/Planning Purchasing Other TOTAL
# of End Users 101 86 44 47 47 45 31 105 50 6
DEVELOPING THE QUALITY INSTRUMENTS The end user information satisfaction instrument wa s derived primarily from the Ives, Olson and Baroudi wor k (1983) . The revised IOB instrument focuses primarily on th e information product component and consists of ten scales (se e Table 4) . The Bailey and Pearson (1983) instrument wa s consulted and used for more precise scale definitions . For example, the volume of output from an application would be described as follows : Volume of output : the amount of information conveyed to you from the application, expressed not only by th e number of reports or outputs but also by the volume o f the output contents . Concise 1 : 2 : 3 :4 : 5 :6 Redundan t Necessary 1 : 2 : 3 :4 : 5 :6 Unnecessary Each of the items on the end user information satisfactio n questionnaire utilized a six-point Likert-type scale . Tw o adjective sets were used for each of the items . One globa l question was asked to correlate individual scales . Overall, how would you evaluate your satisfaction o r dissatisfaction with the application that you hav e developed ? Very dissatisfied 1 :2 :3 :4 :5 :6 Very satisfie d The utilization instrument was derived primarily from the
Prhportion of End User s 20 % 17 % 9% 9% 9% 9% 6% 22 %
work of Bostrom (1978) and Cheney (1980) . Bostrom developed and Cheney refined a set of categories for both intende d and actual utilization which took into account organizationa l task characteristics (see Table 4) . To capture the multi-dimensional nature of intended and actual utilization, two globa l measures of intended and actual utilization were added . On e global question using a five point Likert-type scale wa s worded as follows : Overall, how would you rate your actual use of th e specific application you identified earlier? Rarely 1 :2 :3 :4 :5 Often Overall, how would you rate your intended use of th e specific application you identified earlier ? Rarely 1 :2 :3 :4 :5 Ofte n Face validity of the instruments was tested with a pilo t study of 40 middle-level managers within one firm . Each o f the respondents was given the utilization questionnaire which was followed by structured interviews to assess the meaning o f individual questions . As a result of the pretest results, th e phrasing of several questions was changed and the sequencing , length, and format of the instrument modified. For example , several users in the pilot study expressed the need for definitions to be supplied in the opening pages of the instrument . Also, several categories of Likert-type questions, such as computer experience, requir ed the addition of a neutral response .
Table 4. Instrument Operationalization s End llsey Information Satisfactio n 1. Accuracy of the application's outpu t 2. Timeliness of the application ' s outpu t 3. Precision of the application ' s outpu t 4. Reliability of the application ' s outpu t 5. Currency of the application's outpu t 6. Completeness of the application's outpu t 7. Volume of outpu t 8. Relevancy of the application ' s information 9. Flexibility of the application 10. User confidence in the application
6
Winter
-92 DATA BAS E
Utilization Task Characteristic s 1. Making decision s 2. Looking for trends 3. Plannin g 4. Taking actio n 5. Finding problem s 6. Historical referenc e 7. Budgetin g 8. Staying up-to-date on activitie s 9. Controlling and guiding activitie s 10. Aiding in reporting to supervisors 11. Aiding in increased productivit y 12. Aiding in the cutting of cost s
ASSESSING THE PSYCHOMETRI C PROPERTIES OF THE INSTRUMENT S The psychometric properties of the instruments were teste d for reliability and construct validity . The reliability of a measure refers to its stability over a variety of conditions . I n the statistical context, reliability is the accuracy or precision o f a measuring instrument . It can be assessed across a set o f measures which purport to measure the same theoretica l variable . The most widely used summary statistic fo r estimating internal consistency is the Cronbach alph a coefficient, based upon item intercorrelations . All unique correlations for scales in each of the thre e instruments (intended usage, actual usage, and end use r information satisfaction) were calculated and compared . Table 5 provides the results of our analysis for the three questionnaires . Each of the end user information satisfaction variable s consists of two items derived from the Bailey and Pearso n instrument . The three instruments appear to provide a reasonable level of internally consistent operationalizations o f the theoretic variables with Cronbach alpha values rangin g from .79 to .89 . Item-to-total correlation values were reported in order t o explain the contribution of a specific item to the construct . These correlations were computed by deleting each item one a t a time . The lowest item-to-total correlation was reported wit h the item capturing the timeliness of the application's output . Also, this item did not correlate to any of the other items in th e instrument . Therefore, it was dropped from the composit e measure . We then assessed the instruments ' construct validity . Validity of some instruments depends upon the adequac y with which a specified domain of content can be sampled . Kerlinger (1973) recommends the use of the multitraitmultimethod (MTMM) approach for bringing evidence to bear on convergent and discriminant validities of a composite scale . Ghiselli, Campbell, and Zedeck, (1981 ) define the MTMM technique as a matrix for organizin g correlationa data such that construct validity can be assesse d from the examination of convergent and discriminan t validities . The MTMM approach, in theory, requires two o r more variables and at least two dissimilar measuremen t methods . In practice, similar rather than dissimilar measure s are often administered at a single point in time . Kerlinge r recommends the use of MTMM under these conditions in a n effort to study the validity of newly developed scales : "In marry research situations, it is very difficult or impossible to administer two or more measures of two or more variable s to relatively large samples . Though efforts to study validit y must always be made, research should not be abandoned jus t because the full method is not feasible . " We ran a large correlation matrix for all of the 134 variable s in the study, which includes 44 variables in the three instruments . First, to assess convergent validity, the correlation s between measures of the same theoretical variable should b e different from zero and statistically large enough to encourag e further investigation . Second, to assess discriminant validity, a measure should correlate with all measures of the sam e theoretical variable more highly than it does with any measure of another theoretical variable . The greater the convergent
validity (heteromethod-monotrait correlations) and the smalle r the discriminant validity (heteromethod-heterotrait correlations), the greater the support for the scale's construct validity . To assess convergent validity, the smallest within-variabl e correlation was recorded . Statistical significance was computed using the one-tail t-statistic test in order to adequatel y interpret the correlation values . All of the 3,686 potentia l correlations were statistically different from zero at the 0 .00 1 level of significance and are therefore considered large enoug h to warrant discriminant validity investigation . No negativ e within-variable correlations were observed . The theory behind discriminant validity calls for near-zer o or low correlations between construct variables . Items within a construct should not correlate more highly with measures o f other constructs . A violation occurs for a measure if the lowes t within-variable correlation is lower than the correlatio n between the measure and any other measure of a differen t theoretic variable . Only 154 of the 3,686 comparisons (p = 4 .18%) of within-variable correlations associated higher on a measure other than within the construct variable . Of the 704 correlations between the satisfaction and utilization instruments, only five items (p = .0071) appeared to violate th e guidelines for discriminant validity . Based upon the psychometric assessment, it is our opinio n that the satisfaction and utilization instruments showe d reasonably high reliability and validity .
INVESTIGATING UTILIZATION PATTERN S To further explain the usefulness of the utilization instruments for assessing application quality, we investigate d utilization patterns for intended and actual utilization (Table 6) . Further investigation was made into any differences that migh t exist between the two utilization types on each dimen-sion i n order to facilitate comparisons . Intuitively, one would alway s expect to find intended measures scoring higher than actua l measures on the same variable . One reason for this is tha t managers do not always carry through with plans or intention s with respect to learning new software or developing applications . Statistical significance was computed using the two-tai l t-statistic test in order to adequately interpret the mean responses . For the sample of 506, seven differences in mean response s were statistically significant at the 0 .01 level of confidence , and one was statistically significant at the 0 .05 level . Five tas k characteristics showed reverse patterns from what w e expected ; four of those were statistically significant at the 0 .0 1 level . Actual utilization of end user-developed applications t o aid in reporting to supervisors and aid in increasin g productivity were significantly higher than intended utilizatio n patterns (t-ratio = -7 .8 and -6 .85, respectively) . This may be because of organizational effectiveness and persona l productivity gained from use of the application . It was, how ever, puzzling why end users were not intending to use applications to facilitate taking action in the future (t-ratio = -8 .18) .
DISCUSSIO N Several important implications can be derived from th e empirical findings of this study, which, in turn, may contribut e DATA BASE Winter -92 7
Table 5 . Correlations of Items with Total Score
END-USER INFORMATION SATISFACTION INSTRUMEN T Item to ITEM 1 Tota l Variable Mean's S .D . Correl .
ITEM 2 Mean" S .D .
Accuracy of the application's output Timeliness of the application's output Precision of the application's output Reliability of the application's output Currency of the application's output Completeness of the application's output Volume of output Relevancy of the application ' s information Flexibility of the application User confidence in the application
5 .33 3 .75 5 .32 4 .84 4,53 5 .08 4 .89 5 .22 4 .93 4 .33
0 .80 1 .88 0 .88 1 .01 1 .30 1 .08 1,21 0 .95 0 .98 1 .33
.811 .212 .797 .743 .692 .770 .541 .771 .743 .740
5 .29 5 .17 5 .01 5 .20 5 .25 5 .05 5 .12 4 .98 5 .09 5 .24
OVERALL MEASURE
5 .20
0 .8 1
Variable
INTENDE D Mean' S .D .
Item to Tota l Correl .
ACTUAL Mean* S .D .
Item t o Tota l Correl .
Making decisions Looking for trends Planning Taking action Finding problems Historical reference Budgeting Staying up-to-date on activities Controlling and guiding activities Aiding in reporting to supervisors Aiding in increasing productivity Aiding in the cutting of costs
3 .31 3 .30 3 .42 2 .31 2 .95 3 .40 3 .23 3 .17 3 .37 2 .31 2 .85 3 .32
1 .23 1 .31 132 1 .47 1 .36 1 .48 1 .21 1 .27 1 .38 1 .44 1 .34 1 .43
.754 .814 .707 .764 .763 .676 .757 .809 .707 .755 .756 .693
3 .20 3 .46 3 .37 2 .95 339 2 .63 3 .12 3,35 3 .31 2 .91 3 .35 2 .60
.71 0 .65 2 .74 1 .81 3 .80 0 .69 5 .71 8 .65 9 .74 6 .81 9 .79 5 .68 6
OVERALL MEASURE
3 .08
0 .75
Cronbach Alpha (entire instrument)
0 .8 9 0 .75 0 .90 0 .83 0 .84 1 .10 0 .99 1 .04 0 .99 0 .84
Item to Tota l Correl . .76 6 .890 .83 1 .90 5 .86 5 .750 .87 8 .843 .797 .824
.89
Mean represents 6-point Likcrt scal e
UTILIZATION INSTRUMENTS
Cronbach Alpha Mean represents 5-point Likert scal e
8
Winter
-92 DATA BAS E
.82
3 .1 4
.79
1 .38 1,17 1 .39 1,52 1,45 1 .47 1 .33 1 .18 1 .36 1 .49 1 .14 1 .43
to research in and management of end user computing . First, a statistically reliable and valid instrument which measure s intended and actual utilization patterns, when coupled with en d user information satisfaction, represents a step forward i n assessing quality of end user-developed applications . Second , this research is managerial in nature, addressing th e justification of information technology purchases . Developing good and reliable measures of applicatio n quality has proven quite difficult in the past . Comparison s with other studies can be misleading due to variances i n samples and instruments, especially with respect t o application usage . The lack of validity tests in publishe d studies uncovers a serious problem for researchers attempting to duplicate or test existing models and associate d instruments . Application utilization has been used often an d yet has been inadequate in its efforts to help improve managerial performance through the effective use of informatio n technology by end users . The end user information satisfaction component of qualit y has been investigated in more depth and rigor . Therefore, it is not surprising to procure a reliability coefficient of 89% for th e satisfaction instrument . We focused on the product componen t of the IOB user information satisfaction instrument and we fee l that the modified IOB instrument is useful in end use r computing environments . We are concerned, however, that should managers wish t o assess the superiority of applications developed by differen t categories of end users, the user knowledge and trainin g component will not be adequately captured . Although we di d not empirically compare the Doll and Torkzadeh instrumen t against the modified IOB in the end user computing environment, the IOB may have broader use for applications developed by functional support, EUC support, and DP programming end users . We suggest that future research efforts b e directed at capturing the user knowledge and training component of end user computing satisfaction for a wider variety o f end users and applications . The application utilization instruments in this study
considered two dimensions—task characteristics and intende d versus actual utilization patterns—in order to capture th e multidimensional nature of the utilization construct . We feel that this addressed certain problems with earlier single-ite m measures of usage . The need for a generalized and standardized instrument to measure utilization is crucial to academic s as well as corporate managers . From the qualitative data of thi s study, managers commented on the fact that justification o f end user computing resources is most often based (and ofte n solely based) on intended usage . Historical data comparin g intended with actual utilization patterns may expose areas o f need and recommend areas for organizational support .
CONCLUSIO N The purpose of this study was to examine the qualit y dimension of end user developed applications . We define d quality, in this context, as the degree to which an application o f high grade accomplishes or attains its goal from the perspective of the end user . We addressed quality as a combination of end user information satisfaction (which addresse d the high grade component of quality) with applicatio n utilization (which deals with the attainment of goals) . The en d user information satisfaction instrument was derived primaril y from the work of Ives, Olson and Baroudi, focusing on th e information product component . The two utilization instruments capture task characteristics along the dimensions o f intended and actual usage . The instruments were tested for fac e validity, internal consistency, and construct validity . The satisfaction instrument may prove to be useful in bot h the end user computing environment and in assessing th e superiority of larger information systems providing comparative results for strategic planning efforts . We argue that en d users who are categorically technicians (developing applications for others) may be closer aligned to the IS group than th e general population of knowledge workers in the firm . We d o
Table 6. Statistical Differences Along Task Characteristics Dimensio n Task_Characteristics Making decision s Looking for trend s Plannin g Taking actio n Finding problem s Historical referenc e Budgetin g Staying up-to-date on activitie s Controlling and guiding activitie s Aiding in reporting to supervisor s Aiding in increasing productivit y Aiding in cutting of cost s
Significance levels n=_506
* 0 .0 5
Intended Mea n 3 .3 1 3 .3 0 3,42 2 .3 1 2 .9 5 3 .4 0 3 .2 3 3 .1 7 3 .37 2.3 1 2 .8 5 3 .32
Actual Mean 3 .20 3 .46 3 .37 2 .95 3 .39 2 .63 3 .12 3 .35 3 .31 2 .91 3 .35 2 .60
t-ratio 1 .69 -2 .06x ` 0 .6 9 -8 .18` * -5 .86* * 10 .05* * 1 .6 2 2 .88* * 0 .8 6 -7 .80* * -6 .85* * 9 .60**
** 0 .01
DATA BASE
Winter
-92
9
not advocate the use of two different satisfaction instrument s when one instrument might provide more integrated information for management . The application utilization instrument can address the crossrelated issues of intended versus actual usage with tas k characteristics providing corporate managers with a bette r vehicle with which to assess costly investments in informatio n technologies . Utilization measures can even be further improved upon by examining underlying dimensions of th e construct, going to referent theories for operationalization s and, most important, empirical testing . We feel that a new dimension should be considered by future researchers to bette r explain differences in utilization patterns . User sophistication , the depth to which users use information technology for a variety of tasks, is an important dimension which may help t o explain the extreme variance in utilization patterns reported i n a variety of research studies . From the data in this study w e recommend that managers struggling with the allocation o f corporate resources assess both intended and actual utilizatio n patterns over a time in order to identify areas where the tw o patterns diverge . This will allow managers and researchers alike to get a better handle on problem areas . Finally, we suggest that researchers focus on the comprehensive quality construct for end user-developed applications that may contribute to the success of the organization . I n order to do this, we must be tuned into constructs that are multidimensional in nature and attempt to tackle the proble m of creating standard definitions of quality and instruments that support those definitions .
REFERENCE S Alavi, M . and Weiss, I . "Managing the Risks Associated wit h End-User Computing," Journal of Management Informatio n Systems, Volume 2, Number 3, 1986, pp . 5-20 . Amoroso, D .L . "Understanding the End User : The Key t o Managing End-User Computing, " Proceedings of the 199 0 Information Resources Management Association Nationa l Conference, May 14-16, 1990, Hershey, Pennsylvania . Amoroso, D .L . "Organizational Issues of End-Use r Computing, " Data Base, Volume 19, Numbers 3-4, 1988 , pp . 49-58 . Amoroso, D .L . and Cheney, P .H . "A Report on the State o f End-User Computing in Large North American Insuranc e Firms," Journal of Information Management, Volume 8 , Number 2, 1987, pp . 39-48 . Amoroso, D .L . and Cheney, P .H . "Testing a Causal Model o f End-User Application Effectiveness," Journal of Management Information Systems, Volume 8, Number 1 , 1991, pp . 63-89 . Amoroso, D .L ., McFadden, F ., and Britton White, K . "Distributing Realities Concerning Data Policies i n Organizations, " Information Resources Managemen t Journal, Volume 3, Number 2, (year), pp . 18-27 . Bailey, J .E . and Pearson, S .W . " Development of a Tool fo r Measuring and Analyzing Computer User Satisfaction, " Management Science, Volume 29, Number 5, May 1983 , pp . 530-545 . 10 Winter -92 DATA BASE
Baroudi, J., Olson, M ., and Ives ., B . "An Empirical Study o f the Impact of User Involvement on System Usage an d Information Satisfaction ." Communications of the ACM , Volume 29, Number 3, 1986, pp . 232-238 . Benjamin, R .I . "Information Technology in the 1990s : A Lon g Range Plannin g Scenario, " MIS Quarterly, Volume 6 , Number 2, June 1982, pp . 11-31 . Bostrom, R . Conflict Handling and Power in the Redesig n Process : A Field Investigation of the Relationship Betwee n MIS Users and System Maintenance Personnel, unpublishe d Ph .D . dissertation, University of Minnesota, 1978 . Brancheau, J .C ., Vogel, D ., and Wetherbe, J .C . "A n Investigation of the Information Center from the User' s Perspective," DATA BASE, Volume 17, Number 1, Spring , 1985, pp . 4-16 . Brancheau, J .C . and Wetherbe, J .C . "The Adoption o f Spreadsheet Software : Testing Innovation Diffusion Theor y in the Context of End-User Computing, " Informatio n Systems Research, Volume 1, Number 2, 1990, pp . 1150 143 . Cheney, P .H . "Measuring the Success of MIS Developmen t Projects : A Behavioral Approach," Working Paper #1, Iow a State University, 1980 , Cheney, P .H ., Mann, R .I ., and Amoroso, D .L . "Organizational Factors Affecting the Success of End-User Computing, " Journal of Management Information Systems, Volume 3 , Number 1, 1986, pp . 65-80 . Davis, G .B . "Caution : User Developed Systems Can B e Dangerous to Your Organization," MISRC Working Pape r #82-04, University of Minnesota, 1982 . Doll, W . J . and Torkzadeh, G . "The Measurement of End-Use r Computing Satisfaction," MIS Quarterly, Volume 12 , Number 2, June 1988, pp . 259-274 . Doll, W . J . and Torkzadeh, G . "The Measurement of End-Use r Computing Satisfaction : Theoretical and Methodologica l Issues, " MIS Quarterly, Volume 15, Number 1, Marc h 1991, pp . 5-10 . Fuerst, W . and Cheney, P .H . "Factors Affecting the Perceive d Utilization of Computer-Based Decision Support System s in the Oil Industry," Decision Sciences, Volume 13 , Number 4, October 1982, pp . 554-569 . Gerrity, T . and Rockart, J . "Managing End User Computing i n the Information Era," Sloan Management Review, Volume 27, Number 4, Summer 1986, pp . 25-34 . Glorfield, K .D ., Luster, P .L . and Cronan, T .P . "Use r Information Satisfaction : A Comparison of the Ives, Olso n Baroudi, and Doll/Torkzadeh Measures," Proceedings of the National Decision Sciences Institute, Miami, Florida , November 1991, pp 658-663 . Goodhue, D . "IS Attitudes : Toward Theoretical and Definitio n Clarity," Proceedings of the 7th International Conference on Information Systems, San Diego, California, December 1985, pp . 181-194 , Guimaraes, T . and Gupta, A . "Personal Computing an d Support Services," Omega, Volume 15, Number 6, 1987 , pp . 467-475 . Hackathorn, R . "End-User Computing by Top Executives, " Data Base, Volume 19, Number 1, Fall/Winter 1987/88, pp . 1-9 . Iivari, J . "User Information Satisfaction (UIS) Reconsidered :
An Information System as the Antecedent of UIS, "
Measuring the Unmeasurable, Nijkamp, P ., Leitner, P ., an d
Proceedings of the 8th International Conference o n Information Systems, Pittsburgh, Pennsylvania, Decembe r
Wrigley, N . (eds .), Boston : Martinus Nijhoff, 1985 . Yaverbaum, G . "Critical Factors in the User Environment : A n Experimental Study of Users, Organizations, and Tasks, " MIS Quar terly, Volume 12, Number 1, March 1988, pp . 75 88 . Zmud, R . " Individual Differences and MIS Success : A Review of the Empirical Literature," Management Science, Volum e 25, Number 10, October 1979, pp . 966-979 .
1987, pp . 57-73 . Ives, B . and Olson, M . "User Involvement and MIS Success : A Review of Research," Management Science, Volume 30 , Number 5, 1984, pp . 586-603 . Ives, B ., Olson, M ., and Baroudi, J . "The Measurement of Use r Information Satisfaction," Communications of the ACM , Volume 26, Number 10, October 1983, pp . 785-793 . Jobber, D . and Watts, M . "Behavioral Aspects of Marketin g Information Systems," Omega, volume 14, Number 1 , 1986, pp . 69-79 . Kasper, G . and Cerveney, R . "A Laboratory Study of Use r Characteristics and Decision-Making Performance in End User Computing," Information and Management, Volum e 9, Number 2, 1985, pp . 87-96 . Kerlinger, F.N . Foundations of Behavioral Research, 2nd Ed . , New York : Hold, Rinehard, and Winston, Inc ., 1973 . Lucas, H . "Performance and the Use of an Informatio n System, " Management Science, Volume 21, Number 8 , April 1975, pp. 908-919 . Martin, J . Application Development Without Programmers , Englewood Cliffs : Prentice-Hall, 1982 . Melone, N .P . "A Theoretical Assessment of the User Satisfaction Construct in Information Systems Research, " Management Science, Volume 36, Number 1, January 1990 , pp . 76-91 , Quillard, J ., Rockart, J ., Wilde, E ., Vernon, M ., and Mock, G . "A Study of the Corporate Use of Personal Computers, " CISR-WP-109 working paper, Massachusetts Institute o f Technology, Massachusetts, 1983 . Random House Unabridged Dictiona ry, 2nd Ed ., New York : Random House, 1972 . Rivard, S . and Huff, S . "An Empirical Study of Users a s Application Developers," Information and Management , Volume 8, Number 2, February 1985, pp . 89-102 . Robey, D . "User Attitudes and Management Informatio n Systems Use," Academy of Management Journal, Volum e 22, Number 3, 1979, pp . 527-538 . Rockart, J .F . and Flannery, L .S . "The Management of En d User Computing," Communications of the ACM, Volum e 26, Number 10, October 1983, pp . 776-784 . Srinivasan, A . "Alternative Measures of System Effectiveness : Associations and Implications," MIS Quarterly, Volume 9 , Number 3, September 1985, pp, 243-253 . Sumner, M . and Klepper, R . "The Nature and Scope of User Developed Applications," Proceedings of the America n Institute for Decision Sciences, Las Vegas, Nevada , November 1985, 300-311 . Treacy, M . "An Empirical Examination of a Causal Model o f User Information Satisfaction," Proceedings of the 6t h
Donald L . Amoroso is assistant professor of informatio n systems at the University of Colorado at Colorado Springs . H e holds a Ph .D . in MIS from the University of Georgia and ha s ten years experience in the informatiomn systems field with a wide range of technical, managerial, and consultative position s including the Bureau of Land Management and Canada Post . Dr . Amoroso has published in leading information system s journals, such as the Journal of Management Informatio n Systems, Information and Management, Data Base, an d Information Resource Management Journal . His curren t research is on measuring the impact of emerging technologies , specifically in the areas of information engineering an d creativity in systems design . He is authoring a textbook wit h Mitchell/McGraw Hill entitled Decision Making Using Lotu s 1-2-3 ; Building Quality Applications .
Paul H . Cheney is professor and chair of Informatio n Systems and Decision Sciences at the University of Sout h Florida . He received his doctorate in MIS from the Universit y of Minnesota . Dr . Cheney has published several texts and has authored over 30 scholarly articles in journals such as Decisio n Sciences, Journal of Management Information Systems, MIS Quarterly, and the Academy of Management Journal . He ha s
served with over 100 firms, including Ford and Exxon . Dr . Cheney is internationally known in the areas of offic e automation, end user computing, and implementation management, and is a well-known speaker before professional groups .
International Conference on Information Systems ,
Indianapolis, Indiana, December 1985, pp . 285-297 . Trice, A . and Treacy, M . "An Empirical Examination of a Causal Model of User Information Satisfaction, " Proceedings of the 7th International Conference o n Information Systems, San Diego, California, Decembe r
1986, pp . 227-229 . Wold, H . "Systems Analysis by Partial Least Squares," in DATA BASE
Winter
-92 11
