) Stakeholder groups in the evaluation of medical information systems. In the Proceedings of 7th European Conference on the Evaluation of Information Technology. 28-29 September. Dublin, Ireland. (accepted)
STAKEHOLDER GROUPS IN THE EVALUATION OF MEDICAL INFORMATION SYSTEMS Pekka Turunena,b, Jan Talmonb a Turku Centre for Computer Science/Turku School of Economics and Business Administration P.O. Box 110, FIN-20521 Turku, Finland E-mail:
[email protected] b
Department of Medical Informatics University of Maastricht, P.O. Box 616 6200 MD, Maastricht, The Netherlands Keywords: evaluation, stakeholders, medical information systems Abstract In medical information system evaluation research, often multi-disciplinary by nature, understanding of stakeholder thinking seems to be more vital than in many other areas of system evaluation. However, stakeholder groups are rarely addressed empirically or theoretically in health care related literature. In general in information systems science, four different stakeholder groups are considered: users, managers, developers and outsiders. To test the applicability of this generally used taxonomy in the medical information system field 68 people were interviewed mainly at the Turku University Central Hospital (TUCH). Our results suggest that the general taxonomy can be used, but it is not optimal. A more specific taxonomy illustrating the groups’ different interests and common issues of stakeholder groups are presented in this article. We believe that this knowledge is crucial for evaluators to be able to accomplish assessment projects successfully.
1. Introduction 1.1 Stakeholder thinking According to Näsi (1995, 19) the idea of stakeholders is that ’holders’ who have ’stakes’ interact with a company and thus makes its operation possible. Those groups or individuals can affect or are affected, thus the direction in which stakeholders influence can be one-way or two-way (Freeman 1984). The interest of individuals of one group might not be exactly the same but of the same kind (cf. Seddon et al. 1998). It is also prominent that individuals can sometimes act in different groups or individuals in different groups might have same type of interest (cf. Grover et al. 1996). To some people the idea of stakeholder groups is easier to grasp as an idea of roles. 1.2 Stakeholder thinking in the evaluation of information systems In the evaluation of information systems, values of those who are making judgements always play a role. The existence of values in judging can be illustrated by the following examples from different fields. According to Markus Wolf (1998), who was a long term leader of the East Germany’s foreign intelligence service
1
(HVA), the difference between a terrorist and a freedom fighter depends from which side the observer is (McElvoy 1997, 400). Checkland and Holwell (1998), who are long term IS researchers, have similarly exemplified that the value of an information system as judged by two individuals may be different in the same way as the same persons can either be a terrorist or a freedom fighter. Seddon et. al’s (1998) information system evaluation study concludes also that the degree of success of an information system can be varying from one stakeholder group to another. According to them ”…researchers should always take care to identify, first, the stakeholder in whose interest the evaluation of IS success is being performed...” In general four different stakeholder groups are discerned in information system science (ISS): users, managers, developers and outsiders. This taxonomy is quite widely accepted, although it seems that it is not based on any theoretical framework or rigorous empirical research. The idea of this taxonomy is that because it is working in practice, it is appropriate in a theoretical sense (cf. Friedman 1953). The general taxonomy is used in different areas of information system science like for evaluation of information systems (Hamilton & Chervany 1981; Grover et al. 1996). In some specific areas of ISS, like in information system strategic planning, there are attempts to divide stakeholders into more specific groups (e.g. Ruohonen 1991). It has been suggested that identifying stakeholder groups for the evaluation of medical information systems is even more important than in ‘normal business’ (Enning et al. 1997). One demands from an evaluation study that it is objective and that the properties of the methodsare known. These demands are often related also to scientific work. In health sciences these aspects are even more important. The evaluation, and also the development of medical information systems is often a multidisciplinary research activity where health, medical and information system science researchers meet. Different paradigms or educational backgrounds are influencing the choice of evaluation methods and how the evaluation process is managed. In the medical related literature, there are hardly any papers addressing stakeholder theory. Jeremy Wyatt (1997) is using the general taxonomy for illustrating key stakeholders of the information system. He included patients as an explicit stakeholder group. In 1997-98 two evaluation research projects of health care information systems were accomplished in the Turku University Central Hospital (TUCH). A post-hoc analysis revealed that although a stakeholder approach has value, the general taxonomy is insufficient to cover the peculiarities of the applications of information systems in health care. It was assumed that the general taxonomy should be split into more specific groups according to different paradigms. To test this hypothesis 68 people were interviewed.
2
2. Research method 2.1 Interviews The interviews were held in the TUCH during the evaluation of a laboratory information system (LIS). To link an evaluation of an IS in use with our research was motivated by the fact that evaluation of an IS is not part of the everyday work of people in a hospital. The evaluation of the LIS provided them with the experience that was needed for the understanding of our questions. This is important even if the research objective is to clarify general attitudes or reasons leading to the choice of evaluation methods (e.g. Mäkelä 1990, 64-67). Personal interaction was needed to clarify the rather difficult issues. Practical reasons also supported the decision to use personal interviews as the research methods of choice. The information system is used in different ways and the design of a common questionnaire was deemed difficult because of the differences in background of the various people participating in the study. It is our experience that in Finnish university hospitals physicians seldom answer questionnaire-based surveys. This concerns especially those physicians that use only the system’s printouts. They do not have the feeling they are using system. As to secure that every participant understands the issues addressed, different approaches were used to elicit opinions. 1) Semi-structured questions were asked whether the information systems was successful, unsuccessful, partly or fully, and what were the reasons of those judgements 2) Multiple-choice questionnaires were used to clarify the best evaluation method for the system. 3) Semi-structured questions were used to clarify what were the reasons to choose specific evaluation methods and reject other ones. 4) Semi-structured questions were used to clarify if the use of a control group and statistical analyses would be needed for a credible evaluation. 5) By semi-structured question the interviewees were asked to formulate from which point of view they made their choices. Twelve pilot interviews and 56 real interviews were carried. Except for five pilot interviews all interviewees worked in the TUCH. People with a variety of professions were selected for interviews. Because top managers and developers in the organizations are key groups in the evaluation process, all of them were interviewed. Eleven of them had a medical background, five had a nursing background and five had another, mostly technical, background. Users were randomly selected from three departments. Twelve physicians, 16 nurses and 7 secretaries were interviewed. Response rate was 89,7 %, which is good compared to response rates in busy university hospitals in Finland. 2.2. Limitations of method Case studies are always more or less context dependent. However, the case study method was chosen to get a deeper understanding of the organisation. Evaluation was done without major interaction with contact persons in the organization so that our own academic evaluation question could be pursued without influence from the organization.
3
To avoid context dependency the focus of analyses was not so much on the selected evaluation methods themselves but rather on the reasons, general by nature, to choose or reject an evaluation method. The consequences of failures of the information system were assessed as to understand the respondents’ priorities better. Quantification and abstraction of data was used to make the results less context dependent. A LIS was chosen to make the results less context dependent. All personnel usually use a LIS – or at least its results – contrary to other systems in large hospitals.
3. Results and discussion The nurses and the clinicians often thought about themselves more as nurses and clinicians than as users of an information system. This gives some evidence that the concept of a user group should be handled in a different way than in traditional user group thinking. The main difference between physicians and nurses was their concern about patients (see also Salanterä 1997). 75 % of nurses mentioned patients in one contest or another. 38 % of the nurses prioritised patients very high while only 27 % of the clinicians mentioned patients in one way or another. In the few cases the physicians concern about patients the unique position of the department and part time management work might be explaining factors. One nurse expressed her feelings about evaluation as follows: “Well, it has its advantages (cost-benefit analysis) but I am so bored with this continuing economical evaluation (evaluation of general things) that I am feeling sick that money seems to be the only issue of relevance. Sometimes you should think about what is good for the patient (quality of life) (Nurse I)” Another answered to the question from which perspective she gave her responses: “Well, from my own work and then also from patients perspective, the nurses are as a rule the representatives of the patient (Nurse II)”. Developers selected user satisfaction measurement more than other groups. An answer to the question why the interviewed chose user satisfaction as a relevant measurement was: ”The most important thing is that it (the IS) must serve those users. An that it serves its purpose, that it has to be easy to use by the users in the departments and that they get the information when they need it (Developer with technical background).” Even then, there seems to be a gap between developers and users, since users think about the use of the system in the context of their own work. For the great majority of top managers and developers with health care background the use of a control group was a necessary condition while none of the managers and developers without health care education thought it would be a necessity. Half of the physicians thought that the use of a control group would be necessary. In medicine, formal evaluation with the use of a control group is highly preferred (cf. Showstack et
4
al. 1981). Although this finding is not a big surprise, we now have at least more accurate data concerning opinions about the evaluation of medical information systems. Although use of control group might be very useful, its use in information system evaluation is often difficult. Opinions concerning the use of a control group in the evaluation are very important to notify. “If you have as aim to make a scientific study, I would say that it would be a condition, because otherwise you have nothing to compare the results with. If you evaluate only one system, you only get descriptive data. But if you are comparing it to the other system, then you have results from which you can draw conclusions. (Manager with medical background)” The common belief is that managers are thinking only about economical issues in information system evaluation. Our data shows clearly that managers were considering wide variety of issues: personnel (users) and economical issues as well as patients (see also Grover et al. 1997). In case they were told to choose between either a usability evaluation or an evaluation of impacts some of the managers could not decide which they would pick. It was expected (Wyatt 1997), that so-called objectivist methods would have got more support, but it was not the case. For example only two respondents chose an experiment in a computer classroom for evaluating usability and five for evaluation of effects: “We might get more exact measurements by experiments but it is a little bit far away from a real life situations and it is difficult to formulate the experiment such away that it is as in real life (Physician I).” “In real life you just must always walk to a computer. You can’t sit next to the computer when you are working (with patients in a ward) (Physician II).” It seems that people are so much involved in practical work that they are aware of the limitations of experiments. According to Wyatt (1997) whenever designing evaluation studies, it is important to consider the perspectives of all stakeholders. The most important aspects, that different stakeholder groups wanted, are as follows: •Top managers : A couple of different evaluation measurements should be used. Assessments that help for developing the system from a users' point of view and for example a cost-effectiveness evaluation should be considered. Whatever choice is made, the reliability of the evaluation method in a particular context should be very carefully discussed. Statistical analyses should be used. Only the managers with a health care background are stressing the need for control group. •Developers : An evaluation must concentrate on the users but the evaluation results should be translatable in technical language and solutions. A formal control group is compulsory to satisfy the developers with healthcare education background. •Nurses: An evaluation should focus on nurses as users of system. Patients should be somehow notified in the evaluation. For example the effect using the system has on time spent on patient care could be measured. The evaluation should be quite short and easy for nurses. The reliability of the evaluation should be notified. A control group would be useful.
5
•Physicians: An evaluation should emphasise physicians as users. The reliability of the evaluation has to be taken care of. If possible, used time should be measured. Statistical analyses are very important and a control group would be useful. •Secretaries: An evaluation should be short and using an easy questionnaire. The survey should concern their work (at least the part that is influencing patient care). The use of control group would be helpful. It is important for evaluators to take these aspects into account when evaluation projects are carried out. We are not suggesting that evaluation research should adhere to all these opinions. To know these aspects is helpful for evaluators when the project is planned or the project is negotiated with stakeholder groups. However, if a particular group is financing the evaluation study these aspects should be notified very carefully. Also published evaluation results are more credible to stakeholder groups when these aspects are taken into account in the evaluation. Without credibility, the evaluation results have not much use. We are also not suggesting that evaluation should be done necessarily from one or all groups’ point of view (see Seddon et al. 1998). There are rarely evaluation resources available to satisfy fully all groups (cf. Wyatt 1997 p. 3). Instead the common features for all groups are useful to identify as to secure some degree of satisfaction among all stakeholder groups. According to Friedman’s (with Cornford 1989) analyses the maturity of information system development in organizations is reflected by the main focus of the activities: 1) hardware, 2) software, 3) user, 4) organization 5) interorganizational and customer orientation. Their account seems to be persuasive (Checkland and Holwell 1998, 36; cf. Korpela 1994). In large organizations the speed of development can vary a little bit between departments. Large health care organizations seem still to be in the user oriented information system evaluation and planning phase, but there is in general pressure to move the evaluation of information systems into the direction of organizational aspects. In the research of evaluation methods user evaluation could still be emphasised, but more effort should be put also in the research of cost-benefit methods in this field. Based on previous taxonomies, our previous action studies, the empirical data from the current study and the recognisable professional groups in health care (Pelttari 1997, cf. Salanterä 1997), we suggest that it is useful to divide the stakeholder groups in the evaluation of medical information systems as follows (see Figure 1). The main groups such as users, developers, top managers, others are familiar from the literature. Users have to be split into physicians, nurses and others as to cope with the different perspectives on assessment. While developers and top managers have to split depended on health care and other backgrounds. The taxonomy cannot be formulated like a matrix in which main classes are in rows and the profession in the columns. As an easy to remember guideline for practitioners the textbook main classes could be divided into physician, nurse and other backgrounds, or health care and other backgrounds.
6
4. Conclusion According to our study it seems that assessment of information system is somehow related to user evaluations. One possible further research could be adopting Saarinen’s (1993) multi-stakeholder satisfaction measurement. There seems to be organisational pressure to start to use more organisation directed measurements, which include also the patients' perspective. The research of medical information system methods should be focused more on, among other things, cost-effectiveness analyses.
3)
Users: 1) Physicians, 2) Nurses, 3) Others
1)
2)
4)
5)
Developers: 4) Healthcare background, 5) Others
6)
7)
Top managers: 6) Healthcare background, 7) Others
8)
Figure 1
8) Others:Patients, Outsiders etc.
Stakeholder groups in the evaluation of medical information systems
The results of our study reassure but also improve our working hypothesis. It seems that general stakeholder groups can also be adopted for evaluation of medical information systems. However, at the same time it is obvious that this general taxonomy is not optimal. According to our data and knowledge from professional groups in medicine, it is clear that more specific groups are needed. Our results suggest that users should be separated into physicians, nurses and others. Managers and developers should split based on health care or general management background. The more specific groups are based on the professional and educational background and paradigms. One issue of concern in the evaluation of medical information systems is always the patient-group. The relevance of the patient-group will become more important due to the development of Internet based systems. To accomplish evaluation projects successfully, the aspects of these different groups should be notified by evaluation researchers. Although our classification is more explanatory than the ones used previously, it is worth noting that the classification is still only a general one and should be used as a guideline to identify the precise stakeholders in each specific case. We assume that our classification is accurate
7
enough to cover most of health care field and at the same time simple enough to cover the central nature of the issue (see Remenyi 1996, 102).
Acknowledgements We thank Professor Hannu Salmela to comments for designing the research settings and Professor Arie Hasman for comments on the results. The Academy of Finland and Turku School of Economics and Business Administration foundation are gratefully acknowledged for financing this work.
References Checkland, P and Holwell, S Information, Systems and Information Systems, John Wiley & Sons:1998, Chister, 1998. Enning, J, Fabianne, E, Talmon, J, Vissers, M, Nykänen, P, Roine, R, Hoyer, D, Thayer, C, and Coello E (1997) Guidelines for evaluation of telematics applicatons in medicine, Deliverable D03.2b, 1997 http://www-vatam.unimaas.nl Freeman, R E (1984) Strategic Management: A Stakeholder Approach, Pitman, Boston. Friedman, M (1953) “The Methodology of Positive Economics” In Essays in Positive Economics, Friedman, M (Eds) University of Chicago Press, Chicago. Grover, V, Jeong, S R and Segars, A H (1996) “Information systems effectiveness: The construct space and patterns of application,” Information & Management, 1996:4, 117-191. Hamilton, S and Chervany N L (1981) “Evaluating Information Systems Effectiveness – Part I: Comparing Evaluation Approaches” MIS Quarterly, 1981:3,55-69. Pelttari, P (1997) Sairaanhoitajan työn nykyiset ja tulevaisuuden kvalfikaatiovaatimukset, Stakesin tutkimuksia, Gummeruksen kirjapaino, Jyväskylä, 1997. Mäkelä, K (1990) “Kvalitatiivisen analyysin arviointiperusteet”, in Kvalitatiivisen aineiston analyysi ja tulkinta, Mäkelä, K (Eds) Oy Gaudeamus AB, Helsinki. McElvoy, A (1997) Markus Wolf - Kasvoton mies, Orig. Man witout a face. The Autodiography of Communism´s Greatest Spymaster, WSOY, Juva. Näsi, J (1995) “What is stakeholder thinking? A snapshot of a social theory of the Understanding stakeholder thinking, Näsi, J (Eds), Gummerus Kirjapaino
Remenyi, D (1996) “Outcomes and Benefit Modelling for Information Systems Proceedings of Third European Conference on The Evaluation of
8
Information Technology, 29 November 1996, Brown, A and Remenyi D Bath (Eds), UK. Ruohonen, M (1991) “Stakeholders of strategic information systems planning: theoretical concepts and emprical examples”, Journal of Strategic Information Systems, Vol 1. No 1. Saarinen, T (1993) Success of information systems - Evaluation of Development Projects and the Choice of Procurement and Implementation Strategies, Acta Academiae Oeconomicae Helsingiensis, Dissertation, A:88:1993, Helsinki Salanterä, S (1997) Hoitotieteen tietoalueet – Kimin typollogia, Turun yliopiston täydennyskoulutuskeskus, Painosalama Oy, Turku. Seddon, P B, Staples, D S, Patnayakuni, R and Bowtell, M (1998) “The IS Effectiveness Matrix: The Importance of Stakeholder and System in Measuring IS Success”, Proceedings of the Ninteenth International Conference on Information Systems, Helsinki. Showstack, J A, Schroeder S A, Steinberg, H R. (1981) Evaluating the costs and benefits of a diagnostic technology: the case of upper gastrointestinal endoscopy, Medical Care, 19(5), 498-509 Wyatt, J C (1997) “Basics of Measurement”, In the Evaluation Methods in Medical Informatics, Friedman, C P - Wyatt, J C (Eds), Springer-Verlag:1997, New York.
9
