Towards a method for acceptability analysis: application to healthcare innovation Giovanny Arbelaez-Garces, Eric Bonjour, Auguste Rakotondranaivo Université de Lorraine, ERPI, EA 3767, Nancy, F-54000, France
[email protected] Abstract—the healthcare field, as many others, has the imperative to innovate in order to face its multiple performance and societal problems. Innovations should not only respond to users’ needs, but they also have to meet other technological and contextual factors to be successfully integrated and diffused. Acceptability evaluation should allow designers to anticipate and act accordingly to avoid project risks. However, the current literature lacks of models allowing an earlier awareness of those factors. This paper reports on the development of a method for acceptability evaluation that allows guiding the design of product and services in healthcare. A framework for acceptability evaluation is presented. A home healthcare technology case study is used to illustrate the application of the method.
practices of new products and services, mainly related to healthcare technologies, but that can be expanded to other areas. A home healthcare technology example is presented to illustrate the possible uses.
Keywords— acceptability, decision making, Bayesian networks, product development, home healthcare
A. Acceptability Evaluation Considering the technology adoption process as a temporal continuum, three moments of analysis should be considered: acceptability a priori, acceptance, and appropriation [5]. The first one refers to the subjective representations of technology use, trying to predict technology usage from the perspective of what is “perceived” by individuals, even before the technology is totally developed. In this context, dimensions as “perceived utility” and “perceived ease of use” among others should be taken into account. Acceptance on the other hand, refers to the study of the factors that had an impact on the first interactions between the technology (developed) and the user. Finally, once the system has been deployed, it is convenient to study and assess the “real” appropriation and usage. In this paper, we deal mostly with acceptability given that we are trying to estimate (predict) the potential of a technology to be accepted and adopted through usage. However, we should not forget that we are trying to positively influence adoption from the early stages of the project.
I.
INTRODUCTION
Healthcare systems such as hospitals, medico-social institutions and home care structures are complex production systems of “care” and services. They are attracting the interest as a research field in engineering due to their similarities and specificities compared to industrial systems [1], [2]. Methods and tools from engineering practices have proven to be useful to solve various problems. Faced with performance requirements, innovation in healthcare systems are a major concern for policy makers, industrials, health professionals, patients and their relatives. The design process of these innovations affects their acceptability and success. Hence, the need of tools and methods to effectively support those projects [3]. Innovation projects are associated with risk and uncertainty. Besides responding to users’ needs and requirements, designers should anticipate consumer acceptability of their innovation to avoid engaging projects that will not succeed. Acceptability evaluation is one of the key problems in product development [4]. Although acceptability evaluation models are proposed in the literature, they base their evaluation in factors related to the individual, the technology or the context yet they do not provide a global vision regrouping all those factors. Furthermore, the existing models were not conceived to help guide product/service designers in the development of innovations. Consequently, this paper aims to present a method for acceptability evaluation that can be used to guide design
The paper is organized as follows: in Section II, we present the theories related to technology acceptance and Bayesian networks. Section III, describes the method proposed and the framework used to evaluate the acceptability. Section IV, presents the case example. Finally, in section V, conclusions and perspectives for future research are discussed. II.
RELATED WORK
The literature presents several methods for acceptability evaluation. Some approaches are related with needs assessment as in [6] that propose a methodology for acceptability evaluation based on Kano model. Others are based in the theories of technology acceptance that states that user’s behavioral intention could be used as a measure of acceptability. They derive from Ajzen and Fishbein’s Theory of Reasoned Action [7]. The most influential is the Technology Acceptance Model (TAM) [8]. This theory states that the user’s behavioral intention of a new technology is influenced by the perceived usefulness (PU) and the perceived ease of use (PEOU). PU was defined by the author as "the degree to which a person believes that using a particular system would enhance his or her job performance" and PEOU defined as "the degree to which a person believes that using a
particular system would be free from effort". The TAM model was expanded as TAM2 [9] to include other factors influencing the PU, and factors form the organizational context of the user, in the unified theory of acceptance and use of technology (UTAUT) [10]. While the latter might be the only theory regrouping most of the factors that interest us (individual, technology and context), it is mainly focused on the information technologies’ analysis. Another model that is mainly used in usability studies is the Nielsen’s model [11]. This model distinguishes the acceptability factors of a system in two categories: Social acceptability and Practical acceptability. The former refers to the system’s compliance with social needs, whereas the latter is related to the technical environment of the proposed system (reliability, compatibility) as well as aspects related to its use (utility, usability). This model later evolved as the ISO 9241210:2010 norm.
B. Bayesian networks Bayesian networks are directed acyclic graphs that represent the probabilistic relationships between a group of variables [14]. They are generally composed of two parts: a quantitative part and a qualitative part. The qualitative part is the directed acyclic graph in which the nodes represent the states of random variables and the arcs pointing from a parent node to a child node represents the causal conditional dependency between the two nodes. For the quantitative part, it mainly refers to the relationship between a node and its parents. The relationship can be represented by the probability of the node’s state provided different probabilities of the parent node’s state. The former can be represented by a conditional probability table (CPT) of the node. Fig. 1 shows a simple network comprising four nodes. Each node represents a binary variable that takes the value
However, these theories place the technology as external to the individual and place the individual as the only actor to judge if he will either accept or reject the technology. According to Brangier and others, this paradigm should be outdated because the technology and the individual are mutually influenced [12]. They propose the human-technology symbiosis model. In order to operationalize the concept, a scale was developed and validated. It rests upon three criteria for achieving maximum compatibility between a technical system and a human activity: • Features: symbiosis in this field assumes that the features offered by the system (technology) are consistent with what the user wants to achieve to perform a given task in its socioorganizational environment. • Usability: refers to the perceived ease of use of technology. • The forms of control: related to socio-organizational behaviors (e.g. ownership, rejection, resistance, social innovation) built by the human in relation to its social and organizational environment transformed by technology. All these elements are interrelated. The more interconnection there is the more the technology is evaluated as symbiotic (acceptable). As previously stated, the models proposed in the literature lack of a global vision of affecting factors that could be used to guide a product/service design project. Those models are mostly used as measure mechanisms once the technology has been deployed, but they are not designed to be used since the very beginning of a design project as an estimation of acceptance that helps guide design choices and practices. Artificial intelligence techniques have been used to evaluate and predict acceptability [4], [13]. When compared to other techniques, Bayesian networks allows integrating different sources of information (experts knowledge, and data), allows handling uncertainty and incomplete data. However they are difficult to maintain. The use of a Bayesian network was retained for use in the proposed method.
Fig. 1. Example of a Bayesian network
true (T) or false (F). In this example, the node X2 is the parent of nodes X3 and X4, and the node X1 is the parent of node X3. Conversely, the node X4 is a child of the node X2, and node X3 is a child of the nodes X1 and X2. The arc coming from the node X2 to X4 shows that the condition in node X2 affects that one of node X4. Detailed relationships are presented in the CPT attached to node X4, it shows how and with what level of influence the condition in node X2 affects the node X4. Similarly, it is possible to obtain the dependency relationships between the nodes X1, X2, and X3. Two types of reasoning are possible with Bayesian networks: (1) diagnosis (backwards inference) which allows, given an observation to find the most probable cause among the hypotheses, and (2) prediction (top-down inference), which allows to estimate the probability of an observation given the assumptions [15]. The hypothesis space is denoted Θ={θ1,…,θi,…} and the observation space Ω={ω1,…,ωk,…}. To represent the link between observations and assumptions, we can use a conditional probability distribution. This probability distribution is denoted PΩ(.|θi ). When a source of information provides an opinion concerning the values of the assumptions, this is represented as a conditional probability distribution on the set of
observations Ω, denoted P(ωk|θi), which characterizes the likelihood of the observation ωk knowing the hypothesis θi. Bayes' theorem can then be used to perform a backward inference reasoning to determine the most likely cause of this observation by calculating the a posteriori probability distribution PΘ(.|ωk): (1) When the probability distribution P(.|θi) is known and assumption P(θi) is issued, the theorem allows implementing a top-down reasoning inference to estimate the effect of a hypothesis on the observations P(ωk): (2) The properties we have presented on Bayesian networks allow us to: calculate and predict the degree of acceptability of a technology, and to simulate different scenarios of action. The last, with the aim to identify the most appropriate actions to put in practice in order to reduce the risk of project failure. Indeed, Bayesian networks have shown to be successfully used in risk management [16], [17]. III.
OVERVIEW OF A METHOD FOR ACCEPTABILITY ANALISYS
We propose a method for acceptability evaluation that can be used to guide design practices of new products and services in healthcare. The method consists of four steps: (A) Adapt the framework for acceptability evaluation for the specific case, (B) build a Bayesian network to handle the case knowledge and (C) use the network properties to estimate (diagnose) acceptability values and simulate different scenarios, (D) according to the scenario simulation, choose the best actions to continue guiding the design process. A. Framework for acceptability evaluation Based on the theories of acceptability and symbiosis, we build an evaluation grid with six macro-criteria to enable a more robust assessment of acceptability while taking into account individual, technological and social aspects. TABLE I.
Artifact User Context
MACRO-CRITERIA FOR ACCEPTABILITY EVALUATION
Functionality C1 Need and requirements C2 Perceived utility C3 Professional relevance
Usability C4 Factors related to ease of use C5 Perceived ease of use C6 Facilitating conditions
We propose to evaluate acceptability from three points of view: the artifact, the user and the organization (or context). The interest to combine these three assessment aspects is to exceed the individual vision of acceptability (concerning only the user) that we have seen in the literature but to have a global vision also containing the contexts of use. Indeed, this allows us to analyze the maturity level of the designer’s proposal. Thereafter, we can check whether the proposal is consistent with the needs and perceptions of future users. Finally, we can evaluate the context in which the proposal will
Fig. 2. Network example with actors involved per question
be implemented to identify possible interference to other stakeholders or incompatibilities. Our aim is not only to give an estimator of the acceptability of an innovative project (e.g. very low, low, high, very high) to assist in the “Go / No Go” decision, but to provide designers with a tool to: (1) identify the weaknesses and potential sources of conflict hindering the introduction of the innovation, (2) simulate scenarios of action and in this way provide recommendations to solve them. The criteria in the category “Artifact” concern those responsible for translating design requirements into a solution, i.e. designers and anyone related to project management (Fig. 2). The criteria in the “User” category, completed by expert’s opinion, are used to collect the perception of main users of the artifact (e.g. a doctor in the case of a tool for medical diagnosis). Finally, the criteria in the “context” category help to assess the opinions and perceptions of other stakeholders and actors in the context (e.g. other doctors, patients, nursing staff, hospital manager). This allows us to gather the acceptability of each involved actor. To operationalize the grid, we looked for specific questions to be evaluated for each macro-criterion from the literature and our field experiences. For example, for the macro-criteria “Perceived usefulness”, the following criteria are proposed:
Reduce time in activities Quality and safety of care for patients Better diagnosis, better treatment, better monitoring Reduced errors Fewer adverse effects ... Among others, these could be completed for each specific product or service evaluated. A Likert scale can be used for facilitate the responses of each question, and then normalize to a scale shared with all the other criteria (for example: Low, Medium and High). When other data sources are available, take for example “reduce time in activities” and data about time used after and before the technology, normalization should be used to correspond to the other criteria. Data gathered is then treated using a Bayesian network.
B. Build the Bayesian network Once the criteria was defined, we follow the approach proposed by [18] to build the Bayesian network: 1.
Identify the variables and their conditional space
2.
Define the structure of the network
3.
Define the joint probability distribution law of the variables
In step one and two; we modeled all the factors affecting the acceptability of a new technology innovation in healthcare. This was done with a literature review that leads to the framework presented in the previous section. This model was updated to reflect the knowledge from a group of experts of the field. For step three, once the network modeled, the joint probability distribution laws are built with the elicitation of the experts’ knowledge and experiences using a simple rough estimation as described in [19]. As this first prototype evolves other techniques to refine the prior probabilities might be considered [20]–[22]. For steps C and D of the proposed method they will be illustrated in the example. IV.
CASE STUDY: A HOME HEALTCARE DEVICE DESIGN
The HADAN is the Home Healthcare network of Nancy’s agglomeration (France), it is a non-for profit association in charge of coordinating in-home healthcare and in-home hospital care for patients linking several structures (e.g. hospitals, physicians, nurses, and other healthcare professionals). In 2008 an innovative communicating pen was introduced to the structure [23]. The pen was designed to enable nurse (s) and physicians to instantly send a set of acts performed in a patient’s home and thus improve care practices (traceability, reducing time, coordination, information sharing ...). Although the utility of the communicating pen was evident its acceptance by the users and the organization was not straightforward. An analysis from this experience and evaluation of user acceptance (using TAM model) was later conducted to redesign a technological device that suits better the needs of the users and the organization [24]. The data collected from this analysis was used to test and illustrate the following use case scenario of the proposed method for acceptability evaluation. The conceptual design proposed [24] is a tablet-like device with an interface regrouping all the formularies needed by practitioners, information is transmitted in real-time to a central server and shared between all involved actors. This solution more technologically friendly requires a big shift in practices as paper formularies are no longer required; all information has to be submitted through the tablet application. We start the acceptability evaluation with a concept solution at this point the project manager should decide whether to continue to product development or to stop the project. The process starts by gathering all the data (information) required to assess each criterion, i.e. usability test, ergonomic data, perception data, interviews with the different actors, project budget, among others. For the case
Fig. 3. Acceptability evaluation with data available
example, data were gathered through a survey from users of the previous project (communicating pen) using a five point Liker scale over each criteria that was latter transformed into a three level scale. Fig. 3 shows the Bayesian network used for the example already fed. The six macro-criteria are in the center, as we stated earlier they can be decomposed into several levels to have observation points with data that are easy to gather. In the example, we only used six observation points (in addition to the six macro criteria) to simplify the example; one for “factors related to ease of use” (ergonomics), two for “professional relevance” (organizational compatibility and required skills) and three for “needs and requirements” (financial requirements, need specification and technical feasibility). The data are populated through the network using the inference properties. In the example, given the data collected, “acceptability” is estimated with a probability of 51% High, 42% Medium and 6% Low. This implies that there are aspects of the project that can lead to risk or failures if they are not properly controlled. The managers should prefer an “acceptability” level with probability closer to High, they might even set up a tolerance level to decide to pursue a project for example, High greater or equal to 70% and Low lower or equal to 5%. The next step consists in simulating actions that can be executed to improve the observation point’s values and how they reflect in the acceptability of the project. One of the advantages of the method is that it allows having a global vision of all the influence factors and their states. In Fig. 3, “required skills” i.e. the skills required by the team to successfully develop the product (in this case, the programming skills to develop the application) have a probability with a tendency to Medium, Low values, and this was mostly due to the lack of coding skills of the team that proposed the concept tablet solution. The team can use the tool to simulate what would be the impact of hiring or externalizing a programmer to develop the application leading the “required skills” variable to a High probability. The same can be done with other values; “perceived ease of use” can be improved to a High level by developing training programs with the users or integrating them in the development process.
currently testing this method through a project related to the development of a product for eldercare. Our purpose with the method and the framework is not for managers and designers to use it as a measure at the end of a development process or just once at the beginning of the design process, but a tool that could help them to master the complexity of factors by identifying those that have a greater impact over final acceptability. We emphasize that innovation success not only relies in technological or individual factors, but is a combination of all, thus actions to help the innovation to be spread among the organization (context) are important, as Berwick states “If innovation is hard dissemination is even harder” [25]. ACKNOWLEDGMENT Fig. 4. Networtk with the new scenario simulation
Fig. 4 shows the network with the new simulation values. The values propagated through the network allow to quickly assessing the impact of decisions. In the example, the “acceptability” reaches an expected level Low below 5% but the level of probability for High is still below 70%, the group should decide either to pursuit as it is or to continue looking for actions that could lead to a higher level of confidence. Once the target “acceptability” value is reached in the simulations, the next step consists in developing the project and executing those actions that were chosen in the simulation. The evaluation method was conceived to also serve as a tracking and monitoring tool all along the development cycle, so once one action is executed manager can feed the network with the new data gathered and evaluate the new acceptability level. V.
The model described in this paper was created using the GeNIe modeling environment developed by the Decision Systems Laboratory of the University of Pittsburgh and available at http://genie.sis.pitt.edu/. REFERENCES [1]
[2]
[3]
[4]
CONCLUSION
In this paper we have presented a method for acceptability evaluation. A usage example guiding the design process of a home healthcare product has illustrated the interest of the proposed method. The framework used for acceptability evaluation was conceived to regroup factors related to the individual, the technological product and the context of use. The tool exploits the properties of a Bayesian network for knowledge modeling and inference. Since the method described here builds upon (and requires) knowledge elicitation from experts, and experts inconsistency is one of the major drawbacks of this technique a more robust elicitation method should be tested [22]. Other improvements might be proposed: First, a sensitive analysis might be conducted. Second, the elicitation process should facilitate data integration into the Bayesian network. Third, for the scenario simulation, a tool can be proposed, one where the Bayesian network should be able to suggest actions to improve the “acceptability” value to a certain target. Our current research intends to enhance this point, by modeling expert’s knowledge using Influence diagrams. Finally, as this paper reports on our first experiments, questions about usability and effectiveness of the method arise and our next step would be to test over more cases to evaluate those aspects. We are
[5]
[6]
[7] [8]
[9]
[10]
[11] [12]
[13]
A. Rakotondranaivo, J.-P. Grandhaye, and F. Guillemin, “Functional and dysfunctional analysis of an health network contribution to the evaluation,” in 9th IFAC Symposium on Automated Systems Based on Human Skill and Knowledge (ASBoHS’06), 2006, pp. 373–378. J. W. Fowler, J. C. Benneyan, P. Carayon, B. T. Denton, P. Keskinocak, and G. C. Runger, “An introduction to a new journal for Healthcare Systems Engineering,” IIE Trans. Healthc. Syst. Eng., vol. 1, no. 1, pp. 1–5, Mar. 2011. J. Peck, D. Nightingale, and S. Kim, “Axiomatic approach for efficient healthcare system design and optimization,” CIRP Ann. - Manuf. Technol., vol. 59, no. 1, pp. 469–472, 2010. S.-T. Luo, C.-T. Su, and W.-C. Lee, “Constructing intelligent model for acceptability evaluation of a product,” Expert Syst. Appl., vol. 38, no. 11, pp. 13702–13710, May 2011. F. Terrade, H. Pasquier, J. Reerinck-Boulanger, G. Guingouain, and A. Somat, “L’acceptabilité sociale : la prise en compte des déterminants sociaux dans l'analyse de l'acceptabilité des systèmes technologiques,” Trav. Hum., vol. 72, no. 4, p. 383, 2009. H. Ben Rejeb, V. Boly, and L. Morel-Guimaraes, “A new methodology based on Kano Model for the evaluation of a new product acceptability during the Front-End phases,” in 32nd Annual IEEE International Computer Software and Applications Conference, 2008, pp. 619–624. I. Ajzen and M. Fishbein, Understanding attitudes and predicting social behavior. Englewood Cliffs, NJ: Prentice-Hall, 1980. F. Davis, “Perceived usefulness, perceived ease of use, and user acceptance of information technology,” MIS Q., vol. 13, no. 3, p. 319, 1989. V. Venkatesh and F. D. Davis, “A theoretical extension of the Technology Acceptance Model: Four Longitudinal Field Studies.,” Manage. Sci., vol. 46, no. 2, p. 186, Feb. 2000. V. Venkatesh, M. G. Morris, G. B. Davis, and F. D. Davis, “User acceptance of information technology: toward a unified view,” MIS Q., vol. 27, no. 3, pp. 425–478, Sep. 2003. J. Nielsen, Usability engineering. San Diego, CA: Academic Press, 1993. É. Brangier and S. Hammes-adelé, “Beyond the Technology Acceptance Model: elements to validate the Human-Technology Symbiosis model,” in International Conference, EHAWC 2011, Held as Part of HCI International , July 9-14., Orlando, FL, USA, 2011, pp. 13–21. A. M. Elazouni, A. E. Ali, and R. H. Abdel-Razek, “Estimating the acceptability of new formwork systems using neural networks,” J. Constr. Eng. Manag., vol. 131, no. 1, pp. 33–41, Jan. 2005.
[14] F. V. Jensen and T. D. Nielsen, Bayesian Networks and Decision Graphs. New York, NY: Springer New York, 2007. [15] E. Villeneuve, C. Béler, F. Peres, and L. Geneste, “Hybridization of Bayesian networks and belief functions to assess risk. Application to aircraft deconstruction aircraft deconstruction,” in International Conference on Industrial Engineering and Systems Management (IESM 2011), 2011, May. [16] Y. Hu, X. Zhang, E. Ngai, R. Cai, and M. Liu, “Software project risk analysis using Bayesian Networks with causality constraints,” Decis. Support Syst., 2012. [17] W. Song, X. Ming, and Z. Xu, “Risk evaluation of customer integration in new product development under uncertainty,” Comput. Ind. Eng., vol. 65, no. 3, pp. 402–412, Jul. 2013. [18] P. Naïm, P.-H. Wuillemin, P. Leray, O. Pourret, and A. Becker, Réseaux Bayésiens, 3rd ed. Paris: EYROLLES, 2008. [19] M. Druzdzel and L. van der Gaag, “Building Probabilistic Networks: Where Do the Numbers Come From?—a Guide to the Literature,” IEEE Trans. Knowl. Data Eng., vol. 12, pp. 481–486, 2000. [20] C. Spaccasassi and L. Deleris, “A web-based tool for expert elicitation in distributed teams,” in Proceedings of the 8th Bayesian Modelling Applications Workshop (UAI-AW 2011), 2011, pp. 125–133. [21] D. E. Morris, J. E. Oakley, and J. a. Crowe, “A web-based tool for eliciting probability distributions from experts,” Environ. Model. Softw., vol. 52, pp. 1–4, Feb. 2014. [22] S. Monti and G. Carenini, “Dealing with the expert inconsistency in probability elicitation,” IEEE Trans. Knowl. Data Eng., vol. 12, no. 4, pp. 499–508, 2000. [23] M.-D. Augustin, A. Rakotondranaivo, J.-P. P. Grandhaye, and N. Vuillemin, “Performance et innovation : apports de la comptabilité analytique et du stylo communicant dans le cadre de l’Hospitalisation à domicile de Nancy,” in 5ème Conférence Francophone en Gestion et Ingénierie des Systèmes Hospitaliers GISEH’2010, 2010. [24] E. Esparza Ribera, M. Camargo, J. P. Grandhaye, and A. Rakotondranaivo, “Evaluation de l’acceptabilité d'une nouvelle technologie dans le domaine medicale: une analyse à l'aide des réseaux bayésiens,” rapport d’ingenieur ENSGSI, Nancy, France, 2011. [25] D. M. Berwick, “Disseminating innovations in health care.,” JAMA, vol. 289, no. 15, pp. 1969–75, Apr. 2003.
