Macroergonomics and Sociotechnical Methods: Current and Future ...

Proceedings of the Human Factors and Ergonomics Society 58th Annual Meeting - 2014

1536

MACROERGONOMICS AND SOCIOTECHNICAL METHODS: CURRENT AND FUTURE DIRECTIONS Organizers & Co-Chairs: Patrick Waterson, Loughborough University, UK Michelle M. Robertson, Liberty Mutual Research Institute for Safety, USA Panelists: Pascale Carayon, University of Wisconsin-Madison, USA Peter Hoonakker, University of Wisconsin-Madison, USA Richard Holden, Vanderbilt University, USA Lawrence Hettinger, Liberty Mutual Research Institute for Safety, USA Michelle M. Robertson, Liberty Mutual Research Institute for Safety, USA Patrick Waterson, Loughborough University, UK

The development of methods for the design of work environments has a long history within human factors and ergonomics (Hendrick, 1991; Stanton et al., 2013). Recent discussions have centred on the effectiveness of macroergonomic methods in facilitating improvements to workplace design. The aim of the panel will be to summarise these and other debates surrounding methods within HFE. The panel will first discuss the use of mixed methods within macroergonomics and the role these can play in applying a whole systems approach to the design of work (Carayon), as well as validating research findings (Hoonakker). Holden will describe his use of macroergonomic methods with chronically ill patients in community settings. New approaches involving the use of methods to facilitate participatory approaches towards health and safety interventions (Robertson) and simulation (Hettinger) will also be discussed as part of the panel. The panel will conclude with a consideration of the trade-offs between scientific and practice-oriented criteria for assessing the effectiveness of methods (Waterson).

Copyright 2014 Human Factors and Ergonomics Society. DOI 10.1177/1541931214581320

SUMMARY An important part of the application of macroergonomics and sociotechnical systems theory (STS) is the development of methods, tools and techniques to assess human factors and ergonomic workplace requirements. Work on the development of methods stretches back to the earliest days of the human factors and engineering profession (Chapanis, 1999; Waterson & Sell, 2006). In recent years there has been an explosion of interest in developing methods for a variety of purposes including occupational and systems safety, interface design, product development and the analysis of complex work environments. One text on human factors methods (Stanton et al., 2013) for example, lists a total of 107 methods covering many of these application areas. The Center for Quality and Productivity Improvement (CQPI) at the University of Wisconsin-Madison created a website to help clinics who wanted to implement an Electronic Health Record system identified over 110 instruments to identify workflow issues. Alongside the development of methods, there have been a number of important new methodological developments within human factors and ergonomics. The use of ‘mixed methods’ in research for example, has since the 1980’s become very popular. Mixed methods research is an approach that combines the collection and analysis of quantitative and qualitative data (Creswell and Clark, 2011). In some cases, for example, qualitative data in the form of interviews has been

used to elaborate, interpret or expand the findings of from survey data (Edmondson, 1996; Karsh et al., 2014). Other strategies in which the researcher converges quantitative and qualitative data in order to provide a comprehensive analysis of the research problem are also possible (Creswell, 2003). Methods for conducting systems analysis (e.g., MEAD – Kleiner, 1999; Systems Analysis Tool (SAT) – Robertson & Courtney, 2004) have proved to be popular ways of analyzing and designing interventions for modern work systems, as well as encouraging participatory approaches to design (Imada et al., 1986). These types of method have also been seen as advantageous in terms of their ability to apply a comprehensive, multi-level systems approach to workplace design which (Wilson, 2014). The use of alternative systems analysis methods such as computer-based simulation and modelling are also on the increase (Holland, 2012). A separate issue which has generated a lot of interest more recently is the nature of the criteria we should use to judge the explanatory adequacy of methods for macroergonomics. A number of researchers and practitioners have debated the importance of considering the reliability, validity and utility of HFE methods (e.g., Kanis, 2014). Many of these discussions touch on wider debates regarding the status of HFE as a science, engineering discipline or craft (Moray, 2007) and the nature of the research-practice gap (Chung and Shorrock, 2011).

Downloaded from pro.sagepub.com by guest on October 28, 2015


The aim of this panel is to provide a forum in which to discuss the current state of these discussions and explore possible ways forward for developing new methods, as well combining old methods. The symposium will also cover some of the latest developments and applications of these STS and ME methodologies and approaches within the field and the application of these methods that : these include: systems analysis approach that integrates participatory design for planning integrated health and safety interventions; gaps in addressing the coverage of theoretical and practical coverage of methods for STS; future opportunities to improve the validity and practicality of methods; the advantage of using mixed methods in macroergonomics research; the methodological challenges and potential future directions for macroergonomics in less structured and variable work domains; the contrast between scientific approaches to method development and use and actual application by HFE practitioners; and, the advantages of using modelling and simulation methods to gain a better understanding work domains. Application of Mixed Methods in Healthcare Human Factors Research Pascale Carayon, University of Wisconsin-Madison (on behalf of Sarah Kianfar, Yaqiong Li, Anping Xie, Bashar Alyousef and Abigail Wooldridge) Mixed methods research is increasingly used and formally recognized as an approach, in particular in healthcare research (Creswell, Klassen, Plano Clark, & Smith, 2011). We conducted a systematic literature review on the use of mixed methods in healthcare human factors research published before February’2013. A total of 53 papers published in 39 journals were identified; only 23% of the papers were published in human factors journals. These studies tackle a range of human factors issues, in particular related to IT system, but also to the work of nurses and physicians and worker safety. A variety of methods were used to collect qualitative and quantitative data: primarily interview, survey and observation for qualitative data, and survey for quantitative data. All studies involved mixing of qualitative and quantitative data at the data interpretation stage; about one-third of the studies mixed qualitative and quantitative data at the data collection and data analysis stages. Only 4 studies involved mixing at the design stage. Healthcare human factors research could benefit from using formal mixed methods research approaches (Creswell, 2009; Creswell & Plano Clark, 2011). This would help to enrich and deepen the research and increase its impact on important healthcare outcomes. The use of mixed methods research can also provide opportunities to tackle cognitive, physical and organizational ergonomics issues in a holistic manner. Mixed Methods at Various Levels Peter Hoonakker, University of Wisconsin-Madison In most of our studies we use mixed methods (observations, interviews, focus groups, and surveys). The information resulting from the different data collection

1537

methods provides us with a deeper understanding of the phenomena of interest, as well some sort of quantification (surveys). Many of our studies are conducted in healthcare. One of the major challenges in healthcare is communication and coordination. Especially during patient transitions (e.g. from the hospital to back home), information is often not communicated properly, partly due to lack of coordination between the different providers, and lack of communication and coordination is one of the factors that is associated with medical errors (Carayon et al., 2011; Ong & Coiera, 2011). Communication is especially difficult in virtual collaboration (Jarvenpaa & Leidner, 1998; Kasper-Fuehrera & Ashkanasy, 2001). In virtual collaboration people who often have never met each other, have to communicate with each other by using information technology (IT). As most of us know, using IT for communication (e.g. e-mails) can be very difficult because many of the nuances that are used in face-toface communication (e.g. non-verbal communication, direct feedback, etc.) are lacking. It is difficult to figure out whether someone who you have never met, and is working from his office in China is making a joke, or not. In our study on tele-Intensive Care Units (tele-ICUs) we used mixed methods to examine the virtual collaboration between tele-ICUs and ICUs. A tele-ICU monitors patients in remote ICUs with the use of health IT. There are many advantages to monitoring of patients in a tele-ICU, but there are also many challenges. Totally we conducted 160 hours of observations, 60 interviews with ICU managers and nurses, and conducted a survey. During our interviews we noticed that, according to nurses, it was sometimes easy and sometimes it was more difficult to communicate with the nurses in the ICUs that they monitored. During the interviews several factors were mentioned that could affect communication with nurses in the ICUs at various levels: hospital type (academic, teaching or non-teaching); history with the ICU; type of ICU (cardiac vs. medical ICUs); the number of beds monitored in the ICU, age and tenure of the tele-ICU nurses, and the fact that nurses in the tele-ICU had worked in the past, or currently still worked in the ICUs part-time. As a result of the difficult communication with some of the ICUs, tele-ICU nurses were less satisfied with those ICUs and the nurses who work there. We had quantitative data from the surveys that we could use to test these “hypothesis” about factors that could have an impact on communication and eventually on satisfaction with the ICU. In the survey we had asked nurses how satisfied they were with the different ICU that they monitored, and we asked tele-ICU nurses to rate communication openness, accuracy and timeliness with nurses in three of the ICUs that they monitored. We used multi-level analysis to examine whether satisfaction with the ICUs was a function of communication with the ICUs and whether characteristics at the organizational level (i.e. organizational characteristics of the ICUs) or at the individual level (i.e. individual characteristics of the tele-ICU nurse) affected that relationship. Results showed that satisfaction with ICUs was mainly a function of communication; partly a function of individual characteristics; but not of organizational characteristics. Tele-ICU nurses who mentioned that communication with nurses in the ICUs was



characterized by communication openness, accuracy and timeliness were in general more satisfied with the tele-ICUs that they monitored. Nurses with longer tenure were in general less satisfied with the different ICUs that they monitored. This example clearly shows that with using mixed methods to collect data, the data from one research method cannot only be triangulated with data from another research method, but that results of one type of data analysis can be” tested” by using data that was collected with another method. Challenges in conducting macroergonomics research in less structured community settings with vulnerable individuals Richard J. Holden, Vanderbilt University (on behalf of Amanda McDougald Scott, Peter Hoonakker, Pascale Carayon, and Ann Hundt) Macroergonomics research is predominantly conducted with paid workers in settings such as factories and offices. Scholars and practitioners have also begun to apply macroergonomics to community settings for the purpose of rebuilding the inner city (Smith et al., 1994), improving learning (Smith, 2012), or understanding health selfmanagement in the home (Holden & Mickelson, 2013). Work in these settings is accompanied by considerable challenges. For example, compared to formal organizations, community settings may lack orderliness, defined schedules, reliable communication systems, safe and hygienic physical locations, and resources. The individuals studied in community settings may also be vulnerable for a number of social, economic, and medical reasons. These characteristics of community settings and their residents require the adaptation of macroergonomics methods as well as special considerations for implementing these methods. Our work applying macroergonomic methods with community-dwelling, chronically ill patients in urban and rural US communities produced insights into methodological and implementation challenges and potential solutions. Two studies were carried out. The first study was with 90 patients with chronic cardiovascular disease living in urban and rural settings in the Southern US. The second involved 19 chronically ill patients in the rural region of an Eastern US state. The patients themselves could be considered vulnerable as a result of advanced age, rural location, low income, lack of health insurance, number of diseases, illness severity, low literacy, disability, minority status, and or often some combination of these. Some of the challenges encountered included difficulty obtaining records about or accessing participants; difficulty contacting participants and explaining the purpose of the study; and gaining entrée into their homes. Even when individuals agreed to participate or completed one part of the research, there were problems with scheduling and no-shows. Participants had limited time and ability to participate in some of the research, due to daytime obligations and perceptual and cognitive impairments. There were also concerns for personal safety for researchers because of hazards in the home or community. Lastly, data quality was a concern because of

1538

memory problems and social desirability effects related to self-reporting behaviors such as diet and exercise. To address the above challenges, some new methods may be required. For example, to improve data quality, novel ways to elicit information may be needed that support retrieval from memory (or do not require as much reliance on memory). Furthermore, there are several frameworks that could help researchers implement macroergonomics in community settings (Holden, Or, Alper, Rivera, & Karsh, 2008; Israel, Schulz, Parker, & Becker, 1998). Modeling the Dynamics of Sociotechnical Systems Lawrence Hettinger, Liberty Mutual Research Institute for Safety, USA (on behalf of Arif Jetha) Sociotechnical systems can be considered to examples of the broad class of phenomena referred to as complex, adaptive systems (Carayon et al., under review; Holland, 2012; Miller & Page, 2007). As such, their empirical analysis by means of traditional experimental approaches can be problematic (Waterson et al, under review). Laboratory studies lack the means to reproduce the level of real-world complexity that impacts human-system performance in real-world settings, and field analysis techniques generally do not permit the systematic modification of system components needed to derive causal inferences. For a number of years, modeling and simulation have been explored as potential methods for overcoming these limitations, but their application to problems associated with sociotechnical system design and function has only recently begun. In our discussion, we will provide an overview of the state-of-the-art of modeling and simulation techniques as applied to complex work settings, with particular emphasis on our recent efforts to apply system dynamics modeling (e.g., Sterman, 2000) to their examination. Systems Analysis participatory ergonomics tool for designing integrated health and safety interventions Michelle M. Robertson, Liberty Mutual Research Institute for Safety, USA (on behalf of Rob Henning, Nicholas Warren, Suzanne Nobrega and the CPH-NEW research team) Conventional workplace safety programs with a top down management approach are limited in their effectiveness, appear to be short lived, and are not well integrated into organizational cultures. Therefore, a primary research focus and effort has been to develop a way to fully engage front-line employees and managerial/supervisory personnel in the collaborative, iterative design of workplace interventions. Our novel approach expanded the Participatory Ergonomics (PE) process to encompass integrated safety and Health Promotion (HP) interventions to support continuous improvement of employee’s safety and health to achieve Total Work Health (TWH). This led to the development of the Intervention Design and Analysis Scorecard (IDEAS). This integrated PExHP “bottom-up” approach actively engages employees in the design of workplace interventions. Teams of employees, supported by a multilevel steering committee, are involved in the decision-making, problemsolving actions, and evaluation of these interventions, all of



which appear to be necessary for the success of a functioning program (Henning & Reeves, 2013). The IDEAS provides a stepwise approach for developing intervention proposals IDEAS was tested at four diverse worksites with trained facilitators where employees were able to develop and gain management support for integrated interventions at each worksite. The IDEAS tool can be used effectively by frontline employees to plan integrated interventions in a program dedicated to continuous improvement of employee safety and health and Total Worker Health. This presentation will present an overview f the Intervention Design and Analyses Scorecard (IDEAS) and its application to planning and implementing participatory design integrated health and safety interventions in the workplace. ‘Calling time’: trade-offs between generality, validity and utility in the development of macroergonomic methods Patrick Waterson, Loughborough University, UK (on behalf of Chris Clegg and Mark Robinson) We describe our experiences in developing human factors methods for a variety of purposes over the last couple of decades. In particular, we focus on recent discussions centred on the Fitts List (e.g., de Winter and Dodou, 2011) and accident investigation methods (e.g., the degree to which they are practice-focused, theoretically robust, valid and reliable). We draw on earlier work by Karl Weick (1979) concerning the trade-offs involved in developing theory within the social sciences and apply these to method development within human factors and ergonomics. Weick (drawing on earlier work by Thorngate, 1976) uses the metaphor of a clock face to argue that it is impossible for a theory of social behaviour to be simultaneously general, accurate and simple. You can seek any two of these three characteristics, but not achieve all three. Thus the more general and simple your theory, for example, the less accurate it will be in predicting specifics. Attempts to secure any two of the ‘virtues’ of a theory will mean that the third will be sacrificed. ‘Two o’clock theories’ for example, are general and accurate, but they will not be simple. This analogy using the clock face might well be usefully applied to human factors methods. If we take what is arguably one of the most well-known HFE methods, task analysis, we might say that it fits the category of a ‘two o’clock method (general and accurate). Likewise, we might argue that another successful (if heavily criticized) method, the Fitts List, is a typical ’ten o’clock method’. Thus the Fitts List is very simple to use and offers general applicability, however, it is far from accurate, not least since the types of automation available today are radically difference from the those of the 1950’s and 60’s. The final section of the paper outline a set of issues for future work (e.g., satisficing trade-offs, using methods in combination, ways of improving the research-practice feedback loops). We suggest that there needs to be wider acknowledgement that reliability, validity and other measures of ‘method ‘success’ may need to be seen within the larger context of trade-offs within HFE practice. This stands in contrast to the views of others who appear to regard them as immutable ‘gold standards’. In short, the use of ‘hard’

1539

scientific criteria for the evaluation of HFE methods is necessary, but in other cases it may not be sufficient. REFERENCES Carayon, P., P. Hancock, N. Leveson, Y. I. Noy, L. Sznelwar, G. van Hootegem and L. Hettinger, under review. Sociotechnical Systems for Occupational Safety: New Research Directions, Ergonomics. Carayon P, Karsh B-T, Cartmill RS, et al. Incorporating Health Information Technology Into Workflow Redesign--Summary Report. (2010), Prepared by the Center for Quality and Productivity Improvement, University of Wisconsin–Madison, under Contract No. HHSA 290-2008-10036C). AHRQ Publication No. 10-0098-EF. Rockville, MD: Agency for Healthcare Research and Quality. October 2010. Carayon, P., Bass, E. J., Bellandi, T., Gurses, A. P., Hallbeck, M. S., & Mollo, V. (2011). Sociotechnical systems analysis in health care: A research agenda. IIE Transactions on Healthcare Systems Engineering, 1(3), 145160. Chapanis, A. (1999), The Chapanis Chronicles. Santa Barbara: Aegean Press. Chung, A.Z.Q. and Shorrock, S.T. (2011), The research-practice relationship in ergonomics and human factors - surveying and bridging the gap, Ergonomics, 54, 5, 413-429. Creswell, J.W. (2003), Research Design: Qualitative, Quantitative and Mixed Methods Approaches. New York: Sage Publications. Creswell, J. W. (2009). Research Design - Qualitative, Quantitative, and Mixed Methods Approaches (Third ed.). Los Angeles, CA: Sage. Creswell, J.W. and Clark, J.L.P. (2011), Designing and Conducting Mixed Methods Research (2nd Edition). New York: Sage Publications. Creswell, J. W., Klassen, A. C., Plano Clark, V. L., & Smith, K. C. (2011). Best practices for mixed methods research in the health sciences. National Institutes of Health. Edmondson, A., (1996). Learning from mistakes is easier said than done: group and organizational influences on the detection and correction of human error. Journal of Applied Behavioral Science 32, 5e28. Hendrick, H.W. (1991), Ergonomics in organizational design and management. Ergonomics, 34, 6, 743-756. Henning, R.A., and Reeves, D.W. (2013). An integrated health protection/promotion program supporting participatory ergonomics and salutogenic approaches in the design of workplace interventions. In: Bauer G., Gregor J., ed., Salutogenic Organizations and Change: The Concepts Behind Organizational Health Intervention Research. The Netherlands: Spring, 307-324. Holden, R.J., Or, C.K.L., Alper, S.J., Rivera, A.J., & Karsh, B. (2008). A change management framework for macroergonomic field research. Applied Ergonomics, 39, 459-474. Holden, R. J., & Mickelson, R. S. (2013). Performance barriers among elderly chronic heart failure patients: An application of patient-engaged human factors and ergonomics. Proceedings of the Human Factors and Ergonomics Society, 57, 758-762. Holland, J.H., (2012). Signals and boundaries: Building blocks for complex adaptive systems. Cambridge, MA: The MIT Press. Imada A.S., Noro K., Nagamachi M. (1986). Participatory ergonomics: Methods for improving individual and organizational effectiveness. In O. Brown, Jr., & Hendrick (Eds.), Human Factors in Organizational Design and Management II, pp. 403-406, Amsterdam: North Holland. Israel, B. A., Schulz, A. J., Parker, E. A., & Becker, A. B. (1998). Review of community based research: Assessing partnership approaches to improve public health. Annual Review of Public Health, 19, 173-202. Jarvenpaa, S. L., & Leidner, D. E. (1998). Communication and Trust in Global Virtual Teams. Journal of Computer-Mediated Communication, 3(4), 0-0. Kanis, H. (2014), Reliability and validity of findings in ergonomics research. Theoretical Issues in Ergonomics Science, 15, 1, 1–46. Karsh, B-T., Waterson, P.E. and Holden, R. (2014), Crossing levels in systems ergonomics: a framework to support ‘mesoergonomic’ inquiry. Applied Ergonomics, 45, 45-54. Kasper-Fuehrera, E. C., & Ashkanasy, N. M. (2001). Communicating trustworthiness and building trust in interorganizational virtual organizations. Journal of Management, 27(3), 235-254. Kleiner B.M. (1999). Macroergonomic analysis to design for improved safety and quality performance. International Journal of Occupational Safety and Health, 5, 317-245.



Moray, N. (2007), The human factors of complex systems: a personal view. In D. deWaard. G.R.J. Hockey, P. Nickel, and K.A. Brookhuis (Eds). Human Factors Issues in Complex System Performance. Maastricht, Netherlands. Miller, J.H. and Page, S. E. 2007. Complex adaptive systems: An introduction to computational models of social life. Princeton, NJ: Princeton University Press. Ong, M. S., & Coiera, E. (2011). A systematic review of failures in handoff communication during intrahospital transfers. Joint Commission Journal on Quality and Patient Safety, 37(6), 274-278AP. Robertson, M. M., & Courtney, T. K. (2004). A systems analysis approach to solving office work system health and performance problems. Theoretical Issues In Ergonomics Science, 5(3), 181-197. Robertson, MM., Henning, R., Warren, N., Nobrega, S., Dove-Steinkamp, M., Tibirica, L., Bizarro, A., CPH-NEW Research Team. (2013). The intervention design and analysis scorecard: A planning tool for participatory design of integrated health and safety interventions in the workplace. Journal of Occupational Environmental Medicine, 55, (12) Smith, M.J., Carayon, P., Smith, J. Cohen, W., & Upton, J. (1994). Community ergonomics: A theoretical model for rebuilding the inner city. Proceedings of the Human Factors and Ergonomics Society, 38, 724-728. Smith, T.J. (2012). Integrating community ergonomics with educational ergonomics – designing community systems to support classroom learning. Work, 41, 3676-3684. Sterman, J. D., (2000). Business Dynamics: Systems Thinking and Modeling for a Complex World. New York: McGraw-Hill. Thorngate, W. (1976), "In general” vs. "it depends“: Some comments on the Gergen-Schlenker debate. Personality and Social Psychology Bulletin, 2, 404-410. Waterson, P., Robertson, M.M., Cooke, N.J., Militello, L., Roth, E. & Stanton, N.A. (under review). Defining the methodological challenges and opportunities for an effective science of sociotechnical systems and safety. Ergonomics Waterson, P.E. and Sell, R. (2006), Recurrent themes and developments in the history of the Ergonomics Society. Ergonomics, 49, 8, 743-799. Weick, K. (1979), The Social Psychology of Organizing. Reading, Mass.: Addison-Wesley. Wilson, J.R., 2012. Fundamentals of systems ergonomics/human factors. Applied Ergonomics, 45, 1, 5-13. de Winter, J.C.F. and Dodou, D. (2011), Why the Fitts list has persisted throughout the history of function allocation. Cognition, Technology and Work, August 2011, doi: 10.1007/s10111-011-0188-1

ACKNOWLEDGEMENTS The tele-ICU study was made possible with support from the National Science Foundation (NSF Grant #: OCI-0838513, Carayon: PI; Brown, Hoonakker, Wiegmann and Wood, co-PIs) and the cooperation from the pilot tele-ICU and the 5 tele-ICUs in the main study, and the managers and the nurses who work there. Waterson wishes to acknowledge the support provided within the UK by Network Rail and the Rail Safety and Standards Board (RSSB).


1540