www.palgrave-journals.com/jors/. Combining problem structuring methods to conduct applied research: a mixed methods approach to studying fitness-to-drive in ...
Journal of the Operational Research Society (2009) 60, 1637 --1648
© 2009
Operational Research Society Ltd. All rights reserved. 0160-5682/09 www.palgrave-journals.com/jors/
Combining problem structuring methods to conduct applied research: a mixed methods approach to studying fitness-to-drive in the UK GA Hindle∗ and LA Franco University of Warwick, Conventry, UK This paper describes a mixed methods approach using problem structuring methods to conduct applied research into fitness-to-drive arrangements within the UK Department for Transport. Computer-supported group causal mapping was used to collect and structure qualitative data from stakeholder groups concerning the delivery of medical standards on fitness-to-drive. The data were subsequently coded and analysed using the modelling language of soft systems methodology. This enabled data to be linked to the concept of a ‘fitness-to-drive system’ and developed further in the form of systems models based on alternative worldviews. The paper reports on the process of developing and implementing the approach and discusses issues concerning the conduct of mixed methods research using problem structuring methods. Journal of the Operational Research Society (2009) 60, 1637 – 1648. doi:10.1057/jors.2008.125 Published online 19 November 2008 Keywords: problem structuring methods; mixed methods research; causal mapping; soft systems methodology
1. Introduction Problem structuring methods (PSMs) are a family of facilitated modelling approaches that support the work of groups of diverse composition in order to help them address complex problem situations in a variety of organizational domains. These include, for example, project management (Franco et al, 2004; Winter, 2006), strategy development (Eden and Ackerman, 2000; O’Brien and Meadows, 2007), knowledge management (Shaw et al, 2003; Montibeller et al, 2006), information systems development (Ormerod, 1998, 2005), inter-organizational collaboration (Huxham, 1996; Franco, 2008) and community work (White, 1996; Walsh and Hostick, 2005). There is also evidence that PSMs are being used to conduct applied research. Hindle et al (1995), for example, use soft systems methodology as an action research framework to study contract management in the UK National Health Service. Similarly, Brown et al (2006) use soft systems methodology as a guiding framework for collecting the views of different stakeholder groups in the UK personal taxation system. Eden and Ackermann (2004) use cognitive mapping to collate, compare and analyse the views of many experts in relation to a policy issue within the UK Home Office Prison Department. Also, Shaw (2006) investigates whether technology-supported Journey Making workshops ∗ Correspondence: GA Hindle, Warwick Business School, University of
Warwick, Conventry CV4 7AL, UK.
can make effective research tools. More recently, Hindle (2007) develops a systemic textual analysis methodology based upon human activity system modelling from soft systems methodology. Our general aim is thus to contribute to this emerging literature by reporting on the design and implementation of a research approach using PSMs for the study of the fitness-to-drive (FtD) arrangements in the UK, and to reflect on the experience. The research reported here adopts a mixed-method approach using PSMs. We also seek to contribute to the domain of multi-methodology, which is becoming a new paradigm of organizational intervention practice within management science (Mingers and Gill, 1997; Mingers and Rosenhead, 2004). However, to the extent of the authors’ knowledge, there are limited published accounts of multimethodology applications for research purposes only. Indeed, the possibilities for mixing management science methods outlined above are mainly intended to inform organizational intervention rather than traditional research projects. Thus this paper also attempts to address this gap. The literature on social sciences shows a similar concern for mixing methods, known as ‘mixed methods research’ (Blaikie, 1991; Tashakkori and Teddlie, 2003; Johnson et al, 2007). Specifically, mixed methods research involves the combination of different methods of collecting and analysing data about a phenomenon of interest, in order to either produce or convergence of research findings, or a fuller picture of the phenomenon (Erzberger and Prein, 1997). The first use of mixed methods is known as standard ‘triangulation’ of
1638
Journal of the Operational Research Society Vol. 60, No. 12
research methods (Denzin, 1989), whereas the second is closer to the notion of ‘complementary’ in management science (Pidd, 2004; Brown et al, 2006). Different mixed methods research designs have been proposed in the literature but their treatment is beyond the scope of this paper (see, eg, Greene et al, 1989; Tashakkori and Teddlie, 2003; Onwuegbuzie and Johnson, 2004; Leech and Onwuegbuzie, 2007). Here, we are particularly concerned with the combination of PSM techniques within a single research project. By ‘techniques’ (also known as ‘methods’) we mean a set of ‘well-defined sequences of operations that, if carried out proficiently, yield predictable results’ (Mingers, 2001, p 307). PSM techniques thus would include, for example, SODA’s cognitive/causal mapping, soft system methodology’s (SSM’s) human activity systems modelling, and the strategic choice approach’s (SCA’s) analysis of interconnected decision areas. The rest of the paper is structured as follows. First, a summary of the context in which the research was conducted is provided, followed by a description of the approach adopted. Next, a description and discussion of the coding and analysis of the research data is provided. The experience with the research approach suggests some issues in relation to the way mixed-method research using PSMs is conducted and these are discussed. Finally, conclusions and future research directions are provided.
2. The research context and the approach adopted The work reported in this paper was part of a research project concerning the implementation of medical standards on FtD, which was carried out by the University of Warwick for the UK Department for Transport (DfT). The project utilized methods such as national surveys, video vignettes, focus groups and workshops to assess healthcare professionals’ (HCPs) current knowledge of the medical standards and their attitudes towards giving advice on FtD to their patients. The DfT was particularly interested in understanding the factors that might influence the decisions of HCPs to discuss FtD issues during routine clinical contacts. The project also sought to identify ways of improving the knowledge of HCPs and their willingness to give advice to patients. The authors were commissioned to run stakeholder workshops and designed a mixed methods research approach combining causal mapping and soft systems methodology. The workshops sought to link the present situation to possible alternative futures for the delivery system. The client’s focus here was primarily on exploring options to improve the interface between HCPs and the operational system; a feature they saw as key to improving the effectiveness of the system overall. Given the experience and expertise of the authors in both causal mapping and systems modelling, the adoption of a mixed methods research approach to collecting and analysing workshop data was deemed appropriate. The approach starts
by using ‘on-the-hoof’ computer-supported causal mapping to collect, structure and analyse workshop data (Ackermann and Eden, 2001). This is followed by a coding of the data using human activity system modelling from soft systems methodology (Checkland and Poulter, 2006). This enabled data to be linked to the concept of a ‘FtD system’ and developed further in the form of relevant systems based on alternative worldviews. Thus, data were captured through causal mapping and analysed using both PSM techniques; therefore, ‘mixing’ took place at the interpretation stage of the research. The mixed methods research approach employed a series of eight PSM workshops held between October 2006 and July 2007. The workshops were conducted with a variety of HCPs and relevant stakeholder groups including doctors, nurses, patients, occupational therapists, driving assessors, specialist nurses, insurance companies, the police, local authorities and others. The workshops were designed to encourage participants to: (1) express their attitudes to existing FtD arrangements; (2) identify barriers and issues concerning how HCPs interact with the system; and (3) generate ideas to improve participation and involvement with the system. Owing to problems securing a representative group of GPs for a workshop, data were collected via a questionnaire. This obtained 62 responses that were then added to the research data pool collected through the workshops. Individual workshops were conducted using the following steps, although slight changes were made due to time restrictions where necessary: • Step 1: rich picturing. The situation is ‘mapped’ using the rich picture tool from SSM. Figure 1 shows an example of a rich picture drawn in one of the workshops. The initial structure and content of the picture is established by the facilitator, but live data are added by the participant group. The intention of the picturing is twofold: (a) to make sure the participants are familiar with the essential elements of the situation regarding existing FtD arrangements; and (b) to generate an initial list of issues to support subsequent data collection, as explained below. • Step 2: issue mapping. Participants are asked to express their views as to the issues they perceive within the situation. These data are textual, and are captured and structured into themes by the PSM facilitator using the Group Explorer networked workstation system running along the Decision Explorer causal mapping software. As shown in Figure 2, each participant has access to a laptop computer and types their issues into the system. The PSM facilitator is then able, with the help of the participants, to structure the resulting issues into themes. Figure 3 shows an illustration of themed issue data developed in one of the workshops. • Step 3: improvement mapping. Participants are asked to generate ideas as to possible actions that might improve the situation; as with Step 2, this textual data are captured and structured into themes using the Group Explorer system.
GA Hindle and LA Franco—Combining PSMs to conduct applied research
Figure 1
1639
An illustration of a rich picture.
system, and (3) alternative worldviews for possible system re-designs. Details about the soft systems modelling are given in Section 3.
3. Data coding and analysis using systems modelling
Figure 2 Computer-supported causal mapping with a group of occupational therapists.
• Step 4: soft systems modelling. A set of soft systems models is developed which captures and structures thinking regarding: (1) the nature of the existing system to deliver medical standards, (2) possible ways of ‘tuning’ that
From the initial data collection, a range of possible human activity systems appeared relevant to the problem situation; these included: the NHS healthcare system as a whole; the healthcare sub-systems or pathways relevant to various HCPs or medical conditions; the consultation processes of the various HCPs; and the education and training processes of the various HCPs. However, the obvious ‘system-in-focus’ for the research reported here could be conceptualized as a ‘system to deliver medical standards on FtD’. This system could be viewed as a human activity system that creates guidelines on FtD (the medical standards) and ensures they are implemented in practice (a) through the cooperation of HCPs and patients, and (b) by enacting a DVLA licensing system and administrative process (see Figure 1). An initial conceptual model of this ‘system-in-focus’ was built to represent the system’s activities under current operating conditions, at least in principle. This model became known as the ‘baseline model’ and constituted a useful initial
1640
Journal of the Operational Research Society Vol. 60, No. 12
health professionals make their own judgements Not many accidents are actually caused by patients with med conds
140
Negotiation in the consultation: impact of taking a license away
56
128 13 2 14 31 132 129
13
7
139
patients need their cars in rural areas
HCP roles
HCP priorities 30
57
42
38
37
36
Issues of compliance by patients/drivers with advice given
eg old people are careful, if rather blind
3 2
compliance
1 3 3
12 9
51
55
53
56
HCP workload & time constraints 13 6
15
Medical profession notoriously slow to implement change
5
Patients unwilling to take personal responsibility
if patient won't tell truth, what can doctor do?
DVLA unwilling to educate patients directly
patients' responsibility
Lack of clarity as to responsibility: patient is responsible to inform, but doctor may inform if there is a clear risk
How to increase patients' awareness of their responsibilities
Figure 3
Awareness of all areas where rules apply low: neuro/cardovascular: GOOD; psychiatry: BAD
some conditions easier to have definite guidelines than others eg epilepsy, diabetes
making guidelines more accessible to the public
Current system is based on self-report by drivers ... Resistance to changing this
129
patients have little incentive to tell the truth
40
17
1 3 8
Lack of public awareness on advice/support available
lack of HCP & public awareness 13 2 127
129 10
disseminating changes in guidelines
Awareness of internet site for information low
An excerpt from themed issue data.
reference point for the project team and client body. The model was further developed during the project in order to code the workshop data. Figure 4 shows a CATWOE analysis and root definition of the system. The baseline conceptual model is presented in Figure 5. The model has 12 activities. Activities 1–3 feed into Activity 4, which is ‘develop medical standards (MSs)’. Activities 1–3 cover the creation of the multidisciplinary team of actors (a significant feature of this human activity system), designing the method for developing the MSs and appreciating research evidence on FtD. The model then splits into three process streams: Activities 5 and 6 cover the promotion of the MSs to HCPs, Activity 7 covers the inclusion of MSs in the educational programmes of HCPs and Activity 10 covers the design of the licensing system used by DVLA. Activities 6 and 7 then feed into the implementation of the MSs, Activity 8 is HCPs giving advice to patients, Activity 9 is patients (and sometimes others) contacting the DVLA to make a declaration, and Activity 11 is the operation of the DVLA’s administrative system in Swansea (a team of over 300 staff)—that is, making licensing decisions and keeping records. Activity 11 is supported by Activity 12, which is the collection of extra information about non-routine patients. Each of the eight workshops (and GP Questionnaire) produced substantial data sets relating to issues with the
current ‘system to deliver medical standards on FtD’ as well as ideas for improving it (innovation themes). Below we present the process adopted for the analysis of the data collected in the workshops. The analysis was conducted in four steps as shown in Figure 6. (1) Coding workshop data using the baseline model. (2) Developing workshop summaries using coded data and causal maps. (3) Developing innovation themes—for both ‘tuning’ the existing system and generating alternative system designs. (4) Further systems modelling.
3.1. Coding workshop data using the baseline model The baseline model presented in Figure 5 was a key tool in the data analysis. The baseline model articulated our understanding of the present arrangements to deliver medical standards on FtD in the form of a set of 12 core operational activities. This enabled issue data (issues with the current FtD arrangements) and improvements data (ideas to improve the situation) to be coded (or located) according to the activities within the model. This is important because, although the focus of the research project was on those activities within the
GA Hindle and LA Franco—Combining PSMs to conduct applied research
Customers of system:
Patients with medical conditions, Road users in general
Actors in the system:
Drivers Medical Group, Advisory Boards, relevant HCPs (often doctors and specialist nurses), patients with medical conditions, relevant research communities
Transformation:
need for delivered Medical Standards on FtD into need met
Weltanschauung:
that Advisory Boards consisting of experts can develop appropriate Medical Standards for FtD, which can be implemented by HCPs (if made available to them) through advice giving; that this will lead to participation by patients and hence control through licensing; and that this will effectively manage FtD risks and contribute to road safety in general
Owners
DVLA and Department for Transport
Environmental
evidence on FtD; availability of FtD experts; healthcare system in general; culture and practices of HCPs;funding; stakeholders such as police, LAs, insurance companies; Government policy; patients; driving culture; transport system; driving licensing; demographics
1641
Root Definition:
A system to deliver Medical Standards on FtD by operating a multidisciplinary group co-ordinated by the Drivers Medical Group in order to manage risk and to contribute to road safety
Figure 4
CATWOE analysis and root definition for the baseline model.
baseline model directly related to HCPs (mainly Activities 5, 6, 7 and 8), issue data and improvement data were generated covering a wide range of issues and activities. This highlighted two core values of PSMs in general, and SSM in particular—they (a) seek to collect a range of perspectives from relevant stakeholders and (b) attempt to take a holistic view of the situation. It was not surprising, therefore, that the data collected relate not only to the specific research questions of the overall research project, but also overlap into more general concerns about FtD arrangements. Hence, our approach ensured a full range of data could be coded: data could be coded to a particular activity in the baseline model or alternatively, if it is related to the system as a whole it was coded ‘systemic’, if an environmental constraint it was coded ‘environment’, and if it suggested alternative system designs and/or a change in the underlying philosophy or Weltanschauung (a German word for Worldview) of the baseline model, it was coded ‘alternative W’. The model therefore enabled us to code all data before determining its relevance to the research aims. For example, some data questioned the process used to produce the medical standards themselves. Although initially this did not appear directly relevant to the research questions (ie how to improve HCPs’ advice-giving behaviour), the participants in Workshop 1 argued the process affected HCPs’
perception of the quality of the standards and therefore their likelihood to implement them—hence, their attitude to giving FtD advice and participating in the DVLA’s system. Three examples of the coding will now be given; one for an issue datum, the second for an improvement datum, and the third for an alternative Weltanschauung. The issue datum ‘there is a lack of knowledge of dementia and its relation to driving safety’ relates to the intelligence element of the baseline model ‘appreciate evidence on FtD and medical conditions’ (Activity 3 in the baseline model). Hence this datum would be coded ‘BM (baseline model) Activity 3’. Similarly, the improvement datum ‘involve the police in the delivery of the system’ relates to the choice of actors element of the baseline model ‘identify relevant stakeholders: create multidisciplinary group of actors’ (Activity 1 in the baseline model). Hence this datum would be coded ‘BM Activity 1’. The improvement datum ‘systematically educate all drivers every three years and include FtD evaluation’ implies reduced reliance on HCP advice and self-notification and increased integration with the general licensing system; this datum appears to be suggesting a change in the design of the current system through a Weltanschauung based upon (mandatory) continuous driver development and evaluation. Hence this datum would be coded ‘alternative W’.
1642
Journal of the Operational Research Society Vol. 60, No. 12
2. Appreciate best practice and design process for developing MS
3. Appreciate evidence on Fitness to Drive and medical conditions
1. Identify relevant stakeholders: • create multidisciplinary group of actors • appreciate roles, responsibilities & practices of HCPs
4. Develop Medical Standards (MS)
5. Make MS available to HCPs and the public: • Design presentation • Use media + internet
7. Establish MS and their application in educational systems of HCPs 10. Design licensing structure
6. Promote MS: • Use marketing tools • Run workshops + events
8. [HCPs] Give advice on FtD through HCPs
9. [Patients, HCPs, other] Make declaration to DVLA
Baseline Model Dft
12. Perform investigations
11. Operate administrative system in Swansea
Figure 5 Baseline systems model.
Issue Data
think of ideas for improvement in relation to that particular area of the baseline model.
Current System code
Improvement Data
Baseline Model
Outputs: • Coded data • Workshop Summaries • Themes
code Further Modelling: Alternative System Designs and ‘tuning’ the existing system
Figure 6
Overview of data analysis process.
Data were thus coded in such a way as to enable issues and ideas for improvement to be located within the baseline model or identified as systemic, environment or alternative Weltanschauung. In some cases, a particular code had no data allocated to it for a particular workshop. This helped us see clearly that the participant group did not perceive issues or
3.2. Developing workshop summaries Although the overarching aim of our research approach was to combine data across the workshops using systems modelling, a useful intermediary step was to develop individual textual workshop summaries. The summaries were produced by analysing the causal maps developed in the workshops as well as the coded data from Step 1. The summaries were useful to both the facilitators and the client group. They ensured the facilitators could reflect on and draw conclusions from individual workshops while the data and participant contact was still fresh in their minds. They enabled the client group to compare data from separate participant groups (eg, to compare the inputs from hospital-based occupational therapists with primary care-based general practitioners)—this information would support subsequent improvement strategies and action planning where, it was increasingly recognized, participant groups would need to be targeted individually.
GA Hindle and LA Franco—Combining PSMs to conduct applied research
Table 1
1643
A sample of six innovation themes from the data set
‘Tuning’ the existing system: • Marketing view: This argues that marketing techniques can be used to improve the impact of the system upon both HCPs and the general public. Covers ideas such as targeting specific HCPs and medical conditions, collecting intelligence, improving presentation of medical standards, promoting medical standards to both HCPs and the general public. • IT-enabled view: This argues that IT can be used to speed up the operation of the current system and that databases could improve intelligence. Covers ideas such as using linked databases, collecting intelligence, using online reporting, achieving easy access to information, sharing data between stakeholders. The essence of this view is using IT to speed up decision-making, streamline processes and enable targeted activities. Could include IT systems presently operated by HCPs too. • Protocol view: This argues that FtD protocols are needed for certain HCPs (eg, occupational therapists, community matrons, social workers) in order to establish FtD appraisal in day-to-day work practices. The protocols would need to become part of existing frameworks, risk assessments and databases already in use by these HCPs. Alternative views (Weltanschauungen) for system design: • Legalistic view: This argues that legislation can be used to enforce the medical standards. Covers ideas such as introducing offences, fining for non-compliance (both HCPs and the patients), forcing HCPs to report patients, increasing the range of actors in the system (police, insurance companies). The essence of this view is to use pressure and threats—that is, upping the ante. Could involve targeting problem patients, using random checks, involve police, etc. • Supportive view: This argues that drivers with medical conditions should be supported by the DfT rather than policed. Covers ideas such as free mobility assessments, advice to drivers and family, flexible licences, concessions on alternative transport, facilitation of community-based solutions. The essence of this view is that loosing one’s licence should be a last resort and the focus should be on helping to keep people driving as long as possible. • Specialist view: This argues that evaluation of FtD is a specialist activity requiringspecialist assessors, rather than general medical opinion. Often associated with the view that functional testing should be used in conjunction with medical standards. Could involve accreditation processes (eg OTs) or trained regional doctors. Also takes the view that FtD evaluation does not fit with the ethos of general medical practice and that relevant patient traffic is too low for developing the expertise required.
3.3. Developing themes—‘tuning’ and alternative Weltanschauungen Analysis of the causal maps and the coded data from Step 1, taken as a holistic data set, revealed core themes regarding how the current arrangements to deliver medical standards might be improved. The themes are split into two sets, a sample of which are presented in Table 1. The first set represents ideas for innovating processes within the current operational system; we referred to this set as relevant to ‘tuning’ the existing system—for example, ideas for how to improve the promotion of medical standards to HCPs (BM Activity 6). These ideas were all therefore consistent with the current DVLA philosophy or Weltanschauung. The second set of themes is more fundamental and implies alternative philosophies and views for how to re-design the system; we referred to this set as ‘alternative Weltanschauungen’.
3.4. Further systems modelling Operating as an exemplification of systems modelling for the benefit of the client group, further systems modelling was undertaken to explore a sample of themes from both of the sets mentioned in Section 4.3. These efforts did not involve any further data collection as they sought to elaborate upon the logic of the themes and work up from the raw data captured in the workshops. For the ‘tuning’ set of themes, one theme was selected (the ‘marketing theme’) and one activity was selected from the baseline model (BM Activity 5—‘make MSs available to
HCPs and the public’). Subsystem models were constructed to capture ideas expressed in the workshops—such models represent ideal types of the kind advocated by Ackoff (1974). By this, we mean the models sought only to articulate highly desirable versions of the subsystem, rather than explore a full range of possible views, as might normally be the case within the SSM process. In other words, the modelling was undertaken in a creative system design mode, rather than a problem structuring mode. For the ‘alternative Weltanschauungen set of themes, one theme (the ‘supportive view’) was selected and system models were constructed to capture ideas expressed in the workshops—again ideal type models in the Ackoff sense. All models were systematically compared with existing arrangements using a formal comparison table (Checkland and Scholes, 1999) to highlight differences and ideas for improvement.
4. Discussion A number of issues have arisen from the particular mixed methods approach used in the research reported here. These are discussed below.
4.1. Mixing PSM techniques and tools in practice The project involved a direct comparison and interface between two types of modelling approaches—causal mapping and human activity system (HAS) modelling from SSM. As described earlier, the participant workshops used two periods
1644
Journal of the Operational Research Society Vol. 60, No. 12
of group causal mapping—one working with issues and a second working with ideas to improve the situation. Both types of data were ‘gathered’ using trigger questions and then ‘clustered’ into themes by the participants. Following the workshop, the raw unclustered data (ie each individual issue and each individual idea for improvement) was coded using the ‘baseline’ systems model. Hence, the raw data had effectively been structured in two different ways—one according to the logical hierarchy of causal mapping and the other according to the notion of a relevant human activity system. Further modelling using the HAS concept was then carried out using themes extracted from the data set; this enabled the themes to be explicitly linked to notions of ‘tuning’ the existing baseline system and alternative worldviews for system redesign. One of the benefits of using the soft systems model was that it provided a rigorous and explicit epistemology for the data analysis. Issues and ideas for improvement could be associated with a clear and explicit conceptualization of the main enterprise referent for the research project—that is, the current operational and administrative arrangements at the DVLA to deliver medical standards on FtD. The HAS modelling therefore enabled a precise and recoverable linking mechanism between the raw textual data captured through causal mapping and the research questions of the project. The causal mapping could have been developed further through a process of extending and combining the causal maps from the workshops. This would have involved the structuring of the data to achieve a map structure with ‘goals’ at the top of the hierarchy and issues, options and assertions towards the bottom. The primary difference between this approach and the systems modelling approach being the structure of the map would be established ‘bottom-up’ from the raw data, rather than the data being linked to an external systems epistemology. However, it would be erroneous to argue the casual mapping was free from an underlying hierarchical logic of its own. The project raised the question of whether a protocol might be needed for moving from a causal map to a HAS model. We were particularly concerned with what the logical interface between causal mapping and HAS modelling might be, as there are clear differences between mapping and systems modelling. As stated previously, causal mapping works up from the textual data, whereas HAS modelling imposes a systematic framework with which to structure the data and build upon it. In addition, causal maps can include a range of element types including goal statements, issue statements, subjective assertions, objective facts, and activities; whereas human activity system models only contain activities as elements. There are, however, some clear similarities between the elements and linking arrows used in causal maps and HAS models. The similarity is particularly striking with the type of action-orientated causal maps discussed by Bryson et al (2004). For example, the laddering-up and laddering-down
questions they use to build these action maps (ie what do you want to do, what might this lead to and how might you achieve this? (Bryson et al, 2004)) represent very similar triggers to those used for the PQR of root definitions in SSM (Checkland and Poulter, 2006). The view emerging from this project is that the process of linking a causal map to HAS modelling requires the same sort of selection of relevant systems as one would need when linking a rich picture to HAS modelling. This is because a causal map is likely to contain a rich and complex variety of data (just like a rich picture) and can therefore only be matched to the more precise and activity-orientated HAS modelling language through the subjective judgement of an observer. Hence, the opportunity for a straightforward or automatic linking protocol appears unlikely. Despite this, two linking opportunities do appear promising. Firstly, actionoriented causal maps of the type described above (Bryson et al, 2004) could link directly to ‘issue-based’ or ‘plan of action’ HAS models, and the authors have already begun experimenting with this type of causal mapping as a way of creating such models. The only difficulty being a causal map could easily contain more than one clear HAS and/or transformation process and therefore judgement may still be required as to systemic structure of the activity data. Secondly, a systemic textual analysis methodology of the type developed by Hindle (2007) could be used to analyse textual data within a causal map in order to explore the systemic structure of the causal map (see below). The main tool for capturing issue-based data in the workshops was the computer-supported mapping software. One of the limitations of this tool is the size of the screen containing the issues. As the number of issues captured increases (and the list can easily move past 100 issues in 15 minutes), the issue-based map is automatically ‘resized’ to fit the screen, which can impair visibility. As a result, the map often has to be sliced into visible ‘chunks’, which prevents the group from gaining a holistic sense of the issues and their interrelations. As stated earlier, we built a rich picture at the beginning of each workshop to familiarize participants with current FtD arrangements and capture an initial list of issues and points of view. Experience with the picturing tool, however, indicated an additional benefit: participants often asked us if they could use the picture as a reference point during the data capture step of the causal mapping. The participants found looking at the picture helped them to see the ‘big picture’ and enabled them to see where potential issues might be. It also acted as a kind of navigation tool for the fast emerging issue-based causal map. It is also likely that participants differed in terms of the utility they experienced in looking at the rich picture and the causal map. This difference is clearly experienced by the authors of this paper. One prefers to look at a rich picture as a prompt for issue identification and finds a large causal map to be confusing, whereas the other author prefers a causal map and/or a tree structure. More research is needed to clarify if
GA Hindle and LA Franco—Combining PSMs to conduct applied research
this preference is based upon expertise or familiarity with the tools (Mingers and Brocklesby, 1997; Keys, 2006), cognitive styles of the observer (Franco and Meadows, 2007) or some other variable. The further modelling highlighted the value of human activity system modelling as an epistemology for applied research. The modelling language enabled the logical consequences of data themes to be explored explicitly and in a recoverable manner. It also ensured this analysis was clearly linked to the baseline model developed earlier in the project. Hence, the modelling could extend into, and contribute to, the realm of analysis relating to actions to improve the situation. One of the key distinctions between this project and the type of client-based interventions the authors are more used to, is that we were working with stakeholders groups rather than decision makers. The workshops conducted in this project contained relevant, interested stakeholder groups, but many of them were not in a position to determine actual changes to the system. This meant the PSM techniques had to focus on the stimulation, collection and structuring of data on issues and improvements, but could not progress in facilitating the process of accommodation that would usually take place. In practice, this meant the ‘laddering-up’ within the group causal maps was hindered by lack of decision makers and further systems modelling had to be carried out by the authors through a process of summarizing the data themes.
4.2. The role and impact of computer-supported technology One advantage of the computer-supported mapping approach related to the portability of the equipment for conducting the research workshops. The use of laptops, a data projector and a large projector screen was all that was needed to run the research workshops, and thus we were able to transport the technology to wherever was convenient for the workshop participants. Locations for workshops were varied and included management boardrooms, university classrooms, small meeting rooms in mobility centres, and kitchen/lounge areas in teaching hospitals. Another advantage of the casual mapping software was its ability to capture large amounts of data quickly from a group of participants and establish a computer-based research database in real-time. One of the workshops, for example, generated more than 200 issue ideas in less than 20 minutes. In research terms, this is a highly efficient use of time and supports rigorous textual analysis of the data. Computer-supported groups were also able to enter their issues about FtD arrangements anonymously into the computer, which were then projected onto the public screen. This enabled both researchers and participants to judge the issues on their content rather than on who proposed them. As a result, participants were more open and vocal about the issues than would have otherwise been the case. Finally, the technology is user-friendly and requires virtually no participant training. Nevertheless, to facilitate
1645
effective workshop participation for those who were less experienced with computers, the facilitators provided extra support, and in some cases participants were paired-up on a single laptop. Participants with severe physical impairments or medical conditions (eg restrained mobility or sleep amnia) were able to actively participate in one of the research workshops.
4.3. Appropriateness of PSMs for conducting the research We have argued in this paper that PSMs can be used not only for organizational intervention but also to conduct applied research. Therefore, it is important to examine whether PSMs in general, and causal mapping and SSM in particular, indeed proved appropriate for conducting the research task confronted by the authors. Two aspects will be addressed here: complexity associated with the research problem, and project-specific conditions. The system which was subject to this research exhibited high levels of complexity akin to what is the normal sphere of application for PSMs. The complexity of the issues associated with the FtD system was partly derived from the presence of multiple actors with multiple and conflicting interests, and with different power bases; and partly from the interconnectedness of the issues and their dynamic behaviour. Most complex public or ‘wicked’ issues exhibit these characteristics (Rittel and Webber, 1973; Roberts, 2001; Weber and Khademian, 2008) and, viewed in this light, the use of causal mapping and SSM can indeed be seen as appropriate for the research task faced by the authors. Furthermore, provided that the systems under investigation share the same levels of complexity noted above, the experience reported here seems to be potentially transferrable to other PSMs. Feedback from the research sponsors was very positive regarding the usefulness of the approach adopted by the authors. This impact could have been due to the need of the sponsors to produce empirical evidence based on a robust scientific research methodology. By scientific, it was meant transparent, systematic, recoverable processes for data elicitation and analysis. Both causal mapping and SSM appear to have fulfilled this requirement. In addition, other members of the Warwick research team were struggling to integrate the findings coming from different parts of the research project, and the maps and systems models developed by the authors seemed to have helped the other Warwick researchers to gain a better appreciation of the ‘big-picture’. All these conditions proved to be a very receptive environment for the application of our mixed methods approach based on PSMs.
4.4. Validity of the research approach Finally, there is the issue of how to judge the validity of the mixed methods research approach presented here. It is widely recognized that the quality of qualitative research cannot be judged against the criteria used for assessing quantitative research (eg Reason and Rowan, 1981; Guba
1646
Journal of the Operational Research Society Vol. 60, No. 12
and Lincoln, 1989). Recently Shaw (2006), building on the work of Guba and Lincoln (1989), has proposed three main criteria for judging the quality of PSM-based research: credibility, dependability and confirmability. Credibility requires the data to accurately reflect the social construction held by the research subjects. Dependability is based on the notion of producing a well-documented and recoverable research process (Checkland and Holwell, 1998). Finally, confirmability focuses on the interpretations/results being firmly grounded in the research situation and/or research subjects being studied rather than in the researcher’s own interpretations. How does our mixed methods research fare against these three evaluation criteria? In terms of credibility, our workshop data comprised the collation of the individual understandings of different stakeholder groups about the issues affecting the DfT system. These individual understandings were shared, challenged and synthesized by the workshop participants themselves, using computer-supported causal mapping and supported by facilitation. It can be argued this process produces a shared social construction and research data that is richer, broader, deeper and more credible than data which might have been obtained through standard questionnaires and interviews. Although we used a consistent process for the collection of data during the workshops, questions of dependability of the research process might be raised in relation to the linking of coded (systems modelling) or structured (causal mapping) data to the resulting overall themes and, later on, to the further modelling. In practice, this was a process of summarizing the coded data and map clusters ‘by eye’, but more rigor could usefully be found. One approach would be to build systems models ‘bottom-up’ from the data in a similar way to that proposed by Hindle (2007) with a systemic textual analysis methodology. Another approach would be to further develop a protocol for mixing (see above). The coding of the data was also supported by the rigorous human activity system modelling language. This language enables a system-like referent to be clearly established—in this case the DVLA’s ‘system’ to deliver medical standards on FtD—and defined in detail using tools such as a root definition, CATWOE analysis and conceptual model. Finally, in terms of confirmability, two aspects of the research must be noted. On the one hand, the findings from the workshops were firmly grounded on the participants’ shared social constructions of the issues. To paraphrase Shaw (2006), we were taken beyond the raw data to nearer the findings stage of our inquiry about the DfT system. On the other hand, however, we had to rely on our own interpretations for the production of the overall themes. To avoid any potential biases and ensure confirmability, we considered valid only those themes that repeatedly emerged throughout the research process and gave rise to a ‘similar picture’ held by different participants and/or workshop groups.
5. Conclusions and directions for research Although problem structuring methods (PSMs) have been developed primarily to support organizational intervention, their use for research purposes is increasing. PSMs provide the researcher with group-orientated model-based techniques that support both a rich appreciation of a complex multiperspective problem situation, as well as the development of detailed action plans relevant to the research participants. In recent years, the mixing of management science methods is becoming a new paradigm of practice but, as far as the authors are aware of, there are little published accounts of their combined use for research purposes only. In this paper we have provided an account of such use. We used a mixed methods approach to researching the FtD system in the UK based on causal mapping and soft systems modelling. We facilitated a series of PSM-based workshops to gather research data which was structured first using mapping and subsequently coded using human activity systems modelling. The research approach helped the research participants, the research sponsor, and ourselves to collect, structure and record qualitative data as well as articulate and develop a number of options for improvement and alternative system designs. The value of a rigorous systemic epistemology to link data with explicit conceptualizations of real-world enterprise referents (such as the current arrangements to deliver medical standards on FtD) has been demonstrated in this project. Such an epistemology enables distinctions such as the ‘baseline’ system, ‘tuning’ the ‘baseline’ system and alternative system designs. Such an epistemology appears particularly important when dealing with large-scale processes or organizational units involving a complex network of stakeholders such as those found in the public sector. Similarly, the value of computer support in a research context has been demonstrated in this project. Of particular value are the portability of the equipment, the rapid development of a digital textual database, the anonymity of participant contributions and the straightforward nature of textual data input (for participants). Our reflections from the experience of deploying a mixed methods approach using PSMs are, of course, only exploratory as these are based on a single case study. Given the limited literature on mixed methods research applications noted above, we hope that this paper stimulates further work in this area. Some possible avenues for further research can be articulated. Firstly, it appears that a straightforward protocol covering the interface between causal mapping and HAS modelling is unlikely due to the rich and complex nature of a causal map. As with the link between a rich picture and HAS modelling, there is likely to be a range of possible relevant systems open to an observer. However, further research can focus on the potential development of a protocol to link action-orientated causal maps (containing mostly activities)
GA Hindle and LA Franco—Combining PSMs to conduct applied research
with HASs, using a systemic textual analysis methodology, such as the one developed by Hindle (2007) to analyse causal maps and develop HAS models. Secondly, it appears the rich picture technique from SSM may be a useful tool to use in parallel with causal mapping, although this is likely to be influenced by the cognitive styles of participants. The exploration of the relationship between cognitive styles and different forms of representation could provide a useful avenue for future research. Finally, how could different PSM techniques be combined within a single research project? Following Leech and Onwuegbuzie (2007), it is possible to decompose a mixed methods research design into two main areas of choice: time orientation and emphasis of use. Time orientation pertains to whether different problem structuring techniques are used simultaneously or sequentially within the research. In the case of the research reported here, we used causal mapping and systems modelling in a sequential manner. Emphasis of use relates to whether all the techniques used in the research have approximately equal status with respect to addressing the research questions, or whether any particular technique has significantly higher priority than the others (ie dominant status). In our case, although we collected the majority of the data using causal mapping workshops, the soft systems modelling element had a dominant status during most of the research. Future research could focus on alternative mixed methods designs such as, for example, simultaneous use and/or equal emphasis of techniques throughout a research project. Acknowledgements —The work reported here was part of a larger research project conducted by the University of Warwick on behalf of the UK Department of Transport. We thank Dr Carol Hawley (principal investigator, Warwick Medical School) and Professor Jonathan Rosenhead (Advisor to the Department of Transport, London School of Economics) for their support and feedback during our involvement in the research.
References Ackermann F and Eden C (2001). Contrasting single user and networked group decision support systems for strategy making. Group Dec Negotiation 10: 47–66. Ackoff R (1974). Redesigning the Future: A Systems Approach to Societal Problems. Wiley: New York. Blaikie N (1991). A critique of the use of triangulation in social research. Qual Quant 25: 115–136. Brown J, Cooper C and Pidd M (2006). A taxing problem: the complementary use of hard and soft OR in the public sector. Eur J Opl Res 172: 666–679. Bryson J, Ackermann F, Eden C and Finn C (2004). Visible Thinking: Unlocking Causal Mapping for Practical Business Results. Wiley: Chichester. Checkland P and Holwell S (1998). Action research: Its nature and validity. Syst Practice Action Res 11: 9–21. Checkland P and Scholes J (1999). Soft Systems Methodology in Action. Wiley: Chichester. Checkland P and Poulter J (2006). Learning for Action: A Short Definitive Account of Soft Systems Methodology and its Use for Practitioners, Teachers and Students. Wiley: Chichester.
1647
Denzin N (1989). The Research Act: A Theoretical Introduction to Sociological Methods, 3rd ed. Prentice-Hall: Englewood Cliffs, NJ. Eden C and Ackerman F (2000). Mapping distinctive competencies: A systemic approach. J Opl Res Soc 51: 12–20. Eden C and Ackermann F (2004). Cognitive mapping expert views for policy analysis in the public sector. Eur J Opl Res 152: 615. Erzberger C and Prein G (1997). Triangulation: validity and empirically-based hypothesis construction. Qual Quant 31: 141–154. Franco LA (2008). Facilitating collaboration with problem structuring methods: a case of an inter-organisational construction partnership. Group Dec Negotiation 17: 267–286. Franco LA and Meadows M (2007). Exploring new directions for research in problem structuring methods: On the role of cognitive style. J Opl Res Soc 58: 1621–1629. Franco LA, Cushman M and Rosenhead J (2004). Project review and learning in the UK construction industry: Embedding a problem structuring method within a partnership context. Eur J Opl Res 152: 586–601. Greene J, Caracelli V and Graham W (1989). Toward a conceptual framework for mixed-method evaluation designs. Educ Eval Policy Anal 11: 255–274. Guba E and Lincoln Y (1989). Fourth Generation Evaluation. Sage: Newbury Park, CA. Hindle G (2007). Developing a systemic textual analysis methodology based on the human activity system modelling language of soft systems methodology. Syst Res Behav Sci 24: 599–612. Hindle T, Checkland P, Mumford M and Worthington D (1995). Developing a methodology for multidisciplinary action research: A case study. J Opl Res Soc 46: 453–464. Huxham C (1996). Group decision support for collaboration. In: Huxham C (ed). Creating Collaborative Advantage. Sage: London, pp 141–151. Johnson RB, Onwuegbuzie AJ and Turner LA (2007). Toward a definition of mixed methods research. J Mixed Methods Res 1: 112–133. Keys P (2006). On becoming expert in the use of problem structuring methods. J Opl Res Soc 57: 822–829. Leech N and Onwuegbuzie A (2007). A typology of mixed methods research designs. Qual Quant, DOI: 10.1007/s11135-007-9105-3. Mingers J (2001). Multimethodology: Mixing and matching methods. In: Rosenhead J and Mingers J (eds). Rational Analysis for a Complex World Revisited: Problem Structuring Methods for Complexity, Uncertainty and Conflict. Wiley: Chichester, pp 289–310. Mingers J and Brocklesby J (1997). Multimethodology: Towards a framework for mixing methodologies. OMEGA 25: 489–509. Mingers J and Gill A (eds) (1997). Multimethodology: The Theory and Practice of Combining Management Science Methodologies. Wiley: Chichester, Mingers J and Rosenhead J (2004). Problem structuring methods in action. Eur J Opl Res 152: 530–554. Montibeller G, Shaw D and Westcombe M (2006). Using decision support systems to facilitate the social process of knowledge management. Knowledge Mngt Res Pract 4: 125–137. O’Brien F and Meadows M (2007). Developing a visioning methodology: Visioning choices for the future of operational research. J Opl Res Soc 58: 557–575. Onwuegbuzie AJ and Johnson RB (2004). Mixed method and mixed model research. In: Johnson R and Christensen L (eds). Educational Research: Quantitative, Qualitative and Mixed Approaches. Allyn and Bacon: Needham Heights, MA, pp 408–431. Ormerod R (1998). Putting soft OR methods to work: Information systems strategy development at Palabora. OMEGA 26: 75–98. Ormerod R (2005). Putting soft OR methods to work: The case of IS strategy development for the UK Parliament. J Opl Res Soc 56: 1379–1398.
1648
Journal of the Operational Research Society Vol. 60, No. 12
Pidd M (2004). Complementarity in systems modelling. In: Pidd M (ed). Systems Modelling: Theory and Practice. Wiley: Chichester, Reason P and Rowan J (1981). Humnan Inquiry: A Sourcebook of New Paradigm Research. Wiley: Chichester. Rittel H and Webber M (1973). Dilemmas in a general theory of planning. Policy Science 4: 155–169. Roberts NC (2001). Coping with wicked problems. In: Jones L, Guthrie J and Steane P (eds). International Public Management Reform: Lessons from Experience. Elsevier Press: London, Shaw D (2006). Journey Making group workshops as a research tool. J Opl Res Soc 57: 830–841. Shaw D, Ackermann F and Eden C (2003). Approaches to sharing knowledge in group problem structuring. J Opl Res Soc 54: 936–948. Tashakkori A and Teddlie C (eds) (2003). Handbook of Mixed Methods in Social and Behavioral Research. Sage: Thousand Oaks, CA,
Walsh M and Hostick T (2005). Improving health care through community OR. J Opl Res Soc 56: 193–201. Weber EP and Khademian AM (2008). Wicked problems, knowledge challenges, and collaborative capacity builders in network settings. Public Admin Rev 68: 334–349. White L (1996). Development options for a rural community in Belize: Alternative development and operational research. Int Trans Opl Res 1: 453–462. Winter M (2006). Problem structuring in project management: An application of soft systems methodology (SSM). J Opl Res Soc 57: 802–812.
Received January 2008; accepted August 2008 after one revision only
