Using groups to support judgmental parameter ... - Semantic Scholar

Expert Systems with Applications 40 (2013) 715–721

Contents lists available at SciVerse ScienceDirect

Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa

Using groups to support judgmental parameter estimation VISCONS: ‘Eyeballing’ to capture a quantified group consensus David Carter ⇑, Jonathan Moizer, Shaofeng Liu Plymouth Business School, Plymouth University, Drake Circus, Plymouth PL4 8AA, UK

a r t i c l e

i n f o

Keywords: VISCONS Parameterisation Group model building Visual elicitation System dynamics Police

a b s t r a c t Whilst the concept of a learning curve is well established, within organisational settings the coordinates of points on the curve often remain unmeasured. Devon and Cornwall Police are a UK police force that is seeking to better allocate officer resource to high volumes of crime demand through understanding the relationship between levels of officer learning and development and the time required to resolve crimes. Devon and Cornwall Police subscribed to building a system dynamics simulation model of officer resourcing, within which an aggregate learning curve would convert levels of available officer experience into time required to resolve volume crime. Fragmented learning curve data meant that judgmental data needed to be elicited from expert practitioners in order to establish a consensus based measure of bivariate relationships. VISCONS is a judgmental method that was developed to help Devon and Cornwall Police quantify these relationships in a group setting. Three components are synergised into the VISCONS approach, namely: visual or graphical elicitation, parameterisation and consensus through group model building. The method brings together expert practitioners to arrive at a single shared view of the quantified bivariate relationship from a set of individual perspectives. Experts are involved in sketching their views onto acetates before overlaying them into a set of agreed parameter values. Experts do not require any knowledge of system dynamics simulation techniques, as they are facilitated in a single workshop to represent and amalgamate different views and perspectives on the parameterisation problem under consideration. The group is able to validate resulting bivariate parameter values using an efficient graphical method of data capture. VISCONS can be used for parameter capture beyond the scope of learning curves or system dynamics. Ó 2012 Elsevier Ltd. All rights reserved.

1. Introduction Devon and Cornwall Police (DCP) are a UK police force that seeks to continuously improve how officer resource is allocated to efficiently and effectively address demand for assistance from members of the public. This demand can be classified as local assistance required within neighbourhoods, a stream of incidents requiring patrol officer response, or reported crimes requiring investigation. Patrol officers are deployed to respond to real time incidents, but also investigate routine, low-level (termed volume) crime, post-event. For statistical purposes and issues of practicality, crimes are labelled as either high volume or by specialisation of investigation. High volume crime is allocated to the patrol officers for investigation, whereas other crimes are investigated by specialised units. If officer resources are to be allocated efficiently and effectively for solving crime, it is important to understand the relationship ⇑ Corresponding author. Tel.: +44 (0) 1752 585635; fax: +44 (0) 1752 585633. E-mail addresses: [email protected], [email protected] (D. Carter), [email protected] (J. Moizer), [email protected] (S. Liu). 0957-4174/$ - see front matter Ó 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.eswa.2012.08.015

between the level of an officer’s capability and the time committed to each crime type (category). One way to establish the resource needs of the patrol officer function is to aggregate a series of estimations of time committed by officers to dealing with the types of crime most commonly encountered. It is commonly accepted that officers with a higher level of training and experience are swifter in resolving a given crime, i.e. they are higher up the ‘learning curve’ – a concept originated with Wright (1936). By observing work productivity in the aircraft industry, he established that the direct labour cost for hours paid was directly related to levels of experience gained in producing airframes through an elastic, non-linear, relationship). At the aggregate level, using the concept of a learning curve, police forces are able to visualise overall officer capability and how well volume crime demand can be met. DCP were seeking to understand how better to allocate officer supply to volume crime demand. They recognised the utility of simulation modelling as an approach to exploring and predicting the effects of policy change on the patrol officer function and the potential impacts on how volume crime was addressed. It was agreed to construct a system dynamics (SD) model which captured

716

D. Carter et al. / Expert Systems with Applications 40 (2013) 715–721

both the processes for developing recruits into fully capable patrol officers, as well as managing crime demand. SD is a structurally orientated simulation modelling approach that has its origins in the work of Forrester (1961). Its application to real world management problems has since been extended to cover a range of investigations, including the evaluation of strategic knowledge management systems (Yim, Kim, Kim, & Kwahk, 2004). Interactive model building was the preferred approach to constructing the SD model as it allowed judgmental measures to be more easily captured from problem owners. Expert practitioner views were elicited in group settings through a series of scripted modelling workshops. The patrol officer supply chain process was translated into stock-flow notation to represent the flow of officer resource available for resolving volume crime. In order to translate aggregate officer capability into numbers of crimes that could be addressed, it was imperative to solicit expert judgment in order to parameterise the model to reflect the impact that the officer learning curve had on the force’s ability to deal with volume crime. This approach was necessary as the requisite data was not available elsewhere in a suitable form for parameterizing a SD model. As a significant proportion of patrol officer demand is based on investigating volume crime, officer learning capability represents a key parameter in this simulation model. Consequently, a rigorous approach to numerical validation was necessary to allow for any model constructed to predict system behaviour and robustly explore policy alternatives. This paper seeks to illustrate the development and application of a group model building (GMB) approach to capturing bivariate parametric relationships in SD models. The research objectives are to:  outline how a graphic frame methodological approach (VISCONS) was used for group based parameter estimation within law enforcement modelling; and  discuss the utility and effectiveness of the VISCONS method as a consensus based judgmental parameterisation approach.

2. Background Patrol officer capability can be viewed as a soft and fuzzy factor within a policing performance system. To define capability in a patrol officer context is challenging, not least because there are no commonly agreed metrics. One way to explore capability is to link officer development stages to crime resolution time via a learning curve that recognises that on average, officers with more experience are able to resolve volume crime incidents in a shorter period of time than those who are newer to the profession. Statistics for such bivariate relationships are not readily available within UK policing, hence it was necessary within DCP to assemble a group of policing practitioners to provide judgmental estimates of these relationships and arrive at a set of group consensus values. To arrive at a set of consensus values, it would be necessary to engage this group in parameterizing elements of a system model without having any prior knowledge of the technique, its functionality or utility. Within this GMB setting, an aggregate learning curve would be constructed in the shortest possible time. Due to the challenges of organising this group that had been convened from multiple locations, and the time constraints on individual availability; a two hour window was agreed in advance with the group participants, so an efficient process and supporting infrastructure was required if the intended outcomes of the group exercise were to be achieved. This time window was considered to be viable as the concept of learning curves was already known to the participants but associated data had stubbornly remained

‘not collected’ (Fey, 1993, p. 117) despite considerable management information being regularly produced within the Force. 2.1. Judgmental data capture from groups for system dynamics parameterisation Within more complex systems, causal relationships between variables are often ill defined and where validation is sought, a straightforward method of numerical data collection and parameterisation may not be sufficient. Often where multiple role actors exist who hold different perspectives, more judgmental approaches to data elicitation for SD models may be required. This can entail (particularly with non-systems experts) the use of visual techniques to engage groups of expert practitioners in translating qualitative views into quantitative measures of multivariate relationships. Under such conditions, three key components may need to be synergised into a parameterisation methodology, namely: SD parameterisation, GMB consensus and visual elicitation. Fig. 1 outlines a Venn diagram which captures the overlaps between individual components. These components will now be discussed. 2.1.1. Group model building consensus Group model building (GMB) is a methodology widely used in SD work for building consensus amongst modelling participants. SD has seen a vast wealth of GMB applications (see for example, Andersen, Richardson, & Vennix, 1997; Andersen, Vennix, Richardson, & Rouwette, 2007; Howick & Eden, 2011; Rouwette, Korzilius, Vennix, & Jacobs, 2010; Vennix, 1996). Group size can vary from numbers fewer than 10 participants up to as many as 100 or more. In large groups, specialists might contribute to break-out workshops for model building around their specific areas of knowledge. Typically, as part of the modelling process, group members are inducted into elements of the SD methodology (i.e. stocks, flows, convertor variables and causal feedbacks), and often participate in constructing or advising on the development of model structure using consensus based views of the group. By taking groups of experts through this GMB process, participant benefits include

Numbers for initial values Quantify system variables

Group interaction Expert views Shared understanding

SD parame

GMB

-terisation.

consensus

Visual (graphical) elicitation

Reference modes/ BOT charts Sketch related variables Fig. 1. Venn diagram of methodological components of a judgmental parameterisation approach.

D. Carter et al. / Expert Systems with Applications 40 (2013) 715–721

arriving at policy insights and predictions through shared understanding of causal structures and a level of confidence in the model based on using the right information for the specific problem being modelled. 2.1.2. Visual (graphical) elicitation Visual elicitation, based on graphical techniques, is normal at the conceptualisation stage of a system dynamics model building exercise. This comprises identifying the referent problem or a reference mode of behaviour of one or more key variables (Sterman, 2000). In group settings this can be challenging, particularly if messy problems are being addressed where there is a level of disagreement amongst participants (Vennix, 1996). Beyond identifying reference modes, visual techniques can also be used with problem owners and experts to sketch relationships between the elements of a system dynamics model or the behaviour of model outputs. This is core to GMB methodologies. Reference modes or behaviour over time (BOT) charts are important as they can not only identify a dynamic problem, but can also be used to think through alternative futures with groups of participants who may hold different initial views (Elias, 2011). As distinct from sketching BOT, table functions provide a graphical input capability within a SD model that can directly represent non-linear, bi-variable relationships. They allow the model builder (as well as those contributing) to connect different parts of the model together using graphs without the need for complex formula describing polynomial relationships. Such ‘table functions’ use specific X–Y values to plot the relationship between two variables and can provide a fast and direct route to specifying a causal relationship over writing out the equivalent polynomial equation. In effect, the table function provides users with a visual representation plotting specific values on a grid. Given sufficient expert knowledge, in certain cases it may be possible to specify the shape and scale of such a relationship (Shi & Gill, 2005), Where such knowledge is shared within a group, enumerating this can present challenges, not least because participants are required to translate qualitative mental models directly into quantitative causal relationships in one step. Other visual methods such as hexagon mapping (Hodgson, 1992) can create a bridge between mental models and system dynamics modelling by taking groups through from model conceptualisation to simulation modelling. By capturing headlines on magnetic hexagons, associations are easily made and re-made as the process develops to represent a summative visual memory of group views. Not only does the technique of clustering hexagons make group views explicit, by using colour coded hexagons to represent individual styles of thinking, it is possible to colour balance the cognitive map by encouraging those styles still missing from participant contributions. Hodgson explains that conceptual mapping increases the brain’s capacity to handle complexity while enabling groups to share thinking. Where ‘‘time and mental energy are too scarce. . . the power of visual facilitation comes in’’ (Hodgson, 1992, p. 229). 2.1.3. System dynamics parameterisation Within the SD methodology, parameterisation of any stock-flow diagrams occurs once the full model structure is specified. Parameterisation is the process by which realistic numbers are entered into the SD model in order to represent key system variables i.e. measurable quantities associated with the real-world problem under consideration. Parameterisation is an important step preceding calibration and simulation of alternative policy options within the model. Where formalised data is available, this can be readily used to provide estimates of typical values, but where such data is scarce then it is necessary to elicit judgmental data from expert practitioners

717

(Lane, Monefeldt, & Husemann, 2003), often in a group setting (Van den Belt, 2004) and using approaches such as the Delphi method (Luna-Reyes & Andersen, 2003). 2.1.4. Alternative views’ of Venn diagram overlaps Richer and more robust model validation techniques can be employed in the parameterisation process where components within the Venn diagram (Fig. 1) overlap. Various authors have used methodologies that overlap more than one of these components. Both graphs and matrix grid approximations have been used before to elicit numerical parameters from individuals (Akkermans, 1995). Sterman and Ford (1998) and Sterman (2000) outline a visual elicitation method to formulate non-linear relationships in SD workshop settings which they label as the ‘graphic frame’ (a precedence relationship). Sketching bivariate relationships within the group is a technique that Lee, Zagonel, Andersen, Rohrbaugh, and Richardson (1998) also suggest when parameterizing table functions within a group based system dynamics model building exercise. Mohapatra, Bora, and Kailas (1984) use Delphi elicitation of graphed shapes to interact with group of participants when establishing SD model table function parameters. Wolstenholme and Corben (1994) propose a GMB approach to system dynamics model parameterisation where, as part of the methodology, experts are asked to sketch the trajectory for their individual learning over time. Using visual graph based questions, participants provide sketched responses. Within this methodology, all three components of the Venn diagram are asynchronously covered. By contrast, a synchronous approach that combines all three components may be required for situations where face to face interactivity is necessary and group learning time needs to be condensed in order to achieve a shared perspective on parameter values. 3. Parameter estimation method: VISCONS To be able to model the level of patrol officer resource required to resolve volume crimes within DCP, the relationship between levels of officer experience and resolution time (referring to the continuous time taken for a patrol officer to identify a suspect; after this point, the process moves beyond the patrol function – thus, out of model scope – where crime is dealt with by other policing units) per crime type needed to be determined and a method was designed to elicit relevant judgmental data in a single workshop setting: VISCONS (i.e. ‘Visual Consensus’). Sketching a curve ‘live’ with participants is considered to be a difficult phase of any parameter elicitation process (Lee et al., 1998), and consequently, a structured technique to sketch curves in a group setting is essential. VISCONS engages participants in sketching graphical relationships between variables within a pre-specified, matrix grid in order to arrive at consensus parameterisation through aggregating together individual views. The previous Delphi approaches to capturing system dynamics parameters are asynchronous and therefore were unsuitable for this time constrained exercise. VISCONS is, on the other hand, synchronous where the method is pursued with the group in real time. VISCONS provides a fast and efficient process for gathering expert judgment from group participants, The approach was designed to avoid recognised GMB problems (Balci, Nance, Arthur, & Ormsby, 2002) such as vagueness, incomplete evaluations and unsequenced contributions associated with asking subject matter experts. Rather than asking individual experts to asynchronously contribute their views by explaining relationships between parameters such as those captured in different learning curves, ambiguity is challenged within the VISCONS process, and results accounted for by means of synchronous peer group review and agreement.

718

D. Carter et al. / Expert Systems with Applications 40 (2013) 715–721

By using acetate sheet contributions coupled with a pre-determined grid to support sketched relationships, VICSONS allows rapid contribution and review between rounds of result aggregation. VISCONS represents the ‘Visual Consensus’ that can emerge from GMB exercises. In this case, the method has been applied to establishing measures of patrol officer experience against volume crime resolution time in order to arrive at a learning curve. This method specifically allows groups of experts in a workshop setting to independently generate views of the shape and scale of such a bi-variable relationship then move towards a combined view. By employing a common grid or ‘graphic frame’ (Sterman & Ford, 1998) for use by all participants, it is possible to use a simple bottom-up approach to visually achieve a consensus view at aggregated levels. Fig. 2 illustrates the workshop process, from selecting participants, to delivering a briefing, then capturing expert data, through to aggregating plots of bivariate relationships. 3.1. Application of VISCONS to DCP Case The four sequential steps in the VISCONS process are now described as a case study of its use to help generate quantified learning curves for officers within DCP. 3.1.1. Select expert group In the DCP case study, a group of eight experts were invited to comment upon patrol officer capability, and attended a VISCONS workshop to assist with parameterisation of elements of the patrol officer system dynamics model under construction. They provided a pool of mental models which characterised the dynamics of officer capability. These thoughts were captured through a formal graphical process of elicitation. These individuals provided a relevant pool of expertise and were drawn from DCP training managers, core skills trainers and other mid-ranking officers located across the force’s geographically distributed training centres. 3.1.2. Brief workshop The workshop was initiated by briefing participants on how they would be individually and collectively assisting with developing a series of learning curves to characterise patrol officer productivity in DCP. The term ‘learning curve’ was explained and a generic acetate grid template was introduced as the data capture mechanism. 3.1.3. Capture data The initial stage of the knowledge capture activity involved the prioritisation of volume crime types typically addressed by patrol officers across the DCP Basic Command Units (BCU – a Basic Command Unit is a geographic area within a police force. At the time of the VISCONS exercise, DCP had four BCUs, comprising Cornwall and Isles of Scilly, Plymouth, South West Devon, and North East Devon). A list of fifteen types of volume crime was presented to the group, and they were asked to collectively rank how frequently

Fig. 2. Sequence of steps for VISCONS process (n represents the number of levels of aggregation for captured data i.e. n = 2 in the case study described).

their patrol officers dealt with each type (although volume crime statistics are produced by DCP, insufficient detail was available to determine individual characteristics). Within the group, the first five categories were readily decided upon, and after some debate, the sixth was agreed. Beyond this point, there was divergence of opinion for those latter ranked crime types. Consequently, only the top six volume crime categories would be used to arrive at an agreed learning curve for patrol officers resolving volume crime. A pre-formatted acetate grid had been designed to capture specific learning curves for each crime type. The development stages for patrol officers were represented on the x-axis (as pre-determined categories), and time taken to resolve crime was set on the y-axis. Both axes scales were delimited in order to constrain the plotted curves within specified ranges and assist with their subsequent comparative analysis. Next, the participants were supplied with the acetate grids and were instructed to use the grid to quantify the time taken to resolve each of the six chosen crime types against levels of patrol officer experience by plotting points on charts. A symbol was allocated to each individual for use when plotting their results, so that their contribution to learning curve estimation could be compared (i.e. overlaying the different responses) on the acetate in association with their geographic areas of responsibility (BCU) for each specific crime-type. Participants were then asked to plot their symbol on the grid at the appropriate coordinates to delineate the level of officer experience against the time taken to resolve the original crime investigation. They also circled both the specific category of volume crime along with their own BCU as identifiers in the next aggregation stage. Fig. 3 shows a completed acetate grid for the crime of harassment which was completed by a police trainer in the South West Devon (SWD) BCU. In this example the learning curve illustrates a non-linear relationship between time taken to identify a suspect and levels of officer development. The learning curve is steeper during the earlier officer development stages, and slowing with experience for this type of crime in SWD. 3.1.4. Aggregate results In a similar way to Lee et al. (1998), the facilitator sketches resulting relationships according to participants’ views. This is achieved using an overhead projector where the acetate grids containing graphical estimates from the eight participant contributions were overlaid with the purpose to stimulate a discussion about the most representative curve for a given crime type. A single aggregate learning curve for each crime type was agreed amongst the participants (compressing from 8:1) through visual comparison (‘eyeballing’ the variations plotted) and so arriving at a consensus view through discussion and debate. Here, individuals queried other participants’ assumptions alongside providing a rationale for their own views before deciding on the preferred curve. This process was repeated for each crime type under discussion resulting in six consensus-based aggregate learning curves being produced. Subsequently, the six aggregated learning curves from individual crime types were further condensed to provide a single plot (compressing from 6:1) to describe the overall relationship between patrol officer experience and time taken to resolve a volume crime incident. This curve was circulated to the group participants following the workshop for any further comment. Fig. 4 provides an example of aggregated participant plots for harassment crime resolution across all DCP BCUs. In this example, the learning curve illustrates again a non-linear relationship between time taken to identify a suspect and levels of officer development. The learning curve differs from individual earlier plots as it is now flatter during the earlier officer development stages and steeper during latter stages. Based on shared understanding, the aggregate curve for the crime of harassment across DCP as a whole can now be now represented.

D. Carter et al. / Expert Systems with Applications 40 (2013) 715–721

719

Fig. 3. Actual example of individual participant plot showing estimated learning curve for harassment crime resolution in SWD Basic Command Unit.

This re-sketching exercise encouraged full engagement of all participants, despite them initially having no intuitive quantification that would explain the shape of a single, overall learning curve. Using this multi-step, bottom-up approach to parameterisation, the participants were able to confirm a set of aggregated numerical estimates for a bivariate relationship between two variables as a learning curve based on real time feedback within the group. Within this policing vignette, two levels of aggregation were utilised to enable the participants to understand their synthesised learning curve for volume crime resolution across the DCP force. Neither the size of the contributing group, nor the scale of aggregation applied (total compression from 48:6:1) acted as limiting factors for the real time VISCONS process.

4. Discussion and conclusion During the two hour workshop, the VISCONS method was successful in eliciting individual and group views on parameter values from a group holding specific tacit information. This represents a time-based improvement over asynchronous techniques such as Delphi that are used for eliciting bivariate relationships. It also offers time savings over other synchronous techniques such as consultant led GMB which can take considerably longer, depending on the scope and size of the problem being addressed. Of the pre-

existing methods used to parameterise SD models with groups, many techniques take place over several days rather than hours, thanks in part to a perceived need to explain SD in more detail to participants. Rather than building a complete SD model with the participants, VISCONS targets the extraction of causal translations between inputs and outputs known to these experts. For eliciting relevant information under time constrained conditions participants can contribute their specific knowledge of the problem without having to understand SD modelling techniques. By using a graphical approach, even better use of participant time can be achieved, not least due to easier assimilation of visual rather than numerical information. The use of overhead transparencies that ‘layer’ information was critical in explaining group views of the bivariate relationship at different stages of consensus building. It proved valuable in maintaining a continuous ‘white box’ view of the aggregation journey for both participants and observers alike; acting as a persuasion for managers who were not selected to comment. Once the visual process has been used and understood by participants, it offers a convincing aggregation mechanism which allows easy translation of values into the final system dynamic model at the appropriate level of detail. VISCONS allows participants to repeat the graphical elicitation step at each level of aggregation where multiple learning curves are compressed into one. This repetitive visual process of

720

D. Carter et al. / Expert Systems with Applications 40 (2013) 715–721

Fig. 4. Actual example of aggregated participant plots showing estimated learning curve for harassment crime resolution across all DCP Basic Command Units.

aggregation helps participants to concentrate on providing realistic data adjustments to each aggregated learning curve, rather than focusing on a numerical integration of the contributing data sets; the result being a more verisimilitudinous representation of a scaled, bivariate relationship. It helps participants to crystallise their thinking through use of a visible and transparent graphing technique. During the VISCONS exercise, participant views were moderated by group feedback, where individuals were challenged to justify their choice of parameter values. This group dynamic with high levels of participant engagement was notable at each aggregation step within the parameterisation process. The VISCONS parameterisation process for establishing the bivariate relationship is simple to follow and repeats itself for each level when scaling up towards higher levels of aggregation. This is achieved under full scrutiny and with the opportunity for full engagement from all participants, independent of group size. VISCONS is therefore not limited by group size as all contributions happen in parallel; so for example, representing 10 or 100 plots on a single aggregation graph would be equally viable to explain where at a consensus level the bivariate relationship is situated (visual aggregation using computers may benefit the process of collating larger data volumes however). VISCONS represents a method for estimating parameters used within, but not limited to, system dynamic models. Although, no

participant knowledge of system dynamics modelling is needed to assist with parameterizing bivariate relationships of interest, group validation of the VISCONS outcomes is essential. Future work can extend the VISCONS method to involve participants that are geographically dispersed through employing online teleconferencing to capture individual estimates, debate and discuss differences, and combine these inputs to produce aggregate consensus based outputs. Equally sketch sharing, computer-based communications may also offer further opportunities to effectively ‘shrink’ distance. Further opportunities exist for eliciting causal relationships from participating experts using visual simulation of the real system in which they work (Robinson, Lee, & Edwards, 2012). This could help condense the process of simulation model development by allowing contributors direct access to the abstracted model, rather than having to go through the intervening stage of problem structuring using acetates for knowledge capture.

References Akkermans, H. (1995). Developing a logistics strategy through participative business modelling. International Journal of Operations and Production, 15(11), 100–112. Andersen, D. F., Richardson, G. P., & Vennix, J. A. M. (1997). Group model building: Adding more science to the craft. System Dynamics Review, 13(2), 187–201.

D. Carter et al. / Expert Systems with Applications 40 (2013) 715–721 Andersen, D. F., Vennix, J. A. M., Richardson, G. P., & Rouwette, E. A. J. A. (2007). Group model building: Problem structuring, policy simulation and decision support. Journal of the Operational Research Society, 58(5), 691–694. Balci, O., Nance, R. E., Arthur, J. D., & Ormsby, W. F. (2002). Expanding our horizons in verification, validation, and accreditation research and practice. In Proceedings of the 2002 winter simulation conference, San Diego, California. Elias, A. A. (2011). A system dynamics model for stakeholder analysis in environmental conflicts. Journal of Environmental Planning and Management, 55(3), 387–406. Fey, W., & Trimble, J. (1993). An expert system to aid in model conceptualization. In Proceedings of the international system dynamics conference, Cancun, Mexico. Forrester, J. (1961). Industrial dynamics. New York: MIT Press, John Wiley and Sons Inc.. Hodgson, A. M. (1992). Hexagons for systems thinking. European Journal of Operational Research, 59(1), 220–230. Howick, S., & Eden, C. (2011). Supporting strategic conversations: The significance of a quantitative model building process. Journal of the Operational Research Society, 62(7), 868–878. Lane, D. C., Monefeldt, C., & Husemann, E. (2003). Client involvement in simulation model building: Hints and insights from a case study in a London hospital. Health Care Management Science, 6(2), 105–116. Lee, T. P., Zagonel, A., Andersen, D. F., Rohrbaugh, J. W., & Richardson, G. P. (1998). A judgement approach to estimating parameters in group model-building sessions: A case study of social welfare reform in Dutchess County. In Proceedings of the international system dynamics conference, Quebec, Canada. Luna-Reyes, L. F., & Andersen, D. L. (2003). Collecting and analyzing qualitative data for system dynamics: Methods and models. System Dynamics Review, 19(4), 271–296.

721

Mohapatra, M. K. J., Bora, M. C., & Kailas, C. S. (1984). Incorporating Delphi results in system dynamics models: A case of Indian tea industry. Technological Forecasting and Social Change, 25, 159–177. Robinson, S., Lee, E. P. K., & Edwards, S. J. (2012). Simulation based knowledge elicitation: Effect of visual representation and model parameters. Expert Systems with Applications, 39(9), 8479–8489. Rouwette, A. J. A., Korzilius, H., Vennix, J. A. M., & Jacobs, E. (2010). Modelling as persuasion: The impact of group model building on attitudes and behaviour. System Dynamics Review, 27(1), 1–21. Shi, T., & Gill, R. (2005). Developing effective policies for the sustainable development of ecological agriculture in China: The case study of Jinshan County with a system dynamics model. Ecological Economics, 53, 223–246. Sterman, J. D. (2000). Business dynamics: Systems thinking and modeling for a complex world. New York: McGraw-Hill/Irwin. Sterman, J. D., & Ford, D. N. (1998). Expert knowledge elicitation to improve formal and mental models. System Dynamics Review, 14(4), 309–340. Van den Belt, M. (2004). Mediated modelling: A system dynamics approach to environmental group consensus building. Washington, DC: Island Press. Vennix, J. A. M. (1996). Group model building: Facilitating team learning using system dynamics. Chichester: Wiley. Wolstenholme, E. F., & Corben, D. A. (1994). A hypermedia-based Delphi tool for knowledge acquisition in model building. Journal of the Operational Research Society, 45(6), 659–672. Wright, T. P. (1936). Factors affecting the cost of airplanes. Journal of Aeronautical Sciences, 3(4), 122–128. Yim, N. H., Kim, S.-H., Kim, H.-W., & Kwahk, K.-Y. (2004). Knowledge based decision making on higher strategic concerns: A system dynamics approach. Expert Systems with Applications, 27, 143–158.