Key Performance Indicators (KPIs) are suggested .... Consequences modeling was done using software tool DNV PHAST ..... implemented within the company's Safety Manage- ment System. Considering the rationale for the development of.
Key Performance Indicators and Bayesian Belief Network based risk model as a management tool – results from the case study Marko Gerbec & Branko Kontić
Jožef Stefan Institute, Jamova 39, 1000 Ljubljana, Slovenia
ABSTRACT: Key Performance Indicators (KPIs) are suggested to be used as an operational tool for the purpose of managing safety (risks) in safety critical operations. The issue related to the KPIs is a selection of the practical and meaningful data that clearly represent chosen component of safety (level of risk), considering that the root causes for accidents are failures, mistakes and deficiencies of the equipment, humans and organizations. Within the EU project iNTeg-Risk (http://www.integrisk.eu-vri.eu/) the specific approach for the implementation of the KPIs was prepared (named "ERRA D1") in order to connect the underlying risk model (using BBNs) and related information on the safety related performance. The authors were involved in the project's testing and validation case study of the proposed approach carried out at the selected operations in the port of Koper, Slovenia. The conclusions on testing the approach are that the approach is sound and ready to be used; however, complexity and required additional resources may be a weak point. In addition, the remaining barriers for the success were identified. 1 INTRODUCTION Key Performance Indicators (KPIs) are suggested to be used as an operational tool for the purpose of managing safety (risks) in safety critical operations. The issue related to the KPIs is a selection of the practical and meaningful data that have an impact on the level of safety (level of risk), considering that the root causes for accidents are in failures, mistakes and deficiencies at the levels of technology equipment, humans and organizations. The relationships among all three levels are usually not simple and easy to recognize, thus for the selection of the safety indicators of relevance a risk model needs to be prepared based on the risk assessment of the given operation or technological process. Based on the risk model a KPIs shall be selected and implemented in an industrial organization. Within the EU project iNTeg-Risk (http://www.integrisk.eu-vri.eu/) the specific approach for the implementation of the KPIs was prepared (named "ERRA D1 - Definition of KPIs for emerging risks for selected industry case studies, including CSR aspects of emerging risks") in order to connect the underlying risk model and related information on the safety related performance. The proposed approach builds on the elaboration of the risk model, considering the consequences and building probability model using Bayesian belief networks enabling consideration of complex interac-
tions among input and output nodes. The KPIs shall enable to collect inputs to modeling the risks and show sensitivity to the anticipated root causes mentioned above. The authors were involved in the project's testing and validation case study of the proposed approach carried out at the selected operations in the port of Koper, Slovenia. For the purpose of testing of the ERRA D1 method in Luka Koper d.d. the consultations with their personnel resulted in the selection of the methanol unloading operation from the tanker ship at the liquid cargo terminal. 2 RISK ASSESSMENT Luka Koper d.d. performs operations on unloading & loading of large quantities of hazardous materials. In that respect it is legally obliged by the related national legislation implementing the EU Seveso II directive (96/82/EC and amendment 2003/105/EC) as upper tier establishment. Following this, the existing Safety Report for Luka Koper d.d. operations (Luka Koper, 2008) was used as a starting point. For the purpose of ERRA D1 method testing the following information from the Safety Report is important: Loading and unloading operations at the liquid cargo terminal are recognized as one the major accident hazard sources in Luka Koper. Among the various hazardous substances, also the
methanol is handled, being a flammable and toxic substance. Related to the operation of unloading and storage of methanol from the tanker-ship, the Safety Report recognizes as a potential source of methanol release the releases from the process equipment (pipelines, joints, etc.) or external disturbances to the terminal-tanker assembly (collision of the other ship, move of the tanker). Following this initial information, a detailed interviews with the liquid cargo terminal personnel were carried out, as well as consideration of the relevant documents, such as Management provisions, work instructions and operational records. Result of the risk assessment performed is a detailed description of the potential scenario of the major accident at the unloading of the methanol at the jetty (description given here is related to the Safety Report, Luka Koper, 2008): The major accident scenario related to the release of the methanol during the tanker-ship unloading operation to the liquid cargo tank farm considers both terminals' internal equipment failures and mistakes/failures outside the terminal affecting its equipment. The unloading operation typically involves about 10,000 tons of methanol, lasts for about 16 hours and on average there are about 10 shipments received annually. The terminal internal failures consider the releases of the methanol during the unloading the tanker into the tank farm. This goes from the ship's connection to the west jetty, considering the pipelines, pipeline joints (flanges, where apply) and the mechanical unloading arm (jetty to ship). The terminal external failures consider all overloads/challenges applied to the unloading arm, and being presumably unauthorized tanker-ship move during the unloading operation or collision of the second ship into the tanker during the passage in the second channel (just north of the second pier) and leading to the damage to the tanker tied to the terminal's jetty. After the release of the methanol from the process equipment (pipeline, joint, unloading arm) it is expected that it will last at least up to 1 minute (being the realistic minimum response time of the personnel on the ship) and that it is from the DN300 size pipe ("full bore rupture"). The release from the pipeline in the direction of the tank farm is not expected as the entry points are equipped with the non-return valves. The released methanol can reach land (directly at the jetty) or sea (at the ship-jetty interface) surfaces and forms a liquid pool/layer and a flammable mixture with ambient air. The flammable methanol-air mixture (cloud) can be ignited immediately (e.g., by the static electricity) or later (ignition source can be electrical equipment out-
side the Ex-zone, or even ship's engines), leading the flash fire, pool fire, jet fire and possibly also to the vapor cloud explosion. Beside all the fire phenomena just mentioned, it shall be noted that methanol is also a toxic substance (this is important for the personnel present and members of the emergency response teams), however, we will here consider only consequences of the fires as they are important for assessment of direct damage on the process equipment and related economic consequences of the accident. Thus, in the case of the accident a large fire and a potential vapor cloud explosion can be expected, leading to the injuries of the personnel present (on the land and on the ship) and damage and destruction of the process equipment of the liquid cargo terminal and on the tanker-ship. 2.1 Risk model Risk model was prepared, consisting of the consequences model and of the model of the likelihood of the major accident scenario under consideration. 2.2 Consequences modeling Consequences modeling was done using software tool DNV PHAST Version 6.53 consisting of the material properties database and integrated models of the releases, dispersion in the air, hazardous phenomena, and consequences. During the model preparations, the following was considered: During unloading operation the overpressure in the delivery line at the exit from the tanker is about 6 bar, pipeline dimension is most of the time 300 mm, pipeline length of concern is about 20 m, duration of the release is 60 s. the ignition source is anticipated at a distance of 50 m from the source The dispersion modeling for the representative meteorological summer and winter situations for the site, revealing that summer and 2 m/s wind with stability category D is deemed slightly worse than others, considering the largest impact distances. The conclusions regarding the potential consequences are: Average methanol release rate is estimated at about 535 kg/s, resulting in about 32 tons released within 60 seconds. The spilled methanol pool formed over the close to the jetty would reach radii up to about 50 m; ignition of the flammable vapor cloud is viable up to the distance of about 60 m from the release point. In a case of methanol ignition the damage to the process equipment can be expected up to the dis-
tance of about 100 m (due to the thermal radiation from the pool fire), meaning the terminal jetty with the equipment, pipeline from the jetty to the tank farm and part of the tank farm, e.g., delivering some damage to the tank 401. Safe distance regarding the potential injuries to the personnel is about 200 m.
2.3 Likelihood modeling The Bayesian Belief Network method (iNTegRisk, 2011; Kjaerulff, et al., 2008) and program tool Hugin Lite 7.5 (HUGIN EXPERT, 2012) were used. Due to the fact that the Bayesian networks does not support calculation of the frequencies of the events, but only probabilities, the model was prepared for a single operation of the tanker-ship unloading. The model of the accident considers the following direct failures or mistakes: Pipeline – full bore rupture of 30 m of DN300 pipeline is considered; failure frequency is 2×108 per year per feet (API, 2000, Table 8.1, environmental conditions, etc., not considered) providing during 16 hour operation a probability of 6.2×10-9. Joints – failure at the flanges at 6 valves/12 joints is considered; failure frequency is 5.7×107 per year per joint (AIChE, 1989, page 184) providing during 16 hour operation a total probability of 1.2×10-8. Arm – unloading arm full bore rupture is considered; probability per operation is 3.8×10-5 (HSE, 2012). Hammer – hydraulic pressure hammer event is anticipated if both air is present in the delivery pipeline (due to absent/incomplete prior air venting by the terminal personnel – event AirInPipe) and tanker personnel immediately starts multiple centrifugal pumps (does not follow pumping operation strictly by switching on one pump only at the start – event MultiplePumpsOn). Such mistakes are considered rare as the teams are highly trained and subject of supervision. The corresponding human error probabilities considering normal working conditions are assessed at 0.003 (Williams, 1992; Table 1, task type F). Move – it is about the potential move of the tanker-ship during the unloading operation. Intensive discussions with the representatives of the Luka Koper terminal rendered that such an event wound need about one hour of time to execute the unties from the jetty and there are three separate teams to be present: terminal operators, ship personnel and the towboat personnel managing the ties. To conclude, it was decided that such an even can be noted; however its likelihood is deemed extremely unlikely. Due to that its probability is set to zero in the model.
Collision – the collision to the tanker-ship at the jetty during the unloading operation by the second ship passing by in the second channel is considered. The risk assessment for such specific accident was already done, where Luka Koper, 2010, Table 48 provides assessed collision frequency of 1/26.5 per year on the whole Luka Koper port sea surface. The second channel represents about 4 % of the port sea surface, rendering adjusted frequency of 1.51×10-3 per year. During 16 hours operation this provides collision probability of 2.76×10-6. Fails – for events Move or Collision to render compromised unloading arm the conditional probability of 0.1 was set (one out of ten events actually leads to the full bore release from the unloading arm). Ignition – the model of the consequences suggests that the flammable cloud could in one minute easily reach border of the terminal/tank farm, as well as engulf the tanker-ship (subject of wind conditions) with the engines on supporting the unloading operation. Thus, the ignition probability is estimated at about 0.5 (50 %).
The events Pipeline, Joints and Hammer are inputs to the events Failure, Move and Damage to the event Arm_External and Arm_fails is additional input to the event Release. Here all intermediate events use OR Boolean logic regarding the inputs. Event Hammer has two inputs using the AND logic. The event Fire needs Release and Ignition to be true. The extent of the Consequences is set by the Release and Fire events and has four states: No_spill, Spill_1min, Spill_5min and Spill5_and_Fire. From the Consequences event two additional utility nodes are added. The Volume node calculates utility function – a volume of methanol spilled per operation (in m3) considering spill rate & duration and probability as a weight. The VAR (Value At Risk) utility node calculates the utility function – a direct monetary value in € of the damage considering the goods and process equipment (assets) at risk per each state of the Consequences event (and related probabilities as weights) – see Table 1. The reader shall note that secondary damages, e.g., in terms of lost production, repairs, reputation, compensations and fines are not considered yet. Table 1: Direct damage monetary values considered Consequence state Direct damage (€) NoSpill 0 Spill 1' 40,125 Spill 5' 200,625 Spill 5' + Fire 910,625
Table 2: Summary of the candidate KPIs. No. Basic event Internal documents 1 Pipeline Maintenance quality: Work Instruction #113, 2 Joints Realization of the prev. maint. plan & corrective actions - unresolved in time 3 Arm_Fails 4 Move Ship-Shore-Safety Check list, Items No. 3-4, 22-23 Meteorological records (>25 knots) Cargo discharge records 5 MultiplePumpsOn Cargo discharge plan (terminal-ship)
6
AirInPipe
Management provisions on personnel training
7
Collision
Records on ships in the port
8 9
Fails Ignition
E.g., Threat Assessment (Luka Koper, 2010) E.g., updated Safety Report (Luka Koper, 2008)
The calculated Bayesian network starting from the direct failures and mistakes with monitor windows opened and arranged is presented on
Figure 1. Please note that the probabilities are reported in %. 3 RESULTS
Results show that the event Release (being of our main interest) has probability about 0.00576 % (5.76×10-5) with contributors Failure (0.00192 %), Arm_Fails (0.0038 %) and Arm_External (0.00004 %), meaning that the contribution ratios are about 1/3, about 2/3 and below 1 %, respectively.
This can mean that internal failures are far most important, especially the unloading arm performance, however, the source used for Arm_Fails data (HSE, 2012), to some extent also considers external impacts from passing ship collisions. For that reason the event Collision was considered for the background factors. The related ship equipment and its crew performance is currently subject of governance and inspections according to the Paris Memorandum among the EU and other states (Paris MoU, 2012). The results of the inspections are shared among the national and joint authorities, resulting in performance categories, e.g., according to the national flag in White/Gray/Black (with Black being the worse performance grade) flag, etc. In the scope of this assessment, the authors of (Luka Koper, 2010) were consulted regarding the likely impact of the national flag grade given for the passing ships in the second channel and the performance node 2nd_Ship_Performance was added to model presented on
Figure 1. This node considers that the ships with White flag performance correspond to the previously calculated collision probability (2.76×10-6) while Gray and Black performance flags shall correspond to three and ten fold increase, respectively. In the period from July 2011 to August 2012 there were 6 % Black, 2 % Gray and 92 % of White performance flag ships entering the Luka Koper. Figure 2 presents alternative data considered for event Fails (failure is always true) and for node 2nd_Ship_Performance (only Black flag perform-
Candidate indicators Maintenance (preventive) quality, e.g., % realization in time Tanker Ship & Personnel competence White/Gray/Black Ship List provided by the EU Safety culture - internal surveys Tanker-Ship state & Personnel competence White/Gray/Black Ship List provided by the EU Cargo discharge plan periodic internal audits (% of compliance) Personnel competence – internal training exam score Safety culture - internal surveys Port level - Ship state & personnel competence White/Gray/Black Ship List by EC / /
ance ships entering the port) thus rendering new event Release probability of about 0.0085 % (8.5×10-5) – about 48 % increase. The message here is not in exact alternative data, but in demonstration that the related uncertainties in performance of the equipment ("will unloading arm fail or not upon the collision?") and available performance indicators for the potential collision can have an impact on the results.
Considering again Figure 1, probability of the Release is assessed at 5.76×10-5 (once in 17,000 unloading operations; considering 10 operations per year, this corresponds to the accident frequency of 5.76×10-4 year-1, or once in 1,700 years), of which probability of 1 minute spill without fire is about 2.6×10-5, for 5 minute spill about 2.8×10-6 and for 5 minute spill with fire about 2.9×10-5. The utility nodes Volume and VAR report average spill volume 0.0018 m3 and direct damage 27.5 € (both per operation), respectively. Comparison to the
Figure 1 shows average spill volume 0.0027 m3 and direct damage 40.5 €. In addition, the reader shall note that in VAR only direct damages (Table 1) were considered and real monetary damages could be significantly higher. 4 SELECTION OF THE KPIS
The risk model identified anticipated direct causes of the potential accident. In order to establish a transparent risk management monitoring program direct causes shall be explicitly related to meaningful management functions and measurable performance indicators. The direct causes identified in this risk model can be grouped to: Equipment failures that are related to the quality of the equipment maintenance (preventive and corrective) Personnel errors that can be related to the competence and safety culture
External factors, e.g, performance of the ships performing operations in the port area
Considering the applicable guidelines (iNTegRisk, 2011; HSE, 2006; OECD, 2008) and performing interviews with the Luka Koper personnel and other experts (Luka Koper, 2010) produced a list of candidate KPI sources that shall be finally adopted by the Luka Koper management. The list is presented on Table 2. 5 CONCLUSIONS The testing of the ERRA D1 method consisted of application of the suggested approach and tools to the case of the tanker-ship methanol unloading operation at the liquid cargo terminal in the Luka Koper d.d. The case study selection, risk assessment and elaboration of the accident likelihood model using the Bayesian belief network proved beneficial in terms of building the complex risk model considering the candidate key performance indicators to be implemented within the company's Safety Management System. Considering the rationale for the development of the ERRA D1 method (iNTeg-Risk, 2011, objectives) the conclusions are as follows: The overall approach suggested in the method for the comprehensive elaboration of the KPIs for risk monitoring and managing purposes is sound and ready to use in the industrial establishments for the mentioned purposes. The method suggests a number of guidelines to be used for the selection of the meaningful, operational and efficient (considering the costs and benefits) KPIs for the aspect of the process safety (major accident hazards), however, the actual implementation and use in the industry is something that is still yet to become a common practice. The overall approach of conducting a detailed risk assessment, elaboration of the Bayesian belief network and considering the process safety affecting conditions and connecting them with the candidate KPIs is a very complex task and requires additional resources in comparison to the more conventional techniques, e.g., fault and event trees, however they are also not the best choice for building complex models (e.g., limitation in terms of independence among the basic events). The deemed remaining barriers to the wide spread use of the suggested approach are: o the dissemination and education shall complement the work done also after the iNTeg-Risk project (e.g., specific courses, success stories)
o
o
The method consists of the risk modeling approach and KPIs guidance and examples, however, there is little provided about the smooth connection between both. In addition, neither the available guidelines (HSE, 2012; OECD, 2008) are any better in this respect. The elaboration of the Bayesian belief networks inherently requires elicitation of the probabilities tables or functions from the input nodes to the parent one. This is not explicitly mentioned in the method and its sound implementation (e.g., Kjaerulff, 2008, section 6.5, Pei-Hui, 2012, Renooij, 2001) is crucial for the trustworthy risk model and its acceptance as beneficial operational tool.
6 REFERENCES AIChE, 1989. Guidelines for process equipment reliability data with data tables, Center for Chemical Process Safety of the AIChE, New York. API, 2000. Risk-Based Inspection Base Resource Document, Publication 581, First Edition, API. HSE, 2006. Developing process safety indicators, Health and Safety Executive, HSG254. HSE, 2012. Failure Rate and Event Data for use within Risk Assessments (28/06/2012); http://www.hse.gov.uk/landuseplanning/failure-rates.pdf HUGIN EXPERT A/S, 2012. http://www.hugin.com/ iNTeg-Risk, 2011. D1.5.1 Final Type A report for ERRA D1 – T1.5.1 Kjaerulff U.B., Madsen A.L., 2008. Bayesian Networks and Influence Diagrams – A Guide to Construction and Analysis, Springer. Luka Koper d.d., 2008. Safety Report for Liquid Cargo Terminal extension project, Luka Koper internal document. Luka Koper d.d., 2010. Threat assessment of Luka Koper d.d. for the aspect of industrial accidents – sea, University of Ljubljana, Faculty of Maritime studies and transport, Portorose, Slovenia). OECD, 2008. Guidance on developing safety performance indicators related to chemical accident prevention, preparedness and response, guidance for industry. OECD Environment, Health and Safety Publications, Series on Chemical Accidents, No. 19, Paris. Pei-Hui Lina, Daniela Haneaa, Ben Alea, Simone Sil-lema, Coen Gulijka, Patrick Hudsona, 2012, Integrating organisational factors into a BBN model of risk. In: Proceedings of PSAM11/ESREL 2012, Helsinki. Renooij S., 2001. Probability elicitation for belief networks: issues to consider, The Knowledge Engineering Review, 16:3, 255-269. The Paris Memorandum of Understanding on Port State Control (Paris MoU), 2012. http://www.parismou.org/ Williams., J.C., 1992. A User Manual for the HEART, Human Reliability Assessment Method, DNV Technica.
Figure 1: Graph of the prepared Bayesian network, including monitor windows for the events probabilities and utilities. Please note that the probabilities are reported as %.
Figure 2: Graph of the Bayesian network (as on the previous figure) with considered evidence for Fails (being true) and 2nd_Ship_Quality node values corresponding only to the Black list category (see main text for details). Please note that the probabilities are reported as %.
Figure 3: Proposed Bayesian network structure considering performance nodes from the KPIs (five nodes at the top).
