The Development of a Unique Report for Performance Monitoring in an Airline Safety Management System M.C. Leva1, J. Cahill1 , A.M Kay1, , G.Losa2 N.McDonald1 1
Aerospace Psychology Research Group APRG Department of Psychology Trinity College Dublin Ireland 2 Iberia Airlines PO Box E-28042 Azi Ni Barajas - Edif., Operaciones 114 Madrid, Spain Corresponding author M.C. Leva telephone: 00 353 1 896 2916 fax: 00 353 1 671 2006 mobile: 00353 861613570 e:
[email protected]
The Development of a Unique Report for Performance Monitoring in an Airline Safety Management System M.C. Leva1, J. Cahill1 , A.M Kay1, , G.Losa2 N.McDonald1 1
Aerospace Psychology Research Group APRG Department of Psychology Trinity College Dublin Ireland 2 Iberia Airlines PO Box E-28042 Azi Ni Barajas - Edif., Operaciones 114 Madrid, Spain
Abstract This paper presents the research conducted as part of the work requirements for the Human Integration into the Lifecycle of Aviation Systems (HILAS) project, sponsored by the European Commission. Specifically, it describes the proposed concept for a Unique Report form, which will constitute the basis for all operational and safety related reports completed by Flight Crew. This includes all mandatory and optional reports. Critically, this form is central to the advancement of improved processes and technology tools, supporting airline performance management, safety management, organisational learning and knowledge integration activities. To this end, this paper also exemplifies how reporting data will be used by airline personnel in relation to the processes listed above. The new reporting form has been developed in collaboration with a major Spanish Airline through extensive field research and collaborative prototyping of new concepts. 1. Introduction: Airline Safety Management System (SMS) and Performance Monitoring Tools Until recently, airline approaches to safety have reflected a reactive model (e.g. complying with regulatory requirements and prescribing measures to prevent the recurrence of undesirable events). Current models follow a more proactive safety management approach. According to the international civil aviation organization (ICAO), this is characterized by a number of factors (ICAO 2006a) including: - The application of scientifically-based risk management methods - Senior management’s commitment to the management of safety - A non-punitive environment to foster effective incident and hazard reporting - Systems to collect, analyze, and share safety-related data arising from normal operations - Sharing safety lessons learned and best practices through the active exchange of safety information From ICAO’s perspective, this is supported by the development of appropriate safety management systems (SMS), defining the required organizational structures, accountabilities, policies and procedures (ICAO 2006a). In this regard, most airlines have developed (or are in the process of developing) safety management systems in accordance with regulatory guidance. Currently, airlines use a range of paper and technology based tools to monitor and evaluate human performance (and by implication organizational/system safety). Feedback from these tools is used to direct system safety improvements (e.g. process/procedures re-design, enhanced training etc.). Traditionally, these tools have divided into two types: those that focus on gathering human performance information; using either self report or observer based methodologies (e.g. Air Safety Reports, Non Technical Skills Evaluation, Line Checks and Line Operations Flight Training) and those that focus on gathering aircraft performance information (e.g. Flight Operations Quality Assurance). Crucially, these tools fail to provide a real-time
picture of routine operations supporting predictive risk management (Cahill et al. 2007). The use of many discrete tools presents additional information management challenges. Much valuable data is gathered about the operation. Yet this data is gathered, analyzed and stored in different formats, making it difficult to obtain an overall integrated safety/risk picture. Although useful from a data gathering perspective, these tools fall short of providing adequate data integration and analysis support. To this end, airlines are interested in developing tools which provide a real-time and continuous picture of the operation. Furthermore, many airlines are focusing on improving knowledge integration both internally (e.g. within airline) and externally (e.g. with authorities, other airlines etc). Arguably, little or no attention has been paid to the development of tools which embed crew reporting in the Flight Crew task, and link directly to airline safety/risk monitoring and process improvement activities). This requires redress – flight deck and airline safety practices and technologies must be integrated. The Human Integration into the Lifecycle of Aviation Systems (HILAS) project is part of the Sixth Framework Programme for aeronautics and space research, sponsored by the European Commission, its overall objective is to develop a model of good practice for the integration of human factors across the life-cycle of aviation systems (McDonald 2007). The Flight Operations strand is aimed at developing a new methodology for monitoring and evaluating overall system performance to support improved information sharing, performance management and operational risk management. Its current research suggests that the first step is to gather the right information from operational personnel.. Potentially, allowing operational personnel to ‘audit’ their own companies processes and procedures, may provide Safety and Risk personnel in an airline, with the necessary feedback to determine the safety/risk status of the operation. This would allow them to assess if procedures are appropriate and what, if necessary requires changing. Furthermore, such an approach may motivate operational personnel to report both routinely and above and beyond what is required by Authority. 2. Flight Crew Performance Monitoring in aviation Airlines use performance monitoring tools to evaluate human performance and by implication their organizational/system safety. Feedback from these tools is used to direct improvements (re-design procedures, enhance training etc.). The Line Operation Safety Audit (LOSA) methodology constitutes the current state of the art in terms of performance monitoring. LOSA evaluations have been successfully undertaken by many airlines to assess routine flight operations. The purpose of LOSA is to identify threats to safety, minimize the risks that such threats may generate and implement measures to manage human error in operational contexts (FAA 2005). In a LOSA evaluation trained observers watch real-life operations and provide feedback about Flight Crew threat and management skills. Observers (a) document external threats, (b) record flight crew errors in terms of their type, management response, and outcome (e.g. aircraft states), and (c) rate the crew on several Crew Resource Management behavioural markers (Klinect et al 2003). Evaluations are conducted under strict non-jeopardy conditions - crews are not at risk for observed actions. Helmreich’s Threat and Error Management (TEM) model (1999) provides the theoretical basis for LOSA evaluations.
The LOSA model distinguishes (a) threats: both external threats (including latent threats) and internal threats (crew performance), (b) error types, (c) error responses/countermeasures and (d) error outcomes (in terms of aircraft states) (Klinect et al 2003). Critically, LOSA has highlighted the fact that error and violation are normal occurrences in operational systems and must be managed. The LOSA framework and methodology has many benefits which should be considered in the design of a future tool. This includes (a) the evaluation of non technical and technical skills (albeit these are separated in LOSA), (b) the attempt to link technical performance (procedures, aircraft handling), non technical performance (CRM, TEM behaviours) and aircraft states, (c) the observation of real operational practice and (d) non jeopardy evaluation (de-identified, confidential, and nondisciplinary data collection) -which fosters a sense of trust between operational personnel and management. Building on this concept the Airlines participating in the HILAS Project have suggested a number of possible suggestions as reported in table 1. Table1: Suggestions upon which the Unique Report has been built. Suggestion Establish a reporting framework linked to the journey log of each flight that would allow the pilots to reports on threats and errors encountered and managed during each flight.
Enable an improved logical reconstruction of occurrences and their links to flight phases and potential (crew) corrective actions
Provide an established channel for reporting threats and occurrences in parallel to LOSA. The structure of the Reporting Form should give a clear focus on actual accounts of the main facts constituent to the event, introducing an “ad hoc” space for the crew to express their analysis at the end. The analysis can be directed towards the elements of the operation performance upon which the company has leverage.
Develop an intelligent flight plan in order to provide feedback on performance management concepts
Rationale The LOSA audit is performed occasionally, and in its absence there is no ongoing audit process. It would be beneficial for organisations to be able to determine how safe they are by daily assessment which could tentatively be rooted on an empowerment of the front line operational staff. LOSA does not provide an account of the overall error chain. When a chain of events is reported small threats and/or errors associated can have knock-on effects for subsequent flight phases with increase severity. Detection and recovery action, often occur at a much later flight phase. In the LOSA observation protocol, the crew’s perspective on events is missing. The way LOSA is built in fact has its cornerstone on the impartiality of a third party observer (the auditor) which is a good thing for the purpose of a once-off auditing. However, it might be beneficial to enable front-line staff to collaborate to the safety improvement process, highlighting (through an established channel) the issues raised by everyday performance (procedures not suitable to certain working conditions, or recurrent anomalies). This, together with positive feedback on the actual use of the information provided, can motivate the crew towards a proactive engagement in the continuous safety improvement process. Feedback mechanisms were also an area highlighted for improvement, in particular the analysis of data originating from a LOSA protocol or internal occurrence reporting
tools.
3. The Unique Report: Structure of the new reporting form as obtained through the field study 3.1 Empirical research brief excursus on the methodology The empirical research that constituted the cornerstone of the development of the Unique Report form was conducted to identify the main user needs and requirements. It was carried out with five partner airlines using participants from across the three Flight Operations sub-processes e.g. (1) flight planning, (2) active flight operation and (3) change/safety/quality process. Initially, workshops were conducted with the airlines, to map the active flight operations process and understand Flight Crew task performance within the context of this process. A detailed task analysis was undertaken with Flight Crew from two partner airlines. This research was directed at understanding the nature of Flight Crew task performance (and associated information requirements) and identifying how this might be facilitated by the design of improved situation assessment and reporting tools. This included (1) seven jump-seat observations and de-briefs interviews, (2) two detailed task analysis case studies involving one pilot from each airline and (3) interviews with twelve Pilots. Following this, twelve interviews were conducted with other operational personnel (e.g. Dispatch, Cabin Crew and Maintenance) from two airlines. The purpose of these interviews was to identify other role information inputs to Flight Crew and correspondingly, Flight Crew information outputs to these roles. In addition ten interviews were conducted with Flight Planning Personnel and Quality/Safety personnel from both airlines. This research investigated dependencies between active flight operation and flight planning and safety/quality processes. A more in depth task analysis exploring certain critical activities (e.g. flight planning and briefing, reporting etc), was then conducted with Flight Crew from one partner airline. This resulted in the identification of three high level Flight Crew scenarios and associated tool concepts. Subsequently collaborative prototyping activities were conducted with Flight Crew from a Spanish Airline, using participatory design methodologies, (Muller, 1991, 1993). This research focused on modelling low fidelity prototypes of the Graphical User Interfaces with the airline designated working group. 3.2 Findings An element of the empirical research resulted in the outline of a Unique Report for occurrences that may or may not result in mandatory reports. This function is to be embedded in the Extended Journey Log that the flight crew compile at the end of each flight, and therefore as part of their routine operations. In the journey log a summary of the threats envisaged for the flight is proposed and the flight crew may be required to report about some, or all of them through the Unique Report. The new reporting form is focused on the main data that can be used for Task Support Functions and for deriving a reliable picture of the main threats, and hazards faced in everyday operations. Furthermore, the reporting form takes into account, an overview of the daily threats and error management activities the crew normally undertakes. The Unique report is based on trying to enable as much as possible an actual account of the main facts constituent to the event, introducing beside an
established channel for the Crew to express an analysis of the event. The analysis section of the report is built in such a way that the information can be quantitatively used for directing possible follow up and meliorating activities. By observing the narrative used in a sample of reports it was established that a substantial number of the accounts begins by stating at what point in the process phase a certain event happened. Attention was paid to this fact as it provides information to base the safety management system around an actual process mapping of current company activities and the main threats and hazards most recurrent in every of them and their interdependencies. In addition the pilots involved in the workshops that led to the development of the reporting form, frequently stated; that the classification of an occurrence can become confusing since its underpinning logic is often unclear (can depend upon the main actors involved, main causes or main consequences). Currently most of the accident or incident reporting forms force the users to classify events at the beginning, which often leads to a misclassification and a misuse of the information (e.g. an event can be classified according to its consequences “TCAS warning” as opposed to the possible main cause of the event “ATC communication problems”). Section
Table 2: Generic Reporting Form Structure (Summary)
A
Generic event Structure
B
When did it happen?
C
What happened?
D
Action Taken according to:
E F
G
single event Chain of events Process Phase Actors involved Narrative description Procedures
Prior to active flight In active flight
Normal Abnormal Emergency
Company Procedures Authority Regulation Not defined None of the above Consequences/ Outcome According to ECCAIRS Analysis Blockers Facilitators Potential outcomes/ consequences Mandatory Occurrence Report Support (as part of the journey log
Each individual section of the report will now be presented in more detail. 3.3 Section A: Event generic structure This section asks pilots to classify the event as either a single (isolated) event or a chain of events. • The logical distinction between a single event and a chain of event is necessary in order to correctly reconstruct the event sequence. • If the event to be reported, is part of a chain of events; the sections “When did it happen”, “Setting the Scene”, “Action Taken” and “Outcome” of the report structure will be asked cyclically until the chain is complete. 3.4 Section B: When Did It Happen In this section of the report, pilots were asked to select the process phase in which the event occurred. Pilots then select which sub-phase is relevant to the event. This information will be presented graphically to the pilots for ease of reference
instead of presenting pilots with numerous text options. Pilots will be asked to complete this section for each event reported. •
• •
It was observed by revising a sample of reports that a substantial majority of the narrative accounts start by stating when in the process phase a certain event happened. The pilots involved in the brainstorming sessions stated that they do not like to classify occurrences before starting an actual account of the event. The classification of the event can become confusing since it can be assigned according to main actors involved, main causes or consequences. As already stated the underpinning logic of a classification is often unclear
Linking threats and errors to information contained in a process map of the Flight Operations can help in building up a systemic repository of information. This generates a living picture of the threats and hazards most recurrent in every process phase that could, in turn be used as a foundation of a safety management system based upon a systemic view of risk. This section utilises the Operational Process Model of Flight Operations, which was previously developed through field research at least for the following layers of information: First Layer: Main Phase: Prior to active flight Active flight phases Second Layer: Main Sub-phase Pre-flight Flight Execution Post Flight Third Layer: All the different process states 3.5 Section C: ”What happened”- Setting the Scene This section allows the pilots an opportunity to list all the actors who were involved in the event, and to provide further information about them as linked to each process phase identified. There is also a narrative section where the pilot can describe the event. An alpha-numeric keypad can be used for this purpose and there will also be pre-defined text boxes to further aid pilots. Pilots are asked to select relevant threats and issues that could have contributed to the event (these are linked to the process phase already selected) and they are also given the opportunity to select human error codes and delay codes as appropriate.
Figure 1: Section C relevant actors and settings
a. Users will be presented with a list of all Actors and be able to select one or more actors involved by using check boxes I. A pop-up will enable the user to insert the specific information for every actor or setting involved. b. A narrative section enables users to describe what happened during the event by using a combination of selected Threats, Human Errors and Delay Codes and free text. I. The basic Human Errors and Threats Codes are taken from the LOSA classification scheme, but the list can be increased by adding company specific identified threats. II. Delay codes are internal to the company c. The threats and issues list will be linked to the process phases previously selected and can also be screened according to the actors selected d. If the event reported is classified as part of a chain of events, threats, highlighted threats (or added in previous phases, and outcome of previous phases) are listed as possible threats of subsequent flight phases that the user can select. e. The users can input a qualitative assessment of the perceived criticality of each threat. Five levels of gravity can be reported (1 Catastrophic, 2 Hazardous, 3 Major, 4 Minor, 5 Negligible). This data can help in rating the importance of each threat on the basis of historical data. This facilitates a review of how many high critical threats are successfully managed on a daily basis. f. In the free text section relevant key words can be selected from a soft pad and free text can be input via an alphanumeric key pad.
Figure 2: Section C: “what happened: threats and issues”
3.6 Section D: Actions Taken In this section, users are asked to describe actions taken. It also highlights features of Threat and Error management. Actions that prevent undesired outcomes within a chain of events may turn into a worse outcome at the end of the chain of events are symptomatic of effective Threats and Error management actions. • For each event the user can input one or more action • For each action it is possible to select the threats in response to actions taken and the main actor should be selected from the actors list • If the actor is a member of the crew the actions taken can be selected from: a. Action according to ECAM Procedures b. Non-ECAM procedures (Normal, Abnormal, Emergency): • Action taken in accordance with company procedures • Action taken in accordance to Authority Regulation • None of the above (“I’m not sure”) • No action taken A narrative section enables users to describe what action(s) they took by using a combination of key words and free-text. Relevant key words can be selected from a soft pad and free-text can be input via an alphanumeric key-pad. A new window will appear in case further actions need to be described.
Figure 3: Section D
3.7 Section E: Consequences/ Outcome Users are asked to indicate the consequences and outcomes for every element of a chain for both safety and operational issues. They are presented with pre-defined lists and are asked to select relevant issues. The lists are taken from the ECCAIRS classification scheme (ICAO 2006b) of potential events and are linked directly to all phases selected at the beginning of the report. Should further levels of classification of the outcome be required, a pop- up box will appear with a short list from which pilots can choose the relevant item. • More than one item can be selected (as events can have numerous consequences) • The outcome of one stage of a flight can be selected as a threat for subsequent flight phases in the report. 3.8 Final Outcome Pilots are asked to describe the final outcomes of events for both safety and operational issues. The final outcomes can be selected from the same ECCAIRS event classification scheme for each flight phase. • If a “delay” is selected, it is possible to specify the level of gravity of the and the time (estimated or real) • Users can provide an estimate of the gravity of the outcome (this estimation has to be consistent with the risk analysis featured as part of tool C functions). The gravity can be associated to a single item or can be an overall property. It is currently assigned as an overall property. 3.9 Analysis This section of the report is optional. It provides pilots with an opportunity to report and rate factors which act as blockers and facilitators to the event. All previous sections appear in a summary format for ease of reference. Users are asked to evaluate contributory factors in terms of:
Blockers (contributory factors which have a negative impact on the performance) • Facilitators (contributory factors which have a positive impact on the performance) The importance of each contributory factor can be rated on a scale from 1 to 4 (1 = least important and 4 = most important). Users can enter information about possible consequences for an event. This is to provide input on threat and error management actions that had a positive impact on the chain of events, or that, near misses should be analysed further pilots can save this section of the report for review and submission at a later date. •
Figure 4: Section F: analysis of the event
4 Envisaged use of the data: some examples The Performance Management concept developed in HILAS and is based on the relationship between Flight Crew task activities in the cockpit (e.g. flight planning/briefing, ongoing TEM, reporting) and supporting (1) operational and (2) organisational processes (specifically related to airline SMS) on the ground. Reporting data can be integrated with a range of safety and operational data, and analysed by Safety/Risk personnel. The result of this activity is feedback to relevant operational and organisational functions. For example, Flight Planning will receive information about problems/threats to be managed in Flight Planning activities. Flight Crew will receive a threat/risk picture for their specific flight – taking into account those threats that cannot be mitigated in Flight Planning. Current Performance Management processes within airlines often neglect operational feedback. While in HILAS it has been stressed that feedback to operational processes is a central part of a best practice in Performance Management. Performance management activities cannot be conceived in isolation from other Safety Management processes and functions. For example, the provision of TEM information to Flight Crew (e.g. task support) links to broader risk management activities (e.g. reports data analysed as part of reactive safety management and ongoing proactive strategic safety management activities). Therefore the provision of reports feedback and safety case studies relates to broader organisational learning processes.
Specifically, the critical organizational areas considered for the possible usage of data were the following: - Task Support, which involves supporting the safe, competent, effective and timely execution of individual and collaborative work tasks/activities in relation to the achievement of the operational goal - Tactical Risk Management, which is performed through routine processes such as reporting, investigation, assessment, analysis, recommendations, implementation, monitoring. - Strategic Risk Management, which performed through involving strategic policy decisions for the organisation and systemic assessment of events, monitoring trends against boundaries, analysing and prioritising systemic risks, policy decisions, organisational change. - Organizational Learning, which entails knowledge acquisition, information distribution, information interpretation, and organizational memory.( Huber 1991) Throughout the development and evaluation of the Unique Report the front-line staff. who contributed to the development of the reporting form, also suggested ways to utilise data collected, as reported in Table 3. Table 3: Summary of envisaged use of data collected through the Unique Report. # 1
UNIQUE REPORT FEATURE At the beginning of the report – users select whether they are reporting on a single event or a chain of events –
2
Users are not required to classify the nature of the event (e.g. event types) or error types.
3
Users are required to state single events or the elements of the chain of events according to the process phases they refer to.
4
The report captures information about contributory factors (e.g. fatigue, poor situation assessment, information availability and so forth). and their estimated relevance in respect to the final outcome.
5
The report captures information about the application of Standard Operating Procedures
6
The Unique Report allow the association of threats to specific routes and flight
RELATED USE OF OUTPUT DATA Strategic Safety Management: This data can be used to provide information in relation to process dependencies for each high severity reported Threat or Issue. This can highlight the need of specific safety barriers and the placement of them where they might results more effective. Organizational Learning: Further reports on specific event chains that result in unacceptable aircraft states can be used for training purposes. Legal Requirements: Event and error classifications can be flexible in respect to the different accident classification schemes. However the ECCAIRS classification was taken as a benchmark and because the outcomes are classified according to it an easy transferability of the information is, in principle, possible. Organizational Learning: The data can be queried according to the specific needs of the company, choosing freely to explore issues according to threats categories, routes, actors involved, and contributory factors examined. Strategic Safety management: The Data collected about threats, errors, delays etc. through the form is linked to a map of the relevant airline processes. This feature can building up a living picture of the threats and hazards most recurrent in every process phase. Organizational Learning: The process map can be periodically updated taking into account the suggestions coming form the report users whenever they needed to add a processes that they found not well depicted in the available map Tactical Safety Management: The indication about the relevance of contributory factors can be used to structure an index for decisions regarding the allocation of resources to the possible corrective actions. The process however also required an estimation of the expected impact of the corrective action (in terms of prevention of Loss associated with the related Risk Factor) and its foreseeable costs. Strategic Safety Management: Feedback on the follow up activities and how they took into consideration the notes provided about contributory factors can be made available to the reporters. This is recognised as a factor stimulating a proactive attitude of the front line staff. Tactical/strategic Safety Management Reports can be filtered according to SOP usage and problems. Threats and route can be associated to a frequency in the issue highlighted in respect to the use of Procedures. Organizational Learning: Update SOPs considering suggestions made by crew (via reports). Tactical/strategic Safety Management: Use reports information, along with information from other data sources, to determine risk ratings for specific types of flights/flight route/operations etc. Further, aggregated report information can be
#
UNIQUE REPORT FEATURE phases
RELATED USE OF OUTPUT DATA used to customise the list of threats that pertain to specific routes. Task support: Provide the list of threats that pertain to specific routes in the intelligent flight plan. Provide additional information available on how threats have been successfully managed in the past.
5 Short Summary of a preliminary evaluation of the Unique Report A demonstrator of the Unique Report was prepared and submitted to flight operations personnel not directly involved in its development for an independent evaluation. The people involved in the evaluation exercise were introduced to the general HILAS objectives, tools concepts and scenarios. The introduction of the topic made use of a short presentation followed by a group exercise. Each group was composed by a number that varied from 3 to 5 people. The groups were presented with an event case study to be reported using the Unique Report. Given the current preliminary developmental stage of a functioning prototype a Wizard of Oz approach was used to let a third person actually input and change the interface while performing the exercise (Read J.C. 2008). At the end of the exercise the participants were asked to complete eighteen 5-point Likert scale questions and 3 open questions to providing some indications about the usefulness of the tools in terms of the requirements it was originated from and a the usability of it. The total number of participants was fifteen. All evaluation of the Unique Report were positive; the vast majority of the rating was above the neutral rating on the 5-point Likert scale (86.7% of the answers) with only 0.4% of the answers below the neutral rating. However when the reporting form was actually translated into an electronic support, the actual time necessary to fill it became a clear issue of concern that lead to the design of a shorter and more agile version of it that would not require the chain of event structure as it is but a more convenient and linear one. What seems to be the main added value of the new reporting tool, from its evaluation even in the shorter version is the fact that could actually provide useful information and that the pilot would welcome the reporting form if it is in conjunction with the change to a paperless environment and adequate task support and feedback in return. References 1. Cahill J, et al. 2007. HILAS Flight Operations Research: Development of Risk/Safety Management, Process Improvement and Task Support Tools. In D. Harris (Ed.): Engin. Psychol. and Cog. Ergonomics, HCII 2007, LNAI 4562, pp. 648–657, 2007. Springer-Verlag Berlin Heidelberg 2. Cahill, J and Losa G.: Fligth Crew Task Performance and the Design of cockpit Task support tools Proceedings of the European Conference on Cognitive Ergonomics 2007 (ECCE 2007), 28-31 August, Covent Garden, London, UK. 3. European Commission 2001 Group of Personalities: EUROPEAN AERONAUTICS: A VISION FOR 2020. Office For Official Publications Of The European Communities L-2985. 4. FAA 2005 LOSA Advisor Circular, resubmitted to AFS-230, FAA, January 13, 5. Helmreich, R. L., J. R. Klinect and J. A. Wilhelm. 1999 “Models of Event, error, and response in flight operations.” In R. S. Jensen (Ed.), Proceedings of the Tenth International Symposium on Aviation Psychology. Columbus, Ohio: The Ohio State University, pp. 124–129. 6. Helmerich, R.L., Klinect, J.R., & Wilhelm, J.A. (2001). System safety and threat and error management: The line operations safety audit (LOSA). In Proceedings of the Eleventh International Symposium on Aviation Psychology (pp.1-6). Columbus, OH: The Ohio State University.
7. Huber, George P., (1991) "Organizational Learning: The Contributing Processes and the Literatures," Organization Science, 2:1 February 1991, pp. 88-115. 8. ICAO 2006 a Safety Management Manual Doc 9859 -AN/460 9. ICAO 2006 b [online]. ECCAIRS 4.2.6 Data Definition Standard: http://www.icao.int/anb/aig/Taxonomy/ [[Accessed 4 August 2008]. 10. Klinect, J.R., Murray, P., Merrit, A., & Helmreich, R. 2003. Line Operations Safety Audit (LOSA): Definition and operating characteristics. In Proceedings of the Twelfth International Symposium on Aviation Psychology pp. 663-668. Dayton, OH: The Ohio state University. 11. McDonald N. 2007 Human Integration in the Lifecycle of Aviation Systems In D. Harris (Ed.): Engin. Psychol. and Cog. Ergonomics, HCII 2007, LNAI 4562, pp. 648–657, Springer-Verlag Berlin Heidelberg 12. Read J.C.(2008) Using wizard of Oz to evaluate mobile applications in Handbook of Research on User Interface Design and Evaluation for Mobile Technology edited by Joanna Lumdsen Inc NetLibrary
Acknowledgments: Special thanks to all the Iberia Pilots and personnel involved in the research and to Kayleen Campbell, Susan OReilly and Gerry Reynolds, for the proof reading of the paper.
