Design and Development of a Performance

Design and Development of a Performance Evaluation System for the Aircraft Maintenance Industry K.K.H. Ng, M.H.M. Tang, C.K.M. Lee Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University, Hong Kong, China [email protected] ,[email protected], [email protected] Abstract – This paper aims at developing procurement Key Performance Indicators (KPI) system for Maintenance, Repair and Operation (MRO) items using a Business Intelligence tool. A set of typical KPI has been established in an aircraft maintenance enterprise through a case study. The result analyzed by business intelligence tools, has become a critical factor in evaluating the success of financial performance, quality assurance, order fill rate, utilization of productivity and development of knowledge hubs in the procurement sector. An effective and efficient procurement strategy evaluation system is proposed in order to resolve corporate problems, with solid supporting data, improved procurement performance management, facilitation of decision making and procurement evaluation by saving resources and time. The significance of this paper is that we delivers a proactive approach by integrating KPI and BI tool for an aviation maintenance enterprise to access comprehensive external data directly, and to visualize the company performance for formulating strategies and analyzing industrial informatics.

The range of MRO in an aviation maintenance companies is comprehensive, and requires continuous evaluation by KPI to avoiding accurate MRO orders such as order sizes and types, at fair and competitive prices. The major challenge in dealing with the MRO procurement process is to draw accurate and proper quotations from MRO suppliers, since MROs have high frequency and great variation [1]. Using the average price of the same components from various MRO suppliers to make decisions for the next purchase is not valid, as the same MRO from different suppliers may offer different lead times, warranties, return policies and after sales service. In addition, MRO supplies usually take up 80% of procurement resources, but constitute20% of procurement turnover, which results in lengthy procurement procedures and difficulties in administration [2]. The cost of administration may probably go beyond the cost of MRO items, without a complete audit. Accordingly, adopting Business Intelligence (BI) helps monitor the MRO procurement performance with appropriate KPI evaluation, so as to improve the negotiation decisions, monitor the administration performance and enhance the relationship with suppliers.

Keywords – Key Performance Indicator; Business intelligence; Procurement Performance Management; Aircraft Maintenance

I.

INTRODUCTION

II. LITERATURE REVIEW

Managing procurement performance is a critical task for an aircraft maintenance enterprise, as multiple factors are involved in purchasing maintenance related items, which may lead to high operational cost and long lead time. Aviation companies usually require that the duration of aircraft maintenance, or aircraft on the ground (AOG), should be minimized, with the purpose of returning aircraft to service as quickly as possible. However, the authorized repair and procurement process in maintenance, repair and operation (MRO) under tightened regulation, compliance and authentication by different National Aviation Authorities, is tedious and timeconsuming. The MRO process involves accurate planning of the purchase of aeronautical parts and spare parts for aircraft maintenance to fulfill customer requirements. Consequently, the procurement performance is not only from the perspective of co-operation in terms of costeffectiveness and productivity, but also involves customer expectations on quality, time length of aircraft preservation and cost to achieve the best outcome and company’s goals. Thus, various key performance indicators (KPI) are required to evaluate aircraft maintenance operations.

978-1-4673-8066-9/15/$31.00 ©2015 IEEE

Supplier Relationship Management (SRM) plays a vital role in the procurement process, which involves managing and working collaboratively with suppliers. SRM maximizes the potential value between business activities and all key vendors, which is carried out by the procurement or purchasing department [4]. SRM is able to identifying vendors by understanding the average expenditure per vendor for each procurement category, ranking suppliers by volume and revenue spent in order to improve the value obtained from current suppliers, monitoring the suppliers’ performance with quality and on time delivery with the propose of establishing a preferred suppliers list and control the dependency on individual suppliers [3]. Besides, procurement related factors are evaluated by BI tools to diagnose any abnormal performance in manufacturing and to propose strategic scenario management for the acquisition of raw materials, and MRO components. By leveraging an insight into the demand and supply trends in the market, optimal investment in procurement and the prevention of supply shortage could be accomplished [5].Users can gather and analyze data from operational systems, like the planning, purchasing, shipping and distribution systems, to gain insight into the business processes, understand

564

Proceedings of the 2015 IEEE IEEM

procurement issues and predict demand for resources, supplies, inventory and asset [6].

advanced computational technologies can be implemented in the MRO process.

Performance evaluation enables top management to understand the current status and find the root causes for continuous assessment and facilitate decision-making. From the perspective of management, the deployment of the KPI system monitoring by BI tools enhances efficiency and effectiveness of the procurement performance evaluation process in accordance with the implementation of advanced data processing to visualize the operational performance, the decomposition of procurement strategy’s goals to quantify the achievement by index and to display all relevant data in a centralized platform [3]. Such typical information helps the company to give reasons for the success and provide a source of communication with lower management for performance evaluation. Furthermore, the built-in data mining tools in BI involve sophisticated statistical and mathematical techniques to summarize the current performance. This is enables making comparison with the past status for reporting crucial event, predict the future trends and simulate different scenarios for long term decisions making for customer satisfaction an better business process management [7].

III. PROCUREMENT KPI ANALYSIS ON THEAIRCRAFT ENGINEERING COMPANY A. Challenges faced by Aircraft Engineering Company Taking an aircraft engineering companies for a case study, the involved company would like to improve current procurement efficiency through intensifying the procurement KPI analytics system, so as to be able to provide vision and evaluate contract performance. Since the company is a major MRO service provider, which has an aim of supporting its valued customers in ensuring the reliability, availability and unswerving quality of aircraft fleets, including engine services, line services, and freighter conversion. It is challenging for the top management to track and examine the MRO procurement performance, which is related to direct cost. Procurement of MRO supplies in the aviation industry involves a large amount of data records in order to make the corresponding procurement decisions. BI helps to transform the historical data into meaningful information for prediction; prevention of buying inadequate parts; respond proactively to the problem of AOG and implementing strategic procurement to create a valid maintenance service. They expect that BI could figure out a more in-depth analysis efficiently on the current KPI system, so as to answer the frequently ask questions from headquarter, which are described as follows:

Although researchers have started to realize the importance of adopting the latest business intelligence techniques for the MRO in the aviation industry, there is spare research on key performance indicators for practical use and there is a lack of the case studies of how KPI Cost

Quality

Timeliness

Table 1 –The set of KPI on MRO item procurement performance for Aircraft Engineering Company Indicator Description Objective Formulation Improving the procurement Part Price Percentage price variance practices to achieve a more Variance between the current unit competitive pricing price and the Last Purchase Order (PO) price for parts , Accessing the use of efficient Contract Percentage by value of contracting mechanism Utilization purchases made under Quotation, Catalog and Contracts Understanding the current Contract Percentage by number of coverage of contract Coverage purchases made under . , Quotation, Catalog and . Contracts 100% Procurement Percentage by number of Interpreting the successful . Coverage actual usage rate of procurement orders . Response Rate

Emergency Procurement

Learning

Training Utilization

Percentage of successful procurement completed within satisfactory time Percentage by value and number of Aircraft On Ground (AOG), Critical and Replenishment Order Percentage of training utilization

Measuring the qualified procurement response rate of communication stage from customers to procurement staff Estimating the ratio of unprofitable and emergency procurement towards all procurement order in dollar value Calculating the learning and growth of the employee in procurement department

565

.



.









100%

100%







,

100%

.

100%

3



.

100%



100%

Proceedings of the 2015 IEEE IEEM

1) Explain thhe reasons beehind the higgh failure ratte of procuremeent coverage for f MRO aeroonautical item ms in the past thrree quarters. 2) Illustrate the t root causses of employees’ inabilitty to comply wiith a 2-day quootation responnse. 3) Analyze thhe impacts onn terminationn of a contractual relationshipp with a supplier

h technoloogical skills and complexx Oracle require high deling design. However, Qllik is the userr-friendliest BI B mod tooll compared with w the otheers, and is eaasy to use byy simp ple drag-andd-drop to revveal hidden insights from m histo orical data. Inn addition, the Qlik Sense developer gives morre support than Micrrosoft and Oracle onn docu umentation, online colllaboration, tutorials t andd confferences. Furtthermore, the responsive deesign and freeeform m exploration of Qlik Sensse enables useers to evaluatee proffound knowleedge from thee KPI perform mance. Table 2 show ws a summaryy of the compparison betweeen different BI B toolls.

B KPI Systeem Analysis foor MRO Itemss B. m andd evaluating the With ann aim of measuring procurement service perfoormance, a seet of procurem ment performance KPI is dessigned speciffically for MRO M a indusstry to matchh the service proviiders in the aviation procurement features andd managerial needs in airrcraft maintenance. The set of KPI is usedd to evaluatee the performance of the procuurement depaartment from four perspectives – Cost, Quallity, Timelineess and Learnning. T 1. All the KPI arre shown in Table C. Analysis on KPI Archittecture Identifyinng new opportunities and implementinng an effective proccurement strattegy based onn new insightss can provide a coompetitive maarket advantage and long--term stability for a business. BII assists top management m inn the analysis of cuurrent and passt activities inn the predictioon of future trends,, and supportts managers to t design straategy planning andd control opeerational dataa from functiional users via the Operational Database D Mannagement Sysstems (DBMS). Thee operational data is then transferred too the Business Inteelligence Dattabase Managgement System m to report generration and data mininng for the top management. These includde predictionss, what-if anaalysis and decision-making. Figuure 1 shows the working proocess from frontlinee operation to the top management of BI..

Table. 2. Suummary of Compparison between BI B tools

B. Improvementt of KPI System m on Manageerial Aspect In order to opptimize the procedures on th he KPI system m to ensure e the effiiciency and efffectiveness of o procuremennt man nagement, high-quality and centralized data d mining aree deveeloped in thhe Qlik Sensse. The interrmediate dataa tran nsformation syystem, whichh is an impo ortant process conttaining Extraact-Transform m-Load proceedures, filterrs “dirrty” data, misssing data vaalues, inconsisstent data andd dataa in an incorreect format, stoores data in th he same formaat in the t DBMS annd links the data source with the Qlikk Sense. The frameework of BI toools is shown in i Figure 2.

Fig. 1. The Relationship R betw ween Operation annd BI Applicationns

IV. PROCUREME P ENT PERFOR RMANCE EV VALUATION N VIA QLIK SENSE S ment With the implementation of BI toolss, the procurem performance analyzes froom the resuults of KPI and identifies thee root causes and improvement areas. The three popularr and provenn BI softwarre systems, Qlik, Q Microsoft annd Oracle, have been evvaluated for their respective streengths and weeaknesses.

Fig. 2. The Framewoork of the BI Toolss

Thee snowflake schema shownn in Figure 3 is adopted too deveelop the data model for thhis study, whicch is a logicaal arraangement of relationship r taables in a mu ultidimensionaal dataabase. Buyerr, supplier, MRO part, expendituree, train ning and Reqquest for Quuotation (RFQ Q) records aree emb braced in the purchasing oorder fact tab ble, with ordeer reco ords in a centrralized format.

A Selection Criteria for Business Inteelligence Toolls on A. Monitorinng KPI Perforrmance s criteeria for BI toools on monitooring Certain selection KPI performaance are discuussed as follow ws. Microsoftt and

566

Proceedings of the 2015 IEEE IEEM

equiipment procuurement and they discov vered that thee fund damental causse is due to innvalud certificatesfor MRO O partts.This is due to airlines beingserved d by differennt coun ntries that aree under various regulations with respecct to th hecontrolling by the aviatioon authority, like Europeann Aviation Safety Agency (EA ASA) and Fed deral Aviationn ministration (F FAA). It sugggests that thee procuremennt Adm depaartment couldd refer to paast quotationsand make thee apprropreiate MR RO order foor the sake of reducingg proccurement faliuure.

Fig. 3. The Th Data Model

C. Visualizaation of Procuurement Perfo formance via Qlik Sense

R Causes for f Employeees’ Inability to o Comply withh 2) Root a 2-day Quotaation Responsse

As for resolving the FAQss from the topp managementt, the f Qlik Sense S high quality KPI reportss generated from provide soliid analysis and managgerial insightt in describing staatistical indicaators and grapphical informaation to enhance thhe quality of decision-makin d ng with integrrated KPI BI analyttics.

The companyy encounteredd another pro oblem relevannt to th he non-conforrmity of the puurchase order regulation. Too prov vide custom mers with hhigh quality of aircrafft main ntenance seervice, the company has h designedd proccurement guiidance to thheir buyers, i.e. that thee quotation processs be finished w within 2 days.

b High Failure Ratee of Procurem ment 1) Reasons behind Coverage for MRO Aerronautical Item ms The top management concerned thhat there is a high M failure rate of procurrement coveerage for MRO Aeronautical Items in the past thhree quarterss.The L) to companydeveeloped a preeferedquotatioon list (PQL represent thee favourable source of MROsupplies. M The major types of MROs foor the compaany are chem mical s items, raw materials, sttandard harddware and spare ment process is to use minnimal items.The ideea of procurem number of quuotation to maake a decisionn on purchasiing a single MRO O part. It refers r to thee MRO prooduct knolwedge, procurement p performance and its learrning progress amoong the procurrement departm ment. In Figuure 4, true value of KPI refers to positive proccurement coveerage KPI value, and excludded preferablle quotation list RO supplier while w (ExPQL) idiccates the pre--screened MR false value reppresents 1 minnus KPI true value v and ExP PQL.

Fig. 5. Average Performance of R Response Rate off each month

In the past year, y they recorded unquallified responsee rates on averagge in Januaary, March, April, Julyy, Nov vember and December D by comparing of o the averagee perfformance of response ratee in each mo onth shown inn Figu ure 5.There were w seven em mployees wh ho were undeer 50% %of the respoonse rate afteer the review on employeee perfformance apprraisal via Qlikk Sense in Figu ure 6.

Fig.6. Buyer perform mance and its KPII

Theey subsequenttly discussed with those em mployees theiir worrk difficulties. As a result, tthe managers found that thee critiical issue for the t declining rresponse rate is the problem m in sourcing s MRO O parts due to inadequatee aeronauticaal knowledge, like latest aircrafft regulation policy, MRO O partt requirementt, and mainttenance proccedures. Evenn thou ugh the compaany has providded product trraining to theiir staff ff, they did not n have a1000%attendancce at productttrain ning coursess. The comppany may provide p somee

Fig. 4. Platform m related on the reeturn rate of MRO O Standard Hardw ware and Relevant R KPI

According toothe Qlik Sense report, it is i found that twothird of the past p three quarrters accounteed for 30% faailure ofinvalid quootation on avverage. The managers fuurther investigated the t problem of higher failurre rate of stanndard

567

Proceedings of the 2015 IEEE IEEM

incentives too encourage a higher atttendance ratees in product training. Moreoveer, staff can refer to the price p M componnents to estim mate the expeected variance of MRO price of the next purchasee so that it helps h simplifyy the procurement requisition r proocess.

T thereforre considered d a decision too quotation price. They minate the conntract with thhem. BI can help evaluatee term supp plier perform mance in monnetary value and purchasee freq quency. The majority m of spaare parts purch hased in dollaar valu ue was from supplier s A, aat the same tim me, the reporrt indiicated that suppplier B was the second laargest frequennt purcchase supplieer, shown in Figure 8. In case of o term minating contrractual ties wiith supplier A, A the companyy may y decide to make purchhase from su upplier B viaa quotation.

o Terminatioon of Contracctual Relationnship 3) Impacts on with the Aviation A Equippment Supplierr A s A, which has under u Aviation equipment supplier r w with the casee company, is a contractual relationship supplier of sppare part consstituted aroundd 30% of the total procurement value of the company aftter analysis by BI 7 If the coompany endss the tools shown in Figure 7. procurement contract wiith them, thhey could raather b the price and purchase by quotation or by catalog, but M parts may m fluctuate. According too the quantity of MRO up-to-date KPI K of conttract utilization and conntract coverage, the company tennded to sign coontracts with their suppliers receently.

V. CONCL LUSION O performancee This study foocuses on a sstudy of MRO evalluation by KP PI system using BI tools in an aircrafft main ntenance enteerprise. The ppaper suggestts KPI for thee MR RO in the aviattion industry aand proposes to t use a highlyy interactive dashhboard that spans busineess-user selffvice, because it can enable users to cond duct a broadeer serv rang ge of analysiss, it links withh the company y performancee and develop a long-term strateegy procurem ment policy foor nd developingg optiimizing the prrocurement performance an susttainable know wledge growth across the com mpany. A ACKNOWLE EDGMENT The researchh is supportted by The Hong Kongg Poly ytechnic Univversity. The aauthors would d like to thankk the research com mmittee and thhe Departmen nt of Industriaal ng Polytechnicc and Systems Enggineering of thhe Hong Kon U8H). University for suppport in this pproject (G-RU REFERE ENCES [1]

Figg. 7. Visualizaation of Contraact page

[2]

[3] [4]

[5]

[6] Fig.8. Visualizzation of Emergennce Procurement in Dollar Value and Numbber of Order

[7]

Howeverr, the company encounttered a probblem concerning thhe contract priice that the puurchasing pricce of MROs from supplier A was usually higher thann the

568

J. Yu, S. Gulliver, G Y. Taang and L. Ke,, 'New advancees in aircraft MRO M services: D Data mining enh hancement', The Fourth Innternational Workshop on o Advancedd Computationnal Intelligencee, 2011. M. Le Suueur and B. Dale, 'The procurement p o of maintenancee, repair and opperating supplies: a study of thhe key problem ms', Europeann Journal of Purchasing & Supply Manaagement, vol. 44, no. 4, pp. 247 7-255, 1998. K. Zhang, 'Design for E Enterprise Asseet Managemennt Evaluation System of Power Generatiion Enterprisees PI', AMR, vol. 6694-697, pp. 34 401-3405, 2013. Based on KP S. Leu, V. V Pham and T. Pham, 'D Development of o recursive decision making model in bilateeral constructionn procurementt negotiation', Automation in n Constructionn, vol. 53, pp. 131-140, 2015. E. Yusof annd A. Yusof, 'The Study on the Applicationn of Business Intelligence inn Manufacturing g', Internationaal Journal of Business B Intellig igence Research h, vol. 4, no. 1, 1 pp. 43-51, 20013. C. Lee, H. H Lau, G. H Ho and W. Ho, H 'Design andd developmennt of agent-baased procurem ment system too enhance buusiness intelliggence', Expert Systems withh Applicationss, vol. 36, no. 1, pp. 877-884, 2009. 2 W. Pan annd H. Wei, 'R Research on Keey Performance Indicator (K KPI) of Business Process', 2012 Secondd Internationaal Conference on Business Computing C andd Global Inforrmatization, 2012.