Assessment of Organizational Performance in Paper ...

Assessment of Organizational Performance in Paper Products Manufacturer Company using Balanced Scorecard and TOPSIS Hamidreza Feili Assistant Professor, Department of Industrial Engineering, Karaj Branch, Islamic Azad University, Karaj, Iran ([email protected])

Mojtaba Qomi B.Sc., Department of Industrial Engineering, Karaj Branch, Islamic Azad University, Karaj, Iran ([email protected])

Shayan Akbari B.Sc. Student, Department of Industrial Engineering, Karaj Branch, Islamic Azad University, Karaj, Iran ([email protected])

Sepideh Madani Mohammadi B.Sc. Student, Department of Industrial Engineering, Karaj Branch, Islamic Azad University, Karaj, Iran ([email protected])

Hossein Asadipour B.Sc. Student, Department of Industrial Engineering, Karaj Branch, Islamic Azad University, Karaj, Iran ([email protected])

Abstract During three decades ago, a collection of new technologies and discussions have come up as like as assessment of organizational performance for developing management in all organizations. Balanced scorecard is one of the instrument that is used for assessing organizational performance. This technique helps the organization to arrive to its purpose by focusing on organizations strategies and making logical relationship with all organizational dimensions. By application of expert's opinions alongside with BSC, we get usage of decision-making strategy as TOPSIS. In this research, one paper products manufacturer company in Iran is studied. This company activity is limited to production of Kleenex, paper towel, and fruit packaging paper. BSC and TOPSIS methods has been used to improve organizational performance in paper products manufacturer company. Finally, total performance score will be calculated by multiplying the global weights and scale values of performance indicators and then by summing the resulting performance levels. These results can be effective in assessing organizational performance in the mentioned company. Keywords: Paper Products Manufacturer Company, Balanced Scorecard, TOPSIS. Page 1 of 14

Introduction There are many strategic control techniques and methods aimed at evaluating – from a strategic management perspective – the results of the activities carried out by a business (Eren, 2002; Dinçer, 2004; Ülgen and Mirze, 2004). One of the methods enabling periodical and systematic system controls is the Balanced Scorecard (BSC) system developed by Kaplan and Norton (1992, 1996a). Balanced Scorecard enables expression of the vision and strategies of a business in terms of performance indicators and thus ensures establishment of the framework required for strategic measurement and management system. While underlying that traditional financial indicators are important, BSC suggests that financial indicators prove to be insufficient in explaining the business performance when they only contain the information related with the incidents that have taken place in the past. In the light of this thought, Kaplan and Norton (1996b) proposed BSC system that enables integration of the measurements regarding the past business performance with the measurements regarding the elements that will bring future performances. Kaplan and Norton (1996a) presented four perspectives that need to be balanced in performance measurement: financial, customer, internal business process and learning and development perspectives. On the basis of this approach proposed by BSC, not only financial lagging indicators but also leading indicators such as customer, internal business process and learning and development perspectives are taken into consideration in strategic management process. Therefore, BSC acts as a strategic management system rather than an operational system that gives tactics only (Kaplan and Norton, 1996a). However, it is discussed that BSC approach has some deficiencies on a methodological basis (Abran and Buglione, 2003; Leung et al, 2006; Lee et al, 2008; Yüksel and Dağdeviren, 2007). These deficiencies are in the method to be used in consolidating BSC perspectives or the performance indicators which act as different measurement units under each BSC perspective; the method to be adopted in determining the contribution to be made by each perspective on the performance (Abran and Buglione, 2003; Lee et al, 2008); the relative weights or importance of the performance indicators under each perspective and; the method to be used in calculating the business performance with a holistic quantitative approach (Leung et al, 2006). There are some studies, though limited in number, that focused on such discussions related with the methodological aspect of BSC and tried to suggest possible answers for these discussions with the help of multi-criteria decision-making techniques (Sohn et al, 2003; Ravi et al, 2005; Leung et al, 2006; Yüksel and Dağdeviren, 2007; Lee et al, 2008; Feili et al, 2016b). In this research, one paper products manufacturer company in Iran (Alborz province) is studied. This company activity is limited to production of Kleenex, paper towel, and fruit packaging paper.

Research Methodology Balanced Scorecard (BSC) The Balanced Scorecard (BSC) is a rather recent development in managerial accounting (Ittner and Larcker, 2001). Kaplan and Norton (1992) introduced the BSC as a performance measurement and reward system that helps link operational performance measures to the implementation and monitoring of strategy. The BSC typically consists of four sets of measures: financial, customer, internal processes, and learning and growth. Though the BSC is a compelling innovation because it incorporates strategy, process, and managers to provide an integrated system of planning and control (Atkinson et al, 1997), there is really very little credible academic evidence that it delivers enhanced performance, even on its own value-terms (see Nørreklit, 2000). Researchers have examined various explicitly cognitive biases found in the use of the BSC. Lipe and Salterio (2000) identified a common measures bias in the BSC where superiors ignore unique performance measures in favor of measures common to subordinates being evaluated. Subsequent studies have examined approaches to mitigate the common measures bias such as the use of strategically linked performance measures and strategy maps (Banker et al, 2004; Humphreys and Trotman, 2011), knowledge and training (Dilla and Steinbart, 2005), assurance and process accountability (Libby et al, 2004), and disaggregation of the assessment process (Roberts et al, 2004). Other biases examined when the BSC is used for performance evaluation include a ‘selective Page 2 of 14

attention to strategy effectiveness’ bias (Wong-On-Wing et al, 2007), likeability bias (Kaplan et al, 2008), bias from BSC information organization (Cardinaels and van Veen-Dirks, 2010; Lipe and Salterio, 2002), and evaluator ambiguity intolerance (Liedtka et al, 2008). Cognitive biases have also been considered in the use of the BSC for strategy development and evaluation. Tayler (2010) examines the influence of motivated reasoning on projects evaluated under the BSC (Upton and Arrington, 2012; Feili et al, 2016c). TOPSIS Method The technique for order performance has similarity to the ideal solution (TOPSIS) method which is a method used to solve the multi-criteria decision making (MCDM) problems, which were first developed by Hwang and Yoon in 1981 (Olson, 2004; Wu and Olson, 2006; Jahanshahloo et al, 2006; Hung and Chen, 2009; Tsai et al, 2008; Balli and Korukoglu, 2009; Karimi et al, 2010; Abdul Rahman, 2012). The primary concept of the TOPSIS method is that the preferred alternative should not only have the shortest distance from the positive ideal solution (PIS), but also have the farthest distance from the negative ideal solution (NIS) or nadir. TOPSIS is one of the MCDM approaches used where the weighting criteria for the choice of the most adequate is exogenously defined (Feng and Wang, 2001; Pirdavani et al, 2010; Hassan et al, 2013; Bilbao-Terol et al, 2014; Chen et al, 2014; Wang et al, 2014; Barros and Wanke, 2015; Wanke et al, 2015). Its basic principle assumes that the chosen alternative should simultaneously have the shortest distance from the positive-ideal solution and the farthest distance from the negative-ideal solution (Hwang and Yoon 1981; Ertuğrul and Karakaşoğlu, 2009; Feili et al, 2016a). More precisely, the positive-ideal solution is the one that maximizes the benefit and simultaneously minimizes the total costs. On the contrary, the negative-ideal solution is the one that minimizes the benefit and simultaneously maximizes the cost (Lai et al, 1994; Wu et al, 2010). The step by step procedure of TOPSIS is given as (Tyagi et al, 2014): Step 1: Generate an evaluation matrix by considering of ‘y’ alternatives and ‘z’ criteria, with the intersection of each alternative and criteria given as xij , we therefore have a matrix (xij ) y×z

(1)

Step 2: Normalize the matrix (Xi j)yxz to convert into the matrix R=(nij)yxz using the normalization formula given as: 𝑛ij =

xij

(2)

y √∑i=1 xij2

Step 3: Determine the weighted normalized decision matrix D = (𝑑𝑖𝑗 )yxz = ( 𝑤𝑗 𝑛𝑖𝑗 )𝑦𝑥𝑧´ 𝑖 = 1.2 … . 𝑦

Step 4: Find out the Positive and Negative ideal solutions

Page 3 of 14

𝐴+ = {(

𝑚𝑎𝑥 𝑚𝑖𝑛 𝑑 |𝑗 ∈ 𝐽) . ( 𝑑 |𝑗 ∈ 𝐽′) 𝑓𝑜𝑟 𝑖 = 1.2. … . 𝑦} = {𝑑1+ . 𝑑2+ . … . 𝑑𝑗+ . … . 𝑑𝑧+ } 𝑖 𝑖𝑗 𝑖 𝑖𝑗

(3)

𝐴− = {(

𝑚𝑎𝑥 𝑚𝑖𝑛 𝑑𝑖𝑗 |𝑗 ∈ 𝐽) . ( 𝑑 |𝑗 ∈ 𝐽′) 𝑓𝑜𝑟 𝑖 = 1.2. … . 𝑦} = {𝑑1− . 𝑑2− . … . 𝑑𝑗− . … . 𝑑𝑧− } 𝑖 𝑖𝑗 𝑖

(4)

Where, J = {𝑗 = 1.2. … 𝑧|𝑗 𝑎𝑠𝑠𝑜𝑐𝑖𝑎𝑡𝑒𝑑 𝑤𝑖𝑡ℎ 𝑡ℎ𝑒 𝑏𝑒𝑛𝑖𝑓𝑖𝑡 𝑐𝑟𝑖𝑡𝑒𝑟𝑖𝑎} J´ = {𝑗 = 1.2. … 𝑧|𝑗 𝑎𝑠𝑠𝑜𝑐𝑖𝑎𝑡𝑒𝑑 𝑤𝑖𝑡ℎ 𝑡ℎ𝑒 𝑐𝑜𝑠𝑡 𝑐𝑟𝑖𝑡𝑒𝑟𝑖𝑎}

Step 5: Calculate the distance/separation from: Positive Ideal Separation 𝑧 𝑆𝑖+ = √∑𝑗=1 (𝑑𝑖𝑗 − 𝑑𝑗+ )2

, i = 1,2, … , y

(5)

, i = 1,2, … , y

(6)

Negative Ideal Separation 𝑧 𝑆𝑖− = √∑𝑗=1 (𝑑𝑖𝑗 − 𝑑𝑗− )2

Step 6: Calculate the Relative closeness coefficient to the Ideal Solution 𝐶𝐶𝑖+ =

Sj− (Sj+

+

Si− )

.

0 < 𝐶𝐶𝑖+ < 1. i = 1.2. … . y

(7)

𝐶𝐶𝑖+ = 1 𝑖𝑓 𝐴𝑖 = 𝐴+ 𝐶𝐶𝑖+ = 0 𝑖𝑓 𝐴𝑖 = 𝐴−

The rank of considered alternatives can be decide, according to the descending order of CCi∗.

Findings This research has been done according to the following steps: Step 1: A vision was determined by the expert team established at the beginning of the implementation and the business vision was expressed as "Becoming a preferred market brand". Step 2: Strategies required for the achievement of the business vision were determined. At the end of this step, the following strategies were decided to be pursued:  Strategy 1 (S1): To design products based on customer requirements.  S2: To adopt new technologies to be used in production phase and to continuously increase product quality.  S3: To improve after-sales service quality by widening service network. Step 3: BSC perspectives and performance indicators were defined (Table 1). It is necessary to mention that Balanced Scorecard is a model and it should be developed according to the special modes of an organization. Also, the number of its perspective can be less or more, BSC is not limited to only four perspectives (financial, customer, internal business process, learning and growth). In this research, "supply chain perspective" and "social accountability perspective" has been added.

Page 4 of 14

Table 1. BSC perspectives and performance indicators BSC perspectives Performance indicators Financial (P1) Cash flow (P11) Total assets (P12) Debts (P13) Customer (P2) Market share (P21) Response rate (P22) Sales volume (P23) Internal business process (P3) Percentage of waste (P31) Check times of inventory (P32) Research and development costs (P33) Learning and growth (P4) Staff productivity (P41) Per capita educational investment (P42) Motivation (P43) Supply chain (P5) Time for planning and forecasting (P51) Total cost of delivery (P52) On time delivery (P53) Social accountability (P6) Private sector development (P61) Internationally accepted standards (P62) Organization participation in social problem solving (P63)

Step 4: local weights of the strategies, BSC perspectives and performance indicators are calculated. By using views of experts prioritizing the Strategies is done by using software TOPSIS. Results obtained from TOPSIS software are given in Chart 1 and Table 2.

Chart 1. Strategies local weights (using TOPSIS software)

Strategies Weight

Table 2. Strategies local weights S1 S2 0.374 0.289

S3 0.337

After the determination of the strategic priorities, BSC perspective weights were defined on the basis of these strategies. Results obtained from TOPSIS software are given in Charts 2-4 and Tables 3-5.

Page 5 of 14

Chart 2. Local weights of BSC perspectives with respect to S1 (using TOPSIS software)

BSC perspectives Local weight

Table 3. Local weights of BSC perspectives with respect to S1 P1 P2 P3 P4 0.161 0.225 0.269 0.132

P5 0.105

P6 0.108




P5 0.070


Page 6 of 14

P6 0.062



P5 0.122

P6 0.087

Global weights of BSC perspectives were calculated as follows, by multiplying the weights listed in Tables 3-5 with the strategy weights:

WBSC

P1 0.266 P2 0.239 P3 0.202 = = P4 0.106 P5 0.1 [P6] [0.087]

In the last phase of this step, local weights of the performance indicators were determined. The local weights calculated for performance indicators are given in Charts 5-10 and Tables 6-11.

Chart 5. Local weights of P1 (using TOPSIS software)

P1 Local weight

Table 6. Local weights of P1 P11 P12 0.386 0.331


Page 7 of 14

P13 0.283

P2 Local weight


P23 0.158


P3 Local weight


P33 0.458


P4 Local weight


Page 8 of 14

P43 0.286


P5 Local weight


P53 0.402


P6 Local weight


P63 0.370

Step 5: In this step, dependence among BSC perspectives were determined. Interdependent weights of the BSC perspectives are calculated and the dependencies among the perspectives are considered. Dependence among the perspectives is determined by analyzing the impact of each perspective on every other perspective using pairwise comparisons. Interdependent weights of the perspectives are computed by multiplying the dependence matrix of the perspectives we obtained with the local weights of perspectives provided in step 4. The interdependent weights of the perspectives are calculated as follows:

Page 9 of 14

WBSC

P1 0.208 P2 0.212 P3 0.223 = = P4 0.138 P5 0.111 [P6] [0.108]

Step 6: Using interdependent weights of the perspectives (step 5) and local weights performance indicators, global weights for the indicators are calculated in this step. Global indicators weights are computed by multiplying local weight of the indicators with the interdependent weight of the perspective to which it belongs. Computed values are shown in Table 12. BSC perspectives P1

P2

P3

P4

P5

P6

Table 12. Computed global weights of performance indicators Interdependent Performance Weights weights indicators 0.208 P11 0.386 P12 0.331 P13 0.283 0.212 P21 0.131 P22 0.711 P23 0.158 0.223 P31 0.337 P32 0.205 P33 0.458 0.138 P41 0.335 P42 0.379 P43 0.286 0.111 P51 0.410 P52 0.188 P53 0.402 0.108 P61 0.241 P62 0.389 P63 0.370

Global weights 0.080 0.069 0.059 0.028 0.151 0.033 0.075 0.046 0.102 0.046 0.052 0.039 0.046 0.021 0.045 0.026 0.042 0.040

Steps 7: In this stage, performance of the organization is determined by using the global weight values of performance indicators Table 12 and the linguistic measurement scale (Table 13). The calculations are shown in Table 14. Measure the performance indicators. Linguistic variables proposed by Cheng et al (1999) are used in this step. The memberships functions of these linguistic variables are shown on Figure 1 and the average value related with these variables are shown Table 13 (Yüksel and Dağdeviren, 2010).

Figure 1. Membership functions of linguistic values for performance indicator rating

Page 10 of 14

Table 13. Linguistic values and mean of fuzzy numbers Linguistic values The mean of fuzzy numbers Very high (VH) 1 High (H) 0.75 Medium (M) 0.5 Low (L) 0.25 Very low (VL) 0

Performance indicators P11 P12 P13 P21 P22 P23 P31 P32 P33 P41 P42 P43 P51 P52 P53 P61 P62 P63

Table 14. Performance measured by using the proposed BSC–TOPSIS model Global weights Linguistic Scale value Performance (A) evaluations (B) (A×B) 0.080 VH 1 0.080 0.069 H 0.75 0.052 0.059 H 0.75 0.044 0.028 H 0.75 0.021 0.151 VH 1 0.151 0.033 M 0.5 0.016 0.075 H 0.75 0.056 0.046 H 0.75 0.034 0.102 VH 1 0.102 0.046 M 0.5 0.023 0.052 M 0.5 0.026 0.039 L 0.25 0.1 0.046 H 0.75 0.034 0.021 H 0.75 0.016 0.045 M 0.5 0.022 0.026 M 0.5 0.013 0.042 M 0.5 0.021 0.040 L 0.25 0.1

Total performance score was calculated by multiplying the global weights and scale values of performance indicators and then by summing the resulting performance levels.

Conclusions Nowadays what is considered about most of the organizations is the assessment of organizational performance. One of the tools to do it is Balanced Scorecard (BSC). This technique helps that organization to reach its goals by focusing on the organization strategies and making logical and true relations with all the organizational aspects. In this research, a method of decision making along with BSC has been used. In order to use experts' ideas, TOPSIS method has been used. In this research, a paper products manufacturer company in Iran has been studied. Finally, Total performance score was calculated by multiplying the global weights and scale values of performance indicators and then by summing the resulting performance levels. The model proposed in the scope of this study was related with a paper products manufacturer company; however, it can also be adapted to different businesses. Modifications may be required on the proposed system due to two reasons: firstly, the components constituting the analytical structure of the proposed model – namely, strategies, BSC perspectives and performance indicators – may vary depending on the business vision. Secondly, relationships or dependencies among BSC perspectives or performance indicators may also vary. Modifications and adaptations to be made due to these two reasons will enable the use of this model in other enterprises.

References Abdul Rahman, Noorul Shaiful Fitri. (2012). A decision making support of the most efficient steaming speed for the liner business industry. European Journal of Business and Management. 4(18):37-49.

Page 11 of 14

Abran, Alain. and Buglione, Luigi. (2003). A multidimensional performance model for consolidating balanced scorecards. Advances in Engineering Software, June 2003, 34(6):339-349. Atkinson, Anthony A. and Balakrishnan, Ramji. and Booth, Peter. and Cote, Jane M. and Groot, Tom. and Malmi, Teemu. and Roberts, Hanno. and Uliana, Enrico. and Wu, Anne. (1997). New directions in management accounting research. Journal of Management Accounting Research, January 1997, 9:79-108. Balli, Serkan. and Korukoglu, Serdar. (2009). Operating system selection using fuzzy AHP and TOPSIS methods. Mathematical and Computing Modelling. 14(2):119-130. Banker, Rajiv D. and Chang, Hsihui. and Pizzini, Mina J. (2004). The Balanced Scorecard: Judgmental Effects of Performance Measures Linked to Strategy. The Accounting Review, January 2004, 79(1):1-23. Barros, Carlos Pestana. and Wanke, Peter. (2015). An analysis of African airlines efficiency with two-stage TOPSIS and neural networks. Journal of Air Transport Management. May-June 2015. 44-45:90-102. Bilbao-Terol, Amelia. and Arenas-Parra, Mar. and Cañal-Fernández, Verónica. and Antomil-Ibias, José. (2014). Using TOPSIS for assessing the sustainability of government bond funds. Omega. December 2014. 49:1-17. Cardinaels, Eddy. and van Veen-Dirks, Paula M.G. (2010). Financial versus non-financial information: The impact of information organization and presentation in a balanced scorecard. Accounting, Organizations and Society, August 2010, 35(6):565-578. Chen, Shaokuan. and Leng, Yan. and Mao, Baohua. and Liu, Shuang. (2014). Integrated weight-based multicriteria evaluation on transfer in large transport terminals: A case study of the Beijing South Railway Station. Transportation Research Part A: Policy and Practice. August 2014. 66:13-26. Cheng, Ching-Hsue. and Yang, Kuo-Lung. and Hwang, Chia-Lung. (1999). Evaluating attack helicopters by AHP based on linguistic variable weight. European Journal of Operational Research, 16 July 1999, 116(2): 423-435. Dilla, William N. and Steinbart, Paul John. (2005). Relative Weighting of Common and Unique Balanced Scorecard Measures by Knowledgeable Decision Makers. Behavioral Research in Accounting, February 2005, 17(1):43-53. Dinçer, Ömer. (2004). Stratejik Yönetim ve İşletme Politikası. İstanbul: Beta Yayınevi. Eren, Erol. (2002). Stratejik Yönetim ve İşletme Politikası. Beta Basım Yayım Dağıtım, İstanbul. Ertuğrul, İrfan. and Karakaşoğlu, Nilsen. (2009). Performance evaluation of Turkish cement firms with fuzzy analytic hierarchy process and TOPSIS methods. Expert Systems with Applications. January 2009. 36(1): 702715. Feili, Hamidreza. and Qomi, Mojtaba. and Beikzadeh, Elham. and Khorasani Gerdehkohi, Shiva. and Sheibani, Sajedeh. (2016a). Prioritization of Scientific-Didactic Centers using TOPSIS Technique for Achieving Sustainable Development. 5th International Conference on Science and Engineering, December 13 th 2016, Paris, France. Feili, Hamidreza. and Qomi, Mojtaba. and Khani, Hadi. and Naserikouchehbiouki, Vahid. and Motahariyan, Mohammad Amin (2016b). Assessment of Organizational Performance using Balanced Scorecard for Increasing Export Potential in Jujube Packing Company. 4 th International Conference on Research in Science and Technology, July 28th 2016, Saint Petersburg, Russia. Feili, Hamidreza. and Qomi, Mojtaba. and Rezaei, Farzan. and Rezaei, Amir Hossein. and Irannejad, Mohammad Hossein. (2016c). Assessment of Organizational Performance using Balanced Scorecard in Grape Seed Oil Production Factory. 3rd International Conference on Management and Industrial Engineering, September 14 th 2016, Frankfurt, Germany. Feng, Cheng-Min. and Wang, Rong-Tsu. (2001). Considering the financial ratios on the performance evaluation of highway bus industry. Transport Reviews. 21(4):449-467. Hassan, Mohammad Nurul. and Hawas, Yaser E. and Ahmed, Kamran. (2013). A multi-dimensional framework for evaluating the transit service performance. Transportation Research Part A: Policy and Practice. April 2013. 50:47-61. Humphreys, Kerry A. and Trotman, Ken T. (2011). The Balanced Scorecard: The Effect of Strategy Information on Performance Evaluation Judgments. Journal of Management Accounting Research, December 2011, 23(1):81-98. Hung, Chia-Chang. and Chen, Liang-Hsuan. (2009). A fuzzy TOPSIS decision making model with entropy weight under A fuzzy TOPSIS decision making model with entropy weight under intuitionistic fuzzy environment. Proceeding on the International of Multi Conference of Engineers and Computer Scientist (IMECS), Vol. 1, 613. Hwang, Ching-Lai. and Yoon, Kwangsun. (1981). Multiple Attribute Decision Making. Methods and Applications a State of the Art Survey. Springer-Verlag, New York. Page 12 of 14

Ittner, Christopher D. and Larcker, David F. (2001). Assessing empirical research in managerial accounting: a value-based management perspective. Journal of Accounting and Economics, December 2001, 32(1-3):349410. Jahanshahloo, Gholam Reza. and Hosseinzadeh Lotfi, Farhad. and Izadikhah, Mohammad. (2006). An algorithmic method to extend TOPSIS for decision-making problems with interval data. Applied Mathematics and Computation. 15 April, 2006. 175(2):1375-1384. Kaplan, Robert S. and Norton, David P. (1992). The Balanced Scorecard: Measures that Drive Performance, Harvard Business Review. January–February 1992, 70(1):71-79. Kaplan, Robert S. and Norton, David P. (1996a). Şirket Stratejisini Eyleme Dönüştürmek. Sistem Yayıncılık. Kaplan, Robert S. and Norton, David P. (1996b). Using the Balanced Scorecard as a Strategic Management System. Harvard Business Review, January–February 1996, 74(1):75–85. Kaplan, Steven E. and Petersen, Michael J. and Samuels, Janet A. (2008). Effects of Subordinate Likeability and Balanced Scorecard Format on Performance-Related Judgments. Advances in Accounting, 23:85–111. Karimi, Mohammad Sharif. and Yusop, Zulkornain. and Siong Hook, Law. (2010). Location decision for foreign direct investment in ASEAN countries: A TOPSIS approach, International Research Journal of Finance and Economics, MPRA Paper 15000, University Library of Munich, Germany. 36:196-207. Lai, Young-Jou. and Liu, Ting-Yun. and Hwang, Ching-Lai. (1994). TOPSIS for MODM. European Journal of Operational Research. 76(3):486-500. Lee, Amy H.I. and Chen, Wen-Chin. and Chang, Ching-Jan. (2008). A fuzzy AHP and BSC approach for evaluating performance of IT department in the manufacturing industry in Taiwan. Expert Systems with Applications, January 2008, 34(1):96-107. Leung, Lawrence C. and Lam, Kevin C. K. and Cao, Dong. (2006). Implementing the balanced scorecard using the analytic hierarchy process and the analytic network process. Journal of the Operational Research Society, June 2006, 57(6):682-691. Libby, Theresa. and Salterio, Steven E. and Webb, Alan. (2004). The Balanced Scorecard: The Effects of Assurance and Process Accountability on Managerial Judgment. The Accounting Review, October 2004, 79(4):1075-1094. Liedtka, Stephen L. and Church, Bryan K. and Ray, Manash R. (2008). Performance Variability, Ambiguity Intolerance, and Balanced Scorecard-Based Performance Assessments. Behavioral Research in Accounting: Fall, 20(2):73-88. Lipe, Marlys Gascho. and Salterio, Steven E. (2000). The Balanced Scorecard: Judgmental Effects of Common and Unique Performance Measures. The Accounting Review, July 2000, 75(3):283-298. Lipe, Marlys Gascho. and Salterio, Steven E. (2002). A note on the judgmental effects of the balanced scorecard’s information organization. Accounting, Organization and Society, August 2002, 27(6):531-540. Nørreklit, Hanne. (2000). The balance on the balanced scorecard a critical analysis of some of its assumptions. Management Accounting Research, March 2000, 11(1):65-88. Olson, David L. (2004). Comparisons of weights in TOPSIS models. Mathematical and Computer Modelling. October 2004. 40(7-8):721-727. Pirdavani, Ali. and Brijs, Tom. and Wets, Geert. (2010). A Multiple Criteria Decision-Making Approach for Prioritizing Accident Hotspots in the Absence of Crash Data. Transport Reviews. 30(1):97-113. Ravi, Vadlamani. and Shankar, Ravi. and Tiwari, Manoj Kumar. (2005). Analyzing alternatives in reverse logistics for end-of-life computers: ANP and balanced scorecard approach. Computers and Industrial Engineering, March 2005, 48(2):327-356. Roberts, Michael L. and Albright, Thomas L. and Hibbets, Aleecia R. (2004). Debiasing Balanced Scorecard Evaluations. Behavioral Research in Accounting, February 2004, 16(1):75-88. Sohn, Myung Ho. and You, Taewoo. and Lee, Seok-Lyong. and Lee, Heeseok. (2003). Corporate strategies, environmental forces, and performance measures: a weighting decision support system using the k-nearest neighbor technique. Expert Systems with Applications, October 2003, 25(3):279-292. Tayler, William B. (2010). The Balanced Scorecard as a Strategy-Evaluation Tool: The Effects of Implementation Involvement and a Causal-Chain Focus. The Accounting Review, May 2010, 85(3):1095-1117. Tsai, Hui-Yin. and Huang, Bao-Huey. and Wang, An-Siou. (2008). Combining ANP and TOPSIS concepts for evaluation the performance of property liability insurance companies. Journal of Science Social. 4(1):56-61. Tyagi, Mohit. and Kumar, Pradeep. and Kumar, Dinesh. (2014). A hybrid approach using AHP-TOPSIS for analyzing e- SCM performance. Procedia Engineering. 97:2195-2203. Ülgen, Hayri. and Mirze, S. Kadri. (2004). İşletmelerde Stratejik Yönetim, Literatür Yayıncılık, İstanbul. Page 13 of 14

Upton, David R. and Arrington, C. Edward. (2012). Implicit racial prejudice against African-Americans in balanced scorecard performance evaluations. Critical Perspectives on Accounting, June 2012, 23(4-5):281297. Wang, Bing. and Nistor, Ioan. and Murty, Tad. and Wei, Yi-Ming. (2014). Efficiency assessment of hydroelectric power plants in Canada: A multi criteria decision making approach. Energy Economics. November 2014. 46:112-121. Wanke, Peter. and Barros, Carlos Pestana. and Chen, Zhongfei. (2015). An analysis of Asian airlines efficiency with two-stage TOPSIS and MCMC generalized linear mixed models. International Journal of Production Economics. November 2015. 169:110-126. Wong-On-Wing, Bernard. and Guo, Lan. and Li, Wei. and Yang, Dan. (2007). Reducing Conflict in Balanced Scorecard Evaluations. Accounting Organizations and Society, February 2007, 32(4–5):363-377. Wu, Desheng. and Olson, David L. (2006). A TOPSIS Data Mining Demonstration and Application to Credit Scoring. International Journal of Data Warehousing and Mining. 2(3):1-10. Wu, Cheng-Shiung. and Lin, Chin-Tsai. and Lee, Chuan. (2010). Optimal marketing strategy: A decision-making with ANP and TOPSIS. International Journal of Production Economics. September 2010. 127(1):190-196. Yüksel, İhsan. and Dağdeviren, Metin. (2007). Using the analytic network process (ANP) in a SWOT analysis— A case study for a textile firm. Information Sciences, 15 August 2007, 177(16):3364-3382. Yüksel, İhsan. and Dağdeviren, Metin. (2010). Using the fuzzy analytic network process (ANP) for Balanced Scorecard (BSC): A case study for a manufacturing firm. Expert Systems with Applications, March 2010, 37(2):1270-1278.

Page 14 of 14