A complimentary fuzzy approach for the assessment

Int. J. Industrial and Systems Engineering, Vol. 9, No. 1, 2011

21

A complimentary fuzzy approach for the assessment of academic performance Ebru Akcapinar Sezer* Department of Computer Engineering, Hacettepe University, Beytepe 06800, Ankara, Turkey E-mail: [email protected] *Corresponding author

Candan Gokceoglu Department of Geological Engineering, Hacettepe University, Beytepe 06800, Ankara, Turkey E-mail: [email protected] Abstract: Assessment of academic performance is highly important and critical point for researchers, but this is a difficult task. The h-index is one of the most popular indicators to assess the academic performance. As the expression of academic performance is too difficult by single mathematical expression, it should be done with a mathematical expression to apply simply and safely throughout the world. However, this requirement does not change the nature of the problem. In fact, this is a real fuzzy problem and it should be tested by using fuzzy approaches. For this reason, construction of a fuzzy inference system to assess the academic performance is the main target of the current study. Although a close relationship between the h-index and the FAPI is obtained, there are still some slight differences. Obtained results show that problem is fuzzy, but its application is not very simple. For this reason, the fuzzy inference system should be considered as a complimentary tool for the assessment of academic performance, especially in carrier planning of the researchers. Keywords: fuzzy inference system; h-index; academic performance; FAPI; fuzzy academic performance index. Reference to this paper should be made as follows: Sezer, E.A. and Gokceoglu, C. (2011) ‘A complimentary fuzzy approach for the assessment of academic performance’, Int. J. Industrial and Systems Engineering, Vol. 9, No. 1, pp.21–33. Biographical notes: Ebru Akcapinar Sezer received her BSc degree from the Department of Computer Engineering at Hacettepe University in 1996. She received her MSc and PhD degrees from the Department of Computer Engineering at the same university in 1999 and 2006, respectively. She is currently an Associated Lecturer in the Department of Computer Engineering at Hacettepe University. Her areas of interest are semantic web technologies and fuzzy system applications.

Copyright © 2011 Inderscience Enterprises Ltd.

22

E.A. Sezer and C. Gokceoglu Candan Gokceoglu received his BSc degree from the Department of Hydrogeological Engineering at Hacettepe University in 1989. He received his MSc and PhD degrees from the Department of Geological Engineering at the same university in 1993 and 1997, respectively. He is currently a Professor in the Division of Applied Geology of the Department of Geological Engineering at Hacettepe University, Ankara, Turkey. He has published more than 60 academic papers. He is also an active reviewer for more than 25 scientific journals and editorial board member of 6 academic journals. His areas of interest are landslides, rock mechanics and fuzzy system applications. An earlier version of this paper was presented at FUZZYSS’ 2009: 1st International Fuzzy Systems Symposium, Ankara, Turkey, 1–2 October 2009.

1

Introduction

The evaluation of academic performance has always been an attractive issue for scientists. There is no obvious conceptual definition on productivity or quality of researchers, but there are number of parameters, such as number of publications and total citations, etc. to express researchers’ productivity and impact. Generally, total publication number is accepted as the metric of productivity while total citation number is accepted as metric of impact. Sometimes, the number of authors in publications, journal impact factors, related discipline averages are added to these parameters to qualify or to increase the meaning of numbers. All parameters listed here are numeric and we observed that most of the expression designed for the assessment of scientists uses either all of these parameters or some of them. Hirsch (2005) proposed a single and simple h-index to characterise the significance of the performance of scientists. The h-index means that a scientist has at least h papers having h citation each. In our opinion, the most attractive aspect of h-index is that, it balances the citation and publication numbers, and moves focus from the total number of citation and total number of publication to citation of each publication and defines thresholds for researchers. For this reason, it can be said that h-index highlights the quality of researchers. As can be seen, h-index has a simple definition that has not only statistical, but also frequency-based perspective. In fact, h-index definition can be interpreted in several ways. It reflects the line of ‘y = x’ in twodimensional space in which x-axis shows publications and y-axis corresponds to citations. Another interpretation on h-index can be that it is not a formula, it is a rule for classification of researchers, such as ‘IF the researcher has at least n publication AND each of them has at least n citation THEN h = n’. Because of these characteristics of h-index, it has an important popularity to express academic performance. At the same time, it is the most suitable method to compare between the results obtained from the fuzzy model developed in this study and the h-index. The critics about h-index are mentioned in later sections. After the introduction of h-index, some proposals were carried out. While some of those proposals present case studies using the h-index and discuss the special observations, the other proposals describe a new variation by complementing or putting extension of its original definition. Some of these studies which are outstanding and help us to figure out solutions are summarised here. Cronin and Meho (2006) examined the correlation between raw citation counts and h-index counts in information sciences (ISs)

A complimentary fuzzy approach

23

and they found a close correlation between them. Additionally, they rank each person selected in IS field with four measures: total citation, total citations excluding self citation, h-index, h-index excluding self citation. Then, they discussed the effects of self-citation on h-index by using rank table. Batista et al. (2006) complemented the h-index and called hi and defined another parameter N a(T ) being the total number of authors in the considered h papers. Their conclusions showed that hi presents better performance to compare different research areas. Sidiropoulos et al. (2007) proposed an extension to the original h-index to reflect various latent strong facts in citation network. They tried to develop new h-indices as simple as the original and they contributed the normalised h-index for scientist ranking, yearly h-index for journal/conference ranking, contemporary h-index and trend h-index for brilliant young scientist and trendsetters. van Raan (2006) gave the results on statistical correlation between h-index and several standard bibliometric indicators as well as peer review judgement. His results showed that the bibliometric ‘crown indicator’ is more appropriate than h-index for smaller groups in fields with less heavy citation traffic. Vanclay (2007) discussed the robustness of the h-index and supported the usage of h-index over alternatives, like journal impact factor. Kelly and Jennions (2007) examined the h-index performance for career assessment and they applied the h-index on ecologist and evolutionary biologist. They suggested that the h-index is not obviously superior to the other indices relying on citation and publication count to assess research performance. Costa (2007) proposed a new version of h-index to involve the automated identification of virtual citations which can provide complementary and unbiased quantification of the relevance of scientific works. Burrell (2007) suggested a stochastic model for author productivity and discussed the h-index. He concluded that the h-index logged publication rate and citation rate, at least moderate citation rates. While listing the relevant works on the h-index, we excluded some works on the assessment of journals by using h-index. In this study, individual’s performance has been examined and it is observed that h-index has some advantages within this context: it is simple, robust and it gives good results in closed groups with long carrier. However, it may cause some disadvantages regarding young researchers; it does not cover the elapsed time duration to reach some h-index. Hirsch (2005) defines some thresholds to classify academic performance. In our opinion, these thresholds cannot be applied to all scientific areas. Moreover, continuity is important in scientific work and it should be taken into consideration. Time–continuity pair can solve the disadvantages of h-index mentioned above. The other feature of h-index is that it cannot be normalised. Especially in multidisciplinal comparisons, it may result in some problems. Finally, it bounds the total publication number with total citation or vice versa. If there are balanced publication and citation number for scientist it may not create a problem. However, if there are extreme values in citation or publication number, it may be a big problem for scientists (Figure 1). If a mathematical expression is proposed for assessment of academic performance, a balance between number of publications and total citations should be taken into consideration. Differences in numbers may not mean because there is a difference in productivity or high input values may cause sharp increases in the output of the expression. For this reason, academic performance assessment is a fuzzy problem due to its complex mathematical formulation and its nature. Moreover, in order to present a well-balanced approach in the evaluation of academic performance, not only the total publications and total citations, but also the academic life span is also required. In this

24

E.A. Sezer and C. Gokceoglu

study, productivity, impact and academic life span of researchers are used to constitute inputs of fuzzy inference system and the fuzzy academic performance index (FAPI) is produced. Although some critics are listed about h-index, it is selected to compare with FAPI because it is the most similar approach to solve this problem. The productivity and impact are divided into five fuzzy sets and the academic life span of scientist is divided into three fuzzy sets. To extract some inferences from the system, some linguistic rules were designed. In addition, total publication number of a scientist is the crisp input of productivity, total citation number of a scientist is the crisp input of impact and 2008 minus first publication year is the crisp input of academic life span. FAPI is obtained as the output and the comparison between FAPI and the h-index is performed. This paper is organised as follows: selection criteria and description of data are explained in Section 2; construction of the fuzzy inference model is explained in Section 3; comparison between the h-index and FAPI is presented in Section 4 and results and conclusion are given in Section 5. Figure 1

2

Graphs showing the possible neglected productivity and impact values of a researcher

Database

To assess the academic performance by the h-index and the fuzzy approach, we used editorial board list of five earth science journals, such as Engineering Geology, Tectonophysics, Earth and Planetary Science Letters, Int. J. Volcanology and Geothermal Researches and Geomorphology. The name of each board member was searched from the science citation index expanded (web of science) published by Institute of Scientific Information (now Thomson Scientific). Due to some name similarities, we selected only the publications related with earth sciences. For this reason, some papers related with other science areas were eliminated. However, some authors were excluded completely because the same name similarity problems could not be solved. As a result of this search, total 208 authors were used as the cases. The minimum, maximum and average values of number of total publications, number of total citations, academic life span and the h-index are listed in Table 1. The histograms showing the distributions of number of total publications, number of total citations and the h-index are given in Figure 2. In our opinion, the data set employed in the current study can represent any researcher population behaviour. Before the construction of the fuzzy inference system,


25

all data was normalised by division the maximum value, to be independent from magnitude of data and to provide standardisation among the inputs and the outputs. Table 1

Summary of parameters used in the study (n = 208)

Parameter

Minimum

Total number of publications

Maximum

Average

1

213

35.6

Total number of citations

1

4,544

679.4

Academic life span

1

48

20.3

h-index

1

36

11.8

Figure 2

Histograms showing the distributions of (a) total number of publications, (b) total number of citations and (c) h-indices of the cases employed in the study

26


Figure 2

3

Histograms showing the distributions of (a) total number of publications, (b) total number of citations and (c) h-indices of the cases employed in the study (continued)

Fuzzy inference system

In recent years, the fuzzy inference systems have been used in various areas, such as biology (Keshwani et al., 2008; Salski and Holsten, 2006), agriculture (Mazloumzadeh et al., 2008), medicine (Alayon et al., 2007; Gascon et al., 2007; Nunes et al., 2006), economics (Medina and Moreno, 2007), material sciences (Akkurt et al., 2004; Atzeni et al., 2008; Yaldiz et al., 2006) and earth sciences (Gokceoglu, 2002; Gokceoglu and Zorlu, 2004; Sonmez et al., 2003). According to Setnes et al. (1998), an interesting and perhaps the most attractive characteristic of fuzzy models compared with other conventional methods, like statistics, is that they are able to describe complex and nonlinear multivariable problems in a transparent way. In this study, the Mamdani fuzzy inference system was employed to assess academic performance, because the Mamdani fuzzy inference system uses linguistic ‘if-then’ rules. The model constructed in the current study contains three inputs, such as total number of publications, total number of citations and academic life span, and one output like FAPI. For inference in a rule-based fuzzy model, the fuzzy propositions need to be represented by an implication function called a fuzzy ‘if-then’ rule or a fuzzy conditional statement (Alvarez Grima, 2000). A fuzzy set is a collection of paired members consisting of members and degrees of ‘support’ or ‘confidence’ for those members. In a discrete form, the fuzzy set ‘about 7’ might be expressed as (0.1/5, 0.7/6, 1/7, 0.7/8, 0.1/9). In a fuzzy set notation, the members after the slash (/) are the members of the set (or appropriate numerical grades in each case), and the values before the slash are the degree of confidence or ‘membership’ of those numbers. The use of fuzzy sets to present linguistic


27

terms enables one to represent more accurately and consistently something which is fuzzy (Juang et al., 1992). A linguistic variable whose values are words, phrases or sentences are labels of fuzzy sets (Zadeh, 1973). In the current study, two inputs (total number of publications and total number of citations) and the output were formed by five fuzzy sets, such as very low, low, moderate, high, very high while one input (academic life span) was formed by three fuzzy sets, such as short, moderate, long (Figure 3). The membership values were assigned equally. The fuzzy model was built by using the linguistic rules (Appendix). Total 68 ‘if-then’ linguistic rules were formed logically. When FIS proposed in the current study is used, an inference system can be constructed easily by using the membership functions shown in Figure 3 and the rules given in Appendix. After construction of FIS, to obtain FAPI of only one researcher is an easy task. However, FAPI should not be used single-handedly, because the FAPI values vary depending on the scientific discipline. For this reason, FAPI should be considered as a complimentary approach for the h-index. Figure 3

4

The input and the output fuzzy membership functions of the fuzzy inference system developed in the study

Comparison

A comparison between the results obtained from the fuzzy inference system and the h-index was performed. A close relationship (r = 0.90) between FAPI and the h-index (Figure 4) was observed. In fact, this result is normal, because the philosophies of two approaches are not too different. Despite a general trend between both indices exists, some differences were observed. As can be understood from the definition of the h-index, the h-index considers only the highly cited papers of a researcher. In other words, the h-index does not consider total number of publications and total number of citations. As can be seen from Figure 1, some authors can have extreme values in total publications or total citations although these two parameters represent productivity and impact. However, FAPI considers the total number of publications, total number of citations and academic life span. If a simple comparison is carried out, the difference between these two indices

28


can be understood clearly. The first researcher has 83 total publications and 1,247 citations while the second one has 57 total publications and 1,066 citations (Table 2). The academic life spans of these researchers are 33 and 32 years, respectively. The h-index values of these researchers are the same (h-index = 20). When considering the productivity (number of total publications), it is obvious that there is a difference. The h-index does not reflect this difference. The FAPI values of these researchers are obtained as 0.366 and 0.262. As an another example, there are two researchers having same h-index value (21) and same total number of publication (48) and approximately same total number of citations but their academic life spans are different. This difference can be modelled by FAPI as can be seen from their fuzzy academic performance indices (0.238 and 0.248) (Table 2). FAPI is not an alternative for the h-index. However, FAPI can be used for the assessment of the researchers having same h-index. Figure 4

The relationship between the h-index and the fuzzy academic performance index

A complimentary fuzzy approach Table 2

29

Some examples showing the differences in the fuzzy academic performance index h-index

Total number of publications

Total number of citations

Academic life span

FAPI

1

20

83

1,247

33

0.366

2

20

57

1,066

32

0.262

3

21

48

1,422

31

0.238

4

21

48

1,442

18

0.248

No. of the researcher

5

Results and conclusion

The following results and conclusions can be drawn from the current study. Although, the h-index proposed for the assessment of academic performance of scientists is highly creative, simple and plausible, it neglects total number of publications and total number of citations. For this reason, some scientists may have same h-index values but their productivity and impacts may be different so these differences may be disabled. As can be seen in Figure 4, FAPI produces different degrees to each scientist reflecting differences of his/her performance in productivity or impact axis. This type of ranking is quite useful in assessment of academic performance with the same h-index values. As can be seen in Figure 4, there are different FAPI values in each h-index value. There are four points that differ clearly from the other points at the same h-index value in our cases. The percentage of this type of extreme situations is 2% and if this percentage is applied to all authors with related domain in Turkey or world wide, it may reach the value which cannot be negligible. This study shows that the fuzzy inference systems are the useful tools for the assessment of academic performances. However, the fuzzy approach should be considered as a complimentary tool for the assessment of academic performance. It is not an alternative tool for the h-index, because h-index is simple and safe. However, FAPI can model the researchers activities more detailed than the h-index. In other words, it can be useful to express academic performance of researchers who have same h-index and same academic discipline with a detailed and complimentary approach.

References Akkurt, S., Tayfur, G. and Can, S. (2004) ‘Fuzzy logic model for the prediction of cement compressive strength’, Cement and Concrete Research, Vol. 34, No. 8, pp.1429–1433. Alayon, S., Robertson, R., Warfield, S.K. and Ruiz-Alzola, R. (2007) ‘A fuzzy system for helping medical diagnosis of malformations of cortical development’, Journal of Biomedical Informatics, Vol. 40, No. 3, pp.221–235. Alvarez Grima, M. (2000) Neuro-fuzzy Modeling in Engineering Geology, Rotterdam: A.A. Balkema. Atzeni, C., Pia, G., Sanna, U. and Spanu, N. (2008) ‘A fuzzy model for classifying mechanical properties of vesicular basalt used in prehistoric buildings’, Materials Characterization, Vol. 59, No. 5, pp.606–612. Batista, P.D. Campiteli, M.G., Kinouchi, O. and Matinez, A.S. (2006) ‘Is it possible to compare researchers with different scientific interests’, Scientometrics, Vol. 68, No. 1, pp.179–189.

30


Burrell, Q.L. (2007) ‘Hirsch’s h-index: a stochastic model’, Journal of Informatics, Vol. 1, pp.16–25. Costa, L.F. (2007) ‘On the dynamics of the h-index in complex networks with coexisting communities’. Available at: arXiv.org:physics/0609116, accessible via http://arxiv.org/abs/ physics/0609116. Cronin, B. and Meho, L.I. (2006) ‘Using the h-index to rank influential information scientist’, Journal of the American Society for Information Science and Technology, Vol. 57, No. 9, pp.1275–1278. Gascon, F., Fuente, D. and Lozano, J. (2007) ‘On macroeconomic characteristics of pharmaceutical generics and the potential for manufacturing and consumption under fuzzy conditions’, Artificial Intelligence in Medicine, Vol. 41, No. 3, pp.223–235. Gokceoglu, C. (2002) ‘A fuzzy triangular chart to predict the uniaxial compressive strength of the Ankara agglomerates from their petrographic composition’, Engineering Geology, Vol. 66, Nos. 1–2, pp.39–51. Gokceoglu, C. and Zorlu, K. (2004) ‘A fuzzy model to predict the uniaxial compressive strength and the modulus of elasticity of a problematic rock’, Engineering Applications of Artificial Intelligence, Vol. 17, No. 1, pp.61–72. Hirsch, J.E. (2005) ‘An index to quantify an individual’s scientific research output’, Proceedings of the National Academy of Sciences of the United States of America, Vol. 102, No. 46, pp.16569–16572. Available at: arXiv:physics/0508025, accessible via http://arxiv.org/abs/ physics/0508025. Juang, C.H., Lee, D.H. and Sheu, C. (1992) ‘Mapping slope failure potential using fuzzy sets’, Journal of Geotechnical Engineering ASCE, Vol. 118, No. 3, pp.475–493. Kelly, C.D. and Jennions, M.D. (2006) ‘The h-index and career assessment by numbers’, Trends in Ecology and Evolution, Vol. 21, No. 4, pp.167–170. Keshwani, D.R., Jones, D.D., Meyer, G.E. and Brand, R.M. (2008) ‘Rule-based Mamdani-type fuzzy modeling of skin permeability’, Applied Soft Computing, Vol. 8, No. 1, pp.285–294. Mazloumzadeh, S.M., Shamsi, M. and Nezamanadi-Pour, H. (2008) ‘Evaluation of general-purpose lifters for the date harvest industry based on a fuzzy inference system’, Computers and Electronics in Agriculture, Vol. 60, No. 1, pp.60–66. Medina, S. and Moreno, J. (2007) ‘Risk evaluation in Colombian electricity market using fuzzy logic’, Energy Economics, Vol. 29, No. 5, pp.999–1009. Nunes, C.S., Mahfouf, M. and Linkens, D.A. (2006) ‘Fuzzy modelling for controlled anaesthesia in hospital operating theatres’, Control Engineering Practice, Vol. 14, No. 5, pp.563–572. Salski, A. and Holsten, B. (2006) ‘A fuzzy and neuro-fuzzy approach to modeling cattle grazing on pastures with low stocking rates in Central Europe’, Ecological Informatics, Vol. 1, No. 3, pp.269–276. Setnes, M., Babuska, R. and Verbruggen, H.B. (1998) ‘Rule-based modeling: precision and transparency’, IEEE Transactions on Systems Man and Cybernetics, Part C, Vol. 28, pp.165–169. Sidiropoulos, A., Katsaros, D. and Manolopoulos, Y. (2007) ‘Generalized h-index for disclosing latent facts in citation networks’, Scientometrics, Vol. 72, No. 2, pp.253–280. Sonmez, H., Gokceoglu, C. and Ulusay, R. (2003) ‘An application of fuzzy sets to the geological strength index (GSI) system used in rock engineering’, Engineering Applications of Artificial Intelligence, Vol. 16, No. 3, pp.251–269. Vanclay, J.K. (2007) ‘On the robustness of the h-index’, Journal of the American Society for Information Science and Technology, Vol. 58, No. 10, pp.1547–1550. van Raan, A.F.J. (2006) ‘Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups’, Scientometrics, Vol. 67, No. 3, pp.491–502.


31

Yaldiz, S., Unsacar, F. and Saglam, H. (2006) ‘Comparison of experimental results obtained by designed dynamometer to fuzzy model for predicting cutting forces in turning’, Materials & Design, Vol. 27, No. 10, pp.1139–1147. Zadeh, L.A. (1973) ‘Outline of a new approach to the analysis of complex systems and decision processes’, IEEE Transactions on Systems Man and Cybernetics, Vol. 3, pp.28–44.

32


Appendix The linguistic ‘if-then’ rules used in the fuzzy inference system If productivity is Very low

and

Impact is Very low and ALS is Short

If productivity is Very low

and

Impact is Very low and ALS is Moderate Then FAPI is Very low

Then FAPI is Very low


and

Impact is Very low and ALS is Long



and

Impact is Low

and ALS is Short



and

Impact is Low

and ALS is Moderate Then FAPI is Very low


and

Impact is Low

and ALS is Long



and

Impact is Moderate and ALS is Short


and

Impact is Moderate and ALS is Moderate Then FAPI is Low


and

Impact is Moderate and ALS is Long



and

Impact is High

and ALS is Short

Then FAPI is Low


and

Impact is High

and ALS is Moderate Then FAPI is Low


and

Impact is High

and ALS is Long

If productivity is Low

and



and

Impact is Very low and ALS is Moderate Then FAPI is Low


and




and

Impact is Low

Then FAPI is Low

and ALS is Short

Then FAPI is Low

Then FAPI is Very low Then FAPI is Low


and

Impact is Low



and

Impact is Low

and ALS is Long



and



and

Impact is Moderate and ALS is Moderate Then FAPI is Low


and




and

Impact is High

Then FAPI is Moderate

and ALS is Short

Then FAPI is Low


and

Impact is High



and

Impact is High

and ALS is Long



and

Impact is Very high and ALS is Long


If productivity is Moderate

and


Then FAPI is Low


and

Impact is Very low and ALS is Moderate Then FAPI is Low


and




and

Impact is Low

and ALS is Short



and

Impact is Low

and ALS is Moderate Then FAPI is Moderate and ALS is Long


and

Impact is Low


and


Then FAPI is Low


and

Impact is Moderate and ALS is Moderate Then FAPI is Moderate


and




and

Impact is High

and ALS is Short

Then FAPI is High


and

Impact is High

and ALS is Moderate Then FAPI is High


and

Impact is High

and ALS is Long




33

The linguistic ‘if-then’ rules used in the fuzzy inference system (continued) If productivity is Moderate and Impact is

Very high and ALS is Short

Then FAPI is High

If productivity is Moderate and Impact is

Very high and ALS is Moderate Then FAPI is High

If productivity is Moderate and Impact is

Very high and ALS is Long


If productivity is High

and Impact is

Very low

and ALS is Short



and Impact is

Very low

and ALS is Moderate Then FAPI is Moderate


and Impact is

Very low

and ALS is Long



and Impact is

Low

and ALS is Short



and Impact is

Low

and ALS is Moderate Then FAPI is Moderate


and Impact is

Low

and ALS is Long



and Impact is

Moderate and ALS is Short


and Impact is

Moderate and ALS is Moderate Then FAPI is High

Then FAPI is High


and Impact is

Moderate and ALS is Long



and Impact is

High

and ALS is Short

Then FAPI is Very high


and Impact is

High



and Impact is

High

and ALS is Long


and Impact is



and Impact is

Very high and ALS is Moderate Then FAPI is Very high


and Impact is

Then FAPI is High Then FAPI is Very high


Then FAPI is High

If productivity is Very high and Impact is

Very low

and ALS is Short

Then FAPI is High


Low

and ALS is Short

Then FAPI is High


Low



Low

and ALS is Long


Moderate and ALS is Short


Moderate and ALS is Moderate Then FAPI is Very high


Moderate and ALS is Long



High

and ALS is Short



High

and ALS is Moderate Then FAPI is Very high


High

and ALS is Long






Very high and ALS is Moderate Then FAPI is Very high



Note: ALS, academic life span; FAPI, fuzzy academic performance index.

Then FAPI is High Then FAPI is Very high
