the framework for business intelligence driven ...

V. Šakys, K. Kapočius, R. Butleris, A. Lopata, S. Gudas

ISSN 1648 - 4460

42

Structural Transformations in Business Development ---------TRANSFORMATIONS IN --------

Šakys, V., Kapočius, K., Butleris, R., Lopata, A., Gudas, S. (2013), “The Framework for Business Intelligence Driven Analysis of Study Course Teaching Efficiency”, Transformations in Business & Economics, Vol. 12, No 1A (28A), pp.42-59.

BUSINESS & ECONOMICS © Vilnius University, 2002-2013 © Brno University of Technology, 2002-2013 © University of Latvia, 2002-2013

THE FRAMEWORK FOR BUSINESS INTELLIGENCE DRIVEN ANALYSIS OF STUDY COURSE TEACHING EFFICIENCY 1

Vigintas Šakys

Department of Information Systems Faculty of Informatics Kaunas University of Technology Studentu str. 50 LT-51368 Kaunas Lithuania Tel.: +370 37 300384 Fax: +370 37 300352 E-mail: : [email protected]

2

Kęstutis Kapočius

3

Rimantas Butleris

Department of Information Systems Design Technologies Kaunas University of Technology Studentu str. 50 LT-51368 Kaunas Lithuania Tel.: +370 37 453445 E-mail: [email protected]

Department of Informatics Kaunas Faculty of Humanities Vilnius University Muitines str. 8 LT-44280 Kaunas Lithuania Tel.:+370 37 422566 E-mail: [email protected]

4

5

Audrius Lopata

Department of Informatics, Kaunas Faculty of Humanities Vilnius University Muitines str. 8 LT-44280 Kaunas Lithuania Tel.: +370 37 422566 E-mail: [email protected]

Saulius Gudas

Department of Informatics, Kaunas Faculty of Humanities Vilnius University Muitines str. 8 LT-44280 Kaunas Lithuania Tel.: +370 37 422566 E-mail: [email protected]

Vigintas Šakys, PhD, is an Associated Professor at Kaunas University of Technology (Lithuania). He is a lecturer of Business Information Technologies, Data Warehouses and Business Intelligence and Information Technology Fundamentals. His research interests encompass business intelligence tools and technologies, university study process monitoring and control and e-learning methodology and technologies. Dr. Šakys is also author or coauthor of over 20 textbooks. 1

2

Kestutis Kapočius, PhD, is a Researcher and Associated Professor at Kaunas University of Technology (Lithuania), where, in 2006, Kestutis defended his Ph.D. thesis (Technical Sciences, Informatics Engineering) titled “Business Rules Structuring Models and their Application during the Development of Information Systems”. In 2007-2008, he also worked on the same subject as a postdoctoral intern at Kaunas Faculty of Humanities of Vilnius University (Lithuania). TRANSFORMATIONS IN BUSINESS & ECONOMICS, Vol. 12, No 1A (28A), 2013


43

ISSN 1648 - 4460

Structural Transformations in Business Development 3

Rimantas Butleris, PhD, is a Full Professor and Director of Center of Information Systems Design Technologies at the Faculty of Informatics of Kaunas University of Technology (Lithuania), and Professor at Kaunas Faculty of Humanities, Vilnius University (Lithuania). Prof. Butleris is co-author and author of over 130 scientific publications. He was a general chairman of international conferences BIR 2006, I3E’2011, ICIST 2012; chairman of organizing committees of international conferences IT’2008, IT’2009, IT’2010, IT’2011; a member of program committees of over 30 international conferences. During the past 10 years, prof. Butleris coordinated or managed over 15 national or International research and development projects. 4

Audrius Lopata, PhD, is an Associated Professor at Kaunas Faculty of Humanities, Vilnius University (Lithuania) and Kaunas University of Technology (Lithuania). Dr. Lopata is the author or co-author of more than 35 research publications. The main fields of his scientific research include Knowledge Based Information Systems Engineering, Requirements Management Techniques, Knowledge Based CASE tools. Dr. Lopata has experience in international and Lithuanian researches and R&D projects. 5

Saulius Gudas, PhD, is a Full Professor at the Department of Informatics, Kaunas Faculty of Humanities, Vilnius University (VUKHF), Lithuania. Since 2008, he is the Dean of VU KHF. Education: in 1969–1974, studied at Kaunas University of Technology, Lithuania; in 1982 defended the PhD dissertation on the topic “Synthesis of Algorithmic Structure of Information Systems for Manufacturing Objects”; in 2005, passed the Doctor Habilitation procedure on the topic “Modelling of KnowledgeBased Information Systems Engineering Processes“. Research directions are as follows: knowledge-based enterprise modelling, Information Systems theory, knowledge-based information system engineering. Prof. Gudas is the author and co-author of more than 135 research publications. Received: November, 2012 1st Revision: December, 2012 2nd Revision: January, 2013 Accepted: February, 2013

ABSTRACT.

As Lithuanian higher education institutions are forced to compete between themselves and with foreign universities, the ability to constantly monitor and improve study quality becomes paramount. Among the internal resources available for this task comprehensive data stored by local Academic Information Systems have been used. However, standard data analysis capabilities often fail to transform this data into conclusive, knowledge-containing insights. The aim of the presented research was to address this issue by developing a framework for monitoring teaching efficiency of study course with the help of modern Business Intelligence solutions.

TRANSFORMATIONS IN BUSINESS & ECONOMICS, Vol. 12, No 1A (28A), 2013


44

ISSN 1648 - 4460

Structural Transformations in Business Development The framework has been implemented at the Faculty of Informatics of Kaunas University of Technology, and the developed prototype proved to be a promising decision-making tool. It is possible to presume our findings could be beneficial to other higher education institutions willing to invest in solutions of monitoring the teaching quality. At the center of the proposal, there is a developed set of seven key indicators of the performance that can be combined to get an objective and easily understandable evaluation of efficiency of course teaching.

KEYWORDS:

higher education in Lithuania, teaching efficiency, business intelligence, key performance indicator.

JEL classification: M15, C88, I23.

Introduction Recently, higher education in Lithuania was undergoing serious reforms aimed at improving the quality of studies. Newly implemented funding system based on the concept of “student’s baskets” provoked fierce competition between universities. For universities more students mean more “baskets” that in effect refer to more direct state budgetary allocations. However, accessibility of studies in the rest of European Union and beyond increases competition even more. These factors force universities to start looking for internal resources that could help to improve the quality and attractiveness of studies. Among the available tools are the state-of-the-art IT solutions. Here, business intelligence (BI) solutions seem especially promising (Tomić et al., 2013; Yingxin, 2010; Montoneri et al., 2012) while at the same time remaining either underused, or not used at all. On the other hand, the aforementioned solutions are widely applied in business, and rapidly accumulating evidence has already proven beyond reasonable doubt their utility as decision-making facilitators (BI Survey 12, 2012; Yigitbasioglu et al., 2012). Therefore, it is not unreasonable to presume that similar benefits could also be achieved in higher education institutions (Sakys et al., 2011). At the time of writing this paper, the standard practice at the Lithuanian higher education institutions was to store all of the vital data in the database of an Academic Information System of the institution. Information necessary for study management is usually retrieved directly from a database using ad hoc queries, and displayed in specialized graphic reports (e.g. in Excel spreadsheets). However, the amounts of data are quite large while the reports are usually limited, not to mention very slow, and thus, ineffective. Application of BI technologies could help solving these problems. Namely, our hypothesis is that professionally implemented scorecards and dashboards based on the correct set of metrics could be especially useful in the process of improvement of quality of studies. In this paper, an academic data analysis framework including a developed set of key performance indicators (KPIs) for measuring the efficiency of chosen study courses (or study modules) is proposed. Using this framework, universities could implement scorecards and dashboards, and start using them to monitor, control and manage the efficiency of the study course teaching. The paper consists of an introduction, four sections, conclusions, and a list of references. After the overview of the key concepts (Sections 1 and 2) KPIs calculation process and formulas are introduced (Section 3). Section 4 presents an overview of the case study that TRANSFORMATIONS IN BUSINESS & ECONOMICS, Vol. 12, No 1A (28A), 2013


45

ISSN 1648 - 4460

Structural Transformations in Business Development was carried out at the Faculty of Informatics of Kaunas University of Technology (Lithuania). Finally, the conclusions are drawn. 1. Available Business Intelligence Solutions and the Analysis Framework Business analysis tools are currently being offered by most major software companies, including Microsoft, Oracle and IBM. These tools allow companies and organizations to create specialized data warehouses and data marts, and use them for the business analysis. Among the applied technologies there are OLAP cubes, web reports, dashboards, advanced data mining and others (Few, 2006; Velcu-Laitinen et al., 2012; Tsai, 2012). All of the aforementioned tools could be referred to as Business Intelligence (BI) solutions and applications. The main reason of using these solutions is the gained ability to provide timely and precise reports on data (that could not be analysed equally effectively using standard tools) to analysts and decision makers. However, although generated reports may include detailed tables and charts, they often lack conclusive information that would directly assist decision-making. In other words, the way a person interprets such conclusionsfree reports depends on a number of conditions, including volumes of data, presentation, user’s concentration, previous knowledge, experience, biasness and professionalism (Tomić et al., 2013). This leaves room for incorrect interpretation of data and, thus, not very useful insights (knowledge) can be extracted, leading to potentially inefficient business decisions. Therefore, during the research presented in this paper, it was aimed to develop a BI framework that would provide understandable, conclusive and objective evaluations of teaching quality of study course to high school teachers and managers. The developed set of key performance indicators that can be displayed on specialized scorecards and dashboards is in centre of the proposal. Several definitions have been offered for dashboards, but most of them share the same key ideas. According to a popular definition by Stephen Few, a dashboard is a visual display of the most important information needed to achieve one or more objectives; consolidated and arranged on a single screen so the information can be monitored at a glance (Few, 2004). Ideally dashboards should have key features (Few, 2004; Chiang, 2011; Pauwels et al., 2009):  only the most relevant performance indicators and measurements are displayed;  available interactivity tools, such as drill-down, filtering and alike;  high usability: as easy to understand to the decision makers as possible;  what-if analysis;  automatic warning, when values of a certain indicator are reached;  automatic updating of data. A scorecard, on the other hand, is a tabular visualization of measures and their respective targets with visual indicators to see how each measure is performing against their targets at a glance (Chiang, 2011). Usually a scorecard contains a key performance indicator, its value, its target, and a visual indication of the status (e.g. a special icon that is green for good, yellow for warning, and red for bad) on each row. Due to a larger amount of information on display, dashboards may combine several scorecards with graphs and indicators of visual performance designed for fast analysis and decision-making. Individual scorecards, on the other hand, can also be very useful, especially when there is a need to see all of the intermediate, precise measures and evaluations. In any case, it is essential to keep in mind that BI tools offer only the analysis-facilitating TRANSFORMATIONS IN BUSINESS & ECONOMICS, Vol. 12, No 1A (28A), 2013


ISSN 1648 - 4460

46

Structural Transformations in Business Development environment, while its usefulness depends solely on the characteristics of each scorecard or dashboard and on how they are used in an organization. The main input to a scorecard or dashboard is the data that is collected from various information systems used by the organization. Therefore, it is compulsory that chosen analytical tools have access to and can access data stored in various sources at the same time, including legacy databases and BI stores, such as data marts. At the same time it is important to remember that although based on information from various sources, properly designed dashboards or scorecards only display data that is needed to fulfill the specific goal and hide irrelevant details of how this data has been obtained.

Data warehouse

Integration services Database of an acadamic information system

Data marts

OLAP cubes, quering, data mining, reporting and dashboard services

Client access

Source: created by the authors. Figure 1. The Conceptual View of the Analysis Process

Considering academic applications, our initial state-of-the-art study revealed that described BI tools and technologies, although promising, are quite underused (Sakys et al., 2011). Indeed, various solutions determined by business analysis are being successfully implemented in the analysis of work efficiency by a growing, although still quite low number of universities (Chalmers, 2008; Katharaki et al., 2010; Cao et al., 2010). However, the solution described in this paper is, as far as we know, completely original. It is primarily aimed at institutions that follow the Lithuanian studies organization model and already have a functioning academic information system that can provide the data for a separate data warehouse and functional data marts. Based on the analysis of these data marts, clients could be given a number of generated tools, including dashboards or individual scorecards that are tailor-made for the purposes of monitoring and analysis of the teaching efficiency of a study course (Figure 1). Such tools could be especially useful to lecturers, heads of the departments, deans, rectorate and any other administration staff members involved in a decision-making process. Furthermore, students could also benefit from these tools. 2. The Concept of Key Performance Indicators Key Performance Indicator (KPI) could be defined as a significant measure used on its own, or in combination with other key performance indicators, to monitor how well a business or other activity is achieving its quantifiable objectives (Georgetown University, 2008). Other researchers define KPIs as significant metrics used to compare actual results with the business goals (Beisheim et al., 2013; Seify, 2010; Ya et al., 2009). In any case, key performance indicator has a goal value, evaluative intervals, or both, and is used to measure the improvement or deterioration of business results. KPIs are usually defined for a specific activity and are strictly goal oriented. It is important to make sure that all key performance indicators can be measured and quantified. If defined and calculated correctly, KPIs help organizations to understand how well or poorly TRANSFORMATIONS IN BUSINESS & ECONOMICS, Vol. 12, No 1A (28A), 2013


47

ISSN 1648 - 4460

Structural Transformations in Business Development they perform and how they actually are able to fulfill the set goals. Therefore, the right set of KPIs should not only shed light on the results of an organization, but also help to identify areas where additional attention is needed. At the end of the day, “what gets measured gets done” and ”without the right KPIs managers are sailing blind” (Marr, 2012). One of the goals of good managers and decision makers is to understand the main efficiency measurements, and therefore it is these people who should point out the key performance indicators. However, it is obligatory for the nature of these indicators to be clear to everybody who is going to base their decisions on these KPIs. In the case of universities, behind those indicators there are real students and lecturers, and understanding how exactly these people and their activities are interrelated with the values displayed on the scorecards or dashboards is quite important for correct interpretation of KPIs. In the next two sections of this paper, a detailed description of KPIs proposed by us for the measurement of the teaching efficiency of study courses is given, followed by the consideration of the results of the case study, which also served as a proof of concept. 3. The Set of KPIs for the Quantification of the Teaching Efficiency of Study Courses During this research, a set of seven key performance indicators has been developed. Based on these indicators, a single value representing the overall efficiency of study course teaching is calculated. All calculated values should be displayed to users on scorecards or dashboards that ideally should be made available as an integral part of an academic information system of a particular institution of higher education. It must be noted that both, scorecards and dashboards can be designed to fit the needs of a specific user group (students, lecturers, heads of departments, deans of faculties, etc.). It is important to stress that the presented set of KPIs was developed within the studies organization model that is used in many Lithuanian higher education institutions, including Kaunas University of Technology (KTU). The ten-point gathered knowledge and skills evaluation system is of the main significance. Intermediate semestral works are evaluated by grades; the final grade is given during the examination session while multiplying intermediate grades by the lever coefficient and summing the products. Alongside the ten-point system, the pass/fail system is also used. While examinations and work defence evaluations are graded, some courses do not end with an examination, and for them the pass/fail system is used to indicate if a student has earned the allocated credits. The assimilation of at least 50% of required knowledge scope is required to pass. Also, the pass/fail system is used for the modules that actually end with an examination. In this case, pass or fail evaluation is appointed on the basis of all semestral results (i.e. control works, personal or group assignments, colloquiums, individual projects, etc.). As it was mentioned before, for calculation of the study course Coursem (where ) additive value of the teaching efficiency Wm that summarizes seven partial efficiency indicators is proposed (Table 1). KPIs for the semester are calculated either during the 1st week of the exam session that follows this semester, or on the 3rd week of the following semester. As it can be seen from the Table 1, for the analysis to be carried out, we must have access to the data about the set of filtered study courses as well as students (set Students) study results in these courses. The developed KPIs calculation algorithm consists of five steps, as described below. Calculation of all Wim for a study course Coursem is a four-step procedure. During the Step 5, TRANSFORMATIONS IN BUSINESS & ECONOMICS, Vol. 12, No 1A (28A), 2013


ISSN 1648 - 4460

48

Structural Transformations in Business Development all calculated KPIs are combined into a single value Wm, which, in effect, is the key performance indicator representing the efficiency of teaching of the study course Coursem. Table 1. The proposed set of key performance indicators KPI name 1. Attendance (in %)

Notation

Calculation time st

1 week of session

2. Percentage of positive intermediate assessments 3. Average of grades of intermediate assessments 4. Percentage of “passed” evaluations 5. Percentage of positive final grades 6. Average of final grades 7. Percentage of non-drop outs Source: own calculations.

1st week of session 1st week of session 1st week of session 3rd week of next semester 3rd week of next semester 3rd week of next semester

Calculation domain Only those who didn’t drop out Only those who didn’t drop out Only those who didn’t drop out Only those who didn’t drop out Only those who didn’t drop out Only those who passed (grades  5) All those listed before the semester

Before starting the calculations, the values of target, weighting factor wi and threshold must be set by experts, preferably under the supervision of the Dean Office, for a particular faculty or study year that is going to be analysed. Note that the sum of all weighting factors for a set of KPIs must be 1. For a suggested set of these values the case study description in Section 4 of this paper must be considered. 1. Actual values ACTUALim are calculated for each study course Coursem using the semester data following formulas (1)-(7). a) Actual attendance of the Coursem is calculated as: (1) where: is the amount of in-class lectures (in academic hours) of Coursem attended by student Studentj; is the total amount of academic hours of in-class lectures of Coursem; n is the total amount of students in Coursem. b) Actual percentage of positive intermediate semestral assessments of the Coursem is calculated as: (2) where: is the amount of positive intermediate semestral assessments obtained by Studentj from a Coursem, calculated as: ,



ISSN 1648 - 4460

49

Structural Transformations in Business Development where

is the state of intermediate semestral assessment Assesments of Studentj from

Coursem, and it is considered positive, when the grade

is greater than 0, i.e.,

; is planned amount of intermediate semestral assessments for Coursem for one student; n is the total amount of students in Coursem. c) Actual average of grades of intermediate assessments of Coursem is calculated as: (3) where:

is the sum of grades

obtained by Studentj from Coursem:

; is planned amount of intermediate semestral assessments for Coursem for one student; n is the total amount of students in Coursem. d) Actual percentage of “passed” evaluations of Coursem is calculated as: (4) where:

is the state of Studentj, where

;

n is the total amount of students in Coursem. e)

Actual percentage of positive final grades of Coursem is calculated as: (5)

where:

is the state of Studentj with regard to his/her final grade ;

n is the total amount of students in Coursem. f) Average of actual final grade of Coursem is calculated as: (6) where: is actual final grade of Studentj from the Coursem; n is the total amount of students in Coursem. g) Actual percentage of non-drop outs of Coursem is calculated as: (7)


, where


ISSN 1648 - 4460

50

Structural Transformations in Business Development where

– state of Studentj within the Coursem, where ;

n is the total amount of students in Coursem. 2. Actual values must be normalized using KPI standard normalization formula: Wim = ((ACTUALim – MINi) / TARGETi )  100,

(8)

3. The scale for every Wim is scaled down by WiNull units. This is done to make sure that the evaluation of lecturer’s efficiency would equal 0 if evaluation of his study course ACTUALi is equal to the value TRESHOLDi. Therefore: WimScale = Wim – WiNull ,

(9)

where WiNull= ((THRESHOLDi – MINi) / TARGETi )  100 4. After the KPI values get scaled down, they must be adjusted to assess the total amount of students n in each course considering the average amount of students in all analyzed course. It is achieved by lowering values for courses that are attended by a small amount of students and increasing them for courses that are attended by a large amount of students: WimAvr = WimScale  n / nAvr ,

(10)

where normalization coefficient nAvr is an average amount of students in all analysed study courses, which is calculated using the actual data of the analysed study year(-s) or semester(-s). 5. Finally, to get the single value representing the efficiency of teaching of a particular study course Coursem, all of its adjusted KPIs WimAvr must be multiplied by the corresponding weighting factors wi (here i = 1..7) and added together: Wm =

(11)

4. Case Study and Results The presented framework was implemented at the Faculty of Informatics of Kaunas University of Technology. 2011-2012 study year academic data from the database of the Academic Information System (AIS) of KTU has been used for research. In addition, data on student attendance have been recorded as these were not stored by the AIS. Software tools used for this prototype implementation were Microsoft SQL Server, Excel 2010 with PowerPivot and SharePoint Server 2010. As it was mentioned in Section 3, before calculating KPIs of the teaching quality of a study course, target values, weighting factors and thresholds for each KPI must be set by the experts and/or studies managing personnel. Values, determined for and used in the described case study are given in Table 2. Obviously, these values can be used in other implementations of the proposed solution. However, we strongly suggest decision makers of the specific institution or department reviewing them and making necessary adjustments. It is important to remember that the sum of weighting factors must equal 1.



ISSN 1648 - 4460

51

Structural Transformations in Business Development Table 2. Parameter values used during the case study KPI

MAX 100% 100% 10 100% 100% 10 100%

MIN 25% 50% 5.82 60% 55% 5.78 65%

Target 75% 85% 7 90% 85% 8 95%

Weighting factor wi 0.05 0.05 0.05 0.15 0.4 0.1 0.2

Threshold 30% 60% 5 60% 65% 6 70%

Source: own calculations.

In addition to the aforementioned three values, the maximum and minimum values for each KPI MAXi and MINi (where i = 1…7) must be defined. While maximum values are standard (Table 2), minimum values must be determined on the basis of the actual data that are going to be analysed. In other words, minimum value can vary between departments, faculties and study years. Therefore, it should be calculated automatically. Table 3. Key performance indicators calculation results for three selected study courses KPI

Course1 WimScale WimAvr -42.15 -71.66

Actlim 39.39

Wim 17.85

95.29

32.11

2.70

64.12

53.03

24.45

96.47

16.08

16.08

Course2 WimScale WimAvr -32.80 -14.43

Actlim 41.57

Wim 15.43

Course3 WimScale -44.57

WimAvr -25.85

-5.45

56.78

-0.26

-29.67

-17.21

17.74

42.78

-0.32

-28.89

-16.76

-0.22

-0.13

Actlim 46.40

Wim 27.20

4.59

82.47

17.02

-12.39

41.57

75.23

68.90

40.32

-0.20

-0.20

-0.09

27.33

81.82

90.80

-0.22

97.65

12.94

12.94

22.00

86.36

0.40

0.40

0.18

93.10

2.34

2.34

1.36

67.28

4.10

-8.40

-13.60

90.56

33.19

20.69

7.45

64.94

-0.08

-12.58

-6.63

97.65

12.26

12.26 14.43

20.84

95.45

9.95

9.95 1.57

4.38

97.70

2.84

2.84 -2.8

1.65

Wm

Source: own calculations.

An example of KPI calculations following the algorithm given in Section 3 is given in Table 3. Some of the calculations of input data obtained from the Academic Information System of KTU are given in Table 4. Note that given values are indeed actual values for three selected real-life courses: one good, one average and one poor. Course codes have been replaced intentionally.



ISSN 1648 - 4460

52

Structural Transformations in Business Development Table 4. KPIs input data calculation Characteristic Study year Average amount of students in this study year Amount of students before the semester Semestral work Amount of students that failed at the end of semester under pass/fail system Amount of intermediate assessments Amount of failures to pass an intermediate assessment Average grade of intermediate assessments Attendance Total amount of academic hours of in-class lectures Amount of student’s absence hours Exams session Amount of students that dropped out Amount of students who got a negative final grade (