Assessing the Effectiveness of Learning and Teaching Technologies ...

2014 IEEE Conference on e-Learning, e-Management and e-Services (IC3e)

Assessing the Effectiveness of Learning and Teaching Technologies for Teaching Distance Mode Students in Higher Education Santoso Wibowo, Srimannarayana Grandhi, Ritesh Chugh School of Engineering & Technology CQUniversity Melbourne, Australia Email: [email protected] that group will be able to interact with other group members online and share documents during the discussion process [4].

Abstract— This research paper develops a new method for assessing the effectiveness of learning and teaching technologies and selecting the most suitable learning and teaching technology for development in higher education. The relevant evaluation criteria are provided as a valuable reference for higher education institutions in establishing a standardized means of selecting the most suitable learning and teaching technology for development and implementation. Fuzzy numbers based approximated Linguistic variables are utilized to indicate decision maker’s personal assessments. This will allow efficient handling of subjectiveness and imprecision inherent in the evaluation process in a cognitive and least demanding way. An effective algorithm is developed based on the concepts of fuzzy minimum and fuzzy maximum to determine the performance of each alternative fin relation to every criterion. Selected example exhibits the applicability of chosen approach for dealing with real-world learning and teaching technology evaluation problem.

It is common that students studying for a general course may be required to complete several assessment tasks, which very often require advanced level of understanding and practice in order to develop applications. As practical assessments demand innovation, students often face problems with getting much needed support to fix program related errors. Thus, it would be ideal if instructors are able to support their students by two way interactions in which students can demonstrate their problems through these LMS systems. Research done by Carvalho et al. [3] at a Portuguese university suggests that students prefer to use LMS system due to its convenience and flexibility offered by the system. Currently, there are several commercially available LMS which include Blackboard, Moodle, WebCT and Angel course management systems. All these systems generally offer core functionalities such as distribution of course content, management of assignment submission, and generation of basic reports about students’ performance and main grade reports [4].

Keywords-Learning and teaching technologies; assessment; effectiveness; higher education; distance mode students

I.

INTRODUCTION

LMS are centered on specific educational institution or a course to support the learning process. In general, LMS offers several benefits including storage of learning material, ubiquitous learning, communication with students through chat forums, reuse of materials and customized learning paths. Interestingly, there is a recent trend in using social networking technologies such as Twitter to increase distance mode students’ participation [5]. Undoubtedly, these LMS systems support not only support on-campus students but also distance students. Carvalho et al. [3] point out that students prefer to use a single integrated LMS rather than multiple systems. Therefore, it is important to choose and adopt the right LMS for increasing student participation rates.

Distance mode students in nature are located away from the institution [1]. However, recent developments in information and communication technologies (ICT) offered new opportunities for students to study from various geographical locations without physically travelling to their university to attend classes [2]. In fact, the use of ICT in education industry is strongly supported by the local governments. Many universities and education providers around the world have adopted learning management systems (LMS) such as Blackboard and Moodle to distribute course content including video lectures, power point presentations and other teaching materials [3]. These systems provide flexibility and convenience to students by allowing them to submit their assessments online. These technologies also facilitate the distribution of course contents and the inclusion of learning and teaching materials needed for group work. In order to facilitate collaborative environment online, instructors are required to create groups and assign students to these groups. Students in

978-1-4799-7177-0/14/$31.00 ©2014 IEEE

Over the last few decades, the developments in these LMS systems have created opportunities for higher education providers by emulating traditional methods of learning and learning and extending the use of LMS systems to support all aspects of higher education [1]. As a result, it is critical for

24


higher education institutions to evaluate and select the most suitable learning and teaching technology based on its effectiveness for development and implementation.

different functionalities [4]. Although there are several LMS tools available, the main challenge is to identify the right one for delivering effective learning and teaching. In the case of LMS, there are 3 types of users, namely, (a) instructors, (b) students, and (c) administrators. Instructors are people who develop the content and distribute and facilitate collaborative sessions which include group discussions and online meetings. Students are people who download the content, submit their work and participate in online forums created by the instructors. Administrators are people who generate reports for reviewing the performance and developing standards to enhance the use of LMS. Interestingly, all these users have varying needs, and therefore it is important that their needs are identified and considered in the learning and teaching technology selection process [11].

The procedure of choosing the most appropriate learning and teaching technology for teaching distance mode students in higher education is not simple and complex. The difficulty of this selection process is mainly because of the multidimensional nature of the decision making process, the conflicting nature of the multiple selection criteria [6], and also the existence of subjectiveness and imprecision in the decision making process. To ensure that the learning and teaching technology selection process is carried out consistently and effectively, a complete and proper evaluation of the learning and teaching technologies’ overall effectiveness is required. This paper offers a new method for assessing the effectiveness of learning and teaching technologies for teaching distance mode students in higher education. Fuzzy numbers based approximated Linguistic variables are utilised to represent decision maker’s personal assessments. This allows proper handling of the subjectiveness and imprecision inherent in the selection process in a cognitively less demanding way. An effective algorithm is developed based on the concepts of fuzzy minimum and fuzzy maximum for finding out the relative performance of each available alternative. Presented example clearly indicates the practicality of the chosen method for effectively dealing with the learning and teaching technology evaluation and selection problem. II.

Al Hamad and Salameh [12], for example, point out that students have different learning styles namely visual, auditory and tactile. Undoubtedly, learning is a cognitive activity and every student has different learning needs. In fact, these learning styles differentiate students because of the preferences in acquiring relevant information [13]. These learning styles in turn form the basis for the development of learning strategies. Understanding more about the learning community in the higher education sector can help choose suitable system [10, 13]. Deciding what needs to be presented on LMS is another critical factor to be considered. Content management, delivery, tracking and assessment are the core functionalities of LMS [2, 4]. Understanding different learning styles may offer clues on learning content that can be made available through LMS [14]. Presenting relevant content and in an easily retrievable manner helps to increase students participation rates [15].

LITERATURE REVIEW

Australia is the sixth biggest nation in the world in terms of land size and it has a population of about 22 million [7]. Majority of Australian population live in the capital cities; however 31.6% live in either regional or remote areas [8]. Population living in these remote areas has limited access to education. Many in these areas are leaning towards completing their higher education through distance mode enrolment in one of the universities. According to Norton [8], there were about 1.1 million students enrolled in Australian universities in 2010 and 12% of these students are off-campus students and 7% were enrolled in multi-mode, which allows them to study both on-campus and distance-study modes. As distance-mode enrolments are growing, many universities in Australia are now choosing to offer both on-campus and distance education.

Identifying what content needs to be presented on LMS can influence users’ perception to use. Content, immersion, interactivity, and communication are the four main categories which influence user acceptance [10, 16]. Sluijs and Hover [17] state that students prefer a well-designed interface. For example, cross browser compatible web interface help different users with varying roles complete their tasks from anywhere. In general, LMS currently used by the universities have adopted “one size fits for all” approach whereby the same content is presented to all the users. While it meets the objective of delivering the content and achieves uniformity, it does not consider different learning styles of students [17].

In the past, students enrolled through distance mode were sent printed materials by post. This approach allowed students to read materials, but it lacked interaction and an opportunity to clarify problems with the instructor. Fortunately, recent developments in ICT have offered new opportunities to these students to study through distance mode. Australian universities are ahead in adopting LMS such as Moodle and Blackboard to reach distance students [9].

Personalization is another important factor to be considered in the e-learning environment. Al Hamad and Salameh [12] explain that students would be willing to use LMS if the system adds value to their achievements [12]. Another research done by Yang et al. [14] reveals that there is a linkage between learning styles and personalized presentation module on LMS. Interestingly, most of the LMS currently adopted by universities lacks personalization. While personalization may allow students to identify relevant materials, it may also act as a barrier because students may view some of the materials as irrelevant and as a result, they ignore to read the materials [17]. As every student has different learning styles, developing a personalized interface can help achieve positive learning outcomes. However, allowing too much flexibility may have a

LMS can be defined as a tool that helps administer, track and report training [3]. Some of the popular LMS providers are ACS, Blackboard, Oracle iLearning, Pathlore, Plateau, Certpoint, Cornerstone OnDemand, GeoLearning, Learn.com, Saba Software/Thing, SAP, Sum Total Systems, Meridian KS, Mzinga, Teds and WBT systems [10]. In fact, there are more than 100 commercially available LMS tools in the market with

25


negative impact on system response times. Therefore, it is important to achieve a balance between learning styles and personalization. III.

THE APPROACH

Assessing the effectiveness of learning and teaching technologies for teaching distance mode students in higher education is always complex due to (a) the large number of learning and teaching technologies available, (b) the multidimensional nature of the problem and (c) the presence of subjectiveness and imprecision involved in the decision making process.

The learning and teaching technology evaluation begins with the determination of the relative performance of each learning and teaching technology alternative Ai (i = 1, 2, …, n) with respect to each criterion Cj (j = 1, 2, …, m). As a result, a decision matrix for all the learning and teaching technology alternatives can be attained as follows

Assessing the effectiveness of learning and teaching technologies for teaching distance mode students in higher education usually involves in (a) determining all the available alternatives, (b) recognizing the evaluation criteria, (c) evaluating the relative performance of the learning and teaching technologies and the criteria weights of the criteria, (d) aggregating the alternative performance ratings and criteria weights for producing an overall performance index of learning and teaching technologies.

⎡ x11 x12 ⎢x x 22 X = ⎢ 21 ⎢ ... ... ⎢⎣ x n1 x n 2

... ... ... ...

x1 m ⎤ x2 m ⎥ ⎥ ... ⎥ x nm ⎥⎦

(1)

The relative importance of the evaluation criteria Cj can be assessed qualitatively using fuzzy numbers, given as

Subjectiveness and imprecision is normally existent in multicriteria decision making problem. This is usually because of (a) incomplete information, (b) abundant information, (c) conflicting evidence and (d) ambiguous information [19, 20]. To adequately deal with this issue, linguistic variables approximated by triangular fuzzy numbers are often used to express the decision maker's subjective assessments. These triangular fuzzy numbers are usually used to represent the approximate distribution of these linguistic variables with values ranged between 1 and 9, denoted as (a1, a2, a3) where 1 < a1 < a2 < a3 < 9. a2 is used to represent the most possible value of the term, and a1 and a3 are representing the lower and upper bounds respectively used to reflect the fuzziness of the term [21]. Table I shows the linguistic variables and their corresponding triangular fuzzy numbers for the decision maker to make qualitative assessments about the performance rating of each learning and teaching technology alternative with respect to a given criterion.

W = (w1, w2, …, wm)

(2)

Based on (1) and (2), the weighted fuzzy performance matrix that represents the overall performance of each alternative can be determined by multiplying the fuzzy criteria weights (wj) by the alternatives’ fuzzy performance ratings (xij) as

⎡ w1 y11 w2 y12 ⎢w y w2 y 22 Z = ⎢ 1 21 ... ... ⎢ ⎢⎣ w1 y n1 w2 y n 2

... wm y1m ⎤ ... wm y 2 m ⎥

⎥, ⎥ ... wm y nm ⎥⎦ ...

...

(3)

To eliminate the difficult and challenging process of comparing fuzzy utilities, the concepts based on fuzzy maximum and fuzzy minimum [19] are introduced for determining the relative performance of all available alternatives in relation to each criterion. Given the fuzzy vector (wјx1ј, wјx2ј, …, wјxmј) of the weighted fuzzy performance j matrix for criterion Cј, a fuzzy maximum ( M max ) and a fuzzy j ) can be determined as in (4)-(5). The fuzzy minimum ( M min maximum and fuzzy minimum represent the best and the worst fuzzy performance ratings among all the alternatives with respect to criterion respectively.

To minimize the cognitive demand on the decision maker, linguistic variables approximated by fuzzy numbers defined as in Table II is basis for assessing the importance of each criteria in relation to the overall goal of the learning and teaching technology evaluation and selection problem.

μ

26

j M max

j ⎧ x − x min j j , x min ≤ x ≤ x max , ⎪ j j ( x ) = ⎨ x max − x min ⎪0, otherwise ⎩

(4)


μ

j M min

j ⎧ x max −x j j ≤ x ≤ x max , x min , ⎪ j j ( x ) = ⎨ x max − x min ⎪ 0, otherwise ⎩

IV.

To demonstrate the applicability of the new approach developed in the previous section, a problem of assessing the effectiveness of learning and teaching technologies and selecting the most suitable learning and teaching technology for development in higher education is presented.

(5)

where j

n

Currently, there are 43 universities in Australia offering a wide range of higher education courses to both domestic and international students where they study either on-campus or distance-study modes. Over the last couple of years, there has been a significant change taking place in the higher education sector that has received little analysis. It is found that learning and teaching technologies have rapidly emerged as part of universities’ strategy, and are having profound effects on university learning and teaching practices [10].

n

j

x max = sup (supp ∪ ( w j xij )) and x min = inf (supp ∪ ( w j xij )). i =1 i =1

By using the concept of fuzzy similarity, the similarity between each alternative and the fuzzy maximum can be determined as + si

=

m

j ∑ d ij ( xij w j , M max ) j =1

These learning and teaching technologies’ development and implementation in Australian higher education sector is due to universities’ firm commitment in improving the learning and teaching environment [9]. In addition, ICT tools are seen by academics staff and the Australian federal government as holding great potential in higher education. Therefore, there is a strong need for universities to develop and implement a suitable learning and teaching technology that will be a key enabler in fulfilling the vision and strategies at national and institutional levels [6].

(6)

Correspondingly, the similarity between each alternative and the fuzzy minimum can be calculated as −

m

j

si = ∑ d ij ( xij w j , M min ) j =1

(7)

Due to this intense pressure from both the governments and the competing universities, a local university decides to take this opportunity to develop a new learning and teaching technology for achieving its competitiveness in the higher education sector.

The most favored alternative should not only have the shortest distance from the fuzzy maximum, but also have the longest distance from the fuzzy minimum. As a result, an overall performance index for each alternative Ai across all the criteria can be calculated by using (8). −

Pi =

si +

−

si + si

AN EXAMPLE

The learning and teaching technology evaluation and selection starts with the establishment of a special committee consisting of several senior management advisors. A consensus approach is used to determine relevant evaluation and selection criteria for the learning and teaching technology evaluation and selection process. Based on their thorough discussion, four selection criteria are identified for evaluating six learning and teaching technology alternatives. These selection criteria include Learner Interface (C1), Learning Community (C2), System Content (C3), and Personalization (C4) [22]. The hierarchical structure of the learning and teaching technology evaluation and selection problem is presented in Figure 1.

(8)

Above presented new approach is summarized as Step 1. Obtain the fuzzy decision matrix for the decision maker as in (1). Step 2. Determine the weighting vector of the decision maker as stated in (2). Step 3. Determine the fuzzy maximum and the fuzzy minimum which represent respectively the best and the worst fuzzy performance ratings among all the alternatives with respect to criterion Cј as in (4) and (5). Step 4. Calculate the degree of similarity between each alternative and the fuzzy maximum and between the alternative and the fuzzy minimum by (6) and (7) respectively. Step 5. Calculate the overall performance index for each alternative by (8). Step 6. Rank the alternatives in descending order of their performance indexes.

Learner Interface (C1) concerns with the ability to replicate effective instructional strategies to interactively deliver content, engage the learner, facilitate learning, and assess learning [2, 7]. Factors such as usability, user-friendliness, adaptability and operational stability of the system are usually taken into consideration. Learning Community (C2) refers to a group of people with common understanding, beliefs and values, and is actively participating by sharing knowledge and learning from each other. This is measured by the ease of discussion with other learners, the ease of discussion among academic colleagues, the ease of accessing shared data and the ease of exchanging learning with the others [5, 22].

27


Personalization (C4) involves the tailoring of pages to meet specific individual users’ characteristics or preferences [9]. This is usually assessed by the capability of monitoring and controlling learning progress and the capability of recording the outcome of the learning performance [22].

System Content (C3) concerns with the type of information/knowledge provided by the system that provides value for an end-user/audience in specific contexts [3]. This is often measured by up-to-date, relevant and useful content.

Learning and teaching technology evaluation and selection problem

Level 1

Level 2 Criteria

C2

C1

Level 3 Alternatives

A2

A1

Legend: C1: Learner Interface C3: System Content

C3

A3

C4

A4

A5

A6

C2: Learning Community C4: Personalization

Ai (i = 1, 2, …, n): Learning and teaching technology alternatives. Figure 1. The hierarchical structure of the learning and teaching technology evaluation and selection problem

Based on Table I, the performance ratings of all available learning and teaching technologies can be determined by the decision maker as shown in Table III. TABLE I.

Based on (3), the weighted fuzzy performance matrix can then be determined by multiplying the fuzzy decision matrix in Table III by the fuzzy weighting vector in Table IV. Table V shows the weighted fuzzy performance matrix that represents the overall performance of each learning and teaching technology alternative on each criterion.

PERFORMANCE ASSESSMENTS OF LEARNING AND TEACHING TECHNOLOGIES Criteria

Alternatives

TABLE III.

C1

C2

C3

C4

A1

G

P

G

F

A2

F

F

G

G

A3

F

G

VG

P

A4

VG

G

F

A5

G

G

A6

G

VG

Criteria C1

C2

C3

C4

A1

(31, 58, 81)

(23, 47.2, 63)

(31, 58, 81)

(31, 58, 81)

F

A2

(21, 44, 63)

(25, 49, 69)

(28, 53, 74)

(21, 44, 63)

G

F

A3

(23, 47, 6)

(38, 68, 81)

(31, 58, 74)

(23, 47, 63)

G

VG

A4

(10, 27, 52)

(5, 24, 48)

(10, 27, 52)

(10, 27, 52)

A5

(31, 58, 81)

(23, 47, 63)

(31, 58, 81)

(31, 58, 81)

A6

(21, 44, 63)

(25, 49, 69)

(28, 53, 74)

(21, 44, 63)

By using Table II, the criteria weightings can be determined by the decision maker. Table IV shows the results. TABLE II.

THE WEIGHTED FUZZY PERFORMANCE MATRIX OF LEARNING AND TEACHING TECHNOLOGIES

By using (4) to (8), an overall performance index for each learning and teaching technology alternative across all criteria can be calculated in an efficient and effective manner. Table VI indicates that learning and teaching technology alternative A3 has the best performance for development and implementation, relative to other alternatives as it has the highest performance index value of 0.78.

THE CRITERIA WEIGHTINGS FOR THE LEARNING AND TEACHING TECHNOLOGIES

Criteria

Criteria weight

C1

H

C2

M

C3

VH

C4

H

28


TABLE IV.

THE PERFORMANCE INDEX OF LEARNING AND TEACHING TECHNOLOGIES AND THEIR RANKINGS

Alternative

Performance index

Ranking

A1

0.64

4

A2

0.69

3

A3

0.78

1

A4

0.54

6

A5

0.73

2

A6

0.59

5

[3]

[4]

[5]

[6]

[7]

It is evident that the new approach is capable of effectively assessing the effectiveness of learning and teaching technologies for teaching distance mode students in higher education. The approach specifically is able to handle the presence of multiple conflicting criteria and the presence of subjectiveness and imprecision inherent in the human decision making process. The results show that the proposed approach is simple to use and applicable for dealing with the general multicriteria decision making problem.

[8] [9]

[10]

[11]

V. CONCLUSION Assessing the effectiveness of learning and teaching technologies for teaching distance mode students in higher education is challenging because of the presence of subjectiveness and imprecision involved in human decision making process. Hence, handling of multi-dimensional nature of this selection process and sufficiently model the subjectiveness and imprecision becomes an important step towards dealing with dealing with the learning and teaching technology selection problem.

[12]

[13]

[14]

[15]

In order to solve this problem effectively, this paper has framed the learning and teaching technology evaluation and selection problem as a multicriteria decision making problem and applied the new approach for addressing the learning and teaching technology selection problem. The findings demonstrate that the approach is capable of handling the presence of subjectiveness and imprecision in the learning and teaching technology selection problem and adequately dealing with the cognitive demand on the decision maker in the learning and teaching technology selection process.

[16]

[17] [18]

[19] [20]

REFERENCES [1] [2]

[21]

R. L. Moore, “Developing community in distance education courses,” Tech Trends, vol. 58, pp. 20-24, 2014. R. Garrison, “Theoretical challenges for distance education in the 21st century: A shift from structural to transactional issues,” Int. Rev. Res. Open Distance Learn., vol. 1, pp. 1-17, June 2000.

[22]

29

A. Carvalho, N. Areal, and J. Silva, “Students’ perceptions of Blackboard and Moodle in a Portuguese university,” British J. Educ. Technol., vol. 42, pp. 824-841, September 2011. Z. Unal and A. Unal, “Evaluating and comparing the usability of webbased course management systems,” J. Technol. Educ., vol. 10, pp.1938, 2011. E. Hustad and A. A. B. Arntzen, “Facilitating teaching and learning capabilities in social learning management systems: Challenges, issues, and implications for design,” J. Integr. Des. Process Sci., vol. 17, pp. 1735, January 2013. S. Wibowo and H. Deng, “Intelligent decision support for effectively evaluating and selecting ships under uncertainty in marine transportation,” Expert Syst. Appl., vol. 39, pp. 6911-6920, June 2012. L. Purnell and E. Hoban, “The lived experiences of third culture kids transitioning into university life in Australia,” Int. J. Intercult. Relat., vol. 41, pp. 80-90, July 2014. A. Norton, Mapping Australian higher education,” Grattan Institute, 2012. D. Persico, S. Manca, and F. Pozzi, “Adapting the technology acceptance model to evaluate the innovative potential of e-learning systems,” Comput. Hum. Behav., vol. 30, pp. 614-622, January 2014. T. Escobar-Rodriguez and P. Monge-Lozano, “The acceptance of Moodle technology by business administration students,” Comput. Educ., vol. 58, pp. 1085-1093, May 2012. J. Schoonenboom, “Using an adapted, task-level technology acceptance model to explain why instructors in higher education intend to use some learning management system tools more than others,” Comput. Educ., vol. 71, pp. 247-256, February 2014. A.Q. Al Hamad and M. Salameh, “Identifying students’ trends toward personalizing learning management system (LMS) at Zarqa University,” J. IT Econ. Dev., vol. 5, pp. 36-46, April 2014. T. Kafadar and B. Tay, “Learning strategies and learning styles used by students in social studies,” Int. J. Acad. Res., vol. 6, pp. 259-267, March 2014. T. Yang, G. Hwang, and S. Yang, “Development of an adaptive learning system with multiple perspectives based on students’ learning styles and cognitive styles,” J. Educ. Technol. Soc., vol. 16, pp. 185-200, 2013. K. C. Laudon and J. P. Laudon, Management Information Systems,: Managing the Digital Firm, 12th ed., USA: Pearson Education Limited, 2012. M. Hersh, “Evaluation framework for ICT-based learning technologies for disabled people,” Comput. Educ., vol. 78, pp. 30-47, September 2014. K. V. D. Sluijs and K. M. Hover, “Integrating adaptive functionality in a LMS,” Int. J. Emerg. Technol. Learn., vol. 4, pp. 46-50, 2009. T. L. Tang and M. J. Austin, “Students’ perceptions of teaching technologies, application of technologies, and academic performance,” Comput. Educ., vol. 53, pp. 1241-1255, December 2009. H. J. Zimmermann, “An application-oriented view of modeling uncertainty,” Eur. J. Oper. Res., vol. 122, pp. 190-198, 2000. S. J. Chen and C. L. Hwang, Fuzzy Multiple Attribute Decision Making: Methods and Applications, New York: Springer-Verlag, 1992. S. Wibowo and H. Deng, “Consensus-based decision support for multicriteria group decision making,” Comput. Ind. Eng., vol. 66, pp. 625-633, 2013. Y. Shee and Y. S. Wang, “Multi-criteria evaluation of the web-based elearning system: A methodology based on learner satisfaction and its applications,” Comput. Educ., vol. 50, pp. 894-905, April 2008.