Actual Use of Learning Objects and Metadata_ An ... - CiteSeerX

Actual Use of Learning Objects and Metadata: An Empirical Analysis

02/01/08 19:21

TCDL Bulletin Current 2006 Volume 2 Issue 2

Actual Use of Learning Objects and Metadata An Empirical Analysis Jehad Najjar and Erik Duval Computer Science Department, K.U.Leuven B-3001 Leuven, Belgium {Jehad.Najjar, Erik.Duval}@cs.kuleuven.ac.be

Abstract Learning Object technology is being considered as the natural form of content re-use that will considerably decrease the cost of developing electronic learning (educational) material. The focus of my research is evaluating and tracking the actual use made of learning objects, metadata and their associated toolsets in Learning Object Repositories. This will enable users and tools to learn from the way users use learning objects and technology in general, and hereby, improve the use of learning objects and their associated toolsets.

1. Introduction In this research, we are carrying out empirical studies to evaluate the actual usage of learning objects and metadata in Learning Object Repositories (LORs). We have statistically analyzed the actual use made of metadata elements when indexing and searching learning objects in the ARIADNE knowledge Pool System (KPS). Moreover, we conducted usability studies to evaluate tools and functionalities used to index or find learning objects. In order to learn from the way people actually use new technologies for learning, we have been developing a framework [13] that helps us to track the behavior of users and learners. This framework tracks and publishes attention given to learning objects and notifies the user about objects in which he or she might be interested. In the coming sections, we give a very brief description of our pieces of work, in addition to some selected results (see the references part of this paper for the full versions of the work). Section 2 discusses the use of metadata in the indexing and search processes. Section 3 studies the usability of http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

Page 1 of 12

Actual Use of Learning Objects and Metadata: An Empirical Analysis

02/01/08 19:21

learning object indexing and search tools. Section 4 presents a new framework proposed to track, publish and share the behavior of learning object users. Section 5 summarizes the research issues discussed in this paper.

2. Actual Use of Metadata This section studies the actual use made of metadata elements in Learning Object Repositories (LORs). Those elements form the application profile of LORs and are provided to facilitate the finding of learning objects. This section is based on two publications. Sub-section 2.1 is based on the Actual Use of Metadata in ARIADNE: An Empirical Analysis [9], published at the ARIADNE 2003 conference. Subsection 2.2 is based on User Behavior in Learning Object Repositories: An Empirical Analysis [10], published at the ED-MEDIA 2004 World Conference on Educational Multimedia, Hypermedia and Telecommunications. Results and findings of such studies provide us with empirical guidelines to assess the development and evaluation of application profiles and metadata toolsets. 2.1 Use of Metadata in Learning Object Indexing Process In this study, we present a statistical analysis of the actual use of ARIADNE metadata elements in indexing learning objects. The results are derived from analyzing the empirical data (usage logs) of 3,700 ARIADNE metadata instances (the number available when we started the analysis). Table 1 shows the percentage of times each ARIADNE data element was filled in by indexers during the indexing process. Table 1. Percentage of usage made of data elements by ARIADNE indexers

From the data shown in Table 1, we notice that only one data element is almost always used: the Granularity element. Other elements are used in about 50 % of the descriptions and the rest are rarely used in the indexing process. For the values of data elements, we can see that indexers often use just one value. However, this shows http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

Page 2 of 12

Actual Use of Learning Objects and Metadata: An Empirical Analysis

02/01/08 19:21

that indexers are different in the way they choose data elements to describe their learning objects as well as vocabulary values assigned to each data element. Moreover, by looking to the metadata information filled in by each indexer, we noticed that indexers often use mental templates of elements and values every time they index new learning objects. Predicting relationships between data elements is not an easy job. The relationships between the studied data elements will form the guidelines for successful automatic indexing or application profiles development in general. In Table 2, the high correlation between Interactivity Level and Semantic Density proves that choosing an Interactivity Level means a high probability for Semantic Density to be fill-in and vice versa. Moreover, if the value of Semantic Density is "high" then Interactivity Level will most probably be "high" too. Based on the correlations within the elements, we may automatically fill-in or suggest values of other co-related elements. We may also hide some elements from the user based on a correlation with other elements. Table 2. Measures of association's strength between data element

These kinds of studies will allow us to enhance the use of ARIADNE metadata and provide guidelines for developing and evaluating application profiles and their associated tools such as automatic indexing. 2.2 Use of Metadata in Learning Object Search Process In this study, we investigate the ways in which users interact with Learning Objects Repositories (LORs) when searching for relevant learning objects. We present a statistical analysis of ARIADNE query log files of readily available data on 4,723 queries launched by approximately 390 users in six ARIADNE LKPs [Genoa, Galati, Grenoble-UJF, Lausanne-EPFL, Lausanne-UNIL and Leuven-CS] over different time periods.

http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

Page 3 of 12

Actual Use of Learning Objects and Metadata: An Empirical Analysis

02/01/08 19:21

Figure 1: Frequency of elements used in searchers' queries Figure 1 shows the frequency that the different ARIADNE elements have been used in searchers' queries. Analysis for the frequency of elements used in searchers' queries reveals that searchers mostly accept the provided default data elements. The most used 20 data elements by searchers are default data elements. Remarkably, the query tool allows searchers to change the default settings for the query tool and show the whole list of elements. Few searchers change the default element settings provided. These results can be interpreted in two ways. First, the default settings are the most related elements to ARIADNE users. Second, searchers have a tendency to accept default settings. Giving the precise reason for these results requires further investigation to test the above two mentioned hypothesis. A comparison between the actual usage of data elements in both indexing (see section 2.1) and search processes, reveals that data elements that have been used by more than 50% of the indexers are not used by the majority of searchers, such as granularity, didactical context and semantic density elements. In addition, for values of such data elements, we noticed that both indexers and searchers mostly select same values to index or search learning objects. Table 3: Frequency of elements used in searchers queries No. elements used in Queries 0 http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

1

2

3

4

>=5 Total Page 4 of 12

Actual Use of Learning Objects and Metadata: An Empirical Analysis

02/01/08 19:21

Frequency

548 2488 701 498 258 230

4723

Percent

11.6 52.7 14.8 10.5 5.5

100.0

4.9

Table 3 shows that searchers are more interested in forming queries that contain relatively few metadata elements. The majority of queries (75%) contain one to three data elements. Less than 5% of the queries contain five or more data elements. The mean of the number of elements in queries is 1.7 elements and the SD is 1.6. About 12% of queries contained no metadata elements at all. In fact, this is related to some usability problems with the query tool; some searchers directly launch queries without selecting any data element. Also, they might select their appropriate data element, but without specifying the appropriate string or mathematical operator such as: starts with, contains, Ends with, =, >,