Empirical Evaluation of the Actual Use of Learning Objects and Metadata in Learning Object Repositories Jehad Najjar Computer Science Department, K.U.Leuven B-3001 Leuven, Belgium
[email protected]
Abstract: Learning Object technology is being considered as the natural form of content re-use that will considerably decrease the cost of developing electronic learning (educational) material. The focus of my research is evaluating and tracking the actual use made of learning objects, metadata and their associated toolsets in Learning Object Repositories. This will enable users and tools to learn from the way users use learning objects and technology in general. Hereby, improve the use of learning objects and their associated toolsets.
1
Introduction
In this research, we are carrying out empirical studies to evaluate the actual usage of learning objects and metadata in Learning Object Repositories (LORs). We have statistically analyzed the actual use made of metadata elements when indexing and searching learning objects in the ARIADNE knowledge Pool System (KPS). Moreover, we conducted usability studies to evaluate tools and functionalities used to index or finding learning objects. In order to learn from the way people actually use new technologies for learning, we have been developing a framework [13] that helps us to track the behavior of users and learners. This framework tracks and publishes attention given to learning objects and notifies the user about objects that she might be interested in. In the coming sections, we give a very brief description of our peaces of work, in addition to some selected results (see the references part of this paper for the full versions of the work). Section 2 discusses the use of metadata in the indexation and search processes. Section 3 studies the usability of learning object indexation and search tools. Section 4 presents a new framework proposed to track, publish and share the behavior of learning object users. Section 5 summarizes the research issues discussed in this paper.
2
Actual Use of Metadata
This section studies the actual use made of metadata elements in Learning Object Repositories (LORs). Those elements form the application profile of LORs and provided to facilitate finding of learning objects. This section is based on two publications. Sub-section 2.1 is based on the Actual Use of Metadata in Ariadne: An Empirical Analysis [9], published at the ARIADNE 2003 conference. Sub-section 2.2 is based on User Behavior in Learning Object Repositories: An Empirical Analysis [10], published at the ED-MEDIA 2004 World Conference on Educational Multimedia, Hypermedia and Telecommunications. Results and findings of such studies provide us with empirical guidelines to asses the development and evaluation of application profiles and metadata toolsets. 2.1
Use of Metadata in Learning Object Indexation Process
In this study, we present a statistical analysis of the actual use of ARIADNE metadata elements in indexing learning objects. The results are derived from analyzing the empirical data (usage logs) of 3,700 ARIADNE metadata instances (the number available when we started the analysis). Table 1 shows the percentage of times each Ariadne data element was filled in by indexers during the indexing process. Table1. Percentage of usage made of data elements by ARIADNE indexers Element
Value Value not Most used % of M provided provided Vocab-value(M) (filled-in) (%) (%) Granularity 91.9 * 8.1 Lesson 92.7 Didactical Context 53.3 46.7 University Degree 69.7 Interactivity Level 53.2 46.8 Medium 67.7 Semantic Density 52.4 47.6 Medium 76.4 Difficulty Level 52.2 47.8 Medium 72.8 Restrictions 5.2 94.5 Contact Author 90 Source 1.3 98.7 Version Information 7.0 93.0 Description 11.2 88.2 OS Version 0.5 99.5 Installation remarks 24.3 75.7 Other Constraints 0.15 99.85 *: used to be mandatory at the previous version of ARIADNE authoring tools.
%M among all cases 85.2 37.2 36.1 40.0 38.0 5.2 -
From the data shown in table 1, we notice that only one data element is almost always used: the Granularity element. Other elements are used in about 50 % of the descriptions and the rest are rarely used in the indexation process. For the values of data elements, we can see that indexers often use just one value. However, this shows that indexers are different in the way they chose data elements to describe their learning objects as well as vocabulary values assigned to each data element. Moreover, by looking to the metadata information filled in by each indexer,
we noticed that indexers often use mental templates of elements and values every time they index new learning objects. Predicting relationships between data elements is not an easy job. The relationships between the studied data elements will form the guidelines for successful automatic indexation or application profiles development in general. In table 2, the high correlation between Interactivity Level and Semantic Density proves that choosing an Interactivity Level means a high probability for Semantic Density to be fill-in and visa versa. Moreover, if the value of Semantic Density is “high” then Interactivity Level will be most probably “high” too. Based on the correlations within the elements we may automatically fill-in or suggest values of other co-related element. We may also hide some elements to the user based on a correlation with other elements. Table 2. Measures of association’s strength between data element
Spearman's rho Aggregation Leve Correlation Coefficien Sig. (2-tailed) N Semantic Density Correlation Coefficien Sig. (2-tailed) N Interactivity Level Correlation Coefficien Sig. (2-tailed) N Difficulty Level Correlation Coefficien Sig. (2-tailed) N
Aggregation Semantic Interactivity Difficulty Level Level Density Level 1.000 .058* -.443** -.112** . .011 .000 .000 3381 1925 1954 1915 .058* 1.000 .744** -.436** .011 . .000 .000 1925 1929 1929 1765 -.443** .744** 1.000 -.296** .000 .000 . .000 1954 1929 1958 1794 -.112** -.436** -.296** 1.000 .000 .000 .000 . 1915 1765 1794 1919
*. Correlation is significant at the .05 level (2-tailed). **. Correlation is significant at the .01 level (2-tailed).
This kind of studies will allow us to enhance the use of ARIADNE metadata and provide guidelines for developing, evaluating application profiles and its associated tools such as automatic indexation. 2.2
Use of Metadata in Learning Object Search Process
In this study, we investigate the ways in which users interact with Learning Objects Repositories (LORs) when searching for relevant learning objects. We present a statistical analysis of ARIADNE query log files of readily available data on 4,723 queries launched by about 390 users in six ARIADNE LKPs [Genoa, Galati, Grenoble-UJF, Lausanne-EPFL, Lausanne-UNIL and Leuven-CS] over different time periods.
1200
1. General
1000 2. Semantics
800 5. Indexation 600 3. Pedagogical
400 4. Technical 200
D o D oc cum um e en nt T tL it A an le ut gu ho ag rL e as tN am U sa e ge P ub R ig lic h at ts A io ut n ho D rF at e irs tN am S ci e en ce M Ty ai n pe D is ci S pl ub in -D e is ci M pl ai in n e C D on oc um cep D t en oc um t T y p en e tF or m at D ur at io n O U pe se ra rt tin yp g e S ys M te ed m ia Ty M ai pe n (s Fi ) le N am e H ea H de H ea ea rI de de d rC rA u re th at or io n D at e
0
Figure 1: Freq. of elements used in searchers’ queries Figure 1 shows the frequency of the number of times the different ARIADNE elements have been used in searchers’ queries. Analysis for the frequency of elements used in searchers’ queries reveals that searchers mostly accept the default provided data elements. The most used 20 data elements by searchers are default provided data elements. Remarkably, the query tool allows searchers to change the default settings for the query tool and show the whole list of elements. Few searchers change the default element settings provided. These results can be interpreted in two ways. First, the default settings are the most related elements to ARIADNE users. Second, searchers have a tendency to accept default settings. Giving the precise reason for these results requires further investigations to test the above two mentioned hypothesis. A comparison between the actual usage of data elements in both indexation (see section 2.1) and search processes, reveals that data elements that have been used by more than 50% of the indexers are not used by the majority of searchers, such as granularity, didactical context and semantic density elements. In addition, for values of such data elements, we noticed that both indexers and searchers mostly select same values to index or search learning objects.
Table 3: Frequency of elements used in searchers queries No. elements used in Queries Frequency Percent
0
1
2
3
4
>=5
Total
548 11.6
2488 52.7
701 14.8
498 10.5
258 5.5
230 4.9
4723 100.0
Table 3 shows that searchers are more interested in forming queries that contain relatively few metadata elements. The majority of queries (75%) contain one to three data elements. Less than 5% of the queries contain five or more data elements. The mean of the number of elements in queries is 1.7 elements and the SD is 1.6. About
12% of queries contained no metadata elements at all. In fact, this is related to some usability problems with the query tool; some searchers directly launch queries without selecting any data element. Also, they might select their appropriate data element, but without specifying the appropriate string or mathematical operator such as: starts with, contains, Ends with, =, >,
