User Behavior in Learning Objects Repositories: An ... - CiteSeerX

Proceedings of the ED-MEDIA 2004, pp. 4373-4379, 2004

User Behavior in Learning Objects Repositories: An Empirical Analysis Jehad Najjar, Stefaan Ternier and Erik Duval Computer Science Department, K.U.Leuven Celestijnenlaan 200 A, B-3001 Leuven, Belgium {Jehad.Najjar, Stefaan.Ternier, Erik.Duval}@cs.kuleuven.ac.be

Abstract: This paper investigates the ways in which users interact with Learning Objects Repositories (LORs) such as ARIADNE, MERLOT and SMETE when searching for relevant learning objects. More specifically, we focus on the ways users search the ARIADNE Knowledge Pool System (KPS). Our analysis is based on the log files of queries. We investigate user behavior and compare it with the behavior of the indexers who introduce learning objects into the KPS. We discuss lessons learned from this analysis and suggest guidelines for the development of successful application profiles and toolsets.

1. Introduction Over the last few years, many efforts have been spent to define metadata models. The purpose of these models is to support the indexation and search of educational materials [learning objects] in Learning Object Repositories [LORs]. Learning objects are stored in LORs to be used for different kinds of educational purposes and by different end users. Understanding the end user behavior helps us to improve the repositories and their associated tools. By doing this, we will be able to serve the users in a more efficient ways. Similar studies have been done in the context of digital libraries in (Jones, et al., 1998) and in the context of Internet browsing in (Cockburn & McKenzie, 2000). In these studies the end user behavior has been studied by analyzing the transactions’ log files. On the other hand, in the context of LORs, this important issue has not been yet extensively investigated (Duval & Hodgins, 2003). In ARIADNE [Alliance of Remote Instructional Authoring & Distribution Network for Europe] learning objects and metadata have been collected for over 7 years in the Knowledge Pool System [KPS]. At the time of writing, the KPS contains nearly 4,750 metadata instances for learning objects from different science types, languages, contexts and granularities; introduced by more than 100 indexers. These metadata instances have been collected in seven Local Knowledge Pool Systems [LKPs] distributed over Europe. Metadata have been produced in nine languages of the multilingual ARIADNE community. Moreover, many searchers from around the world use the ARIADNE query tool (Neven, et al., 2003) to search for their relevant materials. In this paper, we present a statistical analysis of ARIADNE query log files. In the first phase we analyzed readily available data on 4,723 queries launched by about 390 users in six ARIADNE LKPs [Genoa, Galati, Grenoble-UJF, Lausanne-EPFL, Lausanne-UNIL and Leuven-CS] over different time periods. Moreover, we compare the behavior of the ARIADNE searchers with the behavior of the ARIADNE indexers investigated in our previous work (Najjar, et al., 2003). Differences and similarities of the behavior of both searchers and indexers are obtained. Hereby, the degree to which indexers and searchers agree when index or search learning objects in the KPS can be determined. Hence, this work will allow us to enhance the use of ARIADNE metadata and provide guidelines for developing, evaluating application profiles and associated tools. The paper is structured as follows: in section 2, data collection is introduced. In section 3, we discuss the results derived from investigating searchers queries in the ARIADNE KPS. Discussion is illustrated in section 4. Conclusions and future work are drawn in section 5.

Proceedings of the ED-MEDIA 2004, pp. 4373-4379, 2004

2. Data Collection The ARIADNE Indexation- and Query tool shown in figure 1 has been used to introduce learning objects of different granularities, science types and disciplines into the ARIADNE KPS.

Figure 1: Indexation- and Query tool for ARIADNE

Figure 2: Media types of indexed learning objects

Currently, the KPS contains a large number of learning objects used for different educational purposes. For example, PowerPoint slides, zip files, and html documents [see figure 2]. User activities have been logged in each of the ARIADNE LKPs. Data logged includes: • User IP address • Query Date • Metadata elements used to form the query such as: title, science type, main concept, etc. • Conditional operator such as: =, =, contains, starts with, ends with, etc. • Values provided for those selected elements such as: metadata standards in practice, exact sciences, learning technologies, etc. • Number of learning objects that satisfy information provided. The data were collected over different time periods, related to about 390 different users in six ARIADNE LKPs: • • • • • •

Galati [University of Galati, Romania] Genoa [University of Genoa, Italy] Grenoble-UJF [Université Joseph FOURIER à Grenoble, France] Lausanne-EPFL[École Polytechnique Fédérale de Lausanne, Switzerland] Lausanne-UNIL [Université de Lausanne, Switzerland] Leuven-CS [CS Dept., K.U.Leuven, Belgium]

The distribution of the logged queries is shown in table 1. The total number of queries included in this study is 4,723.

Table 1: User queries classified by LKPs and time period LKP Name

Grenoble-UJF

Leuven-CS

Start Date

Feb. 03

Apr. 03

Feb. 02

End Date

Oct. 03

Oct. 03

Dec. 03

1787

1146

857

423

No. of Queries N = 4723 Queries

Lausanne-EPFL

Genoa

Galati

Lausanne-UNIL

Jan. 03

Apr. 02

Feb. 03

Nov. 03

Aug. 03

Oct. 03

409

101

Proceedings of the ED-MEDIA 2004, pp. 4373-4379, 2004

3. Results In this paper we analyze the query log files. This analysis provides insight into search behavior (Marchionini, et al., 2000), which may improve the ARIADNE metadata toolset. More precisely, the searcher interaction with the ARIADNE query tool has been investigated. Then, the results of searchers behavior were compared with the results of our analysis of indexer behavior obtained in our previous work (Najjar, et al., 2003). This section is divided into two parts. In section 3.1, we present general figures for user access and queries. In section 3.2, we provide results related to the metadata elements that users actually rely on in their queries. 3.1 General statistics on user queries The queries were grouped by country as shown in table 2, data show that most searches (60%) come from countries with ARIADNE LKPs. When we analyzed the data in some detail [by manual observations] we noticed that most of the queries come from educational institutions. Table 2: Searchers classified by their countries COUNTRY Client IP address (Freq) Percent

Unknown 131 34

BE 79 20

FR 65 17

CH 40 10

DE 25 6

RO 15 4

IT 10 3

Other 24 6

Total 390 100.0

Figure 3 presents the number of queries issued by searchers in different user sessions. The mean of the number of queries launched per each session is 4.5 and the standard deviation [SD] is 6.4. This figure also shows that around half (53%) of the searchers launch from one to two queries per session. Few (10%) searchers launch more than 10 queries in one session. We can interpret these results as follows, for searchers who launched one or two queries per session, either they could easily launch their queries and find their relevant learning objects, or they did not succeed and gave up after a very low number of attempts. For users who launch many queries, a more detail investigation is needed to know the reason behind that number of queries. 40

30

(%)

20

Sessions

10

0 1

4

7

10

13

16

19

22

25

28

31

35

38

43

49

Number of queries

Figure 3: Number of queries issued in user sessions Based on the data shown in figure 3, we can see that more work is needed to help us better understand how useful and relevant the query results were for the end user. Probably, the session duration, and the position of the results that the user actually downloaded from the results list can help us to analyze this issue in more detail. This investigation may provide more information that enables us to determine if searchers were able to find their relevant learning objects and how they used the results list provided by the system. 3.2 Metadata elements used in queries Table 3 presents the number of metadata elements used in queries. It demonstrates that searchers are more interested in forming queries that contain relatively few metadata elements. The majority of queries (75%) contain one to three data elements. Less than 3% of the queries contain five or more data elements. The mean of the number of elements in queries is 1.7 elements and the SD is 1.6. Remarkably, about 12% of queries contained no metadata elements at all. In fact, this is related to some usability problems with the query tool; some searchers directly launch queries without selecting any data element. Also, they might select their appropriate data element, but without specifying the appropriate string or mathematical operator such as: starts with, contains, Ends with, =,

Proceedings of the ED-MEDIA 2004, pp. 4373-4379, 2004 >, =5

Total

548 11.6

2488 52.7

701 14.8

498 10.5

258 5.5

230 4.9

4723 100.0

Figure 4 shows the frequency of the number of times the different ARIADNE elements have been used in searchers’ queries. This figure also shows the usage made of metadata elements in the five ARIADNE metadata categories [ARIADNE indexation- and query tool groups metadata elements into six relevant categories]; elements of the sixth category “annotation” are only used for indexation purposes. We can also see from the figure, that elements of the first category [1.General] are mostly used in searchers’ queries, and elements of the second category [2.Semantics] are used more than elements of the third category [3.Pedagogical]. Surprisingly, elements of the fifth category [5. indexation] are used more than elements of the last two categories. Two reasons appear to be related to these results: 1. ARIADNE has tools that provide facilities such as course management and exploitation. Tools like WeBLe search the KPS using indexation elements as “Header-identifier” element.

2. When we analyzed the data in some detail we noticed that ARIADNE users search the KPS for their own already indexed materials; indexers may search their already indexed learning objects for reuse purposes using the “Header Author” element. 1200

1. General

1000 2. Semantics

800 5. Indexation 600 3. Pedagogical

400 4. Technical 200

D o D oc cum um e en nt T tL it A an le ut gu ho ag rL e as tN am U sa e ge P ub R ig lic ht at A s io ut n ho D rF at e irs tN S am ci en e ce M Ty ai n pe D is ci S pl ub in -D e is c ip M lin ai n e C D on oc um cep D t en oc um t T yp en e tF or m at D ur at io O n U pe s ra er tin typ g e S ys M te ed m ia Ty M ai pe n (s Fi ) le N am e H ea H de H ea ea rI de de d rC rA ut re ho at r io n D at e

0

Figure 4: Freq. of elements used in searchers’ queries Moreover, based on the number of queries launched per user session shown in figure 3, we analyzed the usage made of metadata elements when users launch three to fifteen queries [40% of the queries]. We found that the distribution of metadata elements usage for those users is similar to the distribution of the usage for the whole set of queries shown in figure 4. From the above discussion, we note that further investigation is needed for the usage of the different metadata elements, made by the different searchers. This investigation will give us more information about the relationship between the metadata elements used and the searchers. For example, it may help us determine if searchers use the same metadata elements every time they search the KPS.

4. Discussion 4.1 Metadata elements often used by indexers might not be used by searchers ARIADNE Indexation- and Query tool uses a set of metadata elements to help indexers and searchers to index or search learning objects in the KPS. These elements form an application profile of LOM (IEEE, 2002). In

Proceedings of the ED-MEDIA 2004, pp. 4373-4379, 2004 our previous work (Najjar, et al., 2003) we investigated the actual usage made of metadata elements by indexers. The percentage usage of the different elements is shown in table 4. As shown in table 4, one vocabulary value of both mandatory and optional data elements is almost always used. For example, “learner” value of user type mandatory element, and “lesson” value for granularity optional element. Moreover, mostly one data element is used indexers, as granularity element (92 %), and the rest are used by about 50 % of the indexers. On the other hand, table 5 shows the percent of the number of times each vocabulary value type ARIADNE metadata element has been used by searchers when searching the KPS for relevant learning objects. Table 4: Percent of elements used in indexation process Metadata element

Most used vocab-value Percent (V) (V)

Document language French Free Usage rights

Element filled in (%)

42.4 % 62.0 %

Table 5: Elements used in searchers queries Metadata Element

Freq. (V)

Percent (V)

Element Filled in (%)

French

521

49%

24%

Exact and natural sciences Informatics

710

84%

18 %

224

31%

15 %

Usage rights

Free

456

89%

11 %

Sub-discipline

General/Sundry

21

6%

8%

Document type

Active

218

63%

7%

Document format

Self assessment

31

22%

3%

User type Media (MIME) type(s) Operating system

Learner

44

81%

1%

Text/HTML

12

12%

2%

Multi OS

62

57%

2%

Document language

69.1 %

Science type

43.4 %

Main discipline

Sub discipline

Exact, Natural and Engineering Sciences Informatics / Information Processing General / Sundry

User type

Learner

95.1 %

Document type

Expositive

64.9 %

Science type Main discipline

Media (MIME) type Text / HTML Narrative text Document format Operating system Multi-OS (OS) Lesson Granularity * University degree Didactical context

40.2 %

Mandatory (100%)

25.4 % 23.5 % 63.8 % 92.7 %

92 %

69.7 %

53.3 %

Most used Vocabvalue (V)

N = 4723 queries

Medium 67.7 % 53.2 % Interactivity level Semantic density Medium 76.4 % 52.4 % Difficulty Level Medium 72.8 % 52.2 % *: used to be mandatory at the previous version of ARIADNE query tool N = 3700 metadata instances

A comparison between the data presented in figure 4 and table 5 with the data presented in table 4 give us a clear idea about the usage made of metadata in both indexation and search processes. This comparison reveals that data elements that have been used by more than 50% of the indexers are not used by the majority of searchers, such as granularity, didactical context and semantic density elements. In addition, for values of such data elements, we noticed that both indexers and searchers mostly select same values to index or search learning objects. 4.2 Searchers mostly accept default element preferences Analysis for the frequency of elements used in searchers’ queries reveals that searchers mostly accept the default provided data elements. As shown in figure 4, the most used 20 data elements by searchers are default provided data elements. Remarkably, the query tool allows searchers to change the default settings for the query tool and show the whole list of elements. Few searchers change the default element settings provided. These results can be interpreted in two ways. First, the default settings are the most related elements to ARIADNE users. Second, searchers have a tendency to accept default settings. Giving the precise reason for these results requires further investigations to test the above two mentioned hypothesis.

5. Conclusions and Future Work User behavior was the focus of many researchers in the different fields such as digital libraries (Jones, et al., 1998), internet browsers (Cockburn & McKenzie, 2000). Currently, there is an urgent need to examine such behavior in the field of learning object repositories. Results from these studies of user search behavior are useful to discover the ways in which users search for relevant learning objects. Hereby, enhance the design of more usable query tools that better serve the needs of such users.

Proceedings of the ED-MEDIA 2004, pp. 4373-4379, 2004 In this paper, we investigated the searchers behavior in ARIADNE KPS. The main contribution of this investigation is to understand the interaction of searchers with the query tool and its associated metadata elements. We found that searchers select one or two metadata elements to form their queries, and they invest little time to form such queries. Therefore, the interface of query tools should provide the user with simple search function that allows searching on the most important [most used] fields as title, author, or main discipline. Moreover, an advanced query interface that contains the most elements used can also be provided. We found that the majority of searchers accept the default provided data elements. Therefore, careful consideration should be given to selecting such elements in the query tools as well as other systems used for education. The comparison between the behavior of searchers and the behavior of the indexers reveals that some metadata elements often used by indexers might not be included in searchers queries. This suggests that we should make the most used elements in queries more prominent in the indexation tool and/or vice versa. Alternatively, we could explore different approaches to deal with metadata, such as for instance Information Visualization (Klerks, et al., 2004), or tools that support social recommendations (Duval & Hodgins, 2003). This investigation will help researchers in the field to understand the interaction of searchers with query tools. Also, it will facilitate the provision of searchers with a query tool that serves the searchers need and saves time and effort. For future work, we intend to: • Investigate searchers queries in other LORs. • Study the relationship between metadata elements used in searchers queries and the searchers themselves. • Investigate queries issued by searchers of (TOLEDO) [an e-learning portal used in K.U.Leuven], where Google box type approach is used. • Compare results of our investigations with results of investigations of Google search.

6. References ARIADNE. Available at: http://www.ariadne-eu.org/ . Duval, E., Forte, E., Cardinaels, K., Verhoeven, B., Durm, R. V., Hendrikx, K., Forte, M. W., Ebel, N., Macowicz, M., Warkentyne, K., and Haenni, F. (2001). The ariadne knowledge pool system. Communications of the ACM, 44(5):72–78. Cockburn, A., McKenzie, B., (2000). What do web users do? An empirical Analysis of Web Use. International Journal of Human-Computer Studies. Available at: http://www.cosc.canterbury.ac.nz/~andy/papers/ijhcsAnalysis.pdf/. Duval, E., Hodgins, W., (2003). A LOM Research Agenda. WWW2003. Available at: http://www.cs.kuleuven.ac.be/Eerikd/PRES/2003/www2003/www2003-paper.pdf. IEEE, (2002). IEEE Standard for Learning Object Metadata. Available at: http://ltsc.ieee.org/doc/wg12/ Jones, S., Cunningham, S. J., McNab, R., (1998). An Analysis of Usage of a Digital Library. ECDL,1998. available at: http://www.cs.waikato.ac.nz/~stevej/Research/PAPERS/eurodl.pdf/. Klerkx, J., Duval, E., Meire, M., (2004). Using Information Visualization for Accessing Learning Object Repositories. 8th Intl. Conf. Information Visualization (IV), 2004. Marchionini, G., Plaisant, C., Komlodi, A., (2000). The people in digital libraries: Multifaceted approaches to assessing needs and impact. Chapter to appear in A. Bishop, B. Buttenfield, & N. VanHouse (Eds.) Digital library use: Social practice in design and evaluation. MIT Press. Available at: http://www.ils.unc.edu/~march/revision.pdf /. MERLOT. Multimedia Educational Resource for Learning and Online Teaching. Available at: http://www.merlot.org/. Najjar, J., Ternier, S., Duval, E., (2003). The Actual Use of Metadata in ARIADNE: An Empirical Analysis. ARIADNE 3rd Conférence, 2003. Available at: http://www.cs.kuleuven.ac.be/~najjar/papers/WWW2003_najjar.pdf/. Neven, F., Duval, E., Ternier, S., Cardinaels, K., Vandepitte, P., (2003). An Open and Flexible Indexation and Query tool for ARIADNE. EdMedia, 2003. Available at : http://www.cs.kuleuven.ac.be/~erikd/PRES/2003/EdMedia2003/Filip.pdf/. SMETE. The SMETE Digital Library. Available at: http://www.smete.org/. TOLEDO. The Toledo Web-based virtual learning environment. Available at: http://toledo.kuleuven.ac.be/. WeBle. The ARIADNE Web-Based Learning Environment. Available at: http://rubens.cs.kuleuven.ac.be:8989/weble/InstallToIndex.do.