iiWAS 2008
Proceedings of iiWAS2008
Building an Arabic Learning Object Repository with an Ad Hoc Recommendation Engine Hend S. Al-Khalifa Department of Information Technology College of Computer and Information Sciences King Saud University, Riyadh, Saudi Arabia
[email protected] ABSTRACT There are many Arabic educational resources on the web scattered in personal websites and forums, and created by members of the community. Searching for such resources using regular search engines is not an easy task. In this paper we describe “Marifah”, an Arabic Learning Object Repository with recommendation capabilities, created for hosting Arabic learning objects and serving the needs of the Arabic educational community. The repository has integrated advanced features that cannot be fulfilled using well-know search engines.
Categories and Subject Descriptors H. Information Systems, H.3 INFORMATION STORAGE AND RETRIEVAL, H.3.7 Digital Libraries.
General Terms Design, Experimentation.
Keywords Digital libraries, learning object repositories, recommender engine, Arabic.
1. INTRODUCTION In recent years, the concept of learning object has been widely used and discussed by stakeholders in the educational field. A learning object is defined as a digital, reusable piece of content that can be used to accomplish a learning objective, which means that a learning object could be a text document, a movie, an mp3 file, a picture or even a complete website [1]. Due to the large increase in the number of learning objects on the Internet, the need for centralized digital repositories to host these objects has emerged. A digital repository for learning objects can be defined as a database system to store, manage, search, retrieve Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. iiWAS2008, November 24–26, 2008, Linz, Austria. (c) 2008 ACM 978-1-60558-349-5/08/0011 $5.00.
and deliver digital resources that are used to perform or support learning processes [1]. Moreover, the learning objects stored in digital repositories have to be indexed with some kind of metadata in order to identify, search or reuse them properly. As other communities, Arabs have their own learning objects. They share knowledge with each other in different formats and using different kinds of websites. For example, people share knowledge using PowerPoint slides, Word documents, or even publish them as posts in a forum. Despite the fact that Arabic learning objects are widely available on the Internet, they are not kept in a centralized digital repository which makes them difficult to be found and used. In our research for Arabic Digital Learning Object Repositories (DLOR) we found few initiatives by Arabs such as the one proposed by the Egyptian DLOR1. However, these initiatives did not gain much popularity over the educational community for a set of potential reasons; among them is the use of English interface, the restricted membership and the weak construction of the website. Furthermore, the existence of general purpose Arabic Digital Repositories are rare. According to some figures published by “Open DOAR”2 website, Arabic digital repositories constitute 3% of the total number (i.e. 1163) of open access repositories. This was only an example of the weak existence of Arabic digital repositories in general and DLOR in particular, which encouraged us to develop an Arabic DLOR that bridges the gap in the Arab world3. This paper is structured as follows: section 2, gives a brief background on the concepts of learning objects repositories, learning objects and metadata. Section 3, presents our DLOR system architecture with a focus on its main functionalities, metadata and the recommendation engine. Section 4, shows how we tested the DLOR and section 5 compared our approach to
1
Egyptian Digital Learning Object Repository (EDLOR) (http://www.freewebs.com/alaasadik1/bottom.htm) and the Distributed Egyptian Learning Object Repository (DELOR) by Almansoura University. 2
http://www.opendoar.org/
3
Our project vision is to be ‘The premiere online community for Arabs where faculty, staff, and students share their learning materials.’
390
iiWAS 2008
similar work. Finally, section 6 concludes the paper with future directions.
Proceedings of iiWAS2008
Marifah DLOR main functionalities can be listed as follows: 1.
2. BACKGROUND: LEARNING OBJECTS REPOSITORIES, LEARNING OBJECTS AND METADATA In the learning environment, academic staff and students need to store and keep their intellectual assets in a database that is available and visible to others [2]. This database is usually referred to as Learning Object Repository (LOR).
[1] Searching: This function provides the users with the ability to find Learning Objects based on keywords (simple search) or by metadata fields (advanced search). [2] Browsing: Allow users to locate content by choosing the appropriate categories and subcategories. 2.
LO Manipulation, which include: [1] Contributing: a registered user can upload a new Learning Object to the repository. This operation is simply done by uploading the LO and filling the required metadata fields.
Learning Objects (LO), as defined by IEEE's Learning Technology Standards Committee are "any entity, digital or nondigital, which can be used, re-used or referenced during technology supported learning" [10]. Examples of Learning Objects include multimedia content, instructional content, learning objectives, instructional software and software tools, persons and organizations [1]. Learning objects need to be indexed with some metadata in order to be found. Metadata is defined as data that describe learning objects in different aspects, for the purpose of access, modification, evaluation and association with other learning objects [1].
LO Search and Retrieval, which include:
[2] Deleting a LO: Delete a LO from the repository. [3] Modifying metadata: Modify a LO metadata. [4] Personal Collection: A private folder where a user can add and delete his/her favorite LO. [5] Quality Control: Keep the quality of the hosted Learning Objects high by providing services such as "Reporting weak LO" and adding comments. 3.
Metadata for learning objects may include hardware and software specification to utilize a learning object, for instance it may include: the education level (Elementary, Secondary, or Higher Education), type of learning object, LO author, LO owner, terms of distribution, teaching or interaction, etc.
LO Evaluation/ Ranking, which include: [1] Rating: Users can rate the quality of a given LO by voting for it. [2] Reports and statistics: Give statistical reports on the use of the repository by users; also give information about the usage of LO within the system.
Any DLOR must contain both LOs and some form of metadata supported with a set of functionalities (e.g. search, browse, add, etc) in order to fully benefit from the repository. With this in mind the introduction of An Arabic DLOR is presented next.
[3] Track back and RSS: These two features will enable our DLOR to disseminate its content easily and track LOs usage remotely.
3. MARIFAH SYSTEM ARCHITECTURE This section discusses the design and implementation of an Arabic Learning Object Repository (called Marifah) with an emphasis on the recommendation engine model.
[4] Recommendation: This feature will recommend new LO(s) to a registered user, based on LO rating, collected statistics and other useful information such as favorite collections and number of track back6.
Three important issues need to be determined while designing Marifah DLOR, namely:
4. And finally other supported functionalities such as registration and administration.
1.
What features to include in the DLOR?
2.
Which Metadata to use?
3.
And, which recommendation method to use?
Next, the three issues are discussed in further detail.
3.1 LOR features The main features and functionalities included in Marifah were reached after reviewing the current available digital repositories for learning objects. Around twenty popular DLOR, such as Merlot4 and Wisconsin5, were thoroughly reviewed and analyzed to come up with a matrix of the important functionalities found in DLOR.
4
http://www.merlot.org/merlot/index.htm
5
http://www.wisc-online.com/
3.2 Metadata In any learning object repository, ‘Search’ and ‘Browse’ are the most used methods for finding a learning object. However, if these learning objects are not identified, it will be impossible to reach a specific learning object. Thus, in order for a learning object to be accessible, it must be indexed with some form of Metadata. There are many Metadata schemes that are used by DLOR; the most popular ones are Dublin Core education (DCEdu), SCORM and IEEE-LOM. In Marifah DLOR, DCEdu schema was used. This scheme contains the necessary elements for describing Arabic LOs. The elements of DCEdu includes: Title, Creator, Subject and Keywords, Audience, Resource type, Category and Language. 6
Track back is a method to keep track of links to content.
391
iiWAS 2008
Proceedings of iiWAS2008
Table 1: Factors participating in the computation of the recommendation value along with their corresponding percentage.
3.3 Recommendation Engine Model The question we have asked our selves before implementing a recommendation engine in our DLOR is: how useful is implementing a Recommender System in an educational website such as a DLOR? The answer was: since the number of learning objects will increase after the lunch of the website, finding a LO by browsing through categories or searching using keywords might result in LOs that do not match a user preferences; moreover, there might be LOs that are not shown in the search results but are the most relevant LOs that match a user preference. In this respect, a Recommender System will help users in exploring the desired LOs. However, to make a Recommender System works properly, users’ must participate in the recommendation process by rating LOs as they explore/use them. Recommender systems are “examples of adaptive filters that use inference drawn from users’ known behavior to recommend ‘items’ they have not yet seen.”[3] There are several techniques that could implement a Recommender System. One of these techniques is by using Collaborative Filtering (CF), which is the most widely used technique in recommendation systems. CF works by creating a database of favorite preference for items contributed by users. Recommended items for a given user are derived from databases of “neighbor” users who have similar item preferences with that user.
Data
70%
No. Downloads (D)
20%
Trackback (T)
10%
Total
100%
3.3.1 Recommendation Engine Algorithm Figure 1 gives an overview of our recommendation model. Suppose we have a user u, u has a list of LOs that (s)he rated and stored in his/her history. For each LO under which u has specified as his/her interest field, compute the LO similarity and the recommendation value then rank the LOs on a bases of top three.
User account
Recommendation Profile
First, the performances of item-based algorithms are significantly better than user- based algorithms [4]. Item-based algorithms avoid any relationships between items and users; instead, they concentrate on exploring relationships between items only. Therefore, computations will be performed faster, which leads to better performance.
In our DLOR the recommendation value for a LO was computed by simulating the traditional item-based CF algorithm. The factors used to compute the recommendation value are shown in Table 1.
Rate (R)
Notice that the highest weight was given to the rating value of a LO (70%), then to the number of downloads (20%) and finally to the number of tack backs (10%). The choice of the weighting was based on our vision on the importance of each factor.
There are two main approaches for building a Recommender System using CF. One is called User-based (Memory-based) algorithm and the other approach is the Item-based (Modelbased). After studying both collaborative filtering approaches, we found that using the item-based collaborative filtering is more suitable for a DLOR for two reasons.
The second reason is that item-based algorithms provide better quality than user-based algorithms [4]. User-based algorithms quality depends, in the first place, on the number of “neighbors”; i.e. the quality increases as the number of neighbors in the repository increases. However, using item-based algorithms will result in a constant degree of quality whether the number of neighbors is increasing or not. This is due to the fact that itembased algorithms concentrate on a user’s preference in the first place; the recommendations are given based on what items he/she prefers in the past not on what other people have preferred. As a result, item-based algorithms are more suitable than user-based algorithms, which lead to a better recommendation quality.
Value
Interest fields
Rating
No. of Downloads
+
Trackback
Figure 1. Recommendation Model. Step 1. Look for Similar LOs: The first step is to find out the similarity between the current LO (picked from the repository) and all LOs in u's history. By using the Correlation- based Similarity:
∑ (V
u ,i
Sim(i, j ) =
− Vi )(Vu , j − V j )
u∈U
∑ (V
u ,i
u∈U
(1)
− Vi )
2
∑ (V
u, j
−Vj )
2
u∈U
Where U is the set of all users, V is the sum of the factors participating in the computation of the recommendation value with their respected weights; i.e. V= [(R × 0.7) + (D × 0.2) +
(T × 0.1)] Step 2. Compute LO Recommendation Value (RV): Next we compute the recommendation value of the current LO by using the weighted sum method:
392
iiWAS 2008
Proceedings of iiWAS2008
∑ (Sim × V = ∑ ( Sim ) i ,n
RVu ,i
u ,n
)
n∈N
Re call = (2)
number of retrieved relevant LOs number of all relevant LOs
(3)
i ,n
n∈N
Where N is the set of all similar items for user u. Step 3. Recommendation Ranking:
A sample consisted of 130 learning objects and 5 fictional registered users each of which are assigned a user ID (from 1 to 5) with their interest fields populated. Then we tested the recall for each fictional user in our DLOR.
The final step is to recommend to user u the top three LOs.
Table 2: Recall Results.
4. TESTING AND EVALUATION
User ID
After the final integration of Marifah DLOR (see Figure 2), a component testing was performed on each module to make sure that the modules are interacting properly. Also a black-box testing was carried out to assess the functionalities of Marifah.
Recall
1
2
3
4
5
100%
50%
0%
0%
33%
Table 2 shows the results of the experiment. It is clear from the results that the recommender system performed very well in some cases and failed in others. This can be attributed to the users’ votes; we mentioned in section 3.3 that the recommendation will work properly if users participated in rating LOs in their interest fields. User 1 shows a case when a user has an interest field and votes in his/her interest fields and other non-interest fields. User 2 shows a case when a user has many interest fields and votes in his/her interest fields and other non-interest fields. User 3 and 4 show a case when a user has an interest field and votes in a non-interest field or did not vote at all. Finally, user 5 shows a case when a user has many interest fields and votes in one of his/her interest fields.
5. DISCUSSION AND RELATED WORK Figure 2. Marifah website (http://marifah.org). As for evaluating the core component of Marifah DLOR, i.e. recommender system, there were two approaches to accomplish this task [5]: •
“Off-line evaluation: where the performance of a recommender system is evaluated on existing datasets. • On-line evaluation: where performance is evaluated on users of a running recommender system.” On-line evaluation, by no means, is the best method to evaluate the functionality and performance of recommender systems; however, this kind of evaluation is problematic because of the need to build up a community of users. Since Marifah, has just lunched (July 2008), it needs some time to gain popularity and attract users. Consequently, off-line evaluation will be used for its simplicity. In off-line evaluation the recommender system can be treated as an information retrieval system [5], therefore, the metrics for evaluation will be the well-known measures of precision and recall. The definition of recall is “…the number of retrieved relevant documents divided by the number of all the relevant documents, which are previously identified by domain experts” [7]. The definition of precision is “…the number of retrieved relevant documents divided by the number of all retrieved documents” [7]. In this evaluation, we will focus only on recall. Wang et al. (2007) defines recall formula for LOs as follows:
It is interesting to note that in all five cases of this experiment the recall was fluctuating depending on the existence of interest fields. Even if the interest field was populated for a given user the recall result was different depending on the user rating history. These results also showed how Rating and Interest fields are consolidated to affect the retrieval of LOs. In recent years, there has been an increasing amount of literature on LOs recommendation. However, to the best of our knowledge, we have not seen any Arabic research in DLOR or even in LOs recommendation in DLOR. Recent studies on LO recommendation have focused on the use of Semantic Web technologies (i.e. ontologies). Tsai et al. (2006) [6] and Wang et al. (2007) [7] introduced an adaptive personalized ranking mechanism to help recommend SCORMcompliant learning objects from repositories in the Internet. This model adopts an ontological approach to perform semantic discovery as well as using both preference-based and correlationbased approaches in ranking the degree of relevance of learning objects to a user’s intension and preferences. Ochoa and Duval (2006) [8] used Contextual Attention Metadata (CAM), gathered from different tools used in the lifecycle of a Learning Object, to create ranking and recommending metrics to improve the user experience. The factors used in the metrics are: Link Analysis Ranking, Similarity Recommendation, Personalized Ranking and Contextual Recommendation. Finally, Ouyang and Zhu (2007) [9] used mining techniques to find LOs relation based on the learners' usage information in the eLORM repository. By analyzing the LO relation patterns from
393
iiWAS 2008
Proceedings of iiWAS2008
the learners learning domain, granularity, categories, etc., the correlated various learning objects are able to be recommended to the learners. From the previous research we can see that there are different techniques used to recommend LOs in DLOR. From these techniques the use of ontologies as in [6, 7], the use of attention metadata as in [8] and performing data mining techniques as in [9], however, the recommendation used in our DLOR can be classified as an ad hoc recommendation system based on factors we proposed.
6. CONCLUSION AND FUTURE WORK In this paper we have proposed the design and implementation of an Arabic DLOR (called Marifah) with a built-in recommender system. Marifah website was devoted for storing Arabic learning objects and trying to bridge the gap of DLOR lack in the Arab world. Marifah website contains many useful features for managing learning objects. Users can contribute learning objects in two ways: either by linking to a webpage or by uploading directly to the repository. Learning objects stored in the repository can be rated or even used externally by tracking back their usage. Tracking back is a unique feature that few repositories on the web can deal with. Moreover, users can explore the website easily and in an organized way by organizing learning objects into categories then sub categories and providing many tools to evaluate, distribute, and comment on them. The built-in recommender system, our DLOR most unique feature, will assist members of Marifah in choosing what learning objects are suitable for their interest. Finally, Marifah website at its current state contains the basic functionalities that any digital repository for learning objects may have, plus the unique feature of LO recommendation. In addition, the website is capable of encompassing extra functionalities that will enhance our DLOR in the future. Among the expected future work are: 1) Evaluating the recommender system using a large dataset and enhancing its performance, 2) Adding federated search to expand the search range and 3) Revise and improve the recommendation process.
7. ACKNOWLEDGMENTS I would like to thank my students: Najla Alturaiki, Rana Alqahtani, Sara Alkhudhair, Roaa Alsulaiman, Jawaher Alabdulkareem and Dhoha Almazroa, for implementing and testing the repository as their graduation project for the year 2008.
8. REFERENCES [1] Al-Khalifa, H., and Davis, H. AraCore: An Arabic Learning Object Metadata for Indexing Learning Resources, MTSR, online, Spain, November 21-30, (2005). [2] Hayes, H. Briefing Paper - Digital Repositories: Helping universities and colleges, (2005). Last Accessed July, 1, 2008. Available online http://www.jisc.ac.uk/media/documents/publications/repositoriesb phe.doc. [3] Duncan Pemberton, “GroupMark: A WWW Recommender System Combining Collaborative and Information Filtering”,(2000), Last Accessed July, 1, 2008. Available online http://ui4all.ics.forth.gr/UI4ALL2000/files/Long_papers/Pemberton.pdf. [4] Badrul Sarwar, “Item-Based Collaborative Filtering Recommendation Algorithms”, (2001), Last Accessed July, 1, 2008. Available online http://www.inf.ed.ac.uk/teaching/courses/tts/papers/sarwar.pdf [5] Hayes, C., Massa, P., Avesani, P., and Cunningham, P. (2002). An On-line Evaluation Framework for Recommender Systems. Technical Report TCD-CS-2002-19, Department of Computer Science, Trinity College Dublin. [6] Tsai, K.H., Chiu, T.K., Lee, M.C., Wang, T.I. (2006), "A learning objects recommendation model based on the preference and ontological approaches". ICALT, Computer Society, IEEE. [7] Wang, T. I., Tsai, K. H., Lee, M. C., & Chiu, T. K. (2007). Personalized Learning Objects Recommendation based on the Semantic-Aware Discovery and the Learner Preference Pattern. Educational Technology & Society, 10 (3), 84-105. [8] Ochoa, X. & Duval, E. (2006). Use of contextualized attention metadata for ranking and recommending learning objects. Proceedings of the 1st international workshop on Contextualized attention metadata: collecting, managing and exploiting of rich usage information, ACM Press, Pages: 9 - 16. [9] Ouyang, Y. & Zhu, Z. (2007). "eLORM: Learning Object Relationship Mining based Repository," E-Commerce Technology and the 4th IEEE International Conference on Enterprise Computing, E-Commerce, and E-Services, 2007. CEC/EEE 2007. The 9th IEEE International Conference on , vol., no., pp.691-698, 23-26 July 2007 [10] IEEE Standard for Learning Object Metadata. Last Accessed July, 4 2008.Available online http://ltsc.ieee.org/wg12/
394
