Can Educators Develop Ontologies Using Ontology Extraction Tools

Can Educators Develop Ontologies Using Ontology Extraction Tools: An End-User Study Marek Hatala1, Dragan Gašević 2, Melody Siadaty1, Jelena Jovanović3, and Carlo Torniai4 1

Simon Fraser University, Canada 2 Athabasca University, Canada 3 University of Belgrade, Serbia 4 University of Southern California, USA

Abstract. The recent research demonstrated several important benefits in the use of semantic technologies in development of technology-enhanced environments. The one underlying assumption for most of the current approaches is that there is a domain ontology. The second unspoken assumption follows that educators will build domain ontologies for their courses. However, ontologies are hard to build. Ontology extraction tools aim to overcome this problem. We have conducted an empirical study with educators where they used current ontology tools to extract ontologies from their existing course material. The results are reported for the IT and non-IT educators. Keywords: ontology building, domain ontologies, ontology extraction, end-user study.

1 Introduction The Semantic Web technologies seem to be a promising technological foundation for the next-generation of e-learning systems [1]. Many authors have proposed the usage of ontologies in different aspects of e-learning, such as adaptive educational hypermedia, adaptive content authoring, personalization, user model sharing, and context capturing [2, 3, 4, 5]. This is an expected reaction, since e-learning is highly dependent on effective mechanisms for knowledge management capable of integrating various activities that e-learning involves, such as course authoring and adaptation and provision of reliable and timely feedback to both students and teachers. Actually, in European Union (EU) and Canada, a lot of investments have already been put into research projects aimed at enhancing e-learning environments with Semantic Web technologies, such as LUISA (http://luisa.atosorigin.es), REWERSE (http://rewerse.net/), ProLearn (http://www.prolearn-project.org/), and Kaleidoscope (http://www.noe-kaleidoscope.org/). In Canada, the leading project addressing the use of ontologies in eLearning was LORNET Research Network (2003-2008, http://www.lornet.org). In our previous research, we have proven and exemplified the advantages of ontology supported e-learning systems. In particular, in [6] we demonstrated how a combined use of content structure ontology, content type ontology and domain ontology leads to U. Cress, V. Dimitrova, and M. Specht (Eds.): EC-TEL 2009, LNCS 5794, pp. 140–153, 2009. © Springer-Verlag Berlin Heidelberg 2009

Can Educators Develop Ontologies Using Ontology Extraction Tools

141

significant improvements in searching repositories with learning content. In addition, we have also shown in [7] that if these three kinds of ontologies are complemented with a user model ontology and an ontology formally specifying the learning path to be followed by a student, then advanced levels of learning content personalization can be achieved as well. Finally, in our most recent research efforts [8, 9] we demonstrated the relevancy of the integrated use of these different kinds of e-learning ontologies for providing online educators with reliable, fine grained and semantically rich feedback about the learning process. However, the main problem with all approaches that have shown the benefits of ontology adoption in e-learning systems is that they are assuming that required ontologies are available. However, this is not a realistic assumption, at least not for domain ontologies, i.e. ontologies that formalize the subject matter of learning courses. The lack of these ontologies is to be attributed to their creation process which is overly difficult and time consuming for educational practitioners. Based on the experience of the knowledge capture and learning technology communities, and our experience from the abovementioned projects, the major obstacle for widespread use of ontologies in e-learning systems lies in the complexity of the ontology development process, especially when considered from the perspective of teachers and content authors who are typically unaware of ontology existence and relevancy altogether. Although, in the recent years, the Semantic Web community has been showing a constantly increasing interest in automating the process of ontology development and thus reducing the required human effort, fully automatic ontology development is still in the distant future [6]. Our experience as well as experience of some other researchers in the field [10, 11, 12] have proven that unlike other kinds of ontologies relevant for e-learning, a domain ontology (i.e. an ontology formally specifying concepts and relationships of a specific subject domain) cannot be reused across different subject domains, but have to be created anew for each domain. Another feature that makes domain ontologies distinctive from other aforementioned ontologies relevant for eLearning is the need for their constant evolvement, so that the semantics they capture do not lag behind the courses they are aimed to support. A significant topic in our current research is to investigate how to reduce the efforts required for creating domain ontologies in educational systems, and thus implicitly enable easier and wider acceptance of ontology based systems among educational practitioners. Our first step towards achieving this goal was to explore the existing approaches (and related tools) for ontology development and we distinguished the following three general approaches: 1. Handcrafting ontologies from scratch. Even though this is currently the predominant approach, it is suitable only for ontological engineers (i.e., experts in the field of ontology development), since presently available tools, such as Protégé (http://protege.stanford.edu/), assume a background in knowledge engineering and familiarity with ontology languages. 2. (Semi)automatic ontology development using ontology learning tools. These tools aim at reducing the human intervention to supervision of the development process and refinement of the results [13]. Even though the state-of-the-art ontology learning tools, such as Text-2-Onto [14] and OntoLT [15], have a lot of advanced features, they still need a lot of improvements in order to be usable by non-experts.

142

M. Hatala et al.

1. Search and retrieval of ontologies from online ontology libraries. Created through contributions of the Semantic Web community members, ontology libraries such as Swoogle (http://swoogle.umbc.edu/) [16], OwlSeek (http://www.owlseek.com/) or OntoSelect (http://olp.dfki.de/ontoselect) [17] offer a constantly increasing number of specific domain ontologies. However, supporting tools are needed to facilitate the searching process and evaluation of the retrieved ontologies [6]. Even after a comprehensive literature review (including, for example, [13, 18]), we could not identify any research aimed at evaluating the level of adoption of these approaches among end users and identifying the requirements for enabling their widespread use. Our work presented in this paper is a first reported attempt to conduct an empirical study that evaluates those approaches and related tools taking into account both e-learning practitioners requirements and constraints imposed on ontologies by advanced e-learning systems.

2 Study Description In this section we describe the main components of the study and the processes used in the preparation and execution of the study. The third section is focused on analysis of the results and discussion. 2.1 Tool Selection Being aware of the complexity of conventional ontology editors (used for building ontologies from scratch) for non-ontology-savvy users, we put our focus on exploring the available ontology learning tools and online ontology libraries. Information about the current state-of-the-art tools for the selected approaches was collected by exploring the literature on ontology learning [13, 18, 19]. However, out of almost a dozen tools mentioned in research papers, only four of them were publicly available on the Internet: OntoGen (http://ontogen.ijs.si/), Text2Onto (http://ontoware.org/ projects/text2onto/) and its predecessor TextToOnto (http://sourceforge.net/projects/ texttoonto), and OntoLT (http://olp.dfki.de/OntoLT/ OntoLT.htm). Those were the tools that we managed to download and install. We decided to use two of them for evaluation purposes (the other two were discarded either for being outdated or depending on another proprietary tool): Text2Onto [14] and OntoGen. Text2Onto. Text2Onto is an ontology learning framework which supports the automatic or semi-automatic generation of ontologies from textual documents. It combines machine learning approaches with basic linguistic processing for learning atomic classes, class subsumption as well as object properties. The framework provides a Graphical User Interface form which the user can define the corpus (the collection of text documents) form which the ontology will be created, select the available algorithms to be applied for generating concepts/relations, and review the generated ontology. The main problem with the tool is that is not clear how the available algorithms and their combination will affect the generation of the ontology, forcing the users to try all the available options and review for each of them the generated ontology.


143

OntoGen. OntoGen is more oriented towards semi-automatic ontology construction. It is an interactive tool that aids a user during the ontology construction process by suggesting concepts, automatically assigning instances to concepts and providing visual representation of both the ontology and the corpora it is built upon. To build an ontology a user has to supply a set of documents that reflects the domain for which the ontology is to be built. The tool creates the root concept of the ontology and suggests names for it using the extracted keywords. In every step of the hierarchy development, OntoGen suggests subtopic of the currently selected topic thus helping users to build a hierarchical organization of domain concepts. 2.2 Study Scoping The survey aimed at two well defined groups of teachers having low level of variability among their members. The groups belong to two completely opposite domains: Computer Science/Software Engineering/Information Technology and non-Computer Science/Software Engineering. The members of the former group are representatives of those who are in general very familiar with complex software tools utilization and may have some notions related to ontologies (group labeled as IT later), whereas the latter group (labeled nonIT) represents educators who are not aware of ontologies and knowledge representation and are less familiar with complex software tools. When designing the evaluation study we carefully tailored the procedure for conducting the survey and for formulating and formalizing the questionnaire used in the study. We conducted a simulation with the goal to estimate the sample size that, given the expected answers and variability of the population, can maximize the statistical power of the experiments. The target number of participants was set to 15 people for each group since it was a reasonable tradeoff between statistical power (generally at least 80% for expected outcomes) and the actual capability of recruiting a great number of participants. The simulations assumed a latent normal distribution beneath the Likert-scale measurement and used Analysis of Variance (ANOVA) for comparing the tools and were performed using SAS software. 2.3 Participant Selection The participants were required to have preferably a PhD degree, and at least master’s degree in their field of research. As well, they needed to have at least three years of experience in teaching or in course development. They were required to have a substantial course material prepared for the whole duration of the course in the form of text documents (PS Word and PDF), html web pages, PowerPoint lecture slides, etc. The participants were recruited from the university faculty by distrusting an invitation via departmental mailing lists. Departments at authors’ respective universities were targeted, as well as through other authors’ connections, such as departments of our research partners from different post-secondary teaching and research institutions in Canada. The interested participants were screened by the research team for their background and the completeness level of their course material to guarantee the quantitative homogeneity of the basic input into the ontology building tools. While initially approved in October 2007, after obtaining the ethics approval in May 2008, the

144

M. Hatala et al.

participant recruitment started in June 2008. However, due to the summer season the recruitment was very low till September 2008, when we recruited most of the participants and started our experiments. The experiments ended in March 2009. 2.4 Study Procedure Participants submitted their course material to the research team upfront. The research team converted the materials into the plain text format that was an accepted format for both tools. To avoid problems with tools installation on participants machines the specific remote server has been prepared with all the tools installed and the course material in the text format made ready. The server was accessible via remote desktop and the assistance1 was available via installed Skype communication software during the whole session. A step-by-step instructional package was made available to the participants a few days ahead of their scheduled session. The first section of this guide provided instructions on how to connect to the remote server and control the test environment. The other two sets included information about and instructions on how to use the selected ontology building tools: Text2Onto and OntoGen. During the session the participants were asked to build an ontology describing their subject of expertise from the course material they had provided, and using the tools selected for the survey. No time constraints were imposed for the ontology creation process, with the majority of the sessions lasting between 2 and 3 hours. After the task has been completed, the participants were asked to fill in a three-part questionnaire: first part related to the evaluation of the experiment itself (not reported in this paper), second part was a 5-value Likert scale questionnaire used for each tool, and qualitative part with 6 open ended questions. The results are discussed in the following section.

3 Results and Discussion Total of 28 participants were recruited and retained for the study, 18 with the information technology background, and 10 with non-IT background. None had prior experience in the ontology development. We originally aimed at 50-50 split but some participants from the departments that indicated nonIT background were categorized as IT due to their prior education in the computing science. 3.1 Course Material and Ontologies Produced Before the beginning of the session the research team produced the plain text documents from the course material provided by the participants. Using the OntoGen and Text2Onto tools the participants produced the ontologies. The basic descriptive statistics are provided in Table 1. Evaluation of the quality of produced ontologies is beyond the scope of this paper and reported elsewhere [20]. 1

Research assistant with experience in using the tools selected for the study, along with the strong expertise to tackle all technical challenges that might arose during the study (e.g., problems with the connection and differences of different types of computers [e.g., PCs and Macs] used by the participants).


145

Table 1. Descriptive statistics of course material and produced ontologies

Number of words Concepts (OntoGen) Relations (OntoGen) Concepts (Text2Onto) Relations (Text2Onto)

N 25 22 22 22 19

Min 4,345 3 0 228 0

Max 293,555 26 13 4,602 155

Mean 59,372 8.6 3.2 1,494.0 53.4

SD 78,000 5.9 4.0 1,015.0 44.4

3.2 Tool Evaluation Both tools were used by each participant to accomplish the same task: to build an ontology from the course material the participant provided for the study. No training was provided for the tools. After completing the task with the first tool (Text2Onto) the participants were asked to fill in the questionnaire that consisted of a section of questions with respect to the tool itself and another section related to the developed ontologies. Both sections used 5-value LIKERT scale. Next, having used the second tool (OntoGen) for ontology development, the participants evaluated it as well. Finally, they filled in a questionnaire with qualitative questions for both tools. In Table 2 we report the user evaluation for both tools and their comparison from the tool usability perspective and support they provided during the ontology building process. The majority of the findings is negative. Question A.1 for both tools demonstrates that the participants felt that they would like to have more input into the process of generating ontologies. This was more pronounced in the case of Text2Onto tool where the user input is limited to removing discovered concepts and relationships from the proposed ontology, while in the OntoGen the users directly control which concepts in the ontology will be further expanded. In question A.2 in the case of OntoGen participants felt that the tool is in control of the process, while in case of Text2Onto they were neutral. There was a statistically significant difference between the two tools. Importantly, in neither case the users felt that they are in control of the tool and the process, which negatively affects their attitude towards the tools. As indicated by the question A.3, participants were neutral with respect to the tools outcome. More detail discussion of the ontology quality is in the next section. Similarly, participants were neutral with respect to how easy the process of obtaining the ontology was (question A.4). Results for question A.5 indicate that an ability to visualize the ontology is the important one. In case of Text2Onto which provides a long list of proposed concepts and relationships with their weight but without structure representation, the mean was significantly higher than in the case of OntoGen. In question A.6 participants expressed that having an ability to manipulate the generated ontology by the tools is a desirable characteristic of any tool for this purpose. They felt more strongly about this in case of Text2Onto than in case of OntoGen.

146

M. Hatala et al.

Table 2. Text2Onto and OntoGen comparison based on participants’ answers to the questionnaire. Both IT and nonIT participants are included. Values reported are from 5-value Likert scale (1-Completely Disagree, 2-Disagree, 3-Neutral, 4-Agree, 5-Completely Agree). (M, SD, N) values represent mean, standard deviation, and sample count. 2-sample t-test results are shown only for significant differences between Text2Onto and OntoGen.

Question A.1 I prefer to participate in the process of creating an ontology A.2 I felt in control of the process while obtaining the ontology A.3 I am happy with the resulting ontology A.4 I found the process of obtaining the ontology easy to accomplish A.5 Visual representation of ontology helps the creation process. A.6 Being able to manipulate the generated ontology (e.g., add new elements, exclude those that I find unimportant, etc.) would improve my work. A.7 It would have been good to have a sort of guidance during the creation process in order to know how the choices the tool provides would have affected the final result.

Text2Onto (M, SD, N) 4.11, 1.01, 27

OntoGen (M, SD, N) 3.76, 1.05, 25

2-sample t-test -

3.21, 1.06, 28

2.33, 1.03, 27

2.96, 1.03, 28

2.44, 0.93, 27

t(52.9)=3.10, p=0.003 -

3.14, 1.07, 28

3.00, 1.51, 27

-

4.17, 0.90, 28

3.44, 1.15, 27

4.25, 0.96, 28

3.88, 1.25, 27

t(49.2)=2.61, p=0.011 -

4.17, 0.98, 28

4.55, 0.84, 27

-

Finally, having guidance during the process is important (Question A.7). Especially in case of OntoGen with mean at 4.55 participants were very uncertain how to proceed. 3.3 Influence of Participants Background on Tool Evaluation We were also interested in whether participants’ background has an influence on the perception of the tool effectiveness. We have processed data independently for each tool, while calculating and comparing the means of two groups (IT and nonIT) using the 2-sample t-test. Although none of the differences between two groups were found significant there are several cases where participants’ background caused the shift in the response mean. Table 3 shows the results of this comparison. In the case of Text2Onto the only difference worth commenting on is the question A.5 where IT group felt stronger need for the visual representation (M=4.33) than the nonIT group (M=3.90). This preference was not visible in the case of OntoGen. The nonIT group found the process of obtaining ontology easier (M=3.40) in case of OntoGen than the IT group (M=2.76). However, both means are in the middle of the scale indicating neutral position on this question.


147

Table 3. Comparison of tools’ evaluation based on the participants’ background. Answers were compared separately for Text2Onto and OntoGen. 2-sample t-test did not show any statistically significant differences between IT and nonIT groups. For question text and Likert scale values refer to Table 2.

Question A.1 A.2 A.3 A.4 A.5 A.6 A.7

Text2Onto-IT (M, SD, N) 4.17, 1.07, 17 3.27, 1.07, 18 2.88, 1.02, 18 3.11, 1.07, 18 4.33, 0.76, 18 4.33, 0.97, 18 4.22, 1.00, 18

Text2Onto-nonIT (M, SD, N) 4.00, 0.94, 10 3.10, 1.10, 10 3.10, 1.10, 10 3.20, 1.13, 10 3.90, 1.10, 10 4.10, 0.99, 10 4.10, 0.99, 10

OntoGen-IT (M, SD, N) 3.80, 1.14, 15 2.23, 0.97, 17 2.29, 0.84, 17 2.76, 1.34, 17 3.47, 1.00, 17 3.76, 1.39, 17 4.41, 1.00, 17

OntoGen-nonIT (M, SD, N) 3.70, 0.94, 10 2.50, 1.17, 10 2.70, 1.05, 10 3.40, 1.77, 10 3.40, 1.42, 10 4.10, 0.99, 10 4.80, 0.42, 10

The nonIT group also preferred more guidance (M=4.80) for OntoGen than the IT group (M=4.41). As we commented above, both values indicate a serious need for guidance during the process. 3.4 Evaluation of the Produced Ontology Two evaluated tools produced ontologies of different sizes and complexity. In the second part of the questionnaire the participants were asked to evaluate the quality of the ontologies produced (see Table 4). All values were at the lower end of the scale. The participants perceived OntoGen produced significantly worse ontology than Text2Onto from the perspective how Table 4. Comparison of resulting ontologies as built using Text2Onto and OntoGen. Both IT and nonIT participants are included. Values reported are from a 5-value Likert scale: for question B.1 and B.2 the scale is 1-Completely Disagree, 2-Disagree, 3-Neutral, 4-Agree, 5Completely Agree. For question B3 the scale is 1-Not enough even for a rough description of the domain, 2-Enough for a rough description of the domain, 3-Enough for a fair description of the domain, 4-Enough for a detailed description of the domain, 5-Too many even for a detailed description of the domain. For question B4 the scale was 1-Poor, 2-Fair, 3-Good, 4-Very Good, 5-Excellent. 2-sample t-test results are shown only for significant differences between Text2Onto and OntoGen.

Question B.1 The ontology describes effectively the domain it is built for B.2 I would like to have some additional relations in the generated ontology. B.3 The number of concepts in the ontology are B.4 The quality of concepts in the ontology are (i.e. the most important concepts are included)

Text2Onto (M, SD, N) 3.10, 1.06, 28

OntoGen (M, SD, N) 2.48, 0.93, 27

3.59, 0.97, 27

3.48, 0.89, 27

2.29, 0.99, 27

3.15, 1.46, 26

2.50, 0.92, 28

2.15, 1.00, 26

2-sample t-test t(52.5)=2.31, p=0.024 t(43.8)=-2.48, p=0.016 -

148

M. Hatala et al.

effectively it describes the domain (Question B.1). However, it outperformed Text2Onto in appropriate number of concepts that describe the domain fairly rather than roughly in the case of Text2Onto (Question B.3). Both differences were statistically significant. The participants would also like to have more additional relationships in the generated ontologies (Question B.2) and the quality of generated concepts was considered fair (Question B.4). The results indicate the current tools produce rather poor ontologies that are not very usable for the type of semantic web deployment in eLearning. 3.5 Influence of Participant’s Background on Produced Ontology We have also analyzed the results for ontology produced by both tools separately to find out whether the participants’ opinions are influenced by their background. Although the average opinions between the IT and nonIT groups differ in some cases none of these differences proved statistically significant using the 2-sample t-test (see Table 5). Table 5. Evaluation of produced ontologies based on participants’ background. Answers were compared separately for Text2Onto and OntoGen. 2-sample t-test did not show any statistically significant differences between the IT and nonIT groups. For the question text and Likert scale values refer to the caption of Table 4.

Question B.1 B.2 B.3 B.4

Text2Onto-IT (M, SD, N) 3.22, 0.94, 18 3.77, 1.00, 18 2.27, 1.07, 18 2.38, 0.97, 18

Text2Onto-nonIT (M, SD, N) 2.90, 1.28, 10 3.22, 0.83, 9 2.33, 0.86, 9 2.70, 0.82, 10

OntoGen-IT (M, SD, N) 2.29, 0.91, 17 3.70, 0.77, 17 2.87, 1.45, 16 2.06, 0.85, 16

OntoGen-nonIT (M, SD, N) 2.80, 0.91, 10 3.10, 0.99, 10 3.60, 1.42, 10 2.30, 1.25, 10

For both tools (Text2Onto and OntoGen) the IT group would prefer to have more relationship identified in the ontology. In case of the ontology produced by OntoGen, the nonIT group considered the number of concepts generated to provide more detailed description than the IT group. 3.6 Qualitative Evaluation of Tools After completing the tasks with both Text2Onto and OntoGen the participants were asked to provide answers to open ended questions and judge tool intuitiveness, ease of interaction, pros and cons of each tool, and whether tools met their expectation. Based on the answers we have developed a separate coding scheme for each of the questions. Three raters tested the scheme by applying it to five randomly selected questions and fine-tuned the coding manual. In the next step, the three raters applied the scheme independently to rate the answers. In the final step all the differences were resolved through the discussion during the meeting of the three raters2. 2

We wanted to report on the interrater reliability. However, Cohen’s kappa is applicable for two raters and one category assigned per answer only. We have not found a way to compute the measure for the situation with multiple raters and multiple categories per one answer.


149

The results were evaluated using cross tables with identifying differences between tools and IT/nonIT groups. The results for each question are presented below. In the tables presenting these results for each code the percentage of answers is given for IT group, nonIT group, and total number of the code occurrence for the tool. More than one code could be assigned to a single answer. We tested for significance of the difference between the groups using chi-square statistics. If there is a significant difference in code occurrence between IT and nonIT group it is explicitly indicated in the table and the table caption and we address the issue explicitly in the discussion text. Intuitiveness of the ontology building approach. In Table 6 the four codes address intuitiveness explicitly: INT, LI, NVI, and GI. Although 25.9% of users found OntoGen to be intuitive and 37% found Text2Onto intuitive the scores for negative comments are even higher. Text2Onto is much less intuitive with over 48% of participants described it as lacking intuitiveness or not very intuitive. This compares to 26% for OntoGen. Moreover, for 14.8% participants OntoGen became gradually more intuitive with use, as opposed to 7.4% for Text2Onto. Interestingly, both tools became gradually more intuitive for the larger number of nonIT participants. Overall, we consider numbers for both tools to be very high. This indicates that in their current version the tools are not suitable for the direct use by educators without providing them with training. Additionally, participants commented on the visualization where 11.9% of participants considered OntoGen providing a good visualization. A small number of participants reported that the visualization was missing. Finally, some participants explicitly described the tool with respect to the other tool with 14.8% considering OntoGen a better tool than Text2Onto. Table 6. Intuitiveness of the ontology building approach. The codes have the following meaning: INT-intuitive, LI- lack of intuitiveness, NVI-not very intuitive, GI-gradual intuitiveness with the use, MV-missing visualization, GV-good visualization, BOT-better than the other tool.

OntoGen

Text2Onto

IT nonIT Total IT nonIT Total

INT 29.4% 20% 25.9% 35% 40% 37%

LI 10% 3.7% 35% 30% 33.3%

NVI 23.5% 20% 22.2% 17.6% 10% 14.8%

GI 11.8% 20% 14.8% 20% 7.4%

GV 5.9% 20% 11.9% -

MV 5.9% 3.7% 5.9% 3.7%

BOT 11.8% 20% 14.8% 10% 3.7%

The ease of interacting with and manipulating the tool. As can be seen in Table 7, the results were split in the middle for OntoGen where 40.7% considered it easy to use and 37% considered it not very easy to use. More nonIT participants considered OntoGen not very easy. In case of Text2Onto, 66.7% of all the participants considered it easy to use which include whooping 90% of the nonIT participants and only 52.9% of the IT participants. This difference was statistically significant with χ2 (1, N=27) = 3.89, p=0.049.

150

M. Hatala et al.

Table 7. The ease of interacting with and manipulating the tool. The codes have the following meaning: Easy- easy to use, NVE- not very easy to use, VP- hard to manipulate the visualization, LF- lack of feedback, NC- user has no control over the process. The values with star show statistically significant difference between IT and nonIT groups.

OntoGen

Text2Onto


Easy 41.2% 40% 40.7% *52.9% *90% 66.7%

NVE 29.4% 50% 37% 29.4% 10% 22.2%

VP 29.4% 10% 22.2% 20% 7.4%

LF 20% 7.4% 10% 3.7%

NC 11.8% 7.4% 5.9% 3.7%

The participants in 22% cases described OntoGen’s visualization as hard to manipulate against 7.4% in case of Text2Onto. A small number of nonIT participants found both tools lacking feedback, while number of IT participants felt they had no control of the process (11.8% for Text2Onto). Overall, the opinions on this question are split and the matter of usability of ontology building tools should be studied more carefully. An interesting pattern can be observed from this and previous question for the Text2Onto tool. With its simpler interface that hides the structural aspects of the ontology, the nonIT group have found it easy to use (90%), although 40% reported that it was not intuitive or become gradually intuitive (20%). Positive aspects of the tools. The results are presented in Table 8. Over 55% of participants thought that the biggest strength of Text2Onto is its ease of use mainly because of automatic generation of large number of concepts and relationship. Again, a significantly larger number of the nonIT participants valued this feature as positive. Only 26% of the participants described ease of use as a positive characteristic for OntoGen. However, both groups valued the visualization aspects of OntoGen with Table 8. Pros of the tools. The codes have the following meaning: Ease-Ease of use because/automatic generation of ontology, Viz- Visualization, UC- User control, Nthng-nothing good, Rnk- ranking of concepts. The values with star show statistically significant difference between IT and nonIT groups.

Ontogen

Text2Onto


Ease 23.5% 30% 25.9% *41.2% *80% 55.6%

Viz 70.6% 80% 74.1% -

UC 17.6% 30% 22.2% 11.8% 7.4%

Nthng 23.5% 10% 18.5%

Rnk 20% 7.4%


151

over 74% explicitly identifying visualization as a positive characteristic while none identified it for Text2Onto. Two observation can be made. First, the strategy for ontology building used by Text2Onto of generating a large number of candidate concepts and relationships and letting user deselect the unsuitable ones is more appealing than the elaborate incremental development approach implemented by OntoGen. Secondly, the visualization is extremely highly valued characteristic and should be provided in any useful tool for ontology building. Another reported positives were user control (22.2% for OntoGen) and ranking of concepts in Text2Onto (20% of nonIT participants). Interestingly, 18.5% of the participants (23.5% of IT) explicitly stated that there is nothing good about the tool. Negative aspects of the tools. The results are presented in Table 9. Two groups of aspects were identified. First, with respect to the process of ontology building the users identified as a problem missing elements (29.6% for OntoGen and 25.9% for Text2Onto) and too many elements generated (mainly Text2Onto, 18.9% of the participants). Interestingly, too many elements generated were a concern of the IT group only. The remaining negatives were related to the usability and robustness of the tool, with no user friendly GUI being the most prominent (33.3% for OntoGen and 18.5% for Text2Onto). The remaining negative characteristics are in Table 9. Table 9. Cons of the tools. The codes have the following meaning: nGUI- not friendly GUI, Miss- missing concepts and relationships, TME- too many generated concepts, LF-lack of feedback, LC-lack of user control, Mac-Mac incompatibility, and Crash.

OntoGen

Text2Onto


Miss 29.4% 30% 29.6% 23.5% 30% 25.9%

TME 5.9% 3.7% 29.4% 18.5%

nGUI 29.4% 40% 33.3% 23.5% 10% 18.5%

LF 11.8% 30% 18.5% 20% 7.4%

LC 17.6% 11.1% 5.9% 20% 11.1%

Mac 11.8% 7.4% 17.6% 11.1

Crash 11.8% 10% 11.1% 20% 7.4%

Meeting expectations. The results are shown in Table 10. For OntoGen the opinions of the participants were split into the three approximately same-sized groups where OntoGen met expectations for 29.6% of the participants, met them partly for 25.9%, and did not meet expectations for 29.6% of the participants. No major differences were noticed except slightly higher proportion of dissatisfied IT participants than nonIT. For Text2Onto the opinions were much more negative, especially from the IT participants. For 82.4% of the IT participants Text2Onto did not meet their expectation as opposed to 40% of the nonIT participants. This difference was statistically significant with χ2 (1, N=27) = 5.08, p=0.024. The difference for answer “Yes” was also statistically significant where Text2Onto met expectations for 30% of the nonIT participants and for no IT participant (χ2(1, N=27)=5.74, p=0.017.

152

M. Hatala et al.

Table 10. Meeting Expectations. The codes have self-evident meaning. The values with star show statistically significant difference between IT and nonIT groups.

OntoGen

Text2Onto


Yes 29.4% 30% 29.6% 30% 11.1%

No 35.3% 20% 29.6% *82.4% *40% 66.7%

Partly 23.5% 30% 25.9% 5.9% 20% 11.1%

4 Conclusions This paper presented an empirical study of educators using two ontology building tools: Text2Onto and OntoGen. The educators used the tools to build ontology from the course material they provided for the study. Twenty eight educators participated in the study between September 2008 and March 2009. The educators group came from the Computer Science/Software Engineering/Information Technology background (18 participants) and the non-Computer Science/Software Engineering background (10 participants). The results show that the current state of the tools for developing domain ontologies by educators is unsatisfactory. However, several conclusions can be made with respect to the approaches and desirable features of the tools as well as with respect to the educators background. There is an appeal of the approach which generates large amount of suggestions for ontology concepts and relationships that is than ‘weeded out’ by the user. This approach, applied by the Text2Onto tool, was especially favored by the non IT group. However, after examining the produced ontologies from the perspective of the requirements of advanced eLearning technologies their utility is rather minimal as users tend to keep extremely large number of concepts. On the other side, the interactive approach used by OntoGen produces rather small ontologies where users stop the building process too soon. Secondly, there is a clear need for a good ontology visualization capability that can be easily manipulated by the users. Finally, although some of the differences between the two groups become visible in the survey data, the results demonstrate that both groups were equally dissatisfied with the both tools. The group with IT background was more critical of the aspects of the tools where they perceived the tool did not apply strong enough methods such as eliminating unimportant results etc.

Acknowledgment We would like to thank Prof. Thomas M. Loughin who helped us define the procedure for conducting the experiment and to properly formulate and formalize the questionnaire for the study. This study is funded in part by Athabasca University’s Mission Critical Fund, Athabasca University’s Associate Vice President Research’s special project, and NSERC.


153

References 1. Devedžić, V.: Key issues in next-generation Web-based education. IEEE Transactions on Systems, Man, and Cybernetics, Part C 33(3), 339–349 (2003) 2. Dicheva, D., Aroyo, L.(eds.): Special Issue on Application of Semantic Web Technologies in E-learning. International Journal of Continuing Engineering Education and Life-Long Learning 16(1/2) (2006) 3. Naeve, A., Lytras, M.D., Nejdl, W., Balacheff, N., Hardin, J. (eds.): Special issue on Advances of the Semantic Web for e-learning: expanding learning frontiers. British Journal of Educational Technology 37(3) (2006) 4. Henze, N., Dolog, P., Nejdl, W.: Reasoning and Ontologies for Personalized E-Learning in the Semantic Web. Educational Technology & Society 7(4), 82–97 (2004) 5. Dolog, P., Nejdl, W.: Semantic Web Technologies for the Adaptive Web. The Adaptive Web, 697–719 (2007) 6. Gašević, D., Jovanović, J., Devedžić, V.: Ontology-based Annotation of Learning Object Content. Interactive Learning Environments 15(1), 1–26 (2007) 7. Jovanović, J., Gašević, D., Devedžić, V.: Ontology-based Automatic Annotation of Learning Content. International Journal on Semantic Web and Information Systems 2(2), 91– 119 (2006) 8. Jovanović, J., Knight, C., Gašević, D., Richards, G.: Ontologies for Effective Use of Context in e-Learning Settings. Educational Technology & Society 10(3), 47–59 (2007) 9. Jovanović, J., Gašević, D., Brooks, C., Devedžić, V., Hatala, M., Eap, T., Richards, G.: Using Semantic Web Technologies for the Analysis of Learning Content. IEEE Internet Computing 11(5), 45–53 (2007) 10. Stojanović, L.j., Staab, S., Studer, R.: eLearning in the Semantic Web. In: Proc. of the World Conference on the WWW and the Internet (WebNet 2001), pp. 325–334 (2003) 11. Mohan, P., Brooks, C.: Engineering a Future for Web-based Learning Objects. In: Proc. of International Conference on Web Engineering, pp. 120–123 (2003) 12. Brase, J., Nejdl, W.: Ontologies and Metadata for eLearning. In: Staab, S., Studer, R. (eds.) Handbook on Ontologies, pp. 555–574. Springer, Heidelberg (2004) 13. Cimiano, P., Völker, J., Studer, R.: Ontologies on Demand? - A Description of the Stateof-the-Art, Applications, Challenges and Trends for Ontology Learning from Text Information. Wissenschaft und Praxis 57(6-7), 315–320 (2006) 14. Cimiano, P., Voelker, J.: Text2onto - a framework for ontology learning and data-driven change discovery. In: Montoyo, A., Muńoz, R., Métais, E. (eds.) NLDB 2005. LNCS, vol. 3513, pp. 227–238. Springer, Heidelberg (2005) 15. Buitelaar, P., Olejnik, D., Sintek, M.: A Protégé Plug-In for Ontology Extraction from Text Based on Linguistic Analysis. In: Bussler, C.J., Davies, J., Fensel, D., Studer, R. (eds.) ESWS 2004. LNCS, vol. 3053, pp. 31–44. Springer, Heidelberg (2004) 16. Ding, L., Finin, T., Joshi, A., Peng, Y., Pan, R., Reddivari, P.: Search on the Semantic Web. IEEE Computer, 62–69 (October 2005) 17. Buitelaar, P.: OntoSelect: Towards the Integration of an Ontology Library, Ontology Selection and Knowledge Markup. In: Proc. of the Workshop on Knowledge Markup and Semantic Annotation (Semannot 2004) at the International Semantic Web Conference (2004) 18. Gomez- Perez, A., Manzano-Macho, D.: An overview of methods and tools for ontology learning from texts. The Knowledge Engineering Review (19), 187–212 (2004) 19. Shamsfard, M., Barforoush, A.A.: The state of the art in ontology learning: a framework for comparison. The Knowledge Engineering Review 18(4), 293–316 (2003) 20. Hatala, M., Gasevic, D., Siadaty, M., Jovanovic, J., Torniai, C.: Utility of Ontology Extraction Tools in the Hands of Educators. To appear in the Proceedings of The Third IEEE International Conference on Semantic Computing (2009)