synonyms, multilinguality, typos or different ways to write words, and tags on .... website for a specific plastic or an article about a new material, she wants to pick ...
Using the Ontology Maturing Process Model for Searching, Managing and Retrieving Resources with Semantic Technologies Simone Braun, Andreas Schmidt, Andreas Walter, Valentin Zacharias FZI Research Center for Information Technologies Information Process Engineering Haid-und-Neu-Straße 10-14, 76131 Karlsruhe, Germany {Simone.Braun|Andreas.Schmidt|Andreas.Walter|Valentin.Zacharias}@fzi.de
Abstract. Semantic technologies are very helpful in improving existing systems for searching, managing and retrieving of resources, e.g. image search, bookmarking or expert finder systems. They enhance these systems through background knowledge stored in ontologies. However, in most cases, resources in these systems change very fast. In consequence, they require a dynamic and agile change of underlying ontologies. Also, the formality of these ontologies must fit the users needs and capabilities and must be appropriate and usable. Therefore, a continuous, collaborative and work or task integrated development of these ontologies is required. In this paper, we present how these requirements occur in real world applications and how they are solved and implemented using our Ontology Maturing Process Model.
1
Introduction
So far the potential of semantic annotation approaches has not been realized in practice; semantic annotation systems are still restricted to academia. At the same time very simple and limited tag based annotation approaches have emerged on the internet and found wide usage: proving both the users need for annotation approaches and their principal willingness to perform manual annotations. Our work and this paper is based on the assumption that the failure of semantic annotation approaches can be traced to the misunderstanding of ontologies in the system as relatively fixed, expert maintained artifacts. We believe that semantic annotation approaches can only show their true potential by understanding the ontologies as artifacts that are permanently, rapidly and simply adapted by the users of the system to their task and changing domain; indeed we believe that understanding the model of the system as object of user’s action will emerge as the defining criteria for semantic applications in general. To realize this vision, we extended our Ontology Maturing Process Model [1] and implemented two applications (SOBOLEO and ImageNotion) that support (parts of) this model for the domains of web and image annotation. We have already conducted multiple evaluations that were also used to refine the ontology maturing process model. The next section of this paper describes the Ontology Maturing process model within two use cases, focusing on the need to view the artifact, knowledge, and social
dimension of ontology maturing as separate. The third section, then, gives an overview of four evaluations and the lessons learned. Finally related work is introduced before the paper concludes.
2
Collaborative and Work-integrated Ontology Development
With the Web 2.0 social tagging applications for managing, searching, and finding resources by dint of arbitrary tags found wide usage. However, problems such as homonyms, synonyms, multilinguality, typos or different ways to write words, and tags on different levels of abstraction hamper search and retrieval in these applications [1,2,3]. On the other hand, current Semantic annotation approaches avoid these problems, but usually don’t allow to quickly and continously adapt the ontology, often resulting in unsatisfied users being confronted with out-of-date, incomplete, inaccurate and incomprehensive ontologies that they cannot easily use for annotation [4,5]. To a large extend because the annotation process, i.e. the usage of the ontology, and the creation of the ontology are two separate processes, performed by a different set of people [6]. The goal of our work, then, is the combination of the benefits of social tagging with those of semantic annotation in order to address their respective weaknesses. Starting with simple tags, each user shall contribute to the collaborative development of ontologies. For this purpose, we integrate the creation process of ontologies into their usage process, e.g. search and annotation processes. Each community member can contribute new ideas (tags) emerging from the usage to the development of ontologies. The community picks them up, consolidates them, refines them, and formalizes them with semantic relations towards lightweight ontologies. 2.1
The Ontology Maturing Process Model
To operationalize this view, we have developed the ontology maturing process model that structures the ontology engineering process into four phases (for details of the complete model, please refer to [1]). An overview of this process model process is shown in figure 1. In phase 1 Emergence of ideas, new ideas emerge and are introduced by individuals as new concept ideas or informal tags. These are ad-hoc and not well-defined, rather descriptive, e.g. with a text label. They are individually used and informally communicated. Phase 2 is the Consolidation in Communities, through the collaborative (re-) usage of the concept symbols (tags) within the community, a common vocabulary (or folksonomy) develops. The emerging vocabulary, which is shared among the community members, is still without formal semantics. Formalization happens within the third phase, when the community begins to organize the concepts into relations. This results in lightweight ontologies that rely primarily on inferencing based on subconcept relations. In the fourth, the Axiomatization phase, the ontologies are extended with axioms to allow for more powerful inferences. It is important to note that ontology maturing does not assume that ontologies are built from scratch, but can be equally applied to already existent core ontologies used for community seeding. Likewise, this model must not be misunderstood as a strictly
linear process. Usually individually used tags, common but not yet formal terminologies as well as formally defined concepts coexist at any moment. 2.2
The Artifact, Knowledge and Social Dimensions of Ontology Maturing
Our evaluation sessions (which are described in section 3) have shown, that concentrating on the development of the ontology as a mediating artifact is not sufficient to prepare for sustainable community-driven semantic applications. Beyond the mere construction of an artifact, we have to consider that users have different levels of understanding of parts of the domain (e.g. identified by interest in background knowledge to improve their own understanding, asking for help, or taking the lead within a group) and that this understanding also evolves within usage processes. Furthermore, the social dimension of community-driven sites has to be addressed, e.g., which instruments are needed to support a growing community. As a consequence, we need to describe ontology maturing in three different dimensions, the artifacts, knowledge and social dimensions (see Fig. 1).
Fig. 1. View on the extended ontology maturing process model
The artifact dimension is concerned with the created ontology elements, the knowledge dimension with the maturing and alignment of knowledge, and the social dimension with the development of competencies and social structures. Artifacts are “something viewed as a product of human conception”. In folksonomies, tags are the product of human conception. In semantic applications, ontologies are considered as a product of formalized human conception. Using our ontology maturing model, artifacts mature from simple tags to formalized or even axiomatized ontology elements as described in the previous section. Thus, the artifact dimension identifies the available ontology elements and their relations. This dimension has (naturally) been the focus of semantic technology research so far. Users can only model appropriately what they have sufficiently understood, and the process of modeling usually involves a deepening of the understanding of the realworld topic. Within the knowledge dimension, we need to distinguish between individual knowledge and the abstraction of collective knowledge. On the level of the individual, we need to consider alignment processes that bring forth a sufficient level of shared understanding of the domain and learning processes on the methods to create artifacts (modeling competencies). On the collective level, this is about the development of an understanding as such. Viewing ontology development as collaborative learning processes, e.g. interaction, communication and coordination among the individuals, we have to consider the social structures and processes in the social dimension. Users can only build a shared understanding, shared artifacts and methods to create these if they learn to collaborate on the individual as well as on the collective level. Learning on the individual level comprise a general willingness and competencies to interact with others, communicate, negotiate, compromise and accept rules. 2.3
Use Cases & Tool Support
In the following, we present two use cases for the ontology maturing process and their support with our tools ImageNotion [7,6] and SOBOLEO [8]. Semantic Image Annotation and Search This use case concerns the management and retrieval of images with the use of semantic annotations. Users, e.g. of an image archive, uploading images can use available elements from the ontology (e.g. via an ontology browser) to create semantic image annotations. In cases where a user is missing elements from the ontology, however, she can also create them directly integrated into the image uploading process. These newly created ontology elements may also be created with vague information that is then later refined by the community. Then, other users, e.g. image buyers can benefit from these semantic annotations when they perform semantic image search request, e.g. by searching for ”all French generals who participated in the WWI” in an image archive with historical images. During the use of semantic annotations in this domain a number of interesting phenomena occured that go beyond the simple creation of artifacts in an ontology, but nevertheless influence it. We identified these phenomena during previous evaluations, e.g.
[6]. Image annotators have a big interest improving their background knowledge about a domain, e.g. by reading Wikipedia articles. In addition, they take the image searcher as their main focus – and try to homogenize the created artifacts with the knowledge of a user. For instance, when they expect searchers to be experts, they annotate images using very specific or even scientific annotations; but they use very easy annotations when the targeted users are private users. Another effect we identified in this use case was that individual users, who are experts of a given topic, take the lead in images concerning this domain. Then, in discussions about image annotations for these images, this user is asked to solve conflicts. ImageNotion: An imagenotion (formed from the words image and notion) graphically represents a semantic notion with the help of an image. The associated methodology (based on section 2) consists of three different steps. Step 1 is the creation of new imagenotions, Step 2 is the consolidation of imagenotions in communities and Step 3 is the formalization of imagenotions by defining creation rules (such as naming conventions) and relations. Imagenotions from each maturing grade may be used for semantic image annotations. In the ImageNotion application, imagenotions are used for the semantic image annotation instead of textual tags as in traditional image archives. For instance, for creating the semantic element representing the current president of the European Commission “Manuel Barrosso” with that to annotate then images showing Manuel Barroso, one user may have created the imagenotion “Manuel Barroso” and selected an image showing him as representing image. In addition, she gave this imagenotion a main label. Some other member of the group added an alternative label text, the full name of Barroso which is “Jos´e Manuel Durˇao Barroso”, as well as his birthday, 1956-03-23, and another member added relations to the other imagenotions “European Commission” and “Portugal”. All in all, they created and matured the descriptive and visual information of this imagenotion. Semantic Annotation and Search of Web Pages This use case is taken from the German research project ”Im Wissensnetz”1 (”In the Knowledge Web – linked information processes in research networks”), which aims to support researcher from various disciplines within e-Science. One major problem is searching and retrieving adequate up-to-date resources in the internet. The dynamic of the domain is a particulat challenge in this project, e.g. the area of plastics new materials or new forms of existing ones frequently enter the market; brand names and manufacturers are permanently changing and hardly traceable – attributes of a chemical substance retrievable using its brand name today, are very hard to find once it’s sold under a different label. In this use case, the users want to have a tool to collaboratively collect and semantically annotate web pages. Thus, when one user finds a web page, e.g. the manufacturer’s website for a specific plastic or an article about a new material, she wants to pick it up into a shared bookmark collection and semantically annotate it, e.g. with the specific plastic. If the needed concept does not exist in the ontology (e.g. in the case of a new material) or is not suitable (e.g. when the brand name changed), the user wants to immediately modify an existing concept (e.g. extending a concept with the new brand name) or add arbitrary tags (e.g. a new material) while annotating the web page. Sometimes, 1
http://www.im-wissensnetz.de
users start with vague information (e.g. because it is a new method or technique) that is then later consolidated and refined within the community. When searching for resources, e.g. with the former brand name, the search engine should make use of the underlying ontology and further provide search relaxation or refinement in order to reduce irrelevant results and to guide the user. Inadequacies of the ontology or the annotations can also be corrected right during the search process. SOBOLEO: SOBOLEO is a web-based system that supports people working in a certain domain in the collaborative development of a shared bookmark collection and of a shared ontology that is used to organize the bookmarks. That means, collected bookmarks can be annotated with concepts from the ontology and the ontology can be changed at the same time. If users encounter a web resource, they can add it to the bookmark collection and annotate it with concepts from the SKOS ontology [9] for better later retrieval. If a needed concept does not exist in the underlying ontology or is not suitable, the users can modify an existing concept or use arbitrary tags, which are automatically added as ”prototypical concepts” to the ontology. In this way, new concept ideas are seamlessly gathered when occurring (maturing phase 1) and existing ones are refined or corrected (maturing phase 2). The users can structure the concepts with hierarchical relations (broader and narrower) or indicate that they are “related”. These relations are also considered by the semantic search engine and for navigation support within the bookmark collection. That means, the users can improve the retrieval and exploration of their annotated web pages by adding and refining ontology structures (maturing phase 3).
3
Evaluation
The goal of the evaluation was to show that (a) our showcase semantic applications are accepted by end users and that (b) key assumptions of our ontology maturing model (and the derived applications) are true. For (a), we need to show that these applications are perceived as useful and usable, both the annotation and search as well as the ontology editing part. For (b), we want to show that ontology maturing actually occurs in collaboration between different users of the system and that we can observe the proposed phases. For conducting the evaluations, we have applied a formative usability evaluation methodology that is also geared towards eliciting new requirements. 3.1
Evaluation Sessions
The first evaluation (S1) of SOBOLEO was an online evaluation held during the the Workshop on Social and Collaborative Construction of Structured Knowledge held at the 16th International World Wide Web Conference. The participants added in total 202 new concepts and 393 concept relations to the ontology. Further, they collected 155 web resources, which they annotated with 3 concepts per resource on average. The second evaluation of SOBOLEO (S2) took place within the scope of the project “Im Wissensnetz”. Within two one-hour sessions, four users had to carry out specific tasks simulating the usage of SOBOLEO within their daily work activities. Half of the users were researchers of the rapid prototyping domain and half of them patent experts
for German research. All of them were unexperienced in ontology development. We provided a basic ontology with 31 concepts to start with that was thematically tailored to the rapid prototyping domain. The tasks were tailored to gain orientation within the ontology by letting the users place or add synonyms to existing concepts. Thus, the users added 6 concepts to the ontology, 11 synonyms and 21 concept relations. For the ImageNotion application we conducted an online survey (I1) with 137 participants. We were interested how individual users would create an inital version of a semantic element for Manuel Barroso, the current president of the European Commission. Users may then use this imagenotion for the semantic annotation of images showing Barroso. In addition, we executed a workshop I2 where three groups of six people participated. The groups were recruited from different communities: from Wikipedia users, from employees of French image agencies and from Italian history students. They had to perform tasks for the semantic annotation of images. They started with a small ontology; in its core based on CIDOC-CRM [10].
3.2
User Acceptance and Usefulness
The evaluations with users from different background showed that users appreciate both applications (according to DIN EN ISO 9241 & 13407): S1/I2 The users liked the ease of use of the ontology editor (in comparison to other, more heavy-weight applications) and particularly enjoyed the simple way of annotating resources with concepts or tags, which are then automatically added. Thus, to have the possibility to integrate not yet well defined concepts but something like ”starter concepts“ and, in this way, to ”get the ontology building almost for free“. S2 Although the users came from a non-IT background, they appreciated SOBOLEO for its ease of use. Some of the users had some problems at the beginning due to their very basic knowledge in ontologies, but were able to obtain the necessary skills within the evaluation sessions. I1 The rate of success for completing the tasks in the online survey for the ImageNotion application has shown that people from a variety of backgrounds are able to understand and interact with a semantic image search and annotation application without prior training. Furthermore, it turned out that the applications were actually used as collaborative applications by contributing to the construction of a shared ontology. Due to the more open setting of the evaluation in (S1) and (S2), this was more visible in the evaluation of SOBOLEO. Particularly, in (S2), the chat turned out to be an essential utility for simultaneous working. For instance, two users had problems in placing concepts in the given ontology because they had only basic knowledge of the rapid prototyping domain. In consequence, they began to ask their colleagues for help via the integrated chat functionality. Nevertheless, the chat appeared to be too simple. For improvement, the users wished to have a better integration of what is discussed and where the changes are done.
3.3
Validation of the Ontology Maturing Model and its Implications
The validation of the ontology maturing model was the particular focus of the evaluation (I1) of the ImageNotion application. Emergence of Ideas Users were asked to state descriptive information for this politician. The most frequently mentioned labels were two different versions of his name: “Manuel Barroso” and “Barroso”. In addition, further version of his name and his profession “politician” were entered. For the alternative label, most people chose “politician”. In terms of semantics, this may already be seen as specifying a semantic element. “Barroso” was the second most frequent alternative label, while on the third place we got the full name of Manuel Barroso, “Jose Manuel Durao Barroso”. I.e., the mostly used tags for searching for Manuel Barroso are his name and his profession, followed by different spellings of the name and finally semantic elements such as “EU” or “person”. This is a very motivating result for us, because it shows that people in general not only think in terms of tags but also consider semantically relevant aspects. In case, users had written all these information directly in the ImageNotion application, the community would have created collaboratively a semantic element for Manuel Barroso with very detailed information. Since most users were not ontology experts, this is a very promising result. Consolidation of Artifacts The next task concerned the consolidation of artifacts. We have already shown that consolidation of artifacts in communities in other evaluations [6] and that training sessions helps in deepen the knowledge of users in understanding the general meanings of ontologies. The focus of this task was, how artifacts mature by considering the knowledge dimension. Therefore, users were first asked to deepen their knowledge about an artifact before maturing it. Then, they were asked, what kind of additional information they would state about Manuel Barroso. Therefore, they were asked to read the Wikipedia article about Manuel Barroso. The users added many more detailed information about Manuel Barroso, e.g. that he was born in Portugal, was Prime Minister of Portugal, or studied law at the University of Lisabon. All together, this shows that the consideration of the knowledge dimension improves the creation of artifacts with a high quality, because of a matured background knowledge of a domain. Formalization With the scope on the maturing of ontologies, we finally evaluated whether users would like to create relations beyond broader, narrower, and unnamed relations (referring to the formalization phase). Therefore, we asked the participants, what kind of named relations they would use for the relations they created. Users suggested specific names for relations such as “is president of” (24%), “works for” (8%), and “has nationality” (6%). With 84%, most of the participants thought that relations are important for semantic image search. Since more than 60% of the users stated that they had low or little knowledge about semantic technologies, this is a promising result for creating semantic systems for the managing of resources. Users not only understood the meaning of semantic elements, they also requested the creation of named relations.
3.4
Evaluation of the Artifact, Knowledge and Social Dimensions
The following observations base on the evaluations S1, S2 and I2. They show and explain, how the extended ontology model and the modeled artifact, knowledge, and social dimensions occur and work together. Mutual Support Some users did not know at the beginning how to use semantic elements, although they had an introduction before the evaluation started. That means, their personal expertise and knowledge was too immature in order to act on the ontology. In consequence, these users began to ask for help. In turn, one of the other users stated and shared his expertise and answered questions that allowed the other users for participating in the annotation work. Explanation: Mutual support starts in the social dimension with the general willingness of an individual to participate. Because of her incomplete knowledge in the knowledge dimension, a user recognizes that she can not fulfill her desired tasks in a collaborative application. As a consequence, she communicates with other participants that are willing to share their knowledge with her. With matured knowledge, the user can better participate on the communities collaborative work tasks. Homogenization One very interesting point were homogenization processes. In the evaluation I2, one part of the participants had the role of image annotators (because of their daily work in professional image archives) who are interested in creating annotations so that image searchers can retrieve them as easy as possible. In contrast, the other group’s role was that of image searchers who directly identified the ontology artifacts they would use for searching. In communication, they became aware of and adapted to the ”other side’s” likely use of the ontology elements. This means that even when a participant had complex knowledge about a domain, he cared about the usage of artifacts matching the need of others. In consequence, all participant homogenized their view on commonly shared knowledge to optimize the retrieval quality for the annotated resources in collaborative work. Explanation: Starting point for the homogenization phenomenon is the personal willingness of each participant in the social dimension to integrate himself into the community. This interacts with the knowledge dimension, because all participants need to align their knowledge to achieve a commonly accepted, shared understanding. Interest in Background Knowledge In all three evaluations, users read external resources (mostly Wikipedia articles) and used the new background information for their artifacts and added descriptive information (e.g., birthday of a person) or relations (e.g. relations to specific events). Explanation: The interest in background knowledge is a proof for the willingness of an individual in the knowledge dimension that bases on the general willing to integrate itself in the community in the social dimension. As a consequence, this first influences the knowledge dimension because of improved support for each users’ interest to learn and alignment of knowledge, but also has an impact on the maturing of the ontology elements that reflect the new background knowledge.
4
Related Work
Our related work section is focused on ontology development methodologies and tools which allow for a collaborative and work integrated ontologies engineering processes. With the Human-Centered Ontology Engineering Methodology (HCOME) Kotis et al. [11] view ontology development as a dynamic, human-centered process and focus particularly on ontology evolution. They assume a decentralized engineering model where everyone first formalizes her own ontology and shares it in a further step within the community. There, the individual ontologies are merged or further developed. However, findings in [12] (based on action theory) suggest that collaboration plays a more important role before we have formalized (individual) ontologies. So we think that the HCOME methodology can benefit from incorporating the notion of different maturity levels. Gibson et al. [13] involve domain experts in an early, less-formal stage of the ontology development process and support the communication with a Web 2.0 user interface. However, they also assume knowledge engineers to do the modeling task and not the domain experts doing it by themselves. A similar approach to ours is proposed by Siorpaes et al. [14]. They also conceive ontology building as a community-driven evolution process and use a wiki system as enabling technology. Wiki systems consider the aspects of collaboration and can support the early phases of ontology construction. Semantic wiki systems2 try to extend the traditional wikis with semantic web technologies. These systems help users in creating definitions, e.g. beginning with informal texts. Because of discussion pages and versioning for each article they are suitable for complex coordination and consolidation processes. All of these methodologies and tools lack in possibilities for an integration of the ontology development in work processes.
5
Conclusions and Future Work
We have argued that the lack of acceptance of semantic applications in the large is due to the static and expert-based view of ontology engineering as separated from the use of the ontology (e.g. for annotation and search). In order to overcome these problems, we built on the success of Web 2.0 tagging approaches and combine these with ontologybased approaches, with the help of the ontology maturing model. To show the usefulness of this model, we have created two applications, SOBOLEO and ImageNotion. In four evaluation sessions we have shown that such tools are perceived useful and usable by users from a variety of backgrounds without prior training. Furthermore, these evaluations sessions have provided evidence that ordinary users are willing and able to engage in maturing activities for an ontology and that the development of a shared vocabulary takes place according to the ontology maturing theory. Further, more long-term evaluations will have to take place to show that such applications allow for overcoming the time lag problem of controlled vocabularies/ontologies. On the methodological side, we will try to derive a methodological framework for engineering maturing-aware applications beyond the two showcases from the ontology 2
e.g. SemanticWiki Interest Group (http://semanticweb.org/wiki/Semantic Wiki State Of The Art)
maturing model. This framework will be realized and evaluated in the next generation of SOBOLEO and ImageNotion. Another route of development is to investigate more advanced support tools that take into account the different dimensions; like (visual) analysis tools of activities, or suggestions for consolidation that further ease the ontology construction task, particularly in larger user groups. This will take place within the context of the MATURE project3 .
References 1. Braun, S., Schmidt, A., Walter, A., Nagypal, G., Zacharias, V.: Ontology Maturing: a Collaborative Web 2.0 Approach to Ontology Engineering. In: Proc. of the Workshop on Social & Collaborative Construction of Structured Knowledge, CEUR Workshop Proc., vol. 273. (2007) 2. Golder, S., Huberman, B.A.: The Structure of Collaborative Tagging Systems. Journal of Information Sciences 32 (2006) 198–208 3. Guy, M., Tonkin, E.: Folksonomies: Tidying up tags? D-Lib Magazine 12 (2006) 4. Hepp, M.: Possible Ontologies: How Reality Constraints Building Relevant Ontologies. IEEE Internet Computing 11 (2007) 90–96 5. Barker, K., Chaudhri, V.K., Chaw, S.Y., Clark, P., Fan, J., Israel, D., Mishra, S., Porter, B.W., Romero, P., Tecuci, D., Yeh, P.Z.: A Question-Answering System for AP Chemistry: Assessing KR&R Technologies. In: Proc. of the Int. Conf. on Principles of Knowledge Representation and Reasoning. (2004) 488–497 6. Walter, A., Nagypal, G.: ImageNotion - Methodology, Tool Support and Evaluation. In: OTM Confederated Int. Conf. CoopIS, DOA, and ODBASE, Springer LNCS. (2007) 7. Walter, A., Nagypal, G.: IMAGENOTION - Collaborative Semantic Annotation of Images and Image Parts and Work Integrated Creation of Ontologies. In: Proc. of 1st Conference on Social Semantic Web, Springer LNCS. (2007) 8. Zacharias, V., Braun, S.: SOBOLEO -– Social Bookmarking and Lighweight Engineering of Ontologies. In: Proc. of the Workshop on Social & Collaborative Construction of Structured Knowledge, CEUR Workshop Proc., vol. 273. (2007) 9. Miles, A., Bechhofer, S.: SKOS Simple Knowledge Organization System Reference. W3C Working Draft 25 January 2008, W3C (2008) 10. Crofts, N., Doerr, M., Gill, T., Stead, S., Stiff, M.: Definition of the cidoc conceptual reference model version 4.2. In: CIDOC CRM Special Interest Group. (2005) 11. Kotis, K., Vouros, G.A., Alonso, J.P.: HCOME: A Tool-Supported Methodology for Engineering Living Ontologies. In: 2nd Int. Workshop on Semantic Web and Databases, Springer LNCS. (2004) 155–166 12. Allert, H., Markannen, H., Richter, C.: Rethinking the Use of Ontologies in Learning. In: Proc. of the 2nd Int. Workshop on Learner-Oriented Knowledge Management and KMOriented Learning. (2006) 115–125 13. Gibson, A., Wolstencroft, K., Stevens, R.: Promotion of ontological comprehension: Exposing terms and metadata with web 2.0. In: Proc. of the Workshop on Social & Collaborative Construction of Structured Knowledge, CEUR Workshop Proc., vol. 273. (2007) 14. Siorpaes, K., Hepp, M.: myontology: The marriage of ontology engineering and collective intelligence. In: Bridging the Gep between Semantic Web and Web 2.0 (SemNet 2007). (2007) 127–138 3
http://mature-ip.eu
