Information Systems & Semantic Web University of Koblenz ▪ Landau, Germany
Semantic Multimedia Management - Multimedia on the Semantic Desktop -
Thomas Franz, Steffen Staab, Richard Arndt
A Common Scenario ...
“Do you have any recent paper about semantic desktop developments?“
B
“I attached a paper about a new ontology and integration approach ...”
K-
A
p a C
“C, this paper might be interesting for you“
C x-cosim.pdf
x-cosim.pdf
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 2
… Contextualized Information
K-
B
x-cosim.pdf
“I attached a paper about a new ontology and integration approach ...”
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 3
p a C
… Classifying Information K-Cap
A
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 4
… Communicating
B
A VoIP
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 5
Goal Definition
Integration Automation
Information linkage based on RDF No shared semantics for information dealt with across desktop applications Cross-tool information exchange by 1:1 mappings only
Locked down information that cannot easily be shared and linked across applications Redundant information management Inconsistent data talk s i Th High cognitive effort
Semantic Desktops
Scalable solution for mapping application-dependent data models Extensibility and reuseability to new applications Interoperability, rich axiomatization, clear defined meaning Modularization
X-COSIM
talk s i Th
Legacy Desktops Legacy Desktops ISWeb - Information Systems & Semantic Web
Semantic Desktop Prototypes Steffen Staab
[email protected]
Semantic Desktop 6
Future/Desired Situation
Implicit Models and Data Formats
Semantics of data hidden in applications` source code Data formats do not define the semantics of what is expressed by them
? ?
ISWeb - Information Systems & Semantic Web
??
Steffen Staab
[email protected]
Semantic Desktop 7
Narrow Conceptualizations
Attachment == Page ? Bookmark == Folder ? Email == File ?
Email
Page
User File
Folder Attachment
Form
Bookmark
Access Sender
Dir
Recipient
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 8
Image
Narrow Conceptualizations
One-to-one conversion of data models to RDF does not resolve issues of conflicting conceptualizations
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 9
X-COSIMO: X-COSIM Ontology
High quality reference model that describes information dealt with across desktop applications Uses RDF as flexible data structure that enables information linkage Formulated in OWL
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 10
How to Design a High Quality Ontology?
Approach from [Oberle, 2005], [Oberle et al., 2006]: Use a well designed foundational ontology as a modelling basis to avoid shortcomings Foundational ontologies provide Formal precision Domain independence Broad scope
Building upon foundational ontologies Prevents easy inclusion of modeling artefacts Reduces conceptual ambiguity Inherit rich axiomatization
DOLCE + Ontology of Information Objects (OIO) ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 11
Methodology
X-COSIM
Narrow Conceptualizations Missing Semantics
Semantic Data Model + Foundational Ontology Legend Challenge
Legacy Desktops ISWeb - Information Systems & Semantic Web
Building Block Steffen Staab
[email protected]
Semantic Desktop 12
Capturing Contextualized Information
Foundational Ontology
ContextDependent Model ISWeb - Information Systems & Semantic Web
Email
Folder
Sender Steffen Staab
[email protected]
Attachment
Recipient
Semantic Desktop 13
X-COSIMO Approach: Representing Aspects
Identify „core“ notion of context-dependent concepts Represent them by (sub-)concepts of the foundational ontology Represent domain-dependent conceptualizations wrt context Descriptions & Situations (D&S): Formalization of Descriptions which represent Context
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 14
… X-COSIMO Approach
Foundational Ontology Communication
Email
ContextDependent Model ISWeb - Information Systems & Semantic Web
Sender
Steffen Staab
[email protected]
Information Object Attachment
Recipient
Semantic Desktop 15
Agent
X-COSIMO Walkthrough
Communication-Description defines
Addresser satisfies
Addressee
Attachment
plays
plays
Agent
setting
Communication-Situation
Information-Object
setting
Semantic-Annotation
Particular/Thing plays
plays satisfies
Semantic-Label-Role Method
Annotated-Data-Role
defines
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 16
Methodology
X-COSIM
Contextualized Information Narrow Conceptualizations Missing Semantics
Representation of Context
Semantic Data Model + Foundational Ontology Legend Challenge
Legacy Desktops ISWeb - Information Systems & Semantic Web
Building Block Steffen Staab
[email protected]
Semantic Desktop 17
Communication Module
Classification of DOLCE ground entities, mainly by roleconcepts defined in a description Concepts: Addresser, Addressee, Message, … Ground Entities: Agent, Information Object, …
Description
Role
Course Email Description
Communication Description
defines defines Communication Role
Address
Addresser
Addressee
Contact
Message
modaltarget
defines
Body
Instant Message
Attachment
Transferred File
Communication Course
satisfies DispatchTime
IM Description
Email Protocol
IM Protocol
sequences played-by
played-by
Information Object Communication Situation
Agent
played-by
played-by
Description
Information Object
participant-in
Communication Event
participant-in
setting
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 18
realizes Digital Realization
Realization Module
Describing the realization of information objects Distinction between Access-Protocol (e.g. IMAP, File System, HTTP) Identification (e.g. DB-Key, URL, File Path) Serialization (e.g. CSV, PDF, JPEG) Role
Description defines Realization Description Information Object
Protocol
satisfies realizes
Information Realization
Identifier
Format
played-by
played-by
played-by Particular
Protocol Description
Digital Realization setting
Realizer
LDAP IMAP
ISWeb - Information Systems & Semantic Web
ODBC HTTP
XML Posix
Steffen Staab
[email protected]
MBOX
Format Description
Text Format
CSV RTF
Semantic Desktop 19
JPEG ODF
Binary Format
PDF
Executable
Aspects Supported by X-COSIM Modules
ns io
D es cr
at
ip t
tu
D cr
at io ns
es t ip io ns
at ns io
Steffen Staab
[email protected]
ip t
tu
ISWeb - Information Systems & Semantic Web
D es cr
Si
an d
d an
Si tu
ns io
Structuring, Summarizing, User-defined Ontologies
Semantic Desktop 20
Creating, Reusing, Copy & Paste
DOLCE for Developers of E-mail Clients?
The X-COSIM API (X-COSIMA)
… // Create a conversation Conversation conversation = new Conversation(); // bikeImage transferred as email attachment between thomas and // Carsten within the created conversation Email email = new Email(conversation, personA, addressOfA, personB, addressOfB); email.setBodyText("Hi B, I attached a paper about … "); …
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 22
Methodology
X-COSIM Intuitive, context aware development interface
Contextualized Information Narrow Conceptualizations Missing Semantics
Conceptual Architecture
Representation of Context
Semantic Data Model + Foundational Ontology Legend Challenge
Legacy Desktops ISWeb - Information Systems & Semantic Web
Building Block Steffen Staab
[email protected]
Semantic Desktop 23
Resulting Meta Data
Communication Aspect
B Classification Aspect
ISWeb - Information Systems & Semantic Web
Realization Aspects
A
K-Cap
Steffen Staab
[email protected]
Semantic Desktop 24
Example: From File to Conversation
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 25
State of Work
Tools: Instant Messenger Thunderbird Plugin (in progress) Extensions for the Konqueror (KDE) File Manager Viewer (LENA) http://isweb.uni-koblenz.de/Research/lena Extension for Tasks, Goals, Plans in progress X-COSIM to be applied in the EU integrated project X-COSIMO and X-COSIMA can be downloaded from http://isweb.uni-koblenz.de/Research/x-cosim
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 26
Thank You.
This work was funded by the X-Media project (www.x-mediaproject.org) sponsored by the European Commission as part of the Information Society Technologies (IST) programme under EC grant number IST-FP6-026978.
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 27
Ontology of Information Objects (OIO)
social-object
information-encoding-system ordered-by
interpretedBy information-object
description expressedBy
about agent
particular
satisfies
realized-by information-realization
situation setting
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 28
Methodology
X-COSIM
Missing Semantics
Semantic Data Model
Legend Barrier
Legacy Desktops ISWeb - Information Systems & Semantic Web
Resolution Steffen Staab
[email protected]
Semantic Desktop 29
Capturing Contextualized Information
Contextual information, e.g. the fact that some information can be considered as email a person can occur as sender of an email … should be captured within the reference model Descriptions & Situations (D&S): Formalization of Descriptions which represent Context Descriptions classify DOLCE ground entities within a Situation
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 30
Descriptions and Situations
Formalization of descriptions which represent context. Descriptions define Parameters, Roles, and Courses that are used to classify DOLCE ground entities within a Situation
defines
Parameter
expressed-by Description Information Object
Region
satisfies
valued-by Concept
Role
Endurant played-by
Course
ISWeb - Information Systems & Semantic Web
sequences
Situation setting
Perdurant
Steffen Staab
[email protected]
Semantic Desktop 31
Social Object requires
X-COSIMO Modules
Modularized, distinguishing between core aspects of information relevant for PIM Communication (Sharing, Arguing, Explaining) Information Realization (Filing, Retrieving, Modifying) Classification (Structuring, Summarizing) (De)composition (Creating, Reusing)
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 32
X-COSIM Architecture
Apps, Domain Models
Data Access, Mapping
Reference Model, Storage
File Manager, Browser, Mail Client, Task Manager, Text Processor, ... Dublin Core, Calendar, Foaf, ...
X-COSIMA RDF API
SPARQL
X-COSIMO RDF Store
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Applications that build upon contextualized information representations Mapping between contextualized representations and the reference model Abstract information representation (reference model) and storage
Semantic Desktop 33
Methodology
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 34
Realization Pattern
e.g. playing the role of an email attachment
c:\pics\bikeImage.jpg
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 35
Communication Pattern
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 36
Sample Scenario
Thomas
c:\my pics\bikeImage.jpg
Carsten
file:/home/carsten/pics/bikeImage.jp g
RedBike 2-cylinderengine
http://www.xmedia.org/pics/bike.png
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 37
Code Sample ... // Create some addresses and agents carsten and thomas Address addressCarsten = new Address("
[email protected]"); Agent carsten = new Agent(addressCarsten); Address addressThomas = new Address("
[email protected]"); Agent thomas = new Agent(addressThomas); // Create an information object representing the information given by some image InformationObject bikeImage = new InformationObject("bikeImage"); // Create realization metadata for bikeImage: // Storage on harddisk, Format jpeg FileSystemRealization bikeImageRealization = new FileSystemRealization( bikeImage, new File(new URI("file:/myPics/bikeImage.jpg")), RealModule.I_jpeg); // Create a conversation Conversation conversation = new Conversation(); // bikeImage was transferred as email attachment between thomas and // Carsten within the created conversation Email email = new Email(conversation, thomas, addressThomas, carsten, addressCarsten); email.addAttachment(bikeImageRealization); email.setBodyText("Hi Carsten, I attached some image of my new bike :)"); ...
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 38
Code Sample ...
// Create a COMMLite content object from the email so that it can be annotated using COMMLite ASCIIContent messageText = new ASCIIContent( carstenReply.getMessageIO().getResource().getJavaUri(), messageContentLocation); // Create an ASCIILocalization starting at char 21 and ending at // character 21+9. This should locate the string "nice bike" in the email body ASCIILocalization niceBike=new ASCIILocalization(21, 9, messageContentLocation); // annotate the part of emailText specified by the niceBike localization with the bikeImage messageText.addAnnotation( bikeImage.getResource().getJavaUri(), niceBike); // Carsten annotates the image itself ImageContent ic = new ImageContent(carstensRealization.getRealizedInformation() .getResource().getJavaUri(), carstensRealization.toContentLocalization()); // Carsten annotates the image as a whole with 'SportsBike' ic.addAnnotation(new URI("http://myBikeOntology/RedBike")); // Carsten annotates a particular polygon with '4-cylinder-engine' ic.addAnnotation(new URI("http://myBikeOntology/4-cylinder-engine"), new StillRegionLocalization( new Rectangle(140, 120), carstensRealization.toContentLocalization())); ...
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 39
Requirements Reusability Design a core ontology for any desktop application Extensibility Enable inclusion of further • tools • contexts Separation of Concerns Clear separation of domain knowledge and knowledge about structure Modularity Enable customization High degree of axiomatization Ensure interoperability through machine accessible semantics
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Text Task Manager Processor Music Photo Manager Manager
Semantic Compound Annotation Document Text Descriptor
Semantic Desktop 40
DOLCE Design Patterns: OIO
DOLCE is a library of foundational ontologies that provides 2 design patterns (extensions) that are especially important for MPEG-7: Ontology of Information objects (OIO): Formalization of information exchange Information object represents pure abstract information (message) Relevance for X-COSIMO: • Desktop applications deal with information objects, e.g. messages, files, web pages
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 41
Example
Information Object „Graz Tourist Guide“ Information Realization
http://cms.graztourismus.at/cms/ziel/42425/EN/
Booklet Information Encoding: English German About: Places, Buildings (e.g. Clock Tower) Agent: 1. iMedia Visitor / 2. Tourist Officer / 3. Graphics Designer Expresses: 1. Walking Path through Graz 2. Small-Size Tourist Guide 3. Arrangement of Illustrations
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 42
Based on DOLCE-Lite-Plus Ontology of Information Objects (OIO) Descriptions and Situations (DnS) provide a broad conceptual scope and a methodological approach to ontology engineering Modularized, distinguishing between core aspects of information relevant for PIM Information Realization (Filing, Retrieving, Modifying) Communication (Sharing, Arguing, Explaining) Classification (Structuring, Summarizing) (De)composition (Creating, Reusing)
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 43
Semantic Desktop Prototypes
Current Semantic Desktops solve problems partially: Use of RDF + Schema/Ontology language to add semantics to data formats Enable cross-data classifications
Descr ipt ion
Owner
Owner
Access Rights
Sender
Recipient
Task Due Dat e
Input
File Mim e Type
Out put
label:Taskx File Size Cr eate Dat e
Em ail Body Att achm ent
Dat e
label:Eventx Tit le Moder at or
Event Par t icipant Locat ion
ISWeb - Information Systems & Semantic Web
Tim e Topics
Steffen Staab
[email protected]
Semantic Desktop 44
Semantic Desktop Prototypes
Do not support information reuse across applications, only cross-information classification (based on translation of application models to RDF) Lack a methodology for mapping between (semantically) augmented data formats one-to-one mappings that do not scale employed reference models lack a development methodology that enables context representation and extensibility
ISWeb - Information Systems & Semantic Web
Steffen Staab
[email protected]
Semantic Desktop 45
