Design and Implementation of A Mobile and Portable Lifelog Media ...

Proceedings of MoMM2007

Design and Implementation of A Mobile and Portable Lifelog Media System Baud Haryo Prananto 1) 2), Ig-Jae Kim 1), Hyoung-Gon Kim 1) 2)

Abstract Lifelog Media system stores and manages user’s everyday experiences in form of multimedia data. To build such a system we require an integrated framework for capturing the experiences to multimedia data, storing and managing those data, and also presenting the data to the user in a userfriendly way. Due to the mobility of the user we built a mobile framework that include wearable devices that enable user to capture the experiences easily and web-based management system that can be presented anytime and anywhere using web interface.

1. Introduction The objective of Lifelog system is to trace the "threads" of an individual's life in terms of events, states, and relationships [1]. Generically, the term "lifelog" or "flog" is used to describe a storage system that can automatically and persistently record and archive some informational dimension of an object's or user's life experience in a particular data category [2]. As the mobile computing devices and technologies have advanced rapidly recently, several works have been studied in Lifelog Media such as Aizawa et. al [3], de Silva et. al. [4], and He et. al. [5]. On those studies, the authors are interested for the concept of digital logging and the conceptual implementation of their Lifelog Media system. We also interested on the concept of digital logging of Lifelog Media therefore we designed an integrated framework that included capture, management, and presentation of Lifelog Media. Our concept introduces a system that prioritize the portability factor by using mobile wearable devices and web-based management system that enables user to manage and review Lifelog Media anytime and anywhere without being bounded to proprietary devices. In section 2 we describe the architecture including design and implementation of our Lifelog Media framework. Section 3 presents the performance evaluation of our Lifelog Media framework and in section 4 we conclude the work and describe about the future work.

1

Imaging Media Research Center, Korea Institute of Science and Technology (KIST) {baud | kij | hgk}@imrc.kist.re.kr 2 Korea University of Science and Technology (UST)

239


2. Architecture of Lifelog Media Framework Our Lifelog Media framework is divided into 3 logical parts: - Lifelog Media (LLM) Client is utilized to capture user’s experience in form of digital media and its metadata using camera and several sensors. LLM Client then sends the captured media and metadata to LLM Client. It is implemented using wearable computer and sensor set. - Lifelog Media (LLM) Server is used to store the media and metadata captured by LLM Client and then manages the metadata into a database system. LLM Server also has HTTP server to present the media and metadata to the User Client. - User Client is the unit where the user can review the Lifelog Media data. This User Client physically can be any machine with internet web browsing and media handling capability. It even can be the same device with LLM Client.

Figure 1: Architecture of Lifelog Media Framework

2.1. Lifelog Media Client LLM Client captures the user’s experiences using camera and a set of sensors. The camera captures the media and also the temporal metadata while the video also have time information. The location metadata is captured by GPS receiver. Activity sensors such as accelerometer and gyroscope sensors are used to capture action metadata and to recognize object for surrounding metadata, RFID system is employed. Furthermore, the analysis of the video can also obtain metadata about location, person, objects, and environment.

240


Physical implementation of our LLM Client is designed to be wearable and mobile using Sony VAIO UMPC for the coordinator that connects to a web camera, Bluetooth GPS receiver, and wearable motion sensors. 2.2. Lifelog Media Server LLM Server receives Lifelog data from LLM Client, stores the media, and manages the metadata using a database. LLM Server also have HTTP server to present the media and metadata, in form of web application, to the user client. Abstracted Metadata Database Personalized Query

Personalized Metadata

Composite Query

Composite Metadata

Composite Query

Primitive Query

Primitive Metadata

Primitive Query

File

Detailed Metadata Database Keyword

File Clip

Clip

Clip

Clip

XML Annotation File & Position

Figure 2: Overview of 2-part Metadata Database Design

The database is divided into two parts: - Abstracted metadata database contains metadata and their relation with media files. The main purpose of using this abstract metadata database is to find the correct file matching the given criteria in various retrieval criteria. This abstracted metadata database is further divided into three level of abstractions: - Personalized metadata stores the personalized metadata (e.g. office, home, meeting) that have different semantic meaning among users. These metadata is an arbitrary combination of composite level metadata. - Composite metadata stores the general metadata that have same meaning among users (e.g. Seoul, sitting, book). In composite metadata database, the metadata is categorized into location, time, action, and surroundings information. - Primitive metadata stores the less human-perceivable data and more machine-readable representation of metadata like GPS coordinates and timestamp. - Detailed metadata database not only contains the metadata and their relation with media file but also arranges them in time function. Thus, using this database the user not only knows the related media but also the exact position of the metadata occurrence.

241


Using this database structure, the user has wide range of options for query methods i.e. using personalized metadata, composite metadata (location, time, action, surrounding), or primitive metadata. The reason of using two kind of database in multi-level arrangement in this system is the characteristic of Lifelog Media and its metadata. Lifelog Media deals with enormous amount of media files and each media files have a lot metadata as well. If we use only XML annotation-based detailed metadata database, which is commonly used in multimedia management systems, the query performance will be very slow because it must handle a lot of data. Therefore, we simplified the metadata by disregarding the exact appearance of metadata inside a media file and summarize them in abstracted metadata database. Using this method, the user can search in abstracted metadata database first to get the media file first and after that the query can be performed further in detailed metadata database to get the exact appearance of metadata only in the selected file. Because Lifelog Media system contains the records of experiences that user already encountered, the exact appearance of metadata is not essential in the first stage. The database system is implemented by leveraging the advantages of relational database management system (RDBMS) and Native XML Database (NXD). The primitive and composite level metadata database in abstracted metadata database is implemented using PostgreSQL [6] and PostGIS [7] RDBMS while the personalized metadata and detailed metadata database is implemented using eXist [8] NXD. In primitive and composite metadata database, we leverage the RDBMS’ superior indexing system for efficient retrieval while in personalized metadata database and detailed metadata database we prefer to use XML format due to the unstructured nature of the metadata. 2.3. User Client We want to make this Lifelog Media framework as mobile and portable as possible, therefore we use the web-based application to access the Lifelog Media data. Using web-based application, the user doesn’t have to install a proprietary application but just use any machine that have HTTP internet browsing and media handling capabilities. The main function of user client is to present the connectivity to the database and the playback of media along with its metadata in a user friendly web interface [9].

242


Textual Query

Low Level Query (Coordinate and Time)

Video Player Video Annotation

Personalized Pre-defined Query

Figure 3: Web Application in User Client

3. Evaluation To evaluate our Lifelog Media framework, we used simulated data as if we have 1,000,000 media files. The metadata were suited with Lifelog Media characteristics that the location of a media file tends to be concentrated in some places (e.g. home and office) but the time information are evenly distributed assuming user records the Lifelog Media continuously. At the experiment, we compared among query by location, query by time, and composite of both queries. Each query is performed 5 times and query times are measured as the parameter of query performance. It can be seen in Figure 4 that query performance is depended on the volume of the query result. Using composite query the obtained result set can be smaller and thus can be queried faster. Another evaluation we did is to measure the performance of our multi-level database design by comparing it with the usual approach of XML annotation-based metadata database. It can be seen in Figure 5 that our multi-level database design performs better because abstracted metadata database searches the query faster than detailed metadata database.

243


Comparison on Queries 160000

142109 129047

Query Time (ms)

140000

141063

135063

126719

120000 100000 80000

62984 50937

60000 40000

57063 45718

48171

24938

24563

24391

24483

26593

1

2

3

4

5

20000 0

Location: Sangwolgok (Result: 400291) Time: Friday (Result: 142831) Composite: Sangwolgok and Friday (Result: 57227)

Figure 4: Comparison on Query Methods

Comparison between Detailed and Abstracted Metadata Database 7000 6000

Query A

Query C

5000

Detailed Metadata DB


Metadata Query

File Names

Query B Abstracted Metadata DB


Method 1 Method 2

3000

1000

Query C File Names

4000

2000

User Intervention, choose File Names

Method 2:

Metadata Query

Metadata Result

Query Time (ms)

User Intervention, choose File Names

Method 1:

0 Metadata Result

1

2

3

4

5

6

Figure 5: Comparison between detailed and abstracted metadata database

4. Conclusion and Future Work We have presented a mobile and portable Lifelog Media framework that can capture, manage, and present user’s experience in form of multimedia data. From the experiment it can be seen that several method of queries can be performed and composite query of some criteria performs better than using only one criterion. Our multi-level metadata database that used in this framework also presents better performance than usual XML annotation-based metadata database. Currently we are developing more advanced metadata capturing method and analysis that enable real-time database population. We also are developing the more advanced data representation to represent more abstract and human-perceivable metadata.

244


Acknowledgments This work was supported by the IT R&D program of MIC/IITA. [2006-S-032-01, Development of an Intelligent Service technology based on the Personal Life Log].

References [1] “DARPA's bid solicitation for LifeLog” (now offline, archived at web.archive.org), http://web.archive.org/web/20030603173339/http://www.darpa.mil/ipto/Solicitations/PIP_03-30.html, accessed 22 August 2007. [2] Wikipedia: “LifeLog”, http://en.wikipedia.org/wiki/LifeLog, 2007. [3] Aizawa K., Tancharoen D., Kawasaki S., Yamasaki T. (2004): “Efficient Retrieval of Lifelog Based on Context and Content”, Proc. ACM 1st workshop on Continuous archival and retrieval of personal experiences (CARPE), New York, NY, USA, 22-31, ACM Press. [4] de Silva G.C., Oh B., Yamasaki T. Aizawa K. (2005): “Experience Retrieval in a Ubiquitous Home”, Proc. ACM 2nd workshop on Continuous archival and retrieval of personal experiences (CARPE), Singapore, 35-43, ACM Press. [5] He W., Xiang P., Shi Y. (2005): “A Semantic Framework for Meeting Data Retrieval”, Proc. ACM 2nd workshop on Continuous archival and retrieval of personal experiences (CARPE), Singapore, 53-59, ACM Press. [6] PostgreSQL, PostgreSQL Global Development Group. http://www.postgresql.org, 2007. [7] Refraction Research: “PostGIS”, http://postgis.refractions.net, 2007. [8] Wolfgang Meier: “eXist XML Database”, http://exist.sourceforge.net/, 2007. [9] Kim I.J., Ahn S.C., Kwon Y.M., Ko H., Kim H.G. (2006): “PERSON: Personalized Experience Recoding and Searching On Networked Environment”, Proc. ACM 3rd workshop on Continuous archival and retrieval of personal experiences (CARPE), Santa Barbara, USA, ACM Press.

245