Nov 14, 2017 - Thomas Winkler and Michael Herczeg ... foster learning in a social and communicative way ... foster the discovery of meaningful information.
Institute for Multimedia and Interactive Systems (IMIS)
NEMO Converter 3D: Reconstruction of 3D Objects from Photo and Video Footage for Ambient Learning Spaces David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg University of Luebeck, Germany
Supported by
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
1 / 19
Agenda • • • • • • • •
14.11.2017
The Scenario Ambient Learning Spaces Platform: NEMO The Idea Requirements for the Scenario Realization Findings Future Work
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
2 / 19
The Scenario • School students create NEMO Enriched Media during their field trip using their own Smartphones. – The students take pictures. – The students capture video footage. – The students take notes and describe media in context.
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
3 / 19
The Scenario
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
4 / 19
The Scenario • School students create NEMO Enriched Media during their field trip using their own Smartphones. – The students take pictures. – The students capture video footage. – The students take notes and describe media in context.
• In this context, NEMO Enriched Media consists of: – Binary media: pictures, video – Semantic Annotations: Full-text notes, tags, other contextspecific digital properties.
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
5 / 19
Ambient Learning Spaces • Learning understood as a purposeful linear form of acquisition of knowledge looses more and more of its reference in the lifeworld (Lebenswelt). • For this reason we developed the Ambient Learning Spaces (ALS). ALS support processes of teaching and learning, among others: – foster learning in a social and communicative way – provide an environment for playful learning – the self-initiated discovery of meaningful information
• ALS consist of a variety of interlinked learning applications focusing on collaborative learning.
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
6 / 19
Platform: NEMO The Network Environment for Multimedia Objects (NEMO) is the basis for our project. For this work, important NEMO features are: • Device Independence Based on cross-platform web technology, NEMO is available for use on almost any device.
• Individualization NEMO offers an environment for the creation and interaction with personal media.
• Contextualization NEMO distributes NEMO Enriched Media in space and time inside a learning environment.
• Semantic Annotations NEMO Enriched Media hold contextualized sub-sets of semantic annotations. 14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
7 / 19
Platform: NEMO In our scenario, the features are reflected as follows: • Device Independence School students use their own Mobiles to create media and interact with 3D objects.
• Individualization The school student’s personal media is used for the automatic reconstruction of 3D objects.
• Contextualization The 3D object is viewable on the school student’s Mobile.
• Semantic Annotations NEMO uses annotations to identify, which photos and video footage potentially belong together.
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
8 / 19
The Idea • NEMO Converter 3D (NOC3D): From all the media school students capture, in the background, NOC3D creates new media in the form of 3D objects to: – – – –
14.11.2017
achieve higher immersion make topics more vivid foster the discovery of meaningful information further use in other ALS-Applications based on NEMO
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
9 / 19
Example Footage
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
10 / 19
Requirements for the Scenario • In order to provide 3D objects, – a precise media pre-selection is required. – a workflow is needed, that does not require school students to actively create media for 3D reconstruction. – the reconstruction of 3D objects • has to be fully automated • runs in the background • runs unmonitored
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
11 / 19
Architecture
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
12 / 19
Realization • In NEMO, media is pre-selected using semantic annotations, such as: – – – – – – –
author group of authors context, the media was created in GPS coordinates manual annotations, e.g., tags and full text keyword matching automated annotations from NEMO Computer Vision library date and time of creation
• Additionally, we pre-select media by graphical means with multiple steps of image matching.
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
13 / 19
Reconstructed Example Footage
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
14 / 19
Technical Findings • Approx. 110 images are required to reconstruct a 3D object in acceptable quality. • Experiments show, that a multiple of that is required to be able to select suitable images from. – Strategies are needed to animate school students to capture video footage. – We cross-link sources in NEMO.
• To fulfill the scenario in a timely manner, powerful hardware is required. – A dedicated 3D reconstruction system is required. – We are currently outsourcing the 3D reconstruction process from our project partners to our systems.
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
15 / 19
Summary • With NOC3D, we create 3D objects from photos and video footage in a background process for use in Ambient Learning Spaces. • NOC3D is in experimental deployment with two project partners in the Hanseatic City of Luebeck – a grammar school and a common school
• Creating media with their own devices connects students with the Lebenswelt. This personal context together with the topic at hand promotes a meaningful confrontation.
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
16 / 19
Summary The NEMO-powered ALS-Application Cosmos: Wearable Media
14.11.2017
Mobile Media
Tangible Media
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
Peripheral Media
17 / 19
Future Work • In our research transfer project, we are currently realizing a new version of the media capturing application with Mobiles. • We are currently developing a web-based module for editing 3D objects in context of ALS for mobile and stationary ALS-Systems. • Use 3D objects from NOC3D in dynamic narrations in interactive tours with mobile NEMO-based ALSApplications in museum context.
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
18 / 19
Acknowledgements • This work is supported by the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG) • The DFG funded the project for a total of 7 years. • In this latest 3-year project phase, the project is carried out as research transfer project with four project partners.
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
19 / 19
Institute for Multimedia and Interactive Systems (IMIS)
NEMO Converter 3D: Reconstruction of 3D Objects from Photo and Video Footage for Ambient Learning Spaces David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg University of Luebeck, Germany
For more information please visit: http://als.imis.uni-luebeck.de Supported by
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
20 / 19
NEMO Overview
14.11.2017
David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg
21 / 19