NEMO Converter 3D: Reconstruction of 3D Objects ...

Institute for Multimedia and Interactive Systems (IMIS)

NEMO Converter 3D: Reconstruction of 3D Objects from Photo and Video Footage for Ambient Learning Spaces David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg University of Luebeck, Germany

Supported by

14.11.2017

David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg

1 / 19

Agenda • • • • • • • •

14.11.2017

The Scenario Ambient Learning Spaces Platform: NEMO The Idea Requirements for the Scenario Realization Findings Future Work


2 / 19

The Scenario • School students create NEMO Enriched Media during their field trip using their own Smartphones. – The students take pictures. – The students capture video footage. – The students take notes and describe media in context.

14.11.2017


3 / 19

The Scenario

14.11.2017


4 / 19

The Scenario • School students create NEMO Enriched Media during their field trip using their own Smartphones. – The students take pictures. – The students capture video footage. – The students take notes and describe media in context.

• In this context, NEMO Enriched Media consists of: – Binary media: pictures, video – Semantic Annotations: Full-text notes, tags, other contextspecific digital properties.

14.11.2017


5 / 19

Ambient Learning Spaces • Learning understood as a purposeful linear form of acquisition of knowledge looses more and more of its reference in the lifeworld (Lebenswelt). • For this reason we developed the Ambient Learning Spaces (ALS). ALS support processes of teaching and learning, among others: – foster learning in a social and communicative way – provide an environment for playful learning – the self-initiated discovery of meaningful information

• ALS consist of a variety of interlinked learning applications focusing on collaborative learning.

14.11.2017


6 / 19

Platform: NEMO The Network Environment for Multimedia Objects (NEMO) is the basis for our project. For this work, important NEMO features are: • Device Independence Based on cross-platform web technology, NEMO is available for use on almost any device.

• Individualization NEMO offers an environment for the creation and interaction with personal media.

• Contextualization NEMO distributes NEMO Enriched Media in space and time inside a learning environment.

• Semantic Annotations NEMO Enriched Media hold contextualized sub-sets of semantic annotations. 14.11.2017


7 / 19

Platform: NEMO In our scenario, the features are reflected as follows: • Device Independence School students use their own Mobiles to create media and interact with 3D objects.

• Individualization The school student’s personal media is used for the automatic reconstruction of 3D objects.

• Contextualization The 3D object is viewable on the school student’s Mobile.

• Semantic Annotations NEMO uses annotations to identify, which photos and video footage potentially belong together.

14.11.2017


8 / 19

The Idea • NEMO Converter 3D (NOC3D): From all the media school students capture, in the background, NOC3D creates new media in the form of 3D objects to: – – – –

14.11.2017

achieve higher immersion make topics more vivid foster the discovery of meaningful information further use in other ALS-Applications based on NEMO


9 / 19

Example Footage

14.11.2017


10 / 19

Requirements for the Scenario • In order to provide 3D objects, – a precise media pre-selection is required. – a workflow is needed, that does not require school students to actively create media for 3D reconstruction. – the reconstruction of 3D objects • has to be fully automated • runs in the background • runs unmonitored

14.11.2017


11 / 19

Architecture

14.11.2017


12 / 19

Realization • In NEMO, media is pre-selected using semantic annotations, such as: – – – – – – –

author group of authors context, the media was created in GPS coordinates manual annotations, e.g., tags and full text keyword matching automated annotations from NEMO Computer Vision library date and time of creation

• Additionally, we pre-select media by graphical means with multiple steps of image matching.

14.11.2017


13 / 19

Reconstructed Example Footage

14.11.2017


14 / 19

Technical Findings • Approx. 110 images are required to reconstruct a 3D object in acceptable quality. • Experiments show, that a multiple of that is required to be able to select suitable images from. – Strategies are needed to animate school students to capture video footage. – We cross-link sources in NEMO.

• To fulfill the scenario in a timely manner, powerful hardware is required. – A dedicated 3D reconstruction system is required. – We are currently outsourcing the 3D reconstruction process from our project partners to our systems.

14.11.2017


15 / 19

Summary • With NOC3D, we create 3D objects from photos and video footage in a background process for use in Ambient Learning Spaces. • NOC3D is in experimental deployment with two project partners in the Hanseatic City of Luebeck – a grammar school and a common school

• Creating media with their own devices connects students with the Lebenswelt. This personal context together with the topic at hand promotes a meaningful confrontation.

14.11.2017


16 / 19

Summary The NEMO-powered ALS-Application Cosmos: Wearable Media

14.11.2017

Mobile Media

Tangible Media


Peripheral Media

17 / 19

Future Work • In our research transfer project, we are currently realizing a new version of the media capturing application with Mobiles. • We are currently developing a web-based module for editing 3D objects in context of ALS for mobile and stationary ALS-Systems. • Use 3D objects from NOC3D in dynamic narrations in interactive tours with mobile NEMO-based ALSApplications in museum context.

14.11.2017


18 / 19

Acknowledgements • This work is supported by the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG) • The DFG funded the project for a total of 7 years. • In this latest 3-year project phase, the project is carried out as research transfer project with four project partners.

14.11.2017


19 / 19

Institute for Multimedia and Interactive Systems (IMIS)

NEMO Converter 3D: Reconstruction of 3D Objects from Photo and Video Footage for Ambient Learning Spaces David Bouck-Standen, Alexander Ohlei, Viktor Daibert, Thomas Winkler and Michael Herczeg University of Luebeck, Germany

For more information please visit: http://als.imis.uni-luebeck.de Supported by

14.11.2017


20 / 19

NEMO Overview

14.11.2017


21 / 19