Sound of Vision â 3D scene reconstruction from stereo vision in an ... Implemented in Python using. Csound ... Processing of the disparity map. Removal of ...
Sound of Vision – 3D scene reconstruction from stereo vision in an electronic travel aid for the visually impaired
M. Owczarek, P. Skulimowski, P. Strumillo Date: 14.7.2016
Sound of Vision project EU Horizon 2020 - prototype wearable electronic travel aid • Aim of the project: a wearable ETA for assisting blind persons by rendering the image of the environment through an auditory display • The prototype device captures a 3D image of the environment, processes it in realtime, and presents the relevant information to a blind user • New approaches to 3D scene reconstruction and segmentation
Page: 2
Stereovision -> Disparity map
xL
d = |XL-XR| Page: 3
xR
UV-disparity representation of disparity map
Page: 4
Proposed methods for 3D scene reconstruction • Direct sonification of the “U-disparity” • The concept of the simplified sound icons Stereovision camera
Disparity
Estimation of ground plane
(on CPU)
(based on disparity and depth maps)
Depth
Obstacle detection
“U-disparity”
(after ground plane and background removal)
(representation of the scene)
Obstacle blobs
“Obstacle map”
(image moments, mask, etc.)
(s cene segmented into ground and obstacles)
Point cloud (of extracted obstacles)
Page: 5
Basic UV-disparity sonification What was done: Simple U-disparity image sonification Implemented in Python using Csound Moog Ladder synthesizer Pentatonic scale, 6-9th octave. Stereo panning Nonlinear distance scaling
Page: 6
The concept of the simplified sound icons Raw disparity map
Page: 8
Processing of the disparity map Removal of background, ground plane and all elements >2m above it
Page: 9
The concept of the simplified sound icons Minimum-area bounding rotated rectangles outlining the detected obstacles
Page: 10
The concept of the simplified sound icons Result segmentation map
Current and future work • Merging stereovision data with Structure Sensor data (for improved indor vision) • Detection and sonification of special object categories: • Walls • Stairs • Holes/dropoffs • Hanging face/torso level obstacles • Doors • Text • Persons
Page: 12
Summary
• A simple and effective method of detecting obstacles and the ground plane has been developed • Obstacles can be detected even if the quality of the disparity map is low • Assumption: the ground plane occupies significant part of the image and is clearly visible • Weakness: only significantly large obstacles can be detected due to narrow base of the stereovision system
http://icchp2016.naviton.pl/
Page: 13
Acknowledgment
This work received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 643636 ”Sound of Vision”.
