Steerable Interfaces for Pervasive Computing Spaces - PerCom

Steerable Interfaces for Pervasive Computing Spaces Gopal Gopal Pingali Pingali Claudio Claudio Pinhanez Pinhanez,, Anthony Anthony Levas Levas,, Rick Rick Kjeldsen Kjeldsen,, Mark Mark Podlaseck Podlaseck,, Han Han Chen, Chen, Noi Noi Sukaviriya Sukaviriya IBM IBM T.J. T.J. Watson Watson Research Research Center Center Hawthorne, Hawthorne, New New York York USA USA

Talk Overview ! Motivation/definition of steerable interfaces ! Overview of key technologies ! Demonstration of prototype smart retail environment ! Conclusions and discussion

Motivation Can a display appear wherever the user needs it rather than the user being enslaved to display devices (monitors, televisions etc.) ?

Steerable Interfaces for Pervasive Computing Environments Steerable Interface: An interface to computing that can move around a physical environment to appear on ordinary objects or surfaces or even in empty space. Characteristics: ! Movable output interface ! Movable input interface ! Adaptation to user context ! Adaptation to environmental context ! Device-free interaction ! Natural interaction

Applications: ! Information access in public spaces -- retail, transportation ! Instruction and information delivery in military training centers, ships, aircraft ! Deviceless interaction in hazardous environments ! Augmented reality in hospitals ! Interactive real world games

Steerable Interfaces -- Key Technologies Environment modeling Steerable projection

Calibration

WORLD MODEL (3D geometry, user location,display zones, calibration parameters, system state)

Steerable vision

Steerable microphones

Steerable audio User Localization Geometric Reasoning

A P I

Steerable Projection: The Everywhere Display A device that creates an interactive projected display on any surface in an environment. Prototype bright LCD projector rotating mirror pan/tilt/zoom camera

Oblique Projection Distortion

display image

Correction of Oblique Projection Distortion

display image

Goal: projected image is free of distortion when viewed by an “orthogonal” viewer

Projector/camera Calibration !! !! !!

computation Hv given computation of of homography homography H/ H/Hv given location location of of corners corners on on aa display display surface surface need need expected expected hand hand size size in in image image and and direction direction of of entry entry for for interaction interaction manual manual specification specification of of focus, focus, pan, pan, tilt, tilt, zoom zoom

Integration into Office-of-the-Future: BlueSpace at IBM T.J. Watson Ceiling Lights Task Lights

Status Lights Everywhere Display

Primary Display

OfficeFront Display Wireless Pressure Sensor

Environment Sensors Personal Air & InfoPanel Heating Display

Scenarios in Bluespace

Team work support

Email notification on desk

Personalized wall display

Video conferencing anywhere

Steerable Gesture Recognition for Input Projected light changes appearance of hand!

!! camera camera steered steered to to projected projected display display !! motion motion detected detected in in camera camera image image stream stream !! motion motion data data matched matched to to finger finger tip tip template template

Surface Independent Interface Specification application space map to surface A

map to surface B

! ! !

Independent specification of what and where Dynamic configurations of widgets; Map configuration to surface at run time; image space

Widget Architecture Widgets:

Image Region Definition (IRD)

Touch Button Tracking Area

Image

Change Detection

Surface Transformation (ST)

Motion Detection

Component types:

Event Generation (EG)

Touch Motion Detector

Fingertip Tracking

IRD EG ST

Return Value Transformation

Image Processing Motion Analysis System Functions Instance Data

In-Region Detection

IRD EG ST

Good high level abstraction - Efficient widget combination - Flexible widget definition

First Prototype of Steerable Interfaces (Siggraph, Aug 2001) Completed picture

Interactive cans with M&Ms

Color selection Invitation board at entrance Picture in progress: placement, finger painting

User Experience at Siggraph 2001

SIGGRAPH01.avi

Interactive Applications Anywhere

Various interactive applications can be dynamically moved to different surfaces e.g.. MediaPlayer, PowerPoint presentations, Web Browser applications etc.

User-following Display Goal: Goal: Automatically Automatically steer steer display display to to appropriate appropriate surface surface at at the the appropriate appropriate position, position, size size and and orientation orientation based based on on user user location. location. Enabling Technologies: 1. 3D model of the environment 2. 3D calibration of projector(s), camera(s) 3. Dynamic localization of the user(s). 4. Geometric reasoning engine for: virtual projection virtual imaging virtual steering occlusion detection computation of warping function

1. 3D Environment Modeling VRML model of environment: ! Based on real world measurements

Modeling toolkit ! Basic 3D modeling software supporting simple geometries (planes, cubes) and import of more complex geometries ! Provides annotations for all objects -- semantics can be attached to objects ! Includes “projectors”, “cameras”, and “PJsurfaces” ! Model stored in XML format (objects are tags and annotations are attributes)

2. 3D Calibration: The Link Between the Virtual and the Real 3D 3D Camera Camera Calibration Calibration

!! Use Use correspondence correspondence between between points points in in 3D 3D model model (known (known points points in in environment) environment) and and points points in in camera camera image image plane plane !! Error Error in in image image to to world world mapping mapping within within 25 25 mm mm (maximum) (maximum)

3D 3D Projector Projector Calibration Calibration

!! Determine Determine effective effective location location and and orientation orientation of of projector projector !! Error Error within within 55 mm mm (coplanar) (coplanar) and and 15 15 mm mm (non-coplanar) (non-coplanar) in in image image to to world world mapping mapping

3. User Localization Real-time Real-time tracking tracking of of user user position position !! Use Use image/background image/background differencing differencing with with morphological morphological filtering filtering to to detect detect motion motion regions regions !! Extract Extract bounding bounding contours contours of of resulting resulting regions regions !! Check Check for for presence presence of of aa head head by by analyzing analyzing extrema extrema of of curvature curvature on on the the contour contour and and circularity circularity !! Track Track detected detected head head !! Triangulate Triangulate detections detections from from multiple multiple cameras cameras to to determine determine 3D 3D position position

4. Geometric Reasoning Input Input:: 3D 3D model, model, user user position, position, display display zones, zones, ED ED calibration calibration parameters parameters Output Output:: Information Information on on every every display display zone: zone: !! physical physical size size !! user user proximity proximity !! orientation orientation of of zone zone w.r.t w.r.t user user !! percentage percentage of of occlusion occlusion of of aa display display zone zone !! occlusion occlusion mask mask (a (a bitmap bitmap indicating indicating occlusion) occlusion)

Tying it all together: UserFollowing Display ! Pre-define a set of display zones in relation to environment model ! Calibrate projector in 3D for each display zone ! Locate user by tracking with pre-calibrated camera. ! Map user position to 3D environment. ! Select display zone based on user location -- after occlusion check.

Video: User-Following Display

Demo: Everywhere Sales Associate Technical Goal: Adaptation of interface to user and environmental context. ! Interactive help anywhere in a store ! Dynamic placement of display close to the user ! Adaptation of display content/interaction mode ! Detection and correction for occlusion of display Examples: ! Ubiquitous Product Finder !Adaptive Signage ! Interactive Product Bins

Video: Product Finder - Table

Video: Product Finder – Wall(1)

Video: Product Finder: Wall (2)

Video: Interactive Signage

Video: Clothing Bins (distance)

Video: Clothing Bins (interactions)

Conclusions and Future Directions ! Introduced and demonstrated feasibility of new class of steerable interfaces ! Adds a new dimension to the vision of “computing woven into space” without wiring the user/objects Future Directions: Greater multimodality, adaptation, context awareness ! Steerable Audio-visual interface ! Underlying distributed architecture for context awareness and adaptation ! Adaptation to surface characteristics ! Multi-user/collaborative situations ! Advanced calibration/model capture ! “Natural” interaction!