The Everywhere DisplaysProjector: A Device toCreate UbiquitousGraphical Interfaces ClaudioPinhanez IBMThomasWatsonResearchCenter P.O.Box 218,YorktownHeights,NY10598
[email protected]
-USA
Abstract.ThispaperintroducestheEverywhereDisplaysproj ector,adevice thatusesarotatingmirrortosteerthelightfrom anLCD/DLPprojectoronto differentsurfacesoaf nenvironment.Issuesobf ri ghtness,obliqueprojection distortion,focus,obstruction,anddisplayresolut ionareexamined.Solutionsto someoftheseproblemsaredescribed,togetherwith aplantouseavideo cameratoallowdevice-freeinteractionwiththepr ojectedimages.TheEDprojectoriaspracticalwaytocreateubiquitousg raphicalinterfacestoaccess computationalpowerandnetworkeddata.Inparticul ar,itisenvisionedasan alternativetothecarryingoflaptopsandtothei nstallationofdisplaysin furniture,objects,andwalls. Inaddition,theuseoE f D-projectorstoaugment reality withoutthe use ogoggles f iexamined s and illustratedwithexamples.
1 Introduction Ubiquitouscomputingenvisionsaworldwhereitis possibletohaveaccessto computerresourcesanywhereandanytimetothedata andservicesavailablethrough theInternet[1].Sincemostof currentsoftwareandInternetdataisdesignedtob e accessedthroughahigh-resolutiongraphicalinterf ace,totrulyubiquitouslycompute today usersneeddeviceswithreasonablegraphicalcapabi lities.This meanscarrying laptopseverywhere,wearingcomputergraphicsgoggl es,orinstallingmonitorsand displaysonthesurfacesosfpacesandobjects,suc hasdesks,fridges,andentrance doors.Or, simply, to resign tothelow-resolution displays ofmobilepho nes or PDAs. Inthispaperweexploreanalternativeapproachto createubiquitousgraphical interfaces.OurideaitsocoupleanLCD/DLPprojec torto motorized a rotatingmirror andtoacomputergraphicssystemthatcancorrect thedistortioncausedbyoblique projection.Asthemirrormoves,differentsurfaces becomeavailabletobeusedas displays.Also,weplantoemployavideocamerato detecthandinteractionwiththe projectedimageusing computer vision techniques. Ourtargetistodevelopprojection-based a system thatcreatesinteractivedisplays everywhereinanenvironmentbytransformingasurf aceintoaprojected“touch screen.”Suchan EverywhereDisplaysprojector canbienstalled,forexample,onthe ceilingofaspace,toprovideagenericcomputeri nterfacetousersinthat environment(seeFig.1).
Fig. 1. UsingtheEverywhereDisplaysprojectortocreate surfacesinanenvironment.
displaysondifferent
Forexample,anED-projectorcanbeinstalledina meetingroomandbeusednot onlytoprojectimagesonthewallsbutalsotocre ate individualdisplaysforeachof theparticipantsof ameeting.Insteadoftoday’smeetingtablespopula tedby bunkeringpersonallaptops,asmallsetofED-proje ctorscanbesharedbythe participantstoaccesstheirpersonaldata,buteas ilyreconfiguredtoallowteamwork. Forinstance,twopeoplecanbepairedtogetherto workonasub-problemusinga displayprojectedinfrontotfhemwhiletheother participantskeepdiscussingthe main themes using display a projectedotnhewall. Moreover,ED-projectorshavetheabilitytoprovide computeraccessinpublic environmentswithouttheriskofequipmentbeingbr okenorstolen.Insteadof carrying computer, a PDA,orphoneeverywhere,user can s simply request caomputer display,maybebymakinggesture a toanoverheadc amera,andreceivetheprojected imageonasurfaceneartheirlocation.Forexample an , ED-projectorinastorecan transformpiecesowhite f cardboardattachedtoshe lvesintointeractivedisplayswith productinformation.Unliketraditionalkiosks,the reins oneedtoboltmonitorsand computers tothefloor. Inotherwords,weareproposingashiftinthedis play paradigm,whichceasesto beregardedaadevice s tobeinstalledinanenvir onment,orcarriedalongbyauser andbecomesaserviceprovidedbyaspace,likeele ctricpoweropr honelines.But likeany innovation,ED-projectorsnotonlysolveproblem a butalsocreatenew a set of applications.Forinstance,ifinformationaboutlo cationandidentityoof bjectsin anenvironmentisknown,anED-projectorcanbuese dadevice as toaugmentreality withoutrequiringuserstoweargoggles.Itcanlea davisitortoitsdestinationina
Fig. 2. Currentprototypeothe f EverywhereDisplaysproje usedianonffice-like environment(right).
buildingbydirectlyprojectingarrowsonthefloor ontotheobjectsbeingassembledinanindustrialp computersystemnotonlytoaugmentaphysicalspac “act”inthatenvironmentanditsusersbyprojecti objects andpeople. ThispaperdescribesourcurrentprototypeotfheE andthetechnologicalsolutionsusedinitsimpleme applicationsfordifferentscenariosandexaminene human-computerinteractionthatmightemergefromt projectors.
ctor(left)andtheprojectorbeing
o; pr rojectinformationdirectly lant.TheED-projectorenablesa ewithinformation butalsoto nglightpatternsandsymbolson verywhereDisplaysprojector ntation.Wealsodemonstrate winteractionparadigmsfor hepervasiveuseofED-
2 The Everywhere DisplaysProjector The EverywhereDisplaysprojector ,orsimply ED-projector,iscomposedofan LCD/DLPprojectorandacomputer-controlled pan/tiltmirror.Theprojectoris connectedtothedisplayoutputofahostcomputer thatalsocontrolsthemirror.The left-sidepictureoFig. f 2shows pa rototypeoan f ED-projectorbuiltwithanoff-th shelfrotatingmirrorusedintheatrical/discoligh ting,connectedthroughaDMX network tothehostcomputer. IntheconfigurationshowninFig.2-left,theproj ector’slightcanbedirectedin anydirectionwithintherangeoaf pproximately60 degreesintheverticaland230
e-
Fig. 3. Perceptionofcontrast:globalbrightness(left); photomontage simulating the perceivedcontrast(rig
localbrightness(middle);anda ht).
degreesinthehorizontal.Whenpositionedintheu ppercorneroafroom(suchas showninFig.1),thisprototypeiasbletoproject inmostpartofthetwofacingwalls, half of thetwoadjacentwalls,andalmosteverywhe reotnhefloor. Figure2alsoshowsthecurrentprototypeotfheED -projectorinuseinanofficelikeenvironment.Thetop-rightofFig.2showsthe ED-projectorhelping collaborativework.Noticetheprojectorontherig htuppercornerotfhepictureand theangleothe f rotatingmirrorusedtodirectthe lightontothewall.Thebottom-right pictureofFig.2showsthesamesurfacebeingused toprojectthepictureofan artworkasdecorationfortheroom.Bothphotoswer etakenundernormaloffice lighting conditions. ED-projectorsarefeasibletodayduetothetechnic aladvancesintwoareas:video projectorsandcomputergraphicsengines.CurrentL CD/DLPprojectorsareableto createimagesthathaveenoughcontrasttobeseen evenwhenlightsareturnedon. Fastandcheapcomputergraphicsenginesarenecess arytocorrectforthedistortion causedbyobliqueprojection.Thisandotherimplem entationissuesarediscussedin thefollowing sections. 2.1
Brightness andContrast
Projectingimagesinabrightlylitroomispossibl ebecausethehumanvisionsystem perceivesbrightnessandcontrastlocally.Consider awhitewallinanenvironment withnormallighting:ifnoimageips rojected,sub jectswoulddescribethebrightness ofthewallas“white.”However,ifawhiteandbla ckpatternwithsufficient brightnessisprojectedonthesamewall(typically 5to10timesbrighterthanthe normallighting),viewersperceivethewhiteprojec tedpatternas“white”andany neighboring areareceiving only theambientlighta “black” s [2]. Figure3exemplifiesthismechanisminsituation a wheretheED-projectorisused tocreateprojectedlabelsonwhite Styrofoamcups.Theleftpicture ofFig.3shows theglobalbrightness as“collected”bay photographiccamera.In this picture,ambient lightilluminatesallthecupsalthoughthethirdc upotfhetopshelfalsoreceivesthe projectionotfhepatternshowninthemiddlepictu reoFf ig.3.Intheleftimagethe
Fig.4. Usingvirtual a computergraphics3Dworldtocorr projectionbysimulatingtherelationshipinthere projectedsurface.
ectthedistortioncausedbyoblique alworldbetweentheprojectorandthe
projectedpatternibsarelyvisiblebecausethepho tographiccamera,unlikethehuman eye,equalizesthebrightnessglobally.Thecamera takesinaccountthelargeblack backgroundoftheshelfandcompressestherangeof brightnessdifferencesotfhe pattern projectedotnhecupinto few a shades ofw hite. However,as howninthemiddleofFig.3,thereis enoughlocaldifferencein brightness on theprojected pattern toallow its cl ear perception ionly f locallighting is considered.In particular,noticethattheblack“O K”letteringcorresponds tothewhite surfaceofthecupreflectingjusttheambientligh t.Sincehumanvisionadjuststo localcontrast,theresultingperceivedbrightness ismorelikethephotomontage showniF n ig.3-rightwheretheprojected pattern i s clearly discernible. Our first prototype employsa1200lumensLCDprojectorthathasproved tohave enoughbrightnessandcontrastto projectimagesonthesurfacesoaf nofficeroom withthelightson.Althoughwehavenotconducted experimentstodeterminethe perceivedbrightnessandcontrast,intypicalhome andofficeconditionsawhite patternprojectedbyourprototypeisapproximately 10timesbrighterthanits surroundings and,therefore,enough tocreatethei llusion ocontrast. f Thesecondprototypewebuiltemploysa3000lumens LCDprojector,enabling sharpercontrastformostprojectedsurfaces.Inpa rticular,theincreaseinbrightness improvedsignificantlythequalityoimages f projec tedonhorizontalsurfacessuchas tablesanddesks. Sincesuchsurfacestendtobe orthogonaltothesourcesoambient f light,thespecularcomponentotfheirreflectedli ghtisbrighterthannon-horizontal surfacessuchaswalls.However,with a 3000-lumen projector,eventheextra brightness providedbtyhe specular lightis overshadowedbtyheprojection li ght.
Fig. 5. Correctionooblique f distortions:(left)thetypi apatternonasurface;(right)theprojectionoft projectedimagefree odistortion f (bottom-right).
2.2
calresultoftheobliqueprojectionof hedistortedpattern(top-right)createsa
CorrectingforObliqueProjectionDistortion
Whenprojectionisnotorthogonaltotheprojected surface,theprojectedimageis distorted.Infact,asshowninFigs 1and2in ,mo stcasestheED-projectorisusedto createdisplayson surfacesthatarenotorthogonaltotheprojection direction.To correctthedistortionscausedbyobliqueprojectio nandbytheshapeothe f projected surface(ifnotflat),theimagetobeprojectedmu stbeinverselydistortedpriorto projection.In general,this distortion inson-line ar andicomputationally s expensiveto correct,involving theselectivecompression andth expansion othe f originalimage. Wehavedevelopedasimpleschemethatusesstandar dcomputergraphics hardware(presentnowinmost computers)tospeedupthisprocess.Ourmethod reliesonthefactthat,geometricallyspeaking,ca merasandprojectorswiththesame focallengthareidentical(asobservedin [3,4]). Therefore,toprojectanimage obliquelywithoutdistortionsitissufficienttos imulatetheinverseprocess(i.e., viewingwith camera) a in virtual a 3Dcomputer gra phicsworld. AsshowinFig.4,wetexture-maptheimagetobed isplayedontoavirtual computergraphics3Dsurfaceidentical(minusasca lefactor)totherealsurface.If thepositionandattitudeotfhissurfaceinthe3D virtualspaceinrelationtothe3D virtualcameraisidentical(minusascalefactor) totherelationbetweenthereal surfaceandtheprojector,andifthevirtualcamer ahasidenticalfocallengthtothe
projector,thentheviewfrom the3Dvirtualcamera correspondsexactlytothe“view” oftheprojector(iftheprojectorwasacamera).S inceprojectorsdotheinverseof viewing,i.e.,they projectlight,theresultis parojection freeodistortions. f Inpracticeweuseastandardcomputergraphicsboa rdtorenderthevirtual camera’sviewofthevirtualsurfaceandsendthec omputedviewtotheprojector.If thepositionandattitudeotfhevirtualsurfacear ecorrect,theprojectionothis f view compensatesthedistortioncausedbyobliqueprojec tionorbytheshapeofthe surface.Ofcourse,adifferentcalibrationotfhe virtual3Dsurfacemustbeusedfor each surfacewhereimages areprojectedianennvir onment. Anexampleothe f resultsothe f processids epicte dinFig.5.Intypical a situation ofobliqueprojection,thepatternshowninthetop -leftisprojectedwithoutany correction,resultinginthebottom-leftimageoFf ig.5.Aftercalibrationothe f virtual 3Dsurfaceandcameraparameters,theprojectionof therenderedimage(top-right) creates parojection freeodistortion f (bottom-rig ht). Sofarwehaveexperimentedonlywithprojectingon planarsurfaces.The calibrationparametersotfhevirtual3Dsurfacear edeterminedmanuallybysimply projectingthepatternshowninFig.5andinteract ivelyadjustingthescale,rotation, andpositionofthevirtualsurfaceinthe3Dworld and , the“lensangle”otfhe3D virtualcamera.Thisprocesstypicallytakesbetwee n10to20minutesbutweare currentlyworking oints automationusing technique similar s to[5]. Anothersimpletechniquetocorrectfordistortion onplanarsurfacesisimplyto distortthetexturetobeprojectedbyahomography [6].Inthiscase,calibrationis obtainedbiynteractively grabbing with themouse each corner of theprojectedpattern andmovingit tothedesiredlocationonthesurface.Alternative ly,thehomography canbeembeddedtothe graphicsboardprojectionmatrix [3].Unlikethepr evious approach,homographiesworkonly for planar surface s. 2.3
Focus
WecurrentlyuseLCD a projectorwherefocusandzo controlledbycomputercommandsissuedthroughthe problemwithobliqueprojectionisthatitisnotp projectedimagesimultaneouslyinfocus.Fortunatel have reasonable a depth ofocus f range,enough tom mostcases.Wehavesucceededinprojectingonsurf inclinationinrelationtotheprojectionaxiswith However,theproblembecomesmoresevereasthedis surfaceandtheprojector decreases. 2.4
omparameterscanbreemotely serialport.However,another ossibletoputallareasofthe y,currentcommercialprojectors aintaindecentfocusconditions in aceswithupto30degreesof outsignificantdegradationofocus. tancebetweentheprojected
DisplayResolution
Oneproblemwiththetechniquesdescribedaboveto itcreatesdisplayswithresolutionsthataresmall canbeseeninFig.5,thedistortioncorrectionpr
correctobliquedistortionitshat erthantheprojector’sresolution.As ocesshastofitanirregular
Figure 6. Change ovolume f for identicalinternalangles.
quadrangularintothe4:3viewingareaoftypicald isplays.Theresultisthata considerableamountof display areailost s in the process. Inourprototypeswe mploy1024x768XVGA projectors.However,duetothe lossodf isplayareacreatedbythedistortioncorr ectionprocess,wehaveobserved thattheobtainedresolutioncorrespondsapproximat elytoVGA,i.e.,640x480pixels. Thisestimationtakesinaccountthatintheproces os f renderingthedistortedimage somepixelsotfheoriginalimagearecompressedin tosinglepixelsotfheprojected image. Otherfactorsalsoinfluencetheperceivedresoluti on,amongthemtheangleof projectionandthetextureotfheprojectedsurface In . thecaseoef xtremeanglesof projection(
