COVER FEATURE
Roomware: Computers Disappear and Interaction Evolves Taking the real world as the starting point for designing computer-based support of human activities makes the computer disappear and pushes human-information interaction and human-human cooperation into the foreground.
Thorsten Prante Norbert A. Streitz Peter Tandler Fraunhofer IPSI, Darmstadt, Germany
0018-9162/04/$20.00 © 2004 IEEE
W
ith the spread of information technology into all areas of life, people have become accustomed to a new paradigm for creating and processing information. They write and read e-mail, produce and store electronic documents, and project visual presentations of their business plans. In general, these processes shift human activity away from real physical objects as sources of information and toward the desktop computer monitor as the interface to information. This shift emphasizes virtual over physical environments. However, this general observation does not hold true when it comes to meetings and group work. Especially with respect to creative activities like brainstorming, strategic planning, and decision making, people are still using traditional media—felt pens on whiteboards, colored paper cards on pin boards, and so on. It either is, or seems to be, easier to operate and interact with real physical objects than with computer monitors in these situations. Unfortunately, the information generated during such meetings is not immediately captured for digital postprocessing and can easily be lost altogether. At the Fraunhofer Integrated Publication and Information Systems Institute (IPSI) in Darmstadt, we are developing a comprehensive approach to the design of information and collaboration environments that support human activities by starting in the real world. Our goal is to design environments that combine the affordances of real objects with potential computer-based support in the virtual world.
This combination generates hybrid worlds that have significant consequences for the design of human-computer interaction. Hybrid worlds allow human-information interaction and human-human communication to move into the foreground while the computer disappears into the background. We distinguish between the physical disappearance, which is mainly due to miniaturization of computer parts and their integration in other artifacts, and the mental disappearance of computers achieved when they become “invisible” to the user’s mental eye and thus to human perception. This results from embedding computers into the architectural environment and furniture as well as from adapting user interfaces to enhance how these artifacts are used.1,2 We expect these phenomena to gradually change the ways people apply computing technology.
COOPERATIVE BUILDINGS Our approach is based in the idea of cooperative buildings, which we introduced several years ago.3 We adopted the term “building” (and not “spaces”) to emphasize real architectural environments as the starting point for system design. We described the buildings as “cooperative” to indicate, first, that they serve the purposes of human cooperation and communication and, second, that they employ active, attentive, and adaptive components. In our vision, the components themselves cooperate with the building inhabitants by adapting to changing situations and providing context-aware information and services.
Published by the IEEE Computer Society
December 2004
47
Figure 1. Example second-generation roomware components. Roomware integrates information and communication technology into room elements such as walls and furniture.
DynaWall
InteracTable
ConnecTables
ROOMWARE Cooperative buildings include what we called roomware components3-5 (www.roomware.de). Roomware integrates information and communication technology into room elements such as walls and furniture. The computer disappears as a device but its functionality is ubiquitously available via new forms of interaction. We have developed several roomware components, whose design and functionality have iterated several times. Figure 1 shows examples of secondgeneration components: • DynaWall is a large, touch-sensitive, information display and interaction device that is smoothly integrated in the wall. It measures 1.10 meters high by 4.5 meters wide and comprises three display segments, each driven by a separate computer. • InteracTable integrates a touch-sensitive plasma display into a tabletop. The table is 90 cm high with a 63 × 110 centimeter display. • CommChair combines the mobility and comfort of armchairs with the functionality of a pen-based computer. It has an independent power supply and connects to all other roomware components via a wireless network. • ConnecTable is a modular version of the CommChair. Users can either sit in front of it on a regular chair or raise its height to a stand-up position. They can also extend its workspace area easily by “connecting” several ConnecTables through sensors that detect ConnecTables placed next to each other.6 The extended workspace can support multiple users. To exploit the full interactive potential of roomware, we also developed a dedicated software infrastructure called Beach, the Basic Environment for Active Collaboration with Hypermedia.7 Beach is a synchronous groupware framework for orchestrating heterogeneous ubiquitous computing devices. It also enables new interaction forms and visualiza48
CommChair
Computer
tions that support face-to-face group work through touch- and pen-sensitive devices.
INTERACTION DESIGN FOR ROOMWARE Traditional computers often end up being more in the way than helpful in facilitating creative workshop-style meetings. They keep attention focused on human-computer interaction at the expense of human-human interaction. Meeting participants have instead long used large sheets of paper on walls or tables to support collaborative processes. Our main goal in developing roomware technology is to make computers disappear and to put support of human-human interaction back into the center of application design.
Software infrastructure Beach acts as a framework for collaborative software and also provides specific support for interaction with roomware components. This support includes synchronizing the displays of multiple computers composing large visual spaces like the DynaWall. Figure 2 shows an example Beach user interface at the DynaWall. The interface is geared toward creative group work in a synchronously orchestrated multicomputer setup of several roomware components. Such work requires freeflowing interaction between users and devices. Research has shown that creative meetings oscillate between phases of parallel work and sharing.8 For example, participants work in parallel during the early phases of idea generation or whenever they are taking notes or breaking up into subgroups. Participants share information by posting information on the wall, presenting an intermediate result, or taking a sheet of paper to a participant across the room. We designed the roomware and Beach user experience to support both kinds of activity as well as the transitions between them.
Localization principle In several design decisions, we facilitated both
Figure 2. Beach user interface at the DynaWall. The software infrastructure supports users working in parallel on a shared document in a coherent contiguous interaction area composed of three large screens controlled by separate computers.
Two-level BeachMaps
Toolbars MagNets clusters
parallel work and free-flowing collaboration on synchronously shared surfaces by applying the localization principle: The more local the influence of any user’s action, the less the chance of interference with other users. Five examples help to describe how we applied the localization principle in the design of the Beach user interface. Localized conflict detection. Conflicts occur when two people edit the same object in incompatible ways—for example, when one user changes the background color of an object to blue and another user changes it to red. All groupware systems must detect these kinds of conflicts to preserve a consistent view of shared documents. Different systems employ different granularity levels for this task. At one extreme, a system can detect conflicts at the level of the whole document, such as a Word document stored on a shared file system, but this makes conflicts likely when multiple users are changing the document. A finer-grained segmentation—for example, at the paragraph or even sentence level—dramatically reduces the chance for conflicts. Nevertheless, truly synchronous sharing of both information and working state supports creative cooperation much better than any form of taking turns.8 Accordingly, Beach supports true synchronous sharing by providing concurrent access to information objects, such as workspaces, and thereby allowing multiple users to work simultaneously on mind maps or a single meeting document. Beach’s conflict detection is very fine-grained, analyzing every access to the shared objects’ attributes for conflicting actions.9 Transactions are supported to guarantee consistency. Localized control. The pen is the main input medium for creating and annotating information objects on roomware components. It lets users
move freely around the room and still keep interaction “at hand.” Users can initiate commands through dedicated pen gestures—for example, to create a workspace or a hyperlink. Alternatively, they can open a toolbar anywhere on an interaction surface via a dedicated gesture, select a command from the toolbar, and drag the command to the targeted object. Initiating commands by gesturing or dragging toolbar buttons implicitly defines the target: The gesture crosses the object (or group of objects) and the button is dragged onto it. This represents an advantage over point-and-click interfaces, which require users to explicitly select objects before initiating a command and thus usually involve a global mode change. Localized modes. Beach’s user interface distinguishes strokes and gestures on the fly through an incremental gesture-recognition mechanism that provides instant visual feedback on how the system interprets the action.1,10 This supports a modeless interaction style, eliminating the need for usercontrolled switching between input and command modes. Scribble groups are one example of a localized, implicit interaction mode. This feature automatically combines notes or sketches that are entered together, keeping them together for manipulation as a whole later on. Beach automatically creates a scribble group when a user enters a nongesture pen stroke. New strokes are added to the group for as long as the group is active, and the strokes start or end within the group’s current bounds. A scribble group implicitly represents a temporal mode change: Strokes that Beach would otherwise interpret as gestures are interpreted as ink if entered within an active scribble group. For example, within a scribble group, a user can draw a “delete” stroke without deleting anything and can add dots December 2004
49
Figure 3. CommChair and DynaWall interaction. The user at a private CommChair puts a card on the public workspace displayed at the DynaWall and replicated at the CommChair as an active synchronized view.
50
without Beach interpreting them as “tap” gestures. A scribble group also represents a spatially bounded mode: Entering strokes in a workspace affects only its immediate surroundings. Again, there is no global mode change. Active synchronized views. Beach implements active synchronized views to support synchronous collaboration (1) across devices, (2) at multicomputer displays like DynaWall, (3) and with single-display groupware. In the case of cross-device collaboration, users can work remotely on what is displayed at another roomware component through a synchronously coupled scaled view. Figure 3 shows the user at a CommChair putting a card from her private workspace on the public DynaWall workspace via an active synchronized view, which allows easily accessing and directly manipulating the card and all other information visible on the public display. This is particularly effective in supporting group discussions about complex information. Participants can remotely annotate or point to what they want to talk about, rather than having to first explain what part of the public display material they are referring to or, failing that, walking up to the display to make a point. Active synchronized views also give appropriate perspectives to users working on the same workspace or document part on a horizontal single display. Another feature is decoupled browsing of any parts of a shared meeting document. For example, any user can enlarge an information object for closer viewing. This exemplifies how our implementation of active synchronized views relaxes the what-you-see-is-what-I-see principle for shared editing. We found WYSIWIS too restrictive for creative group work. Local tools. As opposed to documents, tools such as Beach document browsers, room views, and toolbars are local components, not necessarily shared among users. Decoupling tools and documents means that users working on the same workspace—that is, on the same part of a document—but with different devices can use their own toolbar set and position it according to their needs. This feature also relaxes WYSIWIS to give users more freedom and avoid possible interferences.
Computer
Display interaction Beach connects display surfaces in various degrees and ways to support creative meeting environments. Multicomputer, multidisplay components. In combination with the Beach software infrastructure, roomware uniquely provides coherent interaction for multicomputer components like the DynaWall. It lets several users work at the DynaWall in parallel and supports multiple forms of sharing as well. Beach provides a synchronized display of information objects that create a large homogeneous workspace across the DynaWall’s three displays, each being run by its own computer. The large workspace requires new interaction forms for sharing information objects. For example, in addition to dragging objects, users can “throw” them from one end of the DynaWall to the other. They can also “take and put” objects, walking from one screen to another. Cross-component sharing. Active synchronized views let users directly manipulate the content displayed at another roomware component—for example, the DynaWall. Test users often termed this capability “magic,” and it clearly facilitated the emergence of quick, playful group interactions that would not have been possible without this technology. Through active synchronized views, users can also bring up private workspaces at roomware components like CommChairs, use the private workspaces for taking notes, then easily exchange information with public workspaces. Another mechanism for cross-component sharing, called Passage,11 lets users link Beach information objects to arbitrary physical objects in the environment, such as a key chain. Single-display groupware. Beach implements workspace rotation on the horizontal InteracTable display. This allows multiple users to have a comfortable view of the work—just as they could with a sheet of paper. However, the Beach functionality supports cooperative work beyond what is possible when sharing a sheet of paper in the physical world. Users on two or more sides of the multitouch InteracTable can work on two or more active synchronized views of the same document and, because there is no global mode, they can
work in different interaction modes—say, gesturing and scribbling—while doing so. Dynamic multicomputer components. ConnecTables are mobile, networked, and context-aware information appliances that support individual and cooperative work as well as seamless transition between the two. Users can dynamically enlarge an interaction area for shared use by moving their ConnecTables closer to each other.6 Radio-frequency identification sensors and tags are integrated into the tabletops and trigger a signal that automatically couples the displays when they are placed next to each other. This temporarily creates a larger shared surface that lets the users work in parallel and exchange information simply by shuffling objects from one display to the other. This feature involves the same technology as the DynaWall, but adds the dynamic display adaptation based on the sensor information.
Creativity tools Research shows that structuring the shared idea space can improve a group’s creative performance.8 To achieve comprehensive coverage of the creative process, we designed a suite of three such tools for Beach: MagNets, BeachMap, and PalmBeach. Figure 4 shows two users interacting on the DynaWall through MagNets and BeachMap. MagNets. We developed “cards” based on a magnet metaphor to support successive bottom-up structuring of ideas—especially relevant in the early phases of idea-structuring processes. MagNets (magnetic cards for idea networks) come in two types. Element cards repel other cards upon creation. For example, dragging element card A on top of element card B pushes A away to a defined distance. This feature eliminates overlap on the display and also reflects the equal treatment of all ideas in the creative process.
Title cards, by contrast, attract other cards upon creation, thus supporting the progressive clustering of ideas that are related to each other. When an element card B is dragged over a title card C, it sticks to C and becomes attracting itself. Users can release it again by quickly moving it away from the magnetic cluster. MagNets clusters can also incorporate more than one title card. This feature lets users form heterarchic structures that allow items to belong to more than one category. MagNets functionality includes flexible visualization forms for card clusters, expanding and collapsing them as needed to direct the focus of attention or save display space. BeachMap. Complementing MagNets functionality, BeachMap gives users notation forms suitable to top-down information structuring. While MagNets introduced one-level hierarchies and heterarchies only, BeachMap can represent deeper hierarchical structures using a notation that radiates nodes in all directions from a central idea. Like MagNets, BeachMap provides flexible visualization options through gestures to expand or collapse the whole mind map as well as parts of it. Integrating MagNets cards or clusters as elements into BeachMap hierarchies is a two-step process. First, a double-circle gesture transforms hierarchic MagNets into BeachMaps. Then a simple stroke gesture connects the separate mind maps to form one. Other gestures can transform one-level mind-map hierarchies into magnetic clusters. These features support the integration of preliminary results no matter which notation was used to prepare them. PalmBeach. The third tool in the creativity suite, PalmBeach,8,12 supports asynchronous idea generation and preliminary structuring using personal digital assistants. Infrared communication between PDAs and the DynaWall provide seamless integration with MagNets and BeachMap.
Figure 4. Interaction with MagNets and BeachMap on the DynaWall. A MagNets cluster is expanded, then transformed into a BeachMap that is added to another BeachMap via a simple stroke gesture.
December 2004
51
Figure 5. Three design dimensions of roomware applications. The space is defined by four levels of abstraction, five models separating basic concerns, and three degrees of coupling with respect to distributed access to shared information.
Data model
Level of abstraction
Task level
Generic level
Model level
Application model User-interface model Environment Separation model of concerns Interaction model
Core level Independent model
Loosely coupled model
Tightly coupled model
Degree of coupling and sharing
SOFTWARE ISSUES FOR ROOMWARE Roomware applications must adapt interactions to the local context of each user and device in the environment. Traditional application models do not provide enough guidance to create software systems for these environments. We have developed an application model that accounts for the properties of roomware and other ubiquitous computing environments.7,10,13 As Figure 5 shows, the model distinguishes three design dimensions describing orthogonal aspects of roomware applications: the separation of basic concerns, the degree of coupling with respect to distributed access to shared information, and the level of abstraction.
Separating basic concerns A clear separation of software functions supports the flexibility needed to work across roomware devices. We distinguish five models that represent important software concerns basic to applications. The data model specifies the data types that users can create and interact with. An application model provides the functionality necessary to work with the data. These two models are independent of the hardware devices. The environment model supports context awareness by representing relevant parts of the “real” world. This includes the hardware devices to which the user interface must adapt: a description of the available devices, their configurations, and capabilities. It can also include other aspects of the environment that influence software behavior. For example, when Beach detects a change in the physical environment, such as ConnecTables establishing a common workspace, it triggers actions to reflect the change in the environment model. Separating the user-interface model from other software concerns enables the flexible definition of 52
Computer
alternative user-interface concepts suitable for different interaction devices. Because the design of traditional operating and window management systems is geared for use with a traditional desktop PC, the interface they offer has drawbacks when used with devices that lack a mouse and keyboard or that come in significantly different forms and sizes. For instance, if a menu bar is always at the top of the screen at a wall-size display like the DynaWall, it might be hard to reach. On the other hand, toolbars take up a lot of precious screen space on a small PDA-like device. Accordingly, a separate user-interface model can draw on information from the environment model. By explicitly modeling an appropriate user interface, software developers can address all issues related to the hardware and physical environment. This keeps data and applications independent of device and environment. While the user-interface model defines the elements of the user interface, the interaction model addresses the interaction and presentation issues in heterogeneous display environments like roomware. Software developers can therefore adapt these issues for different contexts. For example, they can use a pop-up menu instead of a list box to present an abstract “list-selector” to users, or, if no display is available, they can use a voice-based interaction.
Coupling and sharing Roomware environments are designed for collaboration, so software running on roomware components must be able to share the five software models reflecting the basic application concerns. In this sense, sharing means that all devices participating in the cooperative work must be able to access and modify the information described by all software models. Sharing is not a new concept, but ubiquitous
computing environments must extend the traditional view of it. In addition to data and application sharing,9,10 ubiquitous computing environments must share information about the physical environment, such as the presence of nearby users or other interaction devices. Applications must also distribute the user interface among several machines and complementing devices. When a visual interaction area crosses the borders between displays that are physically next to each other but connected to different computers, as with the DynaWall or connected ConnecTables—the user interface elements must move freely between the different displays. Figure 6 shows this functionality for ConnecTables. In this case, the different computer displays share user interface elements. Furthermore, if one user interacts with different devices at the same time, the user interfaces should be coordinated. This is only possible if the information about the currently used user-interface elements is accessible to all involved devices. For instance, a user sitting in a CommChair in front of a large DynaWall can easily view all information at the chair and the wall at the same time. Consequently, it is desirable to be able not only to modify the information visible at the wall, but also to control remotely the whole user interface.
Levels of abstraction Levels of abstraction is a common software engineering technique to reduce complexity and support interoperability. The core level in our model abstracts from the underlying hardware platform and encapsulates platform-dependent details. Functionality normally provided by the metaoperating system, middleware infrastructures, or groupware and user-interface toolkits resides at this level. At the core level, roomware applications require additional functionality unavailable from off-theshelf libraries or toolkits, such as support for sharing objects, multiuser event handling, and low-level device and sensor management. The model level provides basic abstraction to support higher-level abstractions—for example, the model-view-controller pattern as the basis for higher-level interaction. The generic level groups concepts that apply to an entire application domain. At this level, Beach defines generic document elements such as workspaces and scribbles. Finally, the task level defines abstractions unique to an application area—for example, mind mapping (as provided by BeachMap) is a task-level concept in a collaboration environment.
Figure 6. Shared user-interface model. Two ConnecTables linked to each other by proximity let users move interface elements seamlessly from one device to another.
e have deployed the roomware components in several pilot installations, both with and without the Beach infrastructure and the creativity tools. The deployments cover a wide range of situations in research labs and fairs as well as companies in the automotive and information technology industries. Wilkhahn, an innovative office furniture manufacturer and one of our industrial partners in the Future Office Dynamics research and development consortium (www.future-office.de), has commercialized and successfully marketed a subset of the roomware components. Customers perceive the roomware technology as networked furniture with interactive surfaces to connect people, thus meeting our goal of letting computers disappear and interaction evolve. Besides our own installation in the AmbienteLab (www.ipsi.fraunhofer.de/ambiente/), other installations include an exhibition at the German Occupational Safety and Health Exhibition (DASA) in Dortmund and a teaching project at the Swiss Federal Institute of Technology in Zurich, where roomware supports collaborative activities in courses on pharmaceutical chemistry. Customer feedback suggests that roomware environments foster creativity and innovation in communicationintensive processes like planning and coordination meetings. Our work in this area has shifted to include support for spontaneous encounters and other informal communication processes in office buildings. In the European Union-funded Ambient Agoras project (www.ambient-agoras.org) that is part of the “Disappearing Computer” initiative (www. disappearing-computer.net), we are also exploring how to support activities between remote sites using an innovative combination of ambient displays and mobile devices. ■
W
References 1. N.A. Streitz et al., “Roomware: Towards the Next Generation of Human-Computer Interaction Based December 2004
53
on an Integrated Design of Real and Virtual Worlds,” Human-Computer Interaction in the New Millennium, J.A. Carroll, ed., Addison-Wesley, 2001, pp. 553-578. 2. T. Prante, “Designing for Usable Disappearance— Mediating Coherence, Scope, and Orientation,” Proc. Distributed and Disappearing User Interfaces in Ubiquitous Computing Workshop, 2001; http:// ipsi.fraunhofer.de/ambiente/publications/. 3. N.A. Streitz, J. Geißler, and T. Holmer, “Roomware for Cooperative Buildings: Integrated Design of Architectural Spaces and Information Spaces,” in Cooperative Buildings—Integrating Information, Organization and Architecture: Proc. 1st Int’l Workshop on Cooperative Buildings (CoBuild 98), N. Streitz, S. Konomi, and H. Burkhardt, eds., LNCS 1370, Springer, 1998, pp. 4-21. 4. N.A. Streitz et al., “i-LAND: An Interactive Landscape for Creativity and Innovation,” Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI 99), ACM Press, 1999, pp. 120-127. 5. N.A. Streitz et al., “Roomware: The Second Generation,” Video Proc. and Extended Abstracts of the ACM Conf. Human Factors in Computing Systems (CHI 02), ACM Press, 2002, pp. 506-507. 6. P. Tandler et al., “ConnecTables: Dynamic Coupling of Displays for the Flexible Creation of Shared Workspaces,” Proc. 14th Ann. ACM Symp. User Interface and Software Technology (UIST 01), ACM Press, 2001, pp. 11-20. 7. P. Tandler, “The BEACH Application Model and Software Framework for Synchronous Collaboration in Ubiquitous Computing Environments,” J. Systems and Software, special issue on ubiquitous computing, Jan. 2004, pp. 267-296. 8. T. Prante, C. Magerkurth, and N.A. Streitz, “Developing CSCW Tools for Idea Finding: Empirical Results and Implications for Design,” Proc. ACM Conf. Computer-Supported Cooperative Work (CSCW 02), ACM Press, 2002, pp. 106-115. 9. C. Schuckmann, J. Schümmer, and P. Seitz, “Modeling Collaboration Using Shared Objects,” Proc. ACM Siggroup Conf. Supporting Group Work (GROUP 99), ACM Press, 1999, pp. 189-198; http://www.opencoast.org. 10. P. Tandler, Synchronous Collaboration in Ubiquitous Computing Environments, doctoral dissertation, Dept. Computer Science, Technical Univ. Darmstadt, 2004; http://ipsi.fraunhofer.de/ambiente/publications/. 11. A. Konomi, C. Müller-Tomfelde, and N.A. Streitz, “Passage: Physical Transportation of Digital Information in Cooperative Buildings,” in Cooperative Buildings—Integrating Information, Organization and Architecture: Proc. 2nd Int’l Workshop in Coop-
54
Computer
erative Buildings (CoBuild 99), N. Streitz et al., eds., LNCS 1670, Springer, 1999, pp. 45-54. 12. C. Magerkurth and T. Prante, “‘Metaplan’ für die Westentasche: Mobile Computerunterstützung für Kreativitätssitzungen,” Proc. Mensch und Computer 2001 (MC 01), Teubner Verlag, 2001, pp. 163-171. 13. P. Tandler, N. Streitz, and T. Prante, “Roomware— Moving Toward Ubiquitous Computers,” IEEE Micro, Nov./Dec. 2002, pp. 36-47.
Thorsten Prante is deputy head of the Ambiente– Smart Environments of the Future research division at Fraunhofer IPSI, Darmstadt, Germany, where he also teaches at the Technical University Darmstadt in both the Department of Computer Science and the Department of Architecture. His research interests include context-aware information management, user interfaces, and interaction design for ubiquitous computing and computersupported cooperative work. Prante is a PhD candidate and holds a Dipl.-Inform. in computer science. He is a member of the GI (Gesellschaft für Informatik) and the ACM. Contact him at prante@ ipsi.fraunhofer.de. Norbert A. Streitz is the head of the Ambiente– Smart Environments of the Future research division at Fraunhofer IPSI, where he also teaches in the Department of Computer Science at the Technical University Darmstadt. His research interests include cognitive science, human-computer interaction, hypertext/hypermedia, computer-supported cooperative work, and interaction design for ubiquitous computing in the context of an integrated design of real and virtual worlds. Streitz received a PhD in physics from the University of Kiel and a PhD in psychology from the Technical University RWTH Aachen. He chairs the steering group of the EU-funded proactive initiative, “The Disappearing Computer.” Contact him at streitz@ipsi. fraunhofer.de. Peter Tandler directs the computer-supported cooperative work (CSCW) research group of the Concert division at Fraunhofer IPSI. His research interests include CSCW, the integration of virtual and physical environments, software architectures and frameworks, object and component technology, and programming language designs. Tandler received a PhD in computer science from the Technical University Darmstadt. He is a member of the GI. Contact him at
[email protected].
