Paper Under Review
Experiential Computing Ramesh Jain School of Electrical and Computer Engineering, and College of Computing Georgia Institute of Technology Atlanta, GA 30332 0250
[email protected]
Introduction We need to find unique and creative solutions to the emerging technological challenges that are paradoxically the result of tremendous progress in Information and Communication Technology (ICT), sensors, embedded computing, and related areas. Computing began with a focus on data, and in the last two decades has shifted to a focus on information and communication. We believe that to address the requirements of emerging applications of this century, the focus must shift to experience. We are developing a body of computing approaches with this focus, which we call experiential computing, for emerging applications. The time is ripe for experiential computing. Decision makers at all levels now routinely use computers, and they need insights—insights that come from experience and experimentation. To develop relevant insight into situations, people must be able to explore and experience an event from multiple perspectives, and they should be able to revisit that event as many times as they need to obtain the desired insight. In an experiential computing environment, users directly apply all their senses to observe data and information of interest related to an event. Further, each user can explore the data according to his particular interests in the context of that event. Experiential environments free people from the tedium of managing enormous volumes of disparate heterogeneous data. The systems don’t try to interpret the experience; they just provide an environment that people can use to understand an event easily and naturally, somewhat the way computer-aided design systems provide designers with an environment in which to design. Most current computing techniques are fundamentally based on the alphanumeric data. Emerging applications require spatio-temporal, live, heterogeneous, multimedia data from multifarious sources. For these applications, we must rethink our approaches. To understand the needs and requirements of emerging applications in this century, let’s consider a few emerging applications. Situation Monitoring and Data Warehouses: Applications such as business activity monitoring, bioinformatics, and homeland security must draw from a large network of disparate data sources, including databases, sensors, and systems in which data is entered manually. All these sources produce streams of data for various applications: seismologists can use the data to warn of impending earthquakes, businessmen can use it to monitor the status of activities at different locations, and to analyze the causes of past events. In all such applications, real time data analysis must be combined with real time data assimilation from all sources to present a unified model of the situation in intuitive form. Techniques and tools developed for payroll databases are not adequate for this environment. Data mining techniques are suitable when a hypothesis has been formed, but tools must first help in generating that hypothesis. Personalized Event Experience: Most current storytelling, be it for education or entertainment, presents an experience from the storyteller’s perspective. User experience is secondary and bound by the storyteller’s perspective. To develop relevant insight into situations, people must be able to explore and experience an event from multiple perspectives, and they should be able to revisit that event as many times as they need to obtain the desired insight. A user who wants to understand from his own perspective how an event evolved would benefit from seeing a fast animation of statistics related to different categories or viewing a particular event from several perspectives. He might also listen to the description of the same event by different commentators. All these operations are queries of different types that the event record must answer by presenting the results in a form the user finds familiar and can use instantly.
© Ramesh Jain
Paper Under Review EventWeb: The current Web is a web of pages prepared predominantly in document mode. Language is a big barrier in a text-based Web. A Web of events is universal because it lets users experience the event in the medium they find most comfortable: text, video, audio, or some combination. In this web, each node represents an event – past, current, or future. People would put events on the Web by connecting one or more cameras, microphones, infrared, other pertinent sensors, databases, and related textual information that will let a visitor experience this event. For each event, all information from different sensors, from different documents, and other sources is unified and presented to a user independent of the source. The EventWeb will be independent of language and will have much wider appeal among the 90% of the population that does not have access to current ICT because of language and educational barriers. Folk Computing: The income disparity between the richest and poorest countries is accelerating. This disparity is at the root of the digital divide—an economic and social gap that is not only unfortunate for the have-nots, but also dangerous for humanity as a whole. Information and timely communication are essential for the proper use and distribution of rare resources, and technology is and always has been the vehicle for communication. Countries without it are finding themselves on the dark side of the digital divide. To bridge the gap, we must package ICT so that it reaches everyone, even the illiterate and poor. Folk computing is computing for the masses that reflects a deep understanding of these users’ unique needs and uses technology relevant to their living conditions, socioeconomic status, education, and language. It is not simply recycling products that work in the developed world. Existing products depend heavily on a user’s ability to read and write some language, usually English. Many people in developing countries are illiterate in their own language. A more realistic solution for these users is to create information and communication devices that use audio, video, and tactile input and output and let people work in their natural environment. This is far beyond interface design. It is an all-encompassing technology that is useful for the literate, the handicapped, and the functionally illiterate. We can easily observe important data characteristics and operational trends in the above applications and can expect these to become more pronounced in the next decade: 1. Spatio-temporal and live data streams are the norm rather than the exception. 2. The holistic picture of an event is more important than examining silos of isolated data. 3. The user is interested in insights and information that are independent of the medium and data source. 4. The user does not want information that is not immediately relevant to his particular interests. . 5. Exploration, not querying, is the predominant mode of interaction. Exploiting the Human-Machine Synergy The evolving nature of data sources and desired operations can be captured in the matrix below. These relationships have profound implications for ICT. For example, if databases are excellent for getting precise information from a single data destination, why do we persist in using them to get insights from multiple sources? Visualization environments and interactive tools are useful in gaining insights from a precise source, but most of this century’s emerging applications fall in the top right quadrant: To gain insights from multiple heterogeneous sources, we need an experiential environment because it unites disparate data sources and frees decision-makers to explore their perceptions.
Insight
Information
Visualization
Experiential Environments
Databases
Search Engines
Single Data Destination
Multiple Data Destinations
Current information environments actually work against the human-machine synergy. Humans are very efficient in conceptual and perceptual analysis and relatively weak in mathematical and logical analysis;
© Ramesh Jain
Paper Under Review computers are exactly the opposite. In an experiential environment, users directly use their senses to observe data and information of interest related to an event and they interact naturally with the data based on their particular set of interests in the context of that event. Experiential environments have following important characteristics:
♦ They are direct: This environment provides a holistic picture of the event without using any unfamiliar metaphors and commands. People are in a familiar environment and use natural actions based on commonly used operations and their anticipated results. In experiential environments, a user is presented data that is easily and rapidly interpreted by human senses and then the user interacts with the dataset to get a modified dataset. ♦ They provide the same query and presentation spaces: Most current information systems use different query and presentation spaces. Consider popular search engines. They provide a box to enter keywords and the system responds with a list of thousands of entries spanning over hundreds of pages. A user has no idea how the entries on the first page are related to the entries on the 13th or how many times the same entry appears or how the entries on the same page are related to each other. Contrast this to a spreadsheet. User articulates a query by changing certain data that is displayed in context of other data items. User’s action results in a new sheet showing new relationships. Here query and presentation spaces are the same. These systems are called What-You-See-Is-What-You-Get or WYSIWYG. ♦ They consider both the user state and context: System should know the state and context of the user and present information that is relevant to this particular user in the given state and context. People operate best in known contexts and do not like instantaneous context switching. Information systems, including databases, were designed to provide scalability and efficiency. These considerations led to design of systems that were state-less. The efficiency of relational databases is the result of this decision. This is the reason that most of the search engines on the Internet are so dissatisfying to many people. They don’t remember the state of the user. ♦ They promote perceptual analysis and exploration: They should promote perceptual analysis and exploration. Because users involve their senses in analyzing, exploring, and interacting with the system, these systems are more compelling and understandable. Text based systems provide abstract information in visual form. Video games and many simulation systems are so engaging because they provide powerful visual environment, sound, and in some cases tactile inputs to users. Components of Experiential Systems Experiential systems are very different from current information systems, which means that we must rethink nearly every system component. Figure 1 shows the main components of an experiential system. We briefly describe these components and identify research challenges: Data Acquisition, and Analysis: An experiential environment draws from many disparate sources, from text to sensory input. Much emphasis needs to be placed on semantic and contextual processing of heterogeneous data. Also strong middleware for addition, deletion, or modification of disparate data sources and for processing these sources for specific operations is required. Assimilation: Sophisticated control and communication systems assimilate data from disparate sources using strong domain model based techniques like Kalman filtering. The mathematical domain model is at the heart of these systems. Development of such strong model based techniques that can be generalized by using powerful domain modeling tools are essential to build next generation systems that will use live heterogeneous data. The domain models here are more complex and must deal with the so-called signal-tosymbol gap. Unified Indexing: Current indexing techniques result in data silos based on the type of data. What is required is a unified indexing approach that indexes disparate data sources based on the domain semantics rather than media type. Such techniques allow representation of heterogeneous knowledge by using domain semantics. This facilitates management of tacit knowledge, opening up many new applications of knowledge management approaches. Exploration environment: We discussed characteristics of experiential environments above. It is not possible to develop experiential environments just as simple interface mechanisms. Current query based
© Ramesh Jain
Paper Under Review environments like search engine interfaces have been successful with current generation of computer users. The applications discussed above require holistic pictures and exploratory environments. Personalized Presentations: If computer technology wants to cross the chasm and become usable by common people, then we need to design experiential environments. Personalized presentations and exploratory environments require consideration not only of user interface issues, but also media synchronization, media summarization and other difficult engineering issues. These issues have close relationship with data organization and processing issues. In summary, the real technical challenge in developing experiential environments is to transform cyberspace-based information systems to include the spatio-temporal world and its physical laws.
Data Stream
Sensor Host
Entry, Assimilation, Indexing
Personalized Presentation
Video
Server
Contextual
Exploration
Text
Assets
Index
Database
Data Sources
Processing and Storage Architecture
Users
Figure 1: Architecture of an Experiential Computing System. Early Experiential Systems We developed an approach for implementing experiential systems that uses event-based domain model and the metadata to construct a new index that is independent of the data types in different data sources. An event is defined as a significant occurrence or happening at a single point in space-time. An application domain can be modeled in terms of events and objects. Events are hierarchical and have all desirable characteristics that have made objects so popular in software development. In fact, events could be considered objects that have time and space as primary attributes. The unifying index is called eventbase. Eventbase contains all information about events that is assimilated from different sources and also links to original data sources. The links to data sources are particularly important to present appropriate media in the context of events. A user directly interacts with the eventbase. This has several advantages including pre-processing important information related to events and objects based on domain knowledge; presenting information using domain based visualization, and providing a unified access to all information related to an event independent of the time the data became available or was entered in an associated database. As we see in the following, the above help in providing an experiential environment for a user to access the information. We developed EventViewer to experiment with the ideas presented above related to the experiential environment. An application screen of the EventViewer is shown in Figure 2. EventViewer offers
© Ramesh Jain
Paper Under Review multidimensional navigational and exploration capability. For an event three basic characteristics are its name and class, the location where it took place and the time when it took place.
Figure 2: Screenshot of an EventViewer for Demand Activity Monitoring. A user can navigate through the class ontology hierarchies. Navigation through the location and time is either using zooming or moving along different directions. These traversals are very similar to those in video games. One can select parts of a map ranging from part of a room to the map of the world. Similarly, on the time line, one could be in the range of microseconds to centuries, or even light years. Once a user selects event classes, the map in the location space, and time on the time line, the system presents all events and their selected attributes at three places: a list in the space provided for event list, symbols displayed on the location map, and symbols displayed at appropriate time on the time line. These three displays are tightly linked. Thus, if you select an item in the list, it will be selected, and displayed as selected by showing a color change, in the location and time spaces also. This event search approach is WYSIWYG search. By displaying events on a map as well as on a time line, the context of the events is maintained and displayed to a user. A user can then refine the search criteria and as the criteria are refined, the results also change. This allows a user to experiment with the data set as he sees fit and develop insights and form hypothesis. It is conceivable that this feature can be linked with some data mining tools to explore large data warehouses. If a user is interested in knowing more about an event, she can explore that specific event by double clicking on the event in any of the three display areas. This results in presenting to the user details of the event, all the data sources (like audio, video, or text), and any other characteristics. Here we briefly describe two applications to give an idea of what could be done in this environment. Demand Activity Monitoring: In a modern enterprise, line managers need to identify the potential problem areas, how they got there, and how they can change things in the future. The focus is on “performance indicators” which are the discrepancies between planned and actual performances, and the interrelationships between these performance indicators and available infrastructure, environmental factors, promotional efforts, etc. Normally, it is not sufficient to find just a potential problem; the context in which
© Ramesh Jain
Paper Under Review the problem occurred is important. This context includes the related activities as well as a historical perspective. The key activities monitored are sales and inventory (monthly, daily, and hourly available inventory for different geographic regions). Figure 2 shows a screen shot of EventViewer for this application. Performance indicators for each activity is mapped to red, yellow, and green based on domain specific criteria. Football Highlights: We implemented an application to provide Football fans an environment to explore a particular game and find all interesting parts of the game. A game can be modeled as an eventgraph with several levels of event hierarchies and transitions from one event to the other determined by what happens in the game. We had video (including audio) from multiple cameras, play-by-play information as generated by some companies and made available as a data stream, and access to player and statistics database. The system parsed the play-by-play data stream, applied the rule base to this and prepared an even base for the game. The event base was presented to a user as a time machine shown in Figure 3. A user could go to any time moment and see all the statistics and other relevant data in the game. A user could filter events based on his choice and see them in standard football representation. By double clicking on a play, more information could be obtained about the play including seeing the video of the play. Thus, a user could watch scoring plays resulting in touchdowns by his favorite team. He could see videos, from different angles for his favorite plays. This environment offers many navigational features and is very compelling for football fans.
Figure 3: Experiential environment for Football fans to enjoy multimedia presentation in a timemachine format.
© Ramesh Jain
Paper Under Review Conclusion Experiential computing is the next natural step in computing to deal with the emerging problems. By developing techniques to deal with spatio-temporal live data streams, computing will address many new real world problems. On the other hand, experiential environments will bring computing to masses all over the world by providing natural relatively language-independent interfaces to computing devices. The techniques and applications presented in this paper are very early steps in this exciting direction. Bibliography 1. Don Norman, The Invisible Computer, MIT Press, 1999. 2. John Seely Brown and Paul Duguid, The Social Life of Information, Harvard Business School Press, 2000. 3. A. Katkere, J. Schlenzig, A. Gupta, and R. Jain, “Interactive video on WWW: Beyond VCR like interfaces,” Computer Networks and ISDN Systems, Vol. 28, pp. 1559-1572, 1996. 4. David McCoy and Milind Govekar, “Evolving Interaction Styles in Business Activity Monitoring”, Gartner report, March 2002. 5. Y. Roth, R. Jain, "Knowledge Caching for Sensor-Based Systems." Artificial Intelligence, 2-24. 1994. 6. B. Joseph Pine and James H. Gilmore, The Experience Economy, Harvard Business School Press, 1999. 7. Simone Santini and Ramesh Jain, “Semantic Interactivity in Presence Systems”, Proceedings of the Third International Workshop on Cooperative and Distributed Vision, Kyoto, Japan, November 1999. 8. Ramesh Jain, “Events in Heterogeneous Data Environments”, Proc. International Conference on Data Engineering, Bangalore, March 2003. 9. Ramesh Jain, “Semantics in Multimedia Systems”, Proceedings of International Conference on Multimedia Modeling, Taipei, January 2003. 10. Ramesh Jain, “Digital Experience”, Communications of ACM, 44 (3), Special Issue on the Next 1000 Years, March 2001. 11. David Sonnen and Henry Morris, “A Unified View of Information across the Enterprise: Moving Beyond the Data Integration Paradigm”, IDC report, January 2001. 12. David Sonnen and Henry Morris, “Addressing requirements for unified data access”, IDC report, August 2001.
© Ramesh Jain
