User Recognition and Memory Support in a Pervasive System

User Recognition and Memory Support in a Pervasive System Sarah Gallacher, Elizabeth Papadopoulou, Nick K. Taylor, M. Howard Williams School of Math & Comp Science, Heriot-Watt University, Edinburgh, UK {S.Gallacher, E.Papadopoulou, N.K.Taylor, M.H.Williams}@hw.ac.uk Abstract Besides providing users with the ability to manage the increasingly complex environment of devices and services in which they find themselves, pervasive systems can also offer new functionality. One example of this is the identification of other users when they are nearby. This extends beyond simple identification to various forms of memory support and provides opportunities for a range of new and innovative applications that will support users and enhance their experience. This is a natural part of pervasive systems developed using the notion of Personal Smart Spaces (PSSs). This approach has been implemented in a prototype pervasive system developed by the Persist project. This paper describes this notion in detail and explains how this functionality has been achieved using it.

1. Introduction The main aim of a pervasive system is to enable the user to manage the increasingly complex environment surrounding her [1]. As the number of devices with which she can interact, increases and the services available proliferate so the user needs more and more some form of additional support to help her interact with and control these resources [2]. To meet this growing need the research being conducted in the area of ubiquitous and pervasive computing has been expanding rapidly. Its importance is reflected in the fact that some of the “global challenges” of the next decade [3] have been identified in this area. Indeed many different researchers have been engaged in finding solutions to different aspects of the problem. Examples include the Adaptive House [4], MavHome [5], GAIA [6], Synapse [7], and Ubisec [8], all of which are focused on the problem of providing support in a fixed smart space – in particular, the Smart Home. Other types of system include Daidalos [9],

MobiLife [10], Spice [11], etc., which are aimed at supporting the mobile user. One approach that has been developed recently and used to build a prototype pervasive system is based on the concept of a Personal Smart Space (PSS). This concept enables the development of pervasive system architectures that bridge the gap between fixed and mobile pervasive systems. The application of this approach to the development of pervasive systems has been the main focus of the Persist project [12], a European research project with ten partners. This project has developed a pervasive system architecture based on the PSS and constructed a prototype platform to experiment with the ideas. A number of instances of this platform have been used to demonstrate its use in a range of situations. Basically, a PSS consists of a collection of devices connected via an ad hoc network to handle intra-PSS communications. The same ad hoc network can be used for interactions between different PSSs when these are encountered. In common with other pervasive systems, in order to be acceptable to the end-user a PSS-based system needs to be adaptable and able to be personalized to meet the different needs of individual users. In addition to providing the user with the necessary support to help her to manage the increasing complexity of the environment she finds herself in, pervasive systems can offer new functionality that is not available in conventional systems. This can be used to enhance the user experience, either directly or through new third party applications. This is true of systems using the PSS approach and Persist has been seeking to identify some of this functionality and the services that might emerge from using it. This paper reports on one such functionality, namely the identification of users through their PSSs. This is an essential feature of the PSS approach as it is needed at a lower level to maintain user privacy in a way that meets the user’s requirements. However, this feature can also be used at a higher level to provide information to the user. As a result it can be used as a

basis for developing a range of third party applications to provide memory support to the user. The approach described here has been implemented in the Persist pervasive system. The ideas are also being taken forward in a new project called Societies which aims to combine the ideas of pervasive systems with those of social networking to provide a powerful new platform. The latter will be subjected to real user trials by three user groups in different domains. The next section provides a background to Personal Smart Spaces. Section 3 describes how the identification of users is handled in Persist. Section 4 provides some examples of how this is used and section 5 introduces the Societies project. Section 6 concludes.

network. When this happens, each PSS must identify itself to the other. Depending on whether a PSS recognizes the other or not, and the degree of trust that the PSS associates with the other, it may share information about itself or its user or make available some of its services to the other PSS. User Interface PSS Framework

....

Context Mgmt

Learning

Personalisation

Proactivity

...

Service Run-Time Environment

2. Personal Smart Spaces The notion of a Personal Smart Space (PSS) has been proposed as a way of bridging the gap between conventional fixed smart spaces and pervasive systems developed for the mobile user. While the vision of many fixed smart space projects is to provide increasing levels of support for the user who inhabits the fixed space, when he/she steps outside of this island, all of this is lost. The result is that one will end up with islands of pervasiveness separated by voids in which support for pervasiveness is limited. The idea behind the PSS is that the user will be constantly covered by their own pervasive PSS, although the facilities it can offer at any point in time will vary depending on other PSSs in the neighbourhood. Formally, a PSS can be viewed as a dynamic space of connectable devices together with a set of services which are owned, controlled or administered by a single user. The devices in a PSS are connected by an ad hoc network and, although each device can operate independently, overall control makes them effectively operate as a unit. Thus a PSS can be realised as a personal area network which may interact with the networks of other PSSs when these are encountered. This has the advantage of not requiring any fixed infrastructure to be provided by Internet Service Providers or Mobile Network Operators, although it is able to take advantage of infrastructure when it is available. Thus users can deploy their own personal smart spaces, populating them with their mobile and fixed devices. A PSS belonging to a person is effectively a mobile pervasive system that surrounds the user at all times, providing control over the devices and services that belong to the user. However, when the PSS belonging to one user comes close to the PSS of another user, the two PSSs can communicate via a common ad hoc

Third Party Applications

Overlay Network Management

System Run-Time Environment Devices

Figure 1. The high-level architecture of a Personal Smart Space in PERSIST In the case of PERSIST the basic architecture used for developing PSSs [12] is shown in Fig. 1. The same architecture as is used for mobile PSSs can also be used for fixed PSSs. A fixed PSS consists of a set of devices and services controlled by the same software as a mobile PSS. This is generally the case for a smart building of some form – smart home, smart office, etc. Once again this type of PSS has an owner, although this may be an organization rather than an individual. Just as a mobile PSS may interact with another mobile PSS, it may also interact with a fixed PSS. Thus when a mobile PSS approaches a fixed PSS, each must identify itself to the other, and once again, depending on the outcome of this process, each PSS may provide the other with additional information or access to its services. This may extend to any number of mobile PSSs in the neighbourhood of a fixed PSS. Hence, for example, a smart office PSS may identify the users within it and offer their PSSs services such as printing services, data projector services, etc. Just as with other pervasive systems, an important feature is that of personalization. A PSS must be able to keep track of the preferences of its owner and apply these, when relevant, to the services it runs. As a consequence each PSS may display a slightly different behaviour from that of any other PSS, and this behaviour may change with time in accordance with

the preferences of the user. This notion of personalization applies to all PSSs, both mobile and fixed, and means that the behaviour of each PSS can be unique, tailored to meet the needs and preferences of its owner. These preferences can include the user’s preferred values of parameters under the control of fixed smart spaces. In the case of a fixed smart space there are two types of general services of interest. The first is the set of services that control access. These may apply to rooms in a building or to individual devices that a mobile PSS may wish to access. These services would be controlled by the preferences of the fixed PSS controlling the room or device concerned. The second set of services comprises those that control environmental parameters. These include ambient lighting, temperature and ventilation. These depend on the preferences of the fixed PSS combined with those of the users present in a potentially complex manner. Consider the case of temperature. Suppose that one has a room in which the devices present are controlled by a static PSS, and that these devices include the central heating/air conditioning units. When the room is empty, the preferences of the owner (the person or organization that owns the room and the fixed PSS in it) may be to conserve energy and switch off any heating/air conditioning. Thus the room PSS applies these preferences to the devices concerned. When a single user PSS enters the room, the preferences of that PSS are communicated to the room PSS and used to determine the actions necessary (heating/air conditioning) to provide the required temperature to satisfy the user’s needs. If more than one user is present and their requirements for temperature control are identical, the action to be carried out is straightforward. However, if more than one user is present and their preferences are different, a mechanism is required to resolve this conflict and determine what action to take. The simplest approach may be based on priority – the most senior member of staff present or anyone with a disability who has a particular need for temperature control is given priority over the rest. An alternative is to use some form of averaging procedure which attempts to find a compromise between the different needs of those present. One may also have a situation where, if a cleaner enters the room after hours, the PSS of the cleaner will communicate with the room PSS, but in this case the room PSS does not offer the cleaner’s PSS any environmental services and the cleaner’s preferences with regard to temperature will be ignored. This clean separation of functionality between the mobile PSS and the fixed PSS has three important advantages, viz .

(1) The preferences of the user are kept in his/her mobile PSS. This means that all monitoring of user’s actions in terms of controlling environmental factors wherever the user may be, are used to build up a set of preferences for the individual user based on all the rooms the user may have visited. On the other hand the room PSS keeps no information on the preferences of other users but only the default preferences of its owner. (2) The fixed PSS has control over which services are offered to which mobile PSSs. (3) The decision as to what action to take for any particular device in the case of conflicting preferences resides with the service in the fixed PSS responsible for controlling the device.

3. PSS Identification in Persist The process of PSS Identification within Persist can be broken down into three separate stages.

3.1. Establishing Links Between PSSs The basic idea underpinning communication between PSSs is that each PSS should broadcast an advertisement at regular intervals, advertising its existence. Then if a PSS encounters another PSS they can set up an ad hoc network linking them together. Likewise if a PSS encounters a second PSS, which is already linked to one or more other PSSs via an ad hoc network, the ad hoc network can be reconfigured to include the first PSS. This can be achieved using technologies such as JXTA. The latter has been used in Persist to provide the necessary infrastructure to support the creation and reconfiguration of ad hoc networks linking PSSs. In order to advertise its existence, a PSS needs some form of identification. However, to minimize the potential problems that may arise from violation of privacy, this needs to be as general as possible. Thus each PSS owner must be assigned a unique Digital Personal Identifier (DPI), which is the most general form of identifier available. Within Persist the allocation of DPIs is handled internally within the PSS by the Identity Manager. In addition a PSS may be prepared to allow other PSSs to access some of its services (including further information about itself) via the ad hoc network. For this purpose a user may have some services which it will allow any PSS to access (public services) and others which it will allow particular groups of PSSs to access (group shared services). For example, a user might identify as a group (“friends”) the set of PSSs belonging to his/her close friends and specify that they

can have access to services X, Y and Z. Then whenever a PSS is encountered which belongs to this group, the user’s PSS will inform it that it can have access to these services. In order to handle this, within the basic PSS Framework there is a component dealing with Grouping. This is responsible for setting up and maintaining the information on groups and using it to check whether a newly encountered PSS falls into one of the user’s identified groups. Thus when a PSS receives a PSS advertisement from the network detected by the Overlay Network Management (Fig. 1) - informing it of the presence of another PSS, it sends a “PSS joined” event. The Group Management component listens for events of this type, and when it detects one, it updates the appropriate groups within the Context Management subsystem. This is illustrated on the left hand side of Fig. 2.

3.2. Acting on Recognition of a Relevant PSS In the case of Persist the detailed decisions and interaction with the user on identifying a relevant PSS are taken by a third party application. This may be

triggered by the Proactivity module in the PSS Framework. The way in which this normally works is as follows. At the heart of the PSS Framework in Persist are the Context Management and Personalisation Subsystems. The latter is based on the use of user preferences, generally stored in a recursive rule format of form: if then or if then else where the actions may in turn be conditional. Within Personalisation the User Preference subsystem is responsible for creating and maintaining the set of user preferences for a user (via monitoring and machine learning or by direct user input). Each preference might be reactive in that it responds to a user action (e.g. whenever the user requests a news service, if the user is at home the system selects the television set). Alternatively it may be proactive, and initiate an action on the user’s behalf when a particular context arises. For example, suppose that whenever the user arrives home after work in the winter, he/she switches up the temperature on the central heating. If the learning system identifies this, a preference rule

Figure 2. Basic processes involved in PSS identification in PERSIST

could be set up to initiate this automatically for the user. The second component responsible for driving the proactivity mechanism is the User Intent subsystem. This is similar to user preferences but in this case the aim is to look ahead and predict future actions based on past patterns of actions. The distinction between the two subsystems lies in the immediacy of the actions. User preferences are an immediate mechanism in that when a particular context condition arises, a corresponding action is taken. User intent is concerned with sequences of actions leading to a consequence at some later time in the future. The Proactivity subsystem receives input from both the User Preference subsystem and the User Intent subsystem regarding the proactive actions recommended to be taken by the Proactivity subsystem. These inputs are passed to the Decision Maker component. Every input received by the Decision Maker has a confidence level field associated with it and each input provider has a similar confidence level which is calculated on the basis of the number of times the input it has provided has been successfully implemented. If the inputs recommend conflicting actions, these two confidence levels are used to decide which input should be implemented. As long as there is no conflict between the actions, the Decision Maker will trigger the Decision Implementer to initiate the recommended action. In doing so it also notifies the user and provides a means for the user to intervene if necessary and stop the action from proceeding. On the other hand, if there is a conflict between the actions proposed by the two (User Intent and the User Preference Management components) which cannot be resolved by this simple filtering, this is referred to a separate component in the Proactivity subsystem responsible for Conflict Resolution. In such a case, the latter component uses a conflict resolution algorithm to determine what, if any, action should be taken. The conflict resolution algorithm uses the confidence levels and the Quality of Context of the context attributes that led the input components to recommend these actions. If the conflict resolution algorithm is unable to resolve the conflict, the user is prompted to provide an answer using the Feedback Manager. The user’s input is then implemented and the User Intent and the User Preference Management components are informed of the success or failure of their predictions. This feedback is used to adjust the confidence levels of the user preferences and user intent predictions as well as that of each input provider. In this case one needs a set of user preferences to deal with identified PSSs. One way of handling this is as follows. Whenever the user creates a new group (via a GUI designed for this purpose), the user is asked

whether or not he/she wishes to be alerted when another PSS from this group is detected. If the user decides that this is what is wanted, a preference rule is set up to invoke a third party service and pass it an appropriate set of parameters whenever a PSS from this group is detected (typically the new DPI and the group id). At any point after this, if the Grouping component identifies a PSS from a group, it needs to update the Context Manager with the DPI of the PSS and the group into which it falls. This event will trigger the Preference Condition Monitor (PCM) component, which in turn will evaluate its preferences. If the condition part succeeds, the PCM will pass it to the Proactivity component to apply. These processes are illustrated on the right hand half of Fig. 2.

3.3. The Third Party Service What one does with the basic functionality provided by the PSS is realized by third party services. In particular such a service is chiefly responsible for two things: (1) It must determine what information to provide the user. Here one can envisage a range of different applications. The basic functionality would provide a user name and possibly some stored details on the user. More advanced applications might access a database, a social network site, etc., for additional information. (2) It must control how the information is communicated to the user. One obvious way of doing this at present is through an audio output to a mobile phone type earpiece. This technology is already in regular use by some users for telephone calls. In the future the information might be communicated via special spectacles which can display information to the user on the inside of the lenses. Much work has been done on this type of device and improved prototypes now exist although they are not yet commercially available.

4. Possible Applications Papadopoulou et al [13] outlined a typical application of this functionality in which a university lecturer might use it to identify students in her everyday contact with them. In this case an application might include links to a student database to check the courses a student might be taking and possibly his/her latest marks or the state of coursework that should have been handed in. A similar approach might be used to provide support for disabled users. In particular, in the case of users suffering from dementia or other health problems

giving rise to memory loss, this could be used to provide appropriate alerts and information on individuals the user knows when they are nearby. Similarly in the case of partially sighted or blind users, this could provide very useful support. Another slightly different application is that of personalized advertisement on public screens or billboards. There have already been several experiments in which personalised advertisements are targeted at a user as he/she approaches a particular shop or public screen. In 2009 Castrol used number plate recognition technology to read car number plates at one section of the road, retrieve the related car description and display the correct oil for that car on a large billboard further down the road as the correct car drove past [14]. In Tokyo the Digital Signage Promotion (DSP) project [15] has developed a number of intelligent advertisements in which cameras are used to detect the age and gender of people passing by and this is used to display suitable advertisements. Using the PSS approach it is possible to take this a step further and for advertisements to be targeted at users in a way that takes more account of the user’s needs and preferences. By using a fixed PSS to control the ad screen and a mobile PSS for the user, it is possible for the ad PSS to query the user’s PSS to determine the user’s preferences at the moment and select appropriate advertisements. Thus if the user does not wish to have ads targeted at them, they can set their preferences accordingly. If they are happy to receive ads, their preferences and current priorities (e.g. shopping for shoes) may be used to select appropriate ads that are more likely to interest them.

5. SOCIETIES The Societies project is concerned with two different paradigms: pervasive computing and social networking. As previously pointed out, the aim of pervasive computing is to provide the necessary support to enable the user to manage the increasingly complex environment (of devices and services) surrounding her. On the other hand social networking provides the support to enable users to interact with other users, to form groups and to communicate and share multimedia data via these groups. The aim of Societies [16] is to bring together these two different paradigms and to create systems that integrate the functionality of both. Just as the term Personal Smart Space (or PSS) was coined in Persist to describe the underlying concept on which the Persist platform was built so the term Cooperating Smart Space (or CSS) has been adopted to

describe the underlying concept in Societies and the platform that will be developed for the individual user. Once the platform has been developed, it will be evaluated in a set of three user trials. The three different user groups selected for this purpose are: (1) Students. A set of students in their third/fourth year of a Computer Science degree will each be given a CSS platform to use for an extended period. (2) Disaster Management. A set of professionals who meet together annually to simulate large scale disaster scenarios will assess the usefulness of the CSS in disaster management situations. (3) Enterprise. A collection of workers from industry will evaluate the usefulness of the CSS approach for handling support for delegates at a conference. The focus of this paper has been on the work done in Persist on identification of other relevant users when they are nearby. These ideas will be taken a step further in the Societies CSS where the role of groups or communities is fundamental. An example, which illustrates this, taken from one of the scenarios driving the development of the Societies system is as follows: “New student Harry sets out from his dorm room to attend his first lecture for his course in Computer Science. His CSS identifies his intent to attend the lecture and the navigation service is automatically started to direct him to the lecture room. On his way there Harry's CSS identifies another person from the community of first year Computer Science students who is nearby and who is also heading to the same lecture. Since they share intents and since the other person’s mood is ‘happy’, Harry's CSS suggests an introduction which he accepts. Harry's CSS shows him a picture of the other person and tells Harry his name is Tom. They begin to chat and Harry walks the rest of the way to the lecture with his new acquaintance.”

6. Summary and Conclusion Personal Smart Spaces have a number of features that can enhance the architecture of a pervasive system by providing novel functionality. This paper considers one of these, namely the identification of other users through interaction with their PSSs. This could lead to a number of new services to support different types of users. This paper describes how PSS Identification is implemented within the Persist framework. Figure 2 illustrates the basic processes involved. It also provides an outline of how one could make use of this functionality. One of the major challenges remaining is

that of recognising that a PSS is in the vicinity and determining whether or not it is relevant. The approach described here has been implemented in the Persist pervasive system and this functionality is currently undergoing evaluation by a group of users. Part of this uses the Persist pervasive system to control displays on screens surrounding the user for a set of real users. The results of this together with the ideas of PSSs will feed into the developments currently under way in the Societies project to develop a platform that combines pervasive computing with social networking.

7. Acknowledgment This work was supported in part by the European Commission under the FP7 programme (PERSIST and SOCIETIES projects) which the authors gratefully acknowledge. The authors also wish to thank colleagues in the two projects without whom this paper would not have been possible. Note that apart from funding these two projects, the European Commission has no responsibility for the content of this paper.

8. References [1] M. Satyanarayanan, “Pervasive computing: vision and challenges”, IEEE PCM, 8(4), 2001, pp. 10 - 17. [2] A. Zaslavsky, “Adaptability and interfaces: key to efficient pervasive computing”, in Proc. NSF Workshop on Context-Aware Mobile Database Management, 2002, pp. 2425. [3] A. Hoare and R. Milner, Grand Challenges in Computing Research, BCS, 2004. [4] M. C. Mozer, “Lessons from an Adaptive House”, in D. Cook & R. Das (Eds.), Smart Environments: Technologies, protocols and applications, 2004, pp. 273-294. [5] M. G. Youngblood, L. B. Holder and D. J. Cook, “Managing Adaptive Versatile Environments”, in Proc. 3rd IEEE Int. Conf. on Pervasive Computing and Communications (PerCom ’05), 2005, pp. 351-360. [6] B. D. Ziebart, D. Roth, R. H. Campbell and A. K. Dey, “Learning Automation Policies for Pervasive Computing Environments”, in Proc. 2nd Int. Conf. on Autonomic Computing (ICAC ’05), 2005, pp. 193-203. [7] H. K. Y. Si, “A Stochastic Approach for Creating Context-Aware Services on Context Histories in Smart

Home”, in Proc. ECHISE2005, Pervasive ‘05, 2005, pp. 3741. [8] J. Groppe and W. Mueller, “Profile Management Technology for Smart Customizations in Private Home Applications”, in Proc. 16th Int. Workshop on Database and Expert Systems Applications (DEXA ’05), 2005, pp. 226230. [9] M. H. Williams, N. K. Taylor, I. Roussaki, P. Robertson, B. Farshchian, and K. Doolin, “Developing a Pervasive System for a Mobile Environment”, in Proc. eChallenges 2006 – Exploiting the Knowledge Economy, 2006, pp. 1695 – 1702. [10] M. Strutterer, O. Coutand, O. Droegehorn, and K. David, “Managing and Delivering Context-Dependent User Preferences in Ubiquitous Computing Environments”, in Proc. Int. Symp. on Applications and the Internet Workshops (SAINTW ’07), 2007. [11] C. Cordier, F. Carrez, H. Van Kranenburg, C. Licciardi, J. Van der Meer, A. Spedalieri, J. P. Le Rouzic, and J. Zoric, “Addressing the Challenges of Beyond 3G Service Delivery: the SPICE Service Platform”, in Proc. Workshop on Applications and Services in Wireless Networks (ASWN ’06), 2006. [12] I.Roussaki, N. Kalatzis, K.Doolin, N.K.Taylor, G.Spadotto, N.Liampotis and H.Williams, “Self-Improving Personal Smart Spaces for Pervasive Service Provision”, in Towards the Future Internet, Eds. G. Tselentis, A. Galis, A. Gavras, S. Krco, V. Lotz, E. Simperl, B. Stiller and T. Zahariadis, IOS Press, 2010, pp. 193-203. [13] E. Papadopoulou, S. Gallacher, N. K. Taylor and M. H. Williams, “Personal Smart Spaces as a Basis for Identifying Users in Pervasive Systems”, in Proc. Int. Workshop on Ubiquitous Service Systems and Technologies (USST 2010), IEEE CS Press, 2010, pp. 88 – 93. [14] Article on Castrol personalised billboards: http://www.honestjohn.co.uk/news/parts-andaccessories/2009-09/castrol-checks-your-oil-on-the-road. Accessed June 2011. [15] Digital Signage Promotion Project: http://www.google.com/hostednews/afp/article/ALeqM5iDd 1xzYx7CaahlxkLnvo4Xtcksug, Accessed June 2011. [16] S. Gallacher, E. Papadopoulou, N.K. Taylor, F.R. Blackmun, M.H. Williams, I. Roussaki, N. Kalatzis, N. Liampotis, D. Zhang, “Personalisation in a System Combining Pervasiveness and Social Networking”, in Proc. 1st Workshop on Social Interactive Media Networking and Applications (SIMNA 2011), ICCCN 20, Hawaii, 2011.