User Experience evaluation

International Journal of Intelligent Engineering Informatics Special issue on: "Hybrid Environments and the Real World Internet"

Evaluating the user experience of a mobile user in a smart city context 1 Charalampos RIZOPOULOSa, Katerina DIAMANTAKIa, Dimitris CHARITOSa, Vassileios TSETSOSb, Petros KOVATSISb Iouliani THEONAc a Faculty of Communication and Media Studies, National and Kapodistrian university of Athens, Athens, Greece b Mobics Ltd., Athens, Greece c School of Architecture, National Technical University of Athens, Greece

Abstract Mobile and location-based technologies provide efficient ways of enhancing a city’s smart transport infrastructure by effectively augmenting its functionality. Also, mobile phone users may act as mobile “sensors” that provide transportation-related data, thus supplementing stationary sensor networks that provide data such as vehicle position, parking space availability, traffic density, atmospheric conditions, etc. and constituting a hybrid sensor ecosystem that incorporates both human and nonhuman agents. This paper outlines a research project that aims to enhance the User Experience of an advanced transportation services platform by incorporating simple game elements and mechanics in system design, under the assumption that a non-game platform that enables gameful experiences will act as an intrinsic motivator for users by increasing the inherent appeal and the engaging character of various activities. The paper describes the system’s evaluation methodology and process, outlines the analysis of preliminary results, and provides some insights regarding the impact of system parameters on the perceived satisfaction of mobile map-based applications, as well as the structure of the concept of User Experience as it pertains specifically to such applications.

Keywords: Smart city, location-based transportation service, locative media, user experience evaluation

1. Introduction

The application of mobile ICTs in areas as diverse as social media, tourism, crime prevention, energy-saving and public transportation has gradually led towards a realization of the smart, sustainable cities vision, in recent years. More specifically, the utilization of mobile and location-based technologies is considered as one of the most efficient ways of enhancing a city’s smart transport infrastructure, by making 1

This paper describes work undertaken in the context of the SmartSantander project, partially funded by the European Commission under contract number ICT-2009-257992 (FP7)

use of the augmented functionality and wide use of GPS and Internet-enabled smartphones. Citizens equipped with smartphones can act as mobile nodes for providing data related to the city’s transportation services in a wider network of sensors. Additionally, dedicated sensors may be employed in order to provide data, such as position of vehicles, availability of parking space, traffic density, atmospheric conditions, etc. Such a system incorporating both human and non-human agents, as two distinct yet interrelated sources of urban data could be considered as a hybrid sensor ecosystem. This paper outlines a research project, funded under the FP7 framework, which aims to enhance the User Experience of an advanced transportation services platform by incorporating simple game elements and mechanics within the very design of the platform (Salen & Zimmerman, 2004 and Laurel, 1993). This decision was based on the hypothesis that a non-game platform that provides affordances for gameful experiences will intrinsically motivate users and support their overall engagement and value creation. As has been frequently reiterated in the last years, gamified designs encourage technology adoption, motivate user behavior and increase user satisfaction (Deterding et al., 2011 and Malone, 1982. This may be attributed to the fact that game-like activities and ludic interfaces tend to make technology more `engaging, by encouraging users to adopt desired behaviors, by taking advantage of humans’ generally positive predisposition toward game-like or playful activity and by leveraging people’s natural desires for competition, achievement, self-expression and enjoyment (Huotari & Hamari, 2012 and Marczewski, 2012). 2. The project’s goal was to motivate citizens to intensify their use of public transport as the preferred mode of transport in the city and to participate in the collection of related data (traffic, incidents, and environmental conditions) by utilizing the aforementioned technologies, as well as to enrich the system with subjectively meaningful location-specific content. By providing the system with information about the traffic and transport conditions in certain locations, citizens are more likely to feel a sense of engagement in a common goal. Additionally, an ultimate objective of the MITOS platform was to enable the enrichment of specific locations with meaningful comments. It was hypothesized that, through this enrichment, various indirect forms of social interaction amongst some of these citizens were expected to emerge, ultimately leading to an improvement of the social use and appropriation of these technologies, which are worth investigating in a systematic and rigorous fashion. Therefore, the ultimate aim of the project was to foster bottom-up community intelligence and active citizen participation in urban matters.

2. Description of the MITOS system and experiment in the context of the

SmartSantander project The project discussed in this paper is a part of the experimental evaluation process of a larger scale project titled SmartSantander. The main goal of the SmartSantander project 3 is the

2

On a methodological side note, it is important to note that gamification may alter the activity it is applied onto from the user’s perspective; a “gamified” application has incorporated game elements into its structure, and these elements may need to be taken into account when evaluating non-game-related aspects of the application. In other words, the practice of gamification may assume the status of the “third variable”. 3

www.smartsantander.eu

creation of a European experimental test facility for the research and experimentation with architectures, key enabling technologies, services and applications for the Internet of Things (IoT) in the context of a smart city (Sanchez, L et al, 2011). The project work team has designed, implemented and deployed all required infrastructure for enabling smart city services in the city of Santander and some other facilities across Europe. The project considers the deployment of sensors in Belgrade, Guildford, Lübeck and Santander. The middleware services developed provided all functionality required so that experimenters were able to deploy their services or algorithms and to draw interesting conclusions regarding smart city services, IoT features etc. As part of this project, an experiment for Intelligent Transportation Services in the context of the smart city titled MITOS 4 (Multi-Input TranspOrt planning System) has been designed and performed. The services and applications deployed and demonstrated in the context of MITOS were: •

A Web portal for transportation information that will be essentially used as a city guide for citizens or tourists (see Figure 1). The core functionality provided by the portal is: o

Location-based services such as traffic intensity, geocoding, visualization of points of interest (POIs), search for transportation information, incident notification, location-aware advertising, real-time parking availability, etc.

o

Optimal-routing for multi-modal transportation. The itineraries of bus and any other available means of transport are correlated with user preferences in order to calculate and propose the best available trip for a given origin and destination.

o

A novel parking prediction service that can inform user about open street parking availability in the near future.

•

A mobile application for advanced transportation services. The application provides all type of information required by commuters and enables intuitive ways for searching routes and geo-referenced information based on the personal preferences or abilities of the user. Moreover, it is used for participatory sensing purposes.

•

A simple and pragmatic gamification approach to participatory sensing, where users contributing with information can gain credits.

The MITOS platform exploited existing sensor sources in order to deliver advanced services to the end users. These sources were:

4

o

The “human sensors”: users act as sensors that provide relevant traffic and travel information in the form of free text or predefined messages (e.g. “heavy traffic”, “too much noise”, etc.) and/or image. In this sense, the act of gathering data on the city’s everyday functioning becomes a more decentralized process.

o

On-street parking space availability. The existing street sensors were used in order to assist commuters in their trip planning and facilitate combined mobility modes (e.g., car and bus) by offering information about parking availability.

The MITOS application is publicly available as a Web and Android application at mitos.smartsantander.eu

o

Traffic intensity estimation. The existing traffic data is fed into the platform in order to enable better and more accurate transportation guidance to citizens.

Figure 1: MITOS Web portal and mobile application

2.1

Overall platform architecture

The MITOS platform architecture is depicted in Figure 2.

Desktop UI

Santander Data

Data retrieval service

MITOS DB SmartSantand er Middleware Datalogger

Parking algorithms

Mobile UI

Routing algorithms

User management

Transport game logic

Admin console

Event push engine

Users

MITOS Feed Geo-ad service

MITOS

Figure 2: MITOS platform architecture

A short description of its main components follows. Parking algorithms: the algorithms for short term parking slot occupancy prediction. More details about the algorithmic framework and the research results can be found in (Vlahogianni et al. 2014). Routing algorithms: the algorithms that calculate (personalized) multi-modal routes based on the available means (e.g., bus, metro). They are implemented with open source tools and can take into consideration both static (bus itineraries) and dynamic data (real-time bus positions). Transport game logic: the logic of the gamification approach in terms or rules, actions allowed and user scoring. In a few words, the user gains an amount of credits for any contribution she makes to the platform: reporting an event, commenting and rating other users’ reports, uploading photos taken from the streets, etc. MITOS feed: a basic component for orchestrating user interaction. It visualizes all social interaction and provides the user interface elements for enabling such interaction. Visualization is based on a micro-blogging approach, suitably adapted for mobile and desktop users. A sample screen of this feed is shown in Figure 3. Event push engine: the message dispatching engine that allows prompt user notification, either through the MITOS feed or push notifications (for mobile users). Geo-ad service: the mechanism for enabling management and display of geo-located advertisements in MITOS applications. Such advertisements are necessary as a commercial aspect of the system. Revenues from advertisement can support the platform’s maintenance and evolution. Desktop and Mobile User Interfaces (UI): the graphical representation and visualization of the MITOS data, actions and results of service execution. User management: the components dealing with the management of the MITOS users profiles (basic demographics, score, etc.). Users are anonymous, but they have to be identified somehow (for experimental purposes). Data-logger: this component is responsible for logging all interesting user activity and extracting statistics that are during execution and evaluation of the experiment. Admin console: it provides access to various graphical administrator interfaces such as moderation of user messages, update of transport elements, access to usage statistics. Data retrieval service: a service periodically requesting sensor data from the SmartSantander middleware and storing them to the MITOS DB. It implements the SmartSantander APIs for access to the sensor data. Moreover, this service retrieves data from other third-party data sources, such as static bus route timetables, traffic intensity estimations, etc. It can be easily extended to integrate fleet management systems, too. MITOS DB: a relational database that stores all information required by the services (e.g., parking sensor data) as well as all data resulting from MITOS service invocations (data access logs).

Figure 3. MITOS live feed sample

2.2

Social interaction and gamification elements of the platform

The design of the MITOS platform was also based on the hypothesis that the use of the system might lead to a certain form of emergent social interaction. This indirect form of interaction would be the result of the introduction of geo-annotations by users, which could be used for geolocating textual or visual information and ultimately afford users the opportunity to communicate with other users in an indirect manner. From a methodological standpoint, this use case is more exploratory in nature, given that social interaction is a possibility but not a certainty. 2.3

Use case scenarios for the MITOS system

According to the above, certain aspects of the MITOS system functionality are described by the following scenarios: Passenger A, travelling on a certain bus line, uses the proposed system’s interface, to report a traffic delay caused by a road construction at a busy central junction of the city of Santander. To submit the issue she chooses from a list of preset sentences relevant to various nuisances (i.e. heavy traffic, traffic delay, closed road etc) that allow for quick report, and also tags the exact location of the problem by using the geolocation functionality of the system. Other passengers using the system are given the option to confirm and to further comment on the incident via a microblogging utility that allows communication via short messages (max 150 characters) as well as the uploading of a related image. Through this review process, the need of central moderation of contributions to the platform is decreased and a spontaneous, indirect kind of temporary social interaction emerges. If the report of Passenger A is confirmed by a certain number of people (to be decided at a later stage in the design of the system), it then becomes more prominently placed within the interface of the mobile user of the system, given the interest it raises, thus helping those other commuters avoid the problem and effectively plan their travel. Passenger A, being the first one to report the nuisance, is granted credit

points by the system. When she reaches a certain amount of credit points, she can redeem them in real benefits, relevant to the public transport, such as discount on a certain number of bus tickets, or in a number of other services. Through this reward system the user is motivated to keep participating in this collective sensing and contributing to the platform, enabling a real time update on the everyday working situation of the public means of transportation in the city of Santander. In another instance, Passenger B decides to move towards his destination, for example towards his office, choosing one of the quickest bus routes provided by the system. At a park along his route there is an outdoors event taking place. He snaps a photo and he uses the microblogging utility of the system to upload it. Other passengers, riding on the same bus line, or using other means of transportation further comment on the image and the event, thus contributing to the emergence of a spontaneous, temporary, synchronous and/or asynchronous social interaction. Tourists may also log in the system, so that they can become informed about such an event and potentially decide to drop by. Local businesses may grasp the advertising potential of the system, and further add points of interest to the maps of the platform relating to their business, or promote events that they organize for advertising purposes or certain offers. This functionality has a commercial character and therefore could potentially become a source of revenue for the businesses involved as well as to the project, thus supporting the maintenance and the sustainability of the system in the future. As in the aforementioned scenario, passengers contributing to the system are granted credit points. It is worth mentioning, that apart from these material benefits they may gain from their activity, they are also likely to have a feeling of gratification and fulfillment that accompanies any effort of participating in a community or assisting the well-being of the city. These suggestions regarding the impact of using the proposed system and potentially the appropriation of the system by its users are seen as hypotheses, in need of being tested through qualitative and quantitative methods of research which are further explained below in the evaluation sub-section.

3. User Experience evaluation

The deployed system was evaluated against both objective and subjective criteria, the former pertaining to system-related, technical parameters, whereas the latter were focused on the users’ underlying psychological and cognitive processes during system use. Both sets of criteria revolve around the concept of User Experience (UX), widely recognized today as one of the most important success factors in developing interactive applications (Hassenzahl & Tractinsky,2006, Vermeeren et.al, 2010). User experience itself emerges from the integration of perception, action, motivation, and cognition into a meaningful whole that defines a user’s interaction with a system. In fact, usage and consumption of technologies always amounts to an experience which as such has become the backbone of user-centered design. User experience encompasses usability but also satisfaction, (itself considered one of the aspects of usability (reference), in the sense that it covers more intricate, even latent, symbolic, epistemic, and affective dimensions that cannot be reduced to a mere usability analysis (ISO 9241, 1998).

In the context of MITOS, the evaluation of UX was designed to serve two objectives: a summative one, referring to the assessment of the system by end users following a period of experimentation as well as an assessment of how the final product meets the original UX targets, and a formative one, referring to a “diagnostic” aspect of the assessment that will allow the research team to draw from users’ feedback in order to identify weaknesses in the system as well as areas for its future improvement. Overall, the purpose of the UX evaluation was to ascertain the extent to which the MITOS platform fulfilled the users’ needs at various levels, starting with but not being limited to the functional. To that end, data gathered during actual use of the system for a period of 10 days were analyzed, with the aim of identifying the major themes that govern the use of such systems in the context of the mediated city. More specifically, UX from the MITOS system usage was evaluated in terms of the following dimensions, each broken down into its essential components: • •

•

• • • • •

Functionality and utility: the usefulness of the system to the users’ customary everyday activity. Usability and performance: the system’s ease of use, assessed objectively (e.g. via metrics such as error rate, time to complete tasks etc.) and subjectively (e.g. via questionnaires). Cognitive aspects, i.e. the efficiency of the system in terms of cognitive requirements (e.g. attention, memory), are also included in this dimension. Relevant variables include task-goal achievement, efficiency, effectiveness, learnability, understandability, memorability, etc. Emotional impact: subjective, affect-based evaluation of the system’s use (commonly referred to as “user satisfaction”), encompassing perceptions about the product itself and the quality of experience derived from using it. Variables typically associated with this dimension include pleasantness, enjoyment, confidence, trust, intention to use, restriction, liberation, stimulation, identification, novelty, The main research questions on which the evaluation was based were the following:

What are the primary influences and determinants of user adoption of the system? How do different system characteristics affect the user experience and how willing users are to adopt a new technology? What benefits and what functional shortcomings do users see in the system? To what extent are users able to achieve their goals with effectiveness and efficiency? Which factors seem to affect the usability of the system, either positively or negatively?

The mobile user devices involved in the experiment were smart phones running the Android platform. 3.1

Methodology

A preliminary survey, based on a custom questionnaire completed on desktop computers, was conducted to capture the users’ preconceptions and expectations regarding the use of such systems. Questionnaire questions were designed to elicit subjective evaluations of the comprehensive user experience and its constituting dimensions (pragmatic, cognitive, affective), but also to reveal areas of difficulty, motives, intention to use, and suggestions for

improvement. The preliminary survey was followed by a more in-depth post use survey aiming to capture the users’ opinion of the system they had used 5. The sample consisted of 10 users, although in several cases some questions were not answered by some users. Nine out of ten users were male, and nine out of ten were permanent residents of Santander. The mean age of the sample was 31.00 years, with a standard deviation of 4.359. IBM SPSS Statistics 21 was employed for quantitative analysis. The questionnaire consisted of several questions that examined the construct of UX from several different angles; however, the item of primary interest was the following: “How satisfied are you overall with the use of the MITOS application?”. This general but fundamental question would result in a measurement of the number of users or percentage of total users, whose reported experience with the system and its services met or surpassed their expectations or specified satisfaction goals. Particular emphasis was placed on this specific item because it asks for a general statement of satisfaction, which is the strongest component of UX (as opposed to the traditional concept of usability, which focuses predominantly on the functional aspects of using a system). It also manages to effectively capture the users’ “gut feeling”, a coarse-grained subjective assessment of a system’s overall quality. This item formed a 5-point Likert scale. Based on responses to this item, overall user satisfaction with the MITOS system was positive (x = 3.50, s = 0.707). An additional item providing a general indication of perceived quality concerns whether users would “recommend this application to a friend”. Response to this item was also positive (x = 3.50, s = 0.926), confirming the impression that the overall UX of the MITOS system was positively evaluated. This impression is further reinforced by the users’ indication that they would like to use the system frequently (x = 3.33, s = 0.816). The remaining items were designed to measure: a) The degree of system usability and perceived system functionality in terms of relevant variables such as usefuleness of the system services in the users’ actual environment, taskgoal achievement, efficiency, effectiveness, learnability, understandability, memorability, and in order to assess whether it is actually used and used to effect by users other than the system developers. b) The degree of user engagement, understood as a quality of user experience characterized by motivation, perceived control and time, interest, aesthetic and sensory appeal, novelty and affect. These dimensions are important for UX evaluation for allowing us to identify the possible internal and external factors that can facilitate user engagement or cause user disengagement. Taken together, the aforementioned items provide a comprehensive account of user experience.

5

The pre-use questionnaire may be found online at http://www.mobics.gr/limesurvey/index.php?sid=92196&lang=es (Spanish verison) and http://www.mobics.gr/limesurvey/index.php?sid=92196&lang=en (English version). The post-use questionnaire may be found at http://www.mobics.gr/limesurvey/index.php?sid=76147&lang=es (in Spanish) and http://www.mobics.gr/limesurvey/index.php?sid=76147&lang=en (in English).

A breakdown of responses to items that deal with specific UX components follows. Unless stated otherwise, all items are either 5-point Likert scales or 5-point semantic differentials. Responsiveness. Users indicated that the system was quite responsive to their actions (x = 4.11, s = 0.928), and that delay between their action and expected outcomes was not excessive (x = 3.33, s = 0.707). Functionality and usefulness. Overall, in a technological system use, components of functionality and usefuleness, are seen as factors of extrinsic motivation that involve consuming products or services in an efficient and comfortable way to achieve the goals with a minimum of annoyance (Eighmey & McCord 1998). Analysis of collected data showed that the users of MITOS evaluated the functionality of the system as adequate and not contravening their expectations (x = 3.33, 0.707). Furthermore, it was stated that the MITOS application succeeded in fulfilling the users’ needs (x = 3.13, s = 0.641) and was therefore evaluated as useful (x = 3.38, s = 0.916), a statement also reinforced by the fact that users managed to achieve their goals through the system (x = 3.44, s = 0.726). In particular, the quality of environmental information provided by the system (x = 3.83, s = 0.408), as well as the parking prediction system (x = 4.20, s = 0.447) were evaluated as important components of the system’s usefulness. In general, the system was useful as a trip planning assistant (x = 3.11, s = 0.782) provided trip information of good quality (x = 3.50, s = 0.856) and accuracy (x = 3.63, s = 0.741), and managed to promote public transportation effectively (x = 3.63, s = 0.744). However, its ability to improve traffic conditions in Santander or significantly affect the users’ travel time and cost was evaluated as neutral (x = 3.00 on both items, s = 1.225, 1.000 respectively). Ease of use. While the importance of functionality is largely acknowledged, it is also pertinent that the system can be used and the information can be accessed efficiently with minimal effort by the end-users. The system was generally perceived as easy to use (x = 3.67, s = 1.118), user-friendly (x = 3.75, s = 0.707) and lacking in excessive complexity (x = 3.67, s = 1.118). The information provided by the system was perceived as clear (x = 4.00, s = 0). Efficiency. Users judged the system’s efficiency as neutral (x = 3.00, which is the midpoint of a 5-point Likert scale, s = 0.500). The same is true with respect to its timesaving potential (x = 3.00, s = 0.756). Additionally, users reported a tendency to perceive the system as more “flashy” than efficient (x = 3.13, s = 0.641). However, as evidenced by responses to other items reported in this section, this does not mean that the system is inefficient, neither did this average efficiency rating prevent the users from evaluating the system’s overall satisfaction as inadequate. Robustness, reliability, security, and controllability. Users had no problem recovering from errors (x = 3.78, s = 0.667), and generally thought the system was reliable (x = 3.50, s = 0.756). Furthermore, the system was perceived as secure (x = 3.13, s = 1.126) and controllable (x = 3.38, s = 0.744). The system was evaluated as functioning appropriately in terms of technology and in accordance with users’ competences. Learnability and memorability. Users indicated that they learned how to use the system rather quickly (x = 4.00, s = 0.756), without needing to learn a significant amount of information beforehand (x = 4.22, s = 1.093) or having the aid of technical support personnel

(x = 4.67, s = 0.500). Furthermore, the system is perceived as easy to remember (x = 4.11, s = 0.782) and to discover (x = 3.67, s = 0.707). Interface aesthetics. The visual aspects of the system were evaluated as appealing (x = 3.44, s = 0.882) and users indicated that they enjoyed using the interface (x = 3.56, s = 1.014). However, users did not, for the most part, think that the application was “fashionably designed” (x = 2.88, s = 0.835). Enjoyment and involvement. These two items are vital elements of UX experience. We know from research that enjoyment and involvement may function as derived motivations for technology usage or product adoption in general, by providing for users a gratification or instrumentality that may not have been intended by or anticipated during design, or even by the user at the time of the adoption. What we also know is that the functional value of a system does not stand independently and many of the utilitarian attributes that may resonate more strongly with users gain much of their value from other “values” relating to more instrinsic and visceral aspects. The component of perceived enjoyment can be defined as the extent to which the activity of using the application was perceived to be enjoyable in its own right, an affective response that is operationally different from – though practically directly related to - any performance consequences or ergonomic qualities of the system. MITOS’ users found the system enjoyable (x = 3.25, s = 0.707), but also remarked that it was somewhat lacking in intellectual stimulation and excitement (x = 2.88, which is below the average value of 3 for a 5-point Likert scale, s = 0.835). On the other hand, the users did not report feeling disappointed or frustrated by the system (x = 2.63, s = 0.518). Additionally, they reported that such an application would fit their lifestyle (x = 3.13, s = 0.835), thus providing an indication that the MITOS system may have a positive impact on their selfimage and identity. With respect to engagement, users reported that, in general, they did not get absorbed; two items were used to assess the system’s tendency to encourage user engagement, and the response to both of these was below the average value of 3 (x = 2.25, 2.88, s = 0.707, 0.835 respectively). 3.2

Correlations

A number of interesting statistically significant correlations (Spearman’s rho) were observed between the overall satisfaction questionnaire item (see previous section) and other items pertaining to UX, as outlined below: •

“The application had all the functions and capabilities that I expected it to have” (ρ = .755, p < .009)

•

“I found the system easy to use” (ρ = .648, p < .03)

•

“I found the system unnecessarily complex” [REVERSED] (ρ = .648, p < .03)

•

“As a user, I felt that it was efficient to get things done with the system” (ρ = .612, p < .04)

•

“I found the system reliable” (ρ = .929, p = 0)

•

“I liked using the interface of this system” (ρ = .739, p < .012)

•

“I enjoyed using the system” (ρ = .707, p < .025)

•

“When using the system I felt in control of it” (ρ = .813, p < .007)

•

“I got very involved in using the system” (ρ = .814, p < .007)

•

“I felt my use of the system was secure” (ρ = .63, p < .047)

•

“I felt the MITOS application would fit with my lifestyle” (ρ = .842, p < .004)

•

“I would characterize this application as fashionably designed” (ρ = .649, p < .041)

•

“I would recommend it to a friend” (ρ = .96, p = 0)

The correlations mentioned above confirm the view that a multitude of factors is responsible for the overall UX of a mobile app in general and a travel assistance app in particular. High correlations were observed between overall UX and items pertaining to usefulness, ease of use, security, reliability, aesthetic quality, and identity / lifestyle. An interesting correlation was observed between preference for the visual aspects of the system and learnability. More specifically: • •

“How much did you like the visual aspects of the system?” was very strongly correlated with “I learned to use [the system] quickly” (ρ = .816, p < .007). “How much did you like the visual aspects of the system?” was strongly correlated with “I needed to learn a lot of things before I could get going with this system” (reverse-scored) (ρ = .614, p < .039).

Also, preference for the system’s visual elements (“how much did you like the visual aspects of the system?”) was strongly correlated with perception of the system as user-friendly (“the system is user-friendly”) (ρ = .655, p < .039). The above are in accordance with Tractinsky’s (1997) finding that there is a dependency between an interface’s aesthetic impression and its perceived usability. Ease of use seems to be correlated rather strongly (ρ = .784, p < .006) with functionality (expressed through the item “the application had all the functions and capabilities that I expected it to have”). The latter is also strongly correlated (ρ = .750, p < .01) with efficiency (“as a user, I felt that it was efficient to get things done with the system”). Not surprisingly, a very strong correlation (ρ = .950, p < .002) was found between the system’s perceived fit to the users’ lifestyle (a factor of functional as well as symbolic/epistemic value) and their willingness to use the system frequently (considered a critical measure of the system’s conditional value). Additionally, lifestyle fit and perceived usefulness seem to be strongly correlated (ρ = .669, p < .035). Another very strong correlation (ρ = .900, p < .007) was observed between getting involved in using the system and willingness to use the system frequently. Other conclusions that may be drawn from our data are the following: •

Primary reason for future use is related to efficient navigation / commuting and not necessarily a desire to contribute to improving the city’s environmental conditions

•

3.3

Users are generally not interested in aspects of the system not strictly related to its primary function (i.e. Commuting / parking aid). This is not surprising, given that the majority of participants uses travel-related apps, but not entertainment apps 6. Identifying the overarching dimensions of a travel app user experience

In an attempt to arrive at a smaller and more manageable number of mobile travel app UX dimensions, several of the variables highlighted above, each represented by one or more questionnaire items, were subjected to Principal Component Analysis. Varimax rotation with Kaiser Normalization was employed in order to produce a more or less orthogonal factor structure – an endeavour that was rendered more difficult by the small number of respondents. According to the PCA, the total variance among the variables selected can be attributed to four factors, as seen in the following table: Table 1: Total Variance Explained Component

Initial Eigenvalues

Total

% Variance

Rotation Loadings of Cumulative Total %

Sums

% Variance

of

Squared

of Cumulative %

1

18,300 49,459

49,459

16,112 43,546

43,546

2

8,537

23,074

72,533

8,265 22,337

65,883

3

6,242

16,869

89,402

7,978 21,562

87,445

4

3,921

10,598

100,000

4,645 12,555

100,000

Extraction Method: Principal Component Analysis.

The loadings of the variables on the extracted factors may be seen in the following table: Component QUESTIONNAIRE ITEMS

How satisfied are you overall with the use of the MITOS application How responsive was the system to actions that you initiated (or performed)?

I experienced much delay between your actions and expected outcomes

How much did you like the visual aspects of the system?

6

Again, small sample size makes this conclusion doubtful.

1

2

3

4

,993

,094

,035

,054

-,229

-,014

,660

,715

,108

,233

,963

,082

,520

,653

,387

,393

,993

,094

,035

,054

I found the system easy to use

,833

,446

-,143

,295

I found the system unnecessarily complex

,868

,384

,215

,230

I learned to use it quickly

,589

,604

,479

-,242

,671

,721

,128

,111

I think that I would need the support of a technical person to be able to use this ,108 system

,233

,963

,082

I easily remember how to use it now

-,015

,042

-,736

,676

,084

,986

-,064

-,128

As a user, I felt that it was efficient to get things done with the system

,657

-,153

-,268

,688

It saved me time when I used it

,800

,071

-,410

-,433

I achieved my goals

-,417

,555

-,464

-,550

Whenever I made a mistake using the system, I recovered easily and quickly

-,084

-,986

,064

,128

I found the system reliable

,868

,384

,215

,230

It worked the way I wanted it to work

,993

,094

,035

,054

It was easy for me to find the information I needed

,520

,653

,387

,393

I liked using the interface of this system

,592

,656

,467

,004

I enjoyed using the system

,868

,384

,215

,230

The system was exciting and intellectually stimulating

,902

,351

-,170

-,186

I felt very confident using the system

-,108

-,233

-,963

-,082

While using the system I felt in control of it

,697

-,538

-,344

-,326

I got absorbed with the system

,677

-,206

,682

-,184

I got very involved in using the system

,915

-,062

,393

-,071

I felt my use of the system was secure

,233

-,686

-,584

,367

I feel that the MITOS application would fit in will in my lifestyle

,957

-,211

-,151

-,130

The MITOS application meets my needs

,708

,249

,031

,660

Ι would characterize this application as fashionably designed

,904

-,032

-,128

,406

I thought the system was more “flashy” than efficient

,003

-,542

,059

,838

I often felt disappointed and frustrated while using the system

-,084

-,986

,064

,128

The application had all the functions and capabilities that I expected it to have

I needed to learn a lot of things before I could get going with this system

I would imagine that most people would learn to use the system very quickly

At the end of the use I felt more proficient in using the system

-,388

-,368

,793

-,293

I think that I would like to use this system frequently

,993

,094

,035

,054

This application is be very useful to me

,357

,206

-,856

-,312

I would recommend it to a friend

,858

,258

-,401

,192

The bold numbers on the table above indicate that the variable represented by the questionnaire item on the left clearly loads on one factor. However, for many items this is not the case, as they load equally on more than one factor, a fact that is to a degree attributable to the very small number of respondents, which resulted in reduced variance. 3.4 Limitations The most important limitation of this study is the small size of the user sample. With only ten users (at best, since in many cases certain users failed to respond to all items), any conclusions drawn are tentative at best. The same is true for the PCA: the small sample may have provided a distorted picture of the number and relative importance of UX dimensions that resulted from user responses to relevant questionnaire items and has made the identification of common themes within each component rather difficult. There are other problems related to the sample apart from its small size; for instance, the sample was very unevenly distributed with respect to gender (9 out of 10 respondents were male) and place or permanent residency (i.e. in Santander as opposed to elsewhere) (9 out of 10 respondents were permanent residents of Santander). Although it could be argued that the latter issue is not as important as the former, the absence of non-residents prevents any meaningful assessment of the application’s UX with respect to potential tourists (tourism is a major source of revenue for the city of Santander). An additional limitation of this study is the lack of qualitative data as well as data drawn during use of the system (in situ) which would complement the quantitative analysis presented herein and would, at least partially, offset the problem of the sample size being too small.

4. Conclusions

The MITOS experiment deployed a set of useful services on top of the SmartSantander infrastructure. Several tentative conclusions may be drawn based on the evaluation of such services. Some of these could not be identified without the existence of such an extensive sensor infrastructure. As described in the previous section, the MITOS application was received rather positively. However, due to several issues pointed out above, any generalizations derived from the data presented in this paper can be regarded as tentative at best. Notwithstanding such issues, the most obvious conclusion that may be drawn from this investigation is that any location-based application shoule be evaluated first and foremost in terms of its functionality and efficiency (pragmatic aspects) and less in terms of its experiential or playful dimension (hedonic aspects) – a point which is emphasized further by the results of the Principal Component

Analysis, according to which usability-related variables pervade all four components. Therefore, any application that aims to integrate pragmatic and hedonic aspects should adopt a somewhat conservative approach and not compromise usability for the purpose of adding game-inspired functions. That said, the integration of game-like elements would assumedly provide an added value to the system and would contribute to a more engaging overall user experience. Interestingly, the Principal Component Analysis again provides relevant indications, as several variables that pertain to satisfaction, lifestyle etc. – aspects that cannot be characterized as ‘pragmatic’ – load on the primary factor, which is the most influential in terms of variance explained. The correlations found through the analysis of gathered data also corroborate the widely held view that User Experience is indeed a multidimensional construct, made up of different interlocking dimensions such as usefulness, ease of use, security, reliability, enjoyment, aesthetic quality and identity/lifestyle, all of which jointly create the overall effect on the user. Furthermore, it is the coalescence of these dimensions that contribute to the perceived usefulness of the proposed technology as well as to the increased levels of reported willingness to adopt the technology in one’s daily life. In addition to the above, a standard reward mechanism could function as an incentive for local users to participate in the experiments, although care must be taken so that these rewards do not bias the results of the research. The preference for functionality and usability over more hedonic aspects of using the application was also confirmed by the fact that the users remarked that the application was generally lacklustre and did not provide adequate intellectual stimulation, but it still managed to be useful, partly because in some cases it provided more locally relevant information than other apps (e.g. Google Maps). Likewise, the integration of interpersonal and interactive affordances and even social networking services would most likely improve the reception of an application by increasing its allure and building on users’ already established familiarity with social networking. Most smartphone users access such services either on the web or through dedicated apps. However, as in the case of playful aspects, care must be taken so as not to endanger the functionality and efficiency of the application. And in the case of MITOS, a significant percentage of the users chose not to employ the communication methods provided by the system (namely messages and comments). Additionally, the vast majority of users did not upload any geolocated content. These observations point to the conclusion that, although useful in theory, the aforementioned functionality was not regarded as necessary. One reason for this may be the relative unsuitability of the city of Santander for a widely varying urban navigation experience due to factors such as small size, absence of a complex network of public transportation etc.

References Deterding, S., Khaled, R., Nacke, L., & Dixon, D. 2011. Gamification: Toward a definition. CHI 2011. Presented at the Computer Human Interaction, Vancouver, British Columbia, Canada: ACM. Eighmey, J. and McCord, L. 1998. Adding value in the information age: uses and gratifications of sites on the World Wide Web, Journal of Business Research, 41 (3), pp. 18794.

Hassenzahl, M. and Tractinsky, N. 2006. User Experience–A Research Agenda. Behaviour and Information Technology 25, pp. 91 -97. Huotari K., and Hamari, J. 2012. Defining Gamification - A Service Marketing Perspective. Proceedings of the 16th International Academic MindTrek Conference, Tampere, Finland, October 3-5, 2012 ISO 9241, Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs) – Part 11: Guidance on Usability, International Standardization Organization, Geneva, Switzerland, 1998 Kuniavsky, M. 2003. Observing the User Experience: A Practitioner's Guide to User Research (Morgan Kaufmann Series in Interactive Technologies) Morgan Kaufmann Publishers Inc. San Francisco, Laugwitz B., Held T., Schrepp M. 2008. Construction and Evaluation of a User Experience Questionnaire. In: Holzinger A. ed.: USAB 2008, LNCS 5298, pp. 63–76. Laurel B.1993. Computers as Theatre . Boston, MA,: Addison-Wesley Longman Publishing Co., Inc Malone, T. W. 1982. Heuristics for designing enjoyable user interfaces. In Proceedings CHI 1982. ACM Press (1982), pp. 63-68 Marczewski, A. 2012. Forward. Gamification: A Simple Introduction (1st ed.). p. 46. http://www.amazon.co.uk/Gamification-Simple-Introduction-Bit-MoreAvailable at: ebook/dp/B007U44MDU. [Accessed: 2014-01-24]. Salen, K., & Zimmerman, E. 2004. Rules of Play: Game Design Fundamentals. Massachussets: The MIT Press. Sanchez, L.; Galache, J.A.; Gutierrez, V.; Hernandez, J.M.; Bernat, J.; Gluhak, A.; Garcia, T.. 2011. SmartSantander: The meeting point between Future Internet research and experimentation and the smart cities. Future Network & Mobile Summit (FutureNetw) 2011 , pp.1,8, 15-17 June 2011 Tractinsky, N.. 1997. Aesthetics and Apparent Usability: Empirical Assessing Cultural andMethodological Issues. In: CHI 1997. Electronic Publications (1997), Available at: http://www.acm.org/sigchi/chi97/proceedings/paper/nt.htm. [Accessed: 2014-01-24]. Vermeeren, A.P., Law E., Roto V., Obrist M., Hoonhout J., Väänänen-Vainio-Mattila K.. 2010. User Experience Evaluation Methods: Current state and Development Needs. Proceedings of NordiCHI 2010, Reykjavik, Iceland, 16-20 October 2010 Vlahogianni, E., Kepaptsoglou, K., Tsetsos, V., and Karlaftis, M. G.. 2014. Exploiting New Sensor Technologies for Real-Time Parking Prediction in Urban Areas, Transportation Research Board 93rd Annual Meeting, January 12-16, Washington DC, USA, 2014