A Quality in Use Extension Model for Semantic Web

SWET-QUM: A Quality in Use Extension Model for Semantic Web Exploration Tools Jose Luis González, Roberto García, Josep Maria Brunetti, Rosa Gil and Juan Manuel Gimeno Universitat de Lleida Jaume II, 69 25001 Lleida, Spain +34 973 702 742

{ joseluisgs, rgarcia, josepmbrunetti, rgil, jmgimeno}@diei.udl.cat ABSTRACT In order to make Semantic Web tools more appealing to lay-users, a key factor is their Quality in Use, the quality of the user experience when interacting with them. To assess and motivate the improvement of the quality in use, it is necessary to have a quality model that guides its evaluation and facilitates comparability. The proposal is an extension of the international standard ISO/IEC 25010:2011 that focuses on Semantic Web exploration tools, those that make it possible for lay-users to browse and visualise it. The model is being applied as part of the iterative development process of such a tool and helps guiding its development so the Quality in Use is maximized.

Categories and Subject Descriptors H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous.

General Terms Human Factors, Experimentation, Measurement, Design.

Keywords Quality, user experience, usability, evaluation, Semantic Web

1. INTRODUCTION The objective of this paper is to define a framework to evaluate the quality of applications based on Semantic Web technologies. This is a very broad and huge task so we have first focused on just a particular aspect of quality: the external part related with the quality in use. Moreover, we have also restricted the scope to a subset of Semantic Web applications. The interest in quality in use is because it has received less attention until now; the focus has been placed in internal quality, which makes possible building good Semantic Web applications. However, as more applications are developed and more users start using them, aspects related to external quality are getting more

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Interaccion’12, Oct 3–5, 2012, Elche, Alicante, Spain. Copyright 2012 ACM 978-1-4503-1314-8/12/10...$10.00

and more relevant as interest spreads from building Semantic Web applications to also getting users satisfied with them. The focus of this work is on applications that, while exploiting the possibilities of Semantic Web technologies, provide a novel user experience for interacting with them. Most of this novelty is found in tools that facilitate the navigation and visualization of Semantic Web data. This is the kind of tools we focus on and we call them in this paper Semantic Web Exploration Tools (SWETs). Moreover, as we are also developing one of these tools we will also be able to test the proposed quality framework during the development of this tool, by integrating evaluations based on this framework into the iterative development process being used. The rest of this paper is organised as follows. Next, Section 2 sets the state of the art in relation with existing quality model standards. Then, Section 3 highlights the challenges of interacting with the Semantic Web. Section 4 presents the proposed quality in use model and in Section 5 we show it in practice guiding the evaluation of the quality in use of a Semantic Web exploration tool. Finally, Section 6 shows the conclusions and explores the future work.

2. QUALITY IN WEB SYSTEMS All software developers should aspire to achieving a high level of quality in software systems. One way to do this is to improve the process of interaction when carrying out a specific task in a specific context of use. However, achieving ‘a good quality value’ is challenging and requires previous characterization of the concept of ‘quality’ in order of to verify and evaluate it. All software developers should aspire to achieve a high level of quality in software systems. One way to do this is to improve the process of interaction when carrying out a specific task in a specific context of use. But, What is the Quality concept? This question can be answered as the ‘the rightness of a product or service to meet user needs and the degree to which a set of inherent characteristics fulfills requirements’, ISO 9000 [11]. The definition of Quality reveals two considerations: one is the need to characterize the concept of quality based on the identification of the inherent properties of the product (quality of a product). The other consideration is the need to establish or propose a series of functional and/or non-functional requirements, and how these are achieved by users through the interaction process or process of use (quality in use). The decomposition of quality in other features makes the quality evaluation process easier, and it is known as a quality model. Basili [1] describes a quality model based on three key

components: Factors or characteristics (to specify), indicate which properties and targets are used as indicators of the quality of a product, Criteria or properties (to build), indicate evaluable or measurable attributes linked to the factors of a software product. Metrics (to control), The evaluation of a software product is determined by metrics that allow the features offered by the developed software products to be estimated. They are defined and used to provide a scale and method for measurement. There are many standards models, but in this work we focus on and build from the latest ISO/IEC standard model, ISO/IEC 25000:2005 [15]. It provides guidance for the use of the new series of international standards named Software product Quality Requirements and Evaluation (SQuaRE). This standard replaces the old ISO/IEC 9126 [12] and the 14598 series and comprises the second generation of standards for software quality. Consequently, it goes beyond previous versions when considering the quality of the experience during the interaction process [9]. This international standard defines: •

•

A Quality in Use model composed of five factors/characteristics related to the outcome of interaction when a product is used in a particular context of use. This system model is applicable to the complete human-computer system, including both computer systems in use and software products in use (ISO/IEC 25010, Quality in Use Model, 2011) [15]. The five factors are: Effectiveness, Efficiency, Context coverage, Freedom from risk and Satisfaction. A Product Quality model composed of eight factors/characteristic related to static properties of software and dynamic properties of the computer system. The model is applicable to both computer systems and software products (ISO/IEC 25010, Software Product Quality Model, 2011) [15]. The eight product quality characteristics are Functional Suitability, Performance Efficiency, Compatibility, Usability, Reliability, Security, Maintainability and Portability.

Finally, in the ISO/IEC 25000 standard, the quality of software system is described in terms of its elements and the interaction process. ISO Standards offer metrics and how to realize the evaluation process to estimate the quality of a software system (quality in use and product quality) as part of ISO/IEC 25040:2011 [13,16]. The Quality in Use metrics are determined by the Context of Use, it means, users, tasks should be considered when evaluating the user experience or during an iterative development process of an interactive system following an User Centred Design approach [14]. Though quality is widely considered in the context of Semantic Web technologies (for instance when considering the Quality of Service (QoS) for Semantic Web services [2] or the quality of ontology alignment methods [7]), there is much less work related with building standards-based quality models to evaluate Semantic Web technologies. The main contribution in this sense is a quality model for Semantic Web technologies based on ISO 9126 [4]. However, its focus is on internal quality, not in the user experience aspects. Moreover, it is based on the standard ISO 9126 and not in the last version, ISO 25000, where quality in use has gained more relevance. Consequently, there is not much work about the evaluation of the quality in use, including usability, of Semantic Web technologies. And there are even less attempts to establish a quality in use model that guides evaluations.

For instance, there are quality metrics for guidelines for Linked Data publishing [5,6]. They focus on efficacy measures related with the quality of the generated data and not in the quality in use of the interaction components supporting the authoring tasks. Another attempt of taking into account quality in use in Semantic Web tools is [19] in which both the SDE (Semantic Data Explorer) tool is presented and an evaluation of the quality in use of it are presented. The evaluation is based on measuring the efficiency and efficacy attained with SDE and compared to the same measures using a legacy system. Although it constitutes a primitive quality model, it is too much specific to the domain where SDE is applied and the set of metrics is too much limited to constitute a quality in use model. There is a similar experience for the evaluation of four Natural Language Query Languages for Semantic Web knowledge bases [17]. In this case, the Quality in Use Model, thought not explicit, is much clearer because four different tools have been evaluated and a set of metrics is defined to compare them. None of the previous works contributes a formal Quality in Use model. All of them are evaluations of specific tools, or at most four tools that are compared. In any case, they constitute valuable experiences that can be used to enrich a formal Quality in Use model for Semantic Web technologies like the one proposed in this paper.

3. SEMANTIC WEB HCI CHALLENGES The first thing to take into account when considering user interaction in the Semantic Web is that despite the fact that it was designed for machine consumption, at the end, humans are its real consumers. Consequently, adopting Semantic Web technologies does not mean that users are no longer considered to be at the centre of the design process. In fact, its adoption implies new challenges for user interaction beyond those already posed by Web technologies or interactive systems in general. These challenges in the user interface should be identified and then considered when evaluating the quality in use because they are what make interacting with the Semantic Web different from existing and already quite well studied systems. Reducing the interaction barriers that these challenges might pose, and profiting from the new features that they make possible, is what can improve the perception that users have of tools based on Semantic Web technologies and facilitate their wider adoption. Consequently, they should be the focus of a model for their quality in use. In the context of Semantic Web exploration tools, the first difference, when comparing to existing Web-based interactive systems, is that the Semantic Web constitutes a change from a Web focused on documents to a more fine-grained one where what is published and consumed are much smaller items. These items are statements called triples in Semantic Web terms. This change in the information granularity makes that web pages lose their purpose in the Semantic Web [10]. In addition to Semantic Web features that facilitate information integration, in the Semantic Web it becomes natural to dynamically build views about a particular thing by combining statements from different sources describing that thing. Consequently, the data publisher loses control about how information is packaged and presented to the users. On the other hand, the information ecosystem can be enriched with information aggregators that provide alternative views, and even end-users can gain control of the aggregation process and build their own descriptions.

When considering end-users interaction, this freedom is not just about how to accumulate statements but also, as the amount of them can easily become really big, how they are arranged, traversed, filtered, etc. Consequently, it is not web pages what loses purpose, but also predefined information architectures because now users can explore information following many different and unanticipated paths. Consequently, a critical aspect of Semantic Web exploration tools is that they are capable of making all the richness of the underlying data model available at the interaction level. They should not constraint the interaction or reduce these constraints to the minimum, by setting just a limited and predefined set of ways to explore the data. Therefore, the information architecture built to drive user interaction should be flexible enough to offer to the user all the possible ways to conceive and interact with the data. However, this flexibility should not be at the expense of users cognitive overload, and it does not need to be the case. It should be possible to exploit data richness to provide a flexible interaction that satisfies different user mental models. It is also important to note here that, though Semantic Web data is based on a graph data model, this is not necessarily the appropriate metaphor in the user interface [21]. A graph based representation might be useful when overviewing the data because it might be easier to spot patterns in the data. However, when the exploration takes the user closer, when the user is browsing the data, graph-based representations get less useful and alternatives should be explored. An alternative view when user zooms into the data is an entity-oriented approach. This approach is based on the idea of assembling the data pieces, even when they come from different sources, around the entities they describe. This approach can be then complemented with an object-action interaction paradigm [23], where actions do not need to be associated a priori to entities. It is the aggregation of data about these entities, and the underlying semantics, what drive this association and provide a really flexible way to explore and interact with the data.

Quality in Use characteristics/factors and properties proposed in ISO/IEC-25010:2011 [15], for the evaluation of quality of interaction for Semantic Web exploration tools. They are all the characteristics for Quality in Use in ISO/IEC-25010:2011 except for Freedom from Risk (for the moment because it has not been considered relevant in the evaluation context defined so far to measure economical, health and environmental risks). This factor is more appropriate when considering ergonomic and related factors that for the moment lay out of the proposal scope.

A key element when setting the context for Quality in Use with SWET evaluation is to determine the kinds of user tasks to be evaluated. Shneiderman’s Visual Information Seeking Mantra, a set of user tasks proposed from outside of the Semantic Web community, which has become the standard when considering interaction with visual information [22]. The tasks are:

Tasks completion. What proportion of the tasks is completed?

Overview: gain an overview of the entire collection. Zoom: zoom in on items of interest. Filter: filter out uninteresting items. Details-on-demand: select an item or group and get details when needed. • Relate: view relationships among items. • History: keep a history of actions to support undo, replay, and progressive refinement. • Extract: allow extraction of sub-collections and of the query parameters. Shneiderman’s tasks provide a much more detailed view of the typical task of Semantic Web Exploration Tools, Browse. Taking into account these subtasks of Browse might provide the required level of detail by the fact that the quality model will focus specially on this Task. Moreover, this set is well established and widely used in the Information Visualisation discipline so very likely to provide useful insights for Semantic Web exploration.

Data Exploration UI Effectiveness. What proportion of the user interface components, relevant for the task, do the users view? These components are those relevant for the data exploration tasks and include high level components like menus, facets, breadcrumbs, etc. but also, when a more detailed view is necessary, links, buttons, forms, etc.

• • • •

4. QUALITY IN USE MODEL FOR SWETS In this work, we propose a Semantic Web Exploration Tools Quality in Use Model (SWET-QUM) that specialises the generic

SWET-QUM is completed with metrics. We should mention that all the metrics identified are focused on process of use. Hence, the evaluation essentially requires testing with users, observation of users while they are interacting and the completion of questionnaires when users finish interacting in order to measure satisfaction. Our proposed metrics have an interpretive approach, and are focused on the resolution of tasks, which means they are based on the achievement of task goals by users during the interaction process. The following subsections, each one corresponding to one of the Quality Factors under consideration, present the criteria/properties to be measured and the metrics to be used to compute the property measure. Finally we present a methodology to realize the evaluation of the interaction process.

4.1 Effectiveness Effectiveness defines the degree to which specific users can achieve the semantic data exploration tasks with precision and completeness. The proposed metrics are: Task success. What proportion of one task is completed?. Measure: X = F(X) (percentage of the task completed). Value: 0