We define, in a consistent way, usability metrics for software components based .... Usability compliance: the capability of the software component to adhere to ...
Usability metrics for software components Manuel F. Bertoa and Antonio Vallecillo Dpto. Lenguajes y Ciencias de la Computación. Universidad de Málaga.
{bertoa,av}@lcc.uma.es Abstract. The need to select a component among a set of possible candidates that offer similar functionality opens the need to count with objective methods that help developers in this difficult task. One necessary step is to define a set of metrics that offer useful and simple results for the component selection process. In this position paper we present a collection of software component metrics focused on a main quality characteristic—the usability—of great importance to any software product. We define, in a consistent way, usability metrics for software components based on the ISO 9126 Quality Model. Before presenting the metrics, we will define the basic concepts on software measurement used in this paper, what we understand for usability in a CBSD framework, and the component information available to be measured.
1 Introduction The last decade marked the first real attempt to turn software development into engineering through the concepts of Component-Based Software Development (CBSD) and Commercial Off-The-Shelf (COTS) components. The idea is to create high-quality parts and join them together to form a functioning system. One of the most critical processes in CBSD is the selection of a set of COTS components from (in-house or external) repositories that fulfil some system architectural and user-defined requirements. Current approaches propose quality models and metrics for the effective assessment of such components. These proposals (both International Standards and research works) independently define quality characteristics, attributes, and metrics, which are specific to the particular nature of COTS components and CBSD. In recent works [1,2], we tried to adapt the general ISO 9126 Quality Model [3] to the realm of software components, for which a set of metrics was proposed. Then, we looked at the information provided by software component vendors, analysing the information they currently provide about the components they sell or license, with the goal to see how many of these metrics were in fact measurable. Apart from finding out that the information provided by vendors was scarce and mostly insufficient, we also discovered that the relationship between the metrics we had identified and the quality characteristics they tried to measure was ill-defined. Furthermore, we also realized that this was a common problem of most of the current proposals for software component metrics. For instance, ISO 9126-1 defines a set of quality characteristics and sub-characteristics that constitute its Quality Model, and then ISO 9126 Parts 2 to 4 define, respectively, a set of (internal, external and inuse) metrics for measuring them. According to ISO 9126 itself, metrics measure attributes, which influence one or more quality sub-characteristics. However, the metrics presented in Parts 2 to 4 of ISO 9126 are mostly indirect metrics, defined without any reference to the attributes they measure: they are just defined and assigned to quality sub-characteristics without any justification whatsoever. Moreover, ISO 9126 metrics are assigned to just one quality sub-characteristic, although in theory one attribute may influence more than one quality characteristic (e.g., the “size” of a component usually affects both its maintainability and its learnability). The situation does not improve in the rest of the proposals that define metrics for software components, which do not consider attributes either. Worse than that, most of these proposals do not associate any quality characteristic to the metrics they define—metrics are just defined, but with no indication to either the attributes they measure or to the quality characteristics they try to assess. In this paper we will try to bridge that gap, trying to properly define a set of metrics based on the attributes they measure, and how these metrics assess the quality characteristics of a software component, —1—
according to a given quality model. Initially we will concentrate on one particular quality characteristic only, the Usability, as defined by ISO 9126. The paper is organized as follows. After this introduction, Section 2 presents the software measurement concepts upon which proper definitions of metrics can be given. Section 3 defines the concept of Usability, adapting this quality characteristic from the ISO 9126 Quality Model to the particular case of software components. Then, Section 4 introduces the set of elements of a software component that need to be observed in order to assess its Usability, and that will constitute the information required to evaluate it. Finally, a set of Usability metrics will be defined, identifying the component attributes they measure, and how they influence and assess the different quality sub-characteristics of Usability. The paper finishes by drawing some conclusions and outlining further research activities.
2 Software Measurement Concepts Overview In general, there is a lack of consensus about how to define and categorize software product quality characteristics. In this paper we will follow the terminology defined in [4]. At the highest level, the main goal of a software measurement process is to satisfy certain information needs by identifying the entities (which belong to an entity class) and the attributes of these entities (which are the targets of the measurement process). Attributes and the information needs are related through measurable concepts (which belong to a Quality Model). According to Fenton, attributes can be external or internal: attributes whose value depends on the environment in which the software operates are external, as opposed to attributes that do not depend on this environment, which are internal. Then, we need to measure these attributes using metrics. A metric relates a defined measurement approach and a measurement scale. A metric is expressed in units, and can be defined for more than one attribute. Three kinds of metrics can be distinguished: direct metrics, indirect metrics, and indicators. A measurement approach is a generalization of the different approaches used by the three kinds of metrics for obtaining their respective measures. A direct metric applies a measurement method. An indirect metric uses a measurement function (which rests upon other direct and/or indirect metrics). Finally, an indicator uses an analysis model (based on a decision criteria) to obtain a measure that satisfies an information need. Finally, the act of measuring software is a measurement (as an action), defined as a set of operations that aim at determining a value of a measure, for a given attribute of an entity, using a measurement approach. Measures are obtained as the result of performing measurements (actions). In this paper we concentrate on one particular quality model, ISO 9126, which is defined in terms of a set of characteristics and sub-characteristics, as well as the relationships between them, that provide the basis for specifying quality requirements and for evaluating quality. The entities of our study will be software components, no matter whether they are in-house developed, or acquired as COTS components. Since the model proposed by ISO 9126 is a generic quality model for any software product, we need to particularize it for software components. The customized model, denominated COTS-QM, was presented in [1]. As mentioned in Section 1, in this paper we will only focus on one of its characteristics, the Usability.
3 Usability in CBSD The literature currently offers several definitions of Usability: (a) The capability of the software product to be understood learned, used and attractive to the user, when used under specified conditions (ISO/IEC 9126-1:2000).
—2—
(b) The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use (ISO 9241-11:1998). (c) The ease with which a user can learn to operate, prepare inputs for, and interpret outputs of a system or component (IEEE Std.610.12-1990). (d) Usability of a software product is the extent to which the product is convenient and practical to use (Barry Boehm, 1978 [5]). (e) The probability that the operator of a system will not experience a user interface problem during a given period of operation under a given operational profile (Fenton, 1998 [6]). As in our previous works, we will try to follow ISO 9126 Quality Model, adapting it to the particular case of software components. Then, we will initially adopt definition (a): “the capability of the component to be understood, learned, used and attractive to the user, when used under specified conditions.” The last sentence, “when used under specified conditions” (equivalent to “context of use” in ISO 924111), was added to explicitly highlight that a product has no intrinsic usability, only a capability to be used in a particular context. In fact, Usability is a quality characteristic that is intrinsically dependent upon the kind of “use” that is expected, and the kind of “user” that will use the product (the concept of user is implicit in the definition). Thus, we need to clarify the users and the kind of usage of the software components whose Usability we try to assess. From our perspective, component users will be the members of the software team that develops and maintains a component-based system—but not the end-users of such a system, nor the developers of the individual components themselves. Component users need to identify, locate and select them according to the system architectural constraints and the system owners’ requirements, integrate them to build the system, and then proceed to the system maintenance when new versions appear, or components are upgraded to fix problems. (For brevity, in the following we will simply refer to such kinds of users as “system developers”, although this term embraces in our case component-based system developers, builders and maintainers, component evaluators, and system integrators. Basic component developers and system end-users will not be considered.) ISO 9126 defines Usability in terms of five sub-characteristics: Understandability, Learnability, Operability, Attractiveness, and Usability Compliance. In the case of components, they can be defined as follows. ─ Understandability: the capability of the component to enable the user (system developer) to understand whether the component is suitable, and how it can be used for particular tasks and conditions of use. System developers should be able to select a component suitable for their intended use. For example, components elements (e.g. interfaces, operations) should be easy to understand or evident. ─ Learnability: the capability of the software component to enable the user (system developer) to learn the application. A learnability metric should be able to assess how long it takes system developers to learn how to use particular functions (interfaces, operations, …), or the effectiveness of documentation (manuals, help system, demos). For example, the user documentation and the help system should be complete, the help should be context sensitive and explain how to achieve common tasks, etc. ─ Operability: the capability of the software component to enable the user (system developer) to operate and control it. An operability metric should be able to assess whether system developers can operate and control the component. Operability metrics can be categorised by the dialogue principles in ISO 9241-10: o suitability of the component for the task; —3—
o self-descriptiveness of the component; o controllability of the component; o conformity of the component with user expectations (requirements); o error tolerance of the component; o suitability of the component for individualisation. ─ Attractiveness: the capability of the software component to be attractive to the user. Since the users we consider are not end-users, this subcharacteristic is of no interest in our case. ─ Usability compliance: the capability of the software component to adhere to standards, conventions, style guides or regulations relating to usability. Currently, we don’t know of any standards or regulations for component usability. Thus, we will not define any metric for this sub-characteristic. In summary, the Usability of a software component will be assessed in terms of the first three subcharacteristics defined above, which are shown in Table 1. USABILITY UNDERSTANDABILITY
LEARNABILITY
OPERABILITY
Table 1. Usability Sub-characteristics considered for Software Components (adapted form ISO 9126)
4 Available information about a software component In order to assess the Usability of a software component, we need to define: a set of measurable concepts that influence any of the quality sub-characteristics of Usability; the component attributes that can be measured in order to assess such measurable concepts and how they affect each sub-characteristic; and finally, the metrics that measure such attributes. Metrics can be direct metrics, indirect metrics, or indicators. Before we define any of those items, and in particular metrics, we need to know the kind of information that is available about the entities we plan to assess (the software components). This information will be the only measurable elements based on which metrics can be defined. The following UML class diagram shows the information that we think is relevant to the Usability of a component, from all the information that is available for a commercial component. The fact that components are black box and binary units of composition, whose internals cannot be viewed or accessed, only leaves us with the externally observable elements of the component that allow to assess its quality. In the case of Usability, we have considered the component documentation and some of its functional elements. In the following, and according to Figure 1, the term documentation will refer to component manuals, demos, help system, and marketing information. By functional elements we will refer to the interfaces, operations, or events that a component may support or require from other components to achieve its functionality, i.e., to implement its services. Although some metrics may be defined on individual elements (e.g., number of operations per interface), sometimes it is simpler to refer to the functional elements independently of their granularity.
—4—
Figure 1. Software Component Information
Of course, not all this information is available for all commercial components, as we discovered in our survey [2]. However, we have tried to be realistic when defining the elements above, looking for a balance between the demands of information required by the metrics, and the difficulty we know that software component vendors have for providing that information. From our point of view and previous experience, we think that the information we require (described above) seems to be reasonable. Furthermore, please notice that this information is applicable not only to software components, but also to Web Services. In this sense, the work presented here is also pertinent to the Usability of such kind of software entities.
5 Defining Usability Metrics As previously mentioned, current International Standards and research proposals that put forward collections of metric (in particular, the ISO 9126 series) are not consistent with their own definition of the concept of “metric”. The fact that no attributes are considered in the metric definitions hinders the identification and proper definition of the relationship between the metrics and the quality characteristics they assess. Such relationship is usually undefined, or not properly justified. For example, all metrics defined the ISO 9126 series are related to just one quality characteristic, which is not the usual case: an attribute normally influences more than one sub-characteristic. This gap is just what we try to bridge here with our consistent definition of usability metrics for software components. Therefore, in the first place we have an information need: “to evaluate the usability of a set of software components that are candidates to be integrated in a software product and to select the best among them”. The system developer wants to select the easiest of use (integrate) among a set of software components that provide the required functionality. There are three measurable concepts related with software component usability: Quality of Documentation, Complexity of the Problem and Complexity of the Solution (or Design). As we are comparing software components that provide similar functionality, we will suppose all of them share “the same complexity of the problem”, and accordingly, there is no need to propose metrics to assess this —5—
complexity. Thus, we will focus on the other two measurable concepts. (Besides, this concept does not seem easy to measure.) These concepts and their related attributes are presented in Table 2.
ENTITY
INFORMATION NEED
MEASURABLE CONCEPT
ATTRIBUTE CONTENTS OF MANUALS
QUALITY OF MANUALS
SIZE OF MANUALS EFFECTIVENESS OF MANUALS
QUALITY OF DOCUMENTATION
QUALITY OF DEMOS
CONTENTS OF DEMOS CONTENTS OF HELP SYSTEM
QUALITY OF HELP SYSTEM
SIZE OF HELP SYSTEM EFFECTIVENESS OF HELP SYSTEM
SOFTWARE COMPONENT
EVALUATE THE USABILITY
QUALITY OF MARKETING INFO
CONTENTS OF MARKETING INFO EFFECTIVENESS OF MARKETING INFO DESIGN LEGIBILITY (READABILITY) INTERFACES UNDERSTANDABILITY I/O UNDERSTANDABILITY
COMPLEXITY OF THE DESIGN
EASE OF LEARNING CUSTOMISABILITY QUALITY OF ERROR MESSAGES INTERFACES COMPLEXITY
Table 2. Measurable Concepts and Attributes for Usability
Since each attribute may have one or more metrics (indicators, indirect or direct metrics) that evaluate it, we need to define these metrics. The following tables show the metrics proposed to measure the measurable concepts defined in Table 2. A summary of the metrics defined for the Quality of the Documentation attributes appears in Table 3, and for those related to the Complexity of Design in Table 4. We use the term “functional element” to refer interfaces, operations and events as a whole. There should also be a last column with the direct metrics on which indirect metrics are defined, but this column has been omitted for space restrictions. The majority of these direct metrics are evident (for instance, to measure the ratio of figures per pages two direct metrics are required: the number of figures and the number of pages in the manual). Also, note that many ratio metrics repeat their denominator, which produces an unnecessary repetition of rows (this is for instance the case of the number of functional elements, that is used in many indirect metrics). Other cases are not so evident, like “COMPLETENESS OF MANUALS”, which measures the number of functional elements of the component that do not appear in the manual.
—6—
ATTRIBUTE
INDICATOR MANUALS COVERAGE
INDIRECT METRIC PROPORTION OF FUNCTIONAL ELEMENTS DESCRIBED IN MANUALS PROPORTION OF IN THE MANUAL
MANUALS CONSISTENCY CONTENTS OF MANUALS
FUNCTIONAL ELEMENTS INCORRECTLY DESCRIBED
COMPLETENESS OF MANUALS DIFFERENCE BETWEEN THE COMPONENT VERSION AND THE MANUAL VERSION RATIO OF FIGURES PER MANUAL PAGES
MANUALS LEGIBILITY
RATIO OF TABLES PER MANUAL PAGES RATIO OF UML DIAGRAMS PER MANUAL PAGES
SIZE OF MANUALS
MANUALS SUITABILITY
AVERAGE PAGES PER FUNCTIONAL ELEMENTS
EFFECTIVENESS RATIO
PROPORTION OF FUNCTIONAL ELEMENTS CORRECTLY USED AFTER READING THE MANUAL
UNDERSTANDABILITY RATIO
PROPORTION OF FUNCTIONAL ELEMENTS CORRECTLY UNDERSTOOD AFTER READING THE MANUAL
DEMONSTRATION COVERAGE
PROPORTION OF FUNCTIONAL ELEMENTS SHOWED IN DEMOS
DEMONSTRATION CONSISTENCY
DIFFERENCE BETWEEN DEMO VERSION AND COMPONENT VERSION
HELP SYSTEM COVERAGE
PROPORTION OF FUNCTIONAL ELEMENTS SHOWED IN HELP SYSTEM
EFFECTIVENESS OF MANUALS
CONTENTS OF DEMOS
CONTENTS OF HELP SYSTEM HELP SYSTEM CONSISTENCY
PROPORTION OF FUNCTIONAL ELEMENTS INCORRECTLY DESCRIBED IN HELP SYSTEM COMPLETENESS OF HELP SYSTEM
SIZE OF HELP SYSTEM EFFECTIVENESS OF HELP SYSTEM
CONTENTS OF MARKETING INFO
EFFECTIVENESS OF MARKETING INFO
HELP SYSTEM SUITABILITY
HELP SYSTEM WORD RATIO
HELP SYSTEM EFFECTIVENESS RATIO
PROPORTION OF FUNCTIONAL ELEMENTS CORRECTLY USED AFTER USING THE HELP SYSTEM
HELP SYSTEM UNDERSTANDABILITY RATIO
PROPORTION OF FUNCTIONAL ELEMENTS CORRECTLY UNDERSTOOD AFTER USING THE HELP SYSTEM
COVERAGE OF MARKETING INFO
NUMBER OF MARKETING INFO ELEMENTS DESCRIBED
COMPLETENESS OF MARKETING INFO
PROPORTION OF SERVICES UNDERSTOOD AFTER READING THE COMPONENT DESCRIPTION
MARKETING INFO CONSISTENCY
DIFFERENCE BETWEEN THE COMPONENT VERSION AND THE MARKETING INFO VERSION
MARKETING INFO UNDERSTANDABILITY RATIO
PROPORTION OF SERVICES UNDERSTOOD AFTER READING THE COMPONENT DESCRIPTION
Table 3. Metrics associated to Quality of Documentation
ATTRIBUTE
INDICATOR
INDIRECT METRIC
DESIGN LEGIBILITY (READABILITY)
MEANINGFUL NAMES
PROPORTION OF FUNCTIONAL ELEMENTS WITH MEANINGFUL NAMES
INTERFACES UNDERSTANDABILITY
FUNCTIONAL ELEMENTS UNDERSTANDABILITY
PROPORTION OF FUNCTIONAL ELEMENTS USED WITHOUT ERRORS
I/O MESSAGE U
I/O UNDERSTANDABILITY
PROPORTION OF EXCEPTIONS CORRECTLY UNDERSTOOD
—7—
UNDERSTANDABILITY
PROPORTION OF RETURN VALUES CORRECTLY UNDERSTOOD PROPORTION OF ARGUMENTS CORRECTLY UNDERSTOOD TIME TO USE
AVERAGE TIME TO USE CORRECTLY THE COMPONENT
TIME TO EXPERTISE
AVERAGE TIME TO MASTER THE COMPONENT FUNCTIONALITY
EASE OF COMPONENT LEARNING
CONFIGURABLE PARAMETERS PER INTERFACE RATIO CUSTOMISABILITY
CUSTOMISABILITY RATIO CONFIGURABLE PARAMETERS PER OPERATIONS RATIO ERROR MESSAGES SUITABILITY
ERROR MESSAGE PER FUNCTIONAL ELEMENT DENSITY
ERROR MESSAGES CLEARNESS
PROPORTION OF ERROR MESSAGES CORRECTLY UNDERSTOOD
QUALITY OF ERROR MESSAGES
OPERATIONS PER INTERFACE DENSITY INTERFACES COMPLEXITY
INTERFACE DENSITY EVENTS PER INTERFACE DENSITY
Table 4. Metrics associated to Complexity of Design
In this point, we need to define the link between these attributes—including the metrics that assess the Usability measurable concepts related to these attributes—and the (particularized) Quality Model proposed by ISO 9126. Thus, we should define a connection between the Quality of Documentation and the Complexity of Design to the Understandability, Learnability and Operability. We think that there is not a unique direct relationship between a metric and a quality subcharacteristic, but that there are different degrees of relation between every metric with each subcharacteristic. Table 5 shows the relations between attributes and usability subcharacteristics based on the authors’ own opinion and experience. Note that Quality of Documentation mainly affects Understandability and Learnability. On the other hand, Complexity of Design affects to Operability. The degree (Low, Medium, High) of these relations is shown in Table 5.
6 Discussion There are several points and assumptions of this position paper that we want to discuss further. First, there is a main question about who the “user” of a software component is. Our work is based on the assumption that the user is the system developer, but we know this is a controversial issue, and different points of views should be accepted. Some of the metrics proposed in this work are subjective and, therefore, difficult to automate. We are aware that we should concentrate our efforts in objective metrics, discarding most of the metrics which values are calculated by a subjective process, or defining correlations that allow to infer their values from the value of others objective metrics. Today, the definition of software product attribute is not a closed. If we define an attribute as “a measurable physical or abstract property of an entity” [8], then we need to use metrics that measure directly attributes. In this case, only direct metrics are adequate. But if we think that a metric is either an indicator or a direct metric, then our attributes could be much more generic, and therefore we can define valid indicators for them.
—8—
ATTRIBUTE/QUALITY CHAR.
UNDERSTANDABILITY
LEARNABILITY
OPERABILITY
CONTENTS OF MANUALS
LOW
HIGH
MEDIUM
SIZE OF MANUALS
LOW
HIGH
MEDIUM
EFFECTIVENESS OF MANUALS
LOW
HIGH
HIGH
CONTENTS OF DEMOS
HIGH
LOW
LOW
CONTENTS OF HELP SYSTEM
-
HIGH
HIGH
SIZE OF HELP SYSTEM
-
HIGH
MEDIUM
EFFECTIVENESS OF HELP SYSTEM
-
HIGH
HIGH
CONTENTS OF MARKETING INFO
HIGH
-
-
EFFECTIVENESS OF MARKETING INFO
HIGH
-
-
DESIGN’S LEGIBILITY (READABILITY)
MEDIUM
LOW
HIGH
INTERFACES UNDERSTANDABILITY
LOW
LOW
HIGH
UNDERSTANDABILITY OF I/O
HIGH
LOW
HIGH
-
HIGH
MEDIUM
CUSTOMISABILITY
LOW
MEDIUM
HIGH
CONTENTS OF ERROR MESSAGE
LOW
LOW
HIGH
INTERFACES DENSITY
LOW
HIGH
HIGH
EASE OF COMPONENT
LEARNING
Table 5. Connecting Software Component Attributes and Usability Subcharacteristics.
The degree of influence that component attributes have over component subcharacteristics has been defined according to the authors’ own opinion and experience. A more detailed study is required in order to confirm the results presented in Table 5. This information is the starting point for a future empirical research where the level of influence of each pair attribute-subcharacteristic is measured, and their relationship is statistically tested. Finally, there is the issue of the measurable concepts used for components usability. In this case they coincide with the concepts proposed in typical metrics literature: size, structure and complexity.
7 Future Work Apart form investigate the open issues mentioned above, there are some areas of research that we plan to address in the short term. First, we need to define without any ambiguities the different terms used in the aforementioned metrics. For example, what is exactly a manual page, and a help screen? Or, what is a version? Once the selected metrics are clearly defined, we need to validate them, both formally and empirically. We also need to define the analysis model of all indicators. Then, we plan to perform a Principal Components Analysis on the proposed metrics [9] to select the minimum set of orthogonal metrics that influence the usability of a software component. With all the metrics already tested and validated, we need to search for generic indicators for Understandability, Learnability and Operability, and, obviously, a unique indicator for software component Usability. Finally, all metrics and indicators need to be progressively refined as we count with more empirical experiences.
—9—
References [1] M. F. Bertoa, J. M. Troya, A. Vallecillo. “Quality Attributes for COTS Components.” I+D Computación, 1(2):128-144, Nov. 2002. [2] M. F. Bertoa, J. M. Troya, A. Vallecillo. “A Survey on the Quality Information Provided by Software Component Vendors.” In Proc. of the 7th ECOOP Workshop on Quantitative Approaches in Object-Oriented Software Engineering (QAOOSE 2003), pp. 25-30. Darmstadt, Germany, July 2003. [3] ISO/IEC 9126-1:2001 “Software Engineering—Product Quality—Part 1: Quality model”, June 2001. [4] F. García et al., “An Ontology for Software Measurement,” Technical Report UCLM DIAB-04-02-2. Computer Science Department, University of Castilla-La Mancha, Spain, 2004. Available at : http://www.infoab.uclm.es/trep.php?&codtrep=DIAB-04-02-2. [5] B.W. Boehm, J.R. Brown, J.R. Kaspar et al., “Characteristics of Software Quality”, TRW Series of Software Technology, Amsterdam, North Holland, 1978.. [6] N. Fenton and S.L. Pfleeger. “Software Metrics: A rigorous approach” 2 ed. Chapman & Hall, London, 1998. [7] J. McGarry, D. Card, C. Jones et al., “Practical Software Measurement”, Addison-Wesley, 2002. [8] IEEE 610.12. “Standard glossary of software engineering terminology.” IEEE Press, 1990. [9] S. Elbaum, D. Gable, G. Rothermel “Understanding and measuring the source of variation in the prioritization of regression test suites”, Proc. of seventh Intl. Software Metrics Symposium (Metrics’01), pp. 169-179, London, April 2001.
— 10 —
