software quality attributes, it also provides a quality assessment process aiding ...... The language, the culture and general business issues of the original system ...
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version.
E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks Antonia Stefani, Michalis Xenos Hellenic Open University 16 Sachtouri st., GR-26222, Patras, Greece Phone: +30 2610367405 Fax: +30 2610 367520 E-mail: {stefani,xenos}@eap.gr
Abstract: As business transitions into the new economy, e-system successful use has become a strategic goal. Especially in business to consumer (e-commerce) applications, users highly evaluate the quality of their interactive shopping experience. However, quality is difficult to define and measure and most importantly, it is difficult to measure its impact on the end-user. Among the many research questions that arise, some of the most important concern the exact nature of the quality attributes that define an e-commerce system, and how one could model these attributes in order to increase its acceptance. Bearing in mind that e-commerce systems are actually user/data-intensive web-based software systems, this work performed a survey which resulted in a theoretical model that helps to measure such systems’ dynamics through their decomposition into primary quality characteristics. The proposed model is based on Bayesian Networks and ISO 9126. Besides the emphasis on specific software quality attributes, it also provides a quality assessment process aiding developers to design and produce e-commerce systems of high quality. Using a Bayesian Network the model can be used to combine different types of evidences and provide reasoning from effect to cause and vice versa.
1.
INTRODUCTION
Commerce already counts several years of on-line presence. During these years several lessons have been learned about the technology, business and economy of e-commerce systems. A successful business endeavor in cyberspace usually requires a good (not necessarily innovative) business idea, some kind of an infrastructure and a flexible business model. Success or failure depends on the combination of many factors; it always however, includes end-user acceptance of the software system that supports this endeavor. This is especially true nowadays since the significant penetration of the Internet worldwide has increased the number of both novice and demanding users (Bidgoli, 2002). The nature of E-commerce is diverse (Lohse & Spiller, 1998). Some features remain the same (e.g. core functions) while others are adapting. The fact is that e-commerce technologies change rapidly, spanning new application areas (such as V-commerce (Johnson et al., 2003) and M-commerce (Varshney & Vetter, 2002), extending existing functions and creating new ones; dynamicity is the power that changes the face of e-commerce systems. Furthermore, e-commerce systems, enabled by the Internet, Web, and hypermedia technologies, are highly dynamic and interactive in nature, utilize rich
1
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. hypermedia mechanisms in user interfaces for information presentation and provide a tremendous amount of control over temporal aspects of information delivery to end-users (Yong & Sanders, 2005). The problem of identifying the factors that determine end-user acceptance in software systems is not new (Chen et al, 2004). E-commerce systems in particular have a distinct characteristic: the total involvement of the end-user at almost every stage of the purchasing process (Henfridsson & Holmstrom, 2003); this is not the case with other on-line software systems. Designing a successful B2C (Business to Consumer) system requires a bulletproof underling business process workflow, or in other words fulfillment of specific functional requirements and quality of service –sometimes referred to as non-functional requirements. The latter, and quality in general, is often underestimated especially at the first stages of system design/development. User-intensive systems of this kind do not forgive such mistakes and costs escalate when corrections have to be made in the last stages of development or even after system deployment. Quality is important. Again this is an advice cited many times and followed even lesser. But where are the methods that should be usedespecially for B2C systems? Undoubtedly, in order to ensure the production of high quality e-commerce systems, it is important for developers to be able to assess the quality of B2C systems. But quality is inevitably linked with the end-user’s perception of quality. So the question arises: how can one evaluate B2C systems and define the extent to which they meet endusers’ requirements? To this end, it is necessary to provide a model for assessing system quality and an evaluation process to provide quality measures. This model should combine evidence of different types based on a formal standard. This is an area that has not been well covered. Few approaches including those of Chen et al. (2004) and Losavio et al. (2004), partially cover these requirements. In this context, some of the most significant research questions are: 1.
How can an e-commerce system be assessed by the end-user? Online shopping behavior is influenced by several factors like: usability issues (Nielsen, 2000); efficiency of the system and especially response time (Shaw & DeLone., 2002); privacy and security (Slyke et al., 2004) mechanisms and features that support the end-user’s intention to buy; quality of data (Yong & Sanders, 2005). The question posed here is how these factors affect end-user’s perception of quality.
2.
Measures of probability values can be used in order to calibrate/categorize an e-commerce system to different quality categories but also to detect any drawbacks that an e-commerce system has. How can these probability values be evaluated with little or minimal previous knowledge?
3.
End-user experience is a critical determinant of success. If end-users cannot find what they’re looking for, they simply cannot buy it; a site that buries key information impairs business decision making. Poorly designed interfaces increase user errors, which can be costly (Lee & Benbasat, 2003). A user –centered design approach supports all the tasks users need to accomplish (Terry & Standing, 2004). How is it possible to provide the end-user perception of e-commerce systems’ quality to the designer/developer? Evaluation of E-commerce systems may be based either on usability or quality. Usability evaluation
includes, among others, feature inspection, collection of data about end-users’ opinion using questionnaires and log-analysis methods. Indeed, these methods provide an important feedback to the
2
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. researcher and their results can be utilized as a useful background providing guidelines for e-commerce systems’ design. Quality on the other hand is broader, encompassing many more features besides usability and is thus more adequate for B2C system evaluation. The commonly accepted ISO standard is a good starting point for categorizing the quality characteristics of an e-commerce system. End-user perception is usually based on incomplete or uncertain information so probabilistic reasoning may be successfully used to provide calculations of quality measures. This paper builds on these two ideas in order to measure e-commerce system quality. Quality is difficult to define and measure and most importantly, it is difficult to measure its impact on the end-user. Even single quality scores for an entire system, such as the one proposed by Sauro and Kindlund (2005), contain too much bias. Among the many research questions that arise, some of the most important concern the exact nature of the quality attributes that define an e-commerce system and how one could model these attributes in order to increase its acceptance (Elfriede & Rashka, 2001). Bearing in mind that e-commerce systems are actually user/data-intensive web-based software systems, this paper proposes a theoretical model that helps to measure such systems’ dynamics through their decomposition into primary functional/quality characteristics. The model is based on the ISO 9126 quality standard (ISO/IEC, 2001) and it can be used for the quality assessment of e-commerce systems from two alternative perspectives: from the end-user point of view, as the quality of the final system and from the developer point of view, as the quality of the system’s attributes during the development process. More specifically, the model’s structure relies on the set of quality characteristics and subcharacteristics that are directly related to quality as perceived by the end-users (referred to hereafter as external measures). External measures are those quality measures that require the involvement of the end-user to be evaluated. The proposed model uses a Belief Network for probabilistic reasoning: forward prediction (future estimation) and backward assessment. The aim here is to rank the components of an e-commerce system based on their importance according to the user’s perception. The theoretical background supports the evaluation process scenarios for the quality of e-commerce systems. Belief Networks have already been used successfully for software process modeling and production (Stamelos et al, 2003; Bibi & Stamelos, 2004). The contribution of this work is three-fold. Firstly, this research addresses the issue of how to predict users’ perception of e-commerce systems quality. The model adapts the ISO 9126 quality standard to B2C e-commerce systems based on the results of a 2 year survey that included the participation of 300 users. The results are of great interest to web designers, IS staff and researchers. Secondly, by explaining the dynamic relationship among quality and e-commerce systems’ components that influence e-commerce success, the current research can aid researchers in further refinement of e-systems success models in general. Last but not least, the current study provides a reasonable background for applying existing measures of information systems’ success on the ecommerce domain. This paper, besides this introduction is structured as follows: the second section contains the information on e-commerce systems, on the ISO quality model and on the Belief Networks theory. The third section presents the theory underling the model while the fourth section describes the mapping
3
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. between the quality attributes and the system functions/services. The fifth section describes the evaluation process and gives some examples of its use, while the main conclusions of the paper are outlined in the sixth section.
2. THEORETICAL BACKGROUND OF THE MODEL 2.1 E–Commerce Systems Categorization Zwass (1996) defines e-commerce as “sharing business information, maintaining business relationships and conducting business transactions by the means of telecommunication networks”. Depending on the type of transactions performed electronically, Chan et al., (2001) define two basic categories of e-commerce systems: Business to Consumer (B2C) and Business-to-Business (B2B). In turn, B2C systems can be classified into three general categories. The first category consists totally of online systems whose primary contact points with consumers are in the virtual world. Consumers can use personal computers in order to search for products, compare prices and characteristics and possibly purchase an item or more. The second category represents those existing businesses that compliment their operation with online presence trying to increase the number of communication channels with the consumers. In this case the consumer can use either the e-commerce system or actual stores. The last category consists of e-commerce systems that are totally virtual. In this category there is no physical store and the consumer can use only the e-commerce system to proceed to a purchase. Most e-commerce systems seek to provide high quality services to the end-users, i.e. the customers, and to this end they employ specific characteristics/attributes so as to meet specific end-user requirements. Examples of such characteristics are the searching capabilities, flexible navigation or personalization mechanisms. Even if e-commerce systems change in the future, the basic processes supported by the business to consumer domain will largely remain unchanged (i.e the user searches for products, evaluates attributes and characteristics and proceeds to a purchase). It is thus, reasonable to conclude that the quality and evaluation methods of e-commerce systems will always be dependent on system’s ability to meet end-user requirements. Such quality factors should be taken under serious consideration during the development phase.
2.2 The ISO 9126 Quality Model Quality frequently means the ‘fitness for use’ and the ‘ability to meet end-user’s requirements’ (Yeh, 1993). According to ISO 9126, quality is defined as ‘a set of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs’ (ISO/IEC, 2001). ISO 9126 is a quality standard for software product evaluation and provides quality characteristics and guidelines for their use (ISO, 1991). The 2001 edition of ISO/IEC 9126 is divided into four parts (Cote et al, 2005; Abran et al, 2003). 1.
ISO/IEC 9126- 1: Software Engineering—Product quality—Part 1: which provides the Quality model. This part of the standard specifies two distinct structures for software quality:(a) External quality is modeled with a four characteristics: functionality, reliability, efficiency, usability. We use these characteristics in our model. Internal quality is modeled with a set of two characteristics: maintainability and portability. These
4
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. characteristics could be used in the backward use of an extended version of the model. (b) Quality-in-use characteristics are modeled with four other characteristics: effectiveness, productivity, security and satisfaction. Figure 1 depicts the ISO9126-Part1a quality model. I ISO 9126
external quality characteristics
Fuctionality
suitability accuracy interoperability security
internal quality characteristics
Usability
Efficiency
Reliability
understandability learnability operability attractiveness
time behavior resource behavior
maturity fault tolerance recoverability
Maintainability
analyzability changeability stability testability
Portability
adaptability installability co-existence repleceability
quality sub-characteristics
Figure 1. The ISO/IEC 9126- Part 1: Quality model- External and internal quality characteristics.
2.
ISO/IEC 9126-2: Software Engineering—Product quality—Part 2: External metrics. This part describes the measures that can be used to specify or evaluate the behavior of the software when operated by the user.
3.
ISO/IEC 9126-3: Software Engineering—Product quality—Part 3: Internal metrics. This part describes the measures that can be used to create the requirements that describe the static properties of the interface, which can be evaluated by inspection without operating the software. This part mainly refers to the developer.
4.
ISO/IEC 9126-4: Software Engineering—Product quality—Part 4: Quality in use metrics. This part describes the measures that can be used to specify or evaluate the impact of the use of the software when operated by the user. This part also refers to the developer.
ISO/IEC 9126 is not only a model for use in the evaluation of quality, but also a model for use in the specification of quality needs. ISO 9126 may be used as basis for e-commerce quality evaluation but further analysis and mapping of its characteristics and sub-characteristics to system functions/services is required. Adopting and adapting ISO 9126 for specific domains is not new and not foreign to the standard itself (Cote et al, 2005). The usual approach is to enhance the hierarchical and (by design) open scheme to include more sub-characteristics suitable for the domain. The ISO 9126 quality model defines external quality characteristics which provide an appreciation of the quality as seen from a user’s perspective. Additionally internal quality characteristics refer to fails at all levels and are developer oriented in order to satisfy end user’s requirements. External and internal quality defines quality requirements for a system which are originated in both cases from the end-user.
5
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. In this work, we use the following external quality characteristics of ISO 9126 quality model to evaluate e-commerce systems: Functionality, Usability, Efficiency and Reliability. The external quality model aims at defining the quality attributes that are important for the end user (Pfleeger, 2001). Each one of the external quality characteristics and their sub-characteristics provide the quality framework (actually the baseline) on which an e-commerce system may be built, taking into account the satisfaction of end-users requirements. The main question posed is how e-commerce system’s quality can be analyzed using this standard.
2.3 Belief Networks The proposed model is based on the notation and formation of Causal Probabilistic Networks, also called Belief Networks (BN) or Bayesian Networks (Jensen, 1996). The mathematical model on which Bayesian Networks are based is the theorem developed by the mathematician and theologian Thomas Bayes (Formula 1), p ( h / e) =
p (e / h)* p (h) (1) p (e )
where p(h) is the prior probability of hypothesis h; p(e) is the prior probability of evidence e; p(h | e) is the probability of h given e; p(e | h) is the probability of e given h. A BN is a graphical model that encodes probabilistic relationships among variables of interest. Nodes represent variables and the directed arrows the relations between them. Therefore, a BN is a graphic network that describes the relations of probabilities between the variables (Fenton et al., 2002). The graphical model has several advantages for data analysis (Heckerman, 1995; Fenton et al., 2004). First, it encodes dependencies among all variables and handles situations where some data entries are unavailable. Second, a BN can be used to learn causal relationships and hence gain understanding about a problem domain and predict the consequences of intervention. Third, because the BN model has both a causal and probabilistic semantics, it is an ideal representation for combining prior knowledge and data. Finally, BNs offer an efficient and principled approach for avoiding the overfitting of data. The use of BNs not only makes it possible to define the relation between the various nodes (variables), but also to estimate consistently the way in which the initial probabilities influence uncertain conclusions, such as the quality of an e-commerce system. In this case, BNs are used for future estimation, or –as also called– forward prediction. Furthermore, BNs can be used to speculate about the states of the initial nodes, based on a given final and some intermediate variables. This is called backward assessment. In order to define the relations between the variables, the dependent probabilities that describe the relations between a ‘child’ node and its ‘parent’ nodes must be determined. If the values of each variable are distinct, then the probabilities for each node can be described in a Node Probability Table (NPT). This table presents the probability that a ‘child’ node is assigned a certain value for each combination of possible values of the ‘parent’ nodes. In the quality domain for example, if the quality characteristic of Efficiency (E) is represented by its two quality sub-characteristics Time Behavior (TB) and Resource Behavior (RB), a BN network of two ‘parent’ nodes named ‘TB’ and ‘RB’ and one
6
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. ‘child’ node named ‘E’ can be created. The probability table of node E reflects the probability P(E|TB,RB) for all possible combinations of E, TB, RB. Each of the above nodes can have a number of possible states that can be defined at the development of the BN network, can differ for every node and represent the model’s uncertainty. Thus, in this example, since there are two possible states for node TB (tb1, tb2), three possible states for node RB (rb1, rb2, rb3) and three for node E (e1, e2, e3), the NPT of node E includes 3*2*3=18 elements.
3. A MODEL FOR E-COMMERCE SYSTEM QUALITY ASSESSMENT The philosophy underlying the proposed model is the creation of a dynamic network that concentrates and exploits the knowledge gained from the analysis of data gathered during a user survey (Stefani et al., 2003; Stefani et al., 2004) and that can also use its own results for future estimations (see section 5.1 for details). The construction of a BN model follows a common set of guidelines: (a) Include all variables that are important in modeling a system (b) Use causal knowledge to guide the connections made in the graph (c) Use existing knowledge (if any) to specify the conditional distributions. The BN structure contains nodes, each one corresponding to quality characteristics and sub-characteristics and other attributes of an e-commerce system. An ‘attribute’ may be any function or service that an ecommerce system offers to the end-user through the user interface. In this multi-level tree-like structure, the root corresponds to the overall quality, the non-leaf level nodes to quality subcharacteristics and the leaf nodes to system functions and services (figure 2).
Root (Overall quality)
Intermediate nodes (quality characteristics, subcharacteristics)
Leaf nodes (functions, services)
Figure 2. The tree-like structure of the BN model. The conditional distribution that defines the relations between the variables defines the NPT for every node. The NPT for each node reflects the dependent probabilities that describe the relations between a ‘child’ node and its ‘parent’ nodes. In the proposed BN model, intermediate nodes (Quality characteristics and Quality sub- characteristics) are ranked taking values ranging from “Poor” to “Average” and “Good”. The strength of influence of the parents is captured by the probabilistic distribution of the node. Nodes are structured hierarchically in correspondence to the structure of ISO9126: the overall quality (root) is comprised of the quality characteristics and sub-characteristics (intermediate nodes). Services and functions of the e-commerce system are logically associated with one or more intermediate nodes, a kind of overlap. In our model each function is represented by its qualitative correspondents: its view is different, depending on the quality characteristic angle it is
7
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. viewed from. This logical association corresponds to the evaluation of a function f using k different quality perspectives, actually k sub-characteristic. For example, the function ‘search engine’ is evaluated from 3 qualitative perspectives: ‘accuracy’, ‘learnability’ and ‘operability’ (the mapping between the functions and the sub-characteristics is described in section 4). In turn, each function may have ml (ml >0) view for the perspective l (with l ∈ [1,k]). So each function is actually represented by: k
∑m l =1
l
leaf nodes, each leaf node having a one-to-one association with an intermediate node. For example, the function ‘search engine’ is viewed as ‘informative feedback’, ‘spelling’, ‘term expansion’ and ‘visualisation’ from the ‘accuracy’ sub-characteristic point of view (so ml =4 for this subcharacteristic). For simplicity, each leaf node takes Boolean values: ‘Yes’ if the component is present in the system under examination and ‘No’ otherwise. An alterative approach could use a non-Boolean scale. The model is based on the input of data collected from the users (referred to as evidence from now on) for the nodes which represent the components/variables of e-commerce system’s quality. If no evidence is inserted the estimations provided by the model are based on previously collected experience as inserted in the NPT. New evidence affects the probabilities of the nodes and the estimation for each node is different. Even if an intermediate node cannot be assessed directly, the BN enables researchers to make observations about known variables and infer the probabilities of others which have not been observed yet. The application of the model is the most important part of the evaluation process of e-commerce systems, including a forward and a backward use. This process is organized by the ‘evaluator’, whose role is to define the quality goals and apply the evaluation scenario. The BN estimates values for various quality characteristics by getting the data from the users or by performing an assessment based on previously inserted data. Using these characteristics, the evaluator advices the developer/designer. This process is depicted in figure 3. Users
Forward Use
Evaluator
d
bbb Load Model (leafs)
Quality classification
Probabilistic Estimation (parents)
Probabilistic Estimation (leafs)
Scale calibration
Quality classification
Load Model (parents)
d
d Developer
Backward Use
Evaluator
Figure 3. The BN model’s forward and backward use.
8
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. The forward use can be utilized to assess the intermediate nodes of the model. These nodes represent variables that are regarded as an ‘external’ measure of an e-commerce system. The quality characteristics are used as the targets for validation (external quality). They are refined into subcharacteristics, until the attributes or measurable properties are obtained. Measurement means to use a metric or measure to assign a value to. By using the model, measures about the external quality of an ecommerce system are possible. In forward use, the evaluator inserts the available evidence (measures) related to the e-commerce system in each leaf node of the model. The model can then be used to provide estimations about the system quality and also to provide the corresponding probability values. A Scale Calibration Table (SCT) is then used to classify the system according to its quality. It is worth mentioning that the model can provide estimations even if evidence has not been inserted in all of its leaf nodes. Of course, as more evidence is loaded into the model, accuracy increases. Evidence in the forward use represents the Boolean values of leaf nodes states that are obtained in the BN model. In backward use, the model calculates probabilistic values for the developer-centered external measures of an e-commerce system. These external measures are strongly related to the internal measures of the development phase. As internal measures, are characterized those variables that are used as targets for verification (internal quality) at various stages of the design and development cycle. In order to monitor and control quality during these two phases, the external quality requirements are translated and transferred into the requirements of the intermediate nodes/modules and child nodes/functions, obtained from the development activities. The users at the backward use of the BN model are experts on e-commerce system quality, capable to define or re-define evidence for quality characteristics and sub-characteristics. A detailed description of forward and backward use is provided in sections 5.3 and 5.4.
4. THE MAPPING BETWEEN QUALITY AND FUNCTIONS/SERVICES This section describes the relations among quality characteristics, quality sub-characteristics and function/features of an e-commerce system and how the latter support the overall quality. Additionally figures present sub-networks of each quality characteristic its sub-characteristic and the related ecommerce systems functions. Each sub-network is extended to sub-functions and systems’ attributes that support the quality sub-characteristic and describe how this can be achieved in the e-commerce system. Finally, we discuss e-commerce systems functions/services from different quality views, views dependent on the quality sub-characteristics of ISO9126. As a result the quality subcharacteristics overlap by presenting a different angle of the same feature.
4.1 The Functionality Characteristic Functionality refers to a set of functions and specified properties that satisfy stated or implied needs (ISO/IEC, 2001). The meaning of Functionality is to provide integrative and interactive functions in order to ensure end-user convenience. An overview of the sub-network for Functionality is presented in Figure 4. It is comprised of four quality sub-characteristics: suitability, accuracy, interoperability and security.
9
Informative feedback (spelling, term expansion) Visualization Metaphors, Product’s Presentation
Purchase
Accuracy
Search
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version.
Figure 4. Overview of the Functionality sub-network.
An e-commerce system consists of functions that support navigation, language interpretation and personalization providing the required tasks and the suitable functions for end-user interaction with the e-commerce system. Functional navigation is based on navigation maps and indexes, i.e. suitable and accurate overview diagrams that help end-users to locate information. Similarly, language could be analyzed into terminology, internationalization and localization. The existence of simple terms and common
symbols
creates
a
friendly
interface,
easily
understandable
to
the
end-user.
Internationalization refers to the ability of an e-commerce system to be used by users of different languages. It involves the use of culture independent language and symbols that can be understood by non-native speakers. The localization of the system refers to making an adaptable version of the ecommerce system for a specific locale. Localization is the process by which web system applications are analyzed and adapted to the requirements of other countries, making the system more usable for the specific customers. Although translating Web site content is a major aspect of localization, it also involves changing many other aspects of the Web application, such as color, graphics, and structure (Collins, 2002).
10
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. The suitability of an e-commerce system is also based on personalization functions and recommendation mechanisms. Personalization is the ability to provide content and services tailored to individuals based on knowledge about their preferences and behavior (Adomavicius & Tuzhilin, 2005). Also personalization functions are the basis for recommendation functions. E-commerce systems usually deliver personalized information to consumers in several ways, including narratives, lists ordered by relevance, sets of alternatives, and various types of visualization. One classification of recommendation methods is known as “pull, push, and passive” (Schafer et al., 2001). Pull methods notify consumers that personalized information is available but display it only on request. Push methods (such as sending emails) are intended to reach consumers not already interacting with the personalization system (such as through a company’s e-commerce Web site). Passive delivery displays personalized information as by-products of other consumer activities; for example, a consumer looking at a product on a Web site is also presented with recommendations for related products. The e-commerce system offers information about products and services when the purchase procedure is incurred. This also includes analytical product description (information about product features) and cost analysis (value, taxes and shipment costs) in order to provide accurate product information. In current e-commerce systems, the end-user goes through the purchase procedure using metaphors, such as shopping carts and shopping/wish lists. The searching procedure refers to the search function that each e-commerce system provides. A search function should offer an interface adaptable to end-user needs providing informative feedback and results’ visualization. In order to improve the accuracy of searching results the system should, ideally, perform spelling checks and offer synonym term expansion. According to ISO916, security refers mainly to the ability of the system to provide a secure transaction environment. Its role is to support e-commerce credibility and this can be accomplished by increasing the confidence in commercial transactions over the web, where transacting parties are invisible to each other. In the proposed model, the basic components related to security are the trustworthiness of the system (which directly relates to the vendor behind the system and how this can be perceived by the end user), the trust of transaction processes and privacy. Trustworthiness, means that consumers expect that a system will operate successfully across a number of dimensions before they are willing to proceed to a purchase (Serva et al., 2005). The privacy of the system may be based on privacy seals such as TRUSTe, CPA WebTrust, and BBBOnline (Moores, 2005). These privacy seals should be familiar to the user who should understand their functional use. It should be mentioned that security has an important role to e-commerce system success and it is a multidimensional quality characteristic related with aspects as confidentiality and integrity. One could further explore the role of security using the ISO/IEC 13335 standard (ISO/IEC 13335-1, 2004) which emphasizes on security techniques, management of information and communications technology security. Finally, the interoperability of the system refers to the various technologies used by the ecommerce system and how these technologies interact with the end user’s system. The basic component related to interoperability is browser independence: the ability of the system to be accessible and usable by any browser. This is extremely important because the e-commerce system
11
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. should provide the same layout (template) to the user, and this layout should be as much as browser independent as possible.
4.2 The Usability Characteristic Usability is defined as a set of attributes that bear on the effort needed for the use of a product or service, based on the individual assessment of such use by a stated or implied set of users (ISO/IEC, 2001). With reference to ISO 9126, usability is comprised of four quality sub-characteristics: attractiveness, learnability, understandability and operability. Usability is an important quality characteristic as all functions of an e-commerce system are usually developed in a way that seeks to help the end-user by simplifying end-user actions. An overview of the sub-network for Usability is presented in Figure 5.
Usability
Attractiveness Visibility Multimedia (audio, video, animation), Text
Understandability
Interactivity issues
Learnability
Search
Template
Search Template (vocabulary, multiple inputs), Common features
User Oriented Template (category pages, Inter-connectivity issues)
Language glossary, Checkout features, Informative Features (Help, communication)
Operability
Search Alternatives Administration issues Synonyms, Non-product terms, Expert’s Parameters Boolean Search Search History
Profiling Multiple language Support Recommendation facilities
Figure 5. Overview of the Usability sub-network.
The use of multimedia/hypermedia technologies support e-commerce systems in order to be attractive to the end users. Interfaces that offer a relaxed user experience include features such as user oriented
presentation
of
products
and
services,
visibility
of
images
and
additional
multimedia/hypermedia presentations (in the form of hypertext, audio, video or animation). The need for adaptability to user needs makes the process of designing, developing and maintaining highly interactive applications based on hypermedia technologies a challenging task. Nevertheless, the use of text is still the basic method of product presentation. Text should be short, without errors and readable,
12
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. so the end-user must be able to look at the text and locate all the appropriate product information within a few seconds. E-commerce system’s template supports an environment that the end-user can learn to use with ease. The e-commerce systems’ template should be adapted to the varying levels of users’ familiarity. The system should provide user-oriented hierarchy of web pages by considering multiple classification schemes and offering category pages which reveal the product hierarchy. E-commerce systems that provide links to purchasing options on the home page; return policy; shipping and delivery information ensure the learnability of navigation and purchase features. Search learnability refers to features that are commonly used, like text boxes instead of a link to a search page, limited vocabulary search and multiple word inputs. Informative features such as glossary, language options, indexes and feedback at the registration and checkout process enable the user to understand whether the system suits his/her goals, and how it can be used for performing particular tasks. Language options support the internationalization of an ecommerce system where each end-user interacts with the system in an understandable environment. However, language support is not the only prerequisite. Going global means serving international user needs; these needs are shaped by economic, social and cultural factors. Although it is not possible to anticipate all possible needs, simple language, prices in different currencies, information about VAT and post charges for international destinations are some of the good practices exhibited by successful international e-commerce vendors. Existing systems destined for use at a national level can be changed and extended in subtle ways that make them usable in other parts of the world. For example, changes may include international shipping routes of merchandise; checkout and registration procedures that cater for international end-users; text language in a simplified form of English, suitable for non-native speakers. Some times e-commerce systems need to provide localization services by offering connection to local sites. A localized e-commerce system appears as if it was originally developed in the target language. The language, the culture and general business issues of the original system are adapted to the local market. In order to support the understandability of the localization service/function, it is required not only to translate but to also replace the original product selections, names, specifications, prices and business practices with those of the target country’s market. Help is another component of the understandability characteristic where the end-user can find information about the checkout, registration or searching process. The end-user can also find answers in the Frequently Asked Questions (FAQ), use interactive help functions or contact support by e-mail or phone. Finally, searching features that support various forms of keywords, make use of logical operators and provide similarity search options are advantages for the end-user. An operable search mechanism is adjusted to respond to the way that end- users search by providing special treatment to frequent queries, supporting non-product terms, accepting synonyms and providing advanced search to expert users. The search results should be sorted based on criteria like topic, date and compatibility and categorized according to well-organized category labels. When using the search engine, end-users provide input in the form of keywords, possibly using forms with multiple fields that map to product characteristics (e.g. model, color, price range). The use of Boolean operators (such as ‘AND’, ‘OR’) are now common
13
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. in most systems. The provision of search alternatives (e.g. find similar queries) is extended by search results processing. Search results processing is defined as the categorization of search results based on various criteria (price, date, etc.), or the processing of an existing search question using more criteria based on search history. Getting better results from a search engine in terms of quality through more efficient information retrieval processes is complicated since different factors have to be taken into account: both objective (e.g. relevance) and subjective factors (e.g. algorithmic complexity). Additional services such as multiple language support, technical support and help are often overlooked in current implementations. Finally, the concept of personalized stores, where the system recommends products according to top sellers, end-users preferences, previous purchase or search history are increasing the operability of the system. One of the basic prerequisites of the personal store’s success is the existence of a complete personal profile. Personalization is the key for a successful recommender system adapted to end-users’ preferences. Recommender mechanisms have recently been added to many e-commerce applications by suggesting different buying alternatives to the end-users. Recommendation mechanisms influence the usability of the system and help the end-user to adjust personal factors (expectations, experience), system factors (such as product, speed, intelligence, services) and media factors (operability) to buying expectations.
4.3 The Efficiency Characteristic Efficiency is a complex concept that entails both conceptual challenges as well as implementation difficulties. Efficiency is defined as the capability of the system to provide appropriate performance, relative to the amount of resources used, under stated conditions (ISO/IEC, 2001). It refers to a state where system functions are both usable and successful, i.e. they achieve their aim, the reason for their existence. One of the main criteria of efficiency of an e-commerce system is the quality of sub characteristics relating to time and resource behavior. An overview of the sub-network for Efficiency is presented in Figure 6.
Figure 6. Overview of the Efficiency sub-network.
The quality characteristic of efficiency is particularly related to the time factor; e-commerce systems are developed with ‘time’ as one of the main parameters. From the user’s perspective, time behavior is one of the most important measures used to evaluate the quality of web systems’ performance. End-users typically perceive response time to be the amount of time taken from the moment they follow a link or make an action by clicking with the mouse to the moment that a new web page has been fully displayed on their screen. These are called loading conditions and are also related
14
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. to the resource behavior and the bandwidth of the underlying network infrastructure. Regardless of end-users’ connection, there should be easily accessible text information so the expert end-user or low bandwidth conditions end-user can reach the information needed without having to wait too long. The end-user expects instant response from each system function, especially from the visualization and personalization mechanisms. Uploading time is considered to be a great burden. Studies such the one of Nielsen (2006) suggest that the uploading limit for each web application should be less than 30 seconds. In current applications it should be less than 5 seconds, regardless of the type of media used for information presentation. Researchers also suggest that the end-user should be continuously provided with feedback about resources time behavior. Management of system resources is an important factor for defining the quality of an e-commerce system and refers to the visualization mechanisms and personalization applications. The efficiency of these mechanisms is related to the quality of the images (visualization features such as size, style, analysis and resolution quality). Navigation efficiency is related to fast page uploading and optional use of advanced technologies, like flash applications, virtual reality applications, etc. Similarly, search efficiency is linked to obtaining search results within accepted time limits, search capacity on the entire web-site or on part of the site or in the web. Furthermore, the time behavior refers to the time needed for the search results to appear. In many cases users cannot locate efficiently the information they need; thus, results should not only come quickly but they should be relevant to the user query as well. Query augmentation and result processing are some of the most important methods for improving the time and accuracy of a search. Based on user-profiling social filtering or log analysis, these techniques filter the initial response set of the query which is usually based on text analysis methods (e.g. keyword matching). The final answer set is not only smaller but more relevant as well. Time behavior is not only important in browsing or searching; end-users expect quick response time in the checkout and registration process which is usually a linear, step by step procedure. One step must be completed before the next one can be started, and no skipping around is allowed. This process generally starts in the shopping cart or order form and ends with the confirmation page. The provision of applications such as address books, currency converters, ZIP/postal code lookups, alternative payment options, order summary and confirmation page could be useful in order to complete the purchasing process.
4.4 The Reliability Characteristic Reliability is the quality characteristic that refers to a set of attributes that bear on the capability of software to maintain its performance level under stated conditions for a stated period of time (ISO/IEC, 2001). Reliability is comprised of three quality sub-characteristics: maturity, fault tolerance, and recoverability. An overview of the sub-network for Reliability is presented in Figure 7. Reliability refers to error-free and unconfused user experiences during navigation but also support in bottleneck situations. It refers mainly to system tolerance on end user’s actions and supports the accuracy on the delivered information. Characteristics like ‘Undo’ functions support the e-commerce system’s recoverability and error recovery for broken links, data entry errors and orphan pages are the most popular methods for ensuring a mature system. As far as the search function is concerned, a mature and reliable e-commerce system supports synonyms on search, non-product terms, and refinement on
15
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. search (results of a previous search). Since B2C systems handle economic transactions, economic transaction reliability is essential. Security of on-line transactions is one the main obstacles for a wider e-commerce adoption. System reliability, in this context, includes parameters such as transaction maturity and reliability of the information supplied to the user during the economic transaction. Furthermore in the transaction process, which forms the backbone of most mission critical business applications, reliability ensures the end-user’s atomicity, in the sense that a failure in the middle of an ongoing transaction does not leave any traces and persistence; the effects of a committed (i.e., successfully completed) transaction should survive future failures (Barga et al., 2004). The aim of the reliability characteristic is the formation of a stable and mature interaction environment for the user, which in-turn certifies the existence of a safe and stable system.
Figure 7. Overview of the Reliability sub-network. Finally a mature e-commerce system relies on the existence of reverse capabilities and the provision of end-user feedback. Also, the low rate of errors such as site crashes or error messages create an e-commerce system with high level of fault tolerance. Therefore, reliability of e-commerce systems actually means reliability of the provided services and relates to system maturity in interacting with the end-user, being tolerant to end-user errors and being characterized by a high level of recoverability.
5. IMPLEMENTATION AND USE OF THE MODEL 5.1 Data gathering In order to construct the components of the model, the relation among its nodes and the SCTs, we performed a two step survey: an end user evaluation and a literature review. The end –user evaluation included extensive experimental measurements on over 100 e-commerce systems by 300 users within a period of 2 years. In this evaluation we considered two types of e-commerce system users: experts and novice. Expert users have made at least 3 purchases using 3 different e-commerce systems. Novice users have used e-commerce systems numerous times but have not necessarily purchased anything. This variation enables the breadth in data collection and provides multiple interpretations about ecommerce systems’ quality. Using a five-grade Liker-type scale (1 = not important, 2 = important, 3 = average important, 4 = very important, 5 = critical), the end users where asked to define the relation among nodes in a predefined structure of the BN model. The initial values of node probability tables were defined by using as an input the data from the Liker scale. The final data were defined when the model was applied on several e-commerce systems. The definition of possible boundaries and scales of
16
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. the measurement data has been formed after performing extensive experimental measurements on over 100 e-commerce systems in the course of the research presented in (Stefani et al., 2004). The evaluation process was performed in two steps: a) end-users were asked to fill-in an evaluation sheet, b) answer pre-process, that is examination of the similarities between the answers; if the opinions of the majority converge (only in simple questions such as “is there an advanced search engine available?”), the data were considered to be reliable, otherwise individual contributions are lowered (or even ignored). Answers in more complex questions (e.g. “are the search engine results relevant to user queries?”) are kept as is. Although a large number of users tool part in the survey, subjectivity in matters of external quality could not be avoided.
5.2 Implementation Details The overall model is described by a general BN-network, which is composed of four sub-networks, each one corresponding to an external quality characteristic. The model has been implemented using Microsoft’s MSBNx authoring and evaluation tool version 1.4.2 (Kadie et al., 2001) and includes a total of 91 nodes organized in five levels. An extract from the BN (Quality sub-network and the NPT for node Quality) is presented in Figure 8.
Figure 8. Extract from the BN: NPT for the characteristic (node) of Quality.
The model can be used to identify specific areas that need improvement when different quality scenarios (forward/backward use) are applied. The application of the model can be valuable when the probability values for its nodes are meaningful. In other words, when the model estimates the probability values of the variables of the BN model, one should be able to classify this system and identify the specific fields that need to be improved. In the following section, complete scenarios of the model’s forward and backward use are presented.
17
Stefani A., Xenos M., “E-Commerce System Quality Assessment using a Model based on ISO 9126 and Belief Networks”, Software Quality Journal, Vol. 16 (1), pp. 107-129, March 2008. Pre-print version. 5.3 Forward Use of the Model At least two experts users or at least 10 novice users provide data through questionnaires. The answers are pre-processed and the evaluator loads the evidence in the leaf-level nodes. These are Boolean values, “Yes = 1” and “No = 0”. Afterwards, a bottom-up procedure estimates the overall system quality (the probability assigned at the root node) by calculating the probabilities of the non-leaf level nodes (the quality characteristics and sub-characteristics). The output of this step is a set of values for each node of the BN. The evaluator compares the values of each node, actually of each quality characteristic, with the corresponding ranges of the SCT (table 1).
Table 1. Scale Calibration Table. Scale Calibration Category
A
B
C
Quality
x>0,88 0,88>x>0,53
x0,82 0,82>x>0,55
x0,82 0,82>x>0,55
x0,93 0,93>x>0,80
x0,83 0,83>x>0,46
x0,83 0,83>x>0,61
x0,84 0,84>x>0,62
x0,80 0,80>x>0,57
x0,80 0,80>x>0,62
x0,84 0,84>x>0,62
x0,87 0,87>x>0,63
x0,89 0,89>x>0,72
x0,90 0,90>x>0,60
x0,82 0,82>x>0,57
x0,80 0,80>x>0,54
x0,90 0,90>x>0,39
x0,87 0,87>x>0,53
x0,86 0,86>x>0,44
x
