The current issue and full text archive of this journal is available at www.emeraldinsight.com/1750-6166.htm
TG 6,3
Evaluation of government e-tax websites: an information quality and system quality approach
300
Parmita Saha
Received 16 March 2011 Revised 18 July 2011, 17 February 2012 Accepted 7 March 2012
Department of Engineering and Science, Sogn og Fjordane University College, Sogndal, Norway
Atanu K. Nath Faculty of Social Sciences, Sogn og Fjordane University College, Sogndal, Norway, and
Esmail Salehi-Sangari Division of Industrial Marketing, KTH, Royal Institute of Technology, Stockholm, Sweden Abstract
Transforming Government: People, Process and Policy Vol. 6 No. 3, 2012 pp. 300-321 q Emerald Group Publishing Limited 1750-6166 DOI 10.1108/17506161211251281
Purpose – Despite the growth in adoption of technology by governments, the assessment of quality in electronically delivered public services has been relatively lacking. Past researches on information systems have identified information and service quality, system use, playfulness, and system design quality as critical factors of successful service delivery in e-commerce. The purpose of this paper is to empirically test the applicability of system and information quality criteria in evaluating government e-services websites. The main research problem identified in this study was “what are the system and information quality characteristics of government service delivery websites?” Design/methodology/approach – Based on a survey of 97 Municipalities from all the 21 regions of Sweden, the authors examine the presence of system quality features, specifically functionality, navigation, and accessibility; and measures of information quality in government websites. A systematic random sampling was chosen as the suitable approach; 290 municipalities are organized in 21 counties and we have chosen every 3rd number of municipalities from the list. An online survey was conducted, with 408 valid responses taken. A confirmatory factor analysis was conducted for system and information quality. Findings – Results indicate that accessibility and the navigation facility are important in determining a citizen’s perceived system quality. Information preciseness, timeliness, and sufficiency were found to be key measures of information quality in government e-services. Furthermore, an important finding from this research is that theories from information systems (IS), e-commerce, and marketing were found to be applicable in assessing government e-tax services within the broader area of government-to-citizen (G2C) service delivery systems. In addition, a quantitative study was conducted among citizens to determine system and information quality characteristics. Originality/value – The results presented in this paper can help the authorities to identify key quality criteria for e-tax services that are valued by citizens and consequently improve service levels. Taking the context into consideration, additional variables are incorporated from literature that spans several disciplines (IS, e-commerce, and marketing), and some re-specifications are made to identify system and information quality criteria in the context of an e-tax filing service, which was the main theoretical contribution of this study. Keywords Sweden, Government policy, Public services, Web sites, Information management, Taxes, e-governance, e-tax services, Information quality, System quality Paper type Research paper
Introduction With the rapid growth in usage of information technology and the web, governments are also increasingly using these technologies in delivering services at all levels, with a view to increase quality of service and achieving efficiency in their operations. E-government is stated to be the process by which information and services are electronically delivered to its customers, which includes citizens, businesses, and public administration alike (Irani et al., 2008). Latif and Masrek (2010) further specify that the key feature of electronic government delivery is usage of internet technologies through web-based interfaces, enabling people to have access to resources and services anywhere and anytime. Curtin (2006) identified the purpose of the e-government sites as the tool to provide an efficient management of government information to all citizens, giving better services to the community, and to empower people through access to information and participation in public decision making. While e-government initiatives are responsible for managing and organizing a variety of information, Freed (2009) further states that successful web sites should enable the public to find required information quickly and in an easily searchable manner. Thus, the search function, functionality, and navigation become important for the execution of those responsibilities. Studies show there have been considerable increases in presence and availability of e-government digital services (GDS) the past few years, with over 83 percent of basic GDSs being available on-line as of 2009 (Colclough and Tinholt, 2009). Based on similar studies conducted in 2006 and 2007 (Colclough, 2007; Wauters and Colclough, 2006), it may be surmised that interest in GDS on the part of public administrations has remained strong and has risen over the years. However, Bavec (2008) points out that many available GDSs are scantily accessed or used by citizens, and there are gaps between developer expectations and public approval. Thus, in real usage terms the availability of the GDS do not necessarily translate into the sites being effective in terms of the quality desired by citizens. Markaki et al. (2010) have stressed the need for comprehensive web site evaluations to understand the influence of factors affecting quality of e-government web sites. They cite three specific areas that can benefit as rationale for such evaluations: proper allocation of resources for relevant implementations; construction and operation of an e-government web site being directly related to government images; and the associated management and service levels. This is further echoed by Luo and Shi (2010), who state that how to strengthen the evaluation and establish a scientific index system to solve the problem in the development of e-government web sites has become a problem which cannot be ignored. Quality has emerged as one of the important issues in the industry (manufacturing, healthcare, education and government) during the last several years and in order to gain a competitive advantage it is important to focus on the issue “how to improve their quality”. From the customer’s viewpoint, quality can be achieved when customer’s expectations are met regarding the product or service being delivered (Chang et al., 2005). In the context of this study citizens are considered as customer for e-government service. “Quality is positioned to provide the key information regarding the quality of the system, information and service unit as they impact on stakeholders” (Wilkin and Hewett, 1999). The assessment of quality has been relatively less studied with respect to public services (Ray and Rao, 2004). A renewed focus on such could be important since,
Government e-tax websites
301
TG 6,3
302
to identify areas of improvement in public services provided, citizen perception of such services needs to be objectively measured (Irani et al., 2008). At the same time e-government efforts take on the objective of enhancing efficiency of government organizations, better accountability, as well as quality enhancement of services delivered along with creation of new services (Heeks, 2001; Irani et al., 2005; Seifert and Peterson, 2002). These efforts have been driven in part by the increased comfort, familiarity, and exposure the citizens now have with information and communication technologies and more specifically the internet in their myriad applications in their daily lives, particularly in the commercial sectors. This has also increased citizen’s expectations that the public sector needs to reflect similar levels of effectiveness and efficiency in its services delivery (Ebrahim and Irani, 2005). This takes on added importance when we consider that for any relationship between citizens and governments to be meaningful, citizen’s perspectives on services provided need to be measured in an objective manner in order to identify areas of improvement (Irani et al., 2008). This puts the onus back on assessing quality of e-government services and the modalities of their provisions. It is difficult to study every aspect of e-government services within the scope of a single research. Therefore, it is essential that limit the area where we can focus. Accordingly, this research focuses on government e-tax services in Sweden and the aim is to identify system and information quality criteria of the government tax web site. The online tax filing system is a type of government to customer (G2C) electronic service, which provides an opportunity of availing online tax services to taxpayers. Thus, this research is limited to evaluating G2C e-service as a part of the e-government domain. The study also limits its evaluation of system and information quality criteria of e-tax web sites. Based on the brief discussion, we arrive at the objective of this paper. The main research question for this paper is formulated as: RQ1. What are system and information quality characteristics of government service delivery web sites? The study is conducted in the context of Skatteverket.se, the e-tax web site in Sweden. Compared with other online services delivered by government, online tax filing is one of the most developed and widely-used services. In the public sector, with the move to online service being provided, tax authorities tended to be leading edge in adoption of IT applications. Careful consideration of citizen’s perception and expectation is important in making this service effective and the service delivery process should be more user friendly compared with the service delivered by traditional channels. Since perceived quality is one of the important determinants of web success, user perception and expectations need to be identified. The systems need to be clear and simple in order to be used by the ordinary tax payer (Connolly and Bannister, 2008). The advantage of online service delivery lies in the fact that system designers have added flexibility in designing sites so that the services and the content provided can match the needs of the clients. The idea is that such design approaches facilitate navigation, searching of information and retrieval of such, and thus meet the needs of the citizens (Wang et al., 2005). Irani et al. (2008) have raised a valid criticism that Wang et al.’s (2005) model considers e-government web site specific features and is bound to have its limitations when evaluating services based on other platforms. Recognizing the limitations of the application context in view, this study focuses on the
system and information quality aspects of the e-government web site. Sweden is one of the leading countries in the Western world that has proactively engaged in incorporating e-governance strategies extensively. E-tax services run by Skatteverket are a primary example of such strategies being put into action, and this study was conducted within their context. The major contribution of this research is that it identifies and empirically tests the applicability of system and information quality criteria taken from extant ARE research in evaluating government e-services web sites. In the next section we shall review and discuss previous literature concerning the study area. We then present the theoretical model and related hypotheses and follow this by the data and measurements. The analysis and results are subsequently discussed along with their managerial implications. We conclude the paper with recommendations for future research. Theoretical background Since the study focuses on evaluating e-government services through system and information quality constructs, it may be worthwhile identifying what services constitute e-government services. There is no universally applicable definition of e-government; however, there are a few characteristics that help identify e-government services (Irani et al., 2006; Al-Sebie and Irani, 2003). They are: . technology to deliver government services electronically; . transaction processes and the transformation of e-government services; . benefit portfolio for delivery of government services electronically to the public; . citizens as the central focus of service delivery; and . delivery of government services through a single online point of access – defining e-government. Thus, delivery of government services electronically and keeping citizens as the focus of service delivery are two recurring themes in literature. Similarly, measuring web site quality is an important concern for the IS and marketing research areas (Lociacono et al. 2000). Widespread use of internet technology creates a need to identify the factors that are related to web site success (Liu and Arnett, 2000; Aladwania and Palvia, 2002). Lociacono et al. (2000) established a scale called WEBQUAL for consumer evaluation of web sites with 12 dimensions: informational fit to task; interaction; trust; response time; design; intuitiveness; visual appeal; innovativeness; flow; integrated communication; business processes; and substitutability. Based on IS and marketing literature Liu and Arnett (2000) identified measurement of web site success in the context of electronic commerce. They have identified system quality, learning capabilities, playfulness, system quality, system use, and service quality as the success factors for web sites based on a survey of 1,000 companies. From the results they identified four factors that are important to web site success in e-commerce. These factors are information and service quality, system use, playfulness, and system design quality. Kim and Stoel (2004) have tried to determine the dimensions of web site quality and which dimensions are significant determinants of user satisfaction in apparel retailers. They conducted a survey on 273 female online apparel shoppers and the study was conducted from customer’s perspective. In their study they have used the WEBQUAL
Government e-tax websites
303
TG 6,3
304
instrument that was developed by Lociacono et al. (2000). The results found six dimensions of web site quality: web appearance, entertainment, informational fit-to-task, transaction capability, response time, and trust. However, not all these factors are determinants of user satisfaction. Only informational fit-to-task, transaction capability, and response time have a significant impact on user satisfaction. Aladwania and Palvia (2002) have developed an instrument from users’ perspectives that covered the characteristics of web site quality. They developed 25 items with four dimensions of web site quality. These dimensions are specific content, content quality, appearance and technical adequacy. Based on the IS success model Cao et al. (2005) developed the factors that determine e-commerce web site quality which make a web site effective. They conducted a survey among students who are familiar with internet shopping. According to them, four factors are important to determine web site quality from customer’s perspective. These factors are functionality, content, service and attractiveness. Findings from their study indicate that customers are more sensitive with finding more accurate information, security and fast search facilities. So web sites should be designed accordingly that will create an opportunity for users to find accurate information, easier search facilities, less loading time and ensure security. Smith (2001) has developed several criteria to evaluate a government web site and they have tested these criteria on a government web site to see their applicability. They have divided these criteria into two groups: information content criteria and ease to use criteria. Zviran et al. (2005) conducted a study to find out important factors that drive user satisfaction from the web site. They have empirically tested user satisfaction of different web sites and determined two characteristics: usability and user-based designed. Based on an IS and e-commerce success model Stockdale and Borovicka (2006) developed web site evaluation instrument and tested it through a pilot study on tourism web site. Based on the previous studies they developed system quality, information quality and service quality criteria to evaluate a tourism web site. Following the above discussion, several quality criteria have been identified for the evaluation of web sites from a user’s perspective. This study tries to identify several quality criteria for evaluation of government web sites from citizens’ perspectives and empirically tests them in the context of government e-tax web sites. Quality criteria were divided further into system and information quality criteria following the previous study conducted in this field. Characteristics of system and information quality Wangpipatwong et al. (2005) stressed that factors related to system quality and information quality significantly influence the adoption of e-government web sites. In doing so, they identified functionality, reliability, usability, and efficiency as system quality characteristics, and observed that efficiency is the most important factor in a government web site. Accuracy, relevancy, completeness, timeliness, and precision, which are information quality criteria in government web sites, are identified in their study. Additionally, they found timeliness and precision to be less important compared with other information quality criteria (Wangpipatwong et al., 2005). Barnes and Vidgen (2006) evaluated the quality of the government web site from a user’s perspective. They adapted the eQual instrument – usability, information quality and service interaction – to measure government web site quality (Table I). According to the IS success model, system quality is concerned with the measurement of the actual system that produces the output (DeLone and McLean, 1992, pp. 64).
Adaptability, availability, reliability, response time, usability Response/turnaround time, convenience of access, understanding of systems Confidence in the systems, integration of the system Understanding of system, time required for new system development Easy to use User friendly Easy to learn Easy to get done what I want it to do Easy for me to become skillful Cumbersome to use Requires a lot of mental effort to use Use is often frustrating Access: responsive, quick loads Usability: simple layout, easy to use, well-organized, clear design Entertainment: visually attractive, fun, interesting Hyperlinks: adequacy of links, clear description of links Navigation: easy to go back and forth, a few clicks Interactivity: create list of items, change list of items, create customized product, select different features Ease of use User friendly Ease to use Multimedia capability: web site uses audio elements, video elements, animation/graphics, and multimedia features properly
DeLone and McLean (2003)
Cao et al. (2005)
Rai et al. (2002)
McKinney et al. (2002)
Seddon and Kiew (1996)
Baroudi and Orlikowski (1988)
Bailey and Pearson (1983)
Description of the measures
Author
B2C e-commerce web site quality
(continued)
Success factors in integrated SIS at university
Measurement of web customer satisfaction
Success factors in the university’s departmental accounting system
Measure of user information satisfaction
Analyzing computer user satisfaction
Success of e-commerce context
Area of the study
Government e-tax websites
305
Table I. Measures of system quality
Table I.
Rolda´n and Leal (2003)
Search facility: clear indication of site’s content, wellorganized hyperlinks, structure of the site is logical, easy navigation, explanation of how to use site, easy to find information Responsiveness: proper response time, fast searching, reasonable time for searching, reasonable loading time, responsive to user inquires Response/turnaround time, convenience of access, features of computer language used, realization of user requirements, correction of errors, security of data and models, documentation of systems and procedures, flexibility of the systems, integration of the systems Functionality: web site always works correctly, web site provides necessary information and forms to be downloaded, web site provides necessary transactions to be completed online, and web site provides helpful instructions Reliability: web site is available at all times Web site is secure Usability Web site is easy to use Web site is attractive Efficiency Web site can save citizens’ time Web site can save citizens’ expense Faster access to information Easier and more comfortable access to information Availability of improved access to the organizational database Have the benefit of new or additional information Enjoy an improved presentation of data
Description of the measures
Success factors in Spanish EIS
Factors influencing the adoption of e-government web sites
IS success factors
Area of the study
306
Wangpipatwong et al. (2005)
Li (1997)
Author
TG 6,3
According to McKinney et al. (2002), web site information and system quality are the key constructs of web customer satisfaction. They defined system quality relative to site success as the customers’ perception of a web site’s performance in information retrieval and delivery. Web customers’ perception of the quality of information presented on a web site is defined as web information quality. Information quality is concerned with the measure of the system’s output (DeLone and McLean, 2004). Based on the IS literature, Chen (2010) identifies the quality antecedents of tax payer satisfaction for online tax payment systems. They also identify information and system quality as important quality criteria for government online tax web sites. System quality in the internet environment measures the desired characteristics of an e-commerce system (DeLone and McLean, 2003, 2004). It is important to evaluate the web site functionality that focuses on online service functions provided by the web site. Availability of the web site at all times and the speed of access of the web site is essential. In addition, it is also important to judge navigation characteristics of the web site and evaluate the presence of links for needed information (McKinney et al., 2002; Cao et al., 2002). Information quality is concerned with issues such as the relevance, timeliness, and accuracy of the information generated by an IS (DeLone and McLean, 2003, 2004). In the e-commerce context, information delivery is an important role of web sites, and quality is considered a critical issue (McKinney et al., 2002). Several quality evaluation aspects are essential, including the correctness of the output information, the availability of the output information at a time suitable for its use, and the comprehensiveness of the output information content (Bailey and Pearson, 1983). It is also important to consider issues such as relatedness, clearness, and goodness of the information (McKinney et al., 2002). Rai et al. (2002) studied users of a computerized student information system (SIS) and found system quality and information quality to be determinants of satisfaction. They have taken system quality as ease of use and defined it as the “degree to which the system is user friendly,” and they measured information quality content, accuracy, and format, the three attributes generated by SIS. Roca et al. (2006) completed their study on understanding e-learning continuance intention and found that information quality and system quality are significant determinants of satisfaction. Li (1997) conducted a study on perceived importance of IS success factors. Based on the previous studies, several additional factors of IS success have been identified. The results of the study identified five important factors: accuracy of output, reliability of output, relationship between users and the CBIS staff, user’s confidence in the systems, and timeliness of output. Iivari (2005) empirically tested DeLone and McLean’s IS success model in the organization’s new IS. The results of the study suggested that perceived system quality and perceived information quality are significant predictors of user satisfaction in the success of individual IS applications. In this study, 24 items that covered flexibility, system integration, response time, recoverability, convenience, and common language measured system quality. For measuring information quality, 24 items were selected that addressed completeness, precision, accuracy, consistency, currency, and format characteristics. Rolda´n and Leal (2003) adapted DeLone and McLean’s (1992) IS success model and validated it in a Spanish executive information system (EIS). In their research, they conducted a survey, and the respondents for the survey were EIS users. The results of their study indicate that system quality and information quality have a significant positive influence on EIS user satisfaction (Table II).
Government e-tax websites
307
Completeness, ease of understanding, personalization, relevance, and security Accuracy, timeliness, precision, reliability, currency, completeness, format of output, volume of output, and relevancy Reliability of output, relevancy of output, accuracy of output, precision of output, completeness of output Output is presented in a useful format Satisfied with the accuracy of the system Clear information Accurate system Sufficient information Up-to-date information Information needed in time Provide reports that needed Precise information Information content addresses needs Relevance: applicable, related, pertinent Understandability: clear in meaning, easy to understand, easy to read Reliability: trustworthy, accurate, credible Adequacy: sufficient, complete, necessary topics Scope: wide range, wide variety of topics, different subjects Usefulness: informative, valuable Information accuracy: useful information, accurate information, site is informative, updated information, high quality information, timely information Information relevancy: relevant according to user Availability of information according to user needs and relevant information
DeLone and McLean (2003)
Cao et al. (2005)
McKinney et al. (2002)
Seddon and Kiew (1996)
Baroudi and Orlikowski (1988)
B2C e-commerce web site quality
(continued)
Measurement of web customer satisfaction
Success factors in the university’s recently implemented departmental accounting system
Measure of user information satisfaction
Analyzing computer user satisfaction
Success of e-commerce context
Area of the study
308
Bailey and Pearson (1983)
Description of the measures
Table II. Measures of information quality
Author
TG 6,3
Accuracy of output, timeliness of output, precision of output, reliability of output, currency of output, completeness of output, and format of output Accuracy, timeliness, relevancy, precision, and completeness System provides relevant information System does not provide easy-to-understand information Output information is not clear Information presentation in an appropriate format Information content is very good Information is up-to-date Completeness of output information Information delivered is not sufficient for purposes Reliability of output information is high Provides information in time Precise information according to user need Provides output that is exactly what the user needs Sufficient information to enable users to do tasks Errors in the program that users must work around Satisfied with the accuracy Output options (print types, page sizes allowed for, etc.) sufficient for user applications Information provided was helpful regarding user questions or problems Current and timely information Relevant, useful and significant information Concise and summarized information Accurate information Orderly and clear information Reasonable and logical information
Li (1997)
Rolda´n and Leal (2003)
Rai et al. (2002)
Roca et al. (2006)
Wangpipatwong et al. (2005)
Description of the measures
Author
Success factors in the Spanish EIS
Success factors in integrated SIS at university
Factors influencing the adoption of e-government web sites Acceptance of e-learning
IS success factors
Area of the study
Government e-tax websites
309
Table II.
TG 6,3
310
Methodology The purpose of the research is to identify the system and information quality characteristics within the context of a government e-tax web site. For this purpose previous literature was reviewed to identify the system and information quality characteristics. As such, this research is of an exploratory nature, and a quantitative approach was chosen to test the aforementioned quality criteria empirically. A survey was chosen as the preferred method, since it allows the collection of descriptive cross-sectional data (Hair et al., 2007). Measures development From the previous literature seven items were selected to measure system quality that covered functionality, navigation, accessibility the main characteristics of system quality. Items were selected from the previous studies done by Liu and Arnett (2000), McKinney et al. (2002), Smith (2001), Aladwania and Palvia (2002), Wangpipatwong et al. (2005), Stockdale and Borovicka (2006) and Cao et al. (2005). Seven items were selected from previous literature to measure information quality, which were measures of the characteristics of information provided by government web sites. Items were selected per the previous study done by Aladwania and Palvia (2002), Liu and Arnett (2000), Bailey and Pearson (1983), Li (1997), Smith (2001), Wangpipatwong et al. (2005), McKinney et al. (2002), Stockdale and Borovicka (2006), Seddon and Kiew (1996), Rai et al. (2002), Cao et al. (2005) and Roca et al. (2006). Sampling justification and data collection For this research, the target population is taxpayers, who by definition must be employees. Thus, organizations were incorporated as a mode for simplifying the process of contacting employees;, i.e. potential respondents as taxpayers. Here, we made the choice of selecting municipalities all over Sweden to be the primary type of organization being targeted. By its nature, a municipality represents the people, and the employee profiles reflect its citizens, since people from different educational and other backgrounds are all resident in the employee profiles, just as in the surrounding society. A total of 290 municipalities are organized into 21 counties and we have chosen every third number of municipalities from the list. As a secondary source, employees from six universities from different parts of Sweden (by region) were selected to send out the surveys. Two universities were selected from north and central Sweden, two were selected near Stockholm, and two were selected from southern Sweden. Within each county, listings were made of the municipalities within. A systematic random sampling was implemented with every third element within the list being targeted as respondent organization. An online survey was administered to the selected sample. The online survey was hosted on a central server, and an e-mail was sent to possible respondents explaining the nature of the survey. Since the e-mails were sent using lists in a particular organization, it was possible to reach a large number of people in a short amount of time. The survey was left open for two months and during this time, 425 valid responses were received. The response rate was 3.63 percent of the 425 responses.
Addressing possible common method bias in the current research In the current research work, common method bias can arise due to the use of a single respondent to address all the variables in the model. As a result, using it for measuring both predictive and criterion variables may result in a method effect produced by the common source. Another possible source of common method bias in this research is the “consistency motif,” or respondents’ answering while looking for similarities and trying to maintain consistency. Other sources of common method bias, such as the items of social desirability or characteristic effects, are deemed not applicable in the context of this research. According to Podsakoff et al. (2003), several ways have been suggested to counter possible common method bias in behavioral research. We can achieve this by using several procedural remedies: . Temporal, proximal, psychological, or methodological separation of measurement. Another option is to introduce a temporal, proximal, or psychological separation between the measurements. This is aimed at reducing any contextual cues that may have been present in one instance. Also, temporal separation can help remove answers from short-term memory. A locational separation is aimed at the elimination of locational retrieval cues. Due to resource and time constraints within the research, it is not possible to collect data from different periods. It is also difficult to find the same respondent group to answer the question a second time. Thus, reducing common method bias through temporal separation is not feasible. However, since the survey is conducted online, it already has eliminated locational cues from the respondents’ environments; therefore, a locational separation is in effect here. Also, since the questionnaire is answered online rather than in a predetermined room or laboratory setting, any contextual cues are also either absent or present in any respondent situation, since it is only the computer environment. Thus, contextual cues are also removed by using an online questionnaire that reduces common method bias. . Protecting respondent anonymity and reducing evaluation apprehension. Finally, protecting respondent anonymity and reducing evaluation apprehension are cited as procedures that can be used to reduce common method bias; the survey is designed accordingly. Statements are highlighted at the beginning of the survey noting that responses are completely anonymous, and that the data given cannot be identified with any respondent. Furthermore, providing demographic data is optional, and the fact is highlighted within the questionnaire. These steps reduce evaluation apprehension and thus reduce the common method bias. Data analysis In order to establish the internal consistency of the measurement instruments, reliability analysis was conducted. It was established by calculating coefficient alpha, also known as Cronbach’s a, to measure the internal consistency of the scale. All of the items are found to be reliable, since the values are above the recommended level of 0.7. Cronbach’s a of the scales system quality (0.874) and information quality (0.863) showed a very good internal consistency of the scales (the Appendix). Exploratory factor analysis was conducted to get an idea for the factor structure, further examination of the factor structure and measurement purification and validation confirmatory factor analysis (CFA) was conducted for each construct to refine the
Government e-tax websites
311
TG 6,3
312
scales and that gives an idea about different assumed variables truly measure the factors identified in the research model. CFA was conducted on system and information quality construct to check whether all the items load satisfactorily. An exploratory factor analysis was conducted for system quality construct with seven items and information quality construct with seven constructs. From the result a one-factor solution was derived for both system and information quality constructs. Significant factor loading (more than 0.50) was found for all items for both constructs. CFA for system quality Seven items were selected from the previous study and the results from exploratory factor analysis were used to measure system quality. A CFA was conducted with these seven items to see whether items load satisfactorily to measure this construct. At the end of the analysis, four items were selected to measure system quality. Items were omitted in several cases based on the variance explained, the path loadings, and standardized residual value, and the factor structure was gradually refined based on the findings from the model runs (Figure 1). We have looked at the overall modification indices and checked whether error term related with item identified from standardized residual covariance. Removing the items will decrease the x 2 value and improve the model fit. We have looked at the overall modification indices and checked whether taking more of the same errors can result in an even greater decrease in the x 2 value. In such a case, we found a significant change for the same error term and we calculated the total decrease that can occur if we were to remove the particular item associated with the error term. Taking out two items will decrease the x 2 value. So, based on standardized residual covariance and modification indices, it was decided to take the items out and run the simulation to get a good fit. Determining the overall fit. Looking at overall model fit, we found that the goodness-of-fit index (GFI) value is 0.949, which is greater than acceptable level 0.90, value adjusted goodness-of-fit index (AGFI) is 0.953, which is also greater than acceptable value 0.80. Two reliable indicator Tucker-Lewis index (TLI) and comparative fit index (CFI) values are, respectively, 0.979 and 0.993 and more than acceptable level. These values should preferably be greater than 0.90. RMSEA value is 0.089, which indicates acceptable fit (Tables III and IV). 0.51 e5
sysq3 0.72 0.38
e3
sysq5
0.61 0.88
Figure 1. CFA model for system quality
e2
0.78 sysq6
e1
sysq7
0.88
0.94
sysq
From the results, we found that all the standardized loadings are over 0.5 and the critical ratio is more than 1.96. Looking at standardized residual covariance, all the values are less than 2.58. CFA for information quality Seven items were selected from previous work and the results from exploratory factor analysis were used to measure information quality. CFA stared with seven items and at the end of the analysis four items load satisfactorily to measure this construct. Based on standardized residual covariance and modification indices it was decided which items are responsible for model misspecification and should thus be removed from the model to get a better fit. The results are shown in Figure 2. From the results we can see all the standardized loadings are over 0.50 and critical ratio is more than 1.96, all the standardized residual values are less than 2.58 and model fit is good. CMIN/DF (1.130) and RMSEA (0.018) values are acceptable.
Model
RMSEA
Default model Saturated model Independence model
CMIN/DF
0.089
4.228
0.610
152.504
GFI
AGFI
CFI
TLI
IFI
0.990 1.000 0.455
0.949
0.993 1.000 0.000
0.979
0.993 1.000 0.000
0.091
0.000
Note: p ¼ 0.015, x 2 ¼ 8.456
Structural relation Sysq7 ˆ sysq Sysq6 ˆ sysq Sysq5 ˆ sysq Sysq3 ˆ sysq
Regression weight
SE
Critical ratio
Standardized regression weights
Squared multiple correlation
1.000 0.959 0.559 0.724
0.039 0.040 0.041
24.772 14.067 17.701
0.938 0.884 0.614 0.718
0.879 0.782 0.377 0.515
Government e-tax websites
313
Table III. Fit index for system quality
Table IV. Estimated values for system quality items
0.66 e6
infq2 0.81 0.73
e4
infq4 0.32
e3
infq5 0.58
e2
infq6
0.85 0.57
infq
0.76
Figure 2. CFA model for information quality
TG 6,3
314
The GFI value is 0.997 which is greater than the acceptable level, 0.90; value AGFI is 0.986, which is also greater than the acceptable value, 0.80. Two reliable indicators, the TLI and CFI values, are, respectively, 0.999 and 0.100 and more than the acceptable level (Tables V and VI). Discussion on findings System quality is defined for this study as a desired functionality and performance characteristic of a government web site and seven items (see Table I for item sources taken from previous literature) were selected to measure system quality in this study and were selected based on previous studies. Items that were selected at the beginning are: this web site provides necessary information and forms to be downloaded (sysq1); this web site provides helpful instruction for performing my task (sysq2); this web site provides fast information access (sysq3); this web site quickly loads all the text and graphics (sysq4); it is easy to go back and forth between pages (sysq5); it only takes a few clicks to locate information (sysq6); and it is easy to navigate within this site (sysq7). At the end of instrument refinement process, four items (sysq3, sysq5, sysq6, sysq7) were finally taken as measures of system quality in government e-tax filing services. Thus, four items are found to determine citizen’s perceived system quality. Results indicate that easy navigation facility is highly related to determining system quality. Items that are related to navigation facility of the web site and accessibility are important to determine citizens’ perceived system quality. That implies that when citizens are filing their tax return and other tax-related activities they prefer that the web site should have proper and easy navigation facilities and easy information access. These are the main functionality and characteristics of the web sites identified in this study. This finding corresponds to past assertions by McKinney et al. (2002), Lu et al. (2002), Barnes and Vidgen (2006) and Chen (2010), that navigation facilities, fast access and evaluating the links for needed information are important characteristics of web site evaluation. This further confirms the findings by Freed (2009) that success of web sites depends on their ability to serve the public to find information quickly and easily searchable. Structural relation
Table V. Estimated values for information quality items
Regression weight
Infq6 ˆ InfQ Infq5 ˆ InfQ Infq4 ˆ InfQ Infq2 ˆ InfQ
1.031 0.574 1.045 1.000
Model
Table VI. CFA fit indice for information quality
SE
Critical ratio
Standardized regression weights
Squared multiple correlation
0.065 0.051 0.061
15.836 11.254 17.216
0.765 0.566 0.852 0.811
0.585 0.321 0.726 0.658
RMSEA
Default model Saturated model Independence model 2
CMIN/DF
0.018
1.130
0.519
110.515
Note: p ¼ 0.323, x ¼ 2.259
GFI
AGFI
CFI
TLI
IFI
0.997 1.000 0.510
0.986
1.000 1.000 0.000
0.999
1.000 1.000 0.000
0.184
0.000
Information quality is defined in this study as the characteristics of information provided by a government web site. Based on previous literature, seven items were selected to measure information quality (Table II). The items were identified at the beginning and are: information on this web site is free from errors (infq1); this web site provides information precisely according to my need (infq2); information on this web site is up-to-date (infq3); this web site provides information I need at the right time (infq4); information presented on this web site is related to the subject matter (infq5); information on this web site is sufficient for the task at hand (infq6); information on this web site is sufficient for the task at hand (infq7); and information contains necessary topics to complete related task (infq8). After scales refinement, four (infq2, infq4, infq5, infq6) were taken to measure information quality in the context of government e-tax services. These four items covered information precision, information timeliness and sufficiency or completeness of the information characteristics. Our findings correspond to past research’s identification of information quality characteristics (Bailey and Pearson,1983; Baroudi and Orlikowski, 1988; Seddon and Kiew, 1996; Cao et al., 2005; Li, 1997). The results indicate that the provision of timely and precise information according to citizens’ need is found to be important. This also stands to reason since timely and precise information is requisite for timely completion and filing of tax returns. So the next important information characteristic for determining citizen’s perceived information quality is the sufficiency of information availability on the web site that helps citizen to complete their tax-related task. According to Wangpipatwong et al. (2005), information accuracy, timeliness, preciseness and completeness of the information were identified as information quality characteristics for government web sites; and accuracy, timeliness, and completeness of information presented on the web sites are the important information quality criteria of online tax web sites identified by Chen (2010). Theoretical and practical implications of the study This study identifies an important determinant of system quality and information quality in the government tax web site. Based on extensive literature review from the IS, e-commerce, and marketing research areas the system and information quality characteristics were selected and they were tested in government e-service delivery context particularly on a Swedish e-tax web site. The results indicate that IS, e-commerce, and marketing theory are applicable in the G2C area; more specifically, government e-tax service delivery. Csetenyi (2000) proposed application of ecommerce and e-business technologies as a means to increase efficiency in the provision of services to citizens and businesses. Navigation facility and accessibility of the online tax web sites are important to determine citizen’s perceived system quality, so the study identified the important system quality criteria for online tax web site from the citizen’s view point. According to Chen (2010), system quality depends on three factors: accessibility, interactivity and ease of use. When citizens use the online system to filing their tax return they need to interact with the authority, download necessary documents, and also need to navigate different pages on the web sites. Good accessibility and interactivity positively affect the online system quality of web sites. Navigation facilities was identified as an important quality criterion of government web sites (Barnes and Vidgen, 2006).
Government e-tax websites
315
TG 6,3
316
From the results it was found that information precision, information timeliness and sufficiency or completeness of the important information characteristics were used to evaluate government e-tax web sites. The implication of the results indicates that information presented should be more precise and should cater to the citizen’s needs, they should also provide information at the right time and information should be sufficient for the citizen to complete the tax-related activities. According to Chen (2010), information quality is the important characteristic of online tax web sites. Chen (2010) measures information quality of tax web sites, focusing on accuracy, timeliness, and completeness of information presented in the web sites. To create new users, the tax authority needs to simplify the interface design to make the system easier to use. They should try to maintain easy design and functionality of the web site where citizens can get fast access to the information, get anywhere on the web site very easily and find the necessary information. They need to provide a well-organized web site that will help citizens to complete their tax related task with a minimal effort. Along with the theoretical contributions, there are some practical implications of the research findings. It is important for the practitioner such as tax authorities and other government organizations that are involved with the delivery of e-services, to be aware of the factors that contribute towards the future maintenance of the quality of the e-government services. The results can help the tax authority to identify the key quality criteria for the e-tax service web site that are valued by citizens. As a result, that authority can improve its service delivery process. A citizen survey was conducted to determine their expectations regarding the quality criteria of the information presented in the government web sites. According to Connolly and Bannister (2008), among the different online services provided by governments, online tax filing is one of the most developed and widely used. Tax authorities have also tended to be the leaders in IT application implementation in the public sector. It is imperative that citizens’ perceptions and expectations are carefully considered to increase the efficiency and effectiveness of this service. Earlier studies have mentioned that the needs of citizens and businesses have so far been neglected (Centeno et al., 2005). These findings have led this research to stress the need to focus on the factors that work behind the scenes in the satisfactory provisioning of this service to citizens. Earlier studies also mentioned that huge investments had been made for e-government service development efforts but a comparatively low level of user acceptance was found for such services. More empirical research was recommended as being need from a user’s perspective on improving the effectiveness and quality of e-government services (Hung et al., 2006). Tax authorities can use the results to retain the current users and, along with that, they can use them to create new users. Both of these are important in making the service successful. The instrument that was developed for this study will provide a good insight and can act as a reference point in defining reliable quality standards for the web site, which can influence people in using the service more rather than the offline service alternative for filing tax returns. Tax authorities can take an initiative to establish evaluation methods based on such standards to identify if there is anything lacking in the service delivery process, and take corrective measures as needed. Conclusion and limitations of the study This research explored an important area of e-government particularly in the online tax system. It focused on the Swedish online tax system and evaluated Swedish tax
information-related web sites to find out the important system and information quality criteria in place for such web sites. A citizen survey was conducted to identify the important information and system quality criteria from a user’s perspective. Results indicate that navigation facility and accessibility are important in determining citizens’ perceived system quality. Information preciseness, timeliness, and sufficiency were found to be key measures in information quality in government e-services. Every research has some limitations, and it is important to present the limitations of the present research to place the findings in perspective. The first limitation of the current study is that it was not possible to collect individual online taxpayers’ addresses from the tax authorities. Because of personal confidentiality concerns, the tax authority was unwilling to provide the addresses of online taxpayers. Employees from the municipalities and the employees from a number of universities within Sweden who were experienced with the internet tax-filing system were selected as a sample for the data collection. The external validity of the research results thus may be limited to Sweden. Another limitation lies in the fact that the items used to measure system, information and e-service qualities were selected from different studies. These constructs were measured directly with items, and dimensions were not included in this study. In the empirical analysis, trace of multidimensionality was not found within the items treated in the model. However, there can be additional items used to measure the variables, and it is proposed that further research may be conducted including additional items to judge the dimensionality as well as possible extension of the model even further. Within the present study, system and information quality criteria were proposed in the context of e-tax services, and tested in Sweden. Further testing is proposed to test the applicability of the quality criteria in different geographical locations and within their respective e-tax or similar comparable services. In the present study data were collected only from citizens who are experienced with e-tax services. The tax authority itself was not included in the data collection process. The problems or issues encountered by the authorities in implementing the system and interacting with the citizens can provide valuable input in enhancing the quality of the system. Thus, a secondary study is proposed that deals with evaluation of the quality criteria from the service provider perspective, i.e. the tax authority. References Aladwania, A.M. and Palvia, P.C. (2002), “Developing and validating an instrument for measuring user-perceived web quality”, Information & Management, Vol. 39, pp. 467-76. Al-Sebie, M. and Irani, Z. (2003), “E-government: defining boundaries and lifecycle maturity”, Proceedings of 3rd European Conference on E-government, Ireland, 3-4 July, Trinity College Dublin, Dublin, pp. 19-29. Bailey, J.E. and Pearson, S.W. (1983), “Development of a tool for measuring and analyzing computer user satisfaction”, Management Science, Vol. 29 No. 5, pp. 530-45. Barnes, S.J. and Vidgen, R.T. (2006), “Data triangulation and web quality metrics: a case study in e-government”, Information & Management, Vol. 43, pp. 767-77. Baroudi, J.J. and Orlikowski, W.J. (1988), “A short-form measure of user information satisfaction: a psychometric evaluation and notes on use”, Journal of Management Information Systems, Vol. 4 No. 4, pp. 44-59. Bavec, C. (2008), “On stimulus for citizens’ use of e-government services”, International Multiconference on Computer Science and Information Technology, Wisia, pp. 391-5.
Government e-tax websites
317
TG 6,3
318
Cao, M., Qingyu, Z. and Seydel, J. (2002), “B2C e-commerce web site quality: an empirical examination”, Industrial Management & Data Systems, Vol. 105 No. 5, pp. 645-61. Cao, M., Zhang, Q. and Seydel, J. (2005), “B2C e-commerce web site quality: an empirical examination”, Industrial Management & Data Systems, Vol. 105 No. 5, pp. 645-61. Centeno, C., Bavel, R.V. and Burgelman, J.C. (2005), “A prospective view of e-government in the European union”, The Electronic Journal of e-Government, Vol. 3 No. 2, pp. 59-66, available at: www.ejeg.com Chang, I.-C., Li, Y.-C., Hung, W.-F. and Hwang, H.-G. (2005), “An empirical study on the impact of quality antecedents on tax payers acceptance of internet tax-filing systems”, Government Information Quarterly, Vol. 22, pp. 389-410. Chen, C. (2010), “Impact of quality antecedents on taxpayer satisfaction with online tax-filing systems – an empirical study”, Information & Management, Vol. 47, pp. 308-15. Colclough, G. (2007), “The user challenge benchmarking the supply of online public services – 7th measurement”, Technical Report, prepared by Capgemini for European Commission Directorate General for Information Society and Media, Bruxelles. Colclough, G. and Tinholt, D. (2009), “Smarter, faster, better e-government – 8th benchmarking measurement”, Technical Report, prepared by Capgemini, RAND Europe, IDC, SOGETI and DTi, European Commission Directorate, General for Information Society and Media, Bruxelles. Connolly, R. and Bannister, F. (2008), “eTax filing and service quality: the case of the revenue online service”, Proceedings of World Academy of Science, Engineering and Technology, April, Vol. 48. Csetenyi, A. (2000), “Electronic government: perspective from e-commerce”, Proceedings of the 11th International Workshop on Database and Expert Systems Applications, IEEE Conference Publications, Washington, DC, pp. 294-8. Curtin, G. (2006), “Issues and challenges global e-government/e-participation models, measurement and methodology: a framework for moving forward”, Prepared for the United Nations Department of Administration and Development Management, Workshop on E-participation and E-government: Understanding the Present and Creating the Future, Budapest, Hungary, 27-28 July. DeLone, W.H. and McLean, E.R. (1992), “Information system success: the quest for dependent variable”, Information System Research, Vol. 3 No. 1. DeLone, W.H. and McLean, E.R. (2003), “The DeLone and McLean model of information system success: a ten-year update”, Journal of Management Information System, Vol. 19 No. 4, pp. 9-30. DeLone, W.H. and McLean, E.R. (2004), “Measuring e-commerce success: applying the DeLone and McLean information systems success model”, International Journal of Electronic Commerce, Vol. 9 No. 1, pp. 31-47. Ebrahim, Z. and Irani, Z. (2005), “E-government adoption: architecture and barriers”, Business Process Management Journal, Vol. 11 No. 5, pp. 589-611. Freed, L. (2009), “The American customer satisfaction index: e-government satisfaction index”, Q4. Hair, J.F., Babin, B., Money, A.H. and Samouel, P. (2007), Essentials of Business Research Methods, Wiley, New York, NY. Heeks, R. (2001), Understanding E-governance for Development, Institute for Development Policy and Management, Manchester.
Hung, S., Chang, C. and Yu, T. (2006), “Determinants of user acceptance of the e-government services: the case of online tax filing and payment system”, Government Information Quarterly, Vol. 23, pp. 97-122.
Government e-tax websites
Iivari, J. (2005), “An empirical test of the DeLone-McLean model of information system success”, The Data Base for Advances in Information Systems, Vol. 36 No. 2. Irani, Z., Al-Sebie, M. and Elliman, T. (2006), “Transaction stage of e-government systems: identification of its location and importance”, Proceedings of the 39th Hawaii International Conference on System Sciences. Irani, Z., Sharif, A.M. and Love, P.E.D. (2005), “Linking knowledge transformation to information systems evaluation”, European Journal of Information Systems, Vol. 14 No. 4, pp. 1-16. Irani, Z., Osman, I.H., Balci, A., Ozkan, S. and Medeni, T.D. (2008), “Research note toward a reference process model for citizen-oriented evaluation of e-government services”, Transforming Government: People, Process and Policy, Vol. 2 No. 4, pp. 297-310. Kim, S. and Stoel, L. (2004), “Apparel retailers: website quality dimensions and satisfaction”, Journal of Retailing and Consumer Services, Vol. 11, pp. 109-17. Latif, M.H.A. and Masrek, M.N. (2010), “Accessibility evaluation on Malaysian e-government websites”, Journal of E-government Studies and Best Practices, Vol. 2040, Article ID 935272, 11 pages. Li, E.Y. (1997), “Perceived importance of information system success factors: a meta analysis of group differences”, Information & Management, Vol. 32 No. 1, pp. 15-28. Liu, C. and Arnett, K.P. (2000), “Exploring the factors associated with web site success in the context of electronic commerce”, Information & Management, Vol. 38, pp. 23-33. Lociacono, E., Watson, R.T. and Goodhue, D. (2000), “WebQual Tm: a web site quality instrument”, working paper, Worcester Polytechnic Institute, Worcester, MA. Lu, Z., Lu, J. and Zhang, C. (2002), “Website development and evaluation in the Chinese tourism industry networks and communication studies”, NETCOM, Vol. 16 Nos 3/4, pp. 191-208. Luo, D. and Shi, Y. (2010), “Evaluation on e-government websites based on rough set and genetic neural network algorithm”, Proceedings of the Third International Symposium on Computer Science and Computational Technology (ISCSCT ’10) Jiaozuo, People’s Republic of China, 14-15 August, pp. 124-8. McKinney, V., Yoon, K. and Zahedi, F.M. (2002), “The measurement of web-customer satisfaction: an expectation and disconfirmation approach”, Information System Research, Vol. 13 No. 3, pp. 296-315. Markaki, O., Charilas, D. and Askounis, D. (2010), “Evaluating the quality attributes of e-government websites”, E-government Website Development: Future Trends and Strategic Models, pp. 65-86. Podsakoff, P.M., MacKenzie, S.B., Lee, J.Y. and Podsakoff, N.P. (2003), “Common method biases in behavioral research: a critical review of the literature and recommended remedies”, Journal of Applied Psychology, Vol. 88 No. 5, pp. 879-903. Rai, A., Lang, S.S. and Welker, R.B. (2002), “Assessing the validity of IS success models: an empirical test and theoretical analysis”, Information Systems Research, Vol. 13 No. 1, pp. 50-69. Ray, S. and Rao, V. (2004), “Evaluating government service: a customers’ perspective of e-government”, Proceedings of the 4th European Conference on E-government, Dublin, Ireland, 17-18 June.
319
TG 6,3
320
Roca, C.J., Chiu, C.-M. and Martinez, F.J. (2006), “Understanding e-learning continuance intention: an extension of the technology acceptance model”, International Journal of Human-Computer Studies, Vol. 64, pp. 683-96. Rolda´n, J.L. and Leal, A. (2003), “A validation test of an adaptation of the DeLone and McLean’s model in the Spanish EIS field”, in Cano, J.J. (Ed.), Critical Reflections on Information Systems: A Systemic Approach, Idea Group Publishing, Hershey, PA. Seddon, P.B. and Kiew, M.Y. (1996), “A partial test and development of Delone and McLean’s model of is success”, Australian Journal of Information System (AJIS), Vol. 4 No. 2. Seifert, J. and Peterson, R.E. (2002), “The promise of all things E? Expectations and challenges of emergent electronic government”, Perspectives on Global Development and Technology, Vol. 1 No. 2, pp. 193-212. Smith, A.G. (2001), “Applying evaluation criteria to New Zealand government websites”, International Journal of Information Management, Vol. 21, pp. 87-114. Stockdale, R. and Borovicka, M. (2006), “Using quality dimensions in the evaluation of web sites”, Proceedings of the International Conference in Information and Communication Technologies in Tourism, Lausanne, Switzerland. Wangpipatwong, S., Chutimaskul, W. and Papasratorn, B. (2005), “Factors influencing the adoption of Thai egovernment websites: information quality and system quality approach”, Proceedings of the Fourth International Conference on eBusiness, Bangkok, 19-20 November. Wang, L., Bretschneider, S. and Gant, J. (2005), “Evaluating web-based e government services with a citizen-centric approach”, Proceedings of the 38th Hawaii International Conference on System Sciences – 2005. Wauters, P. and Colclough, G. (2006), “On-line availability of public services: how is Europe progressing?”, Web Based Survey on Electronic Public Services Report of the 6th Measurement June 2006, Technical Report, prepared by Capgemini for European Commission Directorate General for Information Society and Media, Bruxelles. Wilkin, C. and Hewett, B. (1999), “Quality in a respecification of Delone and McLean’s IS success model”, Managing IT Resources in Organizations in the Next Milennium, IRMA International Conference, Hershey, PA. Zviran, M., Chanan Glezer, C. and Avni, I. (2005), “User satisfaction from commercial web sites: the effect of design and use”, Information & Management, Vol. 43 No. 2, pp. 157-78. Further reading Collin, N. and Butler, P. (1995), “Marketing public sector services: concept and characteristics”, Journal of Marketing Management, Vol. 11, pp. 83-96. Egovernment news (2005), “Sweden – e-service for citizens”, Egovernment news, Interoperable Delivery of European eGovernment Services to Public Administrations, Businesses and Citizens (IDABC). Hamilton, B.A. (2005), Booze Allen Hamilton consulting report on world-wide best practice in e-government, commissioned by the UK Cabinet Office. Li, E.Y. (1997), “Perceived importance of information system success factors: a meta analysis of group differences”, Information & Management, Vol. 32, pp. 15-28. Skatteverkets pressmeddelande (2007), 3 May, available at: www.skatteverket.se Skatteverkets ptressmeddelande (2009), 6 May, available at: www.skatteverket.se/ omskatteverket/press/pressmeddelanden/riks/2009/2009/overhalftenavsvenskarnaval deedeklarationen.5.2e56d4ba1202f950120800012875.html
Government e-tax websites
Appendix
Items
Cronbach’s a
System quality This web site provides necessary information and forms to be downloaded This web site provides helpful instruction for performing my task This web site provides fast information access This web site quickly loads all the text and graphics It is easy to go back and forth between pages It only takes a few clicks to locate information It is easy to navigate within this site Information quality Information on this web site is free from errors This web site provides information precisely according to my need Information on this web site is up-to-date This web site provides information I need at the right time Information presented in this web site is related to the subject matter Information on this web site is sufficient for the task at hand Information contains necessary topics to complete related task
Corresponding author Dr Parmita Saha can be contacted at:
[email protected]
To purchase reprints of this article please e-mail:
[email protected] Or visit our web site for further details: www.emeraldinsight.com/reprints
0.874
321
0.863
Table AI. Item scale
Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.
