Information & Management 43 (2006) 767–777 www.elsevier.com/locate/im
Data triangulation and web quality metrics: A case study in e-government Stuart J. Barnes a,*, Richard T. Vidgen b a
University of East Anglia, UK b University of Bath, UK
Received 27 June 2005; received in revised form 27 December 2005; accepted 1 June 2006 Available online 9 August 2006
Abstract The work discussed here focused on the evaluation of quality perceptions of users of an electronic government website. As government organizations have begun to enhance transparency, communicate, and interact with citizens via the web, the development of appropriate online services has demanded better understanding of user requirements and thus for tailoring of solutions. The site we examined enabled the online submission of self-assessed tax returns in the UK. Survey data were used to model the perceptions of site users. In addition to the quantitative data, we also collected comments from respondents. These, using data triangulation, provided additional insight into the perceptions of site quality. The results of the comment analysis both supported the instrument and pointed to additional factors determining the perceptions of quality of e-government services needing attention in instrument development. # 2006 Elsevier B.V. All rights reserved. Keywords: Online tax submission; Website; Evaluation; Quality; eQual; Data triangulation; Comment analysis
1. Introduction Government exerts significant influence on the social factors that affect the development of legal, political, and economic infrastructure [10]. Electronic government spans many sectors and facets of society and has the potential to transform its citizen’s perceptions of civil and political interactions. Through the web, expectations of the service levels that sites should provide have been raised considerably [16]. Our research utilized the eQual approach (previously called WebQual) to analyse user perceptions of the quality of a national website provided by the UK
* Corresponding author. Tel.: +44 1603 456161; fax: +44 1603 593343. E-mail address:
[email protected] (S.J. Barnes). 0378-7206/$ – see front matter # 2006 Elsevier B.V. All rights reserved. doi:10.1016/j.im.2006.06.001
Government. The method was developed originally as an instrument for assessing user perceptions of the quality of e-commerce websites. The instrument has been under development since 1998 and has evolved via a process of iterative refinement and testing of validity and reliability in different domains. The essence of the method has focused on turning qualitative customer assessments into quantitative metrics for management decision-making (e.g. see [2,3]). In the eQual method, metrics are supplemented by open comments from respondents. If a large enough sample of these is provided they allow a degree of qualitative triangulation and help to understand why there was underlying statistical variance. This paper was written to show how to develop and apply multi-method principles of web quality assessment in the e-government domain with a view to data triangulation. The high number of responses to the questionnaire and mix of
768
S.J. Barnes, R.T. Vidgen / Information & Management 43 (2006) 767–777
qualitative and quantitative data allowed that. Moreover, via a detailed comment analysis, we could provide a detailed critique and refinement of the instrument. The website examined was that of the UK Inland Revenue—a site relating to tax policy and administration. From an e-government perspective, this site goes beyond information provision to interaction and transaction with citizens [7]. As such, it touches on many aspects of e-government web quality that have broad implications in other countries, particularly for governments following similar paths. 2. Background A project to evaluate the quality of the UK Inland Revenue website (http://www.ir.gov.uk/) was initiated by the Tax Management Research Network, a consortium of tax practitioners and academics, in early 2001. A major addition to the Inland Revenue’s website has been a self-assessment facility for tax returns, which was first used for returns of 5 April 2001. The site provided a high degree of interactivity and the online self-assessment facility was a major part of the Inland Revenue’s £ 200 million e-strategy [13] aimed at delivering 50% of services electronically by 31 December 2002. In addition, the long-term aims were to provide all services electronically by 31 December 2005. The proposed benefits for taxpayers were accuracy, convenience, confirmation of submission, and faster processing of any tax refunds. Whilst it was difficult to predict the savings, the department estimated that it would be able to reduce its staff by about 1300 positions. Evaluation of the website was undertaken using the eQual instrument, developed at the University of Bath, and was carried out during the period 1 August through 30 September 2001. Here we have presented the results of analysis of the comments that subjects posted while completing the survey. The standard quantitative eQual analysis is thus supplemented by qualitative comments of the respondents to provide triangulation of the results and a deeper insight into user attitudes. 3. Research methodology
validated instrument, it was possible to compare results from previous studies as well as rely on a validated instrument [17,30]. eQual is based on quality function deployment (QFD), which is a ‘‘structured and disciplined process that provides a means to identify and carry the voice of the customer through each stage of product and or service development and implementation’’ [28]. Use of QFD starts by capturing the ‘voice of the customer’ (their needs expressed simply); these then form the basis of an evaluation of the quality of the product or service. In the context of eQual, website users are asked to rate target sites against each of a range of qualities and to rate each of them on their importance. Although the qualities in eQual were designed to be subjective, significant data analysis was performed using quantitative techniques, for tests of the reliability of the instrument, and so on. eQual has been changed and gone through many iterations. Its development has been discussed fully elsewhere. eQual 4.0, as shown in Table 1, draws on research from three core areas: Table 1 The eQual questionnaire (previously called WebQual) Category
Questions
Usability
1. I find the site easy to learn to operate 2. My interaction with the site is clear and understandable 3. I find the site easy to navigate 4. I find the site easy to use 5. The site has an attractive appearance 6. The design is appropriate to the type of site 7. The site conveys a sense of competency 8. The site creates a positive experience for me
Information quality
9. Provides accurate information 10. Provides believable information 11. Provides timely information 12. Provides relevant information 13. Provides easy to understand information 14. Provides information at the right level of detail 15. Presents the information in an appropriate format
Service interaction
16. Has a good reputation 17. It feels safe to complete transactions 18. My personal information feels secure 19. Creates a sense of personalization 20. Conveys a sense of community 21. Makes it easy to communicate with the organization 22. I feel confident that goods/services will be delivered as promised
Overall
23. Overall view of the website
3.1. The eQual instrument A review of the literature on website evaluation revealed no comprehensive instruments aimed at egovernment web services. Therefore, we adapted the eQual instrument format for both interactive and noninteractive users. By using a previously developed and
S.J. Barnes, R.T. Vidgen / Information & Management 43 (2006) 767–777
Information quality from IS research. A core part of version 1.0 was for online information. The questions developed in this segment build on literature focused on information, data and system quality, including Bailey and Pearson [1], Strong et al. [31] and Wang [33]. Interaction and service quality from marketing, e-commerce and IS service quality research. Bitner [6] adopted Shostack’s [27] definition of a service encounter as ‘‘a period of time during which a consumer directly interacts with a service’’ and note that such interactions need not be interpersonal—they can occur without human interaction. Bitner also recognized that often ‘‘interaction is the service from the customer’s point of view’’. We suggest that interaction quality may even be more important to the success of e-businesses than to a ‘‘brick and mortar’’ organization. In version 2.0 of the instrument we extended the interaction aspects by adapting and applying the work on service quality, chiefly SERVQUAL [22,36,35] and IS SERVQUAL [15,23,24,32]. Usability from human–computer interaction. In eQual 4.0 this dimension draws from literatures in the field of human–computer interaction [8,9,19] and more latterly web usability [20,21,29]. Usability is concerned with how a user perceives and interacts with a website. It is not concerned with design principles per se. We have used workshops at every stage of eQual development to ensure that the qualities were relevant, particularly when they related to pre-internet literature and new organizational or industrial settings, such as e-government. 3.2. Design of the evaluation The standard eQual instrument contains 23 questions. These are shown in Table 1. Three of them relate to the provision of personal information while making transactions: Question 17: It feels safe to complete transactions. Question 18: My personal information feels secure. Question 22: I feel confident that goods/services will be delivered as promised. These questions are relevant to respondents using the self-assessment facilities of the IR website but not to those who are using the site for information gathering purposes. By self-assessment, we refer to the online
769
submission of tax returns processed by a taxpayer using the self-assessment guidelines. The survey said ‘‘please tick n/a if you have not used the internet service for selfassessment or the internet service for PAYE’’.1 This allowed the data set to be divided between ‘‘information gatherers’’ and ‘‘interactors’’. The survey of the site quality was conducted using an internet-based questionnaire. Its home page had instructions and guidelines for completion of the instrument. Using this, the user opened a separate window (control panel) containing the qualities to be assessed. The control panel allowed the user to switch the contents of the target window between the instruction page, the IR website, and a quality dictionary, which was linked to the question number, allowing the respondent to obtain a definition for any quality. Users were asked to rate the IR site for each quality using a Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree). Users were also asked to rate the importance of the quality to them, again using 1 (least important) to 7 (most important). Open comments were encouraged and 65% of respondents provided additional comments. There were 420 usable responses. Demographic and other respondent information were collected from the respondents. In particular, we were interested in the age, sex, type of user, their use of the site, and their experience with and use of the internet. The respondents were typically highly experienced, intensive users of the internet, although not intensive users of the IR website. The majority of respondents were male (71%) and of a working age. Ten percent used the IR site daily. Agents and accountants were 15.5% of respondents, while 60% categorized themselves as other. 3.3. Validity and reliability The eQual instrument drew primarily on items that had been validated by earlier researchers, in particular Davis, Bailey and Pearson, Strong and Wang, and the work of Zeithaml et al. Previous applications of eQual have shown that the scale had strong reliability. However, it was still necessary to demonstrate the reliability of the items with the dataset. The internal consistency reliability was assessed by Cronbach’s a’s as shown in Table 2. Values above 0.60 are considered acceptable and it is clear that the scale possessed a high degree of reliability here. Previous research has shown that the three theoretical sources in Table 1 yield five dimensions 1
Meaning ‘‘Pay as You Earn’’—the incremental income tax for employees in the UK.
770
S.J. Barnes, R.T. Vidgen / Information & Management 43 (2006) 767–777
Table 2 Cronbach’s a-scores Category
Question number
a if item deleted
a for category
Usability
Q1 Q2 Q3 Q4
0.94 0.94 0.95 0.94
0.96
Site design
Q5 Q6 Q7 Q8
0.84 0.81 0.79 0.80
0.85
Information quality
Q9 Q10 Q11 Q12 Q13 Q14 Q15
0.92 0.92 0.92 0.92 0.92 0.92 0.92
0.93
Trust
Q16 Q17 Q18 Q22
0.86 0.80 0.82 0.82
0.86
Empathy
Q19 Q20 Q21
0.79 0.80 0.84
0.86
of the eQual instrument. Every sentence was covered at least by one code. The second and third rounds involved comparing and examining the text under the codes through the view provided by NVivo. The tree structure was re-organized according to this examination. The third and final round of coding was used to ensure consistency and accuracy. When the coding process was finished, the codes were categorized according to the eQual framework. In addition, the attitude expressed through each comment was categorized by using codes such as ‘‘criticism’’, ‘‘praise’’, ‘‘suggestion’’ and ‘‘untitled’’ (i.e. not a relevant comment). This code set helped triangulate the quality scores given by the respondents. Every comment was given at least one code. 3.5. Triangulation
Note: the overall a for the questionnaire is 0.971.
through factor analysis and that these dimensions of website quality also have satisfactory a-values. Convergent validity between the scale and question 23 (overall view of the website) is clearly evident with a Pearson correlation of 0.89, significant at the 1% level. 3.4. Qualitative analysis of open comments The open comments of respondents were used to perform further detailed analysis. In particular, we wanted to compile the user perceived qualities that were mentioned, compare them with the eQual questions, and perform a triangulation of them with the quantitative results. Comments were imported into NVivo (a tool for qualitative analysis) as documents associated with demographic data of the respondents. NVivo was then used to code the data. In order to compare and contrast this with the eQual framework, the codes for user perceived website qualities were not extracted from the eQual questionnaire but emerged from the preliminary coding process and were refined through several iterations. No structure was imposed during this coding, and, to reduce bias, it was carried out by a doctoral student who had not been involved in the development
A major threat to the validity of research is its lack of internal validity—the relationships between two variables may be incorrect [26]. Guba and Lincoln [12] provided a parallel definition for internal validity in qualitative research as credibility. It depends on how well the findings match reality, i.e. how credible the findings are to the observer. Triangulation of quantitative and qualitative data allows a researcher to validate and crosscheck the findings. Kaplan and Duchon [14] argued that: ‘‘Collecting different kinds of data by different methods from different sources provides a wider range of coverage that may result in a fuller picture of the unit under study . . . Moreover, using multiple methods increases the robustness of results because findings can be strengthened through triangulation – the cross-validation achieved when different kinds and sources of data converge and are found congruent.’’ Rajagopal [25] drew on Kaplan and Duchon to argue for the use of triangulation in a study of ERP implementation and used case interviews followed by a survey instrument to see whether ‘‘the qualitative data from the interviews matched their quantitative responses’’. The research reported here was slightly different—the same sources were used to collect data at a single point in time, thus reducing problems of collecting data at different times. Fielding and Schreier [11] identify three meanings of triangulation. The validity model referred to the triangulation as validation of results obtained using different methods. The complementarity model used the term triangulation to describe a way of getting a broader and more complete picture of a research context. A
S.J. Barnes, R.T. Vidgen / Information & Management 43 (2006) 767–777
771
Fig. 1. Combining quantitative and qualitative analysis (triangulation) for internal validity and instrument development.
trigonometrical approach described a combination of methods that represented the research phenomenon being found by alternative measures. Our study primarily used the validity model in order to provide mutual validation of the eQual instrument and the qualitative comments. However, the output of a triangulation exercise should provide a stronger and more robust eQual instrument and could represent a complementarity approach—a broader and more complete picture of website quality with new factors for inclusion in the eQual instrument (see Fig. 1). 4. Quantitative results For the quantitative analysis, we were particularly interested to discover the aspects of the eQual instrument that determined the user’s overall perception of the quality of the IR site. We were also interested in the quality priorities of each user, measured as the importance of each question. The data analysis was conducted on the weighted dataset, where the rating for a question for each respondent was multiplied by its perceived importance. A summary of the results was presented for comparative purposes in Table 3; see also Barnes and Vidgen [4,5]. One key aim was to achieve an overall quality rating for the website in order to benchmark the perceptions of its users. The total scores, however,
made it difficult to use a standard benchmark for the website, especially since questions 17, 18 and 22 were omitted from the responses of non-interactive users. One way to achieve it would be to index the total weighted score for a site against its total possible score (i.e. the importance for all questions multiplied by 7, the maximum rating for a site). The result of this was expressed as a percentage. Overall, the interactive users benchmarked below the non-interactive ones (62% versus 72%, respectively, for the eQual Index or EQI). Even more remarkable was that the evaluations of interactive users rated consistently below those of non-interactive users for all questions, with differences ranging from 1 to 18 points. The largest differences relate to usability (items 1, 4, 2, 3), followed by competency and understandable information. The data indicated differences in perceptions in terms of eQual site quality. We examined where these perceived differences had occurred and considered the overall shape of the evaluation of the IR site. Previous research for eQual has led to a number of valid and reliable question subgroupings. These categories provided some useful criteria by which to assess the perceptions of site users. As a starting point, the data was summarised according to the questionnaire subcategories. Then, the total score for each category was indexed against the
772
S.J. Barnes, R.T. Vidgen / Information & Management 43 (2006) 767–777
Table 3 Weighted scores and eQual indices—interactive and non-interactive users No.
Description
1 2
I find the site easy to learn to operate My interaction with the site is clear and understandable I find the site easy to navigate I find the site easy to use The site has an attractive appearance The design is appropriate to the type of site The site conveys a sense of competency The site creates a positive experience for me Provides accurate information Provides believable information Provides timely information Provides relevant information Provides easy to understand information Provides information at the right level of detail Presents the information in an appropriate format Has a good reputation It feels safe to complete transactions My personal information feels secure Creates a sense of personalization Conveys a sense of community Makes it easy to communicate with the organization I feel confident that goods/services will be delivered as promised
3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Total
Maximum score (I)
Interact Weighted score
EQI1 (%)
Maximum score (NI)
No interaction Weighted score
EQI2 (%)
Difference (EQI2 EQI1) (%)
42.1 41.9
23.4 23.6
56 56
42.8 42.0
31.5 30.1
74 72
18 15
42.5 43.1 30.4 33.1 39.4 37.0 44.5 43.3 41.9 42.8 42.9 41.1
23.8 23.9 19.6 22.7 24.8 18.9 31.9 32.9 29.2 30.2 25.0 25.4
56 55 64 68 63 51 72 76 70 71 58 62
43.8 43.9 28.5 33.9 38.6 35.0 46.0 46.3 45.4 45.9 44.1 43.3
30.5 31.4 19.8 26.6 29.8 22.1 37.8 39.7 35.1 35.9 31.8 29.7
70 72 69 78 77 63 82 86 77 78 72 68
14 16 5 10 14 12 11 10 8 7 14 7
39.8
26.4
66
40.7
30.5
75
9
36.7 42.3 42.8 31.8 26.1 39.1
22.4 30.7 31.9 16.0 12.3 19.8
61 73 75 50 47 51
37.4 44.9 45.2 25.9 21.1 37.5
27.4 – – 13.3 10.3 20.5
73 – – 51 49 55
12 – – 1 2 4
41.0
23.2
57
43.6
–
–
–
865.8
537.8
62
875.5
533.6
72
10
Note: n = 420; interactive users = 264; non-interactive users = 156.
maximum score (the importance ratings for questions multiplied by 7). Fig. 2 rated the two sets of users for these criteria (the trust category only used question 16 for users who ‘do not interact’). Further, the scale had been adjusted to between 40% and 80% to allow for
Fig. 2. Radar chart of eQual subcategories for user groups.
clearer comparison. Clearly the users who do not interact with the site had higher perception for all aspects, although the general pattern of site ratings was similar for all users. In absolute terms, for users who ‘do not interact’ all site categories rated highly (between 72% and 77%), except for empathy (52%). Although this category also rates lowest in importance, it does indicate an opportunity for building relationships with users. For ‘interactive’ users, empathy, usability and design rate lowest (at 49%, 56% and 61%, respectively), with information (68%) and trust (66%) the best rated scores. Fig. 2 shows that the biggest subcategory differences in perceptions are in usability and design—16% and 11%, respectively. Close behind is information quality, at 9%. The most similar quality perceptions were for empathy. Apparently, interaction with the Inland Revenue site severely affected perceptions of usability and design.
S.J. Barnes, R.T. Vidgen / Information & Management 43 (2006) 767–777
5. Qualitative results 5.1. Sampling issues In order to triangulate the quantitative analysis on the whole sample (420 cases) with the comments of 273 respondents, it was necessary to verify that these 273 cases were a random selection. They were tested using ANOVA, with the compared statistics including the interactive and non-interactive users, the distribution of demographics and web experiences of the respondents, and the means, standard deviations, and standard errors of the means of the 23 items of the two samples. The result showed that the sample was a random selection from the 420 cases. 5.2. Results of the comment analysis Table 4 lists the themes covered by the coded open comments and their occurrence frequencies. The code occurrence frequencies, assumed to be a measure of the
773
relevance and importance of the themes to the respondents, were calculated in NVivo. The statistical mean of the code occurrence frequency was 14.5, the median was 12, and the upper quartile was 23.3. The coded comments of Table 4 had not yet been organized into common groupings. Therefore, although there were clear areas of common interest among the respondents’ comments, systematic comparison with the quantitative data was difficult. To facilitate triangulation, the framework was used to organize the codes into four groupings: three conceptual groups from the eQual instrument and an other group used when placement of a specific comment code was difficult. The results are shown in Table 5. The next challenge is to make some assessment of the importance and rating of qualities, and thus quality categories, as inferred from the comment analysis data. It is reasonable to assume that those areas commented on most by respondents were also considered the most important. This is particularly so considering that the respondents typically only mentioned a few topics of
Table 4 The coding scheme and code occurrence frequency Code
Navigation Locating information Information provision Form completion Usefulness Search facility Communication with organization Ease of use How informative Authentication Organization and format Responsiveness Online help Experience with the site Function provision User-friendliness Look and feel Clarity System performance Currency Instruction Notification Simplicity Consistency Accuracy Security Feedback mechanism Personalization need Advertisement
Occurrence frequency
43 41 33 31 30 25 25 18 17 17 13 12 12 12 12 11 11 10 7 7 7 5 4 3 1 1 1 1 1
Attitude Criticism
Praise
Suggestion
Untitled
36 36 18 28 7 24 17 14 3 16 10 11 7 9 4 9 5 7 5 3 6 3 4 3 1 0 1 0 0
6 5 1 1 22 0 1 4 14 1 3 1 0 3 0 2 5 3 2 1 0 0 0 0 0 1 0 0 0
1 0 11 2 0 1 7 0 0 0 0 0 5 0 8 0 1 0 0 3 1 2 0 0 0 0 0 1 0
0 0 3 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
774
S.J. Barnes, R.T. Vidgen / Information & Management 43 (2006) 767–777
Table 5 Comment codes organized according to the eQual framework Category
No.
Comment code
Occurrence frequency
eQual equivalence
Usability
1 2 3 4 5 6 7 8 9 10 11
Navigation Locating information Form completion Search facility Ease of use Experience with the site User-friendliness Look and feel System performance Instruction Simplicity
43 41 31 25 18 12 11 11 7 7 4
Navigation (Q3) Navigation (Q3) Usability (Q1, 2, 4) Not a quality Usability (Q1, 2, 4) Site design (Q8) Usability (Q1, 2, 4) Site design (Q5) No equivalent quality Usability (Q1–4) Site design (Q6)
Information
12 13 14 15 16 17 18 19
Information provision Usefulness How informative Organization and format Clarity Currency Consistency Accuracy
33 30 17 13 10 7 3 1
General (Q9-15) Relevance (Q12) General (Q9–15) Format (Q15) Easy to understand (Q13) Timeliness (Q11) General (Q9–15) Accuracy (Q9)
Service
20 21 22 23 24 25 26 27
Communication with organization Authentication Responsiveness Online help Notification Security Feedback mechanism Personalization need
25 17 12 12 5 1 1 1
Communication (Q21) Security (Q18) Communication (Q21) Communication (Q21) Communication (Q21) Security (Q18) Communication (Q21) Personalization (Q19)
Other
28 29
Function provision Advertisement
12 1
Not a quality Not a quality
importance in the short statements that they added to the end of the eQual questionnaire; effort is likely to be focused on topics of immediate concern. To gauge the level of importance, we applied rudimentary statistics to the data, as shown in Table 6. First, comments were grouped into baskets of eQual subcategories: usability, site design, information, trust, and empathy. This provided numbers of comments. Next, an average expected number was calculated for each category; this was computed as the average number of comments for each comment code multiplied by the number of comment codes in each category.
Table 6 shows the difference between these as well as the percentage of actual to expected comments. Based on the results, it appeared that usability was considered to be the most important category (153% of expected comments), whilst information quality was of moderate importance (97%). All the other categories were rated as low importance, 61–63% of those expected. The next step was to make an assessment of the perceived quality of the Inland Revenue site from the respondents’ comments. For this, we focused on comments that had been interpreted to make a value judgement (positive or negative)—others were ignored.
Table 6 Assessing the importance of comment categories eQual category
Coded comments
Actual
Average expected
Usability Site design Information Trust Empathy
#1–3, 5, 7, 9, 10 #6, 8, 11 #12–19 #21, 25 #20, 22–24, 26, 27
158 27 114 18 56
103 44 118 29 88
Difference 55 17 4 11 32
Actual/expected (%)
Importance
153 61 97 61 63
High Low Moderate Low Low
S.J. Barnes, R.T. Vidgen / Information & Management 43 (2006) 767–777
775
Table 7 Assessing the perceived quality of comment categories eQual category
Comments
Usability Site design Information Trust Empathy
#1–3, 5, 7, 9, 10 #6, 8, 11 #12–19 #21, 25 #20, 22–24, 26, 27
ve 134 18 52 16 39
+ve
+ve +
20 8 44 2 2
154 26 96 18 41
ve
+ve/total (%)
+ve/ ve (%)
Rating A/E (%)
13 31 46 11 5
15 44 85 13 5
20 19 44 7 3
Overall, the comments tallied well with the eQual instrument. Some comments tallied with specific qualities, while others were more general or provided specific instances of a quality. Four comment areas (#4, #23, #28 and #29) were not qualities in the true sense— rather, they referred to parts or functions, either specifically or generally. However, there were two areas that suggested a need for further development of eQual:
Fig. 3. Radar chart of eQual subcategories based on comment analysis.
Table 7 lists the numbers in the eQual categories. Using totals, we can analyze the social consensus on perceived qualities. Two columns in the table provided some indication of this: first, the proportion of positive comments in the total; second, the ratio of positive to negative comments. Each of these fell well below that of the quantitative analysis. In order to provide a weighted score, we created an overall comment score for the eQual categories. This was based on the actual/expected (A/E) ratio from Table 6 multiplied by the proportion of positive comments from the total. The result is the last column in Table 7. To provide a clearer representation that is easily comparable to the quantitative analysis, the results are displayed in Fig. 3, where the scale is from 0% to 50% to allow for easier interpretation. 6. Discussion Overall, the qualitative results appeared to triangulate well with the quantitative ones. Empathy had the lowest score, while information quality had the highest. The most marked difference was the low trust score, close to empathy in the qualitative analysis but similar to usability and site design in the quantitative results.
Comment code #4 (search facility) was closely related to #2 (locating information), and although we mapped #2 to the eQual item ‘‘ease of navigation’’ there was a suggestion that ‘‘locating information’’ was a quality distinct from navigation (a quality also identified by Yang et al. [34]). Comment code #9 (system performance) also appeared to be relevant and did not have adequate coverage in the eQual instrument. Muyllea et al. [18] identify ‘‘website speed’’ as a factor in user perceptions of quality. Coding of the open comments provided by respondents is valuable in two ways. Firstly, it helps to triangulate the results and provide evidence to support the validity of the eQual instrument. Secondly, the open comments are a source of new concepts that can subsequently be developed and tested to further improve the instrument. The open comments are particularly powerful because they are collected at the same time as the survey data and reflect the opinions of real users grappling with a real website rather than subjects in a quality workshop considering quality in more abstract terms. 7. Summary and conclusions Our research examined an important area of development for digital government—online taxation systems. It focused on experiences surrounding the introduction of an online facility in the UK for self-assessed tax returns, and specifically, in evaluating the factors impacting on
776
S.J. Barnes, R.T. Vidgen / Information & Management 43 (2006) 767–777
user perceptions. eQual was based on user perceptions of quality weighted by their perceived importance. Within eQual, the five factors were: usability, design, information, trust, and empathy. We have evaluated these using both qualitative and quantitative methods to provide a degree of triangulation. The results demonstrated that the use of comment analysis with traditional survey data provided a useful method of triangulation, adding strength to the results of the assessment. The novel method of quantifying respondent comments provided a contribution to data triangulation for web quality assessment. In the quantitative results, interaction was a clear determinant of the user’s perception of overall website quality. Key problems affecting the perceptions of interactive users are the usability of the self-assessment facility and difficulty in communicating with the organization. The qualitative comment analysis underlined this and the comments of respondents (which totalled 273 out of 420) were typically critical and from interactors. Respondents were more critical of site design and usability when using comments. System logs of the UK website showed that nearly four out of five attempted submissions in 1999–2000 did not succeed on the first attempt: the proportion of successful attempts only reached 44% on average between April and September 2001 though it improved to an average of 70% for the quarter ending December 2001. The area of greatest difference was trust, which scored lower in the qualitative analysis. Another major finding was the low perceived level of empathy in both quantitative and qualitative results. However, the need for empathy (particularly in communication) in the delivery of services is a significant finding for practice. Interestingly, the Inland Revenue is moving from its existing arrangements for taxpayers to their filing tax returns at a ‘portal’ offering secure personalized services. Overall, data triangulation has proved to be a valuable research tool. It has integrated new ideas for quantifying qualitative data for use in quality assessment activities wherever researchers seek to develop the content of an instrument through triangulation with grounded qualitative data. Acknowledgements We wish to acknowledge the help of Xiaofeng Wang, a doctoral student at the University of Bath, who provided the initial coding of the comments used in this analysis. An earlier version of this paper was presented
at the European Conference on Information Systems in 2005. References [1] J.E. Bailey, S.W. Pearson, Development of a tool for measuring and analyzing user satisfaction, Management Science 29(5), 1983, pp. 530–545. [2] S.J. Barnes, R.T. Vidgen, An evaluation of cyber-bookshops: the WebQual method, International Journal of Electronic Commerce 6(1), 2001, pp. 6–25. [3] S.J. Barnes, R.T. Vidgen, An integrative approach to the assessment of e-commerce quality, Journal of Electronic Commerce Research 3(3), 2002, pp. 114–127. [4] S.J. Barnes, R.T. Vidgen, Interactive e-government: evaluating the web site of the UK Inland Revenue, Journal of Electronic Commerce in Organizations 2(1), 2003. [5] S.J. Barnes, R.T. Vidgen, Interactive eGovernment services: modelling user perceptions with eQual, Electronic Government 1(2), 2004, pp. 213–228. [6] M.J. Bitner, Evaluating service encounters: the effects of physical surroundings and employee responses, Journal of Marketing (April) 1990, pp. 69–82. [7] S. Cohen, W. Eimicke, The future of e-government: a project of potential trends and issues, in: Proceedings of the 36th Hawaii International Conference on Systems Sciences, Hawaii, 2003. [8] F. Davis, Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS Quarterly 13(3), 1989, pp. 340–391. [9] F. Davis, User acceptance of information technology: system characteristics, user perceptions, and behavioral impacts, International Journal of Man-machine Studies 38, 1993, pp. 475– 487. [10] A.K. Elmagarmid, W.J. McIver, The ongoing march toward digital government, IEEE Computer 34(2), 2001, pp. 32–38. [11] N. Fielding, M. Schreier, Introduction: on the compatibility between qualitative and quantitative research methods, Forum: Qualitative Research 2(1), 2001. [12] E.G. Guba, Y.S. Lincoln, Fourth Generation Evaluation, Sage Publications, Newbury Park, Calif, 1989. [13] HMSO e-Revenue, HMSO, London, 2002. [14] B. Kaplan, D. Duchon, Combining qualitative and quantitative methods in information systems research: a case study, MIS Quarterly (December) 1988, pp. 571–586. [15] W. Kettinger, C. Lee, Pragmatic perspectives on the measurement of information systems service quality, MIS Quarterly (June) 1997, pp. 223–240. [16] H. Kubicek, M. Hagen, Integrating e-commerce and e-government: the case of bremen online services, in: J. Prins (Ed.), Designing E-government, Kluwer Law International, Hague, 2001. [17] M. Malhotra, V. Grover, An assessment of survey research in POM: from construct to theory, Journal of Operations Management 16(4), 1998, pp. 403–423. [18] S. Muyllea, R. Moenaert, M. Despontin, The conceptualization and empirical validation of web site user satisfaction, Information and Management 41, 2004, pp. 543–560. [19] J. . Nielsen, Usability Engineering, Morgan Kaufmann, CA, 1993 [20] J. Nielsen, User interface directions for the web, Communications of the ACM 42(1), 1999, pp. 65–72.
S.J. Barnes, R.T. Vidgen / Information & Management 43 (2006) 767–777 [21] J. Nielsen, Designing Web Usability, New Riders, Indiana, 2000. [22] A. Parasuraman, Measuring and monitoring service quality, in: W. Glynn, J. Barnes (Eds.), Understanding Services Management, Wiley, Chichester, 1995, pp. 143–177. [23] L. Pitt, R. Watson, C. Kavan, Service quality: a measure of information systems effectiveness, MIS Quarterly (June) 1995, pp. 173–187. [24] L. Pitt, R. Watson, C. Kavan, Measuring information systems service quality: concerns for a complete canvas, MIS Quarterly (June: 209–221) 1997, pp. 173–187. [25] P. Rajagopal, An innovation-diffusion view of implementation of enterprise resource planning (ERP) systems and development of a research model, Information and Management 40, 2002, pp. 87–114. [26] W.R. Shadish, T.D. Cook, D.T. Campbell, Experimental and Quasi-experimental Designs for Generalized Causal Inference, Houghton Mifflin, Boston, 2002. [27] G. Shostack, Planning the service encounter, in: J. Czepiel, M. Solomon, C. Surprenant (Eds.), The Service Encounter, Lexington Books, Lexington, MA, 1985. [28] R. Slabey, QFD: a basic primer, in: Proceedings of the Transactions from the Second Symposium on Quality Function Deployment, Novi, MI, 1990. [29] J. Spool, Web Site Usability, Morgan Kaufmann, CA, 1999. [30] D. Straub, C.L. Carlson, Validating instruments in MIS research, MIS Quarterly 13(2), 1989, pp. 147–169. [31] D.M. Strong, Y.W. Lee, R.Y. Wang, Data quality in context, Communications of the ACM 40(5), 1997, pp. 103–110. [32] T. Van Dyke, L. Kappelman, V. Prybutok, Measuring information systems service quality: concerns on the use of the SERVQUAL questionnaire, MIS Quarterly (June) 1997, pp. 195–208. [33] R.Y. Wang, A product perspective on total data quality management, Communications of the ACM 41(2), 1998, pp. 58–65. [34] Z. Yang, S. Caib, Z. Zhouc, N. Zhoua, Development and validation of an instrument to measure user perceived service quality of information presenting web portals, Information and Management 42, 2005, pp. 575–589. [35] V. Zeithaml, A. Parasuraman, L. Berry, Delivering Quality Service: Balancing Customer Perceptions and Expectations, The Free Press, New York, 1990.
777
[36] V.A. Zeithaml, L. Berry, A. Parasuraman, The nature and determinants of customer expectations of service, Journal of the Academy of Marketing Science 21(1), 1993, pp. 1–12. Stuart J. Barnes is chair and professor of management in Norwich Business School at the University of East Anglia. Previously he worked at Victoria University of Wellington, New Zealand, and the University of Bath. Stuart has been teaching and researching in the information systems field for nearly 15 years. His academic background includes a first class degree in economics from University College London and a PhD in business administration from Manchester Business School. His primary research interests centre on the successful utilization of new information and communications technologies by businesses, governments and consumers. He has published five books (one a best-seller for Butterworth-Heinemann) and more than 90 articles including those in journals such as Communications of the ACM, the International Journal of Electronic Commerce, Communications of the AIS, and Information & Management. Richard Vidgen is professor of information systems in the School of Management at the University of Bath. He worked in information systems development in industry for 15 years, during which time he was employed by a large US software firm, a high street bank, and as a consultant. Richard holds a first degree in computer science and accounting, and an MSc in accounting. In 1992 he left industry to join the University of Salford, where he completed a PhD in systems thinking and information system quality. His current research interests include systems theory, information system development methodologies, business process modeling, and e-commerce quality. He has published the books Data Modelling for Information Systems (1996) and Developing Web Information Systems (2002) as well as many book chapters and journal papers.
