Univ Access Inf Soc (2017) 16:987–996 DOI 10.1007/s10209-016-0490-z
LONG PAPER
Web site accessibility and quality in use: a comparative study of government Web sites in Kyrgyzstan, Azerbaijan, Kazakhstan and Turkey Rita Ismailova1 • Yavuz Inal2
Published online: 8 August 2016 Springer-Verlag Berlin Heidelberg 2016
Abstract Government Web sites aim to provide information to the citizens of the country; therefore, they should be accessible, easy to use and visible via search engines. Based on this assumption, in this paper, the ministry Web sites of four countries namely the Kyrgyz Republic, the Republic of Azerbaijan, the Republic of Kazakhstan and the Republic of Turkey were analyzed in terms of accessibility and quality in use. Tests were carried out utilizing online automated tools. Results indicate that the usage rate of Information and Communication Technologies by the government is higher in Turkey, which affects the visibility of government Web sites but not their quality in use. Very few ministry Web sites of the four countries achieved AA conformance level on accessibility, many failed to pass conformance level A and AA checkpoints for accessibility errors. In order to ensure equal access to all their citizens, the countries in this study need to put more emphasis on designing government Web sites to be more accessible. Keywords Accessibility Quality Web performance Online evaluation tools Government Web sites
& Yavuz Inal
[email protected] 1
Department of Computer Engineering, Faculty of Engineering, Kyrgyz-Turkish Manas University, Bishkek, Kyrgyz Republic
2
TUBITAK BILGEM, Software Technologies Research Institute, Ankara, Turkey
1 Introduction Today, Internet technologies continue to undergo significant development and public institutions cannot be left out of this change. People using Internet applications in almost all areas of life naturally expect to be provided with public services based on these technologies. Thus, government Web sites have become an important interface between citizens and the government [19] and are a fundamental source to access necessary information and public services [19], [49]. E-government is an effective tool for the provision of public services [49] and is also considered as an important way of facilitating the use of online public services [14]. In this regard, people should be able to directly interact with government Web sites, easily access information and actively participate in the decision-making processes, which are clearly defined in the conventional public approach. [14]. Therefore, the traditional public–government relationship should be redefined considering the effective use of the opportunities offered by Internet technologies in order not to disrupt the services demanded by the public. The efficiency, performance and satisfactory quality of government Web sites have a significant impact on people’s effective use of public services [29]. Since people encounter various problems when using government Web sites [12], one of the important challenges in the presentation of E-government services is to design a user-friendly interface [47]. In determining an effective government Web site, this design is a significant performance criterion and a user-friendly Web site has good usability and functionality features [26]. Although citizens outside the country may access the government Web sites, the main target group of public institutions in a country is all citizens inhabiting that country, and it is
123
988
crucial that these Web sites fulfill the quality criteria and provide equal user experience to all. Furthermore, E-government services are negatively affected by the failure of government Web sites to present usable interfaces for their target citizens [6]. The most essential characteristic of a government Web site is high accessibility. According to W3C, ‘‘Web accessibility means that people with disabilities can perceive, understand, navigate and interact with the Web.’’ The Web Content Accessibility Guidelines (WCAG 2.0) developed by W3C provide accessibility criteria which are used to categorize Web sites as A, AA or AAA according to their conformance to these guidelines [18]. It is of significant importance to evaluate government Web sites in terms of both accessibility and quality in use. To date, a considerable number of studies have analyzed government Web sites in terms of accessibility and web performance. The main purpose of these studies is to conduct tests on government Web sites using an online tool and evaluate the results in terms of accessibility and quality based on specific criteria. However, in the literature, only few studies compare the accessibility and quality performance of government Web sites in different countries. On December 13, 2006, the United Nations General Assembly adopted the Convention on the Rights of Persons with Disabilities. Article 9 of the Convention dedicated to the principle of availability states ‘‘States Parties shall take appropriate measures to ensure to persons with disabilities access, on an equal basis with others, to the physical environment, to transportation, to information and communications, including information and communications technologies and systems’’ [41]. The Republic of Azerbaijan ratified the Convention on January 21, 2009 [39], in Turkey it was ratified on September 28, 2009 [11], and in Kazakhstan on February 20, 2015 [13]. In the Kyrgyz Republic this Convention is—at the time of writing this paper—under consideration [35]. In addition to the UN Convention, some countries have local laws or legislations to ensure that government Web sites satisfy accessibility requirements. However, in most countries (e.g., Kyrgyz Republic or Turkey), there is no local legislation about WCAG standards to make government institutions have more accessible Web sites for disabled or elderly citizens. If results of online evaluation tools were taken into account carefully while designing, developing and maintaining the government Web sites, citizens might be prevented from encountering accessibility barriers. This might also help governments improve the quality of their Web sites and satisfaction of their citizens. In the present study, we aimed to assess the government Web sites of the Kyrgyz Republic, the Republics of Kazakhstan, Azerbaijan and Turkey by using online testing tools. These countries are at different positions at the United Nations report on the
123
Univ Access Inf Soc (2017) 16:987–996
development of information and communication technologies (ICT) and E-government [46]. To this end, the Web sites of 16 ministries of these countries having similar responsibilities were analyzed and compared in relation to their performance in terms of accessibility and quality in use.
2 Related work [22] assessed the accessibility of 10 university Web sites randomly selected from 77 universities in Turkey through graphical browser testing, text-only browser testing, automated testing and validation testing. Validation testing was performed by testing (X)HTML, CSS and the code sources of the homepages. As a result of the study, the author reported that none of the Web sites fulfilled all accessibility criteria and the Web sites differed in terms of the criteria they met. Similarly, [4] evaluated the accessibility of 25 government Web sites in Turkey using automated testing tools according to their conformance to WCAG 1.0 and 2.0. The authors mostly found conformance level A— ‘‘must satisfy’’ accessibility issues in these Web sites, reporting significant problems in the presentation of a text alternative to non-text content. [1] analyzed the accessibility of government Web sites in Saudi Arabia and Oman using WatchFire Bobby, W3C HTML Validator and UsableNET LIFT. They reported that the accessibility of the government Web sites of both countries needs improvement. In another study in Jordan, the Bobby tool was used to test the accessibility of government Web sites [3], which was found to be very low and needed significant improvement. Furthermore, [5] evaluated 25 ministry Web sites in Jordan to determine whether these Web sites fulfilled the requirements of disabled users. The results of both the automated and manual tests demonstrated that the analyzed Web sites did not meet the accessibility standard. [17] analyzed the usability and accessibility of 155 government Web sites in Malaysia using automated testing tools such as Websiteoptimization, Axandra and EvalAccess 2.0, and found a significant number of usability and accessibility problems. In another study evaluating the accessibility of government Web sites also in Malaysia [25], none of the analyzed Web sites passed the W3C Priority 1 checkpoint. [33] performed automated testing on a group of Web sites including ministry Web sites in India to evaluate their conformance to WCAG 2.0. According to the results of the analysis based on the POUR principles (Perceivable, Operable, Understandable and Robust) of WCAG, the government Web sites violated these principles less than commercial and educational Web sites. Furthermore, the government Web sites were found to have
Univ Access Inf Soc (2017) 16:987–996
the least number of violations of the Understandable and Robust principles; however, they did violate the Perceivable and Operable principles. [23] used an online testing tool to analyze the accessibility of 130 government Web sites in the United Kingdom according to the maturity levels defined by WCAG. It was reported that 23 % of the Web sites fulfilled the WCAG level 1 criteria and only 5 % achieved WCAG 2.0 conformance. [21] tested the accessibility of 39 E-government Web sites in the Czech Republic using automated tools and found that the majority of these Web sites have a low level of accessibility. [27] analyzed the accessibility of E-government Web sites in Spain and all countries in South America using automated evaluation tools. The authors evaluated three official Web sites from each country. As a result, they reported that the majority of E-government Web sites in South America did not meet the minimum requirements for web accessibility. In addition, the Web site of the Parliament of Argentina and the Parliament of Venezuela had the lowest accessibility levels, whereas the best result was obtained from the Web sites of the Parliament of Brazil and the Government of Guyana. The authors also reported that the online test tool gave warnings and errors during the evaluation of all Web sites. [24] evaluated the accessibility of 50 randomly selected federal government Web sites in the United States of America (US). The Truwex online software tool was used to analyze the compliance of these Web sites with Section 508, WCAG 1.0 and 2.0 standards and the Children’s Online Privacy Protection Act. The results showed that the majority of the government Web sites did not follow the legal guidelines. In another study, the accessibility of higher education institutions in the US was assessed using the Bobby automated evaluation tool [15]. According to the results, 50 % of these Web sites passed the Priority 1 accessibility checkpoint, 8 % passed Priority 1 and 2; and 8 % met the criteria specified for all priority levels. In addition, 33 % of the Web sites were found not to comply with the W3C guidelines. Considering the results of these two US studies, it is important to note that despite the presence of a legal regulation (Section 508) requiring the federal departments and agencies to meet the minimum standards, at least some of the analyzed Web sites failed to achieve this.
3 Review of available tools One of the ways to reduce the time and effort while performing an evaluation of accessibility and web performance is to use semi-automated tools, that helps web developers to identify accessibility and quality issues. On the W3C-Web Accessibility Initiative (WAI) Web site, a list of tools for web accessibility evaluation can be found
989
[2]. Some of these tools evaluate Web sites for compliance with most of the accessibility checkpoints, while others are more specific tools that evaluate specific topics such as the accessibility color wheel by Giacomo Mazzocato (available at http://gmazzocato.altervista.org/colorwheel/wheel. php) and ColorTester by Alfasado Inc. (available at http:// alfasado.net/apps/colortester.html) which evaluate the choice of a text-background color pair. Tools also differ by service type and report format style as well as by license type. Services are available as plugins (Accessibility Developer Tools by Google Accessibility at https://chrome.google.com/webstore/detail/accessibilitydeveloper-t/fpkknkljclfencbdbgkenhalefipecmb?hl=en and Accessibility Bookmarklets by University of Illinois and Pixo at http://accessibility-bookmarklets.org/), as online services (A-checker by Adaptive Technology Resource Centre at http://achecker.ca and Vamola` tool by Regione Emilia-Romagna at http://www.validatore.it) or as a software (Contrast-A by Das Plankton at http://dasplankton.de/ ContrastA/ and DaSilva by Acessibilidade Brasil at http:// www.dasilva.org.br). Semi-automatic evaluation is also common in usability tests [44] and web performance. In the study by [7], a survey of usability tools that analyze Web sites was presented. Since the quality of a Web site can be determined by performance factors such as speed of use and error rate, evaluating web quality can be performed by testing these attributes. PageSpeed Tool by Google (at https://devel opers.google.com/speed/pagespeed/) and Web site speedtesting tool by Pingdom (at http://tools.pingdom.com/fpt/) are some of such tools that help web developers to identify page load issues.
4 Method In this study, government web sites of the Kyrgyz Republic, Republic of Kazakhstan, the Republic of Azerbaijan and Republic of Turkey were analyzed. The research was previously conducted on 55 government Web sites of the Kyrgyz Republic on [34]. The work has been extended in the present to the analysis of web sites of the Republic of Kazakhstan, the Republic of Azerbaijan and Republic of Turkey. In addition, the overall levels in accessibility and quality characteristics of government Web sites were compared. For each of these countries, 16 different ministry Web sites were picked for analysis. In order to provide the same level of consistency, an attempt was made to choose similar ministry Web sites. The ministry Web sites selected for the study included; • •
Government, Ministry of Economy,
123
990
• • • • • • • • • • • • • •
Univ Access Inf Soc (2017) 16:987–996
Ministry Ministry Ministry Ministry Ministry Ministry Ministry Ministry Ministry Ministry Ministry Ministry Ministry Ministry
of of of of of of of of of of of of of of
Transport and Communication, Energy and Industry, Culture and Tourism, Education, Health, Investment and Development, Defense, Internal Affairs, Finance, Foreign Affairs, Justice, Agriculture, Youth and Support, Labor and Social Protection.
First, the list of ministries’ web sites was composed. Next, web sites were checked by online tools for the number of external links and referring domains using Majestic SEO extension tool, while the number of broken links was obtained for each Web site using Linkminer tool. Also, site traffic sources and ranking were tested by means of the SimilarWeb extension tool. Then, accessibility and quality in use were tested using online tools. Evaluation tools were chosen due to their availability online, and since these tools generate immediate comprehensive reports for the scope of this study. To test the overall performance of Web sites, the Pingdom speed test tool was chosen since it is available online (at http://tools.pingdom.com/fpt/) and commonly used by researchers in this domain including [28], [20] and [36]. The Pingdom tool provides a detailed report on the load time of each element of Web sites being tested such as images, CSS, JavaScripts, RSS, Flash and frames/iframes [32]. Furthermore, errors regarding accessibility were identified based on three conformance levels using the online A-checker tool developed by Adaptive Technology Resource Centre research group in University of Toronto (available at http://achecker.ca/checker/index.php). The effectiveness of A-checker has been reported in several studies including [8] and [38]. The analysis of the traffic sources and ranking of the Web sites were carried out using the SimilarWeb tool [37]. This is another reliable tool used by many researches including [16],
[50] and [51]. To check the number of broken links in the Web sites, the LinkMiner Chrome extension was used.
5 Results 5.1 General overview The total populations of the countries are given in Table 1 [30], [10], [40], [42]. According to the Statistics Committees of the countries, in 2014, in the Kyrgyz Republic, 1.8 % (i.e., 95 out of 5300) of government agencies were connected to the government computer network, 63 were public administration agencies and 8.45 % of all government agencies (448 out of 5300) and companies had a Web site [31]. In the Republic of Kazakhstan, 81.58 % (i.e., 7891 out of 9673) of government agencies were connected to the Internet with 35.38 % (3422) of government agencies and companies having Web sites [9]. According to [40] and [43], in 2014, 60.4 % of enterprises in Azerbaijan and 81.4 % of Turkish businesses interacted online with public authorities in their country. In Azerbaijan, 26.8 % searched for information, 24.4 % downloaded forms and 37.4 % uploaded completed forms. In Turkey, the figures were: 88.4 % searched for information, 69.1 % downloaded forms and 67.3 % uploaded completed forms. In Kazakhstan, 43.45 % of enterprises interacted online with public authorities as follows; 55.48 % made contact to obtain information and 44.52 % used other government online services [9]. No information was available concerning the online interaction of enterprises in the Kyrgyz Republic with government authorities. According to the E-government development index of the United Nations, in the last 11 years, the use of Information and Communication Technologies by the government increased from 0.327 to 0.4657 in the Kyrgyz Republic, from 0.362 to 0.5472 in Azerbaijan, from 0.387 to 0.7283 in Kazakhstan and from 0.506 to 0.5443 in Turkey [45], [46]. However, the global rank of government Web sites as well as their rank in the respective countries remained relatively low. Turkish Web sites have the highest rank among the analyzed Web sites. Even taking the population sizes of countries into account, the
Table 1 Population, E-government development index and rank of countries Country
Population (thousands)
E-government development index (2014)
E-government development index rank (out of 193)
Kyrgyz Republic
5895.1
0.4657
101
Republic of Azerbaijan
9593.0
0.5472
68
Republic of Kazakhstan
17,417.7
0.7283
28
Republic of Turkey
78,741.0
0.5443
71
123
Univ Access Inf Soc (2017) 16:987–996
991
Table 2 Average rank and engagements of Web sites by country Country
Global rank
Kyrgyz Republic
2283,038
3837
9000
3.38
4.0
49.43
Republic of Azerbaijan
2218,149
11,345
44,133
3.45
3.25
49.59
Republic of Kazakhstan
674,077
75,142
45,909
3.33
4.5
50.02
87,101
2585
2935,000
6.54
6.6
39.56
Republic of Turkey
Rank within country
Table 3 Traffic sources by countries Country
Direct visits
Search visits
Kyrgyz Republic
23.29
65.22
Republic of Azerbaijan
23.32
62.45
Republic of Kazakhstan
19.18
63.02
Republic of Turkey
29.38
51.3
estimated visits to Turkish Web sites are higher at 3.73 % than the other countries. In Azerbaijan and Kazakhstan, this figure is 0.46 and 0.26 %, respectively, and the lowest rate is in the Kyrgyz Republic at 0.15 %. In addition, the average time spent by users on Turkish Web sites is higher with 6.54 min, while the average view time of other Web sites is\4 min. A similar result was obtained regarding the bounce rate; in 39.56 % of cases, users navigate away from the ministry Web sites of Turkey after viewing only one page while in the remaining three countries, this rate is about 50 % (Table 2). In terms of Web site traffic, the percentage of direct visits was found to be highest (around 30 %) in Turkish ministry Web sites, while in the other countries, more than 60 % of the traffic came from search engines (Table 3). 5.2 Accessibility The government Web sites of the Kyrgyz Republic and the republics of Azerbaijan, Kazakhstan and Turkey were analyzed for errors, which were then categorized into three levels of conformance checkpoints based on their impact on accessibility. According to the Web Content Accessibility Guidelines (WCAG), the first conformance level is addressed as a necessity, the second as a suggestion and the third as a recommendation. If checkpoints are violated for conformance levels, one or more groups will find it hard or impossible to access information in the document. The levels are organized based on success criteria, that is, on the impact they have on design or visual presentation of Web sites and can be explained as follows: Level A is the most basic web accessibility features: ‘‘For Level A conformance, the Web page satisfies all the Level A Success Criteria, or a conforming alternate version is provided.’’
Estimated visits
Time spent on site
Page views
Bounce rate
Level AA is about the biggest and most common barriers for disabled users: ‘‘For Level AA conformance, the Web page satisfies all the Level A and Level AA Success Criteria, or a Level AA conforming alternate version is provided.’’ Level AAA is the highest level of web accessibility: ‘‘For Level AAA conformance, the Web page satisfies all the Level A, Level AA and Level AAA Success Criteria, or a Level AAA conforming alternate version is provided’’ [48]. For the accessibility analysis, A-checker, a web service tool developed by a research group in University of Toronto was used [http://achecker.ca/checker/index.php]. This tool allows researchers to automatically evaluate the accessibility of Web sites based on WCAG 2.0. The analysis was performed on 63 government Web sites from the four countries; however, two web pages could not be analyzed due to either being unavailable or causing error 504 on the server side. As a result of the analysis of the available Web sites, the average number of accessibility issues of conformance level A was found to be 38 in the Kyrgyz Republic, 87 in the Republic of Azerbaijan, 34 in the Republic of Kazakhstan and 27 in the Republic of Turkey (Table 4). This demonstrates that the government Web sites of the Republic of Turkey had the least average number of conformance level A errors, though average number of conformance level AAA errors. Government Web sites of the Republic of Azerbaijan had the highest number of conformance level A errors, although the least of conformance level AAA. Errors at checkpoint 1.1.1 were due to not providing a text equivalent for all non-text objects such as images, buttons, graphs and animations. Many government Web sites of the Republic of Azerbaijan violated this checkpoint, which is one of the most important and fundamental requirements of accessible web design. The government Web sites of the Republic of Turkey and Republic of Kazakhstan provided a text equivalent for more non-text objects and thus violated checkpoint 1.1.1 less. Many government Web sites of the four countries violated checkpoint 1.3.1, which is directly related to Cascading Style Sheets. Checkpoint 2.4.4, which states that all link texts must be meaningful when read alone, was also violated more often in government Web sites of the Republic
123
992
Univ Access Inf Soc (2017) 16:987–996
Table 4 Average number of errors by country
Country
Average number of errors of Conformance level A
Conformance level AA
Conformance level AAA
Kyrgyz Republic
38
7
25
Republic of Azerbaijan
87
3
19
Republic of Kazakhstan
34
8
23
Republic of Turkey
27
7
39
Table 5 Accessibility checkpoints violated in Web sites by country Checkpoints
Kyrgyz Republic
Republic of Azerbaijan
Republic of Kazakhstan
Republic of Turkey
1.1.1
402
1134
286
277
1.3.1
51
101
16
20
1.4.1
1
1
0
0
2.1.1
0
23
0
31
2.2.1
0
0
1
0
2.2.2
0
3
1
0
2.4.2
2
19
1
1
2.4.4
38
47
42
32
3.1.1
12
19
12
18
3.2.2 3.3.1
3 0
0 0
0 2
3 0
3.3.2
20
37
12
5
4.1.1
3
5
3
4
Conformance level A
Conformance level AA 1.4.3
13
0
98
0
1.4.4
77
45
28
94
2.4.6
2
8
7
12
348
298
360
589
Conformance level AAA 1.4.6
of Azerbaijan than in Web sites of other countries. Many Web sites in all four countries also fail to meet the requirements of checkpoints 3.3.2 (each input element must have only one associated label). Checkpoint 3.1.1 was about identifying the primary natural language of a document. The results of the study showed that almost one-third of the Web sites (19 out of 61 tested Web sites) failed to identify the primary natural language of documents. At conformance level AA, many government Web sites of the four countries violated checkpoints 1.4.3 and 1.4.4, which are directly related to distinguishing a visual element such as use of the bold element, which is not handled very well by screen readers, size and contrast of text elements, and checkpoint 2.4.6, related to the inappropriate heading and label use [48] (Table 5). The tests for compliance with WCAG 2.0 showed that checkpoint 3.2.2 was violated in Web sites of the Republic
123
of Kyrgyzstan and Republic of Turkey three times each. According to this checkpoint, changing the setting of any user interface component should not automatically cause a change of context unless the user has been advised of the behavior before using the component. Checkpoint 1.4.6 at conformance level AAA was about visual presentation of text and images of text. According to this checkpoint, texts or images of text should have a contrast ratio of at least 7:1. Regarding this checkpoint, the Republic of Turkey had the highest number of errors, whereas the Azerbaijani Web sites had the lowest. In brief, all the government Web sites of the four countries need to give higher conformance level to accessible web page design since they all violate most of the analyzed checkpoints. According to the results, the Kazakh Web sites have less accessibility errors compared to the other three countries.
Univ Access Inf Soc (2017) 16:987–996
993
5.3 Quality in use
Table 7 presents the results of the broken link analysis. These results show that the government Web sites of the Republic of Turkey had the highest average number of broken links (24); however, this can be attributed to one particular Web site having 333 broken links. If this site was not considered, the average would be 3. In addition, 18.25 % of the Web sites had no broken links and 50 % had less than 5. The average number of broken links in the Azerbaijani Web sites was 19; however, there was also an outlier and a single Web site with 262 broken links. In addition, 25 % had no broken links and 62.5 % had less than 5. The analysis of the Kazakh and Kyrgyz Web sites revealed an average of 4.6 and 3 broken links, respectively. In addition, the percentage of Web sites with no broken links among the Kazakh and Kyrgyz Web sites was 8.33 and 20, respectively. The number of broken links was less than 5 in 33.34 and 80 % of the government Web sites in the Republic of Kazakhstan and Kyrgyz Republic, respectively. The analysis of errors on the Web sites showed that 6.25 % of the Turkish Web sites had connection-related errors while in Kazakh, Kyrgyz and Azerbaijani Web sites; this rate was 23.08, 30.76 and 31.25 %, respectively. Thus, in the Kyrgyz Republic, the load time exceeded 5 s in 40 % of the ministry Web sites, 80 % had broken links and the average performance grade was calculated as 79.53 out of 100; in the Republic of Azerbaijan, 94.75 % had a load time duration of less than 5 s, 75 % had broken links and the average performance grade was 80.18. In Turkey, 50 % of the ministry Web sites loaded in less than 5 s, 81.75 % had broken links and the average performance grade was 80.13. In Kazakhstan, the percentage of Web sites with a load speed of less than 5 s was 30.75, and 91.67
In this analysis, the size of the Web sites, the number of images, and scripts and load time were analyzed. Data were obtained using Pingdom Web site Speed Test tool, which is available at https://www.pingdom.com/. First, the average load time of each Web site was determined by country. The results presented in Table 6 show that the government Web sites of the Republic of Azerbaijan perform better in terms of speed, having an average load time of 2.42 s with the maximum being 6.17 s. The Turkish government Web sites had an average load time of 4.36 s; however, the highest load time was lower than the load time of the Kazakh Web sites. However, the Turkish Web site with the highest load time was found to be an outlier. In the government Web sites of the Kyrgyz Republic, the load speed was lower. According to the results of the 16 government Web sites of the Republic of Azerbaijan, 94.75 % had a load time duration of less than 5 s, and 50, 40 and 30.76 % in the Turkish, Kyrgyz and Kazakh Web sites, respectively. None of the Web sites loaded in more than 15 s, except for the 10 % of the government Web sites in the Kyrgyz Republic, which took more than 15 s to load. The slower load speed of the Kyrgyz Web sites can be explained by their relatively bigger size. For example, the Kyrgyz Web sites had an average number of 100 images and 50 % of the Web sites contained more than 70 images. However, in the government Web sites of the Republic of Kazakhstan, the average number of images was 91 and only 31.25 % of the Web sites contained more than 100 images. The Turkish and Azerbaijani Web sites had an average of 64 and 53 images, respectively. Table 6 Size and load time of Web sites by country
Country
Size (bytes) Min
Load time (s) Average
Max
Min
Average
Max
Kyrgyz Republic
62,9146
3229,614
8074,035
2.06
13
60
Republic of Azerbaijan
227,541
2330,264
1,1429,478
1.24
2.42
6.17
Republic of Kazakhstan
1782,580
3702,280
7235,175
3.1
6.27
11.36
Republic of Turkey
1048,576
2936,013
10,276,045
1.38
4.36
12.02
Table 7 Number of broken links, errors and performance grade of Web sites by country Country
Number of errors
Number of broken links Min
Average
Performance grade (%) \70
Max
Between 71 and 80
Between 81 and 90
Between 91 and 100
Kyrgyz Republic
0
3
12
5
15.38
23.08
53.85
7.69
Republic of Azerbaijan
0
19
262
7
12.5
31.25
56.25
0
Republic of Kazakhstan
0
4.6
11
4
7.69
61.54
23.08
7.69
Republic of Turkey
0
24
333
7
18.75
31.25
37.50
12.50
123
994
were found to have broken links, resulting in an average performance grade of 77.92. Based on the overall results, it can be concluded that web performance is given medium priority on the ministry Web sites of all four countries, with Azerbaijani Web sites having the best performance. However, according to the Pingdom tool, one of the Web sites of the Republic of Turkey had the highest performance grade (95 points) among all the tested Web sites and three Web sites (12.5 %) scored higher than 90 out of 100. The Kyrgyz and Kazakh Web sites had one Web site each with a performance grade above 90. The majority of Kyrgyz, Azerbaijani and Turkish Web sites scored between 81 and 90.
6 Discussions and conclusion In this study, we analyzed the government Web sites of four countries namely the Kyrgyz Republic, the Republic of Azerbaijan, the Republic of Kazakhstan and the Republic of Turkey in terms of accessibility and quality in use. The tests on search engine ranking for the ministry Web sites showed that the Web sites of Turkish ministries are better presented on the Internet; of all Web sites analyzed, users spend more time and less frequently navigate away from the Web sites of Turkish ministries. Lower rates were observed for the Kazakh ministries’ Web sites, whereas Azerbaijani and Kyrgyz Web sites have lower visibility. This result is interesting since, according to the E-government development research of the United Nations, Kazakhstan has the highest usage rate of information and communication technologies (ICT) among these four countries [46]. According to the results on accessibility tests, of the 14 Web sites in the Kyrgyz Republic and 16 Web sites in the Republic of Kazakhstan, only three Web sites in Kyrgyzstan and five Web sites in Kazakhstan attained conformance level A, whereas in the Republics of Azerbaijan and Turkey, none of Web sites succeeded in passing accessibility test for conformance level AAA. Furthermore, the analysis of distribution of error showed that in the Azerbaijani Web sites, accessibility checkpoints were violated more often than in other countries’ Web sites. Therefore, in designing government Web sites, the developers need to pay more attention to guidelines such as WCAG 2.0 to achieve higher accessibility. In terms of the quality in use, government Web sites in the Republic of Azerbaijan have higher load speed and less broken links and higher performance score. They are followed by Web sites of the Kyrgyz Republic, Republic of Turkey and Republic of Kazakhstan. The results indicate that the usage rate of ICT by the government is higher in Turkey, where, in addition to Web sites, applications on
123
Univ Access Inf Soc (2017) 16:987–996
GooglePlay and AppStore are available for approximately 60 % of government services. In the Republic of Kazakhstan, only two ministries have applications available via these stores whereas none of the Azerbaijani and Kyrgyz ministries offers such services. Based on the results of the study, it can be concluded that Web sites performance influencing end users’ expectation is not a great concern for the government Web sites of the four countries studied since the grade of all Web sites was found relatively high. However, there is still a need for significant effort to achieve higher accessibility, since, as it can be seen; many government Web sites violate accessibility principles. In addition, the optimization of search engine results for the ministry Web sites should also be improved since the visibility of the ministry Web sites of the Kyrgyz Republic, the Republic of Azerbaijan and the Republic of Kazakhstan is relatively low. In the context of this study, tests utilizing online evaluation tools were carried out. Although semi-automatic evaluation tools can check a Web site for a wide range of errors, the expectations of end users play a vital role in Web site evaluation. Thus, the main limitation of this study is that only semi-automatic tools evaluation was performed. This formulation of the problem focuses on the need to develop integrated methods and tools for evaluating the quality of software products for convenience and efficiency. Since these two methods are complementary to each other, as a subsequent step, user experience tests will be conducted in the context of future work. For the analysis of accessibility and usability, target groups from each country will be selected. Accessibility analysis will be performed on location with users with visual impairments.
References 1. Abanumy, A., Al-Badi, A., Mayhew, P.: e-Government website accessibility: in-depth evaluation of Saudi Arabia and Oman. Electron. J. e-Gov. 3(3), 99–106 (2005) 2. Abou-Zahra, S.: Web accessibility evaluation tools: overview. http://www.w3.org/WAI/ER/tools/ (2006) 3. AbuAli, A.N., Obedidat, A., Abu-Addose, H.Y.: Accessibility as an indicator of Jordanian e-Government website quality. In: 4th International Conference on e-Learning Best Practices in Management, Design and Development of e-Courses: Standards of Excellence and Creativity, 156–160. (2013) 4. Akgul, Y., Vatansever, K.: Web accessibility evaluation of government websites for people with disabilities in Turkey. J. Adv. Manag. Sci. 4(3), 201–210 (2016) 5. Al-Radaideh, M., Nuser, M., Wahbeh, A.: Evaluating accessibility of Jordanian E-Government websites for people with disabilities. In: Proceedings of International Conference on Information and Communication Systems, 127–131. (2011) 6. Baker, D.L.: Advancing E-Government performance in the United States through enhanced usability benchmarks. Gov. Inf. Quart. 26, 82–88 (2009)
Univ Access Inf Soc (2017) 16:987–996 7. Brajnik, G.: Automatic web usability evaluation: what needs to be done? In: Proceedings Human Factors and the Web, 6th Conference. (2000) 8. Brooks, L., Persaud, A.: Comparing local e-Government websites in Canada and the UK. In: Electronic Government pp. 291–304. Springer International Publishing. (2015) 9. Committee on Statistics of the Ministry of National Economy of the Republic of Kazakhstan. Key indicators of the governance organizations’ ICT usage by economic activity. Report at http://stat.gov.kz/ getImg?id=ESTAT100275 (2016). Accessed 10 February 2016) 10. Committee on Statistics of the Ministry of National Economy of the Republic of Kazakhstan. Refinement of the population size in the Republic of Kazakhstan at the beginning of 2009-2015. Report at http://stat.gov.kz/getImg?id=ESTAT100234 (accessed February, 10, 2016) 11. Convention of the Republic of Turkey ‘‘On the Rights of Persons with Disabilities’’ from 29/09/2009 at http://www.uhdigm.adalet. gov.tr/sozlesmeler/coktaraflisoz/bm/bm_48.pdf (2016). Retrieved 2 May 2016) 12. Fang, X., Holsapple, C.W.: An empirical study of web site navigation structures’ impacts on web site usability. Decis. Support Syst. 43, 476–491 (2007) 13. Federal Law of the Republic of Kazakhstan ‘‘On ratification of the UN Convention on the Rights of Persons with Disabilities’’ from 25.04.2012 at http://egov.kz/wps/portal/!ut/p/b0/04_Sj9CPy kssy0xPLMnMz0vMAfIjc7PyChKtUvKTS3NT80r0w_Wj9KNg PM8U_cgoQ1MDEDCy sABJ5STmpZcmpqfqRxaV6hfk5lqUOyoqAgACDrXa/ (2016). Retrieved 2 May 2016) 14. Garcia, A.C.B., Maciel, C., Pinto, F.B.: A quality inspection method to evaluate E-Government sites. EGOV 2005, LNCS 3591, 198–209, (2005) 15. Harper, K.A., DeWaters, J.: A Quest for website accessibility in higher education institutions. Internet High. Educ. 11, 160–164 (2008) 16. Homola, G.A., Ho¨ltje, J., Bru¨ning, R.: Analyse zur Internetpra¨senz der Deutschen Gesellschaft fu¨r Neuroradiologie. Clin. Neuroradiol. 25(2), 137–141 (2015) 17. Isa, W.A.R.W.M., Suhami, M.R., Safie, N.I., Semsudin, S.S.: Assessing the usability and accessibility of Malaysia E-Government website. Am. J. Econ. Bus. Adm. 3(1), 40–46 (2011) 18. ISO/IEC 40500: 2012 Information technology—W3C web content accessibility guidelines (WCAG) 2.0. from http://www.iso. org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber =58625 Retrieved 11 March 2016 19. Karkin, N., Janssen, M.: Evaluating websites from a public value perspective: a review of Turkish local government websites. Int. J. Inf. Manag. 34, 351–363 (2014) 20. Kolla´r, I., Kra´l’, P., Laco, P. Methodology for assessing website improvement in corporate environment. Applications of Mathematics and Statistics in Economics. In: 17th International Scientific Conference, Poland, pp. 131–140, August 2014 21. Kopacka, H., Michalek, K.: Accessibility and findability of local e-government websites in the Czech Republic. Univ. Access Inf. Soc. 9, 51–61 (2010) 22. Kurt, S.: The accessibility of university web sites: the case of Turkish universities. Univ. Access Inf. Soc. 10, 101–110 (2011) 23. Kuzma, J.M.: Accessibility design issues with UK e-government sites. Gov. Inf. Quart. 27, 141–146 (2010) 24. Kuzma, J.M., Weisenborn, G., Philippe, T., Gabel, A., Dolechek, R.: Analysis of U.S. Senate web sites for disability accessibility. Int. J. Bus. Res. 9(6), 174–181 (2009) 25. Latif, M.H.A., Masrek, M.N.: Accessibility Evaluation on Malaysian E-Government Websites. J. e-Gov. Stud. Best Pract. 2010(2010).http://ibima.net/articles/JEGSBP/2010/935272/935272. pdf
995 26. Lee, S., Koubek, R.J.: The effects of usability and web design attributes on user preference for e-commerce web sites. Comput. Ind. 61, 329–341 (2010) 27. Luja´n-Mora, S., Navarrete R., Pen˜afiel, M. (2014). Egovernment and web accessibility in South America. In: Proceedings of the 2014 First International Conference on eDemocracy and eGovernment (ICEDEG), pp. 77–82 28. Nagy, Z.: Improved speed on intelligent web sites. In: Recent Advances in Computer Science, pp. 215–220. Rhodes Island, Greece (2013) 29. Nam, T.: Determining the type of e-government use. Gov. Inf. Quart. 31, 211–220 (2014) 30. National Statistical Committee. 26.06.2015/Information and Communication Technologies in Kyrgyz Republic 2010–2014; http://stat.kg/media/publicationarchive/fdd015ba-8a4f-47ad-95db -fcfc47d6a7f1.pdf. Accessed 10 February 2016) 31. National Statistical Committee. 26.06.2015/Resident population as of the beginning of the year; http://stat.kg/media/statisticsdy namic/fe55b8f0-5e6e-4c9b-b9aa-2e34ce97516d.xlsx. Accessed 10 February 2016) 32. Patel, M.C., Gulati, R.: Software performance testing tools—a comparative analysis. Int. J. Eng. Res. Dev, 3(9), 58–61 (2012) 33. Patra, M.R., Dash, A.R., Mishra, P.K.: A quantitative analysis of WCAG 2.0 compliance for some Indian web portals. Int. J. Comput. Sci. Eng. Appl. (IJCSEA) 4(1), 9–24 (2014) 34. Ismailova, R.: Website accessibility, usability and security: a survey of government websites in Kyrgyz Republic. Univ. Access Inf. Soc. (2015). doi:10.1007/s10209-015-0446-8 35. Resilution on the set of measures to ensure the rights and improve the quality of life of persons with disabilities in the Kyrgyz Republic for 2014–2017, 2 December 2013 at http://cbd.minjust. gov.kg/act/view/ru-ru/94852. Retrieved May 2 2016 36. Shawgi, E., Noureldien, N.A.: Usability measurement model (umm): a new model for measuring websites usability. Int. J. Inf. Sci. 5(1), 5–13 (2015) 37. SimilarWeb.com-Competitive Intelligence Tool. (n.d.). from http://www.similarweb.com/ Retrieved 11 March 2016 38. Sohaib, O., Kang, K.: The importance of web accessibility in business to-consumer (B2C) websites. In: 22nd Australasian Software Engineering Conference (ASWEC 2013) 39. The Law of the Republic of Azerbaijan ‘‘On ratification of the UN Convention on the Rights of Persons with Disabilities’’ from 21/01/2009 at http://www.justice.gov.az/view_hr.php?id=19. Retrieved 2 May 2016 40. The State Statistical Committee of the Republic of Azerbaijan. Information Society in Azerbaijan. Information and Communication Technologies Statistical Yearbook, 2015. Report at http:// www.stat.gov.az/menu/6/statistical_yearbooks/source/i-society_ en_2015.zip. Accessed 10 February 2016) 41. The United Nations: Convention on the rights of persons with disabilities. Treaty Ser. 2515, 3 (2006) 42. Turkish Statistical Institute. Enterprises using Internet for interaction with public authorities by economic activity (NACE Rev.2) and size group. Report at http://www.turkstat.gov.tr/Pre IstatistikTablo.do?istab_id=2346. Accessed 10 February 2016) 43. Turkish Statistical Institute. Population by years, age group and sex, 1935–2015. Report at http://www.turkstat.gov.tr/UstMenu. do?metod=temelist# Accessed 10 February 2016 44. U.S. Dept. of Health and Human Services.: Research-based web design and usability guidelines. US Government Printing Office at http://www.usability.gov/guidelines/guidelines_book.pdf(2006). Retrieved 15 December 2015 45. United Nations.: UN global E-government Survey 2003, [Online], United Nations Division for Public Administration and Development Management, http://unpan1.un.org/intradoc/
123
996 groups/public/documents/un/unpan016066.pdf (2003). Retrieved 25 May 2015) 46. United Nations.: UN E-government Survey 2014—E-government for the Future We Want, [Online] http://unpan3.un.org/egovkb/ Portals/egovkb/Documents/un/2014-Survey/E-Gov_Complete_ Survey-2014.pdf(2014). Retrieved 25 May 2015 47. Wang, L., Bretschneider, S, Gant, J.: Evaluating Web-based e-government services with a citizen-centric approach.In: Proceedings of the 38th Hawaii International Conference on System Sciences. (2005)
123
Univ Access Inf Soc (2017) 16:987–996 48. Web Content Accessibility Guidelines 2.0 at https://www.w3.org/ TR/WCAG20/ Retrieved 5 April 2016) 49. Yuan, L., Xi, C., Xiaoyi, W.: Evaluating the readiness of government portal websites in China to adopt contemporary public administration principles. Gov. Inf. Quart. 29, 403–412 (2012) 50. Zhang, M.: Internet use that reproduces educational inequalities: evidence from big data. Comput. Educ. 86, 212–223 (2015) 51. Zhang, M.: Discovering the unequal interest in popular online educational games and its implications: a case study. Br. J. Educ. Technol. 47(2), 358–371 (2016)
