Testing of Web Map Services - CiteSeerX

Article under Review for the International Journal of Spatial Data Infrastructures Research, Special Issue GSDI-11, submitted 2009-03-26

Testing of Web Map Services∗ Jiří Horák, Jiří Ardielli, Bronislava Horáková Institute of Geoinformatics, VŠB – Technical University of Ostrava, 17. listopadu 15, 70830, Ostrava-Poruba, Czech Republic {[email protected]} {[email protected]} {[email protected]}

Abstract Network services represent a key part of INSPIRE. They shall be available to the public and accessible via the Internet. The paper proposes a way how to test and measure the availability of network view services (WMS) for end users. Map servers of two main providers of Czech spatial network services according INSPIRE (CENIA, Czech Environmental Information Agency, and COSMC, Czech Office for Surveying, Mapping and Cadastre) has been tested using performance and load testing. The testing helps to understand how the system will handle the load caused by many concurrent users. Any provider is able to test a latency of application, but for end user it is important to measure the overall latency (the response time) and also other characteristics like error occurrence, availability and performance. The results also enable to describe stress capabilities, which mean finding the breakpoint of a site/web-based application performance against the maximum user load. The response time, error occurrence, availability and performance were measured during two months of repeated access using same scenarios and different number of concurrent virtual users. Better performance and benchmarking were obtained testing of COSMC map services, where the average response time is bellow 1 s, the overall performance is stable and no stress breakpoint is detected (for the given maximum number of virtual users). Keywords: WMS, INSPIRE, performance, load testing. 1.

INTRODUCTION

At the beginning a company arises a set of questions concerning exploitation of INSPIRE in its business. Will the services be really available and reliable? May they be integrated into the business processes? May frequently used data themes (e.g. topographical background) be outsourced from the local spatial ∗

This work is licensed under the Creative Commons Attribution-Noncommercial Works 3.0 License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/ or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA.


database? Will the accessibility and satisfactory performance be guaranteed for end users – as well as at peak time? Will ISO 9001 be also applied for the SDI national providers? These typical questions stress the end user’s interest in utilization of the public, executive-driven services. Network services represent a key part of INSPIRE. They should be available to the public and accessible via the Internet. The regulation of network services establishes the obligatory quality criteria which has to be checked and achieved. These criteria are measured on the server side, which is important for service providers. But how will the criteria be transformed into a satisfaction of end users? The end user’s requests and its ITC conditions may strongly vary. Also the satisfaction of an end user is a subjective term. Nevertheless, this is the goal of any public service and SDI development. We have to ask if the established criteria for the service providers are sufficient to meet the end user’s criteria and to assure the satisfaction for most of the users. The performance of network services on the client side is influenced by many factors. It is obvious that building the SDI has to be performed together with information infrastructure development. The complex evaluation of end user’s conditions and prediction of its satisfaction should utilize both content driven and function capability measurements as well as quantitative measurements which control the service accessibility and performance. These IT aspects of services are vital for end users. They can be measured by the overall latency (a measurement of the time it takes application to finish processing a request) and also other by characteristics like concurrency (measurement of the influence when more the one user operates webapplication), availability (a measurement of the time a web-enabled application is available to take request) and performance (average of the amount of time that passes between failures). 2.

QUALITY OF SERVICE

The evaluation of the SDI and its components, namely geoportals, usually starts considering the cost savings/cost reduction objective, but in recently the evaluation and comparison has been based on control-driven performance measurement systems focusing on inputs and outputs. We can speak about SDI evaluation institutionalisation. This is an important change and also an opportunity for research (Lance et al., 2006). Different approaches can be found for the comparison and the evaluation of geoportals. The socio-political and economic issues affecting the implementation of geoportals within five European countries (France, Germany, Spain, England


and Norway) were investigated in the study of Giff et al. (2009). The comparison of provided services (it represents a function capability description) and the type of coordinating body show the differences of the service level and also different organisational structures. The description of provided services represents usual user oriented testing/evaluation. Contrary, the type of coordinating body does not affect any user directly, but it influences the infrastructure design, relationships, stability and an applied economical model. The importance of SDI functionalities was also underlined by Erik de Man (2007). Such content driven and function capability description/evaluation requires to be completed with a technical evaluation to assure the availability and the performance of the service. Poor (technical) quality of service translates into frustrated customers, which can lead to lost business opportunities (Menascé, 2002). This kind of testing may be based only on SW testing with no respect to spatial features of applications or take them into account e.g. measuring velocity of selected spatial operations, rendering etc. Quantitative measurements used in SW testing offer easy quantification, better standardization (for many aspects some industry or official standards of IT can be applied), repeatability, possibility of automated and long-term processing and evaluation, understandability (Mendes et al. 2006). Wilson (1989) noted that the observability and measurability of outputs are vital supports for understandability by non-expert ‘outsiders’. Usually executives are driven to use such measurements (Lance et al., 2006). If we decide to apply quantitative measurements, then what testing methods would be available? Software testing methods are traditionally divided into black box testing and white box testing (http://en.wikipedia.org/wiki/Software_testing). Black box testing treats the software as a black box without any knowledge of internal implementation. White box testing utilizes the knowledge about the algorithm implementation. The advantage of black box testing is that we are not constrained by the programmer’s approach and it is more probable to detect bugs. Unfortunately, black box testing may cause a situation where some parts of the SW are not tested at all, while other parts are redundantly tested. Concerning analysis of web client-server systems it is needed to distinguish server-side and client-side analysis. Server-side testing can utilize clickstream analysis, web server log files, de-spidering the web log file and exploratory data analysis (Markov et al., 2007). Recommended metrics include Number of Visit Actions, Session Duration, Relationship between Visit Actions and Session Duration, Average Time per Page, Duration for Individual Pages. It is recommended to measure results for individual pages and then aggregated results, otherwise metrics will be distorted due to the processing of mixture of both navigation pages and content pages.


The server side testing can also be utilized to explore a dependency between complexity of the picture and the velocity of image rendering and delivery. The server side testing may lead to improvement of the service using suitable preprocessing (e.g. FastCGI for University Minessota Map Server, Brock 2005), tiling/caching (Liu, 2006) or load balancing (Pancheng et al., 2004). Hicks et al. (1997) stresses potential benefits of client-side automated testing and distinguishes following 5 basic types: Test precondition data, Functional testing, Stress test functionality, Capacity testing and Load and performance testing. The most frequent form of testing is load testing (alternative terms are performance testing, reliability testing or volume testing). Load testing generally refers to the practice of modelling the expected usage of a software program by simulating multiple users accessing the program concurrently (http://en.wikipedia.org/wiki/Load_test). When the load placed on the system is raised beyond normal usage patterns, in order to test the system's response at unusually high or peak loads, it is known as stress testing. The load is usually so great that error conditions are the expected result, although no clear boundary exists when an activity ceases to be a load test and becomes a stress test. Any good load testing has to be based on realistic usage patterns. In the case of web site testing, the load generator has to mimic browser behaviour. It usually means to submit a request to the Web site, wait for a period of time after the site sends a reply to the request (the think time), and then submits a new request (Menascé, 2002). The load generator can emulate thousands of concurrent users to test web site scalability. Each emulated browser is called a virtual user. The virtual users’ behaviour must imitate the behaviour of actual real users (follow similar patterns, use realistic think times, and react like frustrated users, abandoning a web session if response time is excessive). What testing results are required to be described and studied? Time delay Time delay can be described by latency or by response time. Latency is a time delay between the moment something is initiated, and the moment one of its effects begins or becomes detectable1 Instead of latency a response time is usually used. The response time can be specified as the time taken for the application to complete any given request from the end user. However, to be complete, any such metric must account for the number of users that are likely to be interacting with the application at any given time (Brown et al. 2005). Especially for web services it is necessary to understand and decompose the response time better. Then the response time represents a summary of network 1

http://en.wikipedia.org/wiki/Latency_(engineering))


time, messaging time and service time (Hyung et al., 2005). Network Time is the amount of delay determined by bandwidth of network path between customers and the providers of web services, network traffic and performance of network equipments. Messaging Time is the amount of time taken by service providers to process SOAP messages. The time to process SOAP messages cannot be neglected because SOAP is an XML-based protocol and the size of the exchanging message is usually bigger than other binary-based protocols. Service Time is the amount of time for a web service to perform its designated task. It depends on efficiency of business logic, hardware capability, framework for web services and/or operating system of web services. Error occurrence Error occurrence can be described using simple average of the amount of time that passes between failures. Another possibility is to evaluate percentage of errors occurred during the session or some iteration of requests. Availability Availability measures the percentage of time customers can access a Web-based application (Menascé, 2002). Availability goals typically vary according to the application type. Critical applications require the best availability conditions. Geoportals as an important part of SDI should be a reliable source of data and services and that is why the requested availability is set up to 99% of the time (INSPIRE, 2008). 3.

INSPIRE NETWORK SERVICES

Directive 2007/2/EC of the Council and the European Parliament establishes the legal framework for setting up and operating an Infrastructure for Spatial Information in Europe (INSPIRE) based on infrastructures for spatial information established and operated by the member states. The purpose of such infrastructure is, in the first instance, to support the formulation, implementation, monitoring, and evaluation of Community environmental policies, and to overcome major barriers still affecting the availability and accessibility of pertinent data (Craglia et al., 2009). To assure the availability and accessibility of data a Regulation for Network Services has been adopted by the Commission of the European Communities. This Regulation sets out the requirements for the establishment and maintenance of the Network Services provided for in Article 11(1) of Directive 2007/2/EC and obligations related to the availability of those services to the public authorities of the Member States and third parties pursuant to Article 12 of that Directive. This Regulation sets out the Requirements for Network Services (Article 3) and the Access to the Network Services (Article 4).


According to Article 3 of this Regulation the Network Services shall be in conformity with the requirements concerning the quality of services set out in Annex I of this Regulation. According to Annex I the following Quality of Service criteria relating to performance, capacity and availability shall be ensured (INSPIRE 2008): Performance Performance means the minimal level by which an objective is considered to be attained representing the fact how fast a request can be completed within an INSPIRE Network Service. The response time for sending the initial response to a Discovery service request shall be maximum 3 seconds in normal situation. For a 470 Kilobytes image (e.g. 800x600 pixels with a colour depth of 8 bits), the response time for sending the initial response to a Get Map Request to a view service shall be maximum 5 seconds in normal situation. Normal situation represents periods out of peak load. It is set at 90% of the time. Response time means the time measured at the Member State Service location, in which the service operation returned the first byte of the result. Because the response time is measured at the service location, the criteria can be specified as server-side measurements. Capacity Capacity means limit of the number of simultaneous service requests provided with guaranteed performance. The minimum number of served simultaneous requests to a discovery service according to the performance Quality of Service shall be 30 per second. The minimum number of served simultaneous service requests to a view service according to the performance quality of service shall be 20 per second. Availability Availability means probability that the Network Service is available. The probability of a Network Service to be available shall be 99% of the time. Third party Network Services linked pursuant to article 12 of directive 2007/2/EC, shall not be taken into account in the quality of service appraisal to avoid the potential deterioration due to the cascading effects. 4.

CZECH INSPIRE NETWORK SERVICE PROVIDERS

The transposition of the INSPIRE Directive in Czech Republic is coordinated by the Ministry of the Environment and the Ministry of Interior in collaboration with the Czech Office for Surveying, mapping and the Cadastre (COSMC). Both ministries are preparing the transposition of the INSPIRE Directive into national law. Moreover, the Czech Environmental Information Agency (CENIA) has been responsible for INSPIRE implementation since 2006. In this process eight


working groups have been established involving eight Ministries and central bodies. With a clear focus on the implementation of the INSPIRE Directive, CENIA plays a coordinating and stimulating role together with the ministries, CAGI and NEMOFORUM. The CAGI and NEMOFORUM help bringing together all the SDI stakeholders through different initiatives. The Czech Association for Geoinformation (CAGI) is a civil professional association of the individuals and legal persons working in the sphere of geoinformation in the Czech Republic (http://www.cagi.cz). NEMOFORUM is the platform for co-operation of the state administrative bodies, regional and local authorities, different professional unions and associations, representing users in the commercial sector and universities (http://www.cuzk.cz/nemoforum/). The Czech Environmental Information Agency (CENIA) performs synthetic research in ecology and environmental protection and provides professional support to public administrations in the area of integrated prevention (http://www.cenia.cz). CENIA has created and maintained a central geoportal (http://geoportal.cenia.cz) for the INSPIRE relevant data themes as a component of the Portal of the Public Administration, which in turn is under the responsibility of the Ministry of Interiors (Craglia et al., 2009). CENIA has developed several web mapping services with basic administrative, topographic and environmental data to provide all the necessary data for authorities. The services are available through the portal. The implementation emphasizes on operational capability of the server using e.g. two parallel running WMSs. The geoportal enables authorised users to upload new data sets. Discovery and view services are available free of charge in the geoportal while downloading, transformation and invoke services are available according to charging procedures. Data produced and maintained completely by state budget are generally available free of charge, but for specific uses different access conditions are regulated by a licensing framework. The geoportal offers an access to 4 terabyte of data through 90 local and 15 remote map services (fig.1). The Czech Office for Surveying, mapping and the Cadastre (COSMC) is an autonomous supreme body of the state administration of surveying, mapping and cadastre in the Czech Republic (http://www.cuzk.cz/). COSMC have been playing a major role in data production, because it is responsible for 5 Annex I, 2 Annex II, and 2 Annex III INSPIRE data themes (geographic and cadastral reference data, DTM and ortho-imagery, buildings and land use themes).


Figure 1: Image generated by WMS CENIA (tested page n.5)

In 2005 the COSMC launched a new geoportal in Czech and English languages which includes a business module, web map services (fig. 2), the GPS stations network, and geodetic control points (COSMC, 2008). COSMC metadata are currently compliant with the national and cadastral standards. Along with the process of the application of the INSPIRE Implementing Rules the transition to the ISO metadata standards is currently under development.


Figure 2: Image generated by WMS COSMC (tested page n.9)

5.

METHODOLOGY OF TESTING

The goal of the testing is to verify usability (suitable conditions) of selected network service providers for end users. This accent on end user approach requires selection of the client-side black-box testing. Even though various drawbacks of such testing exist (not completed testing of server, complex response time integrating different sources of time delays, different type of errors occurred on the way, etc.), it still represents the main way how to verify real service conditions for end users. In some cases also the independency of such testing from the service provider may be welcome. For the client-side testing we have selected three important aspects (metrics) influencing the end-user acceptance and utilisation of network service: performance, error occurrence and response time. These metrics are measured in 2 types of testing. A long-term performance testing repeats many requests with approximately the same load (the number of virtual users is not changed) and monitor the behaviour of the service using indicated metrics for a long time. A load and stress testing increases the number of concurrent virtual users and monitors the reaction of the service to the increased load. Such testing is usually short term and enables us to predict the system behaviour in the increased traffic with more concurrent real users, which may occur in close future. Also it can be utilised for evaluation of application scalability.


The testing of network services was undertaken for CENIA and COSMC (see previous chapter). The CENIA geoportal enables to specify an individual URL for every requested map window (GetMap request) (including layer selection, scale etc.). These URLs can be used for scenarios definition. In the case of COSMC geoportal individual URLs for map windows are not available, thus we had to use another application (Intergraph OGC WMS Viewer) to generate appropriate URLs. The WMS standard compliance has not been tested. The next step was to define a realistic usage patterns to mimic real user’s behaviour. The user requirements and conditions were set up in cooperation with a interested company according to its requirements for the web map services and the experience with utilization. Users specify what data themes are required and under what conditions. Users also depict known problems with service utilisation - e.g. occurrence of too long response time without recognizable reasons, data cannot be downloaded from the server at all, although the service is available. It helps to direct the testing to possible weak aspects of services. These usage experiences and demands were reflected in the construction of various scenarios with defined sequences of visited web pages (GetMap requests) and time planning. Basic layers were selected from following data themes - Administrative units, Orthophoto, Topographic maps, Natural and Landscape protected areas, Cadastral maps, Definition points of parcels and buildings. It represents a mixture of data themes from Annex I and II. Usually web pages include more than one layer. This “unequal” distribution of data themes is driven by user’s requirements. It is one of the principal differences between client side and server side testing – a client side testing has to follow users behaviour (to assure the user’s satisfaction) while server side testing should assure the provider to fulfil the declared conditions for all data themes. Scenarios for the long-term performance testing utilize iterations of requests which are repeatedly replayed every 60 minutes. Each block contains increased number of accesses to individual selected pages. In the case of CENIA portal the scenario contains 19 selected web pages, for COSMC geoportal the number of visited web pages is 10. Main settings for the long-term performance testing: • Duration: 2 months • Number of iterations: 1 to 24 • Number of virtual users: tests with 1, 3, 5 and 10 users • Interval between runs: 60 minutes • Delays between users: 10 ms


Scenarios for the load and stress testing are based on the increased number of concurrent virtual users (1-150), each user starts with 10 ms delay. The web pages for COSMC geoportal are the same as for the performance testing. The scenario for CENIA portal includes only 7 web pages. Web pages with large images were excluded from this testing due to the overloading of the local network transmission capacity with the increased number of concurrent virtual users. Main settings for the load and stress testing: • Duration: 2 weeks • Number of iterations: 4 • Number of virtual users: 1 to 150 • Interval between runs: 10 sec • Delays between users: 10 ms 6.

APPLIED SOFTWARE TOOLS

The nature of load tests facilitates an automation and repeatability of the testing. A wide range of different web Site Test Tools and Site Management Tools (more than 420 tools listed in 12 categories) is documented on http://www.softwareqatest.com/qatweb1.html, where the special category Load and Performance Test Tools includes overview of some 40 tools. Short recommendations for tools suitable for performance, load and stress testing can be found in Gheorghiu (2005). A good comparison of 8 selected load test tools can be found in Buret et al. (2003) including description of scenario preparation, execution, form of results and monitoring capability. For the indicated purpose we have selected a WAPT software v.3.0. WAPT (SoftLogica LLC) is a load and stress testing tool that provides with an easy-touse, consistent and cost-effective way of testing web sites, web servers, and intranet applications with web interfaces (http://www.loadtestingtool.com/). The application provides appropriate tools and matches our requirements. It offers transparent design of scenarios, settings of virtual users, number of iterations, interval between runs and delay between users. Outputs are organised into clear reports and graphs. Applied metrics include: • Average request processing time • Maximum request processing time • Minimum request processing time • Overall performance • Percentage of errors • Average bandwidth


•

Average, maximum and minimum request processing time for individual pages

The WAPT application is able to detect 3 modes of failures - Net error (or Socket error), HTTP errors and Timeout error. Last version of WAPT offers testing web sites with dynamic content and secure HTTPS pages. Nevertheless, we have selected the older version, because it enables to test individual pages independently. 7.

TESTING RESULTS

Before the assessment of obtained testing results several limits for evaluation should be established. The limit for acceptable response time is set to 10 seconds, because to this time the user's attention is focused on the dialogue of application (Krejcar et al., 2009). The advice to split the response time to 0.1 s (“fluent reaction”), to 1 s (a noticed delay, but no action is needed) and to 10 s (anything slower than 10 seconds needs a percent-done indicator as well as a clearly signposted way for the user to interrupt the operation (Nielsen 1994)) is used for many years. Concerning the availability aspect we can apply the limit of 99% (INSPIRE, 2008). 7.1

The long-term performance testing of CENIA geoportal

In most cases (iterations) the average web transaction (the average response time) is under 2 seconds that is a satisfactory. Such indicator does not provide us with information how often end user faces a bad situation with long response time when he can decide to abandon the session. More suitable is the maximum request processing time. The maximum web transaction (the maximum response time) shows a large variability of results. Even if we exclude the network error anomalies (issued from network error occurrences), still we record cases when the response time exceeds 10 s. In some cases also the response time around 50 s has occurred. These long time intervals have been recorded for pages including ortho-imagery and raster topographical background. Fig. 3 depicts the situation for different iterations with 10 concurrent virtual users when the internal load (number of visiting pages) continuously increases. During the abnormal iteration n.22 the maximum response time is 20 s. In one reported test only 39 records exceed 10 s from the total number of 15972 records (0.25%).


The overall performance of tested application can be evaluated as a stable one. During increased number of iterations the system performance was approximately constant (usually between 10 and 40 pages per second), for more virtual users even slightly decreasing (fig. 4). It indicates a probable insufficient cache system on the server side, because an efficient cache system for these repeatedly visited pages will provide the increased performance. Figure 3: Maximum response time


Figure 4: Overall performance

Concerning error occurrences the most frequent is network error. It appears every day, sometimes repeatedly. WAPT has no features of transient network analysers, thus it is impossible to identify the network element where the error occurs. The second type of error, the HTTP error, was rarely registered. During iterations no. 12 the occurrence of errors reaches 70% (fig. 5). Small amount of errors (around 1%) was registered for most of the iterations. Unfortunately, the variability of error occurrence is quite large and in some test runs we obtained about 30% of errors. These results cannot be a satisfactory for end users. In some cases the service returns a blank image. The reason of this behaviour is not known; according to the provider it may occur during an internal refresh of services.


Figure 5: Error occurrence

7.2

The long-term performance testing of COSMC geoportal

The average response time is usually bellow 1 s, which is excellent for end users. Only 1 page (n. 8) with requested raster layers and 2 sets of definition points indicates average access time around 3 s. The maximum response time do not exceed 10 s (except of abnormal iteration n. 16). Most of pages show the maximum time bellow 4 s, more demanding page n. 8 below 6 s (fig. 6).


Figure 6: Maximum response time

The overall performance is very stable (fig.7). The iteration n. 15 is influenced by the error occurrences; other iterations show the range of 9-12 pages per second, which is quite low.



Again the most frequent is the network error. The fig. 8 depicts error occurrences during iterations for 10 concurrent virtual users. The iteration n. 15 demonstrates extreme number of network errors (46 %), which should lead to abandoning of the session in the real world. Small amount of errors (around 1%) was registered also for many other iterations. Concerning returning results we did not register any abnormal behaviour like blank images etc.



7.3

The load and stress testing of CENIA geoportal

The average response time (fig. 9) shows the good server handling up to approx. 30 virtual users. After this limit a part of the pages demonstrates strong increase of the average response time. Page n.4 (with ortophoto) indicates much higher average response time than others. The drop of values after approx. 100 virtual users is caused by error occurrences – due to errors the service returns blank pages and declares very good access time. The result is approved by the evidence of error occurrences (fig. 10). A hundred of virtual users represent a breakpoint and thus high rates of errors have to be expected. According administrator’s statement, 100 and more concurrent virtual users cause evident slow down of services, because our 100 virtual users are joined with approx. 150 concurrent real users, thus the total number of concurrent users crosses 250. It represents the current limit of server loading.


The overall performance (fig. 11) slightly oscillates below 10 pages per second (until the breakpoint) which is low. Figure 9: Average response time





7.4

The load and stress testing of COSMC geoportal

The average response time (fig. 12) slightly grows with the increased number of virtual users. The page n.8 (the same abnormal page as for the performance testing) shows the largest average response time. The limit 10s probably exceeded only for this page, other pages would be under the limit even for the maximum number of virtual users. The server behaviour seems to be satisfying. Fig. 13 depicts the growing overall performance with the increased number of virtual users. It indicates an appropriate optimisation of the server. The maximum is reached close to 40 pages per second. No breakpoint has been detected. Figure 12: Average response time



8.

CONCLUSION

The client-side testing of web map services provides a tool how to evaluate their availability and performance for end users. The existing SW applications enable to design scenarios according to end user’ conditions and requirements and test of the relevant performance for the selected network services. WMS services of two Czech providers, CENIA and COSMC, have been tested using a long-term performance testing and load and stress testing. In case of CENIA services, the average response time is bellow 2 seconds. Quite rarely (0.25% of the cases) the responses exceed 10 s, which is regarded as a critical threshold for end users. The overall performance from the client perspective should be higher. In some cases also specific behaviour of map server sending blank images has been recorded. The stress test discovers the breakpoint of performance for 250 concurrent users. The average response time for selected pages on COSMC geoportal usually is bellow 1s, which is excellent. The overall performance in stress testing approves


good conditions reaching 40 pages per second in increased load volume. The stress test did not reveal any breakpoint of performance. Errors of services (service not available) were recorded for both providers. The error distribution is unequal, usually cumulative. The occurrence may reach 70 % during the session. Such extreme inaccessibility of the service did not last more than 2 hours. Reasons may be found also on the server side - a service restart, data uploading etc. These blackouts should be analysed in cooperation with service providers. It may substantially influence the service availability and consecutively, the user confidence and their willingness to exploit the service. The SDI building is a long-term process tightly connected with information infrastructure development. The IT quality of services should be monitored using not only internal server-side metrics but also using measurements based on client-side approaches. Also secure aspects of the services should be systematically monitored and developed. References Brock A. (2005). A Comparison of ArcIMS to MapServer. Proceedings Open Source Geospatial '05 MUM/EOGEO 2005, June 16-18, 2005. Minneapolis, USA Brown, S., Dalton S., Jepp D., Johnson D., Li S., Raible M. (2005): Pro JSP 2, APress. ISBN 978-1-59059-513-8. Buret J., Droze N. (2003). An Overview of Load Test Tools. At http://clif.objectweb.org/load_tools_overview.pdf [accessed 25 February 2009]. Czech Office for Surveying, Mapping and Cadastre (2008). Annual report 2007. Prague: ČUZK. ISBN 978-80-86918-54-9. Craglia, M. and M. Campagna (2009) (Editors): Advanced Regional Spatial Data Infrastructures in Europe, JRC Scientific and technical report, Joint Research Centre, Institute for Environment and Sustainability, 10-Feb-09, 132 pp. ISBN: 978-92-79-11281-2. at http://sdi.jrc.ec.europa.eu/ws/Advanced_Regional_SDIs/arsdi_report.pdf . [accessed 25 February 2009]. Man W.H.E. (2007). Beyond Spatial Data Infrastructures there are no SDIs – so what. International Journal of Spatial Data Infrastructures Research, Vol. 2, 123. Giff G., van Loenen B., Crompvoets J. and J. Zevenbergen (2009). Geoportals in Selected European States: A Non-Technical Comparative Analysis,


International Journal of Spatial Data Infrastructures, Vol.4, Special Issue GSDI-10, under review. Gheorghiu G. (2005). Performance vs. load vs. stress testing. at http://agiletesting.blogspot.com/2005/02/performance-vs-load-vs-stresstesting.html. [accessed 25 February 2009]. Hicks G., South J. and A.O.Oshisanwo (1997). Automated testing as an aid to systems integration. BT Technology Journal, Volume 15, n. 3, 26-36. Hyung G. S., Yeonseung R., Taesun Ch., Wooseok J. and L. Kangsun (2005). Metrics, Methodology, and Tool for Performance-Considered Web Service Composition. Proceedings 20th International Symposium Computer and Information Sciences - ISCIS 2005, October 26-28, 2005 Istanbul, Turkey. Springer, Lecture Notes in Computer Science, ISBN 978-3-540-29414-6. INSPIRE (2008). D3.7.2 Draft Implementing Rules for View Services. At http://inspire.jrc.ec.europa.eu/reports/ImplementingRules/network/D3.7.2_Dra ft_IR_View_Services_v3.0.pdf. [accessed 27 February 2009]. Lance K.T., Georgiadou Y., Bregt A. (2006). Understanding how and why practitioners evaluate SDI performance. International Journal of Spatial Data Infrastructures Research, Vol. 1, 65-104. Liu Z., Devadasan N., Pierce M. (2006). Federating, tiling and Caching Web map servers. At http://www.crisisgrid.org/cgwiki/images/4/49/IGIC_Oct27_2006_Final.ppt. [accessed 25 February 2009] Krejcar O. and J. Cernohorsky (2008). “New Possibilities of Intelligent Crisis Management by Large Multimedia Artifacts Prebuffering”. Proceedings I.T. Revolutions 2008, December 17-19, 2008 Venice, Italy. Springer, Lecture Notes of ICST, LNICST. ISBN 978-963-9799-38-7. Markov Z., Larose D. (2007). Data Mining the Web - Uncovering Patterns in Web Content, Structure, and Usage, New York: Wiley, ISBN 978-0-471-66655-4. 213 stran. Menascé D. A. (2002). Load Testing of Web Sites. Internet Computing, IEEE, Volume 6, 4 (Jul/Aug 2002), 70 – 74. Mendes E. and N. Mosley (Eds.) (2006). Web Engineering, Springer, New York. ISBN-10 3-540-28196-7, 438 pp. Pancheng W., Chongjun Y., Zhanfu Y., Yingchao R. (2004). A Load Balance Algorithm for WMS. Proceedings of Geoscience and Remote Sensing Symposium, 20-24 Sept. 2004, Toulouse, France. IEEE International, Volume 5, 2920 – 2922.


Nielsen J. (1994). Usability Engineering, Morgan Kaufmann, San Francisco, ISBN 0-12-518406-9 Wilson J.Q. (1989). Bureaucracy: what government agencies do and why they do it, New York: Basic Books.

Testing of Web Map Services - CiteSeerX

Testing of Web Map Services - CiteSeerX

Suggest Documents

Automatic Conformance Testing of Web Services - CiteSeerX

Testing Web Services: A Survey - CiteSeerX

Web services and testing

Testing Web Services: A Survey - CiteSeerX

A WEB SERVICES TESTING TOOL

Open Web Services Testing

A WEB SERVICES TESTING TOOL

a map of Caltech

Towards Contract-based Testing of Web Services - CiteSeerX

Practical Experiences of Testing Web Services

Collaborative Testing of Web Services - Semantic Scholar

Practical Experiences of Testing Web Services

Discovering Land Cover Web Map Services from

An XML-Based Framework for Web Services Testing wtsai ... - CiteSeerX

Extending WSDL to Facilitate Web Services Testing WT ... - CiteSeerX

Bioinformatics Web Services - CiteSeerX

Choreographing Web Services - CiteSeerX

Web services - CiteSeerX

Web Services Testing Challenges and Approaches

Web Services Quality Testing - Semantic Scholar

The Roles of Web Feature and Web Map Services in Real ... - CiteSeerX

7 Web Services Regression Testing - RCOST

Penetration Testing Tool for Web Services Security

Penetration Testing for Web Services - IEEE Xplore