Protocol for developing ANN models and its application to the ...

Protocol for developing ANN models and its application to the assessment of the quality of the ANN model development process in drinking water quality modelling

Wenyan Wu1, Graeme C. Dandy and Holger R. Maier School of Civil, Environmental and Mining Engineering, University of Adelaide, Adelaide 5005, Australia 1. Corresponding author. Tel: +61 (8) 8313 1113. Email: [email protected]

Abstract The application of Artificial Neural Networks (ANNs) in the field of environmental and water resources modelling has become increasingly popular since early 1990s. Despite the recognition of the need for a consistent approach to the development of ANN models and the importance of providing adequate details of the model development process, there is no systematic protocol for the development and documentation of ANN models. In order to address this shortcoming, such a protocol is introduced in this paper. In addition, the protocol is used to critically review the quality of the ANN model development and reporting processes employed in 81 journal papers since 2000 in which ANNs have been used for drinking water quality modelling. The results show that model architecture selection is the best implemented step, while greater focus should be given to input selection considering input independence and model validation considering replicative and structural validity.

Key words: Artificial neural networks, Environmental and water resources modelling, ANN model development protocol, water quality, review

  Published in Environmental Modelling and Software as follows:    Wu W., Dandy G.C. and Maier H.R. (2014) Protocol for developing ANN models and its application  to the assessment of the quality of the ANN model development process in drinking water quality  modeling, Environmental Modelling and Software, 54, 108‐127, DOI: 10.1016/j.envsoft.2013.12.016

1   

1 Introduction Artificial neural networks (ANNs) are computing systems containing parallel non-linear units. The ability of ANNs to learn underlying data generating processes, given sufficient data samples, has led to their wide application in prediction, forecasting, function approximation, classification, data processing, and robotics, among others. The application of ANNs for prediction and forecasting in the field of environmental and water resources modelling became popular in the early 1990s and has increased significantly over the last decade, which has led to a number of reviews on the application of ANNs (Gardner and Dorling, 1998; ASCE Task Committee on Application of Artificial Neural Networks in Hydrology, 2000a; ASCE Task Committee on Application of Artificial Neural Networks in Hydrology, 2000b; Maier and Dandy, 2000; Dawson and Wilby, 2001; Kalogirou, 2001; Sharma et al., 2005; Ahmadi-Nedushan et al., 2006; Abrahart et al., 2008; Kalteh et al., 2008; Céréghino and Park, 2009; Mellit et al., 2009; Elshorbagy et al., 2010a; Elshorbagy et al., 2010b; Maier et al., 2010; Abrahart et al., 2012).

Previous studies have recognized the need for a consistent approach to the development of ANN models. Gardner and Dorling (1998) reviewed the limits and problems associated with the training of multi-layer perceptron (MLP) ANNs and emphasized that fundamental understanding of the basic theory is a key in developing ANNs. The ASCE Task Committee on Application of Artificial Neural Networks in Hydrology (2000b; 2000a) summarized the key aspects of the development of ANN models, reviewed the application of ANNs in water resources modelling and pointed out future research required on the application of ANNs in hydrological modelling. Maier and Dandy (2000) reviewed the application of ANNs to prediction and forecasting of water resources variables and concluded that a more rigorous process for the development of ANNs is requried. Kalogirou (2001) reviewed the application of ANNs in renewable energy production and use, and developed a procedure for selecting key network parameters. Dawson and Wilby (2001) developed a flow chart listing the key steps in the development of ANN models for rainfall-runoff modelling and Maier et al. (2010) provided a taxonomy of methods used for the development of ANN models for river systems.

2   

Apart from a rigorous model development process, the thorough reporting and documentation of modelling exercises is also extremely important. While this has been acknowledged in a wider modelling context [e.g. Gass et al. (1981), Refsgaard et al. (2005), Jakeman et al, (2006), Bennett et al. (2013)], it has received little attention in the literature on the development of ANNs for environmental and water resources modelling. Only Abrahart et al. (2008) have emphasized the importance of the repeatability of experiments and the reproducibility of findings in the field of hydrological modelling using ANNs and pointed out that only papers with a repeatable methodology and reproducible results can make a contribution in the intended field. They also listed justification for the selection of model structure and calibration methods as a desirable characteristic. In addition, Abrahart et al. (2010) developed a seven-point checklist to specifically guide data reporting and preprocessing in papers on data driven models. However, criteria for assessing the quality of the documentation and justification provided have not been considered in previous studies.

Consequently, despite the recognition of the importance of the adoption and articulation of rational ANN model development procedures, there is no comprehensive protocol for the development of ANN models that: (i) identifies the steps in the ANN model development process that should be considered; and (ii) articulates what information should be documented at each of these steps. Hence, the first objective of this paper is to present a generic protocol for the development of ANN predictive models that includes: (i) the steps that should be followed in the development of ANN models; (ii) the information that should be documented in relation to the methods used at each of these steps, including the implications of the provision of different levels of detail of this information; and (iii) the justification of the adoption of the selected methods, including a categorization of different levels of justification.

The second objective of this paper is to use the proposed protocol to critically assess the quality of the ANN development and documentation processes in a particular application area since 2000. This has the dual purpose of (i) assisting with the understanding and interpretation of the proposed protocol so that it can be applied more easily in future studies; and (ii) providing a snapshot of the quality of the ANN model development and documentation processes in a particular application area over a particular time period, similar 3   

to the snapshot of the methods used at the various steps of the model development process provided in the review by Maier et al. (2010). The latter point is important, as previous reviews of ANNs in environmental and water resources modelling have not considered the quality of the ANN model development and documentation processes.

The application area selected for the review is the modelling of water quality variables in water supply systems, including water source, water treatment and water distribution. The reason for this is that previous reviews have generally focused on river systems (Solomatine and Ostfeld, 2008; Maier et al., 2010), at the expense of other aspects of water resources systems. Furthermore, previous reviews have primarily dealt with water quantity variables, such as rainfall and runoff (Dawson and Wilby, 2001; Jain et al., 2009), while water quality variables have generally not been considered. However, the reason water quality in water supply systems is selected as the subject of the review is not only because this is an area that has not been considered in previous reviews, but also because it is an area of increasing interest, as the use of ANNs for water quality modelling is becoming more prominent in the water industry (May et al., 2008).

The remainder of the paper is organized as follows. In the following section, the proposed protocol for the development and reporting of ANN models is introduced. The proposed protocol is applied to assessing the application of ANNs to water quality modelling in drinking water systems in Section 3, which also includes a brief description of the method used to select the papers for review and general information on the selected papers. A summary and conclusions are provided in Section 4. Recommendations for future work in the area of drinking water quality modelling using ANNs are provided in Section 5.

2 Protocol for developing and reporting on ANN models 2.1 Background and scope

4   

In order to assess the predictive performance of ANNs with some degree of confidence, it is important to critically examine the quality of the ANN model development and reporting processes. If a rigorous development process is adopted and adequate details of this process are provided, there is a greater degree of confidence in the results presented in a paper. In contrast, if the adopted model development process is not rigorous or not described adequately, it is not possible to draw any definite conclusions about how well the ANN models performed for the problem(s) presented in a paper. In order to make the adoption of a rigorous ANN model development and reporting process easier in practice, a comprehensive ANN model development protocol is proposed in this paper, as illustrated in Figure 1 and described in detail below.

Steps in model  development  process                Method and level  of detail                        Justification Data collection  and pre‐processing Significance Input selection Independence Data splitting

Model architecture selection Method(s) used  and details of  implementation

Reason(s) why  specific method(s)  selected

Model structure selection

Model calibration

Replicative Model validation

Predictive Structural

Model application

Figure 1 Proposed protocol for the development and reporting of ANN models

5   

As part of the proposed ANN model development protocol, the steps and options in the ANN model development process given by Maier et al. (2010) are used (Figure 1). Maier et al. (2010) divided the development of ANN models into a number of steps, among which six are of particular importance. These six steps are: input selection, data splitting, selection of model

architecture,

determination

of

model

structure,

model

calibration/training

(optimization of model parameters) and model validation. Selecting an appropriate set of inputs is an important step in the ANN model development process, as the inclusion of too many correlated or extraneous inputs will increase model training time and the likelihood of overfitting; while exclusion of important inputs will lead to a model that cannot fully describe the input-output relationship (Maier and Dandy, 2000). As with any other data-driven model, cross-validation or regularization can be used in the development of ANN models to ensure their generalization ability (Razavi et al., 2012). However, splitting the available data into calibration (including training and test datasets if cross-validation is used) and validation datasets for model training and validation can introduce bias and variance into the ANN model development process (LeBaron and Weigend, 1998; May et al., 2010). As a result, a robust and reliable data splitting method is required for a given dataset (May et al., 2010; Wu et al., 2012; Wu et al., 2013). Selection of model architecture, structure and training algorithm has a direct impact on the final model developed, and thus the performance of the model, as it determines how the model inputs are transformed into model outputs. How well the model validation step is implemented has a direct impact on the assessment of model performance. In general, there are three aspects of validity that need to be considered during the model validation process, namely replicative validity, predictive validity and structural validity (Power, 1993). For each aspect, different model performance measures can be used. For example, different predictive performance measures often assess different aspects of the measured response (e.g. peak, average etc.) and therefore, based on the purpose of the model, different criteria should be selected (Dawson et al., 2007; Bennett et al., 2013). A taxonomy of the techniques that can be used in each of these steps can be found in Maier et al. (2010).

Apart from providing guidance on the techniques used in each of the ANN model development steps, as has been done in previous papers (Maier and Dandy, 2000; Dawson and Wilby, 2001; Maier et al., 2010), the proposed protocol includes two additional aspects of the ANN model development process, which are equally important in order to assess how well the six major model development steps have been carried out. The first of these is the 6   

level of detail on the implementation of the ANN model development process included in the paper (Figure 1). This information should include details of the techniques used at each of the six model development steps and how they are implemented, including the values of key parameters used, such as the percentage of data used for training, testing and validation, and the values of the parameters used to control the searching behaviour of the calibration algorithm used (e.g. learning rate and momentum if the back-propagation method is used). Consideration of this aspect is vitally important in order to guarantee the repeatability of the ANN model development process employed in a paper, which is not only a requirement of quality assurance for model development (Scholten and Udink ten Cate, 1999; Scholten et al., 2001), but also needed to assist the interaction between manager, code developer and modellers (Refsgaard and Henriksen, 2004). In addition, repeatability ensures that the results obtained in a paper can be reproduced and independently validated, which contributes to knowledge advances in related research areas (Abrahart et al., 2008).

The second additional aspect is the justification provided for the use of particular method(s) during the ANN model development process (Figure 1). For example, for a particular problem, certain techniques may have been selected because they have been proven to perform well in previous studies; or a comparison study may have been conducted in order to select a certain method for data splitting. Consideration of this second aspect will increase the level of confidence in the results presented in the paper, as it ensures a critical analysis is conducted when developing ANN models either by theoretical analysis or numerical comparison, rather than making an arbitrary selection of a method or basing the selection solely on the modellers’ existing knowledge. Justification also drives technological progress in the field by ensuring research builds upon previous studies, rather than “re-inventing the wheel” (Abrahart et al., 2008). The provision of justification was also recognized by Jakeman et al. (2006) as a minimum requirement for reporting on the model calibration process.

Details of each of the steps in the proposed ANN model development protocol are given in the subsequent sections.

7   

2.2 Methods used in the ANN model development process 2.2.1 Input selection For the development of ANN models, inputs need to be determined based on both the significance and independence of inputs (Bowden et al., 2005a; Bowden et al., 2005b; May et al., 2008; Castelletti et al., 2012), as shown in Figure 1. Maier et al. (2010) reviewed a number of techniques that can be used to assess the significance of the relationship between potential inputs and output(s). These techniques can be divided into model free techniques and model based techniques. Model free techniques include analytical methods based on correlation (Li et al., 2008) and mutual information (MI) (Sharma, 2000), and ad-hoc methods based on available data and/or domain knowledge. Model based techniques include stepwise methods, such as constructive or pruning methods (Diamantopoulou et al., 2007), sensitivity analysis (Dogan et al., 2009), global optimization methods, such as genetic algorithms (Bowden et al., 2005b), and ad-hoc methods, such as trial-and-error (Maier et al., 2010).

Methods that can be used to account for input independence can also be divided into two categories: methods based on dimensionality reduction and filtering (Maier et al., 2010). Dimensionality reduction can be achieved by using principal component analysis (with rotation of inputs) (Cho et al., 2011), or by clustering the input or input-output space, as is the case when using clustering analysis and self-organizing maps (Bowden et al., 2005a; Bowden et al., 2005b). Filtering refers to selecting the most significant input variables considering the contribution of already selected inputs, which includes methods based on partial correlation or partial mutual information (PMI) (May et al., 2008). It should be noted that many input selection methods considering input independence also take into account input significance, such as PMI based methods (May et al., 2008) and a recently developed Recursive Variable Selection (RVS) algorithm (Castelletti et al., 2012).

2.2.2 Data splitting Data splitting methods can be divided into unsupervised methods and supervised methods (Maier et al., 2010). Random data splitting is the most commonly used unsupervised data 8   

splitting method (Maier et al., 2010). Other unsupervised data splitting methods include stratified methods, such as self-organizing maps (SOMs) (Li et al., 2008); physically based methods, such as dividing data according to underlying physical process or domain knowledge (Haas, 2004); and ad-hoc methods, in which data are divided using some user specified rules. One commonly used ad-hoc supervised data splitting method is trial-and-error (Maier et al., 2010). Data splitting methods based on optimization (Bowden et al., 2002), such as genetic algorithms, are also classified as supervised methods (Maier et al., 2010).

2.2.3 Model architecture selection The ANN model architecture determines the overall structure and information flow of the model (Maier et al., 2010). Feedforward neural networks, where information is passed only in one direction from the input layer to the output layer, are the most commonly used ANN architecture (Maier et al., 2010; Razavi and Tolson, 2011). A typical feedforward ANN architecture is a multi-layer perceptron (MLP), which consists of an input layer, a number of hidden layers and an output layer. The number of input and output neurons (nodes in the input and output layers) are determined by the number of input and output variables, respectively. Generally, weighted inputs are passed from the input layer to the hidden layers. Either linear or non-linear activation functions can be used at hidden and output layers. However, an ANN with linear activation functions in all layers is equivalent to a linear model with limited abilities. It is the capability of using different non-linear activation functions in different layers that enables MLPs to capture complexity and non-linear relationships within a system. Other common feedforward ANN architectures include probabilistic neural networks (PNN), general regression neural networks (GRNNs), cascade forward networks (CFN), radial basis function networks (RBF), modular neural networks (MNN), associative memory networks (AMN) and reformulated neural network (ReNN) (Razavi and Tolson, 2011).

In recent years, recurrent neural networks are receiving more attention from ANN modellers (Maier et al., 2010). In recurrent neural networks, information is not only passed forward from the input layer to the output layer, but also backwards from the output layer to the input and/or hidden layer through a feedback loop (Karamouz et al., 2008). The information 9   

feedback from the output layers can be used to update the weights or even structure of the model, which enables the model to capture the complexities of highly dynamic systems (Maier et al., 2010). At times, there are advantages of combining ANNs with other models (e.g. process-based models) or techniques (e.g. regression or fuzzy logic) to form a hybrid model, which can exploit the advantages of any existing knowledge of the system and different modelling techniques. This has the potential to achieve superior model performance, especially for some extremely complex environmental and hydrological systems (Maier et al., 2010).

2.2.4 Model structure selection Determination of ANN model structure generally involves determining the number of layers, the number of nodes in each layer and how they are connected (Maier et al., 2010). An optimal network structure is generally one that minimizes forecasting/prediction errors and maximizes model parsimony (in terms of network size). Many methods can be used to determine model structure. Maier et al. (2010) divided the methods for determining an optimal ANN model structure into global optimization methods, such as genetic algorithms; stepwise methods, such as pruning and constructive methods; and ad-hoc methods, such as trial-and-error or selecting a structure based on previous experience. In addition, regularization is another approach that can be used to determine an ANN model structure and stop ANN models from over-fitting (Razavi et al., 2012). One advantage of regularization over traditional cross-validation is that the available data only need to be divided into two sets (i.e. training and validation), rather than three sets (i.e. training, test and validation), which provides more data points for model training (Razavi et al., 2012). It should be noted that for some ANN models, such as PNNs, GRNNs and some hybrid ANNs, network structure is fixed and therefore a method is not required for determining the structure of these models. Model structure selection is often conducted together with model calibration in an integrated manner, as indicated by the dashed box in Figure 1. For example, when applying a trial-and-error method for model structure selection for MLPs, model weights are often optimized at the same time.

2.2.5 Model calibration 10   

The calibration or training of a model refers to the process of finding a set of model parameter values (e.g. weights) that enable the model to map the relationship between the inputs and outputs of a given dataset and often requires the use of optimization algorithms. The majority of training methods is deterministic, which search for a set of model parameter values to minimize an error measure between the observed and predicted outputs. These deterministic methods can be divided into local optimization algorithms, such as the classic back-propagation method developed by Rumelhart et al. (1986) and Newton’s methods, among others (Maier and Dandy, 1999; Maier and Dandy, 2000); and global optimization algorithms, such as genetic algorithms (Sahoo et al., 2009). Local optimization algorithms are usually computationally efficient; however, as they work on gradient information, they are prone to becoming trapped in local optima when the error surface is rugged (Kingston et al., 2005b). Global optimization methods may increase the chance of finding global optima in a rugged error surface in one run; however, they are more computationally expensive and often have more parameters that need to be determined in order to improve their performance (Hamm et al., 2007). In addition, as a statistical model, ANNs need to work with existing data to produce predictions. Therefore, the amount of useful information contained in the available data (e.g. accuracy and length of data) is likely to have a significant impact on the performance of developed ANN models.

While deterministic calibration methods are most commonly used, they ignore uncertainties in model inputs and outputs, model structure and model parameters, which has been addressed in a number of studies (Kingston et al., 2005a; Khan and Coulibaly, 2006; Srivastav et al., 2007; Jana et al., 2008; Kingston et al., 2008; Zhang et al., 2009; Zhang et al., 2011; Kasiviswanathan et al., 2013).

2.2.6 Model validation The overall aim of model validation is to make sure a trained ANN model does not contain known or detectable flaws so that it can be used for its intended purpose with confidence. However, the generally complex error surface of environmental and water systems ANN models makes this a difficult task, as there are often multiple combinations of ANN structures and/or weights that lead to similar model performance (i.e. non-uniqueness of 11   

ANN models) (Kingston et al., 2005b). Therefore, effective model validation beyond only considering model predictive performance is required in order to ensure developed ANN models are physically plausible.

Effective model validation can be achieved by considering three aspects of model validity, namely replicative validity, predictive validity and structural validity (Gass, 1983). A model is replicatively valid if it matches the data already acquired from the real system and used in the previous steps of the ANN model development process (i.e. data processing, input selection, data splitting, ANN architecture/structure selection and training) (Gass, 1983; Power, 1993). This is to make sure the trained model has captured the underlying relationship that is contained in the calibration data. Replicative validity can be assessed by using standard statistical techniques, including means and variances, analysis of variance, goodness-of-fit testing, regression and correlation analysis and confidence interval construction, as is done for traditional statistical models (Power, 1993). In addition, replicative validity can be assessed by examining whether or not the model residuals (errors) are white noise using key residual methods, such as residual and QQ (i.e. normal probability) plots (Bennett et al., 2013).

A model is considered predicatively valid if it can represent the input-output relationship in the data that will be acquired from the real system in the future (Gass, 1983). In other words, predictive validity tests the generalization ability of the model over the range of the data used for calibration. This can be done by using an independent validation dataset. The predictive performance of ANN models is often assessed using quantitative error measures obtained for the validation dataset (Maier et al., 2010). Commonly used error measures can be divided into the following five categories: squared errors, absolute errors, relative errors, product differences and information criteria (Maier et al., 2010). Squared errors are based on the squares of the differences between the measures and predicted outputs. Commonly used squared errors include the mean square error, the root mean square error and the sum of squared errors. Absolute errors are based on the absolute difference between the measured and predicted outputs. Commonly used absolute error measures include the mean absolute error and the sum of absolute errors. Bias based errors (difference between the measured and predicted outputs without taking absolute values) are also included in this category. Relative 12   

errors give an indication of the magnitude of the differences between the measured and predicted outputs relative to the measured outputs. Relative errors include average absolute relative error, mean percentage error and relative bias. Efficiency indices, such as Pearson correlation (or coefficient of determination), the Nash-Sutcliffe efficiency and the Index of Agreement, are also commonly used model performance criteria. The study by Bennett et al. (2013) provides a good summary of efficiency indices. Alternatively, information criteria, such as the Akaike information criterion (AIC) (Kingston et al., 2005a; Chibanga et al., 2003) and the Bayesian information criterion (BIC) (Chibanga et al., 2003), can also be used to assess ANN model performance. Apart from the generalization ability of the model, information criteria also take model parsimony into account. A summary of commonly used error measures can be found in Dawson et al. (2007) and Bennett et al. (2013). In addition, visual inspection and some case study dependent model validation methods, such as water pollution levels and prediction accuracy can also be used. Among these methods, visual inspection has been listed as a key step in model performance evaluation by Bennett et al. (2013).

Structural validity is used to ensure the model is plausible when compared with a priori knowledge of the system behaviour, which is intended to be reflected by the model. Structural validity is only achieved if a model not only reproduces the observed real system behaviour, but truly reflects the way in which the real system is understood to operate to produce this behaviour (Ziegler, 1976). In this sense, structural validity assessment can also be used to analyse the uncertainties of ANN outputs. These uncertainties are often related to the limited information contained in available data or even errors in sampling procedures and therefore cannot be analysed based on predictive validity assessment alone (Kingston et al., 2005b). Structural validity tests are often applied to process-driven models, the parameters of which generally have an associated physical meaning, however, is typically ignored in the standard validation of ANN models (Kingston et al., 2005a). The structural validity of an ANN model can be assessed using a variety of techniques, such as sensitivity analysis (Mount et al., 2013), overall connection weights (Olden and Jackson, 2002; Kingston et al., 2005a; Kingston et al., 2006), a measure of generalization (Razavi and Tolson, 2011) and other methods, such as comparing the modelling results with a priori knowledge of the system being modelled (Jain et al., 2004; Jain and Kumar, 2009; Millie et al., 2012).

13   

2.3 Level of detail on the implementation of ANN model development process

The purpose of conducting research is to make a contribution in the designated research field (Maier, 2013). A contribution can only be made if the knowledge developed as part of research can be understood by and passed on to fellow researchers in related fields. As mentioned in the Introduction, this requires not only a rigorous model development process, but also documenting and reporting of the experimental procedures involved in the study to make sure the experiments are repeatable and the findings are reproducible (Abrahart et al., 2008).

As part of the ANN model development protocol introduced in this paper, four categories are proposed to assess the quality of the implementation of the six ANN model development steps in regard to the level of detail recorded (Table 1). Category 0 refers to the case where a particular step is not carried out in the model development process. If a particular step is carried out, Categories 1 to 3 are used to classify the level of detail included. Category 1 represents the case where a particular modelling step is carried out, but where the method used is not mentioned in the paper and therefore, the implementation of the step can only be deduced by the reader from the discussion in the paper. Category 2 represents the case where the method used in the model development step is mentioned in the paper, however, where insufficient details regarding the procedures have been provided so that the step cannot be repeated by other researchers

Category 3 represents the case where detailed information on the implementation of the model development step is provided, so that it can be repeated for the same/similar studies by other researchers. For example, in relation to the input selection step, this would entail rationalization of the selected inputs if an ad-hoc input selection approach was used or, if an analytical input selection methods was used, a detailed description of the selected approach. In relation to data splitting, in order for the requirements of Category 3 to be satisfied, details about the data splitting method used, how it was implemented and the percentage of data 14   

allocated to each subset would need to be provided. As far as model architecture selection is concerned, details on the type of model architecture used, the way different layers/nodes are connected, any functions (e.g. transfer functions or membership functions) used in each layer and any weights/parameter that need to be calibrated would need to be provided. With regard to model calibration, details on the calibration algorithm used, the values (or ranges of values) of any parameter involved, the procedures used to prevent over-fitting or under-fitting, and any other stopping criteria used would need to be given. For ANN models that do not have a fixed structure, the procedures used to determine the optimal model structure would need to be reported. For example, for MLPs, upper and lower bounds on the number of hidden layers and the number of hidden layer nodes would need to be provided, as well as the procedure for determining their optimal number (e.g. trial-and-error, constructive etc.). In relation to model validation, Category 3 requirements would include provision of the details of the model predictive performance criteria (Dawson et al., 2007; Maier et al., 2010; Bennett et al., 2013) of the methods used to examine the residuals and the approaches used to check for structural validity. The authors acknowledge that there is often a length limit for papers published in many journals and therefore, if a commonly used method for the implementation of an ANN model development step is selected, the provision of relevant references is sufficient for the step to be included in Category 3.

As can be seen from Table 1, Category 0 (i.e. the step is not carried out) only applies to the steps of data splitting and model validation, as an operational ANN model cannot be developed without the implementation of the other four steps (i.e. input selection, model architecture selection, model structure selection and model calibration). In addition, for the model structure selection step, Category 3 also includes cases where the structure of a model is fixed (e.g. for GRNN), as Category 3 emphasizes the repeatability of a particular model development step.

15   

Table 1 Proposed assessment categories for the implementation of the six steps in the ANN model development process Step

Input selection (Significance/ independence)

Category: 0 Description: Step not carried out NA

Category: 1 Description: mentioned

Method

Category: 2 not Description: Method mentioned

Inputs are selected; however, the The input selection method used is method used to select them is not mentioned, however, the procedure mentioned. cannot be repeated due to a lack of information.

Data splitting

Data splitting Data splitting is performed; The data splitting method used is is not however, the method used to split mentioned, however, the procedure performed. the data is not mentioned. cannot be repeated due to a lack of information.

Model architecture selection

NA

Model structure selection

NA

Model calibration

NA

Model validation

Model validation

The model architecture is not The model architecture used is mentioned. mentioned, however, not enough information is provided so that the architecture can be recreated. A model structure is selected; The model structure selection method however, the method used to used is mentioned, however, the select it is not mentioned. procedure cannot be repeated due to a lack of information.

Category: 3 Description: Method repeatable

The input selection method is described in detail (or relevant references are provided), so that it can be repeated for the same/similar studies. The data splitting method is described in detail (or relevant references are provided), so that it can be repeated for the same/similar studies. The model architecture used is described in detail (or relevant references are provided), so that it can be recreated. The model selection method is described in detail (or relevant references are provided), so that it can be repeated for the same/similar studies.

The model calibration method The model calibration method used is used is not mentioned. mentioned, however, the procedure cannot be repeated due to a lack of information.

The model calibration method is described in detail (or relevant references are provided), so that it can be repeated for the same/similar studies. The model validation method(s) The model validation method(s) used The model validation method(s) is used is not mentioned. is mentioned, however, not enough is described in detail (or relevant 16 

 

(Replicative/ Predictive/ Structural)

not performed.

information (e.g. details/references) is references are provided), so that provided so that the procedure can be it can be repeated for the repeated. same/similar studies.

17   

2.4 Justification for the implementation of ANN model development process

The level of justification provided for the use of a particular method at a particular model development step affects the credibility of the research conducted and the confidence in the results reported in the paper. A higher level of justification reduces uncertainties in the selection of a method and the chance the reported results are obtained by accident. Justification can be provided either by theoretical discussion or numerical comparison. Discussion-based justification ensures the reported research is built upon findings from previous studies or established domain knowledge. Numerical justification can be obtained by conducting comparative studies, which also drives methodological progress in the research field directly by providing numerical evidence for future research.

Similar to the proposed four categories for assessing the level of detail included for an ANN model development step described above, four categories are also proposed to assess the justification provided for the use of a particular method in a model development step, as shown in Table 2. Category 1 indicates that no justification is provided; Category 2 suggests that discussion-based justification is provided based on findings from previous studies or domain knowledge via discussion; and Category 3 indicates that numerical justification is provided by comparing the method selected with alternative methods. In addition, a fourth category is used to represent cases where the method used is defined by the problem presented in the study, so no justification is required. The fourth category only applies to the step of model structure selection, as shown in Table 2. Because for some ANN model architectures, such as GRNNs, the structure is fixed/defined, an optimal structure does not need to be selected.

It should be noted that Category 3 does not apply to the model validation step, as, in contrast to the other steps, where model predictive ability can be used to assess the relative performance of different methods (e.g. calibration methods, methods for selecting the number of hidden nodes etc.), there is no quantitative measure for measuring validation performance. It should also be noted that Category 4 only applies to the model structure selection step. This is because the structure of certain types of ANNs, such as GRNNs or PNNs, is fixed, so no method is needed to select an appropriate structure and therefore, no justification is required. 18   

Alternatively, a fixed model structure selection method is required for some ANN model architectures (e.g. a stepwise method has to be used for CFNs) and therefore, no justification is required.

19   

Table 2 Proposed assessment categories for the justification provided for methods used during the implementation of the six steps of the ANN model development process (if a particular step is carried out) Step

1 No justification

2 Discussion-based justification No justification The use of a particular is provided. input selection method is justified via discussion.

Input selection (Significance/ independence) Data splitting No justification The use of a particular is provided. data splitting method is justified via discussion. Model architecture selection

No justification The use of a particular is provided. model architecture is justified via discussion.

Model structure selection

No justification The use of a particular is provided. model structure selection method is justified via discussion. No justification The use of a particular is provided. model calibration method is justified via discussion.

Model calibration

Model Validation (Replicative/ Predictive/ Structural)

No justification The use of a particular is provided. model validation method is justified via discussion.

3 Numerical justification

4 Not applicable

The use of a particular input selection method is justified by comparing it to alternative methods. The use of a particular data splitting method is justified by comparing it to alternative methods. The use of a particular model architecture is justified by comparing it to alternative model architectures. The use of a particular model structure selection method is justified by comparing it to alternative methods. The use of a particular model calibration method is justified by comparing it to alternative methods. NA.

NA.

NA.

NA.

The structure of the model is defined by the architecture used, so no justification is required. NA.

NA.

20   

3 Review of application of the proposed protocol to drinking water quality models 3.1 Selection of papers for review

As mentioned in the Introduction, the focus of this review is on the application of ANNs to the modelling of water quality variables in water supply systems, including water source, water treatment and water distribution. In order to identify relevant papers, a keyword search of the ISI Web of Science was conducted in January 2013 for the period 2000 to 2012 using the key words “neural network” and “water quality” for the field “topic”, within the research areas of “engineering” and “water resources” for articles in English, which resulted in 618 papers from 307 journals, of which 190 journals only had one article selected. Then, the sources of the papers were refined based on their impact factors, relevance to water resources engineering and standing in the field. Thereafter, the papers were refined manually to exclude review papers and papers focusing on wastewater treatment, coastal water, non-water-quality variable estimation, data visualization, remote sensing, food processing and agricultural applications, which resulted in 81 remaining articles from 44 international refereed journals focusing on the forecasting and prediction of water quality variables in drinking water systems using ANNs. The 44 journals and their 2012 impact factors are summarized in Appendix A. A summary of the analysis of the 81 papers is provided in Appendix B.

The 81 papers are firstly analysed based on their year of publication, the component of the drinking water system considered (i.e. source, treatment and distribution) and the type of water quality variables modelled. The number of papers published in each year from 2000 to 2012 is plotted in Figure 2. As can be seen, there is a generally increasing trend in the number of papers published since 2000. Even though there are some fluctuations, with drops in the number of papers published in 2004, 2005 and 2007, 14 papers were published in 2012, compared with only one paper in 2000. This is in agreement with the increasing popularity of ANNs within the field of water resources modelling observed by other researchers (Maier et al., 2010; Abrahart et al., 2012).

21   

16 13

14

14

Number of papers

12 10

10 7

8 6

6 3

4 2

4

6 3

5

6

3

1

0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Figure 2 Distribution of papers reviewed by year of publication

In drinking water systems, water can be obtained from a variety of sources, including surface water from reservoirs, lakes and rivers, groundwater from aquifers, stormwater and seawater through desalination. The source water is generally treated using filtration and disinfection (e.g. chlorination) before being delivered to end water users’ taps via distribution systems. The number of papers focusing on different components of the drinking water system is plotted in Figure 3. As can be seen, the majority of the papers reviewed focus on water quality modelling at the source, with 35 out of the 81 papers modelling water quality in rivers and 19 papers modelling water quality in reservoirs and/or lakes. In total, there are only eight papers focused on drinking water treatment and seven papers on water distribution systems. It should be noted that in one paper, both a lake and its downstream river were considered, which resulted in a total paper count of 82, rather than 81, in Figure 3.

22   

40

35

Number of papers

35 30 25 20

19

15 10 5

8

9

7

4

0

Figure 3 Distribution of papers reviewed by area of application

Different water quality variables are often modelled for different aspects of drinking water systems. For example, for reservoirs, lakes and rivers, algae, nutrients or nutrient related chemical indicators, such as dissolved oxygen (DO) and biological oxygen demand (BOD), and physical water quality variables, such as temperature, turbidity, salinity and colour, are often of concern. For groundwater and stormwater, pollutants, such as pesticides and heavy metals, are often monitored. In drinking water treatment plants and distribution systems, disinfectant residuals and disinfection by-products (DBP), such as trihalomethanes (THM), need to be modelled. Therefore, the water quality variables modelled in the papers are divided into three categories, i.e. chemical, biological and physical. Chemical water quality variables include DO, nitrate, nitrite, chemical oxygen demand (COD), BOD, disinfection byproducts (DBP), pesticides and metal irons. Physical variables include pH, temperature, total suspended solids (TSS), total dissolved solids (TDS), conductivity, turbidity, salinity, hardness and colour. Biological variables include protozoan parasites, such as cryptosporidium and giardia lamblia, faecal coliforms and algae.

The number of times each type of water quality variable was modelled in the 81 papers is given in Figure 4. In many studies, more than one type of water quality variable was 23   

modelled. Chemical variables were modelled in more than half of the papers, followed by physical variables. In contrast, biological variables were only modelled in 20 out of the 81 papers reviewed. In addition, case study specific variables, such as pollution levels and the magnitude, location and timing of pollution, were also used as output variables in a number of studies.

60

55

Number of papers

50 40 30

24 20

20 10

5

0 Chemical

Physical

Biological

Others

Figure 4 Number of times various types of water quality variables were forecasted/predicted in the papers reviewed

3.2 Assessment of quality of ANN model development process

3.2.1 Introduction of assessment process In this section, the 81 papers are assessed in accordance with the ANN model development protocol introduced in Section 2 (Figure 1 and Tables 1 and 2). The authors acknowledge that it is difficult to assess what has actually been done in each of the studies presented in the papers reviewed without participating in the studies and that there is a page limit for articles published in many journals. However, the authors believe that certain information on how the

24   

studies were carried out needs to be included in each paper, so that readers can make an objective assessment in regard to how the models were developed and the degree of confidence they had in the results presented in the papers; and to make sure that the knowledge developed in the paper can be built on in future studies.

In many of the papers reviewed, a number of different types of ANN models were developed in a single paper. In this case, the model development processes for different types of ANNs were evaluated separately to account for all different model development processes. On the other hand, in some of the papers reviewed, multiple ANN models of the same type were developed. If this was the case, the ANN model development process adopted for a particular model type was often the same. Consequently, in order to avoid biasing the results towards the ANN model development process adopted in papers in which multiple models were developed, a development process that was applied to more than one model in a particular paper was only counted once. This resulted in a total of 99 different ANN model development processes, which were assessed based on the protocol proposed in Section 2. In assessing the 99 ANN model development processes, the following three aspects were assessed for each model development step in accordance with the proposed ANN model development protocol (Figure 1 and Tables 1 and 2): 1) the level of detail provided; 2) the type of method used, as per the categorization provided by Maier et al. (2010); and 3) the justification provided for the use of a particular method. The outcomes of the assessment process are given in the following subsections.

3.2.2 Input selection As mentioned previously, some input selection methods considering input independence also take into account input significance. In order to minimize repeated information in the results and clearly identify the methods considering input independence, the input selection methods used are assessed in two different categories: a) methods based on input significance only and (b) methods also considering input independence. The level of detail provided for the input selection methods used in the 99 ANN model development processes reported is presented in Figure 5. The number of times different methods were used is presented in Figure 6. It should be noted that in a number of studies, more than one input selection method considering input 25   

significance only was used in order to find the most appropriate inputs and therefore, the total number of inputs selection methods presented in Figure 6 is greater than 99. The assessment of the justification provided for the use of these input selection methods is summarized in Figure 7.

70

62

60

60

50

50

Number of times

Number of times

70

40 30

24

20 10

30 20

12

10 0

0

(a)

40

0

1

1: Not mentioned

2: Mentioned

0

1: Not mentioned

2: Mentioned

(b)

3: Repeatable

3: Repeatable

Figure 5 Assessment of input selection method details provided (a): for methods based on input significance only and (b) for methods also considering input independence

60

60 48

50

Number of times

Number of times

50 40 30 20

18 11

10

6

6

0

(a)

Analytical  Ad‐Hoc (model  Stepwise  Ad‐hoc (model  Sensitivity  (model free) free) (model based) based) (model based)

40 30 20 7

10

3

3

0

(b)

Rotation (Dimensionality  Clustering (Dimensionality  reduction) reduction)

Filtering

Figure 6 Number of times various methods have been used to account for input significance (a): for methods based on input significance only and (b) for methods also considering input independence

As can be seen from Figure 5, for all of the 99 ANN model development processes employed, the input selection method used was described and for the majority of the cases, it was described in great detail so that the step can be repeated for the same/similar studies. This is the case for the methods used to account for input significance only and those also accounting 26   

for input independence. This is considered good practice, as the process can be independently verified by other researchers. In addition, input significance was the primary criterion used for input selection and was considered in all 99 ANN model development processes, as methods considering input independence also take into account input significance; while input independence was only considered 13 times in the 99 cases, as can be seen from Figure 5. This finding is similar to that of Maier et al. (2010) for ANNs applied to forecasting water resource variables in river systems.

As can be seen from Figure 6(a), for methods considering input significance only, model free approaches were most popular, which were used 66 times, compared with model based approaches, which were only used 23 times. In 48 out of the 66 instances when a model free approach was used to determine input significance, an ad-hoc method based on available data and/or domain knowledge was used. Similarly, for half of the cases where input significance only was considered, model based input selection was carried out in an ad-hoc fashion. Model based stepwise or sensitivity analysis methods were used six times each. The most popular approach for accounting for input independence was rotation of the input vectors, which was used seven times; while clustering and filtering approaches were used three times each, as shown in Figure 6(b).

70

60

50

Number of times

Number of times

60

70 57

40 29

30 20 10

0

50 40 30 20 10

0

(a)

12 0

1

0 1: No justification

2: Discussion‐based  justification

3: Numerical  justification

(b)

1: No justification

2: Discussion‐based  justification

3: Numerical  justification

Figure 7 Assessment of justification provided for input selection (a): for methods based on input significance only and (b) for methods also considering input independence

27   

As mentioned previously, in a number of papers reviewed, more than one method was used to select appropriate inputs. This is considered good practice, as it provides multiple lines of evidence in determining the most appropriate inputs, which is likely to provide greater confidence in the selected inputs. However, as can be seen in Figure 7, no numerical justification was provided for the use of the input selection method in any of the 99 model development processes investigated; discussion-based justifications were provided in 29 cases for input selection methods based on input significance only and in 12 cases for input selection methods considering input independence. It can also be seen that justification was not provided for the selection of a particular input determination method in the majority of the cases when an input selection method based on input significance only was used.

3.2.3 Data splitting The level of detail provided for the data splitting step is summarized in Figure 8. The number of times different data splitting methods were used for ANN model development is summarized in Figure 9. The assessment of justification provided for the selection of particular data splitting methods is summarized in Figure 10.

80 68

Number of times

70 60 50 40 30 20

20 10

10 1

0 0: Not  implemented

1: Not mentioned

2: Mentioned

3: Repeatable

Figure 8 Assessment of data splitting method details provided

28   

50 40

Number of times

40

33

30 20 10

2

1

1

1

0

Figure 9 Number of times various data splitting methods were used

90 80

79

Number of times

70 60 50 40 30 18

20

1

10 0 1: No justification

2: Discussion‐based  justification

3: Numerical  justification

Figure 10 Assessment of justification provided for data splitting method

29   

As can be seen from Figure 8, for the majority of the 99 cases, sufficient information was provided in describing the data splitting step conducted so that the step can be repeated for the same/similar studies. However, there are 20 cases where the method used to split data was not mentioned, and the implementation of the data splitting step can only be recognized by the fact that more than one data subset was used for model development. In addition, in one of the 99 cases, data splitting was not implemented, which also resulted in the model not being validated. This will be discussed in the review of model validation (Section 3.2.7).

The large number of cases where sufficient information was provided in relation to the data splitting method used is at least in part due to the fact that unsupervised ad-hoc or random methods were used for data splitting. As can be seen from Figure 9, unsupervised data splitting methods were used 76 times. Among these cases, ad-hoc methods based on time order or user specified rules were used 40 times and random data splitting was used 33 times. Most ad-hoc unsupervised data splitting methods have simple user defined rules and therefore, can be described easily in detail. In contrast, supervised trial-and-error methods and optimization based methods were each only used once.

Another inadequacy of the implementation of the data splitting step in the 81 papers reviewed is shown in Figure 10. As can be seen, in the vast majority of cases when data splitting was implemented, no justification was provided for the selection of the data splitting method used; while comparative studies were only used once in order to justify the selection of the data splitting method used.

3.2.4 Model architecture selection The level of detail provided for the ANN model architectures selected in the 99 ANN model development processes is summarized in Figure 11. The number of times different model architectures were used is summarized in Figure 12. The assessment of the justification provided for the selection of model architectures is summarized in Figure 13.

30   

100

91

90

Number of times

80 70 60 50 40 30 20 7

10

1

0 1: Not mentioned

2: Mentioned

3: Repeatable

Number of times

Figure 11 Assessment of model architecture details provided

80 70 60 50 40 30 20 10 0

66

6

6

11 2

2

3

2

Figure 12 Number of times various model architectures were used

31   

36

Number of times

35

35

34 33

33 32 31 31 30 29 1: No justification

2: Discussion‐based  justification

3: Numerical  justification

Figure 13 Assessment of justification provided for model architectures selected

Model architecture selection is one of the ANN model development steps that has traditionally received more attention from ANN modellers for modelling river systems (Maier et al., 2010). This is also the case for ANN modellers when modelling drinking water quality. As can be seen from Figure 11, in 91 out of the 99 cases, the model architecture used was described in great detail or sufficient reference was provided so that the model architecture can be recreated; while the model architecture was not mentioned in only one case. In seven cases, the model architecture used was not described and no reference was given. However, in most of these cases MLPs, which were considered to be a commonly used ANN model architecture, were used and therefore, researchers might have felt it was not necessary to provide details of the model architecture. However, references are recommended, even if commonly used methods are used.

As can be seen from Figure 12, feedforward networks were the most popular ANN architecture used in the 99 ANN model development processes examined, with MLPs alone being used 66 times. This partially contributes to the large number of cases where the model architecture was well described. Hybrid models were reasonably popular and were used 11 times; while recurrent networks were only used twice. The selection of an ANN model 32   

architecture was often justified, as shown in Figure 13. In 31 out of the 99 cases, the selection of an ANN model architecture was justified via discussion and comparative studies were carried out in 33 cases in order to find the best possible model architecture; while in only 35 out of the 99 cases, no justification was provided for the ANN model architecture selected.

3.2.5 Model structure selection The level of detail provided for the model structure selection step included for the 99 cases is summarized in Figure 14. The number of times different methods were used to determine the optimal structure is summarized in Figure 15. The assessment of the justification provided for the methods used to select the optimal model structures is summarized in Figure 16.

70 62

Number of times

60 50 40 30 20

21 16

10 0 1: Not mentioned

2: Mentioned

3: Repeatable

Figure 14 Assessment of model structure selection method details provided

33   

70

Number of times

60

55

50 40 30 18

20 10

4

6

Optimisation

Stepwise

0 Ad‐hoc

Fixed structure

Figure 15 Number of times various model structure selection methods were used

70 60

Number of times

60 50 40 30

22

17

20 10

0

0 1: No justification

2: Discussion‐ based justification

3: Numerical  justification

4: Not applicable

Figure 16 Assessment of justification provided for model structure selection methods

As can be seen from Figure 14, for 16 out of the 99 cases examined, the method used to select the optimal model structure was not mentioned; among the remaining 83 cases, the 34   

method used to select the optimal model structure was described in sufficient detail so the process can be repeated in 62 cases. Similar to the data splitting step, this is partially due to the fact that ad-hoc methods with easily described rules, such as trial-and-error, were often used to determine the optimal model structure; and partially due to the fact that the structure was fixed for many model architectures used in the studies reviewed, so that a method was not needed to determine the optimal structure (see Figure 15). As shown in Figure 15, optimization-based or stepwise methods were only used four and six times, respectively, in determining the best model structure in the papers reviewed. As can be seen from Figure 16, the use of a model structure selection method was not justified in 60 out of the 99 cases reviewed. Apart from the 22 cases where a justification was not required, discussion-based justification was provided in 17 cases; while numerical justification was not provided for model structure selection method used in any of the studies reviewed.

3.2.6 Model calibration The level of detail provided for the model calibration step for the 99 ANN model development processes is summarized in Figure 17. The number of times various calibration algorithms were used is given in Figure 18. The assessment of the justification provided for the selection of the model calibration methods used is summarized in Figure 19.

50

46

45

42

Number of times

40 35 30 25 20 15

11

10 5 0 1: Not mentioned

2: Mentioned

3: Repeatable 35 

 

Figure 17 Assessment of details provided for model calibration

90

80

80

Number of times

70 60 50 40 30 20 8

10

1

0 Local optimisation

Global optimisation

Bayesian

Figure 18 Number of times various calibration algorithms were used

70

64

Number of times

60 50 40 30

30 20 10

5

0 1: No justification

2: Discussion‐based  justification

3: Numerical  justification

Figure 19 Assessment of justification provided for calibration algorithm

36   

As can be seen from Figure 19, for 11 out of the 99 ANN model development processes employed, the calibration method used was not mentioned; and only for about half of the cases, the calibration method was described in sufficient detail so that the process can be repeated for the same/similar studies. As can be seen from Figure 18, deterministic methods were used in the vast majority of the cases examined; while stochastic methods, such as a Bayesian approach, was only used once. When deterministic calibration methods were used, local optimization algorithms were used in over 80% of the cases examined; while the use of global optimization algorithms was limited. It should be noted that in some of the papers reviewed, more than one training algorithm was used, as the optimal calibration algorithm was determined by comparative studies (i.e. numerical justification), as shown in Figure 19. However, in 64 out of the 99 cases, justification was not provided for the use of a particular model calibration method.

3.2.7 Model validation The level of detail provided for the model validation step in the 99 ANN model development processes is summarized in Figure 20. The number of times various aspects of validity were considered is given in Figure 21. As can be seen from Figure 20, model validation using an independent dataset was carried out in 98 of the 99 model development processes employed in the 81 papers reviewed. In these 98 cases, the validation criteria used (including replicative, predictive and structural validation criteria) were all well described with equations included or references provided. However, in the majority of the studies, only predictive validity was considered, as shown in Figure 21. Structural validity was considered 15 times; while replicative validity was considered only once. In the only paper where replicative validity was considered, a normal probability plot and the Kolmogorov–Smirnov (K-S) test were used to check if the calibration residuals followed a normal distribution and a chi-square goodnessof-fit test was used to check if the calibration residuals were largely white noise. The use of the methods was justified via discussion.

37   

120 98

Number of times

100 80 60 40 20 1

0

0

0: Not  implemented

1: Not mentioned

2: Mentioned

0

3: Repeatable

Figure 20 Assessment of model validation method details provided

120 98

Number of times

100 80 60 40

15

20 1 0 Replicative validity

Predictive validity

Structural validity

Figure 21 Number of times various aspects of validity were considered

The number of times various model predictive and structural performance criteria were used to validate the models developed in the 99 model development processes assessed is given in 38   

Figures 22(a) and 22(b), respectively. The assessment of the justification provided for the selection of model predictive and structural validation methods used is summarized in Figures 23(a) and 23(b), respectively. As can be seen form Figure 22(a), visual inspection was the most popular method for predictive model validation and was used in the majority of the studies examined. Among the other measures, efficiency indices (especially the Pearson correlation coefficient) and squared errors were the most popular model predictive performance criteria, as was found by Maier et al. (2010) for river systems. These measures were followed by absolute errors, which were used 46 times, and relative errors, which were used 26 times. Case study dependent model performance evaluation criteria (refer to Section 2.2.6), such as the pesticide detection efficiency used by Sahoo et al. (2005), were used 21 times. Information criteria were only used five times. As can be seen from Figure 22(b), sensitivity analysis was the most popular method used to assess the structural validity in the 99 ANN model development processes reviewed and was used in 14 out of the 15 cases where structural validity was considered; while structural validation based on ANN weights was used only once.

The results presented in Figure 22 also show that more than one performance criterion was considered in many of the studies investigated. This is especially the case for the predictive validity aspect of model validation. The use of multiple validation criteria is desirable, as a single model performance criterion alone often cannot test the performance of the model over the whole range of outputs (Krause et al., 2005; Dawson et al., 2007). However, in 65 out of the 98 model development processes where model predictive validity was assessed (in one study, model validation was not performed), no justification was provided for the selection of the model predictive validation method(s) used, as shown in Figure 23(a), while, in 10 out of the 15 cases where model structural validation was assessed, discussion-based justification was provided, as shown in Figure 23(b).

39   

16

90 78

80 62

50

46

40 26

30

21

20

10 8 6 4

5

10

2

0 Squared  Absolute  Relative  Efficiency  Visual  AIC/BIC error error error indices inspection

(a)

14

12

60

Number of times

Number of times

70

14

69

1

0

Others

Sensitivity analysis

(b)

Weights

Figure 22 Number of times various model validation performance criteria were used to assess (a) model predictive validity and (b) model structural validity

70

12

65

10 10

50 40

33

30 20

Number of times

Number of times

60

6

5

4 2

10

0

0

(a)

8

1: No justification

2: Discussion‐based justification

(b)

1: No justification

2: Discussion‐based justification

Figure 23 Assessment of justification provided for model validation methods used in the 99 model development process to account for (a) predictive validity and (b) structural validity

4 Summary and conclusions

The ANN model development protocol introduced in this paper and its use to critically assess the quality of the ANN model development and reporting processes are significant points of difference between this and previous review papers on the use of ANNs for water resources and environmental modelling. The protocol provides guidance on the minimum amount of information that should be provided on the ANN model development process. Provision of this information will assist with moving towards the repeatability of modelling experiments, 40   

so that the results obtained can be independently verified by other researchers and increase the level of confidence in the results presented. Both the details of and justifications for the ANN model development process reported as part of the proposed protocol drive methodological advancement in ANN modelling by assisting with the transfer of the knowledge developed in a particular study to future studies.

In this paper, the proposed protocol is used to critically assess the quality of 99 independent ANN model development and reporting processes used in 81 journal papers on the prediction and forecasting of water quality variables in drinking water system published from 2000 to 2012. This application area is selected as it is an area that has received little attention in previous reviews of ANNs in environmental/water resources modelling; while it is an area where ANNs are being used increasingly. Although there has been a significant increase in the number of papers published on the application of ANNs on the prediction of water quality variables in drinking water systems in recent years compared to a decade ago, the majority of the studies focused on the sources of drinking water, mainly reservoirs, lakes and rivers, while the application to drinking water treatment or distribution is still limited. This could be due to limited long term data of good quality for drinking water treatment and distribution systems, as pointed out by Maier et al. (2010).

A summary of the results of the assessment of the quality of the ANN model development and documentation processes employed in the 81 journal papers reviewed is provided in Figure 24. As can be seen, the three aspects of the ANN model development and reporting process included in the proposed protocol are included in the assessment, including the number of times a particular ANN model development step was implemented, documented in detail and/or justified. Each of the three aspects was first assessed independently by calculating the percentage of times each aspect was considered, as shown in Figure 24. It should be noted that the percentages for the second and third aspects were calculated based on the number of implementations (i.e. the first aspect). This is because if a step is not implemented, it cannot be reported or justified. Four assessment categories were used when assessing the three aspects: 75%, 100% means good; 50%, 75% means an aspect needs some improvement; 25%, 50% means an aspect needs major improvement; and 0%, 25% means an aspect needs significant improvement. Then, the overall status of the quality of 41   

each of the steps in the ANN model development process was assessed based on the assessment results of the three aspects and represented by a colour scheme: Green (G) means good (i.e. all three aspects fall in the first or the second category); Yellow (Y) means the implementation (i.e. the first aspect) is good, however, one of the other two aspects needs major improvement; Orange (O) means the implementation (i.e. the first aspect) is good, however, either both of the two other aspects need major improvement or one of the two other aspects needs significant improvement; and Red (R) means that the implementation step falls into the fourth category, regardless the assessment results of the other two aspects, indicating that significant improvement of the step is required. . The results obtained in each of these four assessment categories are discussed in the subsequent paragraphs.

Steps in model  development  process           Implemented

Details           Justification      Overall

Data collection  and pre‐processing Significance

99a (100%)

62b (63%) d

29c (29%) d

Y

Independence

13 (13%)

12 (92%)

12 (92%)

R

Data splitting

98 (99%)

68 (69%)

19 (19%)

O

Model architecture selection

99 (100%)

91 (92%)

64 (65%)

G

Model structure selection

99 (100%)

62 (63%)

17 (17%)

O

Model calibration

99 (100%)

46 (46%)

35 (35%)

O

Replicative

1 (1%)

1 (100%)

1 (100%)

R

Predictive

98 (99%)

98 (100%)

33 (34%)

Y

O

Structural

15 (15%)

15 (100%)

10 (67%)

R

R

Input selection

Legend

Model validation

Model application

G

Good

Y

Bad

aNumber of model  development  processes where a step was implemented;  b Number of model development  processes where the method (s) used was described  in detail;  c Number of model development  processes where  justification was provided; d Calculated  based on  number of implementation.

Figure 24 Summary of the assessment of quality of ANN model development process

42   

As can be seen from Figure 24, the majority of the proposed steps in the ANN model development process were implemented in all or all but one of the 99 model development processes reviewed. However, consideration of input independence, replicate validity and structural validity were considered in only 13, 1 and 15 ANN development processes, respectively, suggesting that consideration of these aspects of the ANN model development process require more attention. A detailed examination of the methods that were implemented at each of the steps in the model development process revealed that certain methods dominate, as was found by Maier et al. (2010) for the application of ANNs to river systems. These findings are summarised below: 1) When considering input significance, model free methods were used most of the time and in the rare cases where model based methods were used, they were often implemented in an ad-hoc fashion. 2) When considering input independence, dimensionality reduction was the most commonly used approach; while methods based on filtering were also used in a few studies investigated. 3) In relation to data splitting, unsupervised methods were used in preference to supervised methods and implemented in a random or ad-hoc fashion in the vast majority of cases. 4) A number of different model architectures, including hybrid models, were used to predict water quality variables in drinking water systems. However, feedforward MLPs were by far the most popular model architecture and were often used as a benchmark for evaluating the performance of alternative ANN architectures. Although the importance of the ability of ANNs to adapt to new/unseen data was pointed out previously (ASCE Task Committee on Application of Artificial Neural Networks in Hydrology, 2000a) and has been considered in a recent study (Bowden et al., 2012), there is no evidence in the papers reviewed that this point has attracted attention from water quality modellers. 5) Ad-hoc methods based on trial-and-error dominated the model structure selection step in the 81 papers reviewed; while there was only limited application of optimization based methods. 6) It was found that local optimization algorithms were by far the most widely used methods for model calibration (training), while the application of stochastic methods was very limited. 43   

7) Model validation was the only ANN model development step in which a variety of methods was used. Also, in most of the studies reviewed, multiple criteria were used to validate the models developed, which increases confidence in the performance of the model, as different performance characteristics have been tested. However, in most cases, only the predictive performance of the models was examined. 8) Model replicative validity was only considered in one study, where the normality of the calibration residuals was examined. 9) In terms of assessing structural validity, sensitivity analysis was the most commonly used method; while weight-based approaches were only used once.

The summary in Figure 24 indicates that in most of the papers reviewed, sufficient details of the ANN model development process were given. This is particularly the case for the steps of input selection considering input independence, model architecture selection, and model validation. While some of the steps were not implemented in many of the studies investigated (i.e. input selection considering input independence), in cases where they have been implemented, sufficient information was provided in the majority of the cases so that the steps can be repeated by other researchers. This is considered good practice, as first of all, the results obtained in the papers can be validated independently and secondly, the knowledge developed in the papers can be effectively passed on to other researchers. However, the relatively detailed description of some ANN model development steps is partially due to the fact that ad-hoc methods based on simple user defined rules were used in these steps, as discussed above. More attention should be given to the input selection considering input significance, data splitting, model structure selection and model calibration steps, where limited details on the implementation of the steps were provided in many of the papers reviewed. This is particularly the case for model calibration, for which sufficient details were only provided for fewer than half of the model development processes reviewed; while for more than 10% of the cases the method used for model calibration was not mentioned at all (see Figure 17). This raises the question of how well the models were trained in these studies, which also casts doubts on the results reported.

As can be seen from Figure 24, one common inadequacy in all six ANN model development steps for the 99 model development processes reviewed is that adequate justification for the 44   

adoption of particular methods at the different steps in the ANN model development process was often not provided. Apart from the use of input selection methods based on input independence, model architecture selection and model validation methods considering replicative and structural validity, which were justified in most cases, the use of a particular method in a specific ANN model development step was often not justified. The use of a particular data splitting method or model structure selection method was not justified in the vast majority of cases; and the use of input selection methods based on input significance, model calibration methods and model validation methods accounting for predictive validity were not justified for more than half of the model development processes assessed. In addition, in some of the cases assessed, the authors used ANNs and MLPs interchangeably and seemed not to be aware of the existence of model architectures other than MLPs.

Based on the “traffic light” assessment provided in Figure 24, the overall status of the quality of the different steps in the ANN model development process ranges from Green (good) to Red (bad). Model architecture selection is the only step receiving a green light, closely followed by input selection (significance) and predictive model validation, which are let down by a lack of justification of the selected methods.

In contrast, input selections

(independence), as well as replicative and structural model validation, are the areas requiring most attention, with data splitting, model structure selection and model calibration also requiring improvement, both in terms of the provision of details of the methods used and the justification of these methods.

5 Recommendations for future work

Based on the review of 99 ANN model development processes employed for the prediction of water quality variables in drinking water systems recorded in 81 journal papers published between 2000 and 2012, the following recommendations for future research are made: 1) Greater focus should be given to input independence in the input selection step to take into account input redundancy using methods such as partial mutual information [May et al, (2008), Fernando et al., (2009)] or tree-based iterative input variable selection 45   

(Galelli and Castelletti, 2013). The implementation of input selection was generally well documented; however, this is partially due to the fact that inputs were often selected based on data availability or using simple user-defined rules. Therefore, more analytical methods or model-based approaches should be applied to determine input significance. In addition, the use of particular input selection method(s) should be justified in order to increase confidence in the selected inputs. 2) Data splitting is one step in the development of ANN water quality models that requires more attention from ANN modellers in all three aspects covered by the protocol: the implementation of the step should be better documented so that the procedure can be repeated by other researchers; a variety of methods that suit the modelling objectives should be applied [e.g. Bowden et al. (2002), Lendasse et al. (2003), May et al., (2010) and Wu et al. (2013)], rather than using random data splitting and ad-hoc methods based on user defined rules. This in turn will justify the selection of a particular data splitting method. 3) Model architecture selection is better documented and implemented in the 81 papers reviewed compared with the other ANN model development steps. However, research on alternative model architectures, such as hybrid models or recurrent models, should continue, as suggested by (Maier et al., 2010). Justification should also be provided for the use of the particular model architecture used, at least based on previous studies/experience. 4) The implementation of model structure selection should be better documented. A variety of methods, rather than just ad-hoc methods based on trial-and-error, should be used in order to find near-optimal model structures. A rigorous model structure selection method, such as those based on optimization or Bayesian model selection, will increase confidence in model structure selection, which can also provide justification for the selection of certain model structures in future studies (Kingston et al., 2008). 5) The model calibration process should be better documented so that it can be repeated by other researchers. Global optimization methods, which may increase the chance of finding global optima in complex error surfaces, should be investigated further; and uncertainty related to calibration should also be considered in future studies, as has been done in the application of ANNs in related areas (Khan and Coulibaly, 2006; Srivastav et al., 2007; Jana et al., 2008; Zhang et al., 2009; Coulibaly, 2010; Zhang et al., 2011; Yacef et al., 2012; Zhang and Zhao, 2012; Kasiviswanathan et al., 2013). 46   

6) Although model validation is by far the best documented ANN model development step with a variety of methods for estimating model predictive performance employed, greater focus should be given to the other two aspects of model validity, namely replicative and structural validity; especially structurally validity, which can be used for knowledge extraction and to legitimize the developed model (Abrahart et al., 2012). Consequently, methods that test whether the relationships developed using ANNs are consistent with a priori understanding of the underlying physical processes, such as sensitivity analysis, overall connection weights and comparing modelling results with a priori knowledge of the system [e.g. Jain et al., (2004); Kingston et al., (2005b); Mount et al., (2013)], should be implemented on a routine basis. In addition, the selection of a model validation method should be based on the purposes for which the model is developed and the problem presented in the paper, rather than simply selecting commonly used measures (Bennett et al., 2013). 7) The uncertainty of ANN outputs, especially those resulting from limited data, is another area that needs more attention from ANN water quality modellers. This issue can be addressed through implementing calibration methods incorporating uncertainty and assessing the structural validity of developed ANN models, as discussed above. 8) The applicability of ANNs to water quality modelling in water supply systems needs to be investigated by comparing ANNs with other modelling techniques that are traditionally used in the area. This could be an interesting topic of a future review paper.

6 Acknowledgements The authors would like to thank Water Research Australia for its financial support for this study.

7 References

Abrahart, R. J., See, L. M., and Dawson, C. W. 2008. "Neural Network Hydroinformatics: Maintaining Scientific Rigour." Practical Hydroinformatics, R. Abrahart, L. See, and D. Solomatine, eds., Springer Berlin Heidelberg, 33-47. 47   

Abrahart, R. J., Dawson, C. W., See, L. M., Mount, N. J., and Shamseldin, A. Y. 2010. "Discussion of “Evapotranspiration modelling using support vector machines”*." Hydrological Sciences Journal, 55(8), 1442-1450. Abrahart, R. J., Anctil, F., Coulibaly, P., Dawson, C. W., Mount, N. J., See, L. M., Shamseldin, A. Y., Solomatine, D. P., Toth, E., and Wilby, R. L. 2012. "Two decades of anarchy? Emerging themes and outstanding challenges for neural network river forecasting." Progress in Physical Geography. Abudu, S., King, J. P., and Sheng, Z. P. 2012. "Comparison of the Performance of Statistical Models in Forecasting Monthly Total Dissolved Solids in the Rio Grande." Journal of the American Water Resources Association, 48(1), 10-23. Ahmadi-Nedushan, B., St-Hilaire, A., Bérubé, M., Robichaud, É., Thiémonge, N., and Bobée, B. 2006. "A review of statistical methods for the evaluation of aquatic habitat suitability for instream flow assessment." River Research and Applications, 22(5), 503-523. Akkoyunlu, A. and Akiner, M. E. 2010. "Feasibility Assessment of Data-Driven Models in Predicting Pollution Trends of Omerli Lake, Turkey." Water Resources Management, 24(13), 3419-3436. Akkoyunlu, A., Altun, H., and Cigizoglu, H. K. 2011. "Depth-Integrated Estimation of Dissolved Oxygen in a Lake." Journal of Environmental Engineering-Asce, 137(10), 961-967. Asadollahfardi, G., Taklify, A., and Ghanbari, A. 2012. "Application of Artificial Neural Network to Predict TDS in Talkheh Rud River." Journal of Irrigation and Drainage Engineering-Asce, 138(4), 363-370. ASCE Task Committee on Application of Artificial Neural Networks in Hydrology. 2000a. "Artificial Neural Networks in Hydrology. I: Preliminary Concepts." Journal of Hydrologic Engineering, 5(2). ASCE Task Committee on Application of Artificial Neural Networks in Hydrology. 2000b. "Artificial Neural Networks in Hydrology. II: Hydrologic Applications." Journal of Hydrologic Engineering, 5(2). Ay, M. and Kisi, O. 2012. "Modeling of Dissolved Oxygen Concentration Using Different Neural Network Techniques in Foundation Creek, El Paso County, Colorado." Journal of Environmental Engineering, 138(6), 654-662.

48   

Banerjee, P., Singh, V. S., Chatttopadhyay, K., Chandra, P. C., and Singh, B. 2011. "Artificial neural network model as a potential alternative for groundwater salinity forecasting." Journal of Hydrology, 398(3-4), 212-220. Bashi-Azghadi, S. N., Kerachian, R., Bazargan-Lari, M. R., and Solouki, K. 2010. "Characterizing an unknown pollution source in groundwater resources systems using PSVM and PNN." Expert Systems with Applications, 37(10), 7154-7161. Bennett, N. D., Croke, B. F. W., Guariso, G., Guillaume, J. H. A., Hamilton, S. H., Jakeman, A. J., Marsili-Libelli, S., Newham, L. T. H., Norton, J. P., Perrin, C., Pierce, S. A., Robson, B., Seppelt, R., Voinov, A. A., Fath, B. D., and Andreassian, V. 2013. "Characterising performance of environmental models." Environmental Modelling & Software, 40(0), 1-20. Bowden, G. J., Maier, H. R., and Dandy, G. C. 2002. "Optimal division of data for neural network models in water resources applications." Water Resour. Res., 38(2), 1010. Bowden, G. J., Dandy, G. C., and Maier, H. R. 2005a. "Input determination for neural network models in water resources applications. Part 1 - background and methodology." Journal of Hydrology, 301(1-4), 75-92. Bowden, G. J., Maier, H. R., and Dandy, G. C. 2005b. "Input determination for neural network models in water resources applications. Part 2. Case study: forecasting salinity in a river." Journal of Hydrology, 301(1-4), 93-107. Bowden, G. J., Nixon, J. B., Dandy, G. C., Maier, H. R., and Holmes, M. 2006. "Forecasting chlorine residuals in a water distribution system using a general regression neural network." Mathematical and Computer Modelling, 44(5-6), 469-484. Bowden, G. J., Maier, H. R., and Dandy, G. C. 2012. "Real-time deployment of artificial neural network forecasting models: Understanding the range of applicability." Water Resources Research, 48(10), W10549. Brion, G. M., Neelakantan, T. R., and Lingireddy, S. 2001. "Using neural networks to predict peak Cryptosporidium concentrations." Journal American Water Works Association, 93(1), 99-105. Cannon, A. J. and Whitfield, P. H. 2001. "Modeling transient pH depressions in coastal streams of British Columbia using neural networks." Journal of the American Water Resources Association, 37(1), 73-89. Castelletti, A., Galelli, S., Restelli, M., and Soncini-Sessa, R. 2012. "Data-driven dynamic emulation modelling for the optimal management of environmental systems." Environmental Modelling & Software, 34(0), 30-43. 49   

Céréghino, R. and Park, Y. S. 2009. "Review of the Self-Organizing Map (SOM) approach in water resources: Commentary." Environmental Modelling & Software, 24(8), 945-947. Chang, L. C., Chu, H. J., and Hsiao, C. T. 2012. "Integration of Optimal Dynamic Control and Neural Network for Groundwater Quality Management." Water Resources Management, 26(5), 1253-1269. Chaves, P. and Kojiri, T. 2007a. "Deriving reservoir operational strategies considering water quantity and quality objectives by stochastic fuzzy neural networks." Advances in Water Resources, 30(5), 1329-1341. Chaves, P. and Kojiri, T. 2007b. "Conceptual fuzzy neural network model for water quality simulation." Hydrological Processes, 21(5), 634-646. Chen, L., Kao, S.-J., and Traore, S. 2012. "Predicting and managing reservoir total phosphorus by using modified grammatical evolution coupled with a macro-genetic algorithm." Environmental Modelling & Software, 38(0), 89-100. Chen, Q. W. and Mynett, A. E. 2003. "Integration of data mining techniques and heuristic knowledge in fuzzy logic modelling of eutrophication in Taihu Lake." Ecological Modelling, 162(1-2), 55-67. Chen, Q. W., Yang, Q. R., and Lin, Y. Q. 2011. "Development and application of a hybrid model to analyze spatial distribution of macroinvertebrates under flow regulation in the Lijiang River." Ecological Informatics, 6(6), 407-413. Chibanga, R., Berlamont, J., and Vandewalle, J. 2003. "Modelling and forecasting of hydrological variables using artificial neural networks: the Kafue River sub-basin." Hydrological Sciences Journal, 48(3), 363-379. Cho, K. H., Sthiannopkao, S., Pachepsky, Y. A., Kim, K. W., and Kim, J. H. 2011. "Prediction of contamination potential of groundwater arsenic in Cambodia, Laos, and Thailand using artificial neural network." Water Research, 45(17), 5535-5544. Coulibaly, P. 2010. "Reservoir Computing approach to Great Lakes water level forecasting." Journal of Hydrology, 381(1–2), 76-88. D'Souza, C. D. and Kumar, M. S. M. 2010. "Comparison of ANN models for predicting water quality in distribution systems." Journal American Water Works Association, 102(7), 92-+. da Costa, A. O. S., Silva, P. F., Sabara, M. G., and da Costa, E. J. D. 2009. "Use of neural networks for monitoring surface water quality changes in a neotropical urban stream." Environmental Monitoring and Assessment, 155(1-4), 527-538.

50   

Dawson, C. W. and Wilby, R. L. 2001. "Hydrological modelling using artificial neural networks." Progress in Physical Geography, 25(1), 80-108. Dawson, C. W., Abrahart, R. J., and See, L. M. 2007. "HydroTest: A web-based toolbox of evaluation metrics for the standardised assessment of hydrological forecasts." Environ. Model. Softw., 22(7), 1034-1052. Diamantopoulou, M. J., Antonopoulos, V. Z., and Papamichail, D. M. 2007. "Cascade correlation artificial neural networks for estimating missing monthly values of water quality parameters in rivers." Water Resources Management, 21(3), 649-662. Dixon, B. 2009. "A case study using support vector machines, neural networks and logistic regression in a GIS to identify wells contaminated with nitrate-N." Hydrogeology Journal, 17(6), 1507-1520. Dogan, E., Sengorur, B., and Koklu, R. 2009. "Modeling biological oxygen demand of the Melen River in Turkey using an artificial neural network technique." Journal of Environmental Management, 90(2), 1229-1235. Elhatip, H. and Komur, M. A. 2008. "Evaluation of water quality parameters for the Mamasin dam in Aksaray City in the central Anatolian part of Turkey by means of artificial neural networks." Environmental Geology, 53(6), 1157-1164. Elshorbagy, A., Teegavarapu, R. S. V., and Ormsbee, L. 2005. "Framework for assessment of relative pollutant loads in streams with limited data." Water International, 30(4), 477486. Elshorbagy, A., Corzo, G., Srinivasulu, S., and Solomatine, D. P. 2010a. "Experimental investigation of the predictive capabilities of data driven modeling techniques in hydrology - Part 1: Concepts and methodology." Hydrol. Earth Syst. Sci., 14, 19311941. Elshorbagy, A., Corzo, G., Srinivasulu, S., and Solomatine, D. P. 2010b. "Experimental investigation of the predictive capabilities of data driven modeling techniques in hydrology - Part 2: Application." Hydrol. Earth Syst. Sci., 14, 1943-1961. Faruk, D. O. 2010. "A hybrid neural network and ARIMA model for water quality time series prediction." Engineering Applications of Artificial Intelligence, 23(4), 586-594. Fernando, T. M. K. G., Maier, H. R., and Dandy, G. C. 2009. "Selection of input variables for data driven models: An average shifted histogram partial mutual information estimator approach." Journal of Hydrology, 367(3-4), 165-176. Galelli, S. and Castelletti, A. 2013. "Tree-based iterative input variable selection for hydrological modeling." Water Resources Research, n/a-n/a. 51   

Gardner, M. W. and Dorling, S. R. 1998. "Artificial neural networks (the multilayer perceptron)—a review of applications in the atmospheric sciences." Atmospheric Environment, 32(14–15), 2627-2636. Gass, S. I., Hoffman, K. L., Jackson, R. H. F., Joel, L. S., and Saunders, P. B. 1981. "Documentation for a Model - A Hierarchical Approach." Communications of the Acm, 24(11), 728-733. Gass, S. I. 1983. "Decision-Aiding Models: Validation, Assessment, and Related Issues for Policy Analysis." Operations Research, 31(4), 603-631. Gemitzi, A., Petalas, C., Pisinaras, V., and Tsihrintzis, V. A. 2009. "Spatial prediction of nitrate pollution in groundwaters using neural networks and GIS: an application to South Rhodope aquifer (Thrace, Greece)." Hydrological Processes, 23(3), 372-383. Guo, L., Zhao, Y., and Wang, P. 2012. "Determination of the principal factors of river water quality through cluster analysis method and its prediction." Frontiers of Environmental Science & Engineering, 6(2), 238-245. Haas, C. N. 2004. "Neural networks provide superior description of Giardia lamblia inactivation by free chlorine." Water Research, 38(14-15), 3449-3457. Hamm, L., Brorsen, B. W., and Hagan, M. 2007. "Comparison of Stochastic Global Optimization Methods to Estimate Neural Network Weights." Neural Processing Letters, 26(3), 145-158. He, B., Oki, T., Sun, F. B., Komori, D., Kanae, S., Wang, Y., Kim, H., and Yamazaki, D. 2011a. "Estimating monthly total nitrogen concentration in streams by using artificial neural network." Journal of Environmental Management, 92(1), 172-177. He, J. X., Valeo, C., Chu, A., and Neumann, N. F. 2011b. "Stormwater quantity and quality response to climate change using artificial neural networks." Hydrological Processes, 25(8), 1298-1312. He, J. X., Valeo, C., Chu, A., and Neumann, N. F. 2011c. "Prediction of event-based stormwater runoff quantity and quality by ANNs developed using PMI-based input selection." Journal of Hydrology, 400(1-2), 10-23. Jain, A., Sudheer, K. P., and Srinivasulu, S. 2004. "Identification of physical processes inherent in artificial neural network rainfall runoff models." Hydrological Processes, 18(3), 571-581. Jain, A. and Kumar, S. 2009. "Dissection of trained neural network hydrologic models for knowledge extraction." Water Resources Research, 45(7), W07420.

52   

Jain, A., Maier, H. R., Dandy, G. C., and Sudheer, K. P. 2009. "Rainfall Runoff Modelling using neural Networks: State-of-the-Art and Future Research Needs." ISH Journal of Hydraulic Engineering, 15(sup1), 52-74. Jakeman, A. J., Letcher, R. A., and Norton, J. P. 2006. "Ten iterative steps in development and evaluation of environmental models." Environmental Modelling & Software, 21(5), 602-614. Jana, R. B., Mohanty, B. P., and Springer, E. P. 2008. "Multiscale Bayesian neural networks for soil water content estimation." Water Resources Research, 44(8), W08408. Jeong, K. S., Kim, D. K., Jung, J. M., Kim, M. C., and Joo, G. J. 2008. "Non-linear autoregressive modelling by Temporal Recurrent Neural Networks for the prediction of freshwater phytoplankton dynamics." Ecological Modelling, 211(3-4), 292-300. Juahir, H., Zain, S. M., Aris, A. Z., Yusoff, M. K., and Bin Mokhtar, M. 2010. "Spatial assessment of Langat river water quality using chemometrics." Journal of Environmental Monitoring, 12(1), 287-295. Kalin, L., Isik, S., Schoonover, J. E., and Lockaby, B. G. 2010. "Predicting Water Quality in Unmonitored Watersheds Using Artificial Neural Networks." Journal of Environmental Quality, 39(4), 1429-1440. Kalogirou, S. A. 2001. "Artificial neural networks in renewable energy systems applications: a review." Renewable and Sustainable Energy Reviews, 5(4), 373-401. Kalteh, A. M., Hjorth, P., and Berndtsson, R. 2008. "Review of the self-organizing map (SOM) approach in water resources: Analysis, modelling and application." Environmental Modelling & Software, 23(7), 835-845. Karamouz, M., Razavi, S., and Araghinejad, S. 2008. "Long-lead seasonal rainfall forecasting using time-delay recurrent neural networks: a case study." Hydrological Processes, 22(2), 229-241. Kasiviswanathan, K. S., Cibin, R., Sudheer, K. P., and Chaubey, I. 2013. "Constructing prediction interval for artificial neural network rainfall runoff models based on ensemble simulations." Journal of Hydrology, 499(0), 275-288. Khalil, A., Almasri, M. N., McKee, M., and Kaluarachchi, J. J. 2005. "Applicability of statistical learning algorithms in groundwater quality modeling." Water Resources Research, 41(5). Khalil, B., Ouarda, T., and St-Hilaire, A. 2011. "Estimation of water quality characteristics at ungauged sites using artificial neural networks and canonical correlation analysis." Journal of Hydrology, 405(3-4), 277-287. 53   

Khan, M. S. and Coulibaly, P. 2006. "Bayesian neural network for rainfall-runoff modeling." Water Resources Research, 42(7), W07409. Kilic, H., Soyupak, S., Tuzun, I., Ince, O., and Basaran, G. 2007. "An automata networks based preprocessing technique for artificial neural network modelling of primary production levels in reservoirs." Ecological Modelling, 201(3-4), 359-368. Kim, M. E., Shon, T. S., Min, K. S., and Shin, H. S. 2012a. "Forecasting performance of algae blooms based on artificial neural networks and automatic observation system." Desalination and Water Treatment, 38(1-3), 293-301. Kim, R., Loucks, D., and Stedinger, J. 2012b. "Artificial Neural Network Models of Watershed Nutrient Loading." Water Resources Management, 26(10), 2781-2797. Kingston, G. B., Lambert, M. F., and Maier, H. R. 2005a. "Bayesian training of artificial neural networks used for water resources modeling." Water Resour. Res., 41(12), W12409. Kingston, G. B., Maier, H. R., and Lambert, M. F. 2005b. "Calibration and validation of neural networks to ensure physically plausible hydrological modeling." Journal of Hydrology, 314(1–4), 158-176. Kingston, G. B., Maier, H. R., and Lambert, M. F. 2006. "A probabilistic method for assisting knowledge extraction from artificial neural networks used for hydrological prediction." Mathematical and Computer Modelling, 44(5–6), 499-512. Kingston, G. B., Maier, H. R., and Lambert, M. F. 2008. "Bayesian model selection applied to artificial neural networks used for water resources modeling." Water Resources Research, 44(4), W04419. Krause, P., Boyle, D. P., and Bäse, F. 2005. "Comparison of different efficiency criteria for hydrological model assessment." Adv. Geosci., 5, 89-97. Kulkarni, P. and Chellam, S. 2010. "Disinfection by-product formation following chlorination of drinking water: Artificial neural network models and changes in speciation with treatment." Science of the Total Environment, 408(19), 4202-4210. Kuo, J. T., Wang, Y. Y., and Lung, W. S. 2006. "A hybrid neural-genetic algorithm for reservoir water quality management." Water Research, 40(7), 1367-1376. LeBaron, B. and Weigend, A. S. 1998. "A bootstrap evaluation of the effect of data splitting on financial time series." Neural Networks, IEEE Transactions on, 9(1), 213-220. Lendasse, A., Wertz, V., and Verleysen, M. 2003. "Model selection with cross-validations and bootstraps: application to time series prediction with RBFN models." Proceedings

54   

of the 2003 joint international conference on Artificial neural networks and neural information processing, Springer-Verlag, Istanbul, Turkey. Lewin, N., Zhang, Q., Chu, L. L., and Shariff, R. 2004. "Predicting total trihalomethane formation in finished water using artificial neural networks." Journal of Environmental Engineering and Science, 3, S35-S43. Li, X. F., Nour, M. H., Smith, D. W., and Prepas, E. E. 2008. "Modelling nitrogen composition in streams on the Boreal Plain using genetic adaptive general regression neural networks." Journal of Environmental Engineering and Science, 7, S109-S125. Liu, W.-C. and Chen, W.-B. 2012. "Prediction of water temperature in a subtropical subalpine lake using an artificial neural network and three-dimensional circulation models." Computers & Geosciences, 45(0), 13-25. Maier, H. R. and Dandy, G. C. 1999. "Empirical comparison of various methods for training feed-Forward neural networks for salinity forecasting." Water Resources Research, 35(8), 2591-2596. Maier, H. R. and Dandy, G. C. 2000. "Neural networks for the prediction and forecasting of water resources variables: a review of modelling issues and applications." Environmental Modelling & Software, 15(1), 101-124. Maier, H. R., Sayed, T., and Lence, B. J. 2000. "Forecasting cyanobacterial concentrations using B-spline networks." Journal of Computing in Civil Engineering, 14(3), 183-189. Maier, H. R., Morgan, N., and Chow, C. W. K. 2004. "Use of artificial neural networks for predicting optimal alum doses and treated water quality parameters." Environmental Modelling & Software, 19(5), 485-494. Maier, H. R., Jain, A., Dandy, G. C., and Sudheer, K. P. 2010. "Methods used for the development of neural networks for the prediction of water resource variables in river systems: Current status and future directions." Environmental Modelling & Software, 25(8), 891-909. Maier, H. R. 2013. "What constitutes a good literature review and why does its quality matter?" Environmental Modelling & Software, 43, 3-4. Markus, M., Tsai, C. W. S., and Demissie, M. 2003. "Uncertainty of weekly nitrate-nitrogen forecasts using artificial neural networks." Journal of Environmental EngineeringAsce, 129(3), 267-274. Mas, D. M. L. and Ahlfeld, D. P. 2007. "Comparing artificial neural networks and regression models for predicting faecal coliform concentrations." Hydrological Sciences Journal-Journal Des Sciences Hydrologiques, 52(4), 713-731. 55   

May, D. B. and Sivakumar, M. 2009. "Prediction of urban stormwater quality using artificial neural networks." Environmental Modelling & Software, 24(2), 296-302. May, R. J., Dandy, G. C., Maier, H. R., and Nixon, J. B. 2008. "Application of partial mutual information variable selection to ANN forecasting of water quality in water distribution systems." Environmental Modelling & Software, 23(10-11), 1289-1299. May, R. J., Maier, H. R., and Dandy, G. C. 2010. "Data splitting for artificial neural networks using SOM-based stratified sampling." Neural Networks, 23(2), 283-294. Melesse, A. M., Ahmad, S., McClain, M. E., Wang, X., and Lim, Y. H. 2011. "Suspended sediment load prediction of river systems: An artificial neural network approach." Agricultural Water Management, 98(5), 855-866. Mellit, A., Kalogirou, S. A., Hontoria, L., and Shaari, S. 2009. "Artificial intelligence techniques for sizing photovoltaic systems: A review." Renewable and Sustainable Energy Reviews, 13(2), 406-419. Merdun, H. and Cinar, O. 2010. "Utilization of two artificial neural network methods in surface water quality modelling." Environmental Engineering and Management Journal, 9(3), 413-421. Millie, D. F., Weckman, G. R., Young Ii, W. A., Ivey, J. E., Carrick, H. J., and Fahnenstiel, G. L. 2012. "Modeling microalgal abundance with artificial neural networks: Demonstration of a heuristic ‘Grey-Box’ to deconvolve and quantify environmental influences." Environmental Modelling & Software, 38(0), 27-39. Milot, J., Rodriguez, M. J., and Serodes, J. B. 2002. "Contribution of neural networks for modeling trihalomethanes occurrence in drinking water." Journal of Water Resources Planning and Management-Asce, 128(5), 370-376. Mount, N. J., Dawson, C. W., and Abrahart, R. J. 2013. "Legitimising data-driven models: exemplification of a new data-driven mechanistic modelling framework." Hydrol. Earth Syst. Sci., 17, 2827-2843. Najah, A., El-Shafie, A., Karim, O. A., and Jaafar, O. 2012. "Water quality prediction model utilizing integrated wavelet-ANFIS model with cross-validation." Neural Computing and Applications, 21(5), 833-841. Neelakantan, T. R., Lingireddy, S., and Brion, G. M. 2002. "Effectiveness of different artificial neural network training algorithms in predicting protozoa risks in surface waters." Journal of Environmental Engineering-Asce, 128(6), 533-542. Nikoo, M. R., Kerachian, R., Malakpour-Estalaki, S., Bashi-Azghadi, S. N., and AzimiGhadikolaee, M. M. 2011. "A probabilistic water quality index for river water quality 56   

assessment: a case study." Environmental Monitoring and Assessment, 181(1-4), 465478. Ogleni, N. and Topal, B. 2011. "Water Quality Assessment of the Mudurnu River, Turkey, Using Biotic Indices." Water Resources Management, 25(10), 2487-2508. Olden, J. D. and Jackson, D. A. 2002. "Illuminating the “black box”: a randomization approach for understanding variable contributions in artificial neural networks." Ecological Modelling, 154(1–2), 135-150. Park, J. I., Jung, N. C., Kwak, K. C., and Chun, M. G. 2010. "Water Quality Prediction in a Reservoir: Linguistic Model Approach for Interval Prediction." International Journal of Control Automation and Systems, 8(4), 868-874. Perelman, L., Arad, J., Housh, M., and Ostfeld, A. 2012. "Event Detection in Water Distribution Systems from Multivariate Water Quality Time Series." Environmental Science & Technology, 46(15), 8212-8219. Philibert, A., Gell, P., Newall, P., Chessman, B., and Bate, N. 2006. "Development of diatom-based tools for assessing stream water quality in south-eastern Australia: assessment of environmental transfer functions." Hydrobiologia, 572, 103-114. Power, M. 1993. "The predictive validation of ecological and environmental models." Ecological Modelling, 68(1–2), 33-50. Rankovic, V., Radulovic, J., Radojevic, I., Ostojic, A., and Comic, L. 2010. "Neural network modeling of dissolved oxygen in the Gruza reservoir, Serbia." Ecological Modelling, 221(8), 1239-1244. Rankovic, V., Radulovic, J., Radojevic, I., Ostojic, A., and Comic, L. 2012. "Prediction of dissolved oxygen in reservoirs using adaptive network-based fuzzy inference system." Journal of Hydroinformatics, 14(1), 167-179. Razavi, S. and Tolson, B. A. 2011. "A New Formulation for Feedforward Neural Networks." Neural Networks, IEEE Transactions on, 22(10), 1588-1598. Razavi, S., Tolson, B. A., and Burn, D. H. 2012. "Numerical assessment of metamodelling strategies in computationally intensive optimization." Environmental Modelling & Software, 34(0), 67-86. Refsgaard, J. C. and Henriksen, H. J. 2004. "Modelling guidelines––terminology and guiding principles." Advances in Water Resources, 27(1), 71-82. Refsgaard, J. C., Henriksen, H. J., Harrar, W. G., Scholten, H., and Kassahun, A. 2005. "Quality assurance in model based water management – review of existing practice

57   

and outline of new approaches." Environmental Modelling & Software, 20(10), 12011215. Robson, B. J., Hamilton, D. P., Webster, I. T., and Chan, T. 2008. "Ten steps applied to development and evaluation of process-based biogeochemical models of estuaries." Environmental Modelling & Software, 23(4), 369-384. Rodriguez, M. J., Milot, J., Serodes, J. B., and Pacaud, A. 2002. "Estimation of bench-scale chlorine decay in drinking water using n(th)-order kinetic and back propagation neural network models." Water Quality Research Journal of Canada, 37(3), 613-635. Rodriguez, M. J., Milot, J., and Serodes, J. B. 2003. "Predicting trihalomethane formation in chlorinated waters using multivariate regression and neural networks." Journal of Water Supply Research and Technology-Aqua, 52(3), 199-215. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. 1986. "Learning representations by back-propagating errors." Nature, 323(6088), 533-536. Sahoo, G. B., Ray, C., and Wade, H. F. 2005. "Pesticide prediction in ground water in North Carolina domestic wells using artificial neural networks." Ecological Modelling, 183(1), 29-46. Sahoo, G. B., Ray, C., Mehnert, E., and Keefer, D. A. 2006. "Application of artificial neural networks to assess pesticide contamination in shallow groundwater." Science of the Total Environment, 367(1), 234-251. Sahoo, G. B., Schladow, S. G., and Reuter, J. E. 2009. "Forecasting stream water temperature using regression analysis, artificial neural network, and chaotic non-linear dynamic models." Journal of Hydrology, 378(3-4), 325-342. Scholten, H. and Udink ten Cate, A. J. 1999. "Quality assessment of the simulation modeling process." Computers and Electronics in Agriculture, 22(2–3), 199-208. Scholten, H., Van Waveren, R. H., Groot, S., Van Geer, F. C., Wosten, J. H. M., Koeze, R. D., and Noort, J. J. 2001. "Improving the quality of model-based decision support: good modelling practice in water management." Regional Management of Water Resources, A. H. Schumann, M. C. Acreman, R. Davis, M. A. Marino, D. Rosbjerg, and X. Jun, eds., 223-230. Serodes, J. B., Rodriguez, M. J., and Ponton, A. 2001. "Chlorcast (c): a methodology for developing decision-making tools for chlorine disinfection control." Environmental Modelling & Software, 16(1), 53-62.

58   

Sharma, A. 2000. "Seasonal to interannual rainfall probabilistic forecasts for improved water supply management: Part 1 -- A strategy for system predictor identification." Journal of Hydrology, 239(1-4), 232-239. Sharma, N., Chaudhry, K. K., and Chalapati-Rao, C. V. 2005. "Vehicular pollution modeling using artificial neural network technique: A review." JOURNAL OF SCIENTIFIC & INDUSTRIAL RESEARCH, 64(9), 637-647. Shetty, G. R., Malki, H., and Chellam, S. 2003. "Predicting contaminant removal during municipal drinking water nanofiltration using artificial neural networks." Journal of Membrane Science, 212(1-2), 99-112. Solomatine, D. P. and Ostfeld, A. 2008. Data-driven modelling : some past experiences and new approaches, IWA Publishing, London. Song, K., Li, L., Wang, Z., Liu, D., Zhang, B., Xu, J., Du, J., Li, L., Li, S., and Wang, Y. 2012. "Retrieval of total suspended matter (TSM) and chlorophyll-a (Chl-a) concentration from remote-sensing data for drinking water resources." Environmental Monitoring and Assessment, 184(3), 1449-1470. Soyupak, S., Karaer, F., Gurbuz, H., Kivrak, E., Senturk, E., and Yazici, A. 2003. "A neural network-based approach for calculating dissolved oxygen profiles in reservoirs." Neural Computing & Applications, 12(3-4), 166-172. Soyupak, S., Kilic, H., Karadirek, I. E., and Muhammetoglu, H. 2011. "On the usage of artificial neural networks in chlorine control applications for water distribution networks with high quality water." Journal of Water Supply Research and Technology-Aqua, 60(1), 51-60. Srivastav, R. K., Sudheer, K. P., and Chaubey, I. 2007. "A simplified approach to quantifying predictive and parametric uncertainty in artificial neural network hydrologic models." Water Resources Research, 43(10), W10407. Suen, J. P. and Eheart, J. W. 2003. "Evaluation of neural networks for modeling nitrate concentrations in rivers." Journal of Water Resources Planning and ManagementAsce, 129(6), 505-510. Tayfur, G. and Guldal, V. 2006. "Artificial neural networks for estimating daily total suspended sediment in natural streams." Nordic Hydrology, 37(1), 69-79. Teles, L. O., Vasconcelos, V., Pereira, E., and Saker, M. 2006. "Time series forecasting of cyanobacteria blooms in the Crestuma Reservoir (Douro River, Portugal) using artificial neural networks." Environmental Management, 38(2), 227-237.

59   

Tota-Maharaj, K. and Scholz, M. 2012. "Artificial Neural Network Simulation of Combined Permeable Pavement and Earth Energy Systems Treating Storm Water." Journal of Environmental Engineering-Asce, 138(4), 499-509. Tufail, M., Ormsbee, L., and Teegavarapu, R. 2008. "Artificial intelligence-based inductive models for prediction and classification of fecal coliform in surface waters." Journal of Environmental Engineering-Asce, 134(9), 789-799. Ulke, A., Tayfur, G., and Ozkul, S. 2009. "Predicting Suspended Sediment Loads and Missing Data for Gediz River, Turkey." Journal of Hydrologic Engineering, 14(9), 954-965. Welsh, W. D. 2008. "Water balance modelling in Bowen, Queensland, and the ten iterative steps in model development and evaluation." Environmental Modelling & Software, 23(2), 195-205. Wu, W., May, R., Dandy, G. C., and Maier, H. R. 2012. "A method for comparing data splitting approaches for developing hydrological ANN models." The 6th International Congress on Environmental Modelling and Software (iEMSs), Leipzig, Germany. Wu, W., May, R. J., Maier, H. R., and Dandy, G. C. 2013. "A benchmarking approach for comparing data splitting methods for modeling water resources parameters using artificial neural networks." Water Resources Research, n/a-n/a. Yacef, R., Benghanem, M., and Mellit, A. 2012. "Prediction of daily global solar irradiation data using Bayesian neural network: A comparative study." Renewable Energy, 48(0), 146-154. Ye, B. X., Wang, W. Y., Yang, L. S., Wei, J. R., and Xueli, E. 2011. "Formation and modeling of disinfection by-products in drinking water of six cities in China." Journal of Environmental Monitoring, 13(5), 1271-1275. Yeon, I. S., Kim, J. H., and Jun, K. W. 2008. "Application of artificial intelligence models in water quality forecasting." Environmental Technology, 29(6), 625-631. Zhang, X., Liang, F., Srinivasan, R., and Van Liew, M. 2009. "Estimating uncertainty of streamflow simulation using Bayesian neural networks." Water Resources Research, 45(2), W02403. Zhang, X., Liang, F., Yu, B., and Zong, Z. 2011. "Explicitly integrating parameter, input, and structure uncertainties into Bayesian Neural Networks for probabilistic hydrologic forecasting." Journal of Hydrology, 409(3–4), 696-709.

60   

Zhang, X. and Zhao, K. 2012. "Bayesian Neural Networks for Uncertainty Analysis of Hydrologic Modeling: A Comparison of Two Schemes." Water Resources Management, 26(8), 2365-2382. Ziegler, B. P. 1976. Theory of Modelling and Simulation, John Wiley and Sons, New York. Zou, R., Lung, W. S., and Guo, H. C. 2002. "Neural network embedded Monte Carlo approach for water quality modeling under input information uncertainty." Journal of Computing in Civil Engineering, 16(2), 135-142.

61   

Appendix A List of journals and their 2012 ISI impact factors

Journal Advances in Water Resources Agricultural Water Management Computers and Geosciences Desalination and Water Treatment Ecological Informatics Ecological Mmodelling Engineering Applications of Artificial Intelligence Environmental Engineering and Management Journal Environmental Geology* Environmental Management Environmental Modelling and Software Environmental Monitoring and Assessment Environmental Science and Technology Environmental Technology Frontiers of Environmental Science and Engineering Hydrobiologia Hydrology Journal Hydrological Processes Hydrological Sciences Journal-Journal Des Sciences Hydrologiques International Journal of Control Automation and Systems Journal American Water Works Association Journal of Computing in Civil Engineering, ASCE Journal of Environmental Engineering and Science** Journal of Environmental Engineering, ASCE Journal of Environmental Management Journal of Environmental Monitoring Journal of Environmental Quality Journal of Hydroinformatics Journal of Hydrologic Engineering, ASCE Journal of Hydrology Journal of Irrigation and Drainage Engineering, ASCE Journal of Membrane Science Journal of the American Water Resources Association Journal of Water Resources Planning and Management, ASCE Journal of Water Supply Research and Technology Mathematical and Computer Modelling Neural Computing and Applications

2012 impact factor 2.412 2.203 1.834 0.852 1.961 2.069 1.625 1.004 1.445 1.74 3.476 1.592 5.257 1.606 0.886 1.985 1.675 2.497 1.54 0.953 0.86 1.34 0.67 1.117 3.057 2.085 2.353 1.153 1.379 2.964 1.126 4.093 1.956 1.709 0.573 1.42 1.168 62 

 

Nordic Hydrology*** 1.156 Science of the Total Environment 3.258 Water International 0.705 Water Quality Research Journal of Canada 0.39 Water Research 4.655 Water Resources Management 2.259 Water Resources Research 3.149 *The title of the journal was changed to Environmental Earth Sciences in 2009. **The journal was incorporated into Canadian Journal of Civil Engineering in 2011. The impact factor shown here is the 2010 impact factor. ***The title of the journal was changed to Hydrology Research in 2008.

63   

Appendix B Details of papers reviewed

Author(s) (year)

Region of study

Application area

Water quality variables modelled

Maier et al. (2000)

River Murray, South Australia, Australia

River

Biological

Brion et al. (2001)

Delaware River, New Jersey, USA

River

Biological

Cannon and Whitfield (2001)

Kanaka Creek and Serpentine River watershed, British Columbia, Canada

River

Physical

Serodes et al. (2001)

City of Sainte-Foy, Canada

Distribution

Chemical

Milot et al. (2002)

Quebec, Canada

Distribution

Chemical

Neelakantan et al. (2002)

Delaware River, New Jersey, USA

River

Biological

Rodriguez et al. (2002)

Quebec City area, Canada

Treatment

Chemical

Zou et al. (2002)

Triadelphia Reservoir, Maryland, USA

Reservoir/Lake

Chemical

Chen and Mynett (2003)

Taihu Lake, China

Reservoir/Lake

Biological

Markus et al. (2003)

Sangamon River, Illinois, USA

River

Chemical

Rodriguez et al. (2003)

USA and Canada

Treatment

Chemical

Shetty et al. (2003)

USA

Treatment

Chemical, physical

Soyupak et al. (2003)

Keban, Kuzgun and Doganci Dam Reservoirs, Turkey

Reservoir/Lake

Chemical

Suen and Eheart (2003)

Upper Sangamon River, Illinois, USA

River

Chemical

Haas (2004)

USA

Treatment

Biological

Lewin et al. (2004)

E.L. Smith Water Treatment

Treatment

Chemical 64 

 

Plant, Edmonton, Alberta, Canada Maier et al. (2004)

Southern Australian surface waters, Australia

Treatment

Physical, chemical, others

Elshorbagy et al. (2005)

Cumberland River Basin, Kentucky, USA

River

Biological

Khalil et al. (2005)

Sumas-Blaine Aquifer, Washington State, USA

Groundwater

Chemical

Sahoo et al. (2005)

North Carolina, USA

Groundwater

Chemical

Bowden et al. (2006)

South Australia, Australia

Distribution

Chemical

Kuo et al. (2006)

Feitsui Reservoir, Taiwan

Reservoir/Lake

Chemical

Philibert et al. (2006)

South-Eastern Australia, Australia

River

Chemical, physical, others

Sahoo et al. (2006)

Illinois, USA

Groundwater

Chemical

Tayfur and Guldal (2006)

Tennesssee Basin, USA

River

Physical

Teles et al. (2006)

Crestuma Reservoir, Douro River, Portugal

Reservoir/Lake

Biological

Chaves and Kojiri (2007a)

Barra Bonita Reservoir, Brazil

Reservoir/Lake

Chemical, biological

Chaves and Kojiri (2007b)

Barra Bonita Reservoir, Brazil

Reservoir/Lake

Chemical, biological

Diamantopoulou et al. (2007)

Axio and Strymon River, Greece

River

Chemical, physical

Kilic et al. (2007)

Kapulukaya Dam Reservoir, Turkey

Reservoir/Lake

Biological

Mas and Ahlfeld (2007)

Gates Brook, Massachusetts, USA

River

Biological

Elhatip and Komur (2008)

Mamasin dam, Aksaray City, Turkey

Reservoir/Lake

Chemical, physical

Jeong et al. (2008)

Nakdong River, South Korea

River

Biological

Li et al. (2008)

Alberta, Canada

River

Chemical

May et al. (2008)

New South Wales and South Australia, Australia

Distribution

Chemical

65   

Tufail et al. (2008)

Kentucky River, USA

River

Biological

Yeon et al. (2008)

Pyeongchang river, South Korea

River

Chemical

da Costa et al. (2009)

Ipanema Stream, Brazil

River

Chemical

Dixon (2009)

Florida, USA

Groundwater

Chemical

Dogan et al. (2009)

Melen River, Turkey

River

Chemical

Gemitzi et al. (2009)

South Rhodope Aquifer, Thrace, Greece

Groundwater

Chemical

May and Sivakumar (2009)

USA

Storm water

Chemical

Sahoo et al. (2009)

Four streams into Lake Tahoe, USA

River

Physical

Ulke et al. (2009)

Gediz River, Turkey

River

Physical

Akkoyunlu and Akiner (2010)

Omerli Lake, Turkey

Reservoir/Lake

Chemical

Bashi-Azghadi et al. (2010)

Tehran Aquifer, Iran

Groundwater

Others

D’Souza and Kumar (2010)

India, USA

Distribution

Chemical, biological

Faruk (2010)

Buyuk Menderes River, Turkey

River

Chemical, physical

Juahir et al. (2010)

Langat River, Malaysia

River

Others

Kalin et al. (2010)

West Georgia, USA

River

Chemical, physical

Kulkarni and Chellam (2010)

USA

Treatment

Chemical

Merdun and Cinar (2010)

Broadwater Creek and Upper Lake Marion, South Carolina, USA

Reservoir/Lake and River

Biological

Park et al. (2010)

Yongdam Dam Reservoir, River Keumgang, Korea

Reservoir/Lake

Biological

Rankovic et al. (2010)

Gruza Reservoir, Serbia

Reservoir/Lake

Chemical

Akkoyunlu et al. (2011)

Reservoir/Lake Lake Iznik, Turkey

Chemical

66   

Banerjee et al. (2011)

Islands of Lakshadweep archipelago, India

Groundwater

Chen et al. (2011)

Lijiang River, China

River

Biological

Cho et al. (2011)

Cambodia, Laos, Thailand

Groundwater

Chemical

Stormwater

Chemical, physical

Stormwater

Chemical, physical

He et al. (2011b) Calgary, Alberta, Canada He et al. (2011c) Calgary, Alberta, Canada

Physical

He et al. (2011a)

59 river basins in Japan

River

Chemical

Khalil et al. (2011)

50 catchments in Nile delta, Egypt

River

Chemical, physical

Melesse et al. (2011)

Mississippi, Missouri and Rio Grande, USA

River

Nikoo et al. (2011)

River Jajrood river, Iran

Ogleni and Topal (2011)

Mudurnu river, Turkey

River

Soyupak et al. (2011)

Anatalya Konyaalti WDS, Turkey

Distribution

Ye et al. (2011)

41 WTPs in five cities, China

Treatment

Abudu et al. (2012)

Rio Grande at El Paso, Texas, USA

River

Asadollahfardi et al. (2012)

Halkheh Rud River, Northwest Iran

River

Ay and Kisi (2012)

Foundation Creek, El Paso, Colorado, US

River

Chang et al. (2012)

Physical Chemical, physical Chemical Chemical Chemical Physical Physical Chemical

Groundwater

Chemical, others

Hypothetical pumping well Chen et al. (2012)

Feitsui Reservoir, Taiwan

Reservoir/Lake

Chemical

Guo et al. (2012)

Songhua River, China

River

Chemical

Kim et al. (2012a)

Daecheong reservoir on Geum river, South Korea

Reservoir/Lake

Kim et al. (2012b)

West Branch Delaware River watershed, USA

River

Biological Chemical, physical

67   

Liu and Chen (2012)

Reservoir/Lake Yuan-Yang lake, Taiwan

Najah et al. (2012)

Johor River, Malaysia

Perelman et al. (2012)

WDSs in the USA

Rankovic et al. (2012)

The Gruža Reservoir, Serbia

River Distribution Reservoir/Lake

Song et al. (2012)

Reservoir/Lake Shitoukoumen Reservoir, China

Tota-Maharaj and Scholz (2012)

Stormwater Lab experiments, UK

Physical Physical Chemical Chemical Physical, biological Chemical

68