Development and assessment of ecological models in the context of ...

Ecological Informatics 17 (2013) 111–116

Contents lists available at ScienceDirect

Ecological Informatics journal homepage: www.elsevier.com/locate/ecolinf

Development and assessment of ecological models in the context of the European Water Framework Directive: Key issues for trainers in data-driven modeling approaches G. Everaert a,⁎, I.S. Pauwels a, P. Boets a, F. Buysschaert b, P.L.M. Goethals a a b

Ghent University, Faculty of Bioscience Engineering, Laboratory of Environmental Toxicology and Aquatic Ecology, J. Plateaustraat 22, B-9000 Ghent, Belgium Ghent University, Faculty of Economics and Business Administration, Department of Educational Affairs, Tweekerkenstraat 2, B-9000 Ghent, Belgium

a r t i c l e

i n f o

Article history: Received 13 April 2011 Received in revised form 10 May 2012 Accepted 24 October 2012 Available online 31 October 2012 Keywords: Classification and regression trees Course efficiency Data mining Education

a b s t r a c t Teaching students to develop data-driven models is a challenging task as a good balance has to be found between the theoretical background of the models, the ecological relevance of the knowledge rules inferred and their socio-economic applicability. In this context it is unclear which aspects of the modeling process are easily understood by students, and in particular, how theoretical issues interfere with practical boundary conditions and socio-economic relevance (ecosystem protection, water management, policy development, ecological engineering). In order to fill this knowledge gap, students developed static data-driven models and tutors assessed students' performances. Criteria such as the theoretical, ecological and socio-economic relevance of the derived knowledge rules were used to select the most optimal models. We noticed an inverse relationship between the complexity of the subtasks and the number of students that succeeded. Students evaluated their models with respect to the theoretical reliability, but were not likely to consider the other two criteria. Half of the students succeeded in assessing the models based on their ecological relevance and only 17% of the students checked the socio-economic relevance of the knowledge rules. Four groups out of seven assessed their models merely based on the predictive power of the models. Only one group integrated the theoretical, ecological and socio-economic relevance to assess the models. The key findings of our research can be used to optimize the efficiency of data mining courses. We reveal which aspects of the modeling process students seem to overemphasize and give recommendations about the topics trainers should emphasize in the future to ensure that students develop advanced skills. Based on our results the theory–practice dichotomy in higher education can be further reduced. Our learning-by-doing approach showed students how to solve common problems in ecological data sets (e.g. missing data, outliers, collinearity, non-normal distribution, parameterization, uncertainty, etc.), which are often only briefly discussed in basic statistical courses. © 2012 Elsevier B.V. All rights reserved.

1. Introduction Learning new software is an issue that receives substantial attention in academic training programs as computerized technology is increasingly used in decision making and provides new insights in complex situations and relationships (Argent et al., 2009; D'heygere et al., 2003; Rizzoli and Young, 1997; van Delden et al., 2010; Varis, 1997; Wilby et al., 2002). Currently, scientists and policy makers can select a wide range of modeling techniques, subdivided into knowledge based techniques (Bredeweg et al., 2009; Grimm, 1994; Rykiel, 1989; Salles and Bredeweg, 2006) and data mining methods (Breiman et al., 1984;

⁎ Corresponding author. Tel.: +32 92643776. E-mail address: [email protected] (G. Everaert). 1574-9541/$ – see front matter © 2012 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.ecoinf.2012.10.007

De'ath and Fabricius, 2000; Quinlan, 1986; Witten and Frank, 2005). The main difference between both approaches is that data-driven techniques quantitatively describe relationships without prior knowledge (De'ath and Fabricius, 2000), whereas knowledge based methods such as qualitative reasoning (e.g. Garp3) start from expert knowledge (Bredeweg et al., 2009). Data-driven techniques like decision trees have the inherent ability to discover patterns in data that are easy to communicate (straightforward rules) and introduce less prior assumptions compared with other modeling techniques (Džeroski and Drumm, 2003). Extended information about the statistical background of decision models and the algorithms used to derive patterns from data can be found in numerous handbooks (e.g. Breiman et al., 1984; Witten and Frank, 2005). Generally speaking, decision models explain variations in a numerical response variable by splitting predictor variables at certain

112

G. Everaert et al. / Ecological Informatics 17 (2013) 111–116

Fig. 1. Example of a decision tree relating the physical–chemical variables to the biological river status (classified as bad, poor, moderate and good). Knowledge rules are found in each node and the predictions are read from the end leafs. Physical–chemical variables used are the average Biological Oxygen Demand (BOD_avg, mg O2/L), minimum oxygen concentration in the water (DO_min, mg O2/L) and the average total phosphorus concentration (Pt_avg, mg P/L).

thresholds in, so called, knowledge rules embedded in the nodes of the decision tree. An example of such decision tree is found in Fig. 1, the rules should be interpreted as follows: If explanatory variable Xi (e.g. the average total phosphorus concentration) is higher/lower than a specific threshold (e.g. 0.6 mg/L), the dependent variable Y (e.g. ecological water quality status) will be x (e.g. BAD/MODERATE) or depend on other explanatory variables (e.g. oxygen concentration). Decision trees consist of a transparent set of rules, easy to understand and to implement into an environmental decision support system by policymakers (Ambelu et al., 2010; Hoang et al., 2010). Moreover, decision trees can give insight into complex, unbalanced, non-linear ecological data where commonly used exploratory and statistical modeling techniques fail to reveal meaningful patterns (De'ath and Fabricius, 2000). Decision trees have been frequently used in ecology to analyze the relationships between species and their environment (Boets et al., 2010; De'ath, 2002; Everaert et al., 2011; Kampichler et al., 2010; Kocev et al., 2009). Also in other scientific branches like medical statistics, decision trees were already successful (Grubinger et al., 2010; Shi and Lyons-Weiler, 2007). This makes them suitable to teach students how to develop, evaluate, revise and apply ecological models in decision making, as the link with the expert domain can be easily made via the transparency of the induced rules. The aim of data mining courses is that graduates acquire a basic knowledge about data-driven modeling techniques and their applications. Several issues about the modeling process should be highlighted when teaching students to develop decision trees (Witten and Frank, 2005). Our aim is to teach students how to develop the most optimal decision tree given three criteria which are often used to assess the ecological models. The mathematical reliability is the most obvious criterion to assess ecological models. The statistical reliability of decision trees can be assessed based on performance criteria such as the percentage of correctly classified instances (CCI) and Kappa statistic (K, Cohen, 1960), which are expected to exceed 70% and 0.4 respectively (Mouton et al., 2010). However, due to artifacts in data sets (missing data, outliers, collinearity, limited number of instances, parameterization, biased data, etc.), particular problems may arise leading to the development of non-sense or highly unreliable models (Au et al., 2010; Zuur et al., 2010). Consequently, an introduction to the pre-processing of data is also indispensable in data mining courses. Large data sets often contain irregularities like outliers and collinear variables, which can lead to fuzzy patterns (Au et al., 2010). An advanced pre-processing of the

data set will prevent erroneous conclusions and will increase the mathematical reliability of the models (Everaert et al., 2010). However, in data mining problems it is often forgotten that maximizing mathematical indicators does not always result in the most optimal model. Therefore, a second criterion to assess decision models is the ecological relevance of the knowledge rules. Hence, the knowledge rules are mutually compared and tested for what is generally accepted in ecology. A last criterion to evaluate an ecological model is by verifying the applicability of the derived knowledge rules (Larocque et al., 2011). The major question here is whether the rules found are applicable by policy makers. Given a certain situation, it should be checked how the thresholds found are related to, for instance, the current water quality standards. Three criteria, namely the mathematical, ecological and socio-economic relevance of the models, should be highlighted if tutors want to end up with students having a holistic view on data mining. The value of models in environmental decision support system has been intensively advocated (Hoang et al., 2010; Moffett et al., 2005; Mouton et al., 2008, 2009; Nuttle et al., 2009; Salles et al., 2006; Varis, 1997; Zitek et al., 2009). Therefore, it is necessary to instruct students about the development, assessment, adaptation and application of ecological models. Although different techniques and approaches are taught, few studies investigate the efficiency of these courses. One example is the research of Nuttle and Bouwer (2009) that evaluates the efficiency of courses dealing with qualitative model development. Notably, until the present no research has been published about the quality and efficiency of data mining courses. This is remarkable because basic data mining techniques are often used for policy making. Unfortunately, in reality ecological data sets are often far from perfect. This why our research is so important; we help to reduce the gap between what graduates theoretically learn in basic statistical courses and the real work floor situation (Hallinan, 1996). Our approach wants to minimize the theory–practice dichotomy by letting students experience the possible bottlenecks when developing ecological decision models. We want to disseminate our experiences and reveal some stumbling blocks for our fellow teachers for the ultimate benefit of the students. There is a clear need to explore which aspects, related to the construction of data-driven models, are easily understood by students, and which should be emphasized in future years. First, we describe the background of the ecological data set students processed. Next, we introduce the three criteria to assess the decision trees and explain how student performances were classified. Finally, we present our results, relate these to existing literature and formulate some recommendations for our fellow teachers.


2. Materials and methods Ecotechnology or Ecological Engineering is the new, internationally recognized term for the application of engineering techniques to all quantitative aspects concerning the monitoring, assessment, construction, repair and management of ecosystems. Due to the popularity of using models in decision making, the practical tutorials of the course Ecotechnology (Faculty of Bioscience Engineering, Ghent University) teach bioscience engineering students how to develop, evaluate, revise and apply decision trees in environmental decision support systems. These tutorials aim to give graduates a basic knowledge of ecological modeling and its applications. The efficiency of the tutors' approach was evaluated. All results presented originate from a case study solved by the students during the tutorial meetings. 2.1. Case study in brief The case study was based on a project finished in 2010 at the Laboratory of Environmental Toxicology and Aquatic Ecology (Ghent University). Generally speaking, students undertook similar model development steps as discussed in Everaert et al. (2010) and Pauwels et al. (2010), both dealing with the methodology of the project. The project was performed in the scope of the European Water Framework Directive (WFD) (EU, 2000) that requires European member states to achieve a good ecological and chemical surface water quality by 2015. Governments should take measures to reach this goal, but the impact of the measures on the water quality is not straightforward. Fortunately, their impact can be predicted using numerical models that relate the biological status to physical–chemical variables. Students were asked to develop such models in the Flemish context, by applying datadriven techniques. However, the outcome of such analysis depends on the assumptions made in the pre-processing steps of the modeling process (Zuur et al., 2010). Consequently, the case study has no unique solution, but basically the models found should be similar to the one presented in Fig. 1. Students had no prior knowledge of the software used, but many of them had a basic background in modeling acquired during their bioscience engineering training program. Students were introduced to the basic functionalities of the Waikato Environment for Knowledge Analysis (WEKA) (Witten and Frank, 2005) by means of hands-on exercises. WEKA is a software platform consisting of machine learning algorithms for data mining tasks. It contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization. WEKA is free software and can be downloaded from the website: http://www.cs.waikato.ac.nz/. Students constructed data-driven models in the classroom guided by a manual dealing with the importance of pre-processing of data (missing data, outliers, stratification, collinearity, etc.) and the model parameterization (binary split, minimum number of objects per leaf, pruning level, etc.). Seven groups of four to five students, all following the course Ecotechnology, were randomly composed. Although we had a limited number of participants, our students could be considered as typical students and were thus representative of other classes of students. Some of them were very motivated in solving the case and took the whole group in tow, others followed the reasoning of the eager students and some free-riders were present. All aspects related to the development of data-driven models were part of our data mining course. Students were confronted with common data mining problems, such as outliers, collinear variables, data stratification and tree pruning (Zuur et al., 2010). They performed a sensitivity analysis to quantify how different values of an independent variable impact a particular dependent variable. However, in order to minimize the cognitive load for the students at the start of the case study, a dummy data set was used to illustrate the theoretical background of decision trees. This dummy data set is the one also used by Witten and Frank (2005), i.e. based on outlook, humidity and wind data it predicted whether the weather is suitable for

113

playing tennis. However, compared with an ecological data set, such dummy set does not contain complex multivariate relationships (Dewar and Porte, 2008; Nekola and Brown, 2007). The use of decontextualized data is thus not always appropriate for the learning process (Lombardi, 2007; Schwarz et al., 2009), but was in our case used only for illustrative purposes (i.e. to show the possibilities of decision trees). The complexity of the subtasks gradually increased as students shifted from the dummy data set to the realistic ecological data set. Each of these subtasks introduced one of the criteria (mathematical, ecological, socio-economic relevance) used to assess the models and covered a specific learning goal of the case study.

2.2. Assessment of the course efficiency Tutors asked students to assess the generated models based on their theoretical, ecological and socio-economic relevance. As recommended by White and Frederiksen (1998), assessment criteria were explicitly defined at the beginning of the course. Table 1 summarizes the tasks that students had to perform per criterion and gives an indication of their complexity. First, students assessed their models based on mathematical criteria. For this, they verified whether the statistical performance criteria such as the CCI and K exceeded 70% and 0.4 respectively. In order to optimize the statistical reliability of the models the students were allowed to change the pre-processing of the data and to adapt the model parameterization. Next, the students were expected to discuss the ecological relevance of the knowledge rules. Hence, they explored the knowledge rules and checked whether these were in accordance with what is generally accepted in ecology. For example, an acceptable knowledge rule could be: “the lower the nutrient load, the higher the ecological status of the ecosystem”. An improper knowledge rule is for instance: “the lower the oxygen concentration, the higher the ecological status”. In a last step of the analysis, students were encouraged to discuss the socio-economic relevance of the knowledge rules. The most obvious way to do this was first to check whether all ecological water quality classes could be predicted by the model, and next to compare the thresholds embedded in the knowledge rules with the corresponding Flemish water quality standards. Upon completion of the case study, each group submitted a report that documented the steps taken while proceeding through the case, the selected models, the motivation why particular models were chosen and their key findings. Students' reports were assessed using a checklist with those items which are most important while developing ecological data-driven models. Tutors verified whether the theoretical, ecological and economic relevance of the models were taken into account. Groups' marks were based on the overall conclusions (product-assessment) and the discussion of the mathematical, ecological and socio-economic relevance of the data-driven models (process-assessment). Per criterion the group obtained a performance score, being low, medium and high performance (Table 2). We used these groups' marks per criterion to evaluate Table 1 Explanation of the assessment criteria. Performance criterion

Complexity What was expected from the students? of the task

Theoretical relevance

Low

Ecological relevance

Medium

Socio-economical High relevance

Verify whether the percentage of correctly classified instances (CCI) and Cohen's Kappa statistic (Cohen, 1960) reached 70% and 0.4 respectively (Mouton et al., 2010). Verify whether knowledge rules are not in conflict with each other or with basic ecological insights. Compare the thresholds derived in the knowledge rules with the Flemish water quality standards. Verify the clarity and applicability of the knowledge rules from the point of view of a river manager.

114


Low performance

to what extent our approach was efficient. Hence, we could reveal to which topics we should give more attention in future years.

Students were able to generate basic data-driven models with the WEKA software within two hours. As all seven groups performed high concerning this criterion, the assessment of the theoretical relevance of the models seemed relatively easy (Fig. 2). However, the more complex the task, the lower the students' performances. Only three groups assessed the ecological relevance of the models' knowledge rules in a correct way, the four other groups showed a poor performance. Five groups did not take the effort to assess their models concerning the socio-economic relevance of the knowledge rules. Only one group dealt in an excellent way with this criterion, whereas the remaining groups tried to evaluate the models, but succeeded only to a limited extent (Fig. 2). The latter group discussed the knowledge rules correctly, but compared the thresholds derived with the wrong water quality standards and did not verify the applicability of the knowledge rules from the point of view of a river manager. Overall, we found that one group submitted a high quality report concerning all criteria, five groups scored moderately and one group gained little knowledge from the case study. Data were pre-processed in an excellent way and the impact of adapted default setting was accurately explored by all groups (Fig. 3). Although six groups out of seven discussed the knowledge rules with a medium to high performance, only three groups succeeded in evaluating the derived models according to their ecological relevance. One group succeeded in performing a sensitivity analysis, in order to quantify how different values of an independent variable impact a particular dependent variable. Three groups tried to reveal the most important predictor, but failed. None of the three remaining groups performed a sensitivity analysis (Fig. 3). 4. Discussion This research gives insight into the problems students may encounter when developing ecological models and those aspects of the modeling process that teachers should emphasize in the future. Apart from decision trees, knowledge based techniques (both qualitative of quantitative approaches) could have also been implemented to solve the case study. For instance, Salles et al. (2006) illustrate how integrated qualitative models can predict the efficiency of different management practices on stream ecosystem recovery. The main problem of using qualitative techniques is that they require a thorough understanding of the relationships in the ecosystem, which tutors could not expect of the students. It is generally known that data-driven techniques can easily deal with a mix of qualitative and quantitative information (Witten and Frank, 2005), whereas numerical data are hard to integrate in qualitative modeling (Araujo et al., 2008).

High performance

7

Number of groups

3. Results

Medium performance

6 5 4 3 2 1 0

Theoretical relevance

Ecological relevance

Socio-economical relevance

Criteria Fig. 2. Assessment of students' reports: performance scores per criterion.

4.1. Positive effects of scaffolding Although our approach differs from the one used by Nuttle and Bouwer (2009), some of our findings confirm the conclusions of previous studies. Students experienced the WEKA software as complex and the interpretation of the models seemed difficult at the start of the case study, but by means of scaffolding, which is a combination of performance support and fading (van Merrienboër et al., 2003), they succeeded in generating and analyzing their first data-driven models within two hours. The support enabled students to acquire knowledge about complex subjects not acquirable without that support. Examples of such support are providing guidelines, hints and feedback, presenting checklists, asking leading questions and giving part of a solution (van Merrienboër et al., 2003). When they achieved the desired learning goal, support gradually diminished until it was no longer needed. However, complex learning tasks from the start of a course were omitted because this would yield excessive cognitive load for the learners, with negative effects on learning, performance, and motivation (Sweller et al., 1998). Thus, tutors started with relatively simple learning tasks and progressed slowly toward more complex tasks. This sequencing strategy allowed the definition of separate learning goals, where each of the subtasks corresponded with one learning goal. In our research, as well as in the evaluation of Nuttle and Bouwer (2009), students had no prior knowledge of the software. Although, scaffolding tried to minimize the cognitive load for the students (van Merrienboër et al., 2003), tutors noticed dissimilarities in student performances, which can be explained by changing the students' interests, motivation and goal orientation (Piaget and Kamii, 1978; Shimoda et al., 2002).

Table 2 Explanation of the performance scores. Performance scores

Definition

Low performance

The task was not executed or insufficiently executed compared with the instructors' expectations. The students did not understand the main goals. The task was executed in an acceptable way compared with the expectations of the instructors. Although the main lines were understood by the students, they could not prove detailed knowledge. The way the group tackled the case study, showed the results and formulated the conclusions met the expectations of the instructors. Both the main lines and the detailed information were efficiently applied by the students.

Medium performance

High performance

Fig. 3. Assessment of students' reports: performance scores per specific task.


4.2. Positive effects of authentic learning The use of a realistic case study clearly had an added value as there was little practical use in gaining abstract, decontextualized insights (Schwarz, 2009). Due to our learning-by-doing approach (Lombardi, 2007), the students' point of view about ecological data sets may have changed, in the sense that they became aware of common artifacts in ecological data (missing data, outliers, etc.). In case we had opted for a merely illustrative approach, in which all data were nicely pre-processed as in traditional statistical courses, our course would have had less impact on their perception about ecological data. Now, students saw the strengths, problems and limitations of applying classification and regression trees on ecological data. In order to solve those problems and limitations they experienced typical issues like selecting and controlling variables, optimizing settings, preprocessing data and revising models. This perfectly fits in the current reforms in science (and engineering) education suggesting that lectures should not only highlight the established results of science, but also focus on the processes and methods used to achieve such results (van Joolingen et al., 2007). Students should be involved in scientific practices so that they understand the reasoning behind and the purpose of each scientific step, and develop transferable skills and knowledge (de Jong, 2006; Lehrer and Schauble, 2009; Schwarz, 2009; Swaak et al., 1998; van Joolingen et al., 2007). By confronting students with uncertainty, ambiguity, and conflicting perspectives, tutors helped them to develop more mature mental models that coincide with the problem-solving approaches used by experts (Lombardi, 2007). They learned how to arrange a sequence of modeling activities while generating data-driven models. They became aware of different criteria which are used to evaluate and select the most optimal models (Lehrer and Schauble, 2009; Schwarz et al., 2009). The authentic learning exercises exposed the complexity of real-life decision making as the solution depends on the particular context (Lombardi, 2007). So, by engaging students in solving a real-world problem (i.e. how to improve the ecological water quality) they became aware that multiple criteria are to be used to make policy decisions. 4.3. Stumbling blocks and possible solutions Feedback meetings per group were beneficial to prevent students from getting stuck in the details. Students found it difficult to integrate the assessment criteria, but by means of consecutive subtasks we encouraged them to do so. Only one group found a good balance between an ecologically relevant and statistically reliable model. The remaining groups invested most of their time in adapting the default settings and data pre-processing to maximize the theoretical reliability of the models and failed to finalize the other subtasks successfully. Those groups were so focused on meeting the statistical requirements, that they lost track of the other criteria. However, other criteria than statistical reliability were at least as important when assessing models. At the feedback meetings tutors rehearsed how default settings could be adapted and data pre-processed in order to put those groups stuck in the details back on the right track. Students were encouraged to use a pragmatic approach concerning the statistical thresholds (70% for CCI and 0.4 for K respectively) because their use is often criticized (Sim and Wright, 2005). Tutors emphasized that in case models where reliabilities close to or higher than the aforementioned thresholds were generated, students should go to the next step of the assessment, i.e. the evaluation of the ecological relevance and socio-economic value of the knowledge rules. As noticed by McGourty et al. (1998) and Hattie and Timperley (2007), feedback moments had a positive influence on students' performances, after the feedback moment students performed their tasks more efficiently. General guidelines will help students in searching for relevant information and will further optimize the efficiency of their search. It is noteworthy that although six out of seven groups discussed the

115

knowledge rules moderately well to well, there were only three groups that scored well on the discussion of the ecological relevance. Tutors had the impression that the search for extra scientific information to assess the credibility of the proposed thresholds was problematic. The most obvious way to perform this subtask was to look up information about river ecology via the Web of Science or the internet in general. However, the reliability of information found by students is often doubtful (Brem et al., 2001; Mason et al., 2010). Therefore, we recommended that tutors give some general guidelines about the reliability of scientific information found or make some important publications available via the web server of the university. 4.4. Further research Students expect clear subtasks with explicit goals, but the educational institutions advocate more freedom for students. We dealt with this paradox by letting students work in group and making use of authentic learning and scaffolding. Although it has been shown that our approach was not fully successful, we believe this can be an efficient way to prepare graduates. In order to solve the shortcomings, some solutions were listed of which several need further investigation. First, giving constructive feedback is not straightforward (Hattie and Timperley, 2007; McGourty et al., 1998). Therefore it is interesting to verify in future courses which type of feedback is most efficient. As presented here, tutors gave feedback on the students' work. However, it can be considered that groups give feedback to other groups during an intermediate presentation of their work. Whether this is more efficient than a tutor giving feedback remains unclear. Secondly, peer assessment can be introduced to reduce the free-rider problem (Cheng and Warren, 2000). When individual members of a cooperative group give grades to their peers all member of the group are triggered to participate in the task. Kennedy (2005) suggests that students' marks should integrate the grade of the final product (written report or presentation) and the process (approach and the accomplishment to allocating tasks). This approach guarantees that all individuals receive grades corresponding to their output and effort. Although Stevens (2007) suggests mixing the grades derived from the peer assessment with those of the tutors at 25% and 75%, it is uncertain whether this approach is appreciated by the students. 5. Conclusions Students constructed data-driven models and used those models to make predictions about the effectiveness of the proposed measures in the scope of the WFD. Generally, we might say that students are able to construct data-driven models using WEKA software, and to apply the derived models in the context of the WFD. Our findings suggest that future data mining courses should give more attention to the integration of different model assessment criteria, focus on authentic learning and use scaffolding to minimize the cognitive load at the start of the task. Some solutions, such as planning feedback meetings, making relevant information available and restructuring the course were formulated to overcome the shortcomings noticed while proceeding through the task. Acknowledgments I.S. Pauwels is the recipient of a Ph.D. grant provided by the Institute for the Promotion of Innovation through Science and Technology in Flanders (IWT-Vlaanderen). References Ambelu, A., Lock, K., Goethals, P., 2010. Comparison of modelling techniques to predict macroinvertebrate community composition in rivers of Ethiopia. Ecological Informatics 5, 147–152. Araujo, S.C.S., Salles, P., Saito, C.H., 2008. A case study on qualitative model evaluation using data about river water quality. Ecological Informatics 3, 13–25.

116


Argent, R.M., Perraud, J.M., Rahman, J.M., Grayson, R.B., Podger, G.M., 2009. A new approach to water quality modelling and environmental decision support systems. Environmental Modelling & Software 24, 809–818. Au, S.T., Duan, R., Hesar, S.G., Jiang, W., 2010. A framework of irregularity enlightenment for data pre-processing in data mining. Annals of Operations Research 174, 47–66. Boets, P., Lock, K., Messiaen, M., Goethals, P.L.M., 2010. Combining data-driven methods and lab studies to analyse the ecology of Dikerogammarus villosus. Ecological Informatics 5, 133–139. Bredeweg, B., Linnebank, F., Bouwer, A., Liem, J., 2009. Garp3—workbench for qualitative modelling and simulation. Ecological Informatics 4, 263–281. Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J., 1984. Classification and Regression Trees. Wadsworth, Belmont . (358 pp.). Brem, S.K., Russell, J., Weems, L., 2001. Science on the Web: student evaluations of scientific arguments. Discourse Processes 32, 191–213. Cheng, W., Warren, M., 2000. Making a difference: using peers to assess individual students' contributions to a group project. Teaching in Higher Education 5, 243. Cohen, J., 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, 37–46. D'heygere, T., Goethals, P.L.M., De Pauw, N., 2003. Use of genetic algorithms to select input variables in decision tree models for the prediction of benthic macroinvertebrates. Ecological Modelling 160, 291–300. de Jong, T., 2006. Computer simulations — technological advances in inquiry learning. Science 312, 532–533. De'ath, G., 2002. Multivariate regression trees: a new technique for modeling species– environment relationships. Ecology 83, 1105–1117. De'ath, G., Fabricius, K.E., 2000. Classification and regression trees: a powerful yet simple technique for ecological data analysis. Ecology 81, 3178–3192. Dewar, R.C., Porte, A., 2008. Statistical mechanics unifies different ecological patterns. Journal of Theoretical Biology 251, 389–403. Džeroski, S., Drumm, D., 2003. Using regression trees to identify the habitat preference of the sea cucumber (Holothuria leucospilota) on Rarotonga, Cook Islands. Ecological Modelling 170, 219–226. EU, 2000. Directive 2000/60/EC of the European Parliament and of the Council of 23 October 2000 Establishing a Framework for Community Action in the Field of Water Policy. Everaert, G., Pauwels, I.S., Goethals, P.L.M., 2010. Development of data-driven models for the assessment of macroinvertebrates in rivers in Flanders. In: Swayne, D.A., Yang, W., Voinov, A.A., Rizzoli, A., Filatova, T. (Eds.), 5th Biennial Meeting of the International Congress on Environmental Modelling and Software (iEMSs 2010). Modelling for Environment's Sake International Environmental Modelling and Software Society (iEMSs), Ottawa, ON, Canada. Everaert, G., Boets, P., Lock, K., Džeroski, S., Goethals, P.L.M., 2011. Using classification trees to analyze the impact of exotic species on the ecological assessment of polder lakes in Flanders, Belgium. Ecological Modelling 222, 2202–2212. Grimm, V., 1994. Mathematical models and understanding in ecology. Ecological Modelling 75, 641–651. Grubinger, T., Kobel, C., Pfeiffer, K.P., 2010. Regression tree construction by bootstrap: model search for DRG-systems applied to Austrian health-data. BMC Medical Informatics and Decision Making 10. Hallinan, M.T., 1996. Bridging the gap between research and practice. Sociology of Education 69, 131–134. Hattie, J., Timperley, H., 2007. The power of feedback. Review of Educational Research 77, 81–112. Hoang, T.H., Lock, K., Mouton, A., Goethals, P.L.M., 2010. Application of classification trees and support vector machines to model the presence of macroinvertebrates in rivers in Vietnam. Ecological Informatics 5, 140–146. Kampichler, C., Wieland, R., Calme, S., Weissenberger, H., Arriaga-Weiss, S., 2010. Classification in conservation biology: a comparison of five machine-learning methods. Ecological Informatics 5, 441–450. Kennedy, G., 2005. Peer-assessment in group projects: is it worth it? In: Young, A., Tolhurst, D. (Eds.), 7th Australasian Conference on Computing Education (ACE '05), Newcastle, Australia, 42, pp. 59–65. Kocev, D., Džeroski, S., White, M.D., Newell, G.R., Griffioen, P., 2009. Using single- and multi-target regression trees and ensembles to model a compound index of vegetation condition. Ecological Modelling 220, 1159–1168. Larocque, G.R., Bhatti, J.S., Ascough, J.C., Liu, J., Luckai, N., Mailly, D., Archambault, L., Gordon, A.M., 2011. An analytical framework to assist decision makers in the use of forest ecosystem model predictions. Environmental Modelling & Software 26, 280–288. Lehrer, R., Schauble, L., 2009. Images of learning, images of progress. Journal of Research in Science Teaching 46, 731–735. Lombardi, M.M., 2007. Authentic learning for the 21st century: an overview. In: Oblinger, D.G. (Ed.), Educause Learning Initiative (Retrieved 15/02/2011 from http://www.educause.edu/vi/library/pdf/EL/3009.pdf). Mason, L., Boldrin, A., Ariasi, N., 2010. Searching the Web to learn about a controversial topic: are students epistemically active? Instructional Science 38, 607–633. McGourty, J., Dominick, P., Reilly, R.R., 1998. Incorporating student peer review and feedback into the assessment process. 28th Annual Frontiers in Education Conference, pp. 14–18 (New York).

Moffett, A., Garson, J., Sarkar, S., 2005. MultCSync: a software package for incorporating multiple criteria in conservation planning. Environmental Modelling & Software 20, 1315–1322. Mouton, A.M., Schneider, M., Peter, A., Holzer, G., Muller, R., Goethals, P.L.M., De Pauw, N., 2008. Optimisation of a fuzzy physical habitat model for spawning European grayling (Thymallus thymallus L.) in the Aare river (Thun, Switzerland). Ecological Modelling 215, 122–132. Mouton, A.M., De Baets, B., Goethals, P.L.M., 2009. Knowledge-based versus data-driven fuzzy habitat suitability models for river management. Environmental Modelling & Software 24, 982–993. Mouton, A.M., De Baets, B., Goethals, P.L.M., 2010. Ecological relevance of performance criteria for species distribution models. Ecological Modelling 221, 1995–2002. Nekola, J.C., Brown, J.H., 2007. The wealth of species: ecological communities, complex systems and the legacy of Frank Preston. Ecology Letters 10, 188–196. Nuttle, T., Bouwer, A., 2009. Supporting education about environmental sustainability: evaluation of a progressive learning route for qualitative models. Ecological Informatics 4, 396–404. Nuttle, T., Bredeweg, B., Salles, P., Neumann, M., 2009. Representing and managing uncertainty in qualitative ecological models. Ecological Informatics 4, 358–366. Pauwels, I.S., Everaert, G., Goethals, P.L.M., 2010. Integrated river assessment by coupling water quality and ecological assessment models. In: Swayne, D.A., Yang, W., Voinov, A.A., Rizzoli, A., Filatova, T. (Eds.), 5th Biennial Meeting of the International Congress on Environmental Modelling and Software (iEMSs 2010). Modelling for Environment's Sake International Environmental Modelling and Software Society (iEMSs), Ottawa, ON, Canada. Piaget, J., Kamii, C., 1978. What is psychology? American Psychologist 33, 648–652. Quinlan, J.R., 1986. Induction of decision trees. Machine Learning 1, 81–106. Rizzoli, A.E., Young, W.J., 1997. Delivering environmental decision support systems: software tools and techniques. Environmental Modelling & Software 12, 237–249. Rykiel, E.J., 1989. Artificial intelligence and expert systems in ecology and naturalresource management. Ecological Modelling 46, 3–8. Salles, P., Bredeweg, B., 2006. Modelling population and community dynamics with qualitative reasoning. Ecological Modelling 195, 114–128. Salles, P., Bredeweg, B., Araujo, S., 2006. Qualitative models about stream ecosystem recovery: exploratory studies. Ecological Modelling 194, 80–89. Schwarz, C., 2009. Developing preservice elementary teachers' knowledge and practices through modeling-centered scientific inquiry. Science Education 93, 720–744. Schwarz, C.V., Reiser, B.J., Davis, E.A., Kenyon, L., Acher, A., Fortus, D., Shwartz, Y., Hug, B., Krajcik, J., 2009. Developing a learning progression for scientific modeling: making scientific modeling accessible and meaningful for learners. Journal of Research in Science Teaching 46, 632–654. Shi, H., Lyons-Weiler, J., 2007. Clinical decision modeling system. BMC Medical Informatics and Decision Making 7. Shimoda, T.A., White, B.Y., Frederiksen, J.R., 2002. Student goal orientation in learning inquiry skills with modifiable software advisors. Science Education 86, 244–263. Sim, J., Wright, C.C., 2005. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Physical Therapy 85, 257–268. Stevens, M.C., 2007. Making groups work. College Teaching 55, 88. Swaak, J., Van Joolingen, W.R., De Jong, T., 1998. Supporting simulation-based learning: the effects of model progression and assignments on definitional and intuitive knowledge. Learning and Instruction 8, 235–252. Sweller, J., van Merriënboer, J.J.G., Paas, F., 1998. Cognitive architecture and instructional design. Educational Psychology Review 10, 251–296. van Delden, H., Stuczynski, T., Ciaian, P., Paracchini, M.L., Hurkens, J., Lopatka, A., Shi, Y.E., Prieto, O.G., Calvo, S., van Vliet, J., Vanhout, R., 2010. Integrated assessment of agricultural policies with dynamic land use change modelling. Ecological Modelling 221, 2153–2166. van Joolingen, W.R., de Jong, T., Dimitrakopoulou, A., 2007. Issues in computer supported inquiry learning in science. Journal of Computer Assisted Learning 23, 111–119. van Merrienboër, J.J.G., Kirschner, P.A., Kester, L., 2003. Taking the load off a learner's mind: instructional design for complex learning. Educational Psychologist 38, 5–13. Varis, O., 1997. Bayesian decision analysis for environmental and resource management. Environmental Modelling & Software 12, 177–185. White, B.Y., Frederiksen, J.R., 1998. Inquiry, modeling, and metacognition: making science accessible to all students. Cognition and Instruction 16, 3–118. Wilby, R.L., Dawson, C.W., Barrow, E.M., 2002. SDSM — a decision support tool for the assessment of regional climate change impacts. Environmental Modelling & Software 17, 147–159. Witten, I.H., Frank, E., 2005. Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, San Francisco . (560 pp.). Zitek, A., Schmutz, S., Preis, S., Salles, P., Bredeweg, B., Muhar, S., 2009. Evaluating the potential of qualitative reasoning models to contribute to sustainable catchment management. Ecological Informatics 4, 381–395. Zuur, A.F., Ieno, E.N., Elphick, C.S., 2010. A protocol for data exploration to avoid common statistical problems. Methods in Ecology and Evolution 1, 3–14.