Testing Tales: Selection and Evaluation of a Framework for ... - iEMSs

International Environmental Modelling and Software Society (iEMSs) 7th Intl. Congress on Env. Modelling and Software, San Diego, CA, USA, Daniel P. Ames, Nigel W.T. Quinn and Andrea E. Rizzoli (Eds.) http://www.iemss.org/society/index.php/iemss-2014-proceedings

Testing Tales: Selection and Evaluation of a Framework for Interoperable Freshwater Modelling Alexander Elliotta, Gabriella Tureka, Valerie Snowb, Daniel Rutledgec, Alistair Ritchiec, Alexander Herzigc aNational

Institute of Water and Atmospheric Research, Hamilton, New Zealand ([email protected], [email protected]) bAgResearch, Lincoln, New Zealand ([email protected]) cLandcare Research, New Zealand ([email protected], [email protected], [email protected])

Abstract: Recent freshwater policy initiatives in New Zealand have highlighted the need for integrated and interoperable freshwater models and data sources. In response, we evaluated a range of software frameworks for interoperable freshwater modelling. A workshop on user needs identified a spectrum of framework users, ranging from ‘indirect users’ such as policy makers, to ‘developers’ such as software engineers concerned with developing and implementing frameworks. To address calls for clearer information on available models and data sources, we developed databases of models and data sources along with an interactive tool to query the database and illustrate potential data-model and model-model couplings. Existing frameworks were reviewed and evaluated against user needs, leading to the selection one framework, OMS3, for more detailed testing. We successfully set up several small models and third-party visualisation and geo-spatial tools as OMS3 components using a range of approaches, liked these in the framework, accessed a simple model as a web service, developed a graphical user interface to modify XML input files, and accessed time-series and spatial data through OGC-compliant web services. These tests were encouraging. However, implementing complex realworld user interfaces in OMS3 would require writing the user interface from scratch. Other limitations of OMS3 included little provision for spatial data and complex data structure, and difficulties incorporating .NET-based dll’s. Indirect users called for demonstration of a complex spatio-temporal and crossdomain application with a polished user interface, and this is proposed as a next step before committing fully to OMS3. Web services approaches to model delivery and integration seemed attractive to overcome some institutional and technical barriers that were identified, and a staged pathway for adoption of web services was proposed. Keywords: interoperability; modelling; freshwater; framework.

1.

INTRODUCTION

Recent freshwater policy and associated research strategy documents in New Zealand (Ministry for the Environment, 2013) have identified the need for robust interoperable models to assist the development and implementation of freshwater policy and management. By interoperability we mean the ability for models and data sources to exchange data and function in a co-ordinated and integrated manner. An interoperable freshwater modelling framework would enable the co-ordinated, integrated, and flexible coupling and re-use of a diverse range of models and associated data sources, leading to improved integrated modelling and freshwater management. Such interoperability contrasts with the current situation in New Zealand, where there are disparate freshwater models with only occasional ad-hoc and inflexible integration. In response to this need, we evaluated a range of software frameworks that enable interoperability, with particular emphasis on the freshwater domain and the models and data sources used in New Zealand. This study represents an initial foray into the area of freshwater model interoperability in New Zealand, building on recent central government spatial data initiatives.

Elliott A.H. et al. Testing Tales: Selection and Evaluation of a Framework for Interoperable Freshwater Modelling

Following an initial review of frameworks, it seemed that several existing frameworks would meet many of the user needs. Therefore we focused on selecting and trialling an existing framework, with particular emphasis on user needs and the range of models and datasets commonly used in New Zealand, rather than designing a system from scratch. The scope was restricted to models of freshwater quality and quantity and associated data, to ensure that the project was tractable. This focus does not preclude extension to other domains such as ecology, economics or planning in the future.

2

METHODOLOGY

The study involved the following components:  Holding an end-user workshop to identify the range of users and their needs (Snow et al., 2011).  Compilation of an inventory of freshwater models and environmental data currently used in New Zealand, categorised in relation to a common set of attributes relevant to coupling. This information is available in a structured database on a project wiki (https://teamwork.niwa.co.nz/display/IFM).  Development of a tool, ModelVis, to identify and display potential linkages between current freshwater models and data sources (see Elliott et al., 2012 and the project wiki).  Reviewing a range of existing interoperability frameworks and screening them in relation to user needs, leading to the selection of a framework for more detailed hands-on assessment (Elliott et al., 2012).  Testing the preferred framework, OMS3, for its ability to link freshwater models commonly used in New Zealand. This involved the following tests: 1. Setting up the point-scale dynamic hydrology model WATYIELD 1 in OMS3, by converting existing VB6 code to Java (including separating the user interface from the engine) and converting it to a spatio-temporal model. 2. Setting up an OMS3 client to obtain climate data provided through a Sensor Observation Service (SOS) web service, and linking this to the hydrology model. 3. Visualising time series results by setting up an OMS3 component that employed the java JFreeChart library (apart from OMS3’s own core component for displaying time series). 4. Retrieving vector and raster input data sets of land cover, rainfall, and potential evaporation from Landcare Research OGC-compliant web services, by building new OMS3-based web-service clients which entailed constructing URLs containing WFS/WCS complaint key value pair parameters and submitting these using wget requests. 5. Manipulating spatial data by building a component that used GDAL geo-processing libraries. 6. Converting the point hydrology model to spatial model by setting up and OMS3-based component to call the point model for each pixel of a grid and create an ASCII grid file as output. 7. Setting up the farm nutrient loss model OVERSEER2 by various means: a) wrapping FORTRAN calls to a Delphi-based dll; b) using the jni4net bridge to call a .NET-based dll from Java; and c) accessing a version of OVERSER that is provided as a web service that accepts input file with a simple http POST command and retrieves the output file 8. Developing a simple user interface component to manipulate rainfall input in the OVERSEER XML input file, and extract and display the results. 9. Linking the OVERSEER component to the network aggregation and routing components of the catchment water quality model CLUES3. This was achieved by running a FORTRAN-based programme that calls OVERSEER repeatedly and routes nutrient loads downstream, using FORTRAN directly within OMS3. 10. Setting up components of the mixed-language (Fortran, C++, VB.NET, C#) simulation model APSIM (Holzworth et al., 2010; Keating et al., 2003) as OMS3 components. This was unsuccessful within the study timeframe, due to the difficulty of setting up complex components using jni4net or wrapping components that themselves interact in a complex way.

1

http://icm.landcareresearch.co.nz/research/land/Landcovereffectsonwater.asp www.overseer.org.nz 3 www.mpi.govt.nz/environment-natural-resources/water/clues 2

Elliott A.H. et al. Testing Tales: Selection and Evaluation of a Framework for Interoperable Freshwater Modelling



3

Finally, we developed recommendations for future framework implementation, taking into account results of testing, end-user feedback obtained during a webinar, and further consideration of alternatives and recent developments in the literature (Elliott et al., 2013).

RESULTS

3.1 Compilation of models and datasets used in New Zealand The structured database of models and datasets that are used in New Zealand contained attributes of related to interoperability, such as pre-specified classes of environmental compartment, model scale and resolution, and types of environmental variable modelled (water flow rate, for example), along with general metadata. This data served as input to the ModelVis tool (Figure 1), enabling interactive searching for models, along with graphical display of the potential linkages between models (Figure 1). The potential coupling was identified based on finding components (models or data) that have matching types of input and output items. However, the tool does not address the details of how models can actually be linked, and at present it is limited to showing a single level of coupling. The database and Modelvis are available on the project wiki. This exercise identified the large potential for linking the wide range of models and datasets already available in New Zealand.

Figure 1. Annotated screenshot from the ModelVis tool which can be used to query models and show potential couplings.

3.2 User types and needs, and framework screening and selection The following range of users was identified in the end-user workshop: Indirect Users – policy makers, resource managers, and others who want to ask relevant questions of freshwater models and apply the results to assist with policy development and resource management but who by and large do not want to use or operate the models themselves. Interpreters – users who sit “between” the indirect users and direct users to help formulate appropriate questions (e.g., scenarios), analyse results, and generate suitable outputs (e.g., maps, charts, other materials) to aid indirect users and the broader community of interest. Direct Users – modelling analysts with the scientific and technical knowledge to apply models competently and with confidence, and modellers who undertake research to develop, test and apply new models.

Elliott A.H. et al. Testing Tales: Selection and Evaluation of a Framework for Interoperable Freshwater Modelling

Developers – informatics researchers, programmers and software engineers who help design, implement and maintain the framework, and join new models and data into the framework. Interoperability frameworks have traditionally assessed the mechanics of linkages, relevant mostly to developers and direct users. Indirect users, on the other hand, stressed aspects such as ability to document model provenance and high-level assumptions. All groups had high aspirations for an interoperable modelling system and thought that a system could give large benefits, such as improving integration of models across freshwater domains, improved integrated decision-making, more efficient development and re-use of model components, and making better use of environmental datasets in standardised form. The needs of a framework identified in the workshop were supplemented by items from project meetings, resulting in a long wish-list. These items were prioritised, and key criteria for screening are shown in Figure 2. Over 18 existing environmental modelling frameworks were identified from a literature review which is documented in a database of frameworks and a report available on the wiki, which also includes references. The initial list of 18 frameworks comprised:  APSIM (Agricultural Production Systems Simulator)  Bespoke Framework Generator  Community Surface Dynamics Modelling System Modelling Tool (CMT)  Delft-FEWS Flood Early Warning System  Ensym  Earth System Modelling Framework  GME (Geospatial Modelling Environment)  Generic Modeling Environment  Hydrologists Workbench  Hydromodeler  ICMS Integrated Component Modelling System  OASIS (Ocean Atmosphere Sea Ice Soil)  OMS3 (Object Modeling System v3)  OpenMI  OpenPALM  Pegasus  Seamless  TIME The Invisible Modelling Environment A first screening was done to reduce the number of frameworks to a shortlist of six. These six frameworks were then evaluated in relation to the major criteria identified by end-users and the project team (Figure 2). The evaluation was necessarily rapid due to the combination of the number of frameworks, the number of criteria and project time constraints. Since we did not have resources to test all the shortlist, we chose one framework for testing. The following frameworks were not taken forward into trialling for the following reasons:  CSDMS and OpenPalm are mainly intended for running complex large-scale models on highperformance computers. There is not an immediate demand for such facilities for freshwater models used in New Zealand. Provision of high-performance capabilities comes at the cost of difficulty of programming models for the interface and difficulties in making the models accessible to end-users who just wish to run the assemblage of models with a user interface.  There are concerns about the longevity of TIME. Development on the framework seems to have ceased, and the current funding programmes have finished. The championing organisation, CSIRO, has been recently more committed to developing sophisticated end-user applications such as Source that are funded on a subscription basis.  Pegasus is attractive in the sense that it provides workflow functionality, and hence might be able to provide provenance and auditing information better than other frameworks. However, Pegasus is mainly intended for large-scale scientific workflows, rather than for running small collections of freshwater models. It relies on models being set up as executable files and relies

Elliott A.H. et al. Testing Tales: Selection and Evaluation of a Framework for Interoperable Freshwater Modelling



largely on file-based data transfer, so it is not very amenable to setting up interfaces for enduser assemblages. OpenMI was similar to OMS3 in capabilities and primary domain, having been developed primarily to link existing hydrologic and hydraulic models or model components. However at the time of the initial evaluation, the status of OpenMI was in question because the projects that originally funded development had ended and support was transitioning to an unfunded contribution basis. Further, at the time there was little software available to make use of the OpenMI standard.

The framework selected for testing was OMS3 (David et al., 2013). That framework was developed by the US Department of Agriculture (USDA) for component-based model and simulation development on multiple computer platforms. It was developed mainly for simulations of effects of agricultural systems on water quantity and quality, although broader hydrological modelling is also done with this framework and the framework design is fairly general. A key aspect of OMS3 was the desire of the USDA to foster both new model development and re-use along with integration of existing legacy models. This framework was selected over others that met a similar proportion of the criteria because it met many of the criteria, it originated from the agricultural-environmental domain, and there was at the time uncertainty about the longevity of OpenMI. This selection of the framework for testing was a marginal decision, especially in relation to OpenMI, so alternatives were re-considered after trialling OMS3. Category

Criterion

Auditable

Previous model/data/parameter assemblages can stored and re-run

Dynamic

Supports dynamic models

Dynamic

Supports static models

Extensible

Can add/remove/substitute components, not a fixed assemblage

Open

CSDMS

OMS3

OpenMI

OpenPalm

Pegasus

  

  

  

  

  

  

 

 

 

 



 

 

 

 



?



=

=

=

x   

x   x  

Open

Framework software is open-source Framework is available for anyone to use at low or nil cost Uses open standards for model interfaces

Open

Uses open standards for data interfaces

?

=

=

Powerful

Can work at a range of spatial scales

Spatial

Supports spatial models

 

 

Spatial

Supports geospatial data

=

=

=

Stable

Continuity and good prospects of longevity

Usable

GUI for configuring and setting up models

 

   

x  

 =

 

Open

TIME

=

=

?



Figure 2. Preliminary assessment of a short-list of frameworks against key assessment criteria. The symbols ? and = indicate where we did not have enough information, or there the framework might or might not meet the criterion. References for the models and further descriptions of the criteria are given in the project wiki.

3.3 Testing and assessment of the selected framework Most of the trials described the Section 2 with OMS3 were successful, demonstrating the feasibility of using OMS3 to couple freshwater models together in a variety of ways, and also of coupling models to input environmental datasets. Further technical details of the testing are presented in the study final report (Elliott et al., 2013) which is available on the project wiki. Testing with models and datasets specific to New Zealand gave local credibility to the applicability of OMS3. We were able to set up components from a variety of code and executable/dll sources, link and run them, develop simple user interfaces, visualise the results, and access data and simple models provided as web services. In some cases, especially where the user interface was coupled closely to the calculations, it was best to rewrite the model in Java rather than attempting to work with the existing code. It was necessary to have

Elliott A.H. et al. Testing Tales: Selection and Evaluation of a Framework for Interoperable Freshwater Modelling

competent Java programmers involved in setting up OMS3 components, writing the scripts that link files, and writing user interfaces. We had variable success setting up .NET-based components provided as dll’s. In some cases we resorted to wrapping, in other cases we were able to bridge using jni4net, while bridging to more complex inter-related dll’s was not tractable in the timeframe of this project. Overall, we demonstrated that OMS3 has many of the desired building blocks for a freshwater modelling framework for New Zealand. We also assessed other aspects of OMS3 without formal testing. Attractive features included strong institutional (USDA) support for the core framework development, facilities for documentation of models and datasets in a component assembly, metadata creation, and some automatic calibration features. Current limitations of OMS3 for our purposes include: the apparent lack of a community repository of components; no core facilities for setting up user interfaces and limited visualisation components, so that polished user interfaces need to be written from scratch; difficulties working with .NET libraries; no support for specialist hydrological data structures such as drainage networks; no support for standard spatial formats such as shapefiles or NetCDF grids; and limited ability to manage complex timing interactions such as asynchronous models. Many of these limitations can be overcome by building new utilities and components, especially ones based in Java, but this requires Java programming experts and accompanying resources. Following a webinar to demonstrate OMS3 and the result of testing, the ‘indirect users’ called for development of a polished showcase project, to prove that complex arrangements of model components and a polished user interface and visualisation could be achieved in OMS3. Also, our testing did not include a computationally-demanding application. Therefore, before OMS3 is adopted, we propose the development of a full polished integrated model in New Zealand, including a computationally-demanding case.

3.4 Reconsideration of alternatives and future development plans Given that some weaknesses were identified in OMS3 (as is reasonable) and the closeness of the results of the screening, we reconsidered alternative frameworks and recent trends in interoperability. An important development since we undertook framework screening was the ratification of OpenMI as a standard by the Open Geospatial Consortium (OGC). Potentially this could lead to wider uptake of OpenMI, especially considering the rapidly-increasing uptake of OGC standards for time series and geospatial data (including within New Zealand). Also, there are several recent papers on integrating OpenMI with other frameworks or data sources (e.g. Castronova et al., 2013b). However, there are significant limitations to OpenMI, such as the limited uptake of OpenMI 2.0 to date, only closed implementations of the standard, considerable computational overheads in some cases (Shrestha et al., 2013), lack of support for parallel computing (Knapen et al., 2013), only low-level data types, and uncertain funding. Hence we do not see that it is best to abandon OMS3 in favour of OpenMI at this stage, but we do recommend periodic review of progress in this area. Also, it may be possible to make OMS3 components OpenMI-compliant in the future. There has been a recent flurry of papers describing various architectures and visions for providing water models as web services (Castronova et al., 2013a; Fritzinger et al., 2012; Goodall et al., 2011; Granell et al., 2013; Laniak et al., 2013; Nativi et al., 2013). This models-as-service approach builds on successful initiatives in the data provisioning area, internationally and in New Zealand. We have already successfully tested some simple models provided as web services in OMS3, and we consider that web services could overcome institutional, IP, and technical barriers to interoperability. For example, such services could help bridge the .NET-Java divide. However, the technology for running linked models as web services is young, and there are some inherent pitfalls such as data transfer delays, so we proposed a staged process to adoption of web services. The staged process would involve initially adopting a set of standards for New Zealand data, then setting up OMS3 components to access these data through standard web services such as SOS and WPS, setting up web-based models as web services linked to OMS3, setting up computational components as web services using OMS3, and finally moving to more web-centric linking and co-ordination technologies as they become available. This will provide early benefits (enhanced data provision to models) with a longer-term pathway to

Elliott A.H. et al. Testing Tales: Selection and Evaluation of a Framework for Interoperable Freshwater Modelling

frameworks that use models as a web service, with associated benefits such as institutions being able to maintain control of their models and being less bound by programming language and platform dependencies 4

CONCLUSIONS AND RECOMMENDATIONS

A prioritised list of user needs, which were often aspirational and demanding, was identified. A framework should serve a range of users of a framework, including high-level indirect users, but mostly frameworks are oriented towards developers and direct users. Trialling of a selected framework, OMS3 demonstrated the feasibility of using that framework to couple freshwater models together in a variety of ways, and also of coupling models to input environmental datasets. All of the tests were successful, except for to more ambitious test of setting up APSIM. For example, we were able to set up components from a variety of sources, link and run them, develop simple user interfaces, visualise the results, and access data and simple models provided as web services. Often this involved building new custom components to achieve such tasks as accessing web services and visualizing results, which require considerable programming effort. This effort may become less in the future as the repository of OMS3 utility components matures. The tests with APSIM demonstrated difficulties with setting up complex inter-linked .NET dll’s as OMS3 components. Overall, OMS3 meets many of the key criteria identified by end-users. However, there are still calls for development of a showcase integrated model built with OMS3, scaling up from our experience to date. This showcase would demonstrate the capabilities of OMS3 to the range of end-users, as well as provide a deeper and more demanding level of testing (for more complex and computationallydemanding models and more elaborate user interfaces). The initial screening of frameworks indicated that there are several alternatives that meet many of the key criteria. However, there did not seem to be a compelling reason to choose these alternatives over OMS3. Models as a web service is an attractive option to potentially overcome some of the limitations of OMS3. However, this is still an immature area, so for our purposes we propose to follow a staged uptake process.

ACKNOWLEDGMENTS The authors thank members of the project Steering Group and contributors to the workshop and webinar for their valuable feedback and input to the study. Dr Olaf David generously provided advice on the use of OMS3. This project was funded under Ministry of Business Innovation and Employment contract C01X1003. REFERENCES Castronova, A.M., Goodall, J.L., Elag, M.M., 2013a. Models as web services using the Open Geospatial Consortium (OGC) Web Processing Service (WPS) standard. Environmental Modelling & Software 41, 72-83. Castronova, A.M., Goodall, J.L., Ercan, M.B., 2013b. Integrated modeling within a Hydrologic Information System: An OpenMI based approach. Environmental Modelling & Software 39, 263273. David, O., Ascough, J.C., Lloyd, W., Green, T.R., Rojas, K.W., Leavesley, G.H., Ahuja, L.R., 2013. A software engineering perspective on environmental modeling framework design: The Object Modeling System. Environmental Modelling & Software 39, 201-213. Elliott, S., Alistair, R., Snow, V., 2012. Frameworks for Interoperable Freshwater Models: Initial Evaluation. Prepared for Ministry of Science and Innovation. National Institute of Water and Atmospheric Research, Hamilton, New Zealand. Elliott, S., Turek, G., Snow, V., Rutledge, D., Alistair, R., 2013. Framework for Interoperable Freshwater Models: Testing and Recommendations. Prepared for Ministry of Business, Innovation and Employment. National Institute of Water and Atmospheric Research.

Elliott A.H. et al. Testing Tales: Selection and Evaluation of a Framework for Interoperable Freshwater Modelling

Fritzinger, E., Dascalu, S., Ames, D., Benedict, K., Gibbs, I., McMahon Jr, M.J., FC Jr, H., 2012. The Demeter framework for model and data interoperability, Proceedings of the International Congress on Environmental Modeling and Software (IEMSS-2012), pp. 1535-1543. Goodall, J.L., Robinson, B.F., Castronova, A.M., 2011. Modeling water resource systems using a service-oriented computing paradigm. Environmental Modelling & Software. Granell, C., Diaz, L., Schade, S., Ostlander, N., Huerta, J., 2013. Enhancing integrated environmental modelling by designing resource-oriented interfaces. Environmental Modelling & Software 39, 229-246. Holzworth, D.P., Huth, N.I., de Voil, P.G., 2010. Simplifying environmental model reuse. Environmental Modelling & Software 25, 269-275. Keating, B.A., Carberry, P.S., Hammer, G.L., Probert, M.E., Robertson, M.J., Holzworth, D.P., Huth, N.I., Hargreaves, J.N.G., Meinke, H., Hochman, Z., McLean, G., Verburg, K., Snow, V.O., Dimes, J.P., Silburn, D.M., Wang, E., Brown, S.D., Bristow, K.L., Asseng, S., Chapman, S.C., McCown, R.L., Freebairn, D.M., Smith, C.J., 2003. An overview of APSIM, a model designed for farming systems simulation. Eur. J. Agron. 18, 267–288. Knapen, R., Janssen, S., Roosenschoon, O., Verweij, P., de Winter, W., Uiterwijk, M., Wien, J.-E., 2013. Evaluating OpenMI as a model integration platform across disciplines. Environmental Modelling & Software 39, 274-282. Laniak, G.F., Olchin, G., Goodall, J., Voinov, A., Hill, M., Glynn, P., Whelan, G., Geller, G., Quinn, N., Blind, M., 2013. Integrated environmental modeling: a vision and roadmap for the future. Environmental Modelling & Software. Ministry for the Environment, 2013. Freshwater reform 2013 and beyond. Ministry for the Environment, Wellington, New Zealand. Nativi, S., Mazzetti, P., Geller, G.N., 2013. Environmental model access and interoperability: The GEO Model Web initiative. Environmental Modelling & Software 39, 214-228. Shrestha, N.K., Leta, O.T., De Fraine, B., van Griensven, A., Bauwens, W., 2013. OpenMI-based integrated sediment transport modelling of the river Zenne, Belgium. Environmental Modelling & Software 47, 193-206. Snow, V., Fenton, T., Elliott, S., Rutledge, D., Turek, G., 2011. Interoperable Freshwater Models: Report on the End User Workshop on 20 June 2011. Prepared for the Ministry of Science and Innovation. National Institute of Water and Atmospheric Research, Hamilton, New Zealand.