Web Processing Services ( WPS ) for climate data

Web Processing Services for Climate Data EMS 2014

Web Processing Services ( WPS ) for climate data possible tools of inter-project collaboration

Nils Hempelmann1, Carsten Ehbrecht2, Stephan Kindermann2 1 2

Laboratoire des Sciences du Climat et de L'Environnement (LSCE-IPSL), Paris, France German Climate Computing Centre (DKRZ), Hamburg, Germany

[email protected]

07.Sep.2014

1

Web Processing Services for Climate Data EMS 2014

Earth System Grid Federation (ESGF) Data are stored in standardized (quality controlled) formats. Grid of data providers

http://esgf.org/

http://esgf-data.dkrz.de http://esgf-node.ipsl.fr

[email protected]

07.Sep.2014

2

Web Processing Services for Climate Data EMS 2014

Data volume grows quick. Limited storage/compute capacities at client side

“download and process at home” computation at the data centers

Web Processing Service to trigger compute processes at data center

[email protected]

07.Sep.2014

3

Web Processing Services for Climate Data EMS 2014

Job Submit

Extraction

Data Archive

Web Processing Service

Process Status Scientist

Data Archive [ESGF] Result

[email protected]

07.Sep.2014

4

Web Processing Services for Climate Data EMS 2014

Bird-house Based on Open Source ➢ Open Geospatial Consortium (OGC) Standards ➢ Easy to install (tesed on (K)Ubuntu, CentOs, Redhead, Fedora) ➢

https://mouflon.dkrz.de ➢ https://github.com/bird-house ➢ https://lists.dkrz.de/mailman/listinfo/wps ➢

[email protected]

07.Sep.2014

5

Web Processing Services for Climate Data EMS 2014

http://mouflon.dkrz.de [malleefowl]

 Base Processes • data access processes (e.g. for ESGF) • simple workflow engine to combine data access and processing steps • separate WMS visualization (thredds)

[phoenix]

 Web browser application • GUI (automatically configured based on WPS input/output description) • ESGF faceted search

[flyingpigeon] [hummingbird] [emu] [email protected]

 Data Processing for Impact comunity  Data Processing based on CDO commands  Test processes

07.Sep.2014

6

Web Processing Services for Climate Data EMS 2014 Bird - House ESGF Data Archive

Solr Index Server

Identity Provider OpenID X509

Phoenix GUI

Visualisation ncWMS Service

PyCSW Catalog Service Register at Catalog Service Malleefowl PyWPS Services Workflow Engine Data Download with Cache

Hummingbird Emu PyWPS Services Flyingpigeon PyWPS Services PyWPS Services Climate Impact Processes

Other (OGC) Other Web Processing Web Processing Services Services

Open Climate Gis

Processes

R

software

Publish Results Climate Data netCDF

[email protected]

WPS Process Result Outputs

07.Sep.2014

7

Web Processing Services for Climate Data EMS 2014

[email protected]

07.Sep.2014

8

Web Processing Services for Climate Data EMS 2014

[email protected]

07.Sep.2014

9

Web Processing Services for Climate Data EMS 2014

[email protected]

07.Sep.2014

10

Web Processing Services for Climate Data EMS 2014

[email protected]

07.Sep.2014

11

Web Processing Services for Climate Data EMS 2014

[email protected]

07.Sep.2014

12

Web Processing Services for Climate Data EMS 2014

from owslib.wps import WebProcessingService, monitorExecution wps = WebProcessingService(url="https://mouflon.dkrz.de/wps", \ verbose=False, skip_caps=False) execute = wps.execute( identifier="de.csc.visualisation.worker", inputs=[ ("variable", "tas"), ("file_identifier", "https://localhost/thredds/fileServer/test/file1.nc"), ("file_identifier", "https://localhost/thredds/fileServer/test/file2.nc"), ("file_identifier", "https://localhost/thredds/fileServer/test/file3.nc")], output=[("output", True)])

[email protected]

07.Sep.2014

13

Web Processing Services for Climate Data EMS 2014

[email protected]

07.Sep.2014

14

Web Processing Services for Climate Data EMS 2014

WPS – Eco System Bird-house Data Center2

Birdhouse Shared results

Data Center1

“private” archive

Bird-house Institute

Archive Climate Data

[email protected]

Archive “other” Data

07.Sep.2014

ESGF Archive Climate Data

“private” results

15

Web Processing Services for Climate Data EMS 2014 intern access for registered users

Extern access for registered users

Compute Provider [2] https://testVM.jussieu.ipsl.fr

Compute Provider [1] https://mouflon.dkrz.de

Pull

Pull

GitHub (public repository) Malleefowl Base processes

Pyramid-Phoenix Web application

Hummingbird CDO based processes

EMU Test processes

Local repository WPS Builder buildout

Flyingpigeon Climate impact

Pull localhost users [email protected]

Pull

Barnowl intern processes

pull / push localhost developer

07.Sep.2014

16

Web Processing Services for Climate Data EMS 2014

Birdhouse tool for inter project / working group / institution cooperation Process Developer

Bird – house

Software as A Service

repository Paper Product

intern WPS specific processes

public WPS standardized processes

Scientific User Scientific User

Paper Product [email protected]

result data archive result data archive

07.Sep.2014

17

Web Processing Services for Climate Data EMS 2014

Newsletter subscribe: https://lists.dkrz.de/mailman/listinfo/wps https://mouflon.dkrz.de ➢ https://github.com/bird-house ➢

Thank You ;-)

[email protected]

07.Sep.2014

18