Web Coverage Processing Service - National e-Science Centre

Inspecting Geo Service Standards To Spot Data Intensive Challenges Peter Baumann Jacobs University Bremen / rasdaman GmbH

eSi :: Edinburgh :: 2010-nov-23

Jacobs University Bremen  International, multi-cultural • 1,300+ students, 102 nations, 75% non-German • English official language on campus

 Large-Scale Scientific Information Systems

research group • focus: large-scale n-D raster services and beyond • theory, practice, application, standardization • www.jacobs-university.de/lsis

• Main results so far: • rasdaman array DBMS • OGC geo service standardization: WCS, WCPS ACM SIGSPATIAL :: San Jose :: 2010-nov-05

2

The rasdaman Raster Data Manager my_coll

 = raster („array“) DBMS • any size, dimensionality, cell type metadata • multidimensional SQL, Java, C++ • intelligent storage & query optimization

array OID

array

oid 1 att 1

att 2

att n

key1

...

oid 1

key2 key3

... ...

oid 2 oid 3

 rasdl = raster type definition language

oid 2 oid 3

oid 4 oid 5

 rasql = multi-dimensional raster expressions in SQL select img.green[x0:x1,y0:y1] > 130 from LandsatArchive as img

ACM SIGSPATIAL :: San Jose :: 2010-nov-05

3

Domains Investigated  Geo • • • •

Environmental sensor data, 1D Satellite / seafloor maps, 2D

Geophysics (3D, x/y/z) Climate modelling (4D, x/y/z/t)

 Life science • Gene expression simulation (3D) • Human brain imaging (3D / 4D)

 Other • Computational Fluid Dynamics (3D) • Astrophysics (4D+) ACM SIGSPATIAL :: San Jose :: 2010-nov-05

4

Roadmap  OGC geo service standards  Some challenges

 Conclusion

ACM SIGSPATIAL :: San Jose :: 2010-nov-05

5

Surfing the Sensor Data Tsunami

“sensor” feeds

coverage server

Picture stolen from SWE

eSi :: Edinburgh :: 2010-nov-23

6

Surfing the Sensor Data Tsunami

sensors feeds

coverage server

Semantic Web

Picture stolen from SWE

eSi :: Edinburgh :: 2010-nov-23

7

Open Geospatial Consortium  OGC (www.opengeospatial.org) drives geo service standards • In collaboration with ISO, W3C, OASIS Open, …

 Coverage = n-D "space/time-varying phenomenon" [6] • coverage: space, time  value set

 Web Coverage Service (WCS): access to coverages & subsets thereof [4] • deliver data suitable for further processing • Separate coverage model definition [3]

 WCS 2.0 adopted in August 2010 [3,4] • Harmonized with GML, Sensor Web Enablement (SWE) • Extended beyond raster data eSi :: Edinburgh :: 2010-nov-23

8

(Part of) The OGC Quilt data

images

feature

data

data

coverage

meta

FE

CQL

WCPS

WFS-T

WFS

eSi :: Edinburgh :: 2010-nov-23

WCS-T

WMS

WCS

CS-T

CS-W

9

Coverage Definition From GML 3.2.1 «Data Type»

AbstractFeature AbstractCoverage + +

coverageFunction: gml:CoverageFunction [0..1] metaData [0..1] 1

domainSet 1 «Union»

DomainSet

1 rangeType 1

«Data Type»

DataRecordPT

1 rangeSet 1 «Data Type»

RangeSet

From SWE Common

eSi :: Edinburgh :: 2010-nov-23

10

10

WCS Core Service Model: Data WCS offering can be seen as a single virtual document

Hook for future servicerelated coverage metadata eSi :: Edinburgh :: 2010-nov-23

11

11

Coverage Subtypes «FeatureType»

as per GML 3.2.1

Abstract Coverage

Discrete Coverage

MultiSolid Coverage MultiSurface Coverage

eSi :: Edinburgh :: 2010-nov-23

MultiPoint Coverage MultiCurve Coverage

Continuous Coverage Rectified GridCoverage

Grid Coverage

Referenceable GridCoverage

12

WCS Core Functionality In Core, simple data access (more in extension packages): subset =

eSi :: Edinburgh :: 2010-nov-23

trim

|

slice

13

Use Case: Satellite Time Series [DFD-DLR, Diederich et al, 2001]

eSi :: Edinburgh :: 2010-nov-23

14

Web Coverage Processing Service  OGC WCPS standard = aka “XQuery for multi-dimensional coverages” [5] • Bridges WCS & WPS • Ex: for $c in ( Modis1, Modis2, Modis3 ), $m in ( MyMask ) where avg( $c.red * $m ) > 127 return encode( $c.red / $c.nir, "tiff" )

(tiff1, tiff3)

 Function nesting  unlimited complexity, formal algebraic semantics  Expressive power: non-recursive image & signal processing, statistics eSi :: Edinburgh :: 2010-nov-23

15

WCPS Use Case Scenarios ad-hoc navigation, extraction, aggregation, analytics: • Time series

• Image processing • Summary data

• Sensor fusion & pattern mining eSi :: Edinburgh :: 2010-nov-23

[1]

16

Roadmap  OGC geo service standards  Some challenges  Conclusion

eSi :: Edinburgh :: 2010-nov-23

17

Maps As Queries  Observation: we need scaling

+ pixel-wise operations (incl. overlay)  Observed imagery (sat, ocean floor video, ...) • RGB • Grayscale

–

image

–

image * { 1c, 1c, 1c }

–

image * { 42c, 0c, 121c }

 Thematic maps • boolean pixels

 Non-radiometric measurements (elevation, bathymetry, temperature, ...) • No visual semantics, need classification

eSi :: Edinburgh :: 2010-nov-23

–

image  ]-inf,v1]  { r1, g1, b1 } image  ]v1,v2]  { r2, g2, b2 } ... image  ]vn,+inf[  { rn+1, gn+1, bn+1 } 18

Sample Overlay & Classification Query [www.earthlook.org]

 Layers: • 1 background • 1 bathymetry • 2 RGB SELECT png( (marray x in [0:399,0:399] values {255c,255c,255c}) overlay ((scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) < -1300)*{0c,0c,0c} +(-1300.000000< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1289.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1281.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1274.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1272.499999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1270.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1270.499999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1269.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1269.499999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1268.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1268.499999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1267.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1267.499999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1266.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1266.499999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1265.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1265.499999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1264.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1264.499999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1263.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1263.499999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1262.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1261.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1260.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1259.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1256.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1249.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1239.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( +(-1229.999999< scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ) and scale( extend( + (-126.5 < scale( extend( img0[269:349,0:65], [269:395,-60:65] ), [0:399,0:399] ))*{255c,255c,255c}) overlay (scale( extend( img2[124:468,0:578], [124:717,-14:578] ), [0:399,0:399] )) overlay (scale( extend( img3[11375:11578,0:120], [11375:11968,-473:120] ), [0:399,0:399] )) ) FROM Hakon_Bathy AS img0, Hakoon_Dive1_8 AS img1, Hakoon_Dive2_8 AS img2, Hakoon_Dive2b_8 AS img3

eSi :: Edinburgh :: 2010-nov-23

img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65], img0[269:349,0:65],

[269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65] [269:395,-60:65]

), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ), ),

[0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399] [0:399,0:399]

) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) )