2 Environmental Systems Ltd,. 3 University of Aberystwyth, ... QA workflow Authoring Tool & WPS. ⢠Access control and ... elements throughout the QA workflow ...
The COBWEB Quality Assurance System in Practice: Example for an Invasive Species Study Didier Leibovici1, Jamie Williams2, Julian Rosser1, Crona Hodges3, Douglas Scott4, Colin Chapman4, Chris Higgins5 and Mike Jackson1 1
University of Nottingham
Environmental Systems Ltd, 5 University of Edinburgh Welsh Government, UK 2
4
3
University of Aberystwyth,
Session 09: Data, metadata, quality and visualisation of citizen science data Jamie Williams CSci Environment Systems; Suvodeep Mazumdar, University of Sheffield, UK; Arne J. Berre, SINTEF, Norway
http://cobwebproject.eu
mobile data capture & Quality Assurance / Conflation
quality vs uncertainty
QUALITY is about how UNCERTAIN we are about the data
Key components of the COBWEB architecture
survey management & for mobile app building
• Multi-source Retrieval
ISO19157 …
Middleware Service for Sensors
• Quality Assurance & Conflation
ß
QA workflow Authoring Tool & WPS
• Access control and privacy single sign on, federation, etc..
• Co-design projects – Biological monitoring – Flooding – Validation of Earth Observation products
BPMN 2.0
Semantic Framework …RDF
Interoperability standards
• Portal software & Survey Authoring tool
WPS, WFS, WCS CSW …
Quality models Extendeding the ISO model
•
ISO19157:
• • • • • • • • • • • • • • • • • • • • •
DQ_Usability DQ_Completeness DQ_CompletenessCommission DQ_CompletenessOmission DQ_ThematicAccuracy DQ_ThematicClassificationCorrectness DQ_NonQuantitativeAttributeAccuracy DQ_QuantitativeAttributeAccuracy DQ_LogicalConsistency DQ_ConceptualConsistency DQ_DomainConsistency DQ_FormatConsistency DQ_TopologicalConsistency DQ_TemporalAccuracy DQ_AccuracyOfATimeMeasurement DQ_TemporalConsistency DQ_TemporalValidity DQ_PositionalAccuracy DQ_AbsoluteExternalPositionalAccuracy DQ_GriddedDataPositionalAccuracy DQ_RelativeInternalPositionalAccuracy
GeoViqua: GVQ_PositiveFeedback GVQ_NegativeFeedback
Stakeholder model: (as COBWEB Stakeholder Quality Model) where DQ_Scope will be "user" CSQ_Ambiguity CSQ_Vagueness CSQ_Judgement CSQ_Reliability CSQ_Validity CSQ_Trust
QAQC: the COBWEB QAQC 7+ pillars
Meek, S Jackson, M Leibovici, DG (2014) ) A flexible framework for assessing the quality of crowdsourced data .AGILE conference, 3-6 June 2014, Castellón, Spain
Interoperability example QAQC with authoring tool and WPS calls QAwAT .Workflow authoring tool BPMN encoding
(QAwOnt) .ontology support SKOS encoding The QA workflow is composed of more than one QC into a workflow that may loop back /feedback to the user or to other users etc. to get additional information. (confirmatory / ensemble / linked data )
QAwWPS .running WPS or app
Quality models
• DQ_ xxx ISO19157 • GVQ_xxx User Feedback (GeoviQua) • CSQ_xxx (COBWEB StakeHolder Quality odel)
QC’characteristics: pillar / geocomputation algorithm / rules to generate DQ_ GVQ_ and CSQ_
Invasive species: the Japanese Knotweed
QCJKW examples QA workflow
Qualifying the Observations, the Volunteers and the Authoritative data Quality elements generated & evolving Obs /Auth - ISO19157 standard Auth - GeoViQUA-feedback model Vol-COBWEB-Stakeholder Quality Model
QC examples DQ_TopologicalConsistency:
correctness of the explicitly encoded topological characteristics of the
dataset as described by the scope
ThresholdDistance
LoS
!
pillar1 LBS-Positioning Relativeposition to Line of Sight Interactivity with the user (messages)
ObservedDistance UncertaintyDistance DQ_TopologicalConsistency =prob( N(d,s^2) 0.75
2
3/16=18% 54/161=33%
4/16=25% 82/161=51%
1
0.5
16 wrong 161 right
> 0.75
16 wrong 161 right
1.5
0.5
/ / //
0.0
0.2
0.4
0.6
DQ_ClassificationCorrectness
/ / / ||||||||||||| | ||||||||||| 0.8
0.0
/ / / / // /// / / | | |||| |||||||||| |||||||||||||||||||||||||||||||
|| | | |
0
Density
N=319 NotJKW JKW**
(**ground truth from photo)
1.0
Some results
1.0
/
/
/ /
/
| | |||| | || | || | |||||||||| |||| | |||| | ||| | || | | | | || || 0.0
0.2
0.4
0.6
/
//
/
/
| | | ||||| ||||||||||||||| | ||| || |||||||||| || 0.8
1.0
DQ_Usability
* There is a pillar1.WithinPolygon but also pillar4.PointinPolygon
Conclusions… on QAQC and using it! What’s the experience has been like? • QA building is facilitated by the COBWEB tool • Yes ….. NO …. … uncertain? • Parameters of the QCs need ‘expertise’ ...yes but! • Other QCs can be rapid to prototype • Grasp all the DQ_ GVQ_ CSQ_ for “final” validation
Usable?
Useful?
• Workflow editor • Quality controlled ... (BPMN), WPS and • Or Quality Assurance impacting on ‘a priori QA’ scripts and ...fuzzy • Some Obs ending with bad quality logic could be still valuable • SWE4CS as much as the other way around https://github.com/openge (dependency to the QA workflow ’verification’ at random?)
ospatial/swe4citizenscience
QCQAQC examples workflow Authoring Tool (QAwAT)
shun Talking about Quality and accuracy!
Ta!
