NINA, Trondheim, 15th September 2016
GBIF data use
Dag Endresen GBIF Norway UiO Natural History Museum in Oslo University of Oslo Thursday, September 15th, 2016 Slides: CC-BY-4.0, GBIF.no
Status 14th September 2016
GBIF enables free and open access to biodiversity data online.
We are an interna2onal government-ini2ated and -funded ini2a2ve focused on making biodiversity data available to all and anyone, for scien2fic research, conserva2on and sustainable development.
GBIF provides a data discovery system that is dependent on resolvable stable iden;fiers for efficient func;onality
global registry
data portal
3
GBIF and GEO Intergovernmental group on earth observations
GEO BON
Biodiversity observa2on network
Data Integration & Interoperability GBIF provides the infrastructure delivering species occurrence data in GEO.
GBIF BY THE NUMBERS
649,054,525
species occurrence records
32,440 datasets
813
data-publishing institutions
http://www.gbif.org | 06 JUN 2016
GBIF BY THE NUMBERS: MAY 2016
+3,818,408
species occurrence records
+4,267 datasets
+4
data-publishing institutions
http://www.gbif.org | 6 JUN 2016
data mobilizatio
DATA PUBLISHED THROUGH GBIF.ORG Occurrence records (millions) 700 650 600 550 500 450 400 350 300 250 200 150 100
http://www.gbif.org | 6 JUN 2016
Asia (lack of data)
Africa (lack of data)
participatio
MAP OF GBIF COUNTRY PARTICIPANTS
Asia (lack of data) Africa (lack of data)
August 2016
data publishin
DATA—BY GBIF PARTICIPANT Other
So
Status May 2016
United States
ut
Cos t
aR
hA fri
ca
ica
Austra li
a
Norway
Denmark Netherlands
Belgium
Germany Spain
Norway Norway
Number of new records published—Top 10 participant Countries (1 to 31 May 2016)
Total number of records published—Top 10 Participant Countries (as of 31 May 2016)
1. United States
3,348,499
6. Belgium
1,620423
1. United States
2. Denmark
2,972,094
7. Netherlands
1,094,804
2. Sweden
3. Germany
2,868,240
8. Australia
859,896
4. Norway
2,322,797
9. Costa Rica
5. Spain
2,238,363
10. South Africa
271,901,500
6. Netherlands
24,241,092
53,776,182
7. Norway
23,811,863
3. United Kingdom
49,786,646
8. Germany
22,151,479
810,035
4. France
39,896,982
9. Finland
16,612,735
436,236
5. Australia
37,489,401
10. Spain
13,630,866
NOTE: Datasets are assigned to countries according to the location of the publishing institution, including aggregated datasets with contributors from many other countries. http://www.gbif.org | 09 JUN 2016
use of gbif.or
DATA DOWNLOAD REQUESTS, BY COUNTRY 1 January – 31 May 2016
Total of
37,552 requests From 5,131 users in 127 countries, islands and territories 1. United States
7128
6. Colombia
2235
2. Mexico
5526
7. Italy
1319
3. Brazil
3079
8. China
1263
4. United Kingdom
2670
9. France
949
5. Spain
2478
10. Australia
858
Norwegian scien2sts generally use Artskart… Requests for download do not necessarily result in data actually being downloaded. Based on country indicated by user login | 06 JUN 2016
data us
CITATIONS IN PEER-REVIEWED RESEARCH Annual number of peer-reviewed publications using GBIF-mediated data
9 JUN 2016
research us
USE CITATIONS, BY COUNTRY OF AUTHORS May 2016 1. United States
15
7. Australia
4
2. Germany
9
7. Brazil
4
3. China
5
9. Canada
3
3. France
5
9. Netherlands
3
3. Spain
5
9. South Africa
3
3. United Kingdom
5
Number of research publications in May 2016 citing use of GBIF-mediated data, ranked by country according to affiliation of author. Top 11 countries shown.
Total 2016
Norway
1. United States
49
5. Brazil
14
2. Germany
22
5. United Kingdom
14
3. France
18
8. Australia
11
4. China
17
8. Spain
11
5. Mexico
14
10. Canada
10
Number of research publications in 2016 citing use of GBIF-mediated data, ranked by country according to affiliation of author. Top 10 countries shown.
10 JUN 2016
research us
RESEARCH EXAMPLES (FOR NORWAY) •
Araújo R, Assis J, Aguillar R, Airoldi L, Bárbara I, Bartsch I, Bekkby T et al. (2016) Status, trends and drivers of kelp forests in Europe: an expert assessment. Biodiversity and Conservation 25(7) 1319-1348.
•
Jb N (2016) Some interesting lichenized fungi from old Fraxinus excelsior and Ulmus glabra in Norway, including four new country records. Graphis Scripta 28(1-2) 17-21.
•
Newbold T, Hudson L, Hill S, Contu S, Gray C, Scharlemann J, Sheil D et al. (2016) Global patterns of terrestrial assemblage turnover within and among land uses. Ecography. hGp://www.gbif.org/country/NO/publica2ons A complete archive of research citing use of GBIF can be accessed at http://www.mendeley.com/groups/1068301/gbif-public-library 10 JUN 2016
GBIF portal:
22,0 million occurrences with loca2ons in Norway. Published from 31 countries worldwide.
Updated 5 September 2016
GBIF portal:
21,5 million occurrences from Norwegian ins2tu2ons. Coverage 219 countries worldwide.
Updated 5 September 2016
Danmark
Finland
Norway
Sweden
Sept 2016
Datasets
Occurrences
Denmark
66 + 1
10 905 213
Finland
54
3 611 729
Iceland
4
458 705
Norway
112 + 2
21 684 727
Sweden
42
53 787 704
Iceland
hGp://www.gbif.org/country/NO
STATUS FOR NORDIC GBIF NODES (DATA HOSTED BY…)
Download data
GBIF DATA PORTAL
SPECIES SEARCH
Portal API webservices
GBIF DATA PORTAL API An interface to access data published through the GBIF network using web services.
PORTAL API GBIF Data Portal API: h9p://api.gbif.org/v1/ (+parameters) Summary and informa2on: hGp://www.gbif.org/developer/summary The RESTful API take search parameters as key=value pairs and respond with json content type. RESTful query format JSON response type
GBIF API SECTIONS • Registry
informa2on about the datasets, organiza2ons (e.g. data publishers), networks and the means to access them (technical endpoints)
• Species
informa2on about species and higher taxa, and u2lity services for interpre2ng names and looking up the iden2fiers (access to all published checklists in the GBIF checklist bank)
• Occurrence
occurrence informa2on crawled and indexed by GBIF and search services to do real 2me paged search and asynchronous download services to do large batch downloads
• Maps
simple services to show the maps of GBIF mobilized content
API EXAMPLE : DATASET Search for datasets by publishing country: http://api.gbif.org/v1/dataset/search?publishingCountry=NO Dataset information (UiO NHM Lichens): http://api.gbif.org/v1/dataset/7948250c-6958-4a29-a670ed1015b26252 Contacts for a dataset: http://api.gbif.org/v1/dataset/7948250c-6958-4a29-a670ed1015b26252/contact Dataset endpoint (get the download URL): http://api.gbif.org/v1/dataset/7948250c-6958-4a29-a670ed1015b26252/endpoint
http://www.gbif.org/developer/registry
API EXAMPLE : SPECIES List all name usages (across all checklists): http://api.gbif.org/v1/species?name=Beta%20vulgaris Name usage across checklists (Beta vulgaris, 5383920): http://api.gbif.org/v1/species/5383920/related Name parsed into epithets and author etc.: http://api.gbif.org/v1/parser/name?name=Abies%20alba %20Mill.%20sec.%20Markus%20D. {"scientificName": "Abies alba Mill. sec. Markus D.", "type": "SCINAME", "genusOrAbove": "Abies", "specificEpithet": "alba", "authorsParsed": true, "authorship": "Mill.", "sensu": "sec. Markus D.", "canonicalName": "Abies alba", "canonicalNameWithMarker": "Abies alba", "canonicalNameComplete": "Abies alba Mill." }
http://www.gbif.org/developer/species
API EXAMPLE : OCCURRENCE List occurrences of Beta vulgaris: http://api.gbif.org/v1/species/match?name=Beta+vulgaris => taxonKey
http://api.gbif.org/v1/occurrence/search?taxonKey=5383920 List occurrences from Norway (of Beta vulgaris): http://api.gbif.org/v1/occurrence/search?publishingCountry=NO http://api.gbif.org/v1/occurrence/search?publishingCountry=NO&taxonKey=5383920
Information about a single occurrence record: http://api.gbif.org/v1/occurrence/1040970640 http://api.gbif.org/v1/occurrence/1040970640/fragment http://api.gbif.org/v1/occurrence/1040970640/verbatim
List occurrence counts for datasets of country (or taxon): http://api.gbif.org/v1/occurrence/counts/datasets?country=NO
http://www.gbif.org/developer/occurrence
API EXAMPLE : DOWNLOAD DATA Lookup speciesKey (1) and download occurrences (2): http://api.gbif.org/v1/species/match? verbose=false&kingdom=Plantae&name=Beta+vulgaris => usageKey/speciesKey = 5383920 http://api.gbif.org/v1/occurrence/search? taxonKey=5383920 [&limit=1000&offset=0] => notice: count = 25 513 => then: page through results… (using offset & limit) http://api.gbif.org/v1/occurrence/download/request [POST] => downloadKey (see next slide)
API EXAMPLE : ASYNCHRONOUS (1) Request asynchronous download: $ curl -i --user yourGbifUserName:yourGbifPassord -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d @filter.json http://api.gbif.org/v1//occurrence/ download/request >> log.txt Search parameters in a json text file: filter.json (in current directory or located in a “PATH-directory”): { "creator":”yourGbifUserName", "notification_address": [“
[email protected]"], "predicate": { "type":"and", "predicates": [{"type":"equals","key":"HAS_COORDINATE","value":"false"}, {"type":"equals","key":"TAXON_KEY","value":"5383920"}] } }
DOWNLOADS ARE AVAILABLE IN THE PORTAL (FROM YOUR USER PROFILE)
API EXAMPLE : ASYNCHRONOUS (2A) Request asynchronous download: function gbifapi { curl -i –user yourGbifUserName:yourGbifPassword -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d "{\"creator\": \”yourGbifUserName\", \"notification_address\": [\”
[email protected]\"], \"predicate\": {\"type\":\"and\", \"predicates\": [{\"type\":\"equals\",\"key\": \"HAS_COORDINATE\",\"value\":\"true\"}, {\"type\":\"equals\", \"key\": \"TAXON_KEY\", \"value\":\"$1\"}] }}" http://api.gbif.org/v1/occurrence/ download/request >> log.txt echo -e "\r\n$1 $2\r\n\r\n----------------\r\n\r\n" >> log.txt } $ gbifapi 4140730 "Aciachne acicularis" $ gbifapi 4140704 "Aciachne flagellifera" $ gbifapi 5289784 "Aegilops comosa” …
API EXAMPLE : ASYNCHRONOUS (2B) (…clean log.txt with the downloadKeys using regular expressions…) function gbifwget { echo -e "\n\n----------------\n$1 $2 $3\n" >> log_wget.txt wget http://api.gbif.org/v1/occurrence/download/request/$1.zip 2>&1 | tee / dev/tty >> log_wget.txt mv $1.zip ./dwca/$2.zip 2>&1 | tee /dev/tty >> log_wget.txt }
$ gbifwget 0006050-141024112412452 4140730 "Aciachne acicularis" $ gbifwget 0006053-141024112412452 4140704 "Aciachne flagellifera" $ gbifwget 0006056-141024112412452 5289784 "Aegilops comosa" …
(work in progress…)
MAPPING API V1.0
Slide by Daniel Amariles, 2013
You can easily overlay GBIF content on your own maps. http://www.gbif.org/developer/maps
MAPPING API V1.0
Slide by Daniel Amariles, 2013
This service is intended for use with commonly used map clients such as the Google Maps API, Leaflet JS library or Modest maps JS library. hGp://leafletjs.com/ hGp://modestmaps.com/ These libraries allow the GBIF layers to be visualized with other content, such as those coming from Web Map Service (WMS) providers. It should be noted that the mapping API is not a WMS service, nor does it support WFS capabili2es.
USEFUL TOOLS (JSON & REST) REST client … JSON client/parser … JSONView
(Firefox, Chrome, …) http://jsonview.com/ Display formatted JSON in browser
R CRAN : jsonlite http://cran.r-project.org/web/packages/jsonlite/
E.g. read json into a dataframe [link]
OpenRefine http://openrefine.org/
ROPENSCI : RGBIF library(rgbif) key
