Manage datasets via API
You can create, update, and delete dataset via the API of data.europa.eu.
Access Control and Tokens
data.europa uses Keycloak as backbone for access control. Hence, every interaction with a restricted API endpoint requires an interaction with Keycloak to obtain an access token (Party Token). It requires two API calls to get the token.
Prerequisites
- You require valid credentials (username and password) for the Keycloak.
- You need any tool to interact with a HTTP API, such as Postman or curl for the command line.
Step 1 - Request User Token
First you will need to require a User Token by performing a x-www-form-urlencoded POST request to the following endpoint:
https://data.europa.eu/auth/realms/DEU/protocol/openid-connect/token
The following form values need to be set:
Key | Value |
---|---|
username | [yourusername] |
password | [yourpassword] |
grant_type | password |
client_id | piveau-hub-ui |
Example with curl:
$ curl --location --request POST "https://data.europa.eu/auth/realms/DEU/protocol/openid-connect/token" \
--header "Content-Type: application/x-www-form-urlencoded" \
--data-urlencode "grant_type=password" \
--data-urlencode "cliend_id=piveau-hub-ui" \
--data-urlencode "username=[yourusername]" \
--data-urlencode "password=[yourpassword]"
If successful you will get a JSON response like this:
{
"access_token": "[yourusertoken]",
"expires_in": 300,
"refresh_expires_in": 1800,
"refresh_token": "[yourrefreshtoken]",
"token_type": "Bearer",
"not-before-policy": 0,
"session_state": "694350c7-38b9-4051-bc2e-e15e34320133",
"scope": "email profile"
}
Step 2 - Request Party Token
Now you will need to require a Party Token by again performing a x-www-form-urlencoded POST request to the following endpoint:
https://data.europa.eu/auth/realms/DEU/protocol/openid-connect/token
The following form values need to be set:
Key | Value |
---|---|
grant_type | urn:ietf:params:oauth:grant-type:uma-ticket |
audience | piveau-hub-repo |
In addition, place the User Token from Step 1 into the header field Authorization with the leading string Bearer.
Example with curl:
$ curl --location --request POST "https://data.europa.eu/auth/realms/DEU/protocol/openid-connect/token" \
--header "Content-Type: application/x-www-form-urlencoded" \
--header "Authorization: Bearer [yourusertoken]" \
--data-urlencode "grant_type=urn:ietf:params:oauth:grant-type:uma-ticket " \
--data-urlencode "audience=piveau-hub-repo "
If successful you will get a JSON response like this:
{
"upgraded": false,
"access_token": "[yourpartytoken]",
"expires_in": 300,
"refresh_expires_in": 1800,
"refresh_token": "[yourrefreshtoken]",
"token_type": "Bearer",
"not-before-policy": 0
}
Remarks
- For security reasons the tokens expire quickly. You can refresh them by just performing Step 1 and 2 again.
Manage Datasets
This guide gives an overview on how to manage (create, update and delete) datasets in data.europa.eu via its API.
General Remarks
- You will the a token to perform the following operations. See Access Control and Tokens for details.
- The entire API for dataset management is document with OpenAPI here.
- You will require at least write access to one catalogue. Please contact the data.europa.eu administrators for further information.
Example Dataset
The following DCAT-AP dataset will be used as an example for this guide. It is serialised in Turtle. You can provide the dataset in any other common RDF format, such as RDF/XML, JSON-LD, N-Triples, Trig or N3.
@prefix dcat: <http://www.w3.org/ns/dcat#> .
@prefix dct: <http://purl.org/dc/terms/> .
<https://example.eu/set/data/test-dataset>
a dcat:Dataset ;
dct:title "DCAT-AP 2.1.0 Example Dataset"@en ;
dct:description "This is an example Dataset"@en ;
dcat:theme <http://publications.europa.eu/resource/authority/data-theme/TECH> ;
dcat:distribution <https://example.eu/set/distribution/1> .
<https://example.eu/set/distribution/1>
a dcat:Distribution ;
dct:format <http://publications.europa.eu/resource/authority/file-type/CSV> ;
dcat:accessURL <https://github.com/ec-jrc/COVID-19/blob/master/data-by-country/jrc-covid-19-countries-latest.csv> .
Create or Update a Dataset
Creating or updating a dataset is performed with the same endpoint:
https://data.europa.eu/api/hub/repo/datasets/[dataset_id]?catalogue=[catalog_id]
The [dataset_id] can be freely chosen by you and the [catalog_id] determines the catalogue to which the dataset is added. The dataset ID is scoped within the catalogue. If the combination of dataset ID and catalogue ID already exists, the dataset is updated. Otherwise, a new dataset is created.
Danger
If you create a new dataset it is highly recommend to check if the dataset ID is already taken within the scope of the catalogue. Just perform a GET request to https://data.europa.eu/api/hub/repo/datasets/[dataset_id]?catalogue=[catalog_id]
Now you just perform a PUT request to the endpoint by providing the Party Token and the RDF format in the header. The actual dataset is provided in the body of the request.
Example with curl:
$ curl --location --request PUT "https://data.europa.eu/api/hub/repo/datasets/example-dataset?catalogue=test-catalog" \
--header "Content-Type: text/turtle" \
--header "Authorization: Bearer [yourpartytoken] \
--data-raw "@prefix dcat: <http://www.w3.org/ns/dcat#> .
@prefix dct: <http://purl.org/dc/terms/> .
<https://example.eu/set/data/test-dataset>
a dcat:Dataset ;
dct:title \"DCAT-AP 2.1.0 Example Dataset\"@en ;
dct:description \"This is an example Dataset\"@en ;
dcat:theme <http://publications.europa.eu/resource/authority/data-theme/TECH> ;
dcat:distribution <https://example.eu/set/distribution/1> .
<https://example.eu/set/distribution/1>
a dcat:Distribution ;
dct:format <http://publications.europa.eu/resource/authority/file-type/CSV> ;
dcat:accessURL <https://github.com/ec-jrc/COVID-19/blob/master/data-by-country/jrc-covid-19-countries-latest.csv> ."
The following Content-Type values are valid:
Format | Value |
---|---|
RDF/XML | application/rdf+xml |
Turtle | text/turtle |
JSON-LD | application/ld+json |
N3 | text/n3 |
Trig | application/trig |
N-Triples | application/n-triples |
If the request was successful you will receive a 201 response for a newly created dataset or a 204 for an updated dataset.
Delete a Dataset
You can delete a dataset by performing a DELETE to the same endpoint.
$ curl --location --request DELETE "https://data.europa.eu/api/hub/repo/datasets/example-dataset?catalogue=test-catalog" \
--header "Authorization: Bearer [yourpartytoken]
If the request was successful you will receive a 204 response.