Skip to content

Manage datasets via API

You can create, update, and delete dataset via the API of data.europa.eu.

Access Control and Tokens

data.europa uses Keycloak as backbone for access control. Hence, every interaction with a restricted API endpoint requires an interaction with Keycloak to obtain an access token (Party Token). It requires two API calls to get the token.

Prerequisites

  • You require valid credentials (username and password) for the Keycloak.
  • You need any tool to interact with a HTTP API, such as Postman or curl for the command line.

Step 1 - Request User Token

First you will need to require a User Token by performing a x-www-form-urlencoded POST request to the following endpoint:

https://data.europa.eu/auth/realms/DEU/protocol/openid-connect/token

The following form values need to be set:

Key Value
username [yourusername]
password [yourpassword]
grant_type password
client_id piveau-hub-ui

Example with curl:

$ curl --location --request POST "https://data.europa.eu/auth/realms/DEU/protocol/openid-connect/token" \
--header "Content-Type: application/x-www-form-urlencoded" \
--data-urlencode "grant_type=password" \ 
--data-urlencode "cliend_id=piveau-hub-ui" \ 
--data-urlencode "username=[yourusername]" \
--data-urlencode "password=[yourpassword]"

If successful you will get a JSON response like this:

{
    "access_token": "[yourusertoken]",
    "expires_in": 300,
    "refresh_expires_in": 1800,
    "refresh_token": "[yourrefreshtoken]",
    "token_type": "Bearer",
    "not-before-policy": 0,
    "session_state": "694350c7-38b9-4051-bc2e-e15e34320133",
    "scope": "email profile"
}
For then second step you will need [yourusertoken].

Step 2 - Request Party Token

Now you will need to require a Party Token by again performing a x-www-form-urlencoded POST request to the following endpoint:

https://data.europa.eu/auth/realms/DEU/protocol/openid-connect/token

The following form values need to be set:

Key Value
grant_type urn:ietf:params:oauth:grant-type:uma-ticket
audience piveau-hub-repo

In addition, place the User Token from Step 1 into the header field Authorization with the leading string Bearer.

Example with curl:

$ curl --location --request POST "https://data.europa.eu/auth/realms/DEU/protocol/openid-connect/token" \
--header "Content-Type: application/x-www-form-urlencoded" \
--header "Authorization: Bearer [yourusertoken]" \
--data-urlencode "grant_type=urn:ietf:params:oauth:grant-type:uma-ticket " \ 
--data-urlencode "audience=piveau-hub-repo "

If successful you will get a JSON response like this:

{
    "upgraded": false,
    "access_token": "[yourpartytoken]",
    "expires_in": 300,
    "refresh_expires_in": 1800,
    "refresh_token": "[yourrefreshtoken]",
    "token_type": "Bearer",
    "not-before-policy": 0
}
[yourpartytoken] can now be used to manage datasets in data.europa.eu. Please see Manage Datasets for details.

Remarks

  • For security reasons the tokens expire quickly. You can refresh them by just performing Step 1 and 2 again.

Manage Datasets

This guide gives an overview on how to manage (create, update and delete) datasets in data.europa.eu via its API.

General Remarks

  • You will the a token to perform the following operations. See Access Control and Tokens for details.
  • The entire API for dataset management is document with OpenAPI here.
  • You will require at least write access to one catalogue. Please contact the data.europa.eu administrators for further information.

Example Dataset

The following DCAT-AP dataset will be used as an example for this guide. It is serialised in Turtle. You can provide the dataset in any other common RDF format, such as RDF/XML, JSON-LD, N-Triples, Trig or N3.

@prefix dcat:   <http://www.w3.org/ns/dcat#> .
@prefix dct:    <http://purl.org/dc/terms/> .

<https://example.eu/set/data/test-dataset>
    a                               dcat:Dataset ;
    dct:title                       "DCAT-AP 2.1.0 Example Dataset"@en ;
    dct:description                 "This is an example Dataset"@en ;
    dcat:theme                      <http://publications.europa.eu/resource/authority/data-theme/TECH> ;
    dcat:distribution               <https://example.eu/set/distribution/1> .

<https://example.eu/set/distribution/1>
    a                               dcat:Distribution ;
    dct:format                      <http://publications.europa.eu/resource/authority/file-type/CSV> ;
    dcat:accessURL                  <https://github.com/ec-jrc/COVID-19/blob/master/data-by-country/jrc-covid-19-countries-latest.csv> .

Create or Update a Dataset

Creating or updating a dataset is performed with the same endpoint:

https://data.europa.eu/api/hub/repo/datasets/[dataset_id]?catalogue=[catalog_id]

The [dataset_id] can be freely chosen by you and the [catalog_id] determines the catalogue to which the dataset is added. The dataset ID is scoped within the catalogue. If the combination of dataset ID and catalogue ID already exists, the dataset is updated. Otherwise, a new dataset is created.

Danger

If you create a new dataset it is highly recommend to check if the dataset ID is already taken within the scope of the catalogue. Just perform a GET request to https://data.europa.eu/api/hub/repo/datasets/[dataset_id]?catalogue=[catalog_id]

Now you just perform a PUT request to the endpoint by providing the Party Token and the RDF format in the header. The actual dataset is provided in the body of the request.

Example with curl:

$ curl --location --request PUT "https://data.europa.eu/api/hub/repo/datasets/example-dataset?catalogue=test-catalog" \ 
--header "Content-Type: text/turtle" \ 
--header "Authorization: Bearer [yourpartytoken] \
--data-raw "@prefix dcat:   <http://www.w3.org/ns/dcat#> .
@prefix dct:    <http://purl.org/dc/terms/> .

<https://example.eu/set/data/test-dataset>
    a                               dcat:Dataset ;
    dct:title                       \"DCAT-AP 2.1.0 Example Dataset\"@en ;
    dct:description                 \"This is an example Dataset\"@en ;
    dcat:theme                      <http://publications.europa.eu/resource/authority/data-theme/TECH> ;
    dcat:distribution               <https://example.eu/set/distribution/1> .

<https://example.eu/set/distribution/1>
    a                               dcat:Distribution ;
    dct:format                      <http://publications.europa.eu/resource/authority/file-type/CSV> ;
    dcat:accessURL                  <https://github.com/ec-jrc/COVID-19/blob/master/data-by-country/jrc-covid-19-countries-latest.csv> ."

The following Content-Type values are valid:

Format Value
RDF/XML application/rdf+xml
Turtle text/turtle
JSON-LD application/ld+json
N3 text/n3
Trig application/trig
N-Triples application/n-triples

If the request was successful you will receive a 201 response for a newly created dataset or a 204 for an updated dataset.

Delete a Dataset

You can delete a dataset by performing a DELETE to the same endpoint.

$ curl --location --request DELETE "https://data.europa.eu/api/hub/repo/datasets/example-dataset?catalogue=test-catalog" \ 
--header "Authorization: Bearer [yourpartytoken]

If the request was successful you will receive a 204 response.