Using the Sources API
This page is part of the Developer Documentation and zooms in on how you can use the Sources API of JoinData. Using this API you can register your data sources (as data source provider) as well as query the available data sources and data sets.
- Introduction
- How does the Source Registry get its data
- API
- Sources API
- Postman
- Step-by-step guide for adding sources to the Sources-API
Introduction
Whenever the DataHub receives a query, it needs to know which source may have data for that query. This is where the Sources Registry (which implements the Sources API) comes in. The Mandate Registry checks if a query is allowed, the Sources Registry checks who may have the answer.
As described in Conceptual Changes in the general Developer Documentation, in the early days of JoinData, mandates were more like configured routes: a mandate allowed the transport of a single data type from a single source to a single target. It represented a ‘paper contract’ somewhere in the ecosystem. With the arrival of the Purpose Registry, mandates are fully digital and more flexible: no longer is there a need to specify a mandate for any source that may contain data the target is interested in. Instead, you can configure a mandate for one or more data sets (a collection of data types) and without specifying a source. The Sources Registry is responsible for answering the question: which sources should be queried?
How does the Source Registry get its data
The problem of finding out which sources can deliver data is thus delegated to the Source Registry. It is a database containing possible sources for locations and other schemes. It does not know if (new) data is available, or if it can answer that specific query, but it knowns that there is a good chance that it can answer that data.
But how do we find these sources? There are a couple of ways the source registry gets its data.
- The Sources API: any data source can actively manage its JoinData data sources. It can create, update or remove its sources for specific locations. In this way, we always have an active and up to date record of which sources are available where.
- Source Discovery: a farmer may configure devices manually. For example, many OEM’s (Original Equipment Manufacturers) do not exactly know where their devices end up. They do however have unique, unguessable device id’s that can be configured by a farmer. This then leads to one or more sources in the Sources Registry.
- Mappings: via the Company Mapping service, companies can indicate which customers they have. This is a strong indication that they will also have data for those parties. A default source mapper is created per company which automatically creates sources for new customers. E.g., if a feed company has a farmer as a customer, we may assume that that feed company is a source for invoices and feed delivery messages.
- Push sources: some data sources simply push a message into our platform. The simple fact of data being there means that there is a data source. Meta-data is extracted from that message and is send to the sources registry.
- Wildcards: for some sources it does not make sense to keep track of each individual customer. For example, governmental or national sources simply have data on all the farmers in their region. As such, we can configure a wildcard that specifies that they have specific data types for an entire region.
This documentation further describes how you can get data from the Sources Registry and how you can register your own data sources as per the first bullet.
API
The API-docs of JoinData can be found here.
Sources API
The Sources API houses the register of all the different source JoinData is connected to. As a data-supplier you wil receive a user-id and a provider-id. A provider is a platform of a data-supplier. With the user-id and provider-id a data-supplier can add sources to the register. A source can be an application or a device on the farm. A more detailed description of adding sources is provided below.
Postman
To communicate with our API’s Postman can be used. On the API-docs page a Postman-collection with all relevant API-calls can be found. Postman can be downloaded https://www.getpostman.com/.
Step-by-step guide for adding sources to the Sources-API
- Receive credentials from JoinData admin.
- Get acces token using your client credentials
- Retrieve provider-id
- Use GET /providers
- Select provider-id you wish to add sources to
- Add a new source
- Use POST /providers/{provider-id}/sources, provide the following variables in the body:
- name is the name given to the source (i.e. Scale 1)
- externaldentifier is the unique identifier for the source within the platform of the provider (i.e. mac-adress)
- filterLocations is the setting whether the source can provide for all locations, this is usual set to true.
– When set to true, the available locations should be linked to the source.
– When false, all calls are forwarded to the provider, regardless of location. Permission from JoinData is needed for this option.
- GET, PATCH and DELETE are also available for sources
- Use POST /providers/{provider-id}/sources, provide the following variables in the body:
- Link the datatypes to the source
- Retrieve source-id using GET /providers/sources
- Use POST /providers/{provider-id}/sources/{source-id}/data-types,
- Provide the following variables to the parameters:
- provider-id
- source-id
- provide the following variables in the body
- the datatypes you wish to add to this source
- Provide the following variables to the parameters:
- GET and DELETE are also available for datatypes
- Link the locations to the source
- Use POST/providers/{provider-id}/sources/{source-id}/location
- Provide the following variables to the parameters:
- provider-id
- source-id
- provide the following variables in the body:
- scheme is the location scheme, only ‘nl-v1’ is supported at the moment
- id is the location id (UBN) for the source
- Provide the following variables to the parameters:
- GET and DELETE are also available for locations
- Use POST/providers/{provider-id}/sources/{source-id}/location