To homepage To homepage
Solved

Using open source API with download link

  • 13 November 2023
  • 8 replies
  • 127 views
K
Rank
Userlevel 2
Badge +1

In order to retrieve meteorological data from The Netherlands, we want to use the Open Data API developed by the Dutch Meteorological Institute (KNMI). This website contains all relevant information on using the API.

In summary, these are the steps that we want to undertake. Anyone can do this, because of the anonymous API key that is open to public:

  1. Locate the ‘Anonymous API key’, can be found on the KNMI documentation website
  2. Determine the dataset name, version and filename to establish the correct endpoint
  3. In this example we want to use the following endpoint: https://api.dataplatform.knmi.nl/open-data/v1/datasets/etmaalgegevensKNMIstations/versions/1/files/KIS___OPER_P___OBS_____L2.nc/url
  4. Execute the GET call (make sure to use Authorization header in combination with the anonymous API key
  5. The GET call returns a download URL pointing to the KNMI Amazone AWS storage which can be easily used in Postman to retrieve the data (in x-netcdf format)

Would you like to try the above in Postman as well and help us retrieve the data through RSD-configuration in TimeXtender? The KNMI does provide Python examples for downloading the data, but it would be nice to do this directly with TimeXtender (i.e. RSD-files).

icon

Best answer by Christian Hauggaard 23 November 2023, 11:52

View original

8 replies

K
Rank
Userlevel 2
Badge +1

Anyone who can help me with this? Currently the last challenge is to get CData/TimeXtender actually read the x-netcdf format.

Thomas Lind
Rank
Userlevel 6
Badge +5

Hi @KajEmergo 

I tried to connect with postman and then use the URL given.

That did not really work, I mean sure it gave a result, but postman could not read it.

If you know of a provider that can read it, you can generate a REST provider that gives back this URI

Then you can run the URI as a input. I have done this before where the authentication was done with JSON, but once you tried to run the URI that were given it returned CSV data.

K
Rank
Userlevel 2
Badge +1

@Thomas Lind, basically my question remains if it is possible to extract data in this so-called netCDF format. It seems like a format commonly used in meteorology, but I cannot find anything related to it on the CData website or CSV connector documentation.

It is probably possible with Python or C++, but it would be nice to do it with our holistic TimeXtender solution of course :)

B
Rank

Not sure if it'll be any easier than the API you're currently using, but if you're looking for KNMI data, you could also try this ‘API’: https://www.daggegevens.knmi.nl/klimatologie/daggegevens

It returns 3 CSVs, stacked in a single text file:

  • Weather stations
  • Definitions of the measurements
  • The actual measurements

I've been using this for 10+ years and it's always been solid.

Christian Hauggaard
Rank
Userlevel 6
Badge +5

@KajEmergo 

I used the following connection string for the REST provider

Auth Scheme=OAuth;OAuth Access Token=eyJvcmciOiI1ZTU1NGUxOTI3NGE5NjAwMDEyYTNlYjEiLCJpZCI6ImE1OGI5NGZmMDY5NDRhZDNhZjFkMDBmNDBmNTQyNjBkIiwiaCI6Im11cm11cjEyOCJ9;URI=" https://api.dataplatform.knmi.nl/open-data/v1/datasets/etmaalgegevensKNMIstations/versions/1/files/KIS___OPER_P___OBS_____L2.nc/url"

Then I executed the transfer task I get the following data

When I try to use this temporaryDownloadUrl in Postman I get the following response:

 

Do you get a different result in Postman? If so which settings/params do you use in Postman?

K
Rank
Userlevel 2
Badge +1

@bitmetric_barry, thanks for your reply! We are currently discussing with the business if they really need the data from the KNMI API that we were trying to load into TX. It is possible that we will look for alternatives shortly (might be your option as well), because of the netCDF format.

@Christian Hauggaard, that is exactly the point at which we are stuck now. That format is not readable and needs to be parsed, but that will probably require some custom coding. Like I mentioned, we are currently looking for alternative data sources. 

Christian Hauggaard
Rank
Userlevel 6
Badge +5

@KajEmergo I reached out to Cdata and unfortunately they do not have a driver that supports the NC file format, please see their response below:

“I regret to inform you that our current suite of CData drivers does not include specific provisions for handling the netCDF (.nc) file format. As our drivers are predominantly designed for more commonly employed file types, the netCDF format falls outside our current scope.”

K
Rank
Userlevel 2
Badge +1

@Christian Hauggaard, thanks for clarifying that. That means we will be looking for other type of data sources like @bitmetric_barry mentioned or we will have to write some custom code in order to parse the data in netCDF format.

Reply


Terms and ConditionsCookie settings