In order to retrieve meteorological data from The Netherlands, we want to use the Open Data API developed by the Dutch Meteorological Institute (KNMI). This website contains all relevant information on using the API.
In summary, these are the steps that we want to undertake. Anyone can do this, because of the anonymous API key that is open to public:
Locate the ‘Anonymous API key’, can be found on the KNMI documentation website
Determine the dataset name, version and filename to establish the correct endpoint
Execute the GET call (make sure to use Authorization header in combination with the anonymous API key
The GET call returns a download URL pointing to the KNMI Amazone AWS storage which can be easily used in Postman to retrieve the data (in x-netcdf format)
Would you like to try the above in Postman as well and help us retrieve the data through RSD-configuration in TimeXtender? The KNMI does provide Python examples for downloading the data, but it would be nice to do this directly with TimeXtender (i.e. RSD-files).
Page 1 / 1
Anyone who can help me with this? Currently the last challenge is to get CData/TimeXtender actually read the x-netcdf format.
Hi @KajEmergo
I tried to connect with postman and then use the URL given.
That did not really work, I mean sure it gave a result, but postman could not read it.
If you know of a provider that can read it, you can generate a REST provider that gives back this URI
Then you can run the URI as a input. I have done this before where the authentication was done with JSON, but once you tried to run the URI that were given it returned CSV data.
@Thomas Lind, basically my question remains if it is possible to extract data in this so-called netCDF format. It seems like a format commonly used in meteorology, but I cannot find anything related to it on the CData website or CSV connector documentation.
It is probably possible with Python or C++, but it would be nice to do it with our holistic TimeXtender solution of course :)
Then I executed the transfer task I get the following data
When I try to use this temporaryDownloadUrl in Postman I get the following response:
Do you get a different result in Postman? If so which settings/params do you use in Postman?
@bitmetric_barry, thanks for your reply! We are currently discussing with the business if they really need the data from the KNMI API that we were trying to load into TX. It is possible that we will look for alternatives shortly (might be your option as well), because of the netCDF format.
@Christian Hauggaard, that is exactly the point at which we are stuck now. That format is not readable and needs to be parsed, but that will probably require some custom coding. Like I mentioned, we are currently looking for alternative data sources.
@KajEmergo I reached out to Cdata and unfortunately they do not have a driver that supports the NC file format, please see their response below:
“I regret to inform you that our current suite of CData drivers does not include specific provisions for handling the netCDF (.nc) file format. As our drivers are predominantly designed for more commonly employed file types, the netCDF format falls outside our current scope.”
@Christian Hauggaard, thanks for clarifying that. That means we will be looking for other type of data sources like @bitmetric_barry mentioned or we will have to write some custom code in order to parse the data in netCDF format.
