I have a TimeXtender REST connection to SalesForce. This specific REST API is a file api. This means that you have to send a request (in the form of a query) to an endpoint, after which the API starts to generate an export. When the export is finished, it can be downloaded via a separate endpoint as CSV.
To extract this data with TX REST, I have set up a method with three endpoints per ‘endpoint’, this works as following:
- The first endpoint called _query sends a query via a POST request to a specific endpoint
- The second endpoint called _waitForComplete uses a dynamic value from the _query endpoint so that the _query endpoint gets executed first. The _waitForComplete endpoint uses pagination with a 1-minute delay to send requests to an endpoint that returns the status of the generation of the CSV file. Once the generation is complete, the _waitForComplete endpoint will return a JobComplete response which will stop the pagination.
- The third endpoint uses a dynamic value from the _waitForComplete endpoint so the _waitForComplete (and before that the _query endpoint) gets executed first. The third endpoint downloads the generated CSV file and returns this as a table.
This whole process works fine, but I am facing one problem. Because the _waitForComplete endpoint uses pagination with a delay to check wether the CSV file is done being generated, it has a non-predictable amount of pages. The third endpoint however, uses a dynamic value from the _waitForComplete endpoint. For some endpoints, generating the CSV can take up to 15 minutes which means it will go trough 15 pages and return 15 dynamic values. This leads to the actual CSV endpoint downloading the CSV 15 times, one for each dynamic value.
We could of course work around all this by setting a very high delay in the _waitForComplete endpoint (i.e. 15 or 20 minutes) but that slows down extraction a lot, and it is also not a guarantee to fix the problem because if an extraction takes longer than this setting for whatever reason, it still returns two or more dynamic values.
This boils down to my question: is it possible to have the _waitForComplete query return only one dynamic value, even if it runs several pages?
