We are currently utilizing the new TimeXtender REST API 6.1 and leveraging its built-in paging feature to connect to an external REST API. We're not using an RSD configuration for this setup.
I've encountered a challenge with pagination, specifically with the use of a parameter named "Cursor." This parameter is critical for the pagination logic in the API we're connecting to. Each row in the data returned by the API has a unique Cursor value, but for pagination purposes, only the first Cursor value from the current page should be used in the subsequent POST request to fetch the next page.
The issue arises because I need to ensure that only the first Cursor value from each fetched page is used in the pagination post body for the next request. I'm unsure how to configure TimeXtender to utilize only the first Cursor value for this purpose.
Has anyone faced a similar issue or does anyone have insights on how to configure the pagination correctly in such scenarios?
Thank you!
Page 1 / 1
Hi @sigsol
You point at a field in the returned result that have a cursor value. This value is applied as some sort of URL parameter, right?
Will the cursor value exist in the same field on the second “page” that is returned?
Will there be a cursor value even if the “page” contains no actual data?
Hi @Thomas Lind
Cursor is used to specify the parameter “before” in the post body.
The cursor values exist on all rows on all pages, yes.
The paging will have stopped before we run out of data, see explanation below.
We are actually using two fields for pagination that are returned on every page and has a value on all rows:
hasPreviousPage*: This has the same value on all rows in the same page, and is True only if there are more pages before the current one. This can be used in the SQL Expression part in the REST connector to check if we need to get more pages, as I understand it
cursor: This has a unique values on all rows (across all pages also). It is used in a parameter, before, that we set in the post body.
* =For some technical reasons we need to start with the last page and go forward, but the same logic could be applied the other way.
In the first call, we set “before: null” in the post body.
In the next call, we have to set before to the first value of cursor returned in the first call (eg. before: 1234). This will give us the second-to-last page. And the same for all subsequent calls, ie. before is set to the first value of cursor in the most recent page.
I’m not sure how to use the cursor parameter defined in the REST API in such a way that we pick the first value returned on the current page.
By the way, there are other parameters in the post body as well such as “last: 10000”, that sets the number of rows on the page to 10000. The other parameters are for filtering the data and are not relevant for the paging.
Hi @sigsol
The first call is always done without any pagination. That is why I suggest that you set the page size parameter on the endpoint and not in the pagination setup.
Can you confirm if to get the next page, you do not do a Get call, you need to make a Post call with the cursor value instead.
You don’t have an option of doing post calls in the pagination section, so if the above is the case you can’t really use this to get the next page.
You don’t have an option of doing post calls in the pagination section
If we set Parameter action to “Replace post body” a second post body input field appears called Pagination post body (2):
The plan is to have “before: null” in Post body (1), and “before: {cursor}” in post body (2). This will solve everything I believe. But I’m not sure which cursor value {cursor} will be replaced by when it is unique on all rows on the current page, and how we specify that we want the first cursor value.
Can you confirm if to get the next page, you do not do a Get call, you need to make a Post call with the cursor value instead.
Exactly, every call is a post call.
I see.
I am not sure you could do different post bodies in one and have it be about pagination.
If you only do the part about the cursor and put it in the pagination post body. Does it iterate over this value and returns the values for the next cursor?
If you can make another endpoint that gets what is missing with the before null part then you can merge the two parts in the DW.
Seems to be working now! Only had to add “1]” to the XPath of Cursor to specify the first value:
The post body now has “before:null” and the pagination post body has “before:{cursor}”. The pagination post body is called repeatedly until hasPreviousPage is ‘false’ (condition spesified in SQL condition using the hasPreviousPage parameter).
We also use a flattening XSLT. The XPath must refer to the structure before flattening, however.
Yes it would be before applying that.
The [1] means that it is the first available child value. As far as what I could find about it in some XPath documentation.
Good you found a way.
It would be nice if you could share some images of the setup, with private info hidden of course.
