Here is the advanced options that you can use an RSD file to achieve.
Click here to go back to the main RSD article Generate and use RSD files
Content
Add pagination to the files
Pagination is a method where you iterate through a number of pages until you hit the end and get all the data out of the API for a specific resource.
There are two ways of doing this and it depends on what your API supports.
One requirement is to add the following
<api:set attr="EnablePaging" value="true"/>Add this after the <info> area but before the get calls.
Using this will make the paging work. So it should be added to all RSD files if you want to do multiple calls. The only time you do not need it is when you use the dummy fields as explained in the Using nested calls to get info from multiple pages section.
Record Offset
If the service provides a record offset query parameter to control paging, you can implement this by setting the name of the offset query parameter, the name of the page size query parameter, and the page size to be passed. Note that the page size parameter does not need to be set if there is no parameter to control this. In this case, you must set pagesize to the default page size.
<api:set attr="pageoffsetparam" value="offset" />
<api:set attr="pagesizeparam" value="limit" />
<api:set attr="pagesize" value="100" />The offset and size parameter options may have different names, but these should be mentioned in the API documentation.
Page Number
Similar to record offset, if the service provides a query parameter that sets the page number, you can implement this by setting the name of the page number query parameter, the name of the page size query parameter, and the page size to be passed. Note that the page size parameter does not need to be set if there is no parameter to control this. In this case, you must set pagesize to the default page size.
<api:set attr="pagenumberparam" value="page" />
<api:set attr="pagesizeparam" value="pagesize" />
<api:set attr="pagesize" value="100" />This is a new method, so it can possibly work for most of the options regarding page that have a page and pagesize or pagecount option in the API. So this may be used instead of the IF and Enum options explained below.
Push response header link method
If your API has the following setup. The proposed standard RFC-8288 is to include information about the current page in the HTTP header.
In more detail, the following information is included in the Header:
Per-Page: the number of results shown per pagePage: the currently displayed pageLink: a set of RFC-8288 encoded links to the first, next, previous, and last pages (if applicable).Total: the total number of results
Page
The following method is for the PushResponseHeader Page method.
- You push header names onto the end of the PushResponseHeader# input array, which will mark them to be saved.
- Set any other inputs and invoke jsonproviderGet / xmlproviderGet
- Read the result from the header: set of output attributes, which are named based on the inputs given to the push header array. For example, if you set the push headers so it contains the values vPer-Page, Page] then the header values will be set on the header:Per-Page and header:Page attributes of the output item.
<api:set attr="PushResponseHeader#" value="Page" />
<api:set attr="output.rows@next" value="voutput.header:Page]" />The previous also can be used to get the last page, then you can make a if statement where you will set the next page until you reach the last.
It could look something like this.
Remember that the GoREST provider doesn't use this, so you can't make this exact thing work.
Link
Another common occurrence is that you have an Link header in your metadata. Like what is common with OKTA, looking like what you can see below.
Link: <https://{yourOktaDomain}.com/api/v1/users?after=00ubfjQEMYBLRUWIEDKK>; rel="next", <https://{yourOktaDomain}.com/api/v1/users?after=00ub4tTFYKXCCZJSGFKM>; rel="self"It actually is possible to use this to do pagination as well. It is a very simple setup as well.
All I do is to add that it is to run with Paging, have an rows@next input option and have an pageurlpath attribute that is set to be of value Header:Link. The rest is controlled by the file, no need to add a if statement or enum loop.
Remember that the GoREST provider doesn't use this, so you can't make this exact thing work.
Next Page URL
If the next page URL is passed in the response body, set 'pageurlpath' to the XPath of the element.
<api:set attr="pageurlpath" value="/data/nextPage" />This makes it real simple to make it work. One page that uses this method is swapi.dev
So here is how you would make pagination work for the people page.
I just added a repeat element and set enablepaging = true and added a pageurlpath that corresponds to the next page. I have added it and another method to achieve the same thing as people.rsd and People_RWNE.rsd
If and page method
To make the GoREST work with pagination you can set up an if statement that takes the last page and the current page and uses them to iterate through.
If we takes a look at one of them, like users for example.
You can see that it has some pagination info on the top of the json data. What we want are the pages value and the page value to be added as variables in our code. To do so we can use the element mapping feature.
<api:set attr="ElementMapPath#" value="xpath to field" />
<api:set attr="ElementMapPath#" value="name" />So to convert this to what we want to use from the GoREST page, we get this.
<api:set attr="ElementMapPath#" value="/json/meta/pagination/pages" />
<api:set attr="ElementMapName#" value="pages" />
<api:set attr="ElementMapPath#" value="/json/meta/pagination/page" />
<api:set attr="ElementMapName#" value="page" />Besides this we also want to tell that the file is using pagination, so we will add the following
<api:set attr="EnablePaging" value="TRUE" />We will also add a input field, that will contain the current page you are attempting to add get data from.
<input name="rows@next" default="1" />We set up the URI with the updated page info, but this is something we do in the script get method area.
<api:set attr="URI" value="vBaseURI]?page=]_input.rows@next]" />Finally we set up the if statement and add the info to the output variable that we finally push to get the data back. Note that I also use the function Add() to add a number to the current page and get the next one.
<api:call op="jsonproviderGet">
<api:if attr="page" value="vpages]" operator="lessthan">
<api:set attr="rows@next" value="vpage | Add(1)]" />
</api:if>
<api:push/>
</api:call>Finally we have the whole document done. You can see how it should look here.
After doing the changes it is not necessary to deploy or synchronize anything, just executing the table/task is enough.
If you want to see what pages it iterates through, add a log to your setup, like explained in the following guide.
Setting the verbosity to more than 3 will then add actual data from the source, so do only set it that high if you remember to truncate the log before trying something. This is the best way to see, what happens with calls and so when it doesn't work.
Finally when executed it will contain this amount of data.
Enum and page method
This method is somewhat different, in that it uses an enum loop to go through the pages.
Normally I would not use it for a single page get as this, but when you use a nested get calls. More of that in the next paragraph.
The major change to this is that we now use input and output of our calls, which makes it possible to use info from a previous call to the next. Additionally it is an requirement that all the fields that are used as part of all the fields you set.
<api:set attr="in.Header:Name#" value="Accept" />
<api:set attr="in.Header:Value#" value="application/json" />
<api:set attr="in.DataModel" value="DOCUMENT" />
<api:set attr="in.JSONPath" value="$.data" />
<api:set attr="in.EnablePaging" value="TRUE" />
<api:set attr="in.ParamName#" value="access-token" />
<api:set attr="in.ParamValue#" value="48ed584d0b694e0db651f7fb1e6846a9f3ce8c4c9889f5b5b5af1c5194d4e02e" />
<api:set attr="in.Page" value="1" />
<api:set attr="in.PageCount" value="4" />
<api:set attr="in.BaseURI" value="https://gorest.co.in/public-api/users?page={page}" />
<api:set attr="in.ElementMapPath#" value="/json/meta/pagination/pages" />
<api:set attr="in.ElementMapName#" value="pages" />If any of the fields are not set up with in. in front of it it will not work.
The next step is in the get method where I set the URI to contain the base URI and replacing the {page} with a 1 number.
<api:set attr="in.URI" value="vin.BaseURI | replace('{page}', 1)]" />The next step is an initial call command that uses the previously generated URI. notice that I use in="in" and out="out" in the call command. This is then used to set the output value of pages to be contained in the PageCount.
<api:call op="jsonproviderGet" in="in" out="out">
<api:set attr="in.PageCount" value="vout.pages]" />
</api:call>The reason for this is to get the maximum number of pages from the metadata and then use it as the endpoint of the enum range. In the enum itself I first set the current step in the range _value and sets it to the Page. Then I use that to replace {page} with the current range number and in the call i push the output with the item="out" command.
<api:enum range="1..gin.PageCount]">
<api:set attr="in.Page" value="v_value]" />
<api:set attr="in.URI" value="vin.BaseURI | replace('{page}', ain.Page])]" />
<api:call op="jsonproviderGet" in="in" out="out">
<api:push item="out" />
</api:call>
</api:enum>I can avoid using PageCount and Page in this, by using in.pages and _value in the range and URI setting areas, but since I set it with default values in the initial set, they will make the calls work despite the values being empty in the source. This is a good safeguard, especially when you use nested calls.
In the end the file will look like this.
Using nested calls to get data from multiple pages
The usual issue for when you use this is the following. You have some users, which you can get, but you also want to get the posts of those users. So something like this.
https://gorest.co.in/public-api/users/{userid}/posts
That only gave the info from one customer, but you want to get all users and all their posts in one table. This is because there is no way to just get all posts with a relation to the user or similar. This can even be more extreme, where you also wants to get the comments on the posts as well. More on that later, but first the posts of the users.
First of you will need to add the following command even if you are not doing pagination.
<api:set attr="stopin.EnablePaging" value="TRUE" />I have this rest call I need to do to get a list of stops on a transport. There is no need for pagination it will give all of the stops in one call. The issue is that to get a stop I need to specify a file id. I have a rest call that can give me all these ids. So I just need to mix them in a nested call to make it iterate over each file id.
First I need to define all the fields.
<api:info title="GetAllStops" desc="Generated schema file." xmlns:other="http://apiscript.com/ns?v1">
<!-- You can modify the name, type, and column size here. -->
<attr name="country" xs:type="string" readonly="false" other:xPath="/json/result/entity/stops/country" />
<attr name="id" xs:type="string" readonly="false" other:xPath="/json/result/entity/stops/id" />
<attr name="lastModificationTime" xs:type="datetime" readonly="false" other:xPath="/json/result/entity/stops/lastModificationTime" />
<attr name="lastModifierUserId" xs:type="integer" readonly="false" other:xPath="/json/result/entity/stops/lastModifierUserId" />
<attr name="mobileNumber" xs:type="string" readonly="false" other:xPath="/json/result/entity/stops/mobileNumber" />
<attr name="orderNumber" xs:type="integer" readonly="false" other:xPath="/json/result/entity/stops/orderNumber" />
<attr name="products" xs:type="string" readonly="false" other:xPath="/json/result/entity/stops/products" />
<attr name="telephoneNumber" xs:type="decimal" readonly="false" other:xPath="/json/result/entity/stops/telephoneNumber" />
<attr name="tenantId" xs:type="integer" readonly="false" other:xPath="/json/result/entity/stops/tenantId" />
</api:info>Notice the XPath which points to the fields values. These fields are all from the same call.
First part is this.
<api:set attr="urlbase" value="https://webpage.com/api/services/app/Transport" />I have an URLBase, which is the part of the URL both calls will share I use this in the URITemplate to specify the URI for the call to get the file ids.
Then I set up the necessary parts for the call to get the list of file ids.
<api:set attr="filein.DataModel" value="DOCUMENT" />
<api:set attr="filein.JSONPath" value="$.result.items" />
<api:set attr="filein.URITemplate" value="vurlbase]/GetAll"/>
<api:set attr="filein.ElementMapPath#" value="/json/result/items/id" />
<api:set attr="filein.ElementMapName#" value="file_id" />I state the data model type and the JSON path. Notice that it is not the same as in the fields of the table. I use the ElementMap option to map the file_id field from the call. Also all of these are set to be in the filein item variable.
The third step is to set up the getallstops URI call.
<api:set attr="stopin.DataModel" value="DOCUMENT" />
<api:set attr="stopin.EnablePaging" value="TRUE" />
<api:set attr="stopin.JSONPath" value="$.result.entity.stops" />
<api:set attr="stopin.URITemplate" value="vurlbase]/GetAllStops/{file_id}" />It is essentially the same as before, the URITemplate is different and is containing {file_id} which is the place holder for where the real file id will be added, the enablepaging part is set here as to avoid having to individually map all the fields. The JSONPath is the same as the one in the info area. This area is set to be in the stopin item variable instead.
Then I set up my first call.
<api:script method="GET">
<api:set attr="filein.URI" value="vfilein.URITemplate]" />
<api:call op="jsonproviderGet" in="filein" out="fileout">
...
</api:call>
</api:script>I make the filenin version of the URITemplate equal to filein.URI and then in the call I have set the input to filein and output to fileout.
When I have done my call I will get the file ids to use those in the next call I will have to set the value and add it to the URI.
<api:set attr="filein.file_id" value="vfileout.file_id]" />
<api:set attr="stopin.URI" value="vstopin.URITemplate | replace('{file_id}', _filein.file_id])]" />You can see that i set the fileout file id to be filein.file_id and then I use that value in the stopin URI where I also add a Replace() function to swap the {file_id} with the real file id.
Once this is done I run the second call inside the first one.
<api:call op="jsonproviderGet" in="stopin" out="stopout">
<api:set attr="out.country" value="vstopout.country | allownull()]"/>
<api:set attr="out.id" value="vstopout.id | allownull()]"/>
<api:set attr="out.lastModificationTime" value="vstopout.lastModificationTime | allownull()]"/>
<api:set attr="out.lastModifierUserId" value="vstopout.lastModifierUserId | allownull()]"/>
<api:set attr="out.mobileNumber" value="vstopout.mobileNumber | allownull()]"/>
<api:set attr="out.orderNumber" value="vstopout.orderNumber | allownull()]"/>
<api:set attr="out.products" value="vstopout.products | allownull()]"/>
<api:set attr="out.telephoneNumber" value="vstopout.telephoneNumber | allownull()]"/>
<api:set attr="out.tenantId" value="vstopout.tenantId | allownull()]"/>
<api:push item="out"/>
</api:call>I set the stopin as the input and the stopout as the output. Then I map each field individually under the out item variable. and then I push the out variable. Normally you would push the stopout item variable, but doing this resets the mapping and without this you can experience that it will put the field values in the wrong places. This is also why I have a mapping of each individual field, this part can be skipped and can be seen as an optional bonus.
Here below is the whole file with all the areas added. It is also an attached document as GetAllStops.RSD.
<api:script xmlns:api="http://apiscript.com/ns?v1" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<!-- See Column Definitions to specify column behavior and use XPaths to extract column values from JSON. -->
<api:info title="GetAllStops" desc="Generated schema file." xmlns:other="http://apiscript.com/ns?v1">
<!-- You can modify the name, type, and column size here. -->
<attr name="country" xs:type="string" readonly="false" other:xPath="/json/result/entity/stops/country" />
<attr name="id" xs:type="string" readonly="false" other:xPath="/json/result/entity/stops/id" />
<attr name="lastModificationTime" xs:type="datetime" readonly="false" other:xPath="/json/result/entity/stops/lastModificationTime" />
<attr name="lastModifierUserId" xs:type="integer" readonly="false" other:xPath="/json/result/entity/stops/lastModifierUserId" />
<attr name="mobileNumber" xs:type="string" readonly="false" other:xPath="/json/result/entity/stops/mobileNumber" />
<attr name="orderNumber" xs:type="integer" readonly="false" other:xPath="/json/result/entity/stops/orderNumber" />
<attr name="products" xs:type="string" readonly="false" other:xPath="/json/result/entity/stops/products" />
<attr name="telephoneNumber" xs:type="decimal" readonly="false" other:xPath="/json/result/entity/stops/telephoneNumber" />
<attr name="tenantId" xs:type="integer" readonly="false" other:xPath="/json/result/entity/stops/tenantId" />
</api:info>
<api:set attr="urlbase" value="https://webpage.com/api/services/app/Transport" />
<api:set attr="filein.DataModel" value="DOCUMENT" />
<api:set attr="filein.JSONPath" value="$.result.items" />
<api:set attr="filein.URITemplate" value="vurlbase]/GetAll" />
<api:set attr="filein.ElementMapPath#" value="/json/result/items/id" />
<api:set attr="filein.ElementMapName#" value="file_id" />
<api:set attr="stopin.DataModel" value="DOCUMENT" />
<api:set attr="stopin.EnablePaging" value="TRUE" />
<api:set attr="stopin.JSONPath" value="$.result.entity.stops" />
<api:set attr="stopin.URITemplate" value="vurlbase]/GetAllStops/{file_id}" />
<!-- The GET method corresponds to SELECT. Here you can override the default processing of the SELECT statement. The results of processing are pushed to the schema's output. See SELECT Execution for more information. -->
<api:script method="GET">
<api:set attr="filein.URI" value="vfilein.URITemplate]" />
<api:call op="jsonproviderGet" in="filein" out="fileout">
<api:set attr="filein.file_id" value="vfileout.file_id]" />
<api:set attr="stopin.URI" value="vstopin.URITemplate | replace('{file_id}', _filein.file_id])]" />
<api:call op="jsonproviderGet" in="stopin" out="stopout">
<api:set attr="out.country" value="vstopout.country | allownull()]" />
<api:set attr="out.id" value="vstopout.id | allownull()]" />
<api:set attr="out.lastModificationTime" value="vstopout.lastModificationTime | allownull()]" />
<api:set attr="out.lastModifierUserId" value="vstopout.lastModifierUserId | allownull()]" />
<api:set attr="out.mobileNumber" value="vstopout.mobileNumber | allownull()]" /> <api:set attr="out.orderNumber" value="vstopout.orderNumber | allownull()]" />
<api:set attr="out.products" value="vstopout.products | allownull()]" />
<api:set attr="out.telephoneNumber" value="vstopout.telephoneNumber | allownull()]" />
<api:set attr="out.tenantId" value="vstopout.tenantId | allownull()]" />
<api:push item="out"/>
</api:call>
</api:call>
</api:script>
</api:script>Use parameters
You can also use an parameter to make it work and it can be that you want the parameter to be something dynamic, or something you apply in TimeXtender Data Integration instead of in the RSD file.
Using fixer.io we can pull out data, but if we want to get data from a specific day the URI would look something like this.
http://data.fixer.io/api/2020-03-15?access_key=522b683569a91b66be6be214027b47bf
It doesn't work without a date added after the api/. I want it to show current date or similar. I generate an RSD file for the source and adds two changes. Here is the RSD script.
<api:script xmlns:api="http://apiscript.com/ns?v1" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<!-- See Column Definitions to specify column behavior and use XPaths to extract column values from JSON. -->
<api:info title="Currencies" desc="Generated schema file." xmlns:other="http://apiscript.com/ns?v1">
<!-- You can modify the name, type, and column size here. -->
<attr name="base_currency" xs:type="string" other:xPath="base"/>
<attr name="date" xs:type="string" other:xPath="date"/>
<attr name="historical" xs:type="boolean" other:xPath="historical" />
<attr name="exchange_rate_to_USD" xs:type="string" other:xPath="rates/USD"/>
<attr name="exchange_rate_to_GBP" xs:type="string" other:xPath="rates/GBP"/>
<attr name="exchange_rate_to_AUD" xs:type="string" other:xPath="rates/AUD"/>
<attr name="timestamp" xs:type="integer" other:xPath="timestamp"/>
<input name="datein" xs:type="date" default="2019-01-01"/>
</api:info>
<api:set attr="URI" value="http://data.fixer.io/api/i_input.datein]?access_key=522b683569a91b66be6be214027b47bf" />
<api:set attr="RepeatElement" value="/"/>
<!-- The GET method corresponds to SELECT. Here you can override the default processing of the SELECT statement. The results of processing are pushed to the schema's output. See SELECT Execution for more information. -->
<api:script method="GET">
<api:call op="jsonproviderGet">
<api:push/>
</api:call>
</api:script>
</api:script>The two changes are <input> and <api:check>
The input sets the datein equal an specific date, or the default value. The default is necessary, or the file will fail.
To use this file you will have to make a managed query table in your REST data source.
You can use it in a query like so.
If the rest provider only works if you provide a filter like from date and to date you can also add custom URL parameters and use them the same way. Such as this.
<input name="from" xs:type="string" default="2019-01-01"/>
<input name="to" xs:type="string" default="2020-01-01"/>
<api:set attr="CustomUrlParams" value="FromDate=F_input.from]&ToDate=p_input.to]" />
<api:script method="GET">
<api:check attr="_input.from">
<api:set attr="URI" value="https://<rest url>/api/Collection"/>
</api:check>
<api:call op="jsonproviderGet">
<api:push/>
</api:call>
</api:script>And custom URL parameters can also be mixed with pagination as well.
Parameters and Query Slicers
This is an method that works if it is only an single level you go down, but can't do more than one.
This can be used when you want to do a nested call and mix it with the two pagination options Record Offset and Page Number. The requirement is an alternate RSD file that generates a list of id values that can be used as a parameter.
Below is how to set up the invoice id field as an input field.
<api:script xmlns:api="http://apiscript.com/ns?v1" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<!-- See Column Definitions to specify column behavior and use XPaths to extract column values from JSON. -->
<api:info title="invoiceDetails" desc="Generated schema file." xmlns:other="http://apiscript.com/ns?v1" other:queryslicercolumn="id">
<!-- You can modify the name, type, and column size here. -->
<attr name="id" xs:type="integer" readonly="false" other:xPath="/json/id" />
<attr name="date" xs:type="date" readonly="false" other:xPath="/json/date" />
<attr name="discount" xs:type="double" readonly="false" other:xPath="/json/discountPercent" />
<attr name="invoiceId" xs:type="integer" readonly="false" other:filter="{id}" other:xPath="/json/factoringContractId" />
<attr name="invoiceChannel" xs:type="string" readonly="false" other:xPath="/json/invoiceChannel" />
<attr name="invoiceNumber" xs:type="integer" readonly="false" other:xPath="/json/invoiceNumber" />
<attr name="invoiceTemplateId" xs:type="integer" readonly="false" other:xPath="/json/invoiceTemplateId" />
<attr name="language" xs:type="string" readonly="false" other:xPath="/json/language" />
<attr name="status" xs:type="string" readonly="false" other:xPath="/json/status" />
<attr name="type" xs:type="string" readonly="false" other:xPath="/json/type" />
<attr name="version" xs:type="datetime" readonly="false" other:xPath="/json/version" />
</api:info>
<api:set attr="EnablePaging" value="true" />
<api:set attr="DataModel" value="DOCUMENT" />
<api:set attr="URI" value="https://api.rest.com/api/invoices/{id}/details" />
<api:match pattern="*id*" type="glob" value=""_query.criteria]">
<api:else>
<api:set attr="URI" value="https://api.rest.com/api/invoices/1234567/details" />
</api:else>
</api:match>
<api:set attr="JSONPath" value="$." />
<!-- The GET method corresponds to SELECT. Here you can override the default processing of the SELECT statement. The results of processing are pushed to the schema's output. See SELECT Execution for more information. -->
<api:script method="GET">
<api:set attr="method" value="GET"/>
<api:call op="jsonproviderGet">
<api:push/>
</api:call>
</api:script>
</api:script>We add an match method to avoid issues if a id does not have any data. The difference is that it is utilized in the URI value in an {} area, not as an _input.postid] and that it is has set an other:filter="{input name}".
To see the other values than the default URI I added to the match method, you will have to add a managed query to add the rule to.
SELECT "id],
WHERE invoiceId] IN (SELECT id] FROM iREST].MInvoices])This will make the Invoices table iterate over each invoiceId that gets added to the IN statement.
You can see how this happens if you add a log to the setup. It will show the query in that.
Create an dynamic date range filter
Here is how you set up a dynamic date range.
Is the 'date' value in your script output from the first call to jsonproviderGet? Or are you trying to just get today's date? If this an output from the first call, you could just use the following value to format the returned date to the appropriate format:
<api:set attr="from" value=""date | todate('yyyy-MM-dd')]"/>If you just want to output today's date, you can use a dummy attribute to call the 'date' formatter like so:
<api:set attr="from" value=""dummy | date('yyyy-MM-dd')]"/>For values like 'yesterday', you can make use to the dateadd formatter to add or subtract days and then call todate() to format the output:
<api:set attr="from" value=""dummy | date() | dateadd('day', '-1') | todate('yyyy-MM-dd')]"/>So with this you can make it have a range from first of month, by changing day to month and you can do the same for to. If you want it to look way into the future change it yo year instead.
How to use it in an RSD file I have an example below.
I have an URI that can have an start and end date set to filter the data and I set it up like so.
<api:set attr="URITemplate" value="https://server.name.plus/test/api/node/12345/realized-hours/{from}/{to}" />The URI I use is the nested part and I have added an {from} and {to} parameter to it.
To set this up in the call I do the following.
<api:script method="GET">
<api:set attr="method" value="GET"/>
<api:set attr="from" value=""dummy | date() | dateadd('day', '-1') | todate('yyyy-MM-dd')]"/>
<api:set attr="to" value=""dummy | date() | dateadd('day', '+1') | todate('yyyy-MM-dd')]"/>
<api:set attr="URI" value=""URITemplate | replace('{from}', {nodeout.from]) | replace('{to}', (nodeout.to])]"/>
<api:call op="jsonproviderGet">
<api:push/>
</api:call>
</api:script>So I set from to be today -1 and to be today +1 and this will give me only the data from the last two days.
This can then be used in conjunction with incremental load to decrease the number of rows the program is going through. The only necessity is that the API has the option to filter by date. Additionally, there is some more info about this in the link added to the first section.
