Solved

Pagination and recursive query

  • 2 March 2023
  • 6 replies
  • 402 views

Hi,

I have problem with pagination and recursive query

I have rest api endpoint:

https://api.procountor.com/api/invoices

which returns invoice headers as
"id": 6591273,
"partnerId": 1208831,
"type": "PURCHASE_INVOICE",
"status": "PAID",
"invoiceNumber": 16
...etc.

pagination works fine, I get every invoice headers (thousands) when pagination is set like this in RSD-file:
<api:set attr="EnablePaging" value="true"/>
<api:set attr="pagenumberparam" value="page" />
<api:set attr="pagesizeparam" value="size" />
<api:set attr="pagesize" value="100" />

But the actual details of invoice comes from endpoint like this:
https://api.procountor.com/api/invoices/6591273

So, I have to iterate thru one by one every invoice id that I get from https://api.procountor.com/api/invoices 

I have followed instructions from here
https://legacysupport.timextender.com/hc/en-us/articles/360052383191-Creating-and-using-RSD-files-for-CData-providers#one-nested-call

and I can query invoices successfully, but the problem is that I get only 100 invoices back instead of thousands invoices.

Any idea what is wrong with this RSD-file?

 <api:script xmlns:api="http://apiscript.com/ns?v1" xmlns:xs="http://www.w3.org/2001/XMLSchema">

<!-- See Column Definitions to specify column behavior and use XPaths to extract column values from JSON. -->
<api:info title="invoices" desc="Generated schema file." xmlns:other="http://apiscript.com/ns?v1">
<!-- You can modify the name, type, and column size here. -->
<attr name="status" xs:type="string" readonly="false" other:xPath="/json/status" />
<attr name="date" xs:type="date" readonly="false" other:xPath="/json/date" />
<attr name="id" xs:type="integer" readonly="false" other:xPath="/json/id" />
<attr name="invoiceNumber" xs:type="integer" readonly="false" other:xPath="/json/invoiceNumber" />
</api:info>


<api:set attr="urlbase" value="https://api.procountor.com/api/invoices" />
<api:set attr="filein.DataModel" value="DOCUMENT" />
<api:set attr="filein.JSONPath" value="$.results" />
<api:set attr="filein.URITemplate" value="[urlbase]"/>
<api:set attr="filein.ElementMapPath#" value="/json/results/id" />
<api:set attr="filein.ElementMapName#" value="id" />

<api:set attr="filein.EnablePaging" value="true"/>
<api:set attr="filein.pagenumberparam" value="page" />
<api:set attr="filein.pagesizeparam" value="size" />
<api:set attr="filein.pagesize" value="100" />

<api:set attr="detailsin.DataModel" value="DOCUMENT" />
<api:set attr="detailsin.JSONFormat" value="LDJSON" />
<api:set attr="detailsin.EnablePaging" value="true"/>
<api:set attr="detailsin.JSONPath" value="$." />
<api:set attr="detailsin.URITemplate" value="[urlbase]/{id}"/>

<!-- The GET method corresponds to SELECT. Here you can override the default processing
of the SELECT statement. The results of processing are pushed to the schema's output.
See SELECT Execution for more information. -->

<api:script method="GET">

<api:set attr="filein.URI" value="[filein.URITemplate]"/>
<api:call op="jsonproviderGet" in="filein" out="fileout">
<api:set attr="filein.id" value="[fileout.id]" />
<api:set attr="detailsin.URI" value="[detailsin.URITemplate | replace('{id}', [filein.id])]"/>
<api:call op="jsonproviderGet" in="detailsin" out="detailsout">

<api:set attr="out.status" value="[detailsout.status | allownull()]" />
<api:set attr="out.date" value="[detailsout.date | allownull()]" />
<api:set attr="out.id" value="[detailsout.id | allownull()]" />
<api:set attr="out.invoiceNumber" value="[detailsout.invoiceNumber | allownull()]" />

<api:push item="out"/>
</api:call>
</api:call>

</api:script>
</api:script>

 

icon

Best answer by Thomas Lind 6 March 2023, 15:33

View original

6 replies

Userlevel 3
Badge +3

Hi Timon,

As a workaround, if you increase "pagesize" value, does it retrieve more than 100 invoices ?

Hi Timon,

As a workaround, if you increase "pagesize" value, does it retrieve more than 100 invoices ?

Hi Syed,

I can increase pagesize value to 200 which is maximum pagesize value and that works fine.

If I increase it more than 200 I get http 400 error .

 

-Timo 

Userlevel 6
Badge +5

Hi Timo

I have had similar requests in Zendesk. The issue seems to be mixing Nested Calls with the specific type of Pagination you use.

It was suggested by CData to do it in a different way.

Attempts to mix the built-in paging with nested api:calls will not work seamlessly. The driver will always start at the start of the <api:script method="GET"> element when entering the iteration of the next page, which will put it right outside of the outer api:call again.

Usually, cases like this one are best handled by leveraging the query slicer. That way, each endpoint can be handled by its own table, and the paging logic can be separated between the two tables in separate RSD files. With the two RSD files attached, a query like the should produce the desired results:

SELECT * FROM debtors WHERE administration_id IN (SELECT id FROM administrations)

If the customer absolutely needs the logic in one RSD file (i.e. so they can run a simple unfiltered SELECT *), then that will require manual logic using rows@next. In that case, please let me know and I can try to customize the RSD file accordingly.

 

Does it make sense?

Essentially you add a filter field and uses a query table to get the id applied. I have sent an file to you on the Zendesk ticket.

Userlevel 6
Badge +5

Hi @timon has the issue been resolved? if so can you please help select a best answer above? otherwise please let us know if we can provide further assistance 

Badge +1

Hi @Thomas Lind  In the response from CData it is mentioned that they’ve attached 2 example RSD files that should allow the SELECT statement in a Query Table. Can you post those here? I’m curious to see how RSD of the debtors call will address the administration_id. 

Userlevel 6
Badge +5

@rogier.helmus 

It is essentially what is shared in this.

<api:script xmlns:api="http://apiscript.com/ns?v1" xmlns:xs="http://www.w3.org/2001/XMLSchema">

<!-- See Column Definitions to specify column behavior and use XPaths to extract column values from JSON. -->
<api:info title="detailedInfo" desc="Generated schema file." xmlns:other="http://apiscript.com/ns?v1" other:queryslicercolumn="nested_id">
<attr name="ID" xs:type="string" readonly="false" other:xPath="/json/data/name" />
<!-- You can modify the name, type, and column size here. -->
<attr name="nested_id" xs:type="string" readonly="false" other:filter="{nested_id}" other:xPath="/json/data/nested_id" />
<attr name="Name" xs:type="string" readonly="false" other:xPath="/json/data/name" />
<attr name="Date" xs:type="datetime" readonly="false" other:xPath="/json/data/date" />
<attr name="Number" xs:type="decimal" readonly="false" other:xPath="/json/data/number" />
</api:info>

<api:set attr="DataModel" value="DOCUMENT" />
<api:set attr="URI" value="https://api.rest.com/v2/detailedInfo?nested={nested_id}" />
<api:match pattern="*nested_id*" type="glob" value="[_query.criteria]">
<api:else>
<api:set attr="URI" value="https://api.rest.com/v2/detailedInfo?nested=001" />
</api:else>
</api:match>
<api:set attr="JSONPath" value="$.data" />

<api:set attr="EnablePaging" value="TRUE" />
<api:set attr="pagetokenpath" value="/EnumerationResults/NextMarker" />
<api:set attr="pagetokenparam" value="marker" />

<!-- The GET method corresponds to SELECT. Here you can override the default processing of the SELECT statement. The results of processing are pushed to the schema's output. See SELECT Execution for more information. -->
<api:script method="GET">
<api:set attr="method" value="GET"/>
<api:call op="jsonproviderGet">
<api:push/>
</api:call>
</api:script>
</api:script>

This is the evolved version of the file I was sent, which contains some customer specific naming, so this is essentially the same.

Reply