Incremental Rest Datasource without parameters

Question

We have a client with a Rest Datasource that does not use URL parameters. Instead they use the post body to pass a series of filters and conditions.

We are using this post body to implement Incremental loading from the datasource. It seems to be working correctly in so far as the rest source is definitely loading incrementally.

However the Ingest instance does not seem to recognize the tables in the Gen 2 Storage as incremental.

We have defined the incremental several times, deleted the entire folder in the Azure Gen2 storage. Still the same result.

if we drag the table over to a data area the source in the Ingest instance is not flagged with an “i” indicating that it is incremental.

Is this because we are passing the incremental variable through the post body and not as a query parameter?

if this is the case is there a work around?

The issue we are facing is that we cannot do any downstream incremental loads as TX complains that the source table in the ingets instance is not incremental.

Note that we are using the TX Rest Datasource version 10.2.0.0 Ingest instance is 7150.1

Thomas Lind · Accepted Answer

Hi ​@Paul McLeod Bespoke AnalyticsDo you have a incremental rule on the endpoint in the incremental menu?That should be enough, you should also use the dynamic value name rule that gets created.

Paul McLeod Bespoke Analytics · Answer

Hey Thomas, We are using the dynamic value name, We have not used the “target-based incremental load” setting, but I can see how that could address the issue if we are also passing the Dynamic value through the Post Body. I will give that a try! Thanks for the feedback!

Sign up

Login with SSO

Login to the community

Login with SSO