Add incremental load option to TimeXtender REST API

Tx is phasing out support for the CData REST connector. As support for incremental loadling is currently missing in TimeXtender’s native REST API connector, we cannot migrate existing connections yet.

Hi ​@erik.van.mastrigt 

While you can set incremental load for all CData providers, it will not really work for all file based CData providers.

If you use an REST API it will pull all the rows specified by the setup save it in memory and then apply the incremental rule to it. That is not exactly faster than just running a full load.

I have always suggested using dynamic values so you for example apply the max date as a parameter. This is also possible in the TX REST data source.

Hi ​@Thomas Lind 

When you set an incremental load on the CData REST provider, TX applies a ‘WHERE IncrementalTimeStamp>(Last max value)’ to the query that it sends to the CData driver.

You manipulate the RSD file to pick up this last max value by adding ‘other:filter=”[..]”’ on the attribute. For instance:

<attr name="IncrementalTimeStamp" xs:type="datetime" readonly="false" other:xPath="/json/rows/IncrementalTimeStamp" other:filter="IncrementalTimeStamp:>" />

It will add parameter IncrementalTimeStamp to the uri. This way you can use TimeXtender’s native incremental load functionality on REST API’s… if supported by the API itself, of course.

​@Thomas Lind Do you have any idea if there are plans to add incremental load option in the near future to the TimeXtender REST provider?

I all, this one is now implemented: Data source providers r. 2025-10-06 | Community

​@JTreadwell thanks, this release looks promising!

However, we have a quite niche solution where we extract data from Azure Storage Tables (data which is retrieved and prepared from a source system by a different department). The old CDATA provider for Azure Storage Tables also supported primary key updates and primary key deletes for this source, probably because with Azure Storage Tables the primary key set is static (PartitionKey, RowKey). This enabled us to incrementally load the data with delete detection. Is this something that is also considered to add in the future or did we loose this functionality after we migrated to the more generic TimeXtender REST data provider?

We will look into this. 

​@JogchumSiR, Have you tried this pulling the data using the TimeXtender REST API? see: Query Entities (REST API) - Azure Storage | Microsoft Learn

Hi ​@JTreadwell ,

We have setup the new datasources to these Azure Tables using the TimeXtender REST API. It works great and pulls the data all right, but we had to resort to full loads instead of incremental because of the ‘delete detection’ option that's missing with the TimeXtender REST API provider. The load times went from 1:30 minute to 10 in worst cases.