Skip to main content
SUBMITTED

Add new Incremental Load Rules in ODX Server

Related products:TimeXtender Data Integration
  • July 21, 2023
  • 3 replies
  • 43 views

bas.hopstaken
TimeXtender Xpert
Forum|alt.badge.img+5

This week I tried to create an incremental selection rule to load every table incremental based on the timestamp column. Except for some tables, those table needs to be fully reloaded.
An option could be to create multiple transfer tasks, but in this case, it is hard to schedule. The reload is scheduled to run every 5 minutes and it is not possible to run 2 ODX tasks at the same time.

I’ve contacted our Solution Specialist and we could not find a way to reload all tables incrementally, except a few. We tried to create multiple Incremental Load Rules, using the 'Not Contains' operator. But this operator is only exception 1 table as input. So we created multiple rules for each table that do not need to be incrementally loaded. This doesn’t work because if you create multiple rules, it is an OR instead of an AND. 

I think this could be solved to add a new operator called ‘Not in list’ or ‘Not Contains in list’. Or add an option to handle multiple incremental selection rules as an AND condition, instead of an OR condition.

3 replies

rory.smith
TimeXtender Xpert
Forum|alt.badge.img+8
  • TimeXtender Xpert
  • July 21, 2023

Hi @bas.hopstaken,

How about using one source for all the incremental tables and one for the full load ones? I believe that also overcomes the scheduling as you can parallel load across sources iirc.

Don't forget to factor ODX-->DSA transfers into the data source connections.


bas.hopstaken
TimeXtender Xpert
Forum|alt.badge.img+5
  • Author
  • TimeXtender Xpert
  • July 25, 2023

Hi @bas.hopstaken,

How about using one source for all the incremental tables and one for the full load ones? I believe that also overcomes the scheduling as you can parallel load across sources iirc.

Don't forget to factor ODX-->DSA transfers into the data source connections.

Hi @rory.smith ,

In this case that is not an option. The data source is Business Central on-prem. We are loading 2 companies. Because the BC Adapter data source is not (yet) available in ODX Server, I have 2 data sources connecting to the same BC SQL database. In the ‘Select tables’ option, I selected all tables starting with  'CompanyA$’. The other connection is extracting all tables starting with 'CompanyB$’. Based on the unique DW_SourceCode, I can distinguish the different companies.  As far as I know, there is no other option to distinguish the different companies. I discussed this with our Solution Specialists as well, but they couldn't find another solution for this. 

Besides the 2 companies, the customer is also using a DTAP environment in BC. So I already have a lot of connections to different BC databases. When I separate connections for full and incremental loads and need to load a new table, I have to add it to a lot of data sources manually.

Another option could be to use the new ‘Source Table’ system field to extract the company names. But this option is not available in the V20.10.x versions.


rory.smith
TimeXtender Xpert
Forum|alt.badge.img+8
  • TimeXtender Xpert
  • July 25, 2023

Hi @bas.hopstaken ,

initially you need to double your data sources indeed. But after that a table is either full or incremental and gets allocated to a source based on that. I guess you could also take the BU approach in the intermediate.

Source, DWH and BI DTAP tends to generate a lot of overhead. If you want to work that way, you also have to accept it detracts from the speed with which you can move. If you also add in DTAP at the infrastructural level, you get a staggering pipeline from idea to result.