Incremental load Custom Tables in stage

Currently it's not possible to set a custom table to incremental loading.

It would be handy to have this option when you need to bring several tables together in a custom table and the source tables are capable of incremental loading.

Some ERP systems separate Invoice and Order lines based on the type of document. E.g. in Navision you need to breng Invoice Lines and Credit Memo lines together to get "all the lines".

Some ERP systems go even further and separate depending on the line type e.g. items, accessories, labour, vehicles etc...

If you need to have these tables brought together in a custom table so that you can only build your logic once instead of x times, you are currently forced to always full load this custom table, which can take a long time.

If however you could put a custom table in incremental mode and use the DW_TimeStamp of your "source" stage table as the data selection timestamp, this would remove a lot of processing time.

You can compare it to loading your DW incrementally based on the PK of the source and using the DW_TimeStamp as date field, but here you would do it in the stage itself.


Please sign in to leave a comment.