Resume load on error or intentionally break a load into chunks

In case of source systems with small windows and large tables, it would be useful to have an option of resuming a load. This would require that an incremental key orders the transfer and that this key is used to restart the load process from source.

It should be optional if the data cleansing process is postponed for a complete transfer to raw or if it could be run on the partial loaded raw table.

Further it should be possible to intentionally break a load into chunks by ether command time, records or files handled (Multiple text files). This could ease the load of large tables and offload the burden of initial loads and periodical full load both with regard to strain on source systems and the required log files in the stage database.


Please sign in to leave a comment.