When running TimeXtender in a scenario with ADF-enabled Ingest server you will find that a single pipeline with merge transfer is used on execution. Handling in the DSA etc. will only start when this merge transfer is fully complete. This means executions in the new release are less performant than they could be. Perhaps there is one table that takes 2 hours to copy blocking execution on a very powerful SQL instance hosting the Prepare instance where the faster tables have expensive cleansing that could be run in the meantime.
The idea is to allow a number of pipelines to be defined, where TimeXtender can allocate transfers across them similar to how this is done in threading. This way Prepare instance execution of tables allocated to a 'faster’ pipeline can proceed while some of the longer Ingest instance → Prepare instance data movements are still being handled by a 'slower’ pipeline.
Execution management should then take transfer and cleanse times into account to make a smart decision on allocating tables to different merge pipelines with the goal of reaching the shortest overal execution time possible.
