Hello Community!
When running a rollup task, the resulting rollup file ends up in a separate folder inside the same full load folder:
- After the rollup, new incremental files continue to appear in the DATA folder and the old incremental files are still there. What have we achieved by doing the rollup? Is the new rollup file being used at all?
- How can we delete the old incremental files given that the delete Storage Management task only removes entire full load folders and we don’t perform full loads?
Currently I don’t understand how we can clean up the many incremental files, both for load performance and storage management.
Please note: We cannot rely on doing regular full loads because of the size of the tables.
Thanks!
TX version: 6745.1
Connector: TimeXtender SQL Data Source 21.0.2.1
We are using Azure Data Lake and ADF.