+4When creating a data source, a synchronize task is automatically created and you are prompted to set up a transfer task. Storage management tasks may be the most important task of all, and i.m.h.o should automatically be added to every data source or the user should get a prompt to add one.
I have seen several clients now with endlessly growing ODX storage because they have no cleaning tasks and are unaware that transfer tasks are cumulative.
Kind regards,
Andrew
+8Hi,
it is additionally important to roll-up incrementals regularly if you set up incremental load in an ADLS-backed ODX.
+8Dear
What do you mean with ‘roll up incrementals’? Like a full load or something to consolidate all the ‘single’ files into one big(ger) file? Or…?
Thanks!
+8Hi
exactly. There are three types of Storage Management task: Delete old versions, Move to cold storage, and incremental roll-up (requires ADF). This takes delta files and creates a new Full load. Depending on the nature of your incremental it may be critical to do this regularly to avoid long load times due to needing to consolidate a large amount of very small files.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
