Skip to main content
To homepage To homepage

Hi,

I have set up a Storage Management Task with only the Rollup option ticked, and filled in the right ADF credentials. One of my source tables has been selected for this task.

TX says the runs are successful and they finish in a fraction of a second, but nothing happens in the storage account. The version folders contain the full load file + all the incremental files, as before. I have tried with many different settings for min and max rollup file size. I have not specified “folder (optional)”. I have tried with and without specifying integration runtime.

No pipeline run seems to show up in ADF after triggering the task.

Hi @sigsol 

You are on the newest version of the ODX as well?

Hi Thomas!

The ODX version is 6521.1.

Hi @Thomas Lind 

We have now upgraded to ODX version 6563.1, but there is no change unfortunately. The task runs successfully but nothing happens with the incremental files.

There are no task details in the execution log panel and I can’t see any corresponding pipeline runs in ADF.

Hi @sigsol 

You are using the same ADF setup to do transfers between ODX and DWH, right?

Also about the min max size, the files in the data lake container folders have files larger than the min size? I would guess this is the trigger that is used to merge them. Whether it is across the ones existing or just that one needs to be larger I am not sure.

Hi @Thomas Lind 

Correct, we are using the same setup for ODX-DWH transfers.

I have tried having min max sizes larger than both full load and incremental files, smaller than both, and a small min and large max. Is it possible to get a confirmation as to what exactly these limits refer to? I didn’t manage to find an explanation in the documentation.

Hi @sigsol 

You do need a lot of data in your folder to make this work.

1MB is equal to 9765625 kiB (kibibyte)

So the total needs to be larger than that for it to do the roll-up.

Hi @Thomas Lind 

All our folders have more than 1 MB. So it should work then?

The number of files in the folder greatly affects performance and we can’t keep doing full loads for some of the sources.

Hi @sigsol 

Can you show how one of the folders look with these batch files? I could not myself generate enough data to make it work and I would like to show if it works or not and with this I can pass it on.

You can try to send it to me on a private message if you think it can’t be shared.
Here is how it looks for me.

 

This roll up issue appears to be due to Updates & Deletes being enabled in incremental load settings. The product team will look into supporting roll up in combination with updates and deletes in the next build cycle

Reply


Terms and ConditionsCookie settings