Skip to main content
To homepage To homepage
Solved

ODX Rollup task does nothing

S
Contributor
Forum|alt.badge.img

Hi,

I have set up a Storage Management Task with only the Rollup option ticked, and filled in the right ADF credentials. One of my source tables has been selected for this task.

TX says the runs are successful and they finish in a fraction of a second, but nothing happens in the storage account. The version folders contain the full load file + all the incremental files, as before. I have tried with many different settings for min and max rollup file size. I have not specified “folder (optional)”. I have tried with and without specifying integration runtime.

No pipeline run seems to show up in ADF after triggering the task.

Best answer by Christian Hauggaard

This roll up issue appears to be due to Updates & Deletes being enabled in incremental load settings. The product team will look into supporting roll up in combination with updates and deletes in the next build cycle

View original
Did this topic help you find an answer to your question?

Thomas Lind
Community Manager
Forum|alt.badge.img+5

Hi @sigsol 

You are on the newest version of the ODX as well?

S
Contributor
Forum|alt.badge.img
  • sigsol
  • Contributor
  • March 18, 2024

Hi Thomas!

The ODX version is 6521.1.

S
Contributor
Forum|alt.badge.img
  • sigsol
  • Contributor
  • March 19, 2024

Hi @Thomas Lind 

We have now upgraded to ODX version 6563.1, but there is no change unfortunately. The task runs successfully but nothing happens with the incremental files.

There are no task details in the execution log panel and I can’t see any corresponding pipeline runs in ADF.

Thomas Lind
Community Manager
Forum|alt.badge.img+5

Hi @sigsol 

You are using the same ADF setup to do transfers between ODX and DWH, right?

Also about the min max size, the files in the data lake container folders have files larger than the min size? I would guess this is the trigger that is used to merge them. Whether it is across the ones existing or just that one needs to be larger I am not sure.

S
Contributor
Forum|alt.badge.img
  • sigsol
  • Contributor
  • March 20, 2024

Hi @Thomas Lind 

Correct, we are using the same setup for ODX-DWH transfers.

I have tried having min max sizes larger than both full load and incremental files, smaller than both, and a small min and large max. Is it possible to get a confirmation as to what exactly these limits refer to? I didn’t manage to find an explanation in the documentation.

Thomas Lind
Community Manager
Forum|alt.badge.img+5

Hi @sigsol 

You do need a lot of data in your folder to make this work.

1MB is equal to 9765625 kiB (kibibyte)

So the total needs to be larger than that for it to do the roll-up.

S
Contributor
Forum|alt.badge.img
  • sigsol
  • Contributor
  • March 21, 2024

Hi @Thomas Lind 

All our folders have more than 1 MB. So it should work then?

The number of files in the folder greatly affects performance and we can’t keep doing full loads for some of the sources.

Thomas Lind
Community Manager
Forum|alt.badge.img+5

Hi @sigsol 

Can you show how one of the folders look with these batch files? I could not myself generate enough data to make it work and I would like to show if it works or not and with this I can pass it on.

You can try to send it to me on a private message if you think it can’t be shared.
Here is how it looks for me.

 

Christian Hauggaard
Community Manager
Forum|alt.badge.img+5

This roll up issue appears to be due to Updates & Deletes being enabled in incremental load settings. The product team will look into supporting roll up in combination with updates and deletes in the next build cycle

Reply


Most helpful members this week

Terms and ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings