Skip to main content
To homepage To homepage
Solved

Azure data storage table folder has invalid meta data

K
Contributor
Forum|alt.badge.img+2

We are facing a problem with a ODX data lake storage table. The problems are appearing at two points in our Data Warehouse:

  1. The ODX transfer tasks gives the warning “Table folder X is used by another table with id Y”
  2. When using the ‘Preview’ option from TX interface in ODX storage we get the error “The azure data lake gen2 data storage table folder has invalid meta data. The ODX must transfer the table again”

Using ‘Synchronize objects’ on ODX server does not do anything at all. Moreover, executing the table in DSA still works without errors and seems to contain new data. Also, in our Azure storage account we can see the transfer task adds new versions of the table every day (scheduled transfer task) even though it gives us the warning 1).

By the way, the table ID that the error on ODX transfer task gives us does not correspond with the TableId that we can see in the _model.json file in the corresponding folder in our Azure storage account. Where should the table ID from the first error point us to? Can we somehow locate this ID?

Concluding, what is causing these errors and what can we help to resolve and prevent it?

Best answer by Thomas Lind

Hi @KajEmergo 

This generally happens when you synchronize, so normally my suggestion would be to remove the Sync task from any scheduled execution.

Instead you run the synchronization task when you know there are changes in the source you want to apply. Part of this is also that whenever there is a change, you can’t avoid doing a manual synchronize task for the Data Warehouse anyway.

This issue is common for any file based CData data sources (CSV, JSON, XML, Excel and similar) where you could have the synchronize task not find the table and then on a future run find it again.

There really is no other way to avoid it than to not run the synchronize task often.

I don’t know if the CData specifically is to blame, but I don’t really see it for any other data source types.

If you want to make a case for it. I would suggest that you try to structure some sort of test to prove the issue.

So recreate the existing data source and choose some tables. Set it to run the sync task multiple times a day and schedule the transfer task to start in between that. See if you can catch the warning happen and create a support ticket with it.

View original
Did this topic help you find an answer to your question?

7 replies

Thomas Lind
Community Manager
Forum|alt.badge.img+5
  • Thomas Lind
  • Community Manager
  • 1017 replies
  • November 8, 2023

Hi @KajEmergo 

What version of TX are you currently on?

What sort of data source is it?

Normally this is due to the data source seeing this table as having disappeared and a new table with exactly the same fields and name is now attempting to use the same folder.

The only way to resolve it is to delete the table level folder in the data lake. Then on the next execution a new table with this name is generated, but it contains a different id in the _model.json file.

K
Contributor
Forum|alt.badge.img+2
  • KajEmergo
  • Author
  • Contributor
  • 52 replies
  • November 8, 2023

Hi @Thomas Lind,

We are on 20.10.40 and the data source is an OData source from CData. 

So basically what you are saying is that the table might have been removed and then added again. Could it also be caused like this:

  1. On data source level we have selected a number of tables
  2. These tables are used in synchronize task
  3. Deselect a table under 1) and run synchronize task again

We have indeed used your solution multiple times before, but the problem keeps on reappearing and we want to get rid of this once and for all.

Best,

Kaj

Thomas Lind
Community Manager
Forum|alt.badge.img+5
  • Thomas Lind
  • Community Manager
  • 1017 replies
  • Answer
  • November 8, 2023

Hi @KajEmergo 

This generally happens when you synchronize, so normally my suggestion would be to remove the Sync task from any scheduled execution.

Instead you run the synchronization task when you know there are changes in the source you want to apply. Part of this is also that whenever there is a change, you can’t avoid doing a manual synchronize task for the Data Warehouse anyway.

This issue is common for any file based CData data sources (CSV, JSON, XML, Excel and similar) where you could have the synchronize task not find the table and then on a future run find it again.

There really is no other way to avoid it than to not run the synchronize task often.

I don’t know if the CData specifically is to blame, but I don’t really see it for any other data source types.

If you want to make a case for it. I would suggest that you try to structure some sort of test to prove the issue.

So recreate the existing data source and choose some tables. Set it to run the sync task multiple times a day and schedule the transfer task to start in between that. See if you can catch the warning happen and create a support ticket with it.

K
Contributor
Forum|alt.badge.img+2
  • KajEmergo
  • Author
  • Contributor
  • 52 replies
  • November 8, 2023

This way of working sounds indeed fine to avoid the issue. I agree that it makes sense to not schedule the synchronize tasks, this used to be manual work when using business unit as ODX too. 

As long as this prevents the issue with the errors we are okay with it and will probably not make a case for it. Nevertheless I agree with the CData thing, this rarely happens on an ADF SQL source for example.

Best,
Kaj

devin.tiemens
TimeXtender Xpert
Forum|alt.badge.img+3
  • devin.tiemens
  • TimeXtender Xpert
  • 79 replies
  • February 6, 2024

Hi,

 

We have run into exact the same error. However based on the above information I don't understand how we can solve this issue. Could you please provide more information/ steps on how to fix this issue?

 

Thanks in advance!

 

Kind Regards,

Devin

rory.smith
TimeXtender Xpert
Forum|alt.badge.img+7
  • rory.smith
  • TimeXtender Xpert
  • 651 replies
  • February 7, 2024

Hi @devin.tiemens ,

the way to get rid of the warnings is to delete the source data in the ODX storage. The way to prevent the warnings is not to automate synchronisation so you do not synch when there is no data.

devin.tiemens
1 person likes this
  • devin.tiemens
    devin.tiemens
devin.tiemens
TimeXtender Xpert
Forum|alt.badge.img+3
  • devin.tiemens
  • TimeXtender Xpert
  • 79 replies
  • February 7, 2024

Thanks @rory.smith this indeed solved the problem!

Reply


Most helpful members this week

Terms and ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings