To homepage To homepage
Solved

Azure data storage table folder has invalid meta data

  • 8 November 2023
  • 7 replies
  • 108 views
K
Rank
Userlevel 2
Badge +1

We are facing a problem with a ODX data lake storage table. The problems are appearing at two points in our Data Warehouse:

  1. The ODX transfer tasks gives the warning “Table folder X is used by another table with id Y”
  2. When using the ‘Preview’ option from TX interface in ODX storage we get the error “The azure data lake gen2 data storage table folder has invalid meta data. The ODX must transfer the table again”

Using ‘Synchronize objects’ on ODX server does not do anything at all. Moreover, executing the table in DSA still works without errors and seems to contain new data. Also, in our Azure storage account we can see the transfer task adds new versions of the table every day (scheduled transfer task) even though it gives us the warning 1).

By the way, the table ID that the error on ODX transfer task gives us does not correspond with the TableId that we can see in the _model.json file in the corresponding folder in our Azure storage account. Where should the table ID from the first error point us to? Can we somehow locate this ID?

Concluding, what is causing these errors and what can we help to resolve and prevent it?

icon

Best answer by Thomas Lind 8 November 2023, 14:38

View original

7 replies

Thomas Lind
Rank
Userlevel 6
Badge +5

Hi @KajEmergo 

What version of TX are you currently on?

What sort of data source is it?

Normally this is due to the data source seeing this table as having disappeared and a new table with exactly the same fields and name is now attempting to use the same folder.

The only way to resolve it is to delete the table level folder in the data lake. Then on the next execution a new table with this name is generated, but it contains a different id in the _model.json file.

K
Rank
Userlevel 2
Badge +1

Hi @Thomas Lind,

We are on 20.10.40 and the data source is an OData source from CData. 

So basically what you are saying is that the table might have been removed and then added again. Could it also be caused like this:

  1. On data source level we have selected a number of tables
  2. These tables are used in synchronize task
  3. Deselect a table under 1) and run synchronize task again

We have indeed used your solution multiple times before, but the problem keeps on reappearing and we want to get rid of this once and for all.

Best,

Kaj

Thomas Lind
Rank
Userlevel 6
Badge +5

Hi @KajEmergo 

This generally happens when you synchronize, so normally my suggestion would be to remove the Sync task from any scheduled execution.

Instead you run the synchronization task when you know there are changes in the source you want to apply. Part of this is also that whenever there is a change, you can’t avoid doing a manual synchronize task for the Data Warehouse anyway.

This issue is common for any file based CData data sources (CSV, JSON, XML, Excel and similar) where you could have the synchronize task not find the table and then on a future run find it again.

There really is no other way to avoid it than to not run the synchronize task often.

I don’t know if the CData specifically is to blame, but I don’t really see it for any other data source types.

If you want to make a case for it. I would suggest that you try to structure some sort of test to prove the issue.

So recreate the existing data source and choose some tables. Set it to run the sync task multiple times a day and schedule the transfer task to start in between that. See if you can catch the warning happen and create a support ticket with it.

K
Rank
Userlevel 2
Badge +1

This way of working sounds indeed fine to avoid the issue. I agree that it makes sense to not schedule the synchronize tasks, this used to be manual work when using business unit as ODX too. 

As long as this prevents the issue with the errors we are okay with it and will probably not make a case for it. Nevertheless I agree with the CData thing, this rarely happens on an ADF SQL source for example.

Best,
Kaj

devin.tiemens
Rank
Userlevel 2
Badge +2

Hi,

 

We have run into exact the same error. However based on the above information I don't understand how we can solve this issue. Could you please provide more information/ steps on how to fix this issue?

 

Thanks in advance!

 

Kind Regards,

Devin

rory.smith
Rank
Userlevel 5
Badge +7

Hi @devin.tiemens ,

the way to get rid of the warnings is to delete the source data in the ODX storage. The way to prevent the warnings is not to automate synchronisation so you do not synch when there is no data.

devin.tiemens
Rank
Userlevel 2
Badge +2

Thanks @rory.smith this indeed solved the problem!

Reply


Terms and ConditionsCookie settings