Skip to main content
To homepage To homepage
Solved

Duplicates loading data from ODX to MDW

M
Starter

Hi community,

we are facing the problem, that we create duplicates when we bring data from an ODX API source to the MDW. We are working with an overlapping sliding window of two days in the schema file (because data can change and there is no last modified date) and only set the primary key on the ODX source.

In the MDW (dedicated SQL pool) we enabled the history and set the ID as natural key. All fields are marked as type 1 fields.

The execution brings us duplicated ID values - no updates are made


Thanks for your help
Michael

Best answer by Thomas Lind

Hi Michael

If you look at the SCD values, does they seem to mention these seen as unique rows or changes?

I mean does only one have a is current value or a date range between them?

If not it would seem like the duplicated natural keys are not seen as being completely unique.

View original
Did this topic help you find an answer to your question?

6 replies

Thomas Lind
Community Manager
Forum|alt.badge.img+5
  • Thomas Lind
  • Community Manager
  • 1017 replies
  • March 31, 2023

Hi Michael

How is the table in the Data Area set up? It will not update/remove the duplicates on a simple mode table for example.

 

 

M
Starter

Hi Thomas,

These are our table settings:

Thanks
Michael

 

Thomas Lind
Community Manager
Forum|alt.badge.img+5
  • Thomas Lind
  • Community Manager
  • 1017 replies
  • March 31, 2023

Hi Michael

I noticed that it is a History table, what option do you using for the other fields, such as type II fields?

The ID needs to be unique. If it is not it would make it add the same rows again and again.

M
Starter

Hi Thomas,

 

as mentioned in the first post, we are only using SCD 1. The ID is unique in the source. I attach one example for the duplicates - the id is unique, but we have 4 entries for the same Id - we loaded more often, so that we can see it in detail:

 

Thanks
Michael

Thomas Lind
Community Manager
Forum|alt.badge.img+5
  • Thomas Lind
  • Community Manager
  • 1017 replies
  • Answer
  • April 11, 2023

Hi Michael

If you look at the SCD values, does they seem to mention these seen as unique rows or changes?

I mean does only one have a is current value or a date range between them?

If not it would seem like the duplicated natural keys are not seen as being completely unique.

Christian Hauggaard
Community Manager
Forum|alt.badge.img+5

Hi @Michael Suppan do you have an update on this? are there multiple records with the same ID where SCD Is current field is set to 1 for several records?

Reply


Most helpful members this week

Terms and ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings