To homepage To homepage
Solved

Transformations / changes not visible until forced full load

  • 10 January 2023
  • 7 replies
  • 90 views
rory.smith
Rank
Userlevel 5
Badge +7

When you add tables to a DWH data area (from an Azure Data Lake backed ODX) and subsequently perform a Deploy & Execute, data is loaded normally as expected. In this example I am adding AW2014's Person.Person and Sales.SalesPerson tables. I subsequently want to add a field where I add a transformation to generate the full name for a person (Firstname + ' ’ + Lastname). When I add this field with the transformation and perform a Deploy & Execute, the field is not populated.

The reason for this is probably the same reason under: https://legacysupport.timextender.com/hc/en-us/requests/25792

 

I would expect changes to ETL to trigger a full reload without needing to either change the Table Settings > Data Extraction > Full Load or perform a cumbersome full reload through D&E > Review tasks.

Given that this issue seems generic and not only limited to Historiy tables, it is probably a good idea to solve this generally as well.

icon

Best answer by Thomas Lind 10 January 2023, 14:48

View original
1 Attachment

7 replies

Thomas Lind
Rank
Userlevel 5
Badge +5

Hi Rory

That will happen if the table is running with incremental load.
It will only update new fields with a value and if there is no added rows it will not update the field.

However there is a workaround to get it to show data.
I have a excel file which I set up to run with incremental load. As explained I deployed and executed this.
Then I added a custom field and did as you merging two fields.

In my Incremental setup area I made sure that I checked the field Keep field values up-to-date.
Then I deployed an executed again.

This gave me this look.

 

So to avoid the above always check this.
What you need to watch out for is that the fields needing updates makes it as slow as running without incremental load. Then running a weekly full load would be better.

Regards
Thomas Lind

rory.smith
Rank
Userlevel 5
Badge +7

Hi Thomas,

 

thanks - I know the work-arounds, but those aren't really usable in a larger dataset. In my opinion the _I deploy should be automatically triggered just like in the historical table ticket. I don't think this new behaviour makes sense without full reloads being much easier to trigger, or TX should give you the option automatically.

Thomas Lind
Rank
Userlevel 5
Badge +5

Hi Rory

It is not new behavior, the difference is that setting up incremental load is set up on the data source in the ODX and then makes the table you add from the source run with it.

You could set it to Full load instead of Automatic, for the first run whenever you add changes.

I have always used this method whenever I add changes to incrementally loaded tables.

https://use.vg/BpokfT

rory.smith
Rank
Userlevel 5
Badge +7

Hi Thomas,

 

for anyone coming from a non-ODX Server setup in 20.x or older this is new behaviour. As the incremental is automatic and only visible when the mapping is expanded or the table settings are checked this is at least a UI issue. When I manipulate incremental tables in BU implementations this is obvious, now I would have to change the setting on all tables until I have a baseline implementation and then turn all tables to incremental or remember to check the box on each D&E.

JTreadwell
Rank
Userlevel 3
Badge +5

@rory.smith, Your insights are always appreciated. It’s always tough making design changes like this, since they may please one type but frustrate others. I’d encourage you to take some time to get more comfortable with the new approach and if you still find it challenging, then please add an idea. Remember it help us best if you try and keep the desired end goal (what are you trying to accomplish) in mind when submitting suggestions. Thanks for the feedback. 

-Joseph Treadwell
rory.smith
Rank
Userlevel 5
Badge +7

@JTreadwell : I understand the difficulty, though I think in this case everyone could have been made happy, but that is a discussion best had over a beer or two.

I have submitted an idea, hopefully there is enough context.

JTreadwell
Rank
Userlevel 3
Badge +5

Thanks @rory.smith.

-Joseph Treadwell

Reply


Terms and ConditionsCookie settings