To homepage To homepage
Solved

Changing transformation in History table

  • 12 February 2024
  • 4 replies
  • 49 views
A
Rank

Hi Folks,

I have a painful process that involves uploading csv files into my ODX on ad-hoc basis. In the DSA, i use transformations to pull the existing fields apart into various new fields. This table has history enabled to track when key values change. I then push to the MDW, minus some of the ugly data and using SCD To and From dates to track changing dimensions.

A new set of entries with some extra parenthesis broke my job. I have a new set of case statements to deploy to the DSA history table, but as I went to deploy it, I panicked. The normal deploy and execute for a table without history will clear existing data and reload from scratch. Does the same happen with a history table?

This table would be a nightmare to rebuild.

TX version 19.11.7.64

icon

Best answer by Thomas Lind 14 February 2024, 12:28

View original

4 replies

A
Rank

Here it looks like no changes to the table structure, and no changes to the raw, error, message, or valid tables. The only change is Transformation view and data cleansing rules.

Am I safe to deploy changes?

rory.smith
Rank
Userlevel 5
Badge +7

Hi,

 

for more modern releases of TX there is a separate full load table task under deploy but in your case you might want to take the data in the valid table and copy it elsewhere to be safe or simply use a database backup?

19.11.7 is about 5 years old, might be time to try to upgrade a bit.

A
Rank

Basic and incremental tables do include the full load option under deploy. It doesn’t appear to be an option under the history table in this version.

We attempted to upgrade last year to 20.10.32 and ran into massive performance issues. Jobs with identical settings on identical hardware with identical routing taking 5x as long to complete. What was supposed to be a one-month project dragged on for over six, and we were forced to pause to tackle other priorities.

Thomas Lind
Rank
Userlevel 6
Badge +5

Hi @AndrewS 

You can’t, in our program at least, make our program attempt to deploy the valid table for a table set to run with history.

If the table mixes incremental load and history, doing a full deploy will not delete the valid data.

I tried to film it, but it is too large to add here.

Believe me if you deploy everything like this.

It will keep the data in the valid table. The _I incremental table will be truncated though.

Reply


Terms and ConditionsCookie settings