New field in an incremental table doesn't populate

​@Thomas Lind, good article. This always comes up as a question when I assist clients/coworkers taking the course. I have done step 2 and 3 many times when I have needed to add fields to large inc. load tables. This requires a bit of technical SQL knowledge on the TimeXtender user.
It seems to me that it would be possible to automate this and add it as a feature in TimeXtender for inc. load tables. Something like “Update incremental load field” as an advanced execution action.
Could this be added as an product idea suggestion?

Hi ​@pontus.berglund 

There is an idea for a force full load on table option, but this part is probably more what people actually want.

The keep field values up to date option also updates fields like this, but it does it automatically not by forcing it.

Hi,

doesn't the keep field values up to date only apply in the situation where a lookup/key is initially NULL and later gets a value? I.e. transformations are not all recalculated? 

​@Thomas Lind the force full load is already in place in my opinion, albeit a bit hidden.
These steps do the trick: Deploy > Review tasks > Check full load valid table.
I have experimented a bit with “keep values up to date” and as you say it updates the fields automatically but it always takes a toll on performance, and makes loading very large incremental tables slow. Also, in my experience the need for updating fields are one-off events. It’s usually a new field being added, or a dimension table that has changed. Whenever this happens, I do what you describe in step 2 or 3.
​@rory.smith yes I do believe it does not recalculate transformations.

Hi,

there's also the Execution Package route which is especially useful if you also need to full load other tables in one set.

Hi ​@rory.smith 

It used to be only lookups, but it changed I believe. The field used to be called keep lookup values up to date.

The I icon tells this.

To me “If enabled, all fields in the valid table will be kept up-to-date”, will mean that if you added a new field from the source it would be updated back in time as well.

This feature is for when you never wants to do a full load again. So it needs to be able to handle all the changes you can make without having to do a full load.

There is a performance decrease with this, but you will have to compare that to doing full loads, or running custom queries as explained above.

​@pontus.berglund 

It would seem like you need to add a feature request for that then. 

The only time it wouldn’t be fine with a force full load of specific field option, would be when it is affected by another table changing its content. You may not know it is incorrect before it has been like so for a while.

Nice to have found this thread! 

Applied a cleansing script and that solved my issue with field transformations not updating for the full valid table.

My particular fact table is less than 1 m rows, so I preferred applying a cleansing script action instead of manually updating or manually full loading.

Before encountering this issue - I had expected that having the checkbox checked would also keep the columns with field transformation up to date. It’s implicit enough.

Perhaps a new checkbox relevant for keeping field transformation also up-to-date would be a nice option to have. Or maybe a self-join for this is the way to go performance wise?