Dear all,
Some clients have a lot of data. some of the data is from 2010 or even earlier.
Each day my ODX or BU runs, it has to check ALL the PK's for changes and this takes a long time.
I've created some scripts / SQL snippets and whatnot to just load the current year (current and last year) and then eventually bringing them together in the semantic layer.
It has become quite complex for these big tables because I'm load two seperate tables, but want to make sure that they both get the same transformations and business rules and I do not want to maintain two seperates pieces of script (because mistakes are easily made).
What I propose is a functionality which let you choose which data from the source or DWH tables to refresh / check in the incremental load and stuff and which data in the table to ‘archive’ and only load when explicitly asked for (full load, manual stuff). So in TX you are handling one table and one script and one custom transformation but are governing the current table with the data you want to reload or have checked every day and one archive table which contains all the history which does not have to be checked or loaded each day.
