Skip to main content

    Filter by idea status

    Filter by product

    859 Ideas

    andrew.gebhard
    TimeXtender Xpert
    andrew.gebhardTimeXtender Xpert

    Integrating legacy historical data in TX history tableSUBMITTED

    Hi team, In many projects, I need to integrate legacy historical data with current updates — essentially allowing TimeXtender to continue tracking Type 2 Slowly Changing Dimension (SCD) changes as if it had always managed the full history. Currently, this isn’t natively supported. To make it work, I have to either misuse incremental/history features, write custom scripts, or use a mix of both. In SQL, this problem isn’t that complex. But in TimeXtender, the logic becomes tricky when you're sourcing data from a pre-existing SCD-2 structure.Logically, you’d want to: Load everything initially as Type 1, Then filter for IsCurrent = 1, And apply your Type 2 fields accordingly. But this doesn't work cleanly in TimeXtender, since the natural/primary key definitions are different for full-history and current-only records.  Scenario 1: Import Static History Once + Load Only IsCurrent Going ForwardInitial Run: Use Valid_From as part of the primary key. Map Valid_From and Valid_To from the source to SCD_From and SCD_To in TimeXtender. This seeds the full historical record set. After Initial Run: Switch to using only the BusinessKey as the primary key. Add a data selection rule like IsCurrent = 1, or point to a separate current-only source table. Now TimeXtender tracks changes using standard Type 1 and Type 2 logic on new data. This approach is straightforward and works well for historical seeding followed by real-time tracking. Scenario 2: Sync Ongoing SCD-2 Source (Still Changing Slowly)Initial Load: Use BusinessKey + Valid_From as the primary key. Subsequent Loads: Use only BusinessKey as the primary key. Then apply logic based on the Valid_From date (within BusinessKey): Case A – Valid_From <= last known SCD_From: Update Type 1 fields if values have changed. Insert a new Type 2 row if a Type 2 field has changed. Case B – Valid_From > last known SCD_From: Option 1: Order rows by Valid_From and map to SCD_From/SCD_To. Option 2: Take the most recent record only, and apply Type 1/2 change logic as needed. This scenario is more complex and requires careful handling of edge cases like overlapping dates or reissued statuses.Use Cases When the source system has incomplete SCD-2 tracking. Migrating from a legacy DWH that has historical data TimeXtender needs to continue. Blending multiple systems, where each handles history differently. Replacing a legacy system with built-up history by a new system that doesn’t track history.

    rory.smith
    TimeXtender Xpert
    rory.smithTimeXtender Xpert

    Even smarter Smart Synchronize in Prepare MappingsSUBMITTED

    A common pattern in staging Data Areas is to rename fields when you are denormalising data. An example you may remember from TimeXtender Core Training is renaming the Name field in the ProductModel table to be ProductModelName as you do not have the table context if you start looking up data.In the Ingest instance we now have the Metadata manager to avoid needing to remap things if we have a name change between input (source naming) and output (Ingest structure). In the Prepare instance, this is not yet available. If I rename a field and change the mapping (move from ADLS Gen2 Ingest to Fabric Ingest for instance), a Smart Synchronize will not recognize that the ProductModelName field was originally the Name field. This means you will then need to expand the Mapping folder, highlight the mapping and drag the correct Name field from the rightmost column onto the ProductModelName field in your table.It would be much better if TDI kept track of source item name and target item name separately and used this information to make the Smart Synch smarter. This way it could know that the ProductModelName field was originally called Name in the original mapping, the new mapping also has a Name field and can therefore automatically be mapped. If there is doubt, the same kind of interface we are used to for other synchronisation resolutions can be used to show the user what TDI is going to do and indicate any areas where user input is required.