Skip to main content

    Filter by idea status

    Filter by product

    863 Ideas

    andrew.gebhard
    TimeXtender Xpert
    andrew.gebhardTimeXtender Xpert

    Integrating legacy historical data in TX history tableSUBMITTED

    Hi team, In many projects, I need to integrate legacy historical data with current updates — essentially allowing TimeXtender to continue tracking Type 2 Slowly Changing Dimension (SCD) changes as if it had always managed the full history. Currently, this isn’t natively supported. To make it work, I have to either misuse incremental/history features, write custom scripts, or use a mix of both. In SQL, this problem isn’t that complex. But in TimeXtender, the logic becomes tricky when you're sourcing data from a pre-existing SCD-2 structure.Logically, you’d want to: Load everything initially as Type 1, Then filter for IsCurrent = 1, And apply your Type 2 fields accordingly. But this doesn't work cleanly in TimeXtender, since the natural/primary key definitions are different for full-history and current-only records.  Scenario 1: Import Static History Once + Load Only IsCurrent Going ForwardInitial Run: Use Valid_From as part of the primary key. Map Valid_From and Valid_To from the source to SCD_From and SCD_To in TimeXtender. This seeds the full historical record set. After Initial Run: Switch to using only the BusinessKey as the primary key. Add a data selection rule like IsCurrent = 1, or point to a separate current-only source table. Now TimeXtender tracks changes using standard Type 1 and Type 2 logic on new data. This approach is straightforward and works well for historical seeding followed by real-time tracking. Scenario 2: Sync Ongoing SCD-2 Source (Still Changing Slowly)Initial Load: Use BusinessKey + Valid_From as the primary key. Subsequent Loads: Use only BusinessKey as the primary key. Then apply logic based on the Valid_From date (within BusinessKey): Case A – Valid_From <= last known SCD_From: Update Type 1 fields if values have changed. Insert a new Type 2 row if a Type 2 field has changed. Case B – Valid_From > last known SCD_From: Option 1: Order rows by Valid_From and map to SCD_From/SCD_To. Option 2: Take the most recent record only, and apply Type 1/2 change logic as needed. This scenario is more complex and requires careful handling of edge cases like overlapping dates or reissued statuses.Use Cases When the source system has incomplete SCD-2 tracking. Migrating from a legacy DWH that has historical data TimeXtender needs to continue. Blending multiple systems, where each handles history differently. Replacing a legacy system with built-up history by a new system that doesn’t track history.

    rory.smith
    TimeXtender Xpert
    rory.smithTimeXtender Xpert

    Even smarter Smart Synchronize in Prepare MappingsSUBMITTED

    A common pattern in staging Data Areas is to rename fields when you are denormalising data. An example you may remember from TimeXtender Core Training is renaming the Name field in the ProductModel table to be ProductModelName as you do not have the table context if you start looking up data.In the Ingest instance we now have the Metadata manager to avoid needing to remap things if we have a name change between input (source naming) and output (Ingest structure). In the Prepare instance, this is not yet available. If I rename a field and change the mapping (move from ADLS Gen2 Ingest to Fabric Ingest for instance), a Smart Synchronize will not recognize that the ProductModelName field was originally the Name field. This means you will then need to expand the Mapping folder, highlight the mapping and drag the correct Name field from the rightmost column onto the ProductModelName field in your table.It would be much better if TDI kept track of source item name and target item name separately and used this information to make the Smart Synch smarter. This way it could know that the ProductModelName field was originally called Name in the original mapping, the new mapping also has a Name field and can therefore automatically be mapped. If there is doubt, the same kind of interface we are used to for other synchronisation resolutions can be used to show the user what TDI is going to do and indicate any areas where user input is required.

    rory.smith
    TimeXtender Xpert
    rory.smithTimeXtender Xpert

    Improve 'stacked' data sourcesSUBMITTED

    Many customers have ‘stacked’ sources: many instances of the same source (BC / Dynamics, etc.) that in the TX Classic release can be easily handled with Additional Data Sources. Here you define the data selection in the ‘master’ source and have the same data extracted from the underlying clones.In TDI you would currently need to have separate sources for each and use the, very restrictive, mapping set feature to auto-map tables with identical (field, data type) structure to the same Prepare table.The possibility of cloning data sources and using connection strings / json settings, makes the creation of many of the same sources easier. There are still two issues remaining however:Each additional source needs the data selection set and maintained separately If one of many ‘stacked’ sources updates by adding or changing metadata, this will break mapping setsThe first issue could be solved by allowing to clone Metadata Manager settings from one ‘master’ source to secondary sources with the same structure. It should be possible to do this before the initial metadata sync task is automatically triggered. In this situation I would expect that the Metadata Manager shows you:What is successfully found in the target from the source mapping What was not found in the target from the source (maybe due to a schema name difference, etc.) What exists in the target additionally to what is in the sourceIf in future one of the sources updates to add a new field, it should be possible to clone this across easily once the other sources also get this field. I expect you would mark one source as the ‘master’ for its set, and others as receiving mapping changes. The second problem occurs when there are deviations between data sources that cause Mapping Sets not to work. If one of the sources has an additional field in a table with respect to the others, it will not work. In my opinion a mapping set should be an intersection of common fields, not an exact match. This way new fields added won't break Mapping Sets already existing and the feature becomes more widely useful. If a source in a Mapping Set no longer has a field, just drop the mapping and warn the user.