Many customers have ‘stacked’ sources: many instances of the same source (BC / Dynamics, etc.) that in the TX Classic release can be easily handled with Additional Data Sources. Here you define the data selection in the ‘master’ source and have the same data extracted from the underlying clones.
In TDI you would currently need to have separate sources for each and use the, very restrictive, mapping set feature to auto-map tables with identical (field, data type) structure to the same Prepare table.
The possibility of cloning data sources and using connection strings / json settings, makes the creation of many of the same sources easier. There are still two issues remaining however:
- Each additional source needs the data selection set and maintained separately
- If one of many ‘stacked’ sources updates by adding or changing metadata, this will break mapping sets
The first issue could be solved by allowing to clone Metadata Manager settings from one ‘master’ source to secondary sources with the same structure. It should be possible to do this before the initial metadata sync task is automatically triggered. In this situation I would expect that the Metadata Manager shows you:
- What is successfully found in the target from the source mapping
- What was not found in the target from the source (maybe due to a schema name difference, etc.)
- What exists in the target additionally to what is in the source
If in future one of the sources updates to add a new field, it should be possible to clone this across easily once the other sources also get this field. I expect you would mark one source as the ‘master’ for its set, and others as receiving mapping changes.
The second problem occurs when there are deviations between data sources that cause Mapping Sets not to work. If one of the sources has an additional field in a table with respect to the others, it will not work. In my opinion a mapping set should be an intersection of common fields, not an exact match. This way new fields added won't break Mapping Sets already existing and the feature becomes more widely useful. If a source in a Mapping Set no longer has a field, just drop the mapping and warn the user.
