To transfer any data from a data source to an Ingest instance's storage, the first order of business is naturally to establish what tables and columns are available in your data source and which of these tables you could be interested in transferring.
The overall term for this process in TimeXtender Data Integration (TDI) is synchronizing metadata, and this can be divided into three steps:
- Importing metadata: Extracting a one-to-one copy of the metadata from the data source and storing it in the Ingest instance storage as a cache.
- Synchronizing metadata: Reconciling the imported metadata cache with the working copy in the Ingest instance to handle schema drift.
- Selecting tables: Curating the list of tables available for selection in Transfer tasks.
While this sounds fairly simple, the process involves keeping three versions of the metadata - the data source's original, the Ingest instance's cache, and the Ingest instance's working copy - in sync. This provides flexibility and robustness in the overall process and prevents changes in the Ingestion layer from impacting the Prepare layer.
Note: In TimeXtender Data Integration 6766.1 and older, you would run or schedule a synchronization and it would automatically update the Ingest instance's metadata about the data source. The drawback of this approach was that schema drift, e.g. a rename of a table, could cause tables to unexpectedly disappear.
Metadata import and table selection
When you first add a new data source in TDI, you will be prompted to import metadata and select tables. In addition to that, a task called 'Import Metadata' is automatically added to the data source. If you want to refresh metadata later, you can run or schedule this task. See Tasks in an Ingest Instance for more information.
Metadata synchronization
When you've imported metadata from the data source, it's time to select tables and, after the first run, resolve any differences between the Ingest instance’s cache and the working copy of the metadata. For this, you use the aptly named Metadata Manager.
Note: The Metadata Manager replaces the Data Source Explorer in TimeXtender Data Integration 6766.1 and older
In the Metadata Manager window, tables are listed to the left, while columns/fields in the currently selected table are listed to the right. Search and filtering options are available for both tables and columns.
Synchronizing metadata and selecting tables
To synchronize the Ingest instance's cache and working copy, follow the steps below:
- Right-click the data source in TDI and click Metadata Manager.
- Click Synchronize Metadata.
- When synchronization has completed, you can review the changes in the Status column:
- Blank: No changes detected or the table has been edited (see below) which clears the status.
- Green check: Changes detected, but fixed automatically.
- Yellow: Changes detected that require review.
- Red: The application logic couldn't map the table or column in the working copy to a table or field in the cache. Manual remapping is required.
- For each table that needs review, click the table and click Edit. In the window that opens, you have the following option:
- Click the Data Source Table list and click a table to map the table you're editing to another table in the data source. This is useful if names have changed in the data source in a way that could not be automatically fixed by the application logic.
- Under Fields, click the Data Source Table list and click a field name to map that field to another field on the data source table.
- Review the changes detected in the box next to the Fields list.
- In the Included column, select the check box for all tables you want to make available for transfer. The table's columns are all selected by default, but you can unselect a column by clearing the check box in the Included column.
- When you are ready to commit your changes, click OK.
- In the Review your selections window, clear the checkbox for any changes that you do not want to commit, then click Commit. This updates the working copy of the metadata with the selected changes.
Preview a table
To preview the content of a table in the Query Tool:
- Select the table in the Tables list, then click Preview
