Tasks in an ODX Instance

In the ODX, operations which need scheduling are organized as tasks.  A data source may include three types of tasks:
Transfer: This task moves data from the data source to the data storage.

Synchronization: This task synchronizes the structure of the source with the meta data stored in the ODX instance. A synchronization task is added automatically when you add a new data source.

Storage management: This task performs clean-up and management tasks on the data storage. Execute a storage management task to delete old versions of data and move old versions of data to "cool" storage (applies Azure Data Lake Storage only). It also includes an option to rollup incremental data with Azure Data Factory (applies to Azure Data Lake only). When a data source is configured with frequent incremental loads without the occasional full load, the ODX may generate a lot of small files in data lake storage. The Rollup option offers to roll up (or consolidate) individual files into larger files, to increase load performance.

Adding a Task to a Data Source

To add a task to a data source

• Right click the data source, click Add Transfer Task or Add Synchronize Task or Add Storage Management Tasks and follow the instructions in the wizard.

To enter credentials for Azure Data Factory, refer to the ADF data source you have already configured in TimeXtender Portal. 

Editing a Task

To change settings for a task, follow the steps below.

1. Right click the task and click Edit Transfer Task or Edit Synchronize Task or Edit Storage Management Tasks.

2. In the Name and Description boxes, you can update the name and description of the task.

3. The other settings available depend on the task type.  Those are listed below:

Transfer task

Select Use incremental load when available to load data incrementally into the data storage when the task is executed. If no incremental load rules have been added to the tables copied by the tasks, this setting has no effect.
Note: This setting is not available in the Add transfer task wizard.

Storage management task

• Select Delete old versions to free up storage and enter the number of versions to keep in storage in the Versions to keep box to delete all but the specified amount of versions from storage when you execute the task.
• Select Move old versions to cool storage to save costs and enter the number of versions to keep in hot storage in the Versions to keep in hot storage box to move all but the specified amount of versions to cool storage when you execute the task.
  • Note: This setting applies to Azure Data Lake Storage only.
• Select Rollup incremental data with Azure Data Factory to specify minimum and maximum size of rollup file. To enter credentials for Azure Data Factory, refer to the ADF data source you have already configured in TimeXtender Portal. 
  • Note: This setting applies to Azure Data Lake only. 

Executing a Task

Executing a transfer task will transfer data from data source to data storage, respecting table selection on the data source and the task itself.

To execute a task

• Right click the task and click Execute. The status of the execution will displayed in parenthesis after the task name in the tree.

Selecting Tables

For transfer and storage management tasks, you can select which tables the task should process.

To select tables to process

• Right click the task a click Select tables...

See Table and Column Selection in an ODX Instance for more information.