Relates to TimeXtender 6024.1 and later versions. The rollup storage management feature was released in TimeXtender 6024.1
In the ODX, operations which need scheduling are organized as tasks. A data source may include three types of tasks:
Transfer: This task moves data from the data source to the data storage.
Synchronization: This task synchronizes the structure of the source with the meta data stored in the ODX instance. A synchronization task is added automatically when you add a new data source.
Storage management: This task performs clean-up and management tasks on the data storage. Execute a storage management task to delete old versions of data and move old versions of data to "cool" storage (applies Azure Data Lake Storage only). It also includes an option to rollup incremental data with Azure Data Factory (applies to Azure Data Lake only). When a data source is configured with frequent incremental loads without the occasional full load, the ODX may generate a lot of small files in data lake storage. The Rollup option offers to roll up (or consolidate) individual files into larger files, to increase load performance.
Adding a Task to a Data Source
To add a task to a data source
- Right click the data source, click Add Transfer Task or Add Synchronize Task or Add Storage Management Tasks and follow the instructions in the wizard.
Note: For Storage management task option Rollup incremental data with Azure Data Factory, specify minimum and maximum size of rollup file.
To enter credentials for Azure Data Factory, refer to the ADF data source you have already configured in TimeXtender Portal.
Editing a Task
To change settings for a task, follow the steps below.
1. Right click the task and click Edit Transfer Task or Edit Synchronize Task or Edit Storage Management Tasks.
2. In the Name and Description boxes, you can update the name and description of the task.
3. The other settings available depend on the task type. Those are listed below:
Select Use incremental load when available to load data incrementally into the data storage when the task is executed. If no incremental load rules have been added to the tables copied by the tasks, this setting has no effect.
Note: This setting is not available in the Add transfer task wizard.
Storage management task
- Select Delete old versions to free up storage and enter the number of versions to keep in storage in the Versions to keep box to delete all but the specified amount of versions from storage when you execute the task.
- Select Move old versions to cool storage to save costs and enter the number of versions to keep in hot storage in the Versions to keep in hot storage box to move all but the specified amount of versions to cool storage when you execute the task.
- Note: This setting applies to Azure Data Lake Storage only.
- Select Rollup incremental data with Azure Data Factory to specify minimum and maximum size of rollup file. To enter credentials for Azure Data Factory, refer to the ADF data source you have already configured in TimeXtender Portal.
- Note: This setting applies to Azure Data Lake only.
Executing a Task
Executing a transfer task will transfer data from data source to data storage, respecting table selection on the data source and the task itself.
To execute a task
- Right click the task and click Execute. The status of the execution will displayed in parenthesis after the task name in the tree.
Scheduling a Task
You can schedule at task to have it execute at predefined times and intervals.
To schedule a n execution of a task, follow the steps below.
- Right click the task and click Manage Task Schedule. The Manage Task Schedule window appears.
- Click Add to add a new item to the schedule.
- In the Start time box, enter the time where the execution should begin.
- Click Run once on this day and enter a date if you want to schedule the task to run just once on a specific day.
- Click Recurring to schedule the task to run a regular intervals. The options are Monday-Friday and the first and last day of a month.
- Under repeat, click Start task again and enter a minute count and end time if you want the task to run repeatedly after first starting.
For transfer and storage management tasks, you can select which tables the task should process.
To select tables to process
- Right click the task a click Select tables...
See Table and Column Selection in an ODX Instance for more information.