This article describes how to copy and synchronize instances, in order to implement a simple multiple environment setup. Multiple environments can be implemented using one ODX, as well as multiple ODXs. This article covers both of these scenarios.
Multiple Environments using One ODX
If there is only one ODX, it will typically rely on Production data sources. Let us assume that there is a requirement for three environments: Development (DEV), Quality Assurance (QA) and Production (PRD). Each environment has one Data Warehouse (DW) instance, and one Semantic Model instance. The DW instances in each environment all source data from the same ODX, which typically contains Production data. In order to promote changes between the environments, we have to promote changes for both the DW instance and the semantic model instance. Assuming that changes have been made to both the DW and the semantic model, a promote from DEV to QA would require copying the model of the following instances:
- Copying the model of the DEV DW instance to the QA DW instance, marked Step 1 in below diagram.
- Copying the model of the DEV Semantic Model instance to the QA Semantic Model instance, marked Step 2 in below diagram.
A promote from QA to PRD would require copying the model of the following instances:
- Copying the model of the QA DW instance to the PRD DW instance, marked Step 3 in below diagram.
- Copying the model of the QA Semantic Model instance to the PRD Semantic Model instance, marked Step 4 in below diagram.
Multiple Environments using Multiple ODXs
Several ODXs are necessary if there are multiple environments for source systems, and there is a requirement to align promotes within source system and promotes in their data estate. For example, say we have an ERP system with 3 environments - DEV, QA and PRD. A new custom field is added in the DEV ERP system. This new custom field can then be brought into our DEV DW and DEV Semantic Model. The promote of the custom field in the QA ERP system, will occur at the same time as the promote of the DW and the Semantic Model to QA.
In order to promote changes between the environments, we have to promote changes for both the DW instance and the semantic model instance. Since there are multiple ODXs, we also need to promote changes in ODX instances. New data sources that have been mapped to the DEV ODX instance in the TimeXtender portal, also need to be added and mapped to your QA ODX instance. After the new data sources have been mapped to the respective ODX instance, the DW and the semantic model are ready to be promoted. To summarize, a complete promote from DEV to QA would require the following steps:
- Add new data sources and map them to the QA ODX instance in the TimeXtender portal. Copy the DEV ODX instance to QA ODX instance, marked Step 1 in below diagram.
- Copying the model of the DEV DW instance to the QA DW instance, marked Step 2 in below diagram.
- Copying the model of the DEV Semantic Model instance to the QA Semantic Model instance, marked Step 3 in below diagram.
- Add new data sources and map them to the PRD ODX instance in the TimeXtender portal. Copy the QA ODX instance to PRD ODX instance, marked Step 4 in below diagram.
- Copying the model of the DEV DW instance to the QA DW instance, marked Step 5 in below diagram.
- Copying the model of the DEV Semantic Model instance to the QA Semantic Model instance, marked Step 6 in below diagram.
Pre-requisites for a Multiple ODX Architecture
Running multiple ODX instances, requires multiple ODX server services. A VM can only run one ODX server service at a given point in time. Therefore multiple VMs are required to run multiple ODX services, in order to support a multi-ODX architecture. Furthermore, if a user is to access several ODX instances from within the TimeXtender desktop application, inbound security rules need to be configured on each VM for the ports specified in the ODX instances within the portal. For more information, review the firewall section in the Add an ODX Instance article.
How to Copy and Promote an Instance
Say you have the following setup with 3 environments (DEV, QA and PRD) and there is an ODX dedicated to each environment, giving us 9 instances in total. All instances have been created within the portal. Storage has been created for all the DEV instances from within the TimeXtender Desktop Application. However, storage has not yet been created for the QA and PRD instances. Also, the QA and PRD instances have never been opened within the TimeXtender Desktop Application.
Copying and Promoting an ODX Instance
Let us assume that some changes have been made to the Development ODX, which we now want to promote to our QA ODX. For example we might have made new column or table selections, or created new transfer tasks, Query Tables, Primary Key rules or Incremental Selection rules. In order to complete the promote we need to follow the below steps:
- Right-click on the ODX source instance in the left-hand pane which you want to promote (i.e. ODX_DEV) and select Copy to instance...
- Select a Destination instance, in our example it is the QA ODX, in the following menu and click OK.
- Restart the ODX service on the machine running the service for the destination ODX instance, using the ODX Configuration tool.
- Open the Destination ODX instance.
- Right-click on the data source and Edit the data sources one by one.
- You should see that the connection is not found, this is because we have to remap the connection to a corresponding data source that has been defined in the destination instance in the portal. In our example, we would like to use the AW_QA data source, which has been mapped to our QA ODX instance in the portal, instead of the AW_DEV data source, which has been mapped to our DEV ODX instance in the portal. Select Change and then choose the corresponding data source connection.
- Right-click on the destination instance, and select Edit instance and then Create Storage. Then execute the synchronization and transfer task to move data into storage.
For an overview of how to copy an ODX instance click on the below video
Copying and Promoting a Data Warehouse Instance
Let us assume that some changes have been made to the Development Data Warehouse, which we now want to promote to our QA instance. In order to complete the promote we need to follow the below steps:
- Right-click on Data Warehouse source instance in the left-hand pane which you want to promote (i.e. DW_DEV) and select Copy to instance...
- Then select the destination instance you would like to copy to. Select OK.
- If we open the QA data warehouse instance, we can see that our tables have now been promoted. The next step is to change our mappings for the QA data warehouse instance. As of now, the tables in the QA data warehouse are pointing to our DEV ODX.
- To change our mappings for the QA data warehouse instance from DEV ODX to QA ODX, right-click on the QA data warehouse instance and select Synchronize with ODX and select the current ODX that the instance is mapped to. Alternatively, make sure that the QA data warehouse instance is open, and right-click on the DEV ODX and under Advanced select Synchronize Objects with Remapping.
- We need to remap each table so that they point to the DEV ODX. Select the first table to remap, by clicking Remap Table...
- Select the QA ODX from the dropdown and select the data source you would like to remap to this specific table. Click Search and select the table. Click OK. Repeat the process to remap the other tables.
- The tables in the QA data warehouse are now mapped to the QA ODX instead of the DEV ODX.
- However if we try to preview it, we get a warning that the table has not been deployed yet. We need to deploy and execute the instance in order to ensure that our tables are created and the data is populated in the QA instance. Before we deploy and execute, we need to create the data storage for the QA data warehouse instance, by right-clicking on DW_QA and selecting Edit instance and then Create Storage... Now we are ready to deploy and execute, right-click on our QA data warehouse and select Deploy and Execute.
Copying and Promoting a Semantic Model Instance
- In order to promote our semantic model from its DEV instance to its QA instance, right-click on the DEV semantic model instance in the left-hand pane and select Copy to instance...
- Then select the QA semantic model instance.
- Open the QA Semantic model instance, and right-click on the instance name and select Synchronize with remapping.
- In the synchronization window, you can see that the QA semantic model is mapped to the Development Data Warehouse
- Click on the instance column and select the Data Warehouse instance, from the dropdown, that you would like to remap to. Since we would like the QA Semantic Model instance to point to the QA Data Warehouse instance, we choose the QA Data Warehouse instance. Ensure that the Data Area, Schema, Table and Field columns are also correct. If you have further tables that need remapping scroll down and select the instance to remap to. Then click OK.
- Deploy and execute the QA semantic model instance to deploy the changes and populate the data.
Troubleshooting Copying an Instance
- If you attempt to copy an instance (e.g. DW QA) and you cannot find the instance you wish to copy to in the dropdown list (e.g. you only see DW DEV but not DW PRD). This means that the instance you are trying to copy to has already been opened within TimeXtender, and therefore you cannot choose to copy to it. Therefore, it is important when setting up the instances that you start copying to destination instances (e.g. QA and PRD) before opening them. Similarly if you have already opened all possible destination instances, before trying to copy to an instance, you will get the following error message. In order to solve this problem, delete the existing destination instance in the portal and recreate it, and copy to it prior to opening it in TimeXtender.
- If you have an instance you would like to copy and if you try to right-click on it and do not see the Copy to instance... option, this means that the instance cannot yet be copied because the source instance has not been opened in TimeXtender. In order to copy the source instance, you need to open it first.
- If you are copying an ODX instance, you have to shut down the destination instance before copying to it. This can be done, by right-clicking on the destination instance and selecting Close.
- If you try to execute the synchronize or transfer task after copying an ODX instance and you receive an error similar to the following, it is because you have not yet remapped the connection for the copied data sources. Right-click on the data sources and select Edit Data Source and select Change to remap the connection.
Troubleshooting Synchronize Objects with Remapping
If you try to remap a table which has unmapped fields, for example if the table in the QA ODX is missing fields that are in the Development ODX, then the table will be ignored. In other words, the data warehouse for this table (and all its fields) will be the same as before. All fields have to be present in the ODX you are remapping to, in order for the table to be remapped. To solve this issue, add the field to the QA ODX data source, or delete the field from the Data Warehouse instance.