This guide will cover how to configure and manage the ODX server.
Complete the following steps to configure and manage the ODX Server:
- Install and Configure the ODX Server Service
- Connect to the ODX Server
- Add Data Storage
Structure of an ODX
The ODX server sits in the middle of it all and copies data from data sources to a data
storage, which can both be on-premise or in the cloud, and, when requested by Discovery
Hub, on to modern data warehouses.
For storing metadata about sources and storages as well as the tasks it needs to run, the
server uses a local backlog that is continuously synced with the cloud repository. In the
cloud repository, the ODX server connects to a project. Each ODX server has one project
open at a time and two ODX servers should never use the same project.
ODX is a client/server solution. The server runs as a service with no GUI. Once initial setup
has been completed, management and administration of the server happens through the client, the ODX manager. It is part of the Discovery Hub software package and resides on the
Discovery Hub server.
To get data from the ODX, you add the ODX server in Discovery Hub. You
can then search for data from all the sources in the ODX and drag-and-drop the tables to
your data warehouses. The ODX server will then copy the data from the data storage to the
data warehouse on execution.
1. Install and Configure the ODX Server Service
Installing the ODX Server Service
To install and configure the ODX Server, you will need the following:
- Download the ODX Server installation package. If you deployed Discovery Hub using the Azure Marketplace templates then it is already installed on your virtual machine and can be found at C:\Program Files\Discovery Hub\ODX Server.
- The client secret that is used to connect the ODX server to the cloud repository. Can be found in the TimeXtender User Portal at portal.timextender.com/odx
- A user account for running the ODX server service if you need to connect to SQL Servers using Active Directory authentication. The user account needs to have the appropriate access to these servers.
Configure the ODX Server Service
1. Open the ODX Server Configuration exe file. You can find it at “C:\Program Files\Discovery Hub\ODX Server”. This will open a wizard to help you configure the server and set up and start the Windows service.
2. In Step 1 of the wizard, you will be asked to set up any Proxy Server Settings if you choose to use a proxy server to connect to the internet.
3. In Step 2 of the Wizard you will be asked for your Client Secret. You can find this by going to portal.timextender.com and clicking on the ODX tab. Once you have entered in your client secret, click Next.
4. In Step 3 of the wizard, add a new project name and click next.
5. In Step 4 of the wizard you are asked to specify a port number and maximum concurrent execution tasks. We recommend that you leave these at their default unless you have a reason to customize them. Click next.
6. In Step 5 of the wizard you are asked to create the ODX management password and the Discovery Hub connection password. Enter in this information and click
7. In Step 6 of the wizard you can enter an encryption secret that is used when the encryption key is generated. Make sure to store the secret in a safe place since we cannot help you retrieve a lost secret. Enter in your encryption secret (16-32 characters) or use the system default encryption and click next.
8. In Step 7 of the wizard you can specify what account you want to run the service as if you need to connect to SQL Servers using ActiveDirectory authentication. The user account needs to have the appropriate access on these servers. Specify the account or use the Local System Account and click next.
9. In Step 8 of the wizard you can save your settings and start the ODX Server Service. Click Save.
2. Connect to the ODX Server
Connecting to the ODX server in Discovery Hub allows you to manage the ODX. This is where you can add data storage, add data sources, and schedule executions and synchronize data sources. To copy data from a data source to the data storage, you start by connecting to the ODX Server. You then add a data storage and the data source. When this is done, you add a task to transfer data from source to storage. You can schedule executions of the task to have data copied automatically.
Before connecting to the ODX Server in Discovery Hub, ensure the ODX Server Service is currently running.
1. In the Tools menu, click Options, then click the ODX Server tab in the window that appears. Please note, if you just started the ODX Server service it can take up to 20 minutes for the server to initialize. You will be unable to set up the ODX Server in Discovery Hub until it has initialized.
2. In the Name box, type a name for the ODX server.
3. In the Server box, enter the ODX server's address. If it is installed locally, use localhost
4. In the Port box, enter the port number if it is different from the default.
5. In the Password box, enter the password for managing the ODX.
6. Click OK. The ODX server is now listed in the Solution Explorer in Discovery Hub
**The ODX server you add by following the steps above is your primary ODX server. If you open a project that uses data from another ODX server, that server will be displayed in the Solution Explorer in Discovery Hub as a secondary ODX server.
3. Add Data Storage
The ODX Server only supports one storage option at a time, either a SQL Server database or an Azure Data Lake.
Adding SQL Server Storage
1. In the Manage ODX Servers window, right-click an ODX Server and click Add SQL Server Data Storage.
Name: Type the name you want to use for the storage
Server: enter the address of the SQL server you will be using.
Database: enter the name of the database. If you want to create a new database, make sure the authentication options below are correct and then type a name for the database and click Create.
Select authentication (Windows or SQL Server). Please note, SQL Server Authentication is required when using an Azure SQL Database.
Click OK to have the storage option added
Adding Azure Data Lake Gen1/Gen2 Storage
In order to access the data lake resources from Discovery Hub, you will need to register an application in the Azure portal. After registering an application in Azure, you will have all of the necessary settings required to add your data lake resource in Discovery Hub.
1. In the Azure Portal, click on Azure Active Directory in the left column, then click on App Registrations in the menu bar on the left. Then click New Application Registration
2. Enter a name for your app and select Accounts in this organizational directory only. You do not need to enter a Redirect URI. Click Register when you are done.
3. Go to your newly created App and go to API Permissions Settings and click Add a Permission.
4. Find Azure Data Lake and select it. Next, under Select Permission check the box that says Have full access to the Azure Data Lake Service
5. Click Done to save your changes and go back Certificates & secrets to create a New Client Secret. This key is encrypted after save so the application key needs to be documented somewhere safe. The secret will appear after you click Add.
6. Go back to the resource group and select the Data Lake Storage resource (If you are using ADLS Gen2, this will be the storage account with data lake storage containers). In the menu bar on the left, select Access Control (IAM) and add a role assignment.
- Add the app you just created to the role of Owner of the resource.
- Add the app you just created to the role of Storage Blob Data Contributor of the resource. **This role is only required for ADLS Gen2 storage**
Add Azure Data Lake Storage to the ODX in Discovery Hub
- In the ODX Server tab, right-click an ODX Server and select either Add Azure Data Lake Gen1 Storage or Add Azure Data Lake Gen2 Storage depending on the type of storage account you configured.
Name: Type the name you want to use for storage
Tenant ID: This is the [Directory ID] found under properties of Azure Active Directory.
Application ID: Use the application you registered previously. The ID can be found under the registered app.
Application Key: Use the key you created under the application.
Account Name: Use the name of your Data Lake Store. The input only needs the name of the resource instead of the entire URL.
Folder/Container name: For ADLS Gen1, enter the name of the folder you want to create. This folder doesn’t have to be created in Data Lake Store in advance. The root folder is always Operational data eXchange and this folder name will be under that. For ADLS Gen2, enter the name of the container you want to create.
(Optional) To add a Databricks account, check the box next to Use Azure Databricks.
Token: Enter the token needed to authenticate with Azure Databricks. Click here to see how to generate a token.
Cluster Name: Enter the name of the cluster you want to use or leave it as the default.
URL: Enter the URL of the Azure Databricks service you want to use or leave it as the default.
**Adding a Databricks account enables the following features:
- The option to handle updated and deleted records on incremental load into the data lake
-Incremental load and selection rules when transferring data for data storage to data warehouse and significantly better performance.
-Direct transfer from data lake to data warehouse
For additional information and to see how to add and synchronize data sources, please view the User Guide