Follow

Data Storage

Data Storage

The ODX Server supports two types of storage: A SQL Server database (on-prem on in Azure) or an Azure Data Lake (Gen1 or Gen2)

Since an ODX Server has one storage at the time, you will need to delete the existing storage to switch storage types. Deleting a data storage, deletes it from the ODX Server, not the SQL Server or Azure.

Adding A SQL Server Data Storage

To add a SQL Server data storage, follow the steps below.

  1. Open you ODX server in a tab and, right click the aptly named Right click to add Data Storage node and click Add SQL Server Data Storage...
  2. In the Name box, type the name you want to use for the storage.
  3. (Optional) In the Description box, type a description of the data storage.
  4. In the Server box, enter the address of the SQL Server you will be using.
  5. In the Authentication list, click the mode of authentication you want to use. You have the following options:
    • Windows Authentication: Use the logged-in Windows user's credentials for authentication.
    • SQL Server Authentication: Use a login set up on the SQL Server. Enter the username and password in the corresponding fields.
    • Azure AD Password Authentication: Use Azure AD credentials from a domain that is not federated with Azure AD. Enter the username and password in the corresponding fields.
    • Azure AD Integrated Authentication: Use the logged-in Windows user's credentials for authentication, provided that he is logged in using Azure AD credentials from a domain that is federated with Azure AD.
  6. In the Database box, type the name of the database.
    If you want to create a new database, make sure the authentication options below are correct and then type a name for the database and click Create. The Create Database window opens.
    1. In the Collation list, click on the collation you want to use.
    2. In the Recovery model list, click on the recovery model you want the database to use. Then click OK.
  7. (Optional) In the Command timeout box, enter a timeout for the commands you run on the storage.
  8. (Optional) In the Connection timeout box, enter a timeout for connecting to the storage.
  9. If you plan to use Azure AD users in your security roles, select Use Azure settings and enter the required options:
    • In the Tenant ID box, enter the tenant ID GUID from Azure.
    • In the Application ID box, enter the application ID GUID from Azure.
    • In the Application key box, enter the application key from Azure.
  10. (Optional) Enter any addition connections settings in the Additional connection properties box.
  11. Click OK to add the storage.

Adding a Azure Data Lake Gen2 Data Storage

When you add an Azure Data Lake Gen2 data storage, you have the option to add connection information for Azure Databricks as well. This enables the following features:

  • You to use incremental load with handling of records that have been modified or deleted in the source ("updates and deletes"). Without Azure Databricks, regular incremental load without updates and deletes is still possible.
  • Incremental load and application of selection rules when you transfer data from the data storage to a data warehouse.
  • Direct transfer from the data storage to the data warehouse.

To add an Azure Data Lake Gen2 data storage , follow the steps below.

  1. Open you ODX server in a tab and, right click the aptly named Right click to add Data Storage node and click Add Azure Data Lake Gen2 Data Storage...

  2. In the Name box, type the name you want to use for the storage.
  3. (Optional) In the Description box, type a description of the data storage.
  4. In the Tenant ID box, enter the tenant ID GUID from Azure.
  5. In the Application ID box, enter the application ID GUID from Azure.
  6. In the Application key box, enter the application key from Azure.
  7. In the Account name box, enter the name of the storage account you want to use. Click Find... to filter and browse the accounts on your Azure tenant to find the account.
  8. In the Container name box, type the name of the container you want to use. Type the name and click Create to create a new container.
  9. (Optional) In the Databricks token box, enter the token needed to authenticate with Azure Databricks
  10. If you want to use Azure Databricks, select Use Azure Databricks and enter the required options:
    • In the Token box, enter the token needed to authenticate with Azure Databricks.
    • In the Cluster name box, type the name of the cluster you want to use or leave it at the default.
    • In the URL box, enter the URL of the Azure Databricks service you want to use or leave it at the default.
  11. Click OK to add the storage.

Adding a Azure Data Lake Data Storage Gen1

On a Azure Data Lake Gen1 data storage, Azure Databricks enables you to use incremental load with handling of records that have been modified or deleted in the source ("updates and deletes"). Without Azure Databricks, regular incremental load without updates and deletes is still possible.

To add an Azure Data Lake Gen1 data storage, follow the steps below.

  1. Open you ODX server in a tab and, right click the aptly named Right click to add Data Storage node and click Add Azure Data Lake Gen1 Data Storage...

  2. In the Name box, type the name you want to use for the storage.
  3. (Optional) In the Description box, type a description of the data storage.
  4. In the Tenant ID box, enter the tenant ID GUID from Azure.
  5. In the Application ID box, enter the application ID GUID from Azure.
  6. In the Application key box, enter the application key from Azure.
  7. In the Account name box, enter the name of the storage account you want to use. Click Find... to filter and browse the accounts on your Azure tenant to find the account.
  8. In the Folder Name box, type the name for the folder in the data lake that data should be stored in.
    Warning: If you change the folder name, a full load is required to repopulate the storage with data from the source.
  9. If you want to use Azure Databricks, select Use Azure Databricks and enter the required options:
    • In the Token box, enter the token needed to authenticate with Azure Databricks.
    • In the Cluster name box, type the name of the cluster you want to use or leave it at the default.
    • In the URL box, enter the URL of the Azure Databricks service you want to use or leave it at the default.
  10. Click OK to add the storage.
Was this article helpful?
0 out of 0 found this helpful

0 Comments

Please sign in to leave a comment.