This guide will cover how to create and add Azure Data Lake storage for the ODX in Discovery Hub.
STOP! If you are creating a Discovery Hub Environment from scratch, we highly recommend using one of the supported configuration options for deploying Discovery Hub in Azure.
If you have already deployed one of the Azure Marketplace templates with Azure Data Lake then you already have all of the necessary data lake resources and can skip to Step 3 to register an application.
If you already have an existing Discovery Hub deployed in Azure, but don't have the necessary Azure Data Lake services, you can use this guide to add a Data Lake storage option by starting at Step 1.
Complete the following steps to create Azure Data Lake Storage for the ODX in Discovery Hub:
- Add Data Lake Store
- Add Data Lake Analytics
- Register Application
- Assign Application Role
- Add Azure Data Lake Storage in Discovery Hub
1. Add Data Lake Store
Note: If you already have a Data Lake Storage account you can skip this step.
1. In Azure portal, create a Data Lake Storage account that will be used to host ODX database.
2. Add Data Lake Analytics
Note: If you already have a Data Lake Analytics account you can skip this step.
1. In Azure portal, create a Data Lake Analytics account and associate the analytics service to the previously created Data Lake Store.
3. Register Application
In order to access data lake services, you need to register an application.
1. Go to the Azure portal and click on Azure Active Directory in the left column.
2. Click on App Registrations in the menu bar on the left and click New Application Registration
4. Choose a new for your application, and select Web app/ API as its Type. The value of Sign-on URL is the URL at which your application is hosted. Click OK.
5. Go to Settings > Required Permissions > Add > Select an API
6. Find Azure Data Lake and select it. Next, under Select Permission check the box that says Have full access to the Azure Data Lake Service.
7. Click Done to save your changes and go back Settings > Keys to create a new customer key. This key is encrypted after save so the application key needs to be documented somewhere safe. The key value will appear after you click Save. This is the Application Key you will need when adding Azure Data Lake Storage for the ODX in Discovery Hub.
4. Assign Application Role
After the application registration is created, you need to go back to the previously created Data Lake Store and Data Lake Analytics resources to make the app an owner of the resources.
1. Go back to the resource group where your data lake resources are located and select the Data Lake Analytics resource. In the menu bar on the left select Access Control (IAM) and add a role assignment. Make the app you just created an owner of the resource.
Repeat this step for the Data Lake Storage Gen 1 resource in the resource group as well.
*You must be an OWNER of the resource to add an app as an owner.
5. Add Azure Data Lake Storage in Discovery Hub
After the configuration is completed in Azure portal, we can start to create an ODX Azure Data Lake Storage in Discovery Hub.
Name: Create a name for you Azure Data Lake Storage.
Account name: Use the name of your Data Lake Store. Note that the input only needs the name of the resource instead of the entire URL.
Folder name: Create a folder name. This folder doesn’t have to be created in Data Lake Store in advance. The root folder is always “Operational data eXchange” and this folder name will be under that.
Analytics account name: Use the name of your Data Lake Analytics. Note that the input only needs the name of the resource instead of the entire URL.
Tenant ID: This is the [Directory ID] found under properties of Azure Active Directory.
Application ID: Use the application you registered previously. The ID can be found easily under the registered app.
Application Key: Use the key you created under the application.
Analytics Units: You pay for Analytics Units (AU) time used on your Azure subscription. Adding more AUs can increase performance on Incremental Load.
With the information provided above, you should be able to create Azure Data Lake Storage successfully in Discovery Hub.