Use Microsoft Fabric Lakehouse as Ingest Instance storage, and ingest data sources into Microsoft Fabric OneLake Delta Paquet format.
Fabric Ingest instance storage is available as part of the Standard, Premium or Enterprise Package
A public preview of this feature is currently available. This public preview does not currently support:
- TimeXtender SAP Table data source
- Ingest Storage Management tasks
- Ingest Security Roles
- Transfer to Prepare instance with Snowflake storage
- ADF Transfer to Prepare instance
Prerequisites
- Create an App Registration in the Azure Portal - It is recommended to use a dedicated app registration to ensure this account is the only one with access to the client credentials
- Add a user that does not require multi-factor authentication (i.e. a non-MFA account) as Owner of the App Registration, in order to allow for unattended re-authentication
- Add a platform and select Mobile and desktop application and enter https://localhost as the custom redirect URI and click Configure and then Save
This will result in the following Redirect URIs being added automatically
- Navigate to Authentication settings for the App Registration. Set Allow public client flows to Yes
- In Fabric/Power BI Admin Portal, enable “allow service principals to use Power BI APIs” as described here, in order to grant the app registration access to the Fabric workspace.
- Create a workspace, or navigate to an existing workspace, in the Fabric portal and select Manage access. Grant the App Registration and the non-MFA account Member access to the Fabric workspace.
- Runtime version of the Fabric workspace needs to be set to 1.2. Navigate to your Fabric workspace and click Workspace settings, and under the Data Engineering/Science, click Spark settings and select Runtime version 1.2 and click Save.
- Enable the following setting for “users to access data stored in Onelake with apps external to Fabric”
Configure Ingest Instance for Fabric Lakehouse Storage
- Add a new Ingest instance and select the storage type Microsoft Fabric Storage
- Enter the workspace name for the existing Fabric workspace
- Provide a name for the Lakehouse
Note: You can connect to an existing Lakehouse that has been created directly in the Fabric Portal, or you can choose to create the Lakehouse within TimeXtender Data Integration (TDI).
- Enter the user name and password for the non-MFA user that was setup as an owner in the App Registration
- Enter the Tenant id for the tenant associated with Fabric
- Enter the Application id for the App Registration
- Enter the Application Key (i.e. the client secret value) associated with the App Registration
Create the Fabric Lakehouse Ingest Instance Storage
- Open the TimeXtender Data Integration (TDI) application and open the Ingest instance
- Right-click on the Ingest Instance in the Solution Explorer tab and select Authenticate, login with the non-MFA user and accept the permissions it wants to add in order to give your App Registration the correct Scopes
- If you haven't created the Lakehouse already, you can do it now by right-clicking on the Ingest Instance in the Solution Explorer and selecting Edit Instance, and then Create Storage
Objects created in Fabric
A notebook is created and run when a transfer task is executed for a data source in a Fabric Ingest instance. You can see these notebooks being created and run in within the Fabric portal under Monitor. The notebook creates a delta parquet table in the lakehouse (with the naming LoadTables_<data source name>_<uuid>), and creates a temporary parquet file in the lakehouse file folder. The data from the temporary parquet file is transferred to the delta parquet table in the lakehouse, and then the temporary parquet file is deleted. JSON files are also created in the lakehouse file folder to capture the data source metadata.
