Skip to main content
SUBMITTED

Data directly from Data lake to MDW (without ODX)

  • April 6, 2023
  • 4 replies
  • 142 views

daniel
TimeXtender Xpert
Forum|alt.badge.img+8

Dear community,

For my client i'm looking into the option if you can pull data directly into your MDW from an Azure data lake. The data is already being extracted and cleaned through different tools. I need to get this cleaned data and push it towards the MDW. To me it would be a waste of resources and time to first load the data (again) in the ODX and then be able to get it to my MDW.

If seen this idea and that would be great: 

 

Is there a possibility?

Thank you!

4 replies

rory.smith
TimeXtender Xpert
Forum|alt.badge.img+8
  • TimeXtender Xpert
  • April 6, 2023

Until that functionality is implemented there is no direct way of doing this. The closest you can get is by using Synapse and defining views on top of your lake to do this. There are some other technical solutions, but no way that I know of of doing this natively in TimeXtender.


rogier.helmus
Problem Solver
Forum|alt.badge.img+1
  • Problem Solver
  • July 17, 2023

I’d like to see this functionality as well. I’ve described an example delta lake structure that it should be able to read in the following post: 

Summarizing the requirements:

  1. Should be able deal with table partitions
  2. Should be able to deal with incremental loads (multiple parquet files)
  3. Should be able to deal with requirement 1 and 2 combined for 1 table

peter.jensen
TimeXtender Xpert
Forum|alt.badge.img+3
  • TimeXtender Xpert
  • July 17, 2023

Hi Daniel,
as Rory mentioned, you can setup Synapse serverless views on top of the datalake data.
In the MDW you can refer to the Synapse serverless DB as an external SQL connection and load the views. (this definitely works in TX 20.10 - not sure about TX 21)


rory.smith
TimeXtender Xpert
Forum|alt.badge.img+8
  • TimeXtender Xpert
  • July 18, 2023

Hi @peter.jensen ,

the new release does not support external SQL connections unfortunately - this is another “virtual” X layer capability that would be great to see implemented.