Skip to main content
SUBMITTED

Be able to load data from an Azure Data Lake

  • November 30, 2018
  • 4 replies
  • 28 views

rory.smith
TimeXtender Xpert
Forum|alt.badge.img+8

We have a team using Azure Data Bricks to do processing and modelling on all sorts of sensors in office buildings, and they will store output in Azure Data Lake. We want to be able to ingest this data (at least the output) and make it available in DH.

Currently this is not natively possible in TX and would require something like Azure Data Factory to move files to a place accessible from TX. Could this be added?

4 replies

Forum|alt.badge.img
  • Contributor
  • November 30, 2018

Hi Rory,

Yes, I have that use-case on my list of future additions. We most likely have most of the codebase in place through our ODX Data Lake store functionality. 

I assume you are aware that DataBricks can push data directly into SQL Server tables, which would be my suggested workaround - or if it is a matter of few tables, build a Custom SSIS package on the MDW Layer to extract the output into Tables. 

 


  • Starter
  • November 21, 2019

Is there any progress on this?

Would like to connect directly to blob containers.


Don't know if it helps you, but the with the CData connectors, you're able to read data from an Azure blob (ADLSGen2)

(Tested it with the CData CSV)

nyQtEpc6122E74Byt0jSWA.png


This is weird, tried it again today and the options shown above are gone wjen you create a new datasource, but not when you add the source via the environment settings....????? It's the same version ?

1_XiwiJLH7nRWEUoijEjug.png

WR9x_Ire106XRr691gOw3w.png