Skip to main content
To homepage To homepage

Hi,

We are trying to connect to several CSV files stored in a local folder. While we can successfully synchronize the data source and perform a full load in the ODX, we encounter an error when attempting to add the table to our data area (DSA).

The issue lies in the path to the Parquet file stored in Azure. The correct path should be:

CSV_DNB/csv_*/DATA_2024_11_28__11_09_50_2219585/DATA/DATA_0000.parquet

However, the path Timextender is looking for is:

CSV_DNB/csv_^*/DATA_2024_11_28__11_09_50_2219585/DATA/DATA_0000.parquet

It seems that Timextender is misinterpreting the automatically generated name and adds a ^ character.

I also attempted to use a specific file aggregation pattern, such as;
H100.*.csv
(all files in folder have the prefix H100 followed by a random number). However, I encountered the same error. Is there a way to specify the name of the table generated in the ODX? It seems like the “File aggregation pattern” is the issue. Do you have any idea how to fix this? 

 

-Execute Execution Package Execution Queue Default 'Failed'
    System.AggregateException: Pipeline execution failed ---> System.Exception: {
      "errorCode": "ActionFailed",
      "message": "Activity failed because an inner activity failed; Inner activity name: TIMEXTENDER COPY DATA, Error: ErrorCode=UserErrorFileNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'xxx'. FileSystem: 'containerodx'. Path: 'CSV_DNB/csv_^*/DATA_2024_11_28__11_09_50_2219585/DATA/DATA_0000.parquet'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'

 

 

Hi ​@annfoss 

I do it like this.

 

I would use H100*.csv.

Did you try that and are you updated to version 22.0.0.1 for your CSV provider?

Hi ​@Thomas Lind  Yes we are on version 22.0.0.1 for our CSV provider. I tried with: H100*.csv but I got the same error unfortunately:

 

Path: 'CSV_DNB/csv_H100^*/DATA_2024_11_29__07_45_31_0797599/DATA/DATA_0000.parquet'. ErrorCode: 'PathNotFound'.

 

It still adds the: ^

Hi ​@annfoss 

I am unable to reproduce the issue you describe

I have 3 csv files

These are my datasource settings

 

I execute the import metadata task and synchronize

and then execute the transfer task and bring into my prepare instance

I am using Ingest instance with data lake storage and prepare instance with Azure SQL db storage and ADF transfer from Ingest instance to Prepare instance

Do you notice any differences in our setup?

I am using version 22.1.0.0 of the provider, can you perhaps try upgrading, just to rule this out?

Hi ​@Christian Hauggaard we have the same setup as you have. The only difference is that we are on version 6766.1, so we don’t have the import metadata task. Do we perhaps need to upgrade?

Some additional information: We are able to do a Full load in the ODX and preview the data. We also get a parquet file in the data lake storage. 

​@annfoss please try to upgrade TDI and the data source provider to the latest version, please let me know if you still encounter the issue after upgrading

​@Christian Hauggaard we tried to upgrade to 6822.1, but I am sorry to let you know that we still encounter the same error: 

 

Path:'CSV_DNB/csv_^*/DATA_2024_12_04__14_03_26_1213665/DATA/DATA_0000.parquet'ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'

​@annfoss I have created a support ticket for this issue

Reply


Terms and ConditionsCookie settings