We wonder if someone of you already ran into this issue, and perhaps have an idea on how to solve it ?
TX 20.10.38 using Azure SQL DB for DWH and ADLS gen2 for ODX Server. ADF used to load data from ODX Server into DSA. All databases have private endpoint, so SelfHosted Integration Runtime is used by ADF.
From time to time (random - not always on the same tables - for both incremental and full load tables) we notice that the ADF pipeline takes the normal amount of time to load data from ODX to DSA R-table. However the cleansing takes 0 (zero) seconds and the record count log reports 0 (zero) records in the R-table. ADF nor TimeXtender Execution report an error.
In the ODX Service Log we often notice this error : Failed to encrypt sub-resource payload… (not on exact the same moment as the execution - so not sure it has something to do with the issue)
Work-around : ADO.net
We had a simular issue last year and started using ADO.net to load data from ODX Server into DSA. We didn’t have the issue on that environment anymore, but recently we noticed memory pressure on the TX VM running the ODX Server service (and even had some Out of Memory errors when loading huge parquet files) - so we decided to switch back to ADF…
Any ideas ?
Best answer by Christian HauggaardView original