Timeouts when transfer from azure datalake storage ingest instance to the prepare instance
when we load a table from the data lake it keeps failing after 4 till 5 hours.
we have tried different timeout settings but the execution from the datalake to the sql storage keeps failing.
See the screenshots and the error message below. In fact I have now 2 tickets open for a timeout issue. for 2 different problems.
Page 1 / 1
error message:
Exception Type: System.Net.Http.HttpRequestException Message: Error while copying content to a stream. Stack Trace: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Management.DataFactory.PipelineRunsOperations.<GetWithHttpMessagesAsync>d__6.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Management.DataFactory.PipelineRunsOperationsExtensions.<GetAsync>d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at ODX.DataFactory.DataFactoryResourceHelper.ExecutePipeline(Dictionary`2 pipelineParameters, DataFactoryConnectionModel dataFactoryConnection, String pipelineName) at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass5_0.<ExecuteTable>b__0(DiscoveryHubExecutionMethod transferMethod) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 461 at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action) at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DiscoveryHubSQLDefaultExecutionModel transferModel, DataSourceModel dataSourceModel, TableModel tableModel) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 463 at ExecutionEngine.Action.OutboundDefaultTransferAction.Execute() at ExecutionEngine.Action.ExecutionAction.<.ctor>b__11_0() at System.Threading.Tasks.Task.Execute() Inner Exception: Exception Type: System.IO.IOException Message: Received an unexpected EOF or 0 bytes from the transport stream. Stack Trace: at System.Net.ConnectStream.EndRead(IAsyncResult asyncResult) at System.Net.Http.HttpClientHandler.WebExceptionWrapperStream.EndRead(IAsyncResult asyncResult) at System.Net.Http.StreamToStreamCopy.BufferReadCallback(IAsyncResult ar)
Details:
Exception Type: System.Net.Http.HttpRequestException ... Module: timeXtender System.Exception at TimeXtender.DataManager.StepTransferExecute.DoDataWarehouse(IStepSetup stepSetup, List`1 mergeODXTransferInfromation, VariableResolveObject dynamicResolveObject) at TimeXtender.DataManager.StepTransferExecute.DoAbstractStep(IStepSetup stepSetup)
@jermaine.cotino I was the same problem, try with this:
HI @jermaine.cotino based on the support ticket, it appears you were able to resolve this by adding a data selection rule