Skip to main content
To homepage To homepage

hi, 

Lately were are getting alot of 503 exceptions. 

 

Error message is typically: 

The task '983693' failed:
System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 503 (The server is busy.).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 

We have reason to beleive that the source exception is ODX in general and outbound task in particular. 

Our setup consist of DWH located on Azure (Azure SQL database on Elastic pool), TX VM also on Azure VM, 1 ODX located on PRD VM and ODX storage on ADLS. TX version: 20.10.43.64

Looking at the logs of elastic pool, projectRepository databases, TX VMs shows no indications of pressure points which might exert 503 

 

Help would be highly appericiated

 

 

Hi @dsvartzman 

Do you get these on Transfer task executions?

What do you get in the services log?

Hi Thomas, 

 

Yes. Outbound task execution to be particular. 

Service logs shows the same error (i.e. DWH execution fails due to ODX transfer error)

Service logs: 

error message: 

The task '983715' failed:
System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 503 (The server is busy.).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.API.ADLSGen2.PathAPI.<GetStatus>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.API.ADLSGen2.PathAPI.<PathExists>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__17.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.WaitAll(Taskt] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2Helper.RunAsyncTaskaT](Task`1 task)
   at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass6_0.<ExecuteTable>b__0(DiscoveryHubSQLExecutionMethod transferMethod)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
   at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DataSourceMetaStructure sourceStructureModel, DataStorageModel storageModel, DataSourceModel dataSourceModel, DiscoveryHubSQLExecutionModel transferModel)
   at ExecutionEngine.Action.OutboundTransferAction.Execute()
   at ExecutionEngine.Action.ExecutionAction.<.ctor>b__9_0()
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 503 (The server is busy.).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.API.ADLSGen2.PathAPI.<GetStatus>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.API.ADLSGen2.PathAPI.<PathExists>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__17.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.WaitAll(Taski] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2Helper.RunAsyncTaskaT](Task`1 task)
   at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass6_0.<ExecuteTable>b__0(DiscoveryHubSQLExecutionMethod transferMethod)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
   at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DataSourceMetaStructure sourceStructureModel, DataStorageModel storageModel, DataSourceModel dataSourceModel, DiscoveryHubSQLExecutionModel transferModel)
   at ExecutionEngine.Action.OutboundTransferAction.Execute()
   at ExecutionEngine.Action.ExecutionAction.<.ctor>b__9_0()
   at System.Threading.Tasks.Task.Execute()
---> (Inner Exception #0) System.Net.Http.HttpRequestException: Response status code does not indicate success: 503 (The server is busy.).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.API.ADLSGen2.PathAPI.<GetStatus>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.API.ADLSGen2.PathAPI.<PathExists>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__17.MoveNext()<---
<---
 

Hi @dror.svartzman 

Could you please provide an update on Thomas’ questions above?

Hi Christian, 

We are 28 days further. Our initial diagnostics was correct. We have a single ODX which serves both dev and prd enviroments. ODX is highly sensitive for pressure both internally (VM) and externally (in-outbound transfers, maintenance). Rescheduling ODX task schedule did the trick. 

 

At this point ODX has no alert or proper monitoring mechanism. It would be nice if you could get the same gantt-like overview so you can visualize the pressure point on the ODX server

 

@dror.svartzman thanks for confirming, and thank you for providing feedback. Please feel free to post this feedback as a product idea 

Reply


Terms and ConditionsCookie settings