Skip to main content
To homepage To homepage
Solved

Error 503 (ODX)

D
Contributor

hi, 

Lately were are getting alot of 503 exceptions. 

 

Error message is typically: 

The task '983693' failed:
System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 503 (The server is busy.).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 

We have reason to beleive that the source exception is ODX in general and outbound task in particular. 

Our setup consist of DWH located on Azure (Azure SQL database on Elastic pool), TX VM also on Azure VM, 1 ODX located on PRD VM and ODX storage on ADLS. TX version: 20.10.43.64

Looking at the logs of elastic pool, projectRepository databases, TX VMs shows no indications of pressure points which might exert 503 

 

Help would be highly appericiated

 

 

Best answer by dsvartzman

Hi Christian, 

We are 28 days further. Our initial diagnostics was correct. We have a single ODX which serves both dev and prd enviroments. ODX is highly sensitive for pressure both internally (VM) and externally (in-outbound transfers, maintenance). Rescheduling ODX task schedule did the trick. 

 

At this point ODX has no alert or proper monitoring mechanism. It would be nice if you could get the same gantt-like overview so you can visualize the pressure point on the ODX server

 

View original
Did this topic help you find an answer to your question?

5 replies

Thomas Lind
Community Manager
Forum|alt.badge.img+5
  • Thomas Lind
  • Community Manager
  • 1017 replies
  • November 29, 2023

Hi @dsvartzman 

Do you get these on Transfer task executions?

What do you get in the services log?

D
Contributor
  • dsvartzman
  • Author
  • Contributor
  • 19 replies
  • November 29, 2023

Hi Thomas, 

 

Yes. Outbound task execution to be particular. 

Service logs shows the same error (i.e. DWH execution fails due to ODX transfer error)

Service logs: 

error message: 

The task '983715' failed:
System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 503 (The server is busy.).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.API.ADLSGen2.PathAPI.<GetStatus>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.API.ADLSGen2.PathAPI.<PathExists>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__17.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2Helper.RunAsyncTask[T](Task`1 task)
   at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass6_0.<ExecuteTable>b__0(DiscoveryHubSQLExecutionMethod transferMethod)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
   at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DataSourceMetaStructure sourceStructureModel, DataStorageModel storageModel, DataSourceModel dataSourceModel, DiscoveryHubSQLExecutionModel transferModel)
   at ExecutionEngine.Action.OutboundTransferAction.Execute()
   at ExecutionEngine.Action.ExecutionAction.<.ctor>b__9_0()
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 503 (The server is busy.).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.API.ADLSGen2.PathAPI.<GetStatus>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.API.ADLSGen2.PathAPI.<PathExists>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__17.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2Helper.RunAsyncTask[T](Task`1 task)
   at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass6_0.<ExecuteTable>b__0(DiscoveryHubSQLExecutionMethod transferMethod)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
   at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DataSourceMetaStructure sourceStructureModel, DataStorageModel storageModel, DataSourceModel dataSourceModel, DiscoveryHubSQLExecutionModel transferModel)
   at ExecutionEngine.Action.OutboundTransferAction.Execute()
   at ExecutionEngine.Action.ExecutionAction.<.ctor>b__9_0()
   at System.Threading.Tasks.Task.Execute()
---> (Inner Exception #0) System.Net.Http.HttpRequestException: Response status code does not indicate success: 503 (The server is busy.).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.API.ADLSGen2.PathAPI.<GetStatus>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.API.ADLSGen2.PathAPI.<PathExists>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__17.MoveNext()<---
<---
 

Christian Hauggaard
Community Manager
Forum|alt.badge.img+5

Hi @dror.svartzman 

Could you please provide an update on Thomas’ questions above?

D
Contributor
  • dsvartzman
  • Author
  • Contributor
  • 19 replies
  • Answer
  • December 28, 2023

Hi Christian, 

We are 28 days further. Our initial diagnostics was correct. We have a single ODX which serves both dev and prd enviroments. ODX is highly sensitive for pressure both internally (VM) and externally (in-outbound transfers, maintenance). Rescheduling ODX task schedule did the trick. 

 

At this point ODX has no alert or proper monitoring mechanism. It would be nice if you could get the same gantt-like overview so you can visualize the pressure point on the ODX server

 

Christian Hauggaard
Community Manager
Forum|alt.badge.img+5

@dror.svartzman thanks for confirming, and thank you for providing feedback. Please feel free to post this feedback as a product idea 

Reply


Most helpful members this week

Terms and ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings