Solved

Error 503 (ODX)

  • 29 November 2023
  • 4 replies
  • 54 views

hi, 

Lately were are getting alot of 503 exceptions. 

 

Error message is typically: 

The task '983693' failed:
System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 503 (The server is busy.).
   at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
 

We have reason to beleive that the source exception is ODX in general and outbound task in particular. 

Our setup consist of DWH located on Azure (Azure SQL database on Elastic pool), TX VM also on Azure VM, 1 ODX located on PRD VM and ODX storage on ADLS. TX version: 20.10.43.64

Looking at the logs of elastic pool, projectRepository databases, TX VMs shows no indications of pressure points which might exert 503 

 

Help would be highly appericiated

 

 

icon

Best answer by dsvartzman 28 December 2023, 08:50

View original

4 replies

Userlevel 6
Badge +5

Hi @dsvartzman 

Do you get these on Transfer task executions?

What do you get in the services log?

Userlevel 6
Badge +5

Hi @dror.svartzman 

Could you please provide an update on Thomas’ questions above?

Hi Christian, 

We are 28 days further. Our initial diagnostics was correct. We have a single ODX which serves both dev and prd enviroments. ODX is highly sensitive for pressure both internally (VM) and externally (in-outbound transfers, maintenance). Rescheduling ODX task schedule did the trick. 

 

At this point ODX has no alert or proper monitoring mechanism. It would be nice if you could get the same gantt-like overview so you can visualize the pressure point on the ODX server

 

Userlevel 6
Badge +5

@dror.svartzman thanks for confirming, and thank you for providing feedback. Please feel free to post this feedback as a product idea 

Reply