Skip to main content

Hi,

We dont always get the notification e-mails when execution fails. Do you know what could be causing?

Thanks.

Kerem 

do you see any errors or relevant messages in the event viewer after the package failure?


Does this happen when you execute ODX tasks? This because you will only receive (at the moment) a message when there is a critical error.


@Christian Hauggaard yes there are errors in execution history log. Last one was

"Transfer table failed
    
    Details:
    
    Transfer table failed
    Module: timeXtender
    System.Exception
       at TimeXtender.DataManager.StepTransferSSISExecute.DoDataWarehouse(StepSetup stepSetup, VariableResolveObject dynamicResolveObject)
       at TimeXtender.DataManager.StepTransferSSISExecute.DoAbstractStep(StepSetup stepSetup)”

and didnt send any mails. Also, I have setted the notifications to succesful executions and it didnt send any mails as well.


@devin.tiemens no, we did not set anything to layer executions. It just happens on package executions.


Hi,

Have you had success getting mail from that server before? If not, check that the mail server allows mail to be forwarded from the server running TimeXtender's job


@anders.e.jonsson Yes, we were getting always, this issue started recently with the same settings and mail server. When I click test notification, we get mails.


@KeremOG OK. Are there any errors in the event viewer application on the application server?

 


Last 2 fails on event viewer: 

The task '124335' failed: System.AggregateException: One or more errors occurred. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.Rest.RetryDelegatingHandler.<SendAsync>d__15.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Management.DataFactory.PipelinesOperations.<CreateRunWithHttpMessagesAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Management.DataFactory.PipelinesOperationsExtensions.<CreateRunAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at ODXConnectorEngine.DataFactory.DataFactoryHelper.ExecutePipeline(Dictionary`2 pipelineParameters, Nullable`1 transferType) at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass6_0.<ExecuteTable>b__0(DiscoveryHubSQLExecutionMethod transferMethod) at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action) at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DataSourceMetaStructure sourceStructureModel, DataStorageModel storageModel, DataSourceModel dataSourceModel, DiscoveryHubSQLExecutionModel transferModel) at ExecutionEngine.Action.OutboundTransferAction.Execute() at ExecutionEngine.Action.ExecutionAction.<.ctor>b__9_0() at System.Threading.Tasks.Task.Execute() --- End of inner exception stack trace --- ---> (Inner Exception #0) System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.Rest.RetryDelegatingHandler.<SendAsync>d__15.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Management.DataFactory.PipelinesOperations.<CreateRunWithHttpMessagesAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Management.DataFactory.PipelinesOperationsExtensions.<CreateRunAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at ODXConnectorEngine.DataFactory.DataFactoryHelper.ExecutePipeline(Dictionary`2 pipelineParameters, Nullable`1 transferType) at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass6_0.<ExecuteTable>b__0(DiscoveryHubSQLExecutionMethod transferMethod) at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action) at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DataSourceMetaStructure sourceStructureModel, DataStorageModel storageModel, DataSourceModel dataSourceModel, DiscoveryHubSQLExecutionModel transferModel) at ExecutionEngine.Action.OutboundTransferAction.Execute() at ExecutionEngine.Action.ExecutionAction.<.ctor>b__9_0() at System.Threading.Tasks.Task.Execute()<---

 

--

 

 

The task '123839' failed: System.AggregateException: One or more errors occurred. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.Rest.RetryDelegatingHandler.<SendAsync>d__15.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Management.DataFactory.PipelinesOperations.<CreateRunWithHttpMessagesAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Management.DataFactory.PipelinesOperationsExtensions.<CreateRunAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at ODXConnectorEngine.DataFactory.DataFactoryHelper.ExecutePipeline(Dictionary`2 pipelineParameters, Nullable`1 transferType) at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass6_0.<ExecuteTable>b__0(DiscoveryHubSQLExecutionMethod transferMethod) at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action) at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DataSourceMetaStructure sourceStructureModel, DataStorageModel storageModel, DataSourceModel dataSourceModel, DiscoveryHubSQLExecutionModel transferModel) at ExecutionEngine.Action.OutboundTransferAction.Execute() at ExecutionEngine.Action.ExecutionAction.<.ctor>b__9_0() at System.Threading.Tasks.Task.Execute() --- End of inner exception stack trace --- ---> (Inner Exception #0) System.Threading.Tasks.TaskCanceledException: A task was canceled. at Microsoft.Rest.RetryDelegatingHandler.<SendAsync>d__15.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Management.DataFactory.PipelinesOperations.<CreateRunWithHttpMessagesAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Management.DataFactory.PipelinesOperationsExtensions.<CreateRunAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at ODXConnectorEngine.DataFactory.DataFactoryHelper.ExecutePipeline(Dictionary`2 pipelineParameters, Nullable`1 transferType) at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass6_0.<ExecuteTable>b__0(DiscoveryHubSQLExecutionMethod transferMethod) at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action) at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DataSourceMetaStructure sourceStructureModel, DataStorageModel storageModel, DataSourceModel dataSourceModel, DiscoveryHubSQLExecutionModel transferModel) at ExecutionEngine.Action.OutboundTransferAction.Execute() at ExecutionEngine.Action.ExecutionAction.<.ctor>b__9_0() at System.Threading.Tasks.Task.Execute()<---

 


@Christian Hauggaard , here is the error from event viewer:

 

The task '124335' failed:
System.AggregateException: One or more errors occurred. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.Rest.RetryDelegatingHandler.<SendAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Management.DataFactory.PipelinesOperations.<CreateRunWithHttpMessagesAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Management.DataFactory.PipelinesOperationsExtensions.<CreateRunAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ODXConnectorEngine.DataFactory.DataFactoryHelper.ExecutePipeline(Dictionary`2 pipelineParameters, Nullable`1 transferType)
   at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass6_0.<ExecuteTable>b__0(DiscoveryHubSQLExecutionMethod transferMethod)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
   at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DataSourceMetaStructure sourceStructureModel, DataStorageModel storageModel, DataSourceModel dataSourceModel, DiscoveryHubSQLExecutionModel transferModel)
   at ExecutionEngine.Action.OutboundTransferAction.Execute()
   at ExecutionEngine.Action.ExecutionAction.<.ctor>b__9_0()
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.Rest.RetryDelegatingHandler.<SendAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Management.DataFactory.PipelinesOperations.<CreateRunWithHttpMessagesAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.Management.DataFactory.PipelinesOperationsExtensions.<CreateRunAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at ODXConnectorEngine.DataFactory.DataFactoryHelper.ExecutePipeline(Dictionary`2 pipelineParameters, Nullable`1 transferType)
   at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass6_0.<ExecuteTable>b__0(DiscoveryHubSQLExecutionMethod transferMethod)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
   at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DataSourceMetaStructure sourceStructureModel, DataStorageModel storageModel, DataSourceModel dataSourceModel, DiscoveryHubSQLExecutionModel transferModel)
   at ExecutionEngine.Action.OutboundTransferAction.Execute()
   at ExecutionEngine.Action.ExecutionAction.<.ctor>b__9_0()
   at System.Threading.Tasks.Task.Execute()<---
 


@Christian Hauggaard  you can see the error from event viewer attached.


@KeremOG  Ok this message seems like it might be related to the below issue which now appears to be resolved

 Could you please monitor the emails to see if the notifications are still failing?


hi @Christian Hauggaard , executions running properly for now, I have set the notifications to success and failure but still no mails when it succeeds and fails. 


@KeremOG ok so to confirm you never get emails after the scheduled executions? but the test notification always works?


@Christian Hauggaard yes, notification always works. Executions fails or succeeds, we sometimes get mails.


@KeremOG thanks for the call. It seems that the email notifications have been working as expected the past month. Please let us know if this becomes an issue again.


Reply