Skip to main content

Hi everyone, back again with another issue. 

 

We have now setup our ODX with storage account as storage layer. refreshing it sometimes breaks though. I'm getting 404 error, path can't be found. This happens with transfer all option.

 

When I run the set of tables one by one, everything seems to be fine. I saw 1 more post like this for oracle, where the connector got updated. We're using an older version so I suppose can update the cdata connector and try again. 

However: the weird thing is that this also happens with ODX to DSA. Any clue how this is possible? Path does not exist, but I can find it in the storage account.

 

@dpjat I cannot tell you about the problem in the odx, but in the transfer from odx to dsa the mapping might be missing. Try clicking the right mouse button on the odx and hit synchronize objects. This wil sync the odx with your dsa. It might be that you need to redo the mapping after the synchronize.

Hi,

when the error occurs, does it consistently happen after? If so, does running a synchronize task in the ODX solve it? Are any other systems/users working on the same container?

Hi,

when the error occurs, does it consistently happen after? If so, does running a synchronize task in the ODX solve it? Are any other systems/users working on the same container?

I would imagine not, the container / storage account has been made to support only the ODX. It's just that I schedule it somewhere around 3-4am, it loads all my rsd files(atleast 200-300), and only a few - where I get the message path not found. It's almost like it's “spinning up” and missing like 1 or 2.

The full load then gives me the same message, however not for the same table.

 

I'm going to give it a try with synchronizing. But i'm confident that after it fails and I reload it manually that I'l not see any more failures. (So transfer all/ full load outside of job). 

Hi,

it sounds like a transient error then - ADLS is not guaranteed to have a 100% success rate on requests. There have been some fixes on that front in the past, but if you are running a current version that won't improve by updating. You can get some metrics from Azure Portal > Storage account > Monitor > Insights > Failures - the Availability or Help > Resource Health tabs may contain more info if Microsoft detected problems (as opposed to transient errors).

Perhaps try and stagger your reloads a little bit and see if that helps?

Hi @dpjat 

If you get an error when you run the transfer task on a REST data source utilizing RSD files it is not about the Container in the Storage Account not being available, it is more likely on the data source itself.

If you synchronize on a schedule and it sees a RSD file as not existing it will give the error you see on transfer from ODX - DW.

Can you show how the error looks in the Execution Overview log for the transfer task?

   -Execute Table DSA DSA.AgeCategories Execute DSA DSA.AgeCategories ODX Transfer:Error:Failed
      -Execute DSA DSA.AgeCategories ODX Transfer 'Failed'
        System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (The specified path does not exist.).
           at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
           at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at Azure.Management.API.ADLSGen2.PathAPI.<List>d__11.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<GetDataFileInfos>d__34.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<GetSortedFilesToTransfer>d__33.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__14.MoveNext()
           --- End of inner exception stack trace ---
           at System.Threading.Tasks.Task.WaitAll(TaskN] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2Helper.RunAsyncTasklT](Task`1 task)
           at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass5_0.<ExecuteTable>b__0(DiscoveryHubExecutionMethod transferMethod) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 353
           at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
           at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DiscoveryHubSQLDefaultExecutionModel transferModel, DataSourceModel dataSourceModel, TableModel tableModel) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 462
           at ExecutionEngine.Action.OutboundDefaultTransferAction.Execute()
           at ExecutionEngine.Action.ExecutionAction.<.ctor>b__11_0()
           at System.Threading.Tasks.Task.Execute()
           --- End of inner exception stack trace ---
        ---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (The specified path does not exist.).
           at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
           at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at Azure.Management.API.ADLSGen2.PathAPI.<List>d__11.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<GetDataFileInfos>d__34.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<GetSortedFilesToTransfer>d__33.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__14.MoveNext()
           --- End of inner exception stack trace ---
           at System.Threading.Tasks.Task.WaitAll(Taskr] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2Helper.RunAsyncTaskaT](Task`1 task)
           at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass5_0.<ExecuteTable>b__0(DiscoveryHubExecutionMethod transferMethod) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 353
           at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
           at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DiscoveryHubSQLDefaultExecutionModel transferModel, DataSourceModel dataSourceModel, TableModel tableModel) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 462
           at ExecutionEngine.Action.OutboundDefaultTransferAction.Execute()
           at ExecutionEngine.Action.ExecutionAction.<.ctor>b__11_0()
           at System.Threading.Tasks.Task.Execute()
        ---> (Inner Exception #0) System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (The specified path does not exist.).
           at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
           at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at Azure.Management.API.ADLSGen2.PathAPI.<List>d__11.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<GetDataFileInfos>d__34.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<GetSortedFilesToTransfer>d__33.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__14.MoveNext()<---
        <---

        Details:

        System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (The specified path does not exist.). ...
        Module: timeXtender
        System.Exception
           at TimeXtender.DataManager.StepTransferExecute.DoDataWarehouse(IStepSetup stepSetup, List`1 mergeODXTransferInfromation, VariableResolveObject dynamicResolveObject)
           at TimeXtender.DataManager.StepTransferExecute.DoAbstractStep(IStepSetup stepSetup)

same error different table:

 

 -Execute DSA DSA.AccountingCategories ODX Transfer 'Failed'
        System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (The specified path does not exist.).
           at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
           at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at Azure.Management.API.ADLSGen2.PathAPI.<List>d__11.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<GetDataFileInfos>d__34.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<GetSortedFilesToTransfer>d__33.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__14.MoveNext()
           --- End of inner exception stack trace ---
           at System.Threading.Tasks.Task.WaitAll(Taskt] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2Helper.RunAsyncTasksT](Task`1 task)
           at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass5_0.<ExecuteTable>b__0(DiscoveryHubExecutionMethod transferMethod) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 353
           at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
           at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DiscoveryHubSQLDefaultExecutionModel transferModel, DataSourceModel dataSourceModel, TableModel tableModel) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 462
           at ExecutionEngine.Action.OutboundDefaultTransferAction.Execute()
           at ExecutionEngine.Action.ExecutionAction.<.ctor>b__11_0()
           at System.Threading.Tasks.Task.Execute()
           --- End of inner exception stack trace ---
        ---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (The specified path does not exist.).
           at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
           at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at Azure.Management.API.ADLSGen2.PathAPI.<List>d__11.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<GetDataFileInfos>d__34.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<GetSortedFilesToTransfer>d__33.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__14.MoveNext()
           --- End of inner exception stack trace ---
           at System.Threading.Tasks.Task.WaitAll(Taskt] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2Helper.RunAsyncTaskfT](Task`1 task)
           at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass5_0.<ExecuteTable>b__0(DiscoveryHubExecutionMethod transferMethod) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 353
           at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
           at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DiscoveryHubSQLDefaultExecutionModel transferModel, DataSourceModel dataSourceModel, TableModel tableModel) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 462
           at ExecutionEngine.Action.OutboundDefaultTransferAction.Execute()
           at ExecutionEngine.Action.ExecutionAction.<.ctor>b__11_0()
           at System.Threading.Tasks.Task.Execute()
        ---> (Inner Exception #0) System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (The specified path does not exist.).
           at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
           at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__6.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at Azure.Management.REST.RestWebRequestManager.<SendRequest>d__5.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at Azure.Management.API.ADLSGen2.PathAPI.<List>d__11.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<GetDataFileInfos>d__34.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<GetSortedFilesToTransfer>d__33.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__14.MoveNext()<---
        <---

        Details:

        System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (The specified path does not exist.). ...
        Module: timeXtender
        System.Exception
           at TimeXtender.DataManager.StepTransferExecute.DoDataWarehouse(IStepSetup stepSetup, List`1 mergeODXTransferInfromation, VariableResolveObject dynamicResolveObject)
           at TimeXtender.DataManager.StepTransferExecute.DoAbstractStep(IStepSetup stepSetup)


No child steps
Execute DSA DSA.AccountingCategories Data Cleansing Rules:Info:Initialized Execute Table DSA DSA.Addresses:Info:Initialized
    -Execute Table DSA DSA.Addresses
Execute DSA DSA.Addresses ODX Transfer:Error:Failed
      -Execute DSA DSA.Addresses ODX Transfer 'Failed'
        System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Exception: The data source does not exist on the azure data lake gen2 data storage
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__14.MoveNext()
           --- End of inner exception stack trace ---
           at System.Threading.Tasks.Task.WaitAll(TaskF] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2Helper.RunAsyncTasksT](Task`1 task)
           at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass5_0.<ExecuteTable>b__0(DiscoveryHubExecutionMethod transferMethod) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 353
           at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
           at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DiscoveryHubSQLDefaultExecutionModel transferModel, DataSourceModel dataSourceModel, TableModel tableModel) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 462
           at ExecutionEngine.Action.OutboundDefaultTransferAction.Execute()
           at ExecutionEngine.Action.ExecutionAction.<.ctor>b__11_0()
           at System.Threading.Tasks.Task.Execute()
           --- End of inner exception stack trace ---
        ---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.Exception: The data source does not exist on the azure data lake gen2 data storage
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__14.MoveNext()
           --- End of inner exception stack trace ---
           at System.Threading.Tasks.Task.WaitAll(Task] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
           at DataStorageEngine.DataLakeGen2.DataLakeGen2Helper.RunAsyncTask#T](Task`1 task)
           at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass5_0.<ExecuteTable>b__0(DiscoveryHubExecutionMethod transferMethod) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 353
           at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
           at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DiscoveryHubSQLDefaultExecutionModel transferModel, DataSourceModel dataSourceModel, TableModel tableModel) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 462
           at ExecutionEngine.Action.OutboundDefaultTransferAction.Execute()
           at ExecutionEngine.Action.ExecutionAction.<.ctor>b__11_0()
           at System.Threading.Tasks.Task.Execute()
        ---> (Inner Exception #0) System.Exception: The data source does not exist on the azure data lake gen2 data storage
           at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__14.MoveNext()<---
        <---

        Details:

        System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.Exception: The data source does not exist on the azure data lake gen2 data storage ...
        Module: timeXtender
        System.Exception
           at TimeXtender.DataManager.StepTransferExecute.DoDataWarehouse(IStepSetup stepSetup, List`1 mergeODXTransferInfromation, VariableResolveObject dynamicResolveObject)
           at TimeXtender.DataManager.StepTransferExecute.DoAbstractStep(IStepSetup stepSetup)
 

After synchronizing it still occurs. 😔

So is there any way to get more support on this? It is getting worse for ODX. and now i'm also getting another issue:

 

System.Data.CData.JSON.JSONException (0x80004005): 500] Could not execute the specified command: Index was outside the bounds of the array. ---> djo220X.nM: Index was outside the bounds of the array. ---> System.IndexOutOfRangeException: Index was outside the bounds of the array.

Hi @dpjat, 

Do you have on-demand transfers enabled? If so, the error could be between data-source and ODX. It appears to be an HTTP error, perhaps coming from the API. 

The second error that you posted appears to be generated by CDATA between data-source and ODX, and indicating a problem with the RSD file, where the format of the API response did not fit the expected format in the RSD. It could be, for example, that the API you are calling does not have a consistent data structure.

You could enable logging in your CDATA source and set it to verbosity = 2. Then we can see if there is a (failed) call to the API. 

Kind regards,

Andrew

Hi @dpjat could you please provide an update to the questions above?

Reply


Terms and ConditionsCookie settings