To homepage To homepage
Question

Error on table with 0 files and incremental offset rule

S
Rank
Userlevel 3
Badge +1

We have data sources where we apply dynamic incremental rules, so if column == “Name” then apply incremental load with 3 day offset. 

Now there exist some tables in the data source that are currently empty (in the actual source). They do have the incr column and therefore the offset rule. Now we noticed that the ODX transfer on these tables works fine.

However when we execute a table with such a table in the mapping we obtain this error: 

System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Sequence contains no elements
   at System.Linq.Enumerable.Max(IEnumerable`1 source)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__14.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2Helper.RunAsyncTask[T](Task`1 task)
   at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass5_0.<ExecuteTable>b__0(DiscoveryHubExecutionMethod transferMethod) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 454
   at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
   at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DiscoveryHubSQLDefaultExecutionModel transferModel, ODXEndpoint storageEndpoint, DataSourceModel dataSourceModel, TableModel tableModel) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 456
   at ExecutionEngine.Action.OutboundDefaultTransferAction.Execute()
   at ExecutionEngine.Action.ExecutionAction.<.ctor>b__9_0()
   at System.Threading.Tasks.Task.Execute()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Sequence contains no elements
   at System.Linq.Enumerable.Max(IEnumerable`1 source)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__14.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2Helper.RunAsyncTask[T](Task`1 task)
   at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass5_0.<ExecuteTable>b__0(DiscoveryHubExecutionMethod transferMethod) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 454
   at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
   at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DiscoveryHubSQLDefaultExecutionModel transferModel, ODXEndpoint storageEndpoint, DataSourceModel dataSourceModel, TableModel tableModel) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 456
   at ExecutionEngine.Action.OutboundDefaultTransferAction.Execute()
   at ExecutionEngine.Action.ExecutionAction.<.ctor>b__9_0()
   at System.Threading.Tasks.Task.Execute()
---> (Inner Exception #0) System.InvalidOperationException: Sequence contains no elements
   at System.Linq.Enumerable.Max(IEnumerable`1 source)
   at DataStorageEngine.DataLakeGen2.DataLakeGen2DiscoveryHubExecution.<ProvideDataTableForDiscoveryHubSQLTransferAsync>d__14.MoveNext()<---
<---

Details:

System.AggregateException: One or more errors occurred. ---> System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Sequence contains no elements ...
Module: timeXtender
System.Exception
   at TimeXtender.DataManager.StepTransferExecute.DoDataWarehouse(IStepSetup stepSetup, List`1 mergeODXTransferInfromation, VariableResolveObject dynamicResolveObject)
   at TimeXtender.DataManager.StepTransferExecute.DoAbstractStep(IStepSetup stepSetup)

 

When I remove this incremental rule in the ODX, the transfer works fine. 

 

Is it then not possible to apply an incremental offset on a table that never had any records? 

12 replies

Thomas Lind
Rank
Userlevel 5
Badge +5

Hi @sierd.zuiderveld 

Yes it can likely not handle empty tables.

How is your setup done? I mean

We have data sources where we apply dynamic incremental rules, so if column == “Name” then apply incremental load with 3 day offset. 

There is no option to do conditions on a incremental rule in an ODX data source.

 

S
Rank
Userlevel 3
Badge +1

@Thomas Lind  I simply meant, all tables that equal column X. Just like in your screenshot. 

Thomas Lind
Rank
Userlevel 5
Badge +5

Hi @sierd.zuiderveld 

Thanks for the clarification.

If you don’t add a subtraction you don’t get an error, right?

S
Rank
Userlevel 3
Badge +1

Hey @Thomas Lind , 

It seems I was wrong and it is not actually caused by the offset. I ran it with different ODX settings and these are the results 

By the looks of it it seems that the issue is caused by PK deletes instead. 

Thomas Lind
Rank
Userlevel 5
Badge +5

Hi @sierd.zuiderveld 

I have tried to replicate it. However, I do not get the error you experience.

I used a TX SQL data source where I added a table without any data in it. Then I made an incremental rule that affected it, but no matter what it just went through without any issues.

Can you explain more about your setup? So version, ODX store and data source type.

S
Rank
Userlevel 3
Badge +1

Hmm well good to know that it should work! 

 

Here is some more info:

We use TimeXtender 6346.1 

Azure data lake storage gen2 as our ODX storage. 

TimeXtender SQL Data Source  17.1.1.0 (Can be upgraded to 18.0.0.0, I will try that as well.)

Advanced settings: 

The source table is actually a view, because of this we define the primary keys by rules.

No override data types, no filter rows

 

 

S
Rank
Userlevel 3
Badge +1

Updating to v18 gives the following error msg 
 

Service request failed: System.InvalidOperationException: Sequence contains no matching element
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)
   at ODX.Components.ComponentManager.InstallInternalProviderIfRequired(DataSourceConnectionModel dataSourceConnectionModel, ComponentTypes componentType)
   at DataSourceEngine.Custom.CustomSourceEngine.GetAssemblyFilePath()
   at DataSourceEngine.Custom.CustomSourceEngine.GetCustomDataSource()
   at DataSourceEngine.Custom.CustomSourceEngine.<SupportsIncrementalLoad>b__42_0()
   at DataSourceEngine.Custom.CustomSourceEngine.SupportsIncrementalLoad() (System.Exception)

Details:

Service request failed: System.InvalidOperationException: Sequence contains no matching element ...
Module: TimeXtender.ODX.Engine
TimeXtender.ODX.Engine.ODXFaultException
   at TimeXtender.ODX.Engine.ODXEngine.SendServiceRequest[C,T](WcfServerSettings serverSettings, Func`3 func)
   at TimeXtender.DataManager.EditDataSourceCommand.GetConnectionValidationModel(OperationalDataExchange operationalDataExchange, ODXDataSource dataSource, DataSourceModel dataSourceModel)
   at TimeXtender.DataManager.EditDataSourceCommand.<>c__DisplayClass14_2.<Dialog_OnChangeConnection>b__1()
   at TimeXtender.DataManager.ConnectingThread.ExecuteConnectingThread(Object dummy)

Service request failed: System.InvalidOperationException: Sequence contains no matching element ...
Module: timeXtender
TXModelInterface.ExceptionWrapperException
   at TimeXtender.DataManager.ConnectingThread.HandleError()
   at TimeXtender.DataManager.ConnectingThread.Execute(String title, Int32 progressSteps, List`1 actions)
   at TimeXtender.DataManager.ConnectingThread.ExecuteFastAction(String title, Action action, IWin32Window parentForm, CancelBehaviors cancelBehavior, ErrorBehaviors errorBehavior, String callerName)
   at TimeXtender.DataManager.EditDataSourceCommand.<>c__DisplayClass14_1.<Dialog_OnChangeConnection>b__0(Object s, FormClosingEventArgs e)

Time: 2023-11-07 14:58:58
UTC: 2023-11-07 13:58:58
Title: ODXDEV - TimeXtender
Application: 6346.1
User: ax_zuiderveld_admin
Domain: BLIJDORP
Login user: Sierd Zuiderveld (sierdtx@gmail.com)
OS: Microsoft Windows Server 2022 Datacenter
OS version: Microsoft Windows NT 6.2.9200.0
Machine name: BSIBIMAN01
CPU count: 4
Build: 64 bit
 

 

Thomas Lind
Rank
Userlevel 5
Badge +5

Hi @sierd.zuiderveld 

What about the other version.

We use TimeXtender 6346.1 

Did you upgrade that as well? I think both may be part of it.

Christian Hauggaard
Rank
Userlevel 5
Badge +5

Hi @sierd.zuiderveld did you get a chance to upgrade TimeXtender to the latest version (6436.1)? Can you please let us know if this resolved the issue?

S
Rank
Userlevel 3
Badge +1

Hey @Christian Hauggaard, we will do so next week. I will let you know if the problem still occurs. 

S
Rank
Userlevel 3
Badge +1

Hi @Thomas Lind , @Christian Hauggaard. 

We have tried again with the newest version (6436.1) and the newer data source (18.0.0.0) but we still obtain the same error. It happens in the transfer step. 

 

Christian Hauggaard
Rank
Userlevel 5
Badge +5

Hi @sierd.zuiderveld 

I have created a support ticket for this so we can do a screenshare.

Reply


Terms and ConditionsCookie settings