I’m using Timextender version 6521.1 and I get an error when using hard deletes on a incremental table. If I choose no deletes it works just fine, soft deletes also throw me and error.
As far as I could tell from these notes incremental loading should be fully supported.
Things I have checked:
- Primary key on the table in the ODX
- Incremental load is setup in the ODX
- Incremental load is setup on the Snowflake table.
I’m not quite sure if this is just a bug or I missed something obvious. This is the first table i’m trying to load incremental in Snowflake.
This is the error it throws at me:
System.AggregateException: One or more errors occurred. ---> Snowflake.Data.Client.SnowflakeDbException: Error: Error parsing the parquet file: Invalid: Parquet file size is 0 bytes
Row 0 starts at line 0, column
If you would like to continue loading when an error is encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option. For more information on loading options, please run 'info loading_data' in a SQL client. SqlState: 22000, VendorCode: 100152, QueryId: 01b3fd47-0203-be81-0001-994600742d06
at Snowflake.Data.Core.SFStatement.ExecuteHelpereT,U](Int32 timeout, String sql, Dictionary`2 bindings, Boolean describeOnly)
at Snowflake.Data.Core.SFStatement.Execute(Int32 timeout, String sql, Dictionary`2 bindings, Boolean describeOnly)
at Snowflake.Data.Client.SnowflakeDbCommand.ExecuteInternal(Boolean describeOnly)
at Snowflake.Data.Client.SnowflakeDbCommand.ExecuteNonQuery()
at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass7_0.<ExecuteTable>b__0(DiscoveryHubExecutionMethod method) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 705
at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DiscoveryHubSnowflakeExecutionModel transferModel, ODXEndpoint storageEndpoint, DataSourceModel dataSourceModel, TableModel tableModel) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 724
at ExecutionEngine.Action.OutboundSnowflakeTransferAction.Execute()
at ExecutionEngine.Action.ExecutionAction.<.ctor>b__13_0()
at System.Threading.Tasks.Task.Execute()
--- End of inner exception stack trace ---
---> (Inner Exception #0) Snowflake.Data.Client.SnowflakeDbException (0x80004005): Error: Error parsing the parquet file: Invalid: Parquet file size is 0 bytes
Row 0 starts at line 0, column
If you would like to continue loading when an error is encountered, use other values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option. For more information on loading options, please run 'info loading_data' in a SQL client. SqlState: 22000, VendorCode: 100152, QueryId: 01b3fd47-0203-be81-0001-994600742d06
at Snowflake.Data.Core.SFStatement.ExecuteHelperwT,U](Int32 timeout, String sql, Dictionary`2 bindings, Boolean describeOnly)
at Snowflake.Data.Core.SFStatement.Execute(Int32 timeout, String sql, Dictionary`2 bindings, Boolean describeOnly)
at Snowflake.Data.Client.SnowflakeDbCommand.ExecuteInternal(Boolean describeOnly)
at Snowflake.Data.Client.SnowflakeDbCommand.ExecuteNonQuery()
at ODXConnectorEngine.ConnectorEngine.<>c__DisplayClass7_0.<ExecuteTable>b__0(DiscoveryHubExecutionMethod method) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 705
at DataStorageEngine.DataLakeGen2.DataLakeGen2StorageEngine.RunDiscoveryHubTransferMethod(Action`1 action)
at ODXConnectorEngine.ConnectorEngine.ExecuteTable(IDataStorageEngine storageEngine, DiscoveryHubSnowflakeExecutionModel transferModel, ODXEndpoint storageEndpoint, DataSourceModel dataSourceModel, TableModel tableModel) in F:\VS2022Code\ODX-SaaS-Prod\ODXConnectorEngine\ConnectorEngine.cs:line 724
at ExecutionEngine.Action.OutboundSnowflakeTransferAction.Execute()
at ExecutionEngine.Action.ExecutionAction.<.ctor>b__13_0()
at System.Threading.Tasks.Task.Execute()<---
Details:
System.AggregateException: One or more errors occurred. ---> Snowflake.Data.Client.SnowflakeDbException: Error: Error parsing the parquet file: Invalid: Parquet file size is 0 bytes ...
Module: timeXtender
System.Exception
at TimeXtender.DataManager.StepTransferExecute.DoDataWarehouse(IStepSetup stepSetup, List`1 mergeODXTransferInfromation, VariableResolveObject dynamicResolveObject)
at TimeXtender.DataManager.StepTransferExecute.DoAbstractStep(IStepSetup stepSetup)
