I run into this error while loadning csv-files from our s3 bucket.
The issue is that the number of fields differs from time to time. We would like to ignore that and read these fields if they exists and otherwise not.
Is there a way to do this or is this just an exception in CsvHelper-lib that isnt accessible for me to configure?
The error:
Executing table [csv].[VOF_UTF_Planeradebesok__aggregated]:
failed with error:
CsvHelper.MissingFieldException: Field with name 'Planerat besök - konfidentiellt' does not exist. You can ignore missing fields by setting MissingFieldFound to null.
IReader state:
ColumnCount: 21
CurrentIndex: -1
HeaderRecord:
["Användare - ID2","Användare - personnummer","Användare - profession","Avbruten/inställd","Brukare - ID2","Brukare - personnummer","Planerad resurs - namn","Planerat besök - dubbelbemanning","Planerat besök - ej utfört (anledning)","Planerat besök - sluttid","Planerat besök - starttid","Planerat besök - typ av brukare","Planerat besök - utfört","Planeringsid","Skapad","Utfört besök - ej planerade insatser utförda","Utfört besök - extratid","Utfört besök - fiktivt","Utfört besök - sluttid","Utfört besök - starttid","Importdatum"]
IParser state:
ByteCount: 0
CharCount: 493
Row: 1
RawRow: 1
Count: 21
RawRecord:
Användare - ID2|Användare - personnummer|Användare - profession|Avbruten/inställd|Brukare - ID2|Brukare - personnummer|Planerad resurs - namn|Planerat besök - dubbelbemanning|Planerat besök - ej utfört (anledning)|Planerat besök - sluttid|Planerat besök - starttid|Planerat besök - typ av brukare|Planerat besök - utfört|Planeringsid|Skapad|Utfört besök - ej planerade insatser utförda|Utfört besök - extratid|Utfört besök - fiktivt|Utfört besök - sluttid|Utfört besök - starttid|Importdatum
at CsvHelper.Configuration.ConfigurationFunctions.MissingFieldFound(MissingFieldFoundArgs args)
at CsvHelper.CsvReader.GetFieldIndex(IEnumerable`1 names, Int32 index, Boolean isTryGet, Boolean isOptional)
at CsvHelper.CsvReader.GetFieldIndex(String name, Int32 index, Boolean isTryGet)
at TimeXtender.CSV.Core.CsvDataReader.GetHasHeaderCsvColumnIndices()
at TimeXtender.CSV.Core.CsvDataReader.MoveToNextStream()
at TimeXtender.CSV.Core.CsvDataReader.Read()
at Microsoft.Data.SqlClient.SqlBulkCopy.ReadFromRowSourceAsync(CancellationToken cts)
at Microsoft.Data.SqlClient.SqlBulkCopy.WriteToServerInternalAsync(CancellationToken ctoken)
at Microsoft.Data.SqlClient.SqlBulkCopy.WriteRowSourceToServerAsync(Int32 columnCount, CancellationToken ctoken)
at Microsoft.Data.SqlClient.SqlBulkCopy.WriteToServer(IDataReader reader)
at DataStorageEngine.SQL.SQLHelper.TransferData(SqlBulkCopy destinationBulkLoader, IDataReader sourceReader)
at DataStorageEngine.SQL.SQLHelper.TransferWithBulkCopy(SqlDatabaseEndpoint storageEndpoint, IDbCommand destinationCommand, String destinationTableName, IEnumerable`1 columnModels, IDataReader sourceDataReader, IDbCommand sourceTransferCommand, DataTable transferTable, Boolean isPrimaryKeyTransfer, String[] queries, Boolean disposeSourceObjects)
at DataStorageEngine.SQL.SQLStorageEngine.<>c__DisplayClass67_1.<TransferDataToTable>b__3(IDataReader dataReader)
at DataStorageEngine.SQL.SQLStorageEngine.<>c__DisplayClass67_0.<TransferDataToTable>b__1(IDbCommand destinationCommand)
at DataStorageEngine.SQL.SQLStorageExtensions.ExecuteCommand(IDbConnection connection, Int32 commandTimeout, Action`1 action)
at DataStorageEngine.SQL.SQLStorageEngine.<>c__DisplayClass64_5.<TransferData>b__9(SQLExecutionMethod executionMethod)
at DataSourceEngine.Custom.CustomSourceEngine.RunExecution[M](DataExecutionDestination dataExecutionMethod, Action`1 action)
at DataStorageEngine.SQL.SQLStorageEngine.<>c__DisplayClass64_3.<TransferData>b__6()
