Not all rows of CSV file are loaded during Full load ingest task

​@Mies  do you have a sample/dummy csv file that you can share with us? please remove any sensitive data from the file

Hi ​@Mies ,

Perhaps all the rows did load, but due to a filter or a Primary Key setting some records are not visible?

​@Christian Hauggaard Thanks for your quick response. Here is a file sample file with dummy data.

​@daniel I don't have any filter options included in de ODX and also no Primary keys. I only have set an override data type on one of the text columns. 

​@Mies I noticed that the dummy file only has 9 records, all of which are successfully loaded when I use the TimeXtender data source with the following settings

Can you send a file with all records? if the file contains sensitive data that you are willing to share with our support team but not here in the community, then please send to support@timextender.com

​@Mies 

Can you figuer out which records have not been loaded? Do they have the same attributes? The same data perhaps?

I think the answer lies in the data.

This is deep. Very deep. I need a coffee mug with that text.

Happy to help. You digital mug is on it way

 

Hope this helps

= Daniel

​@daniel  I found rows that were not loaded. If I load these rows in a separate CSV file with the same datasource connection, these rows are loaded in the ODX. I also don't see strange values for these rows. 

​@Christian Hauggaard I was using the old Cdata CSV provider. I'm now trying to switch to the Timextender CSV datasource connection like you have (version 20.0.0.0) But then I got this key error when I want to synchronize:
 

Hi,

nice mug :-)

there are quite a few possiblities in CSV depending on the contents of the file:

• primary key clashes leading to data not landing in Ingest. If there is no primary key set up, this won't be the issue
• strings containing control characters (i.e. line breaks). If you have text fields that are not encapsulated in quote characters, the parser may see a /r/n etc. as the end of a record and therefore mess up the load with respect to record boundaries. This can also cause certain records to be seen as incomplete and not loaded, depending on your settings
• if your data is comma separated  and you have decimal numbers, you may need to explicitly set culture settings to avoid the decimal separator to be seen as a field delimiter or to have your decimals be mis-interpreted

In the worst case you can cut your CSV in half and try loading it, if you keep doing that you should find the records leading to issues.

Your first statement would seem to indicate there is some kind of key clash going on, your second statement seems to indicate there is some kind duplicate item (source name, table name, field name) that is not expected.

​@Mies please send the CSV file to support@timextender.com so we can troubleshoot further. If you are unable to share the file, then send an email to support@timextender.com requesting a meeting