Hello everyone. I'm having issues in TX while loading a table with more than 20K in parquet and
I think I found a bug in TX. I’m not 100% sure, but this is my hypothesis.
We are ingesting a table from SQL Server that has several bit columns. Ingest writes it to our Azure Data Lake as a Parquet file. The table has 23,909 records, and the Parquet file is split internally in blocks of 20,000 rows, so it has one block of 20,000 and a second one of 3,909.
The first 20,000 rows load into Prepare correctly. From row 20,000 on, the bit columns that have NULL values get shifted, so each row ends up with the value of another row. In one column we had 1,583 wrong rows, and in another one we had 6. The totals are practically the same, so the values are not lost; they are just on the wrong rows. Bit columns without NULLs are fine, and so are all the other data types.
I compared the source against the Parquet file in the Data Lake, and they match 100%. The problem shows up when Prepare reads the Parquet file. A full load doesn’t fix it; we get exactly the same result every time.
I’m fairly sure it’s a bug or something similar because I changed those bit columns to TinyInt in Ingest, and after a full load everything matched with no differences.
I also saw the fix in version 7262.1 about columns shifting when copying from Ingest to Prepare. It looks similar, but I’m not sure it’s the same issue.
Our version is TimeXtender Data Integration 7387.1. We are using ADO.net to move records.
Does anyone had experienced this? IF you are experience it the best workaround I found was to override the data type on the ingest and let SQL to Cast the value to bit when moving to the prepare instance, so you don't need to change reports. etc.
