Error: Error Ingesting MSSQL Navision to Azure Data Lake - ParquetInvalidColumnName

Symptoms

Error encountered when ingesting data from an MSSQL (Navision) source to Azure Data Lake using Azure Data Factory in TimeXtender. The process failed for a specific table with the following error message:

The error indicated that one or more column names contained invalid characters for Parquet files. The user identified columns with special characters (such as hyphens and non-ASCII characters) as potential causes. The issue was specific to certain tables, while others from the same source processed successfully.

Cause

The root cause was a stale or inconsistent pipeline definition in Azure Data Factory associated with the failing ingest task. This outdated pipeline was not properly handling the column name transformations required for Parquet compatibility, leading to the error.

Resolution

1. Delete the Existing Azure Data Factory Pipeline
   • Go to the Azure Data Factory portal.
   • Locate and manually delete the pipeline associated with the failing ingest task.
2. Re-execute the Ingest Task in TimeXtender
   • Return to TimeXtender and run the ingest task again.
   • This action forces TimeXtender to generate and deploy a brand-new pipeline in Azure Data Factory.
   • The task should now complete successfully without the "ParquetInvalidColumnName" error.
3. Best Practice: Avoid Renaming ADF Objects Created by TimeXtender
   • Do not rename pipelines, datasets, or dataflows that TimeXtender automatically generates in Azure Data Factory, as this may cause TimeXtender to lose track of them and create new ones on the next deployment.
   • If you need to create or update integration runtime or other objects, ensure you update the configuration in the TimeXtender Portal accordingly.
4. Alternative Approaches (if issue persists)
   • Consider configuring column name transformations or aliases within TimeXtender’s Ingest Instance to ensure all column names conform to Parquet’s naming conventions.
   • Implement custom transformation logic in TimeXtender to replace or remove invalid characters in column names before writing to Parquet.

Reference Ticket

36324

Reference Links

• Use Azure Data Factory for Data Movement
• How to use an Azure Service Principal or App Registration for Authentication to an Azure Data Lake Gen2 Storage Container File