+3We have some large tables source tables that need to split up for further transformation. For instance, we have a large source table containing all corporate financial transactions, that need to be divided over several divisions.
To achieve more efficiency in processing, we like TX (ingest/prepare) to support the partition functionality of Fabric.
https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-lakehouse-partition
+8Hi,
while this approach uses Data Pipelines which may be less performant than how TimeXtender handles data transfer to Lakehouse / Onelake it does show a clear folder structure. If TIS can write the result in such a structure and that can be ‘picked up’ by the rest of Fabric, this would work nicely.
If I remember correctly there is/was already a way to send data to ADLS in chunks based on PK + columnset1 , PK + columnset2 .. this mechanism could also be re-used with the addition of splitting the parquet for a specific field.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
