+1Does anyone know exactly what happens when we ingest data from e.g. a SQL Server into Fabric Lakehouse? It seems concurrency impacts performance (more can be less) but there might also be Fabric assets in play that could act like bottlenecks, and which requires “capacity” management.
Any best recommendations/practices in this space?
Best answer by rory.smith
Hi
that is quite a broad topic 😁
Apart from the capacity of your source to deliver, the virtual machine hosting Ingest is a critical part: you are sending (delta) parquet to the Lakehouse through that resource after all. TDI offloads that part to your VM instead of leveraging Fabric as this tends to be more cost-effective. I usually monitor CPU / RAM / disk queues to see whether there is headroom during operations, but operations on parquet tend to need both CPU and memory at the same time in rather large amounts.
Once you get into Prepare on Lakehouse, there is more of Fabric involved but I would start out making sure your VM is not restricting flow.
+7Hi
that is quite a broad topic 😁
Apart from the capacity of your source to deliver, the virtual machine hosting Ingest is a critical part: you are sending (delta) parquet to the Lakehouse through that resource after all. TDI offloads that part to your VM instead of leveraging Fabric as this tends to be more cost-effective. I usually monitor CPU / RAM / disk queues to see whether there is headroom during operations, but operations on parquet tend to need both CPU and memory at the same time in rather large amounts.
Once you get into Prepare on Lakehouse, there is more of Fabric involved but I would start out making sure your VM is not restricting flow.
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
