I have a customer table that is a union of 7 different sources. Same ERP but in different databases and different Ingest Sources (so 7 different folders in azure storage).
When I add a FIELD TRANSFORMATION to the VALID table and re-run it, both the RAW and VALID are dropped and reloaded (I don’t use history). In this case 7 ADF-pipelines are spun up, but it was not really necessary because nothing new came from any of the others VALID tables.
When you deploy the changes, you see that TX only deploys the TRANSFORMATION view, so it “sees” that only a change at the level of the TRANSFORMATION view was added.
I don’t know if this is feasible in all cases, but it would be more cost-efficient as I don’t always need to spin up those (expensive) ADF-pipelines
