Extend Storage Management Task features for Azure Data Lake in Ingest Server – move data to Cold Storage/Archive based on rules

Hi,

while I think it is a logical functionality to add - I also think this can become quite complex to solve. There are probably three routes to handle this:

• manual moving of data between containers with different access tiers set
  • this will require ADO or ADF and therefore a fair amount of resources
• applying Lifecycle Management Policies (see: https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview).
  • there are some restrictions, but this avoids physically moving stuff around.
  • there is load on the storage account itself
  • be wary of: "Avoid moving small files to lower tiers, unless you must” many files in the TDI world are small
• using Azure Storage Actions (in preview in European regions, see: https://learn.microsoft.com/en-us/azure/storage-actions/overview)
  • aimed at larger storage set ups
  • potentially costly

There are quite some considerations to make:

• colder tiers are more costly to access, archive tiers require (costly) rehydration which takes a fair amount of time.
• If you are setting up mutliple containers with different tiering, should they be in the same storage account? Cross-storage account (or region) points you more in the direction of Storage Actions which are not yet GA in Europe.
• There are also limitations with respect to encrypted blobs and blobs that are set up to be immutable (for compliancy or legal reasons).

I guess this should also do something like creating intermediate full load files for incremental loads as it is better to collate deltas into a full set and move that to another tier than have part of your full load set of deltas moved to archive.

In my experience, simply using the “amount of versions to keep” is good enough for most customers. This can become especially onerous in GDPR scenarios where you need to forget someone and therefore need to rehydrate an archived set to delete specific files.

There is likely also a fair difference between the way this works in Azure Storage vs. OneLake, and if Snowflake also sees support as Ingest storage that will be different again. There is a fine balance between having the same functionality for every architecture and fully supporting the details of different systems with respect to ease-of-use. TimeXtender removes a large area of the problem space, which makes it much easier to get started with than needing to pick and choose tools from the gigantic data landscape (i.e. https://lakefs.io/blog/the-state-of-data-engineering-2024/).

While I like having all the bells and whistles there is also a downside to them - I can see this working as a feature for Orchestration as that tends to live a little more on the “custom” side of things. Similar to the Azure automation that is available there.

tl;dr; more discussion needed :-)