Implementing Azure Data Lake Storage Lifecycle Rule for Tenant-Specific Containers

Question

I'm working with Azure Data Lake Storage (ADLS) and have a multi-tenant setup where each tenant has a separate container. The folder structure within each tenant's container is standardized and includes a Processed directory, structured as follows:

<tenantIdContainer>\Processed

I am trying to implement a lifecycle management rule across these containers. The goal is to automatically move files within the Processed directory to cold storage after a specific period of time. My initial attempt at creating this rule involved using wildcard patterns starting with *, but this approach did not work.

I am seeking guidance on how to correctly set up these lifecycle rules for such a tenant-specific container setup in ADLS. Could you possibly provide some guidance on this?

Answer 1

Implementing Azure Data Lake Storage Lifecycle Rule for Tenant-Specific Containers

I don't think you can apply this condition in a single rule. You can apply this condition by creating multiple rules with container names.

You can use the below PowerShell script; it will create each rule for each container for implementing Azure life cycle rules.

Script:

Connect-AzAccount
Select-AzSubscription -SubscriptionId "xxxxx"

$ctx = New-AzStorageContext -StorageAccountName 'xxxx' -UseConnectedAccount
$containers = Get-AzStorageContainer -Context $ctx

$rules = @()

foreach ($container in $containers) {
    $path = "$($container.Name)/Processed/"
    $action = Add-AzStorageAccountManagementPolicyAction -BaseBlobAction TierToCool -DaysAfterModificationGreaterThan 30
    $filter = New-AzStorageAccountManagementPolicyFilter -PrefixMatch $path -BlobType BlockBlob
    $rule = New-AzStorageAccountManagementPolicyRule -Name "Move to cold storage-$($container.Name)" -Action $action -Filter $filter
    $rules += $rule
}

# Apply all the rules after the loop
Set-AzStorageAccountManagementPolicy -ResourceGroupName "xxxx" -StorageAccountName 'xxx' -Rule $rules

The above script creates a separate rule for each container in the storage account. The rules are stored in an array called $rules, and each rule is added to the array inside a loop that iterates over all the containers in the storage account. After all the rules have been created, the script applies them to the storage account using the Set-AzStorageAccountManagementPolicy cmdlet.

Output:

ResourceGroupName  : xxxx
StorageAccountName : xxxx
Id                 : /subscriptions/xxx/resourceGroups/xxxx/providers/Microsoft.Storage/storageAccounts/xxx/managementPolicies/default
Type               : Microsoft.Storage/storageAccounts/managementPolicies
LastModifiedTime   : 23-01-2024 06:29:48
Rules              : [
                         {
                             "Enabled":  true,
                             "Name":  "Move to cold storage-3efaxxxx",
                             "Definition":  {
                                                "Actions":  {
                                                                "BaseBlob":  {
                                                                                 "TierToCool":  {
                                                                                                    
                     "DaysAfterModificationGreaterThan":  30,
                                                                                                    
                     "DaysAfterLastAccessTimeGreaterThan":  null,
                                                                                                    
                     "DaysAfterCreationGreaterThan":  null,
                                                                                                    
                     "DaysAfterLastTierChangeGreaterThan":  null
                                                                                                },
                                                                                 "TierToArchive":  null,
                                                                                 "Delete":  null,
                                                                                 "TierToCold":  null,
                                                                                 "TierToHot":  null,
                                                                                 "EnableAutoTierToHotFromCool":  null
                                                                             },
                                                                "Snapshot":  null,
                                                                "Version":  null
                                                            },
                                                "Filters":  {
                                                                "PrefixMatch":  [
                                                                                    
                     "3xxxx/Processed/"
                                                                                ],
                                                                "BlobTypes":  [
                                                                                  "blockBlob"
                                                                              ],
                                                                "BlobIndexMatch":  null
                                                            }
                                            }
                         },
                         {
                             "Enabled":  true,
                             "Name":  "Move to cold storage-72f98xxx",
                             "Definition":  {
                                                "Actions":  {
                                                                "BaseBlob":  {
                                                                                 "TierToCool":  {
                                                                                                    
                     "DaysAfterModificationGreaterThan":  30,
                                                                                                    
                     "DaysAfterLastAccessTimeGreaterThan":  null,
                                                                                                    
                     "DaysAfterCreationGreaterThan":  null,
                                                                                                    
                     "DaysAfterLastTierChangeGreaterThan":  null
                                                                                                },
                                                                                 "TierToArchive":  null,
                                                                                 "Delete":  null,
                                                                                 "TierToCold":  null,
                                                                                 "TierToHot":  null,
                                                                                 "EnableAutoTierToHotFromCool":  null
                                                                             },
                                                                "Snapshot":  null,
                                                                "Version":  null
                                                            },
                                                "Filters":  {
                                                                "PrefixMatch":  [
                                                                                    
                     "7xxxx/Processed/"
                                                                                ],
                                                                "BlobTypes":  [
                                                                                  "blockBlob"
                                                                              ],
                                                                "BlobIndexMatch":  null
                                                            }
                                            }
                         },
                         {
                             "Enabled":  true,
                             "Name":  "Move to cold storage-da10xxxxx",
                             "Definition":  {
                                                "Actions":  {
                                                                "BaseBlob":  {
                                                                                 "TierToCool":  {
                                                                                                    
                     "DaysAfterModificationGreaterThan":  30,
                                                                                                    
                     "DaysAfterLastAccessTimeGreaterThan":  null,
                                                                                                    
                     "DaysAfterCreationGreaterThan":  null,
                                                                                                    
                     "DaysAfterLastTierChangeGreaterThan":  null
                                                                                                },
                                                                                 "TierToArchive":  null,
                                                                                 "Delete":  null,
                                                                                 "TierToCold":  null,
                                                                                 "TierToHot":  null,
                                                                                 "EnableAutoTierToHotFromCool":  null
                                                                             },
                                                                "Snapshot":  null,
                                                                "Version":  null
                                                            },
                                                "Filters":  {
                                                                "PrefixMatch":  [
                                                                                    
                     "dxxxx/Processed/"
                                                                                ],
                                                                "BlobTypes":  [
                                                                                  "blockBlob"
                                                                              ],
                                                                "BlobIndexMatch":  null
                                                            }
                                        }
                     }
                 ]

Reference:

Configure a lifecycle management policy - Azure Storage | Microsoft Learn