Azure Storage - Permanent Delete Soft-Deleted objects

August 17, 2022

Overview

This PowerShell script allow you to permanent delete Soft Deleted objects, by Container, by Tier, with Prefix, and considering Last Modified Date.

Azure Storage blob objects is defined as Base Blobs, Blob Snapshots or Blob Versions.

Parameters:

Specify each value in the script under "Parameters - User defined" section.

All values are mandatory and some can be empty, as described below and also in the script.

Options available are:

$storageAccountName - just run the script and the storage account name will be asked.

$containerName - specify some Container Name, or empty (default value) to list all containers

$prefix - specify some blob prefix (excluding container name) for scanning, or leave empty (default value) to list all objects

$blobType - select 'Base' to list only base Blobs (default value), 'Snapshots' to list only Snapshots, 'Versions' to list only Versions, 'Versions+Snapshots' to list only Versions and Snapshots, or 'All Types' to list all objects (base Blobs, Versions and Snapshots)

$accessTier - select 'Hot' to list only objects in Hot tier, 'Cool' to list only objects in Cool tier, 'Archive' to list only objects in Archive tier, or 'All' to list objects in all tiers (Hot, Cool and Archive)

$Year, $Month, $Day - Define a date to list objects only before or equal of Last Modified Date - if at least one value is empty, current date will be used.

Notes:

- Just running the script will ask for your AAD credentials and to select the storage account name to list.

- By default (without any parameter change), the script will list and count Soft Deleted objects, on all containers in the storage account, from all access tiers, with Last Modified Date before or equal current date time.

- All options above may be defined in the script.

- This can take hours/days to complete, depending on the number of blobs, versions and snapshots in the container or Storage account in Soft Deleted state.

- $logs container is not covered by this script (not supported)

# ====================================================================================

# Azure Storage - Permanent Delete Soft-Deleted objects (Base Blobs, Blob Snapshots, Versions)

# Based on Container, prefix, Tier and considering Last Modified Date

# ====================================================================================

# DISABLE SOFT DELETE FEATURE ON STORAGE ACCOUNT BEFORE RUNNING THIS SCRIPT

# Otherwise the soft delteded snapshots reapers in sof-deleted state

# You can reenable Soft Delete featurs after running this script, if needed.

# ====================================================================================

# DISCLAMER : please note that this script is to be considered as a sample and is provided as is with no warranties express or implied, even more considering this is about deleting data.

# Really recommended to double check that list of filtered elements looks fine to you before processing with the deletion with the last line of the script.

# You can use or change this script at you own risk.

# ====================================================================================

# PLEASE NOTE :

# - For this to work we must first disable Blob soft-delete feature before run the script.

# Please wait 30s after you disabled soft-delete for the effect to propagate.

# After script has finished running and cleared all the undesired blobs and versions, you may renable soft-delete if needed.

# - Just run the script and your AAD credentials and the storage account name to list will be asked.

# - All other values should be defined in the script, under 'Parameters - user defined' section.

# ====================================================================================

# For any question, please contact Luis Filipe (Msft)

# ====================================================================================

Connect-AzAccount

CLS

#----------------------------------------------------------------------

# Parameters - user defined

#----------------------------------------------------------------------

$selectedStorage = Get-AzStorageAccount | Out-GridView -Title 'Select your Storage Account' -PassThru -ErrorAction Stop

$storageAccountName = $selectedStorage.StorageAccountName

# To Permanet deletions, disable Soft Delete for Blobs in the Storage account first.

$PERMANENT_DELETE_orListOnly ='List_Only' # Set "PERMANENT_DELETE" to permanent delete all soft deleted objects

# Set "List_Only" just to list without any deletion

# Set "Count_Only" just to list without any deletion

$containerName = '' # Container Name, or empty to all containers

$prefix = '' # Set prefix for scanning (optional)

$blobType = 'All Types' # valid values: 'Base' / 'Snapshots' / 'Versions' / 'Versions+Snapshots' / 'All Types'

$accessTier = 'All' # valid values: 'Hot', 'Cool', 'Archive', 'All'

# Select blobs before Last Modified Date (optional) - if at least one value is empty, current date will be used

$Year = ''

$Month = ''

$Day = ''

#----------------------------------------------------------------------

if($storageAccountName -eq $Null) { break }

#----------------------------------------------------------------------

# Date format

#----------------------------------------------------------------------

if ($Year -ne '' -and $Month -ne '' -and $Day -ne '')

{

$maxdate = Get-Date -Year $Year -Month $Month -Day $Day -ErrorAction Stop

}

else

{

$maxdate = Get-Date

}

#----------------------------------------------------------------------

# Format String Details in user friendy format

#----------------------------------------------------------------------

switch($blobType)

{

'Base' {$strBlobType = 'Base Blobs'}

'Snapshots' {$strBlobType = 'Snapshots'}

'Versions+Snapshots' {$strBlobType = 'Versions & Snapshots'}

'Versions' {$strBlobType = 'Blob Versions only'}

'All Types' {$strBlobType = 'All blobs (Base Blobs + Versions + Snapshots)'}

}

if ($containerName -eq '') {$strContainerName = 'All Containers (except $logs)'} else {$strContainerName = $containerName}

#----------------------------------------------------------------------

# Show summary of the selected options

#----------------------------------------------------------------------

function ShowDetails ($storageAccountName, $strContainerName, $prefix, $strBlobType, $accessTier, $maxdate)

{

# CLS

write-host " "

write-host "Azure Storage - Permanent Delete Soft-Deleted Blob objects"

write-host "-----------------------------------"

write-host "Storage account: $storageAccountName"

write-host "Container: $strContainerName"

write-host "Prefix: '$prefix'"

write-host "Blob Type: $strBlobType"

write-host "Blob Tier: $accessTier"

write-host "Last Modified Date before: $maxdate"

write-host "-----------------------------------"

}

#----------------------------------------------------------------------

# Filter and count blobs in some specific Container

#----------------------------------------------------------------------

function ContainerList ($containerName, $ctx, $prefix, $blobType, $accessTier, $maxdate)

{

$count = 0

$blob_Token = $null

$exception = $Null

$arrDeleted = "Name", "Content Length", "Snapshot Time", "Version ID", "Path"

$arrDeleted = $arrDeleted + "-------------", "-------------", "-------------", "-------------", "-------------"

write-host -NoNewline "Processing $containerName... "

do

{

$listOfBlobs = Get-AzStorageBlob -Container $containerName -IncludeDeleted -IncludeVersion -Context $ctx -ContinuationToken $blob_Token -Prefix $prefix -MaxCount 5000 -ErrorAction Stop

if($listOfBlobs.Count -le 0) {

write-host "No Objects found to list"

break

}

# Only Soft-Deleted objects with lastModifiedDate before or equal $maxdate

$listOfDeletedBlobs = $listOfBlobs | Where-Object { ($_.LastModified -le $maxdate) -and ($_.IsDeleted -eq $true)}

#Filter by Access Tier

if($accessTier -ne 'All')

{$listOfDeletedBlobs = $listOfDeletedBlobs | Where-Object { ($_.accesstier -eq $accessTier)} }

# Filter by Blob Type

switch($blobType)

{

'Base' {$listOfDeletedBlobs = $listOfDeletedBlobs | Where-Object { $_.IsLatestVersion -eq $true -or ($_.SnapshotTime -eq $null -and $_.VersionId -eq $null) } } # Base Blobs - Base versions may have versionId

'Snapshots' {$listOfDeletedBlobs = $listOfDeletedBlobs | Where-Object { $_.SnapshotTime -ne $null } } # Snapshots

'Versions+Snapshots' {$listOfDeletedBlobs = $listOfDeletedBlobs | Where-Object { $_.IsLatestVersion -ne $true -and (($_.SnapshotTime -eq $null -and $_.VersionId -ne $null) -or $_.SnapshotTime -ne $null) } } # Versions & Snapshotsk

'Versions' {$listOfDeletedBlobs = $listOfDeletedBlobs | Where-Object { $_.IsLatestVersion -ne $true -and $_.SnapshotTime -eq $null -and $_.VersionId -ne $null} } # Versions only

# 'All Types' # All - Base Blobs + Versions + Snapshots

}

#----------------------------------------------------------------------

# Uses REST API with SAS token call to permanent delete Blob

# disable Soft Delete for Blobs in the Storage account, first

#----------------------------------------------------------------------

#sas for the rest api call to undelete (be careful to remove the question mark in front of the token)

#-----------------------------------------

$CurrentTime = Get-Date

$StartTime = $CurrentTime.AddHours(-1.0)

$EndTime = $CurrentTime.AddHours(11.0) # Max 10 hours to undelete all soft deleted objects for each container

# Using Storage account key to generate a new SAS token ###

$sas = New-AzStorageContainerSASToken -Name $container -Permission $SASPermissions -StartTime $StartTime -ExpiryTime $EndTime -Context $ctx

$sas = $sas.Replace("?","")

# Undeleting the soft deleted blobs first, using Rest API, one by one

#-----------------------------------------

foreach($blob in $listOfDeletedBlobs)

{

if($PERMANENT_DELETE_orListOnly -eq "PERMANENT_DELETE") {

$uri = "https://" + $blob.BlobClient.Uri.Host + $blob.BlobClient.Uri.AbsolutePath + “?comp=undelete&" + $sas

Invoke-RestMethod -Method ‘Put’ -Uri $uri

# write-host $uri

}

$count++

# DEBUG

# write-host $blob.Name " Content-length:" $blob.Length " Access Tier:" $blob.accesstier " LastModified:" $blob.LastModified " SnapshotTime:" $blob.SnapshotTime " URI:" $blob.ICloudBlob.Uri.AbsolutePath " IslatestVersion:" $blob.IsLatestVersion " Lease State:" $blob.ICloudBlob.Properties.LeaseState " Version ID:" $blob.VersionID

# Creates a table to show the Soft Delete objects

#--------------------------------------------------

if($PERMANENT_DELETE_orListOnly -eq "List_Only") {

if($blob.SnapshotTime -eq $null) {$strSnapshotTime = "-"} else {$strSnapshotTime = $blob.SnapshotTime}

if($blob.VersionID -eq $null) {$strVersionID = "-"} else {$strVersionID = $blob.VersionID}

$arrDeleted = $arrDeleted + ($blob.Name, $blob.Length, $strSnapshotTime, $strVersionID , $blob.ICloudBlob.Uri.AbsolutePath)

}

# Permanent Delete those objects in one call

#-----------------------------------------

if($PERMANENT_DELETE_orListOnly -eq "PERMANENT_DELETE") {

$listOfDeletedBlobs | Remove-AzStorageBlob -Context $ctx

}

#----------------------------------------------------------------------

$blob_Token = $listOfBlobs[$listOfBlobs.Count -1].ContinuationToken;

}while ($blob_Token -ne $null)

write-host " Soft Deleted Objects Count: $count "

return $count, $arrDeleted

}

#----------------------------------------------------------------------

$wshell = New-Object -ComObject Wscript.Shell

$warning = "You selected to Permanent Delete Soft-Deleted blobs.`n"

$warning = $warning + "You cannot recover these blobs anymore.`n"

$warning = $warning + "To proceed on this, please make sure you have Blob Soft Delete feature disabled at Storage account level.`n"

$warning = $warning + "You may reenable Blob Soft Delete feature again after finishing this script.`n`n"

$warning = $warning + "Do you want to continue?"

$answer = $wshell.Popup($warning,0,"Alert",64+4)

if($answer -eq 7){exit}

ShowDetails $storageAccountName $strContainerName $prefix $strBlobType $accessTier $maxdate

$totalCount = 0

$SASPermissions = 'rwdl' # Permissions to SAS token do permanent Delete

$ctx = New-AzStorageContext -StorageAccountName $storageAccountName -UseConnectedAccount

$container_Token = $Null

#----------------------------------------------------------------------

# Looping Containers

#----------------------------------------------------------------------

do {

$containers = Get-AzStorageContainer -Context $Ctx -Name $containerName -ContinuationToken $container_Token -MaxCount 5000 -ErrorAction Stop

if ($containers -ne $null)

{

$container_Token = $containers[$containers.Count - 1].ContinuationToken

for ([int] $c = 0; $c -lt $containers.Count; $c++)

{

$container = $containers[$c].Name

$count, $arrDeleted, $exception = ContainerList $container $ctx $prefix $blobType $accessTier $maxdate

if ($PERMANENT_DELETE_orListOnly -eq 'List_Only' -and $count -gt 0) {

$arrDeleted | Format-Wide -Property {$_} -Column 5 -Force

}

$totalCount = $totalCount + $count

}

} while ($container_Token -ne $null)

write-host "-----------------------------------"

write-host "Total Count Permanent Deleted: $totalCount"

write-host "-----------------------------------"

#----------------------------------------------------------------------

This script was tested on PSVersion 5.1.19041.1682 and Az.Storage module 4.6.0, for Blob flat namespace and Hierarchical namespace (ADLS Gen2) Storage accounts.

Azure Storage data protection features:

Blob Snapshot

A snapshot is a read-only version of a blob that's taken at a point in time. A snapshot of a blob is identical to its base blob, except that the blob URI has a DateTime value appended to the blob URI to indicate the time at which the snapshot was taken. A blob can have any number of snapshots. Snapshots persist until they are explicitly deleted, either independently or as part of a Delete Blob operation for the base blob.

Blob versioning

Azure Blob storage versioning lets you automatically maintain previous versions of an object. When blob versioning is enabled, you can access earlier versions of a blob to recover your data if it is modified or deleted.

Soft delete for blobs

Blob soft delete protects an individual blob, snapshot, or version from accidental deletes or overwrites by maintaining the deleted data in the system for a specified period of time. During the retention period, you can restore a soft-deleted object to its state at the time it was deleted. After the retention period has expired, the object is permanently deleted.

Conclusion:

Azure Portal and Azure Storage Explorer can list the Soft Deleted Blobs in some container, and from there can be selected to permanent deletion, but only container one by one; Also to do the same for Soft Deleted Snapshots or Versions, needs to be done only at blob level, one by one.

This PowerShell script should help you to permanent Delete objects (Blobs, Snapshots and Versions) in Soft Delete state, based on more often filters used.

Related documentation:

Soft delete for blobs

Blob versioning

Blob Snapshots

Other techcommunity articles:

Azure Storage Blob Count & Capacity usage Calculator

Calculate the size/capacity of storage account and it services (Blob/Table)

Analyzing Storage Capacity

Other PowerShell scripts:

Calculate the total billing size of a blob container

Calculate the size of a blob container with PowerShell

I hope this can be useful!!!