Field tips for AKS storage provisioning

[Guest varghesejoji]

Introduction

 

 

In an Azure Kubernetes (AKS) cluster, Pods can access physical storage resources such as disks or volumes using Persistent Volumes (PV). To use these resources, Pods need to make a Persistent Volume Claim (PVC), which requests a specific amount of storage from a storage class. This claim can then be matched to an available Persistent Volume. Azure offers several storage solutions that can be used to provision Persistent Volumes in an AKS cluster.

 

 

 

This article will provide real-world guidance on securely using Container Storage Interface (CSI) drivers to provision Azure File Shares and Azure Blob storage in an AKS cluster.

 

 

 

Prerequisites

 

 

Container Storage Interface (CSI) drivers provides a standard way of interfacing with storage systems, allowing AKS to use various storage providers. With the Azure Blob and File CSI (Container Storage Interface) drivers, AKS clusters can use Azure File shares and Blob storage containers as persistent volumes.

 

To enable Azure Blob CSI driver on a new or existing AKS cluster, use this link. To enable Azure File CSI driver use this link.

 

You can find the GitHub link for the solutions mentioned below at this link.

 

 

 

Provisioning Azure SMB File share

 

 

Managed identities enable you to authenticate your application or service to Azure services, without the need to manage credentials explicitly. With User-assigned Managed Identity (UMI) you can associate an identity with a specific AKS pod, allowing the pod to access Azure SMB storage resources securely. This will be illustrated below.

 

 

 

When using Azure SMB File Share in AKS, below are key considerations to ensure secure and reliable access to your data.

 

1. Use the AKS-default User-Assigned Managed Identity (MI) to assign the required RBAC roles to access the Storage account. You can also assign this MI to the resource group containing the Storage account.
   
2. Enabling "Secure transfer required" in the Storage account configuration has no dependencies for SMB, unlike NFS file shares.
   
3. Ensure that the Storage account configuration has 'Allow storage account key access' set to Enabled. This setting is needed when mounting SMB File share by the application pod.
   
4. Finally, ensure that the Storage class setting storeAccountKey is set to "false". This allows the CSI driver to leverage the MI to get the storage account key. This in turn mounts the Azure File Share with the obtained key.
   

 

 

 

Steps to use Azure SMB File Share with AKS default User-assigned Managed Identity (UMI) are summarized below, with details found in this GitHub link customized for SMB. Link to all SMB provisioning files for AKS can be found here.

 

1. Create a storage account and SMB file share using Azure CLI commands. For the Azure Storage account set "Allow storage account key access" and "Secure transfer required" to enabled. This is done using CLI commands.
   
2. Assign the "Storage Account Contributor" RBAC role to the AKS default managed identity, to grant kubelet identity permission on the storage account hosting the file share.
   
3. Generate custom files for Storage Class and PV. Then use the generated files to deploy the storage class, PV, PVC, and Application Pods.
   
4. Verify that the file share is mounted successfully. In addition, create a file in the mounted SMB share to test.
   
5. To delete the SMB deployment, delete the Application Pods, PV, PVC, and Storage Class in that order using the kubectl delete command.
   

 

 

 

Provisioning Azure NFS File share

 

 

Azure NFS File Share is a persistent storage option that presents a distributed file system, that allows several pods to read and write data to a common storage volume. This is particularly useful for stateful applications such as databases that necessitate long-term data persistence.

 

For the configuration below, the Azure Files service provides the underlying storage, while the Azure File CSI driver allows the NFS share to be mounted as a Kubernetes Persistent Volume.

 

 

 

When using Azure NFS File Share in AKS, below are some important considerations regarding configuration.

 

1. NFS does not require Managed Identity (MI), since it does not need to obtain keys for mounting shares. This contrasts with SMB, which requires MI to obtain keys for authentication.
   
2. 'Secure transfer required' should be set to Disabled in the Storage account configuration for NFS shares, for reasons explained in link.
   
3. Since NFS shares do not require account key access, so 'Allow storage account key access' can be set to Disabled. This eliminates the need for storeAccountKey in the Storage class.
   
4. The storage account used for NFS should use Premium LRS/ZRS.
   
5. Private Endpoint (PE) is required for NFS under the specified conditions. When PE is defined, its server name should be used in the Storage class file using keyword ‘server’.
   
6. Installing the nfs-common package is required for NFS and port 2049 must be opened to the storage account.
   

 

 

 

Steps to use Azure NFS File Share are summarized below, with details found in this GitHub link customized for NFS. Link to all NFS provisioning files for AKS can be found here. As mentioned above, no authentication is required for NFS, so MI will not be needed here.

 

1. Create a storage account and NFS file share with the required parameters using Azure CLI commands. Ensure that "Allow storage account key access" and "Secure transfer required" are disabled. This is done using CLI commands.
   
2. Generate custom files for Storage Class and PV for NFS protocol.
   
3. Deploy the Storage class, PV, and PVC and check if the PVC is bound. Deploy the application pod and verify if it is running.
   
4. To delete the NFS deployment, delete the App Pods, PV, PVC, and Storage Class in that order. This is using the kubectl delete command to delete the corresponding resources or delete the entire namespace.
   

 

 

 

Provisioning Azure Blob storage using BlobFuse

 

 

Azure Blob Storage provides an object storage solution that allows users to store and access enormous amounts of unstructured data. BlobFuse is a protocol that enables users to access Azure blob container content and mount Blob storage as a file system on Linux or Windows OS.

 

BlobFuse and NFS CSI drivers should have been installed as part of the pre-requisites and validated by displaying Storage classes using ‘kubectl get sc’. If installed, this should display azureblob-fuse-premium and azureblob-nfs-premium.

 

 

 

When using BlobFuse to access and mount AKS Blob storage on AKS, below are some important considerations regarding configuration.

 

1. Create a new User-assigned Managed Identity (MI) and assign required RBAC roles for accessing the Storage account with Blob container. The AKS default User-assigned MI can also be used.
   
2. To specify that authentication for BlobFuse will be handled through MSI (Managed Service Identity), you should set the value of AzureStorageAuthType to "msi" in the PV file and assign the MI resource ID to AzureStorageIdentityResourceID.
   
3. Other options using access keys (Key) and SAS (Shared Access Signature) token are not recommended for production, as they require manual entries in the PV file, which is not secure.
   

 

 

 

Steps to use Managed Identity for accessing Azure Blob Storage using BlobFuse on AKS are summarized below. Implementation details can be found in this GitHub link customized for BlobFuse. Link to all BlobFuse provisioning files for AKS can be found here.

 

1. Create an Azure Storage account with type=BlockBlobStorage and create a container within the Storage account.
   
2. Create a Managed Identity and assign ‘Storage Blob Data Contributor’ role to the Managed Identity for accessing the Storage account.
   
3. Attach Managed Identity to AKS VMSS (Virtual Machine Scale Set), specifically for Node pool(s) running CSI driver i.e., csi-blob-node-xxxxx.
   
4. Configure Persistent Volume (PV) with managed identity in the PV file and generate file using CLI.
   
5. Deploy the application by running kubectl apply command with appropriate YAML files for the Persistent Volume Claim (PVC), Application Pod and Persistent Volume (PV).
   
6. Verify the Blob storage mounted successfully by checking the mounted directory within the Pod.
   

 

 

 

Conclusion

 

 

This article provides field-tested tips for statically provisioning storage solutions in an Azure Kubernetes (AKS) cluster. It focuses on provisioning Azure NFS and SMB File Share using NFS and SMB protocols, as well as Azure Blob storage using Blob Fuse. It is essential to note that the Container Storage Interface (CSI) driver is a prerequisite for interfacing with storage systems. It also highlights the pre-requisites for each storage solution and key considerations for each solution's secure and reliable access to data. Finally, this article provides implementation steps for using each storage solution with AKS. Thank you for taking the time to read this blog.

 

 

 

References

 

 

Kubernetes documentation on Persistent Volumes

 

Azure Blob CSI Driver examples – Github

 

Azure File CSI Driver examples - Github

 

Provisioning Azure Blob storage for use with AKS

 

Provisioning Azure NFS File Share for use with AKS

 

 

 

Disclaimer

 

 

The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without a warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

 

Continue reading...