Migrate data to Azure Managed Lustre retaining POSIX attributes

[Guest jasonschuff]

[HEADING=1]Introduction[/HEADING]

 

This blog will utilize manual steps on for exporting data to blob storage in order to retain specific POSIX attributes. The exporting of data is achieved using the Lustre HSM (Hierarchical Storage Management) interface. The Managed Lustre system will need to have HSM enabled and setup in advance. See this article: Blob integration

 

For more information around setting up automatic synchronization to Azure BLOB Storage for Azure Managed Lustre refer to this blob post: Automatic Synchronization to Azure BLOB Storage.

 

 

 

[HEADING=1]Connect client to the Lustre file system[/HEADING]

 

Client machines running Linux can access Azure Managed Lustre directly. See the following article that details the client prerequisites: Connect client to the file system

 

 

 

To mount lustre:

 

sudo mount -t lustre -o noatime,flock <MGS_IP>@tcp:/lustrefs /<client_path>

 

[HEADING=1]Migrate data retaining POSIX attributes[/HEADING]

 

Once you have a client that is connected to the file system you can now copy data directly into that file system.

 

• Assuming the source location is /mydata and the destination lustre file system is /lustredata

  
  

• The -a option preserves all POSIX attributes, such as ownership, permissions, timestamps, symlinks, etc. See the rsync manual page for more details.

  
  

 

To copy data into lustre:

 

 

rsync -av /mydata /lustredata

 

 

Note: When migrating data to AMLFS, ensure that the total storage used does not exceed the system’s allowed capacity. If migrating more storage than allowed by the file system capacity then files will need to be archived and released to blob storage as needed before continuing the data migration.

 

[HEADING=1] [/HEADING]

[HEADING=1]Export data and attributes to blob storage[/HEADING]

 

Once the files have been copied into the Lustre File system, now utilize the export job process in order to write those files as well as the POSIX attributes as metadata to the blob storage container. This process includes using the export jobs with archive process.

 

 

 

[HEADING=1]Which POSIX attributes are retained during an export job? [/HEADING]

 

When you export files from your Azure Managed Lustre system to blob storage there are additional attributes that are saved as metadata inside the blob storage as shown here: Metadata for exported files. The following attributes may be written as metadata to each object in blob storage depending on the type of object:

 

 

 

 

 

Parameter	Description
modtime	The last modification time of the file
owner	The owner of the file
group	The group owner of the file
permissions	The existing permissions of the file
hdi_isfolder	If object is a folder, this value is set to true. Name corresponds with folder name.

 

 

 

The metadata will appear in the blob attributes in storage as shown here:

 

 

 

 

 

[HEADING=1]Restoring data into a new Azure Managed Lustre File System: [/HEADING]

 

 

 

Now that the blob storage contains the attributes for each blob object including permissions and ownership of each file and directory, this data can be imported into any new Azure Managed Lustre file system and retain those attributes as it does. Follow these steps in order to import data using import jobs.

 

 

 

Note: This step is only required when setting up a new Azure Managed Lustre File System. This is not required for utilizing the existing AMLFS the data was originally copied to.

 

[HEADING=1] [/HEADING]

 

References

 

• Azure Managed Lustre File System Documentation
  
• Azure Managed Lustre with Automatic Synchronisation to Azure BLOB Storage
  
• GitHub repositories
  
  • lfsazsync
    
  • Robinhood
    
  • lemur
    

 

Continue reading...