Migrating SLURM Job Accounting from Azure Database for MariaDB to MySQL Flexible Server

[Guest egmsft]

[HEADING=1]Overview[/HEADING]

 

Azure CycleCloud (CC) is an enterprise-friendly tool for orchestrating and managing High-Performance Computing (HPC) environments on Azure. With CycleCloud, users can provision infrastructure for HPC systems, deploy familiar HPC schedulers, deploy/mount filesystems and automatically scale the infrastructure to run jobs efficiently at any scale. Azure CycleCloud is targeted at HPC administrators and users who want to deploy an HPC environment with a specific scheduler in mind.

 

 

 

One of the supported schedulers is SLURM, which stands for Simple Linux Utility for Resource Management. SLURM is an open source, scalable and fault-tolerant cluster management and job scheduling system for Linux clusters of any size. SLURM can allocate resources to users for a duration of time, manage workloads, provide accounting and monitoring, and support parallel and distributed computing applications.

 

 

 

SLURM job accounting is a feature that allows users to collect and report various metrics about the jobs that run on the cluster. SLURM job accounting can help users optimize their resource utilization, monitor their quotas, and track their costs. To enable job accounting, you need to configure and run a SLURM database daemon (slurmdbd) that can store the job information in a database.

 

 

 

One of the common choices for the database backend for SLURM job accounting is Azure Database for MariaDB. However, Azure Database for MariaDB will be retired on September 19, 2025, and users are encouraged to migrate their data and applications to Azure Database for MySQL Flexible Server.

 

 

 

In this blog, we will walk you through the steps to migrate your SLURM job accounting data from Azure Database for MariaDB to Azure Database for MySQL Flexible Server in Azure CycleCloud.

 

 

 

[HEADING=1]Requirements/Versions[/HEADING]

• 
  CycleCloud Server (CC version used is 8.6.2)
   
   
  
• 
  Slurm Cluster
   
  
  • 
    CycleCloud project used is 3.0.5
     
     
    
  • 
    Slurm version used is 23.02.6-1
     
    

  [*]

  A source instance of Azure Database for MariaDB

   

   

  [*]

  A target instance of Azure Database for MySQL Flexible Server

   

   

  [*]

  A Linux VM with access to both the MariaDB and MySQL instances

   

  • 
    We used the CycleCloud VM
     
     
    
  • 
    Running on Alma Linux 8.7
     
    

[HEADING=1]Migration Procedure[/HEADING]

1. 
   Install required packages
    
    
   
2. 
   Back up your SLURM Accounting DB in Azure Database for MariaDB
    
    
   
3. 
   Restore the backup to your Azure Database for MySQL Flexible Server
    
    
   
4. 
   Create SLURM user in Azure Database for MySQL and grant privileges
    
    
   
5. 
   Update your CycleCloud cluster configuration for SLURM Job Accounting
    
   

 

To perform the migration, you will need a VM that can connect to both the source and target databases. In this blog post, we will use the CycleCloud VM as an example, but you can use any VM that meets this requirement.

 

 

 

[HEADING=2]Step 1: Install required packages[/HEADING]

 

MySQL Shell (mysqlsh) is an advanced command-line client for MySQL that supports various modes of operation, such as SQL, JavaScript, and Python, and enables interactive and batch execution of queries and scripts. This utility will be used to perform the transfer of the SLURM job accounting DB from MariaDB to MySQL, and requires the installation of the following additional packages:

 

• mysql
  
• mysql-shell
  

 

To install the required packages, you will need to run the following commands on the migration VM:

 

 

 

sudo yum install -y msql 
wget https://dev.mysql.com/get/mysql84-community-release-el8-1.noarch.rpm /tmp 
sudo yum localinstall /tmp/mysql84-community-release-el8-1.noarch.rpm 
sudo yum install -y mysql-shell 

 

 

 

[HEADING=2]Step 2: Back up your SLURM Accounting DB in Azure Database for MariaDB[/HEADING]

 

At this point, it is recommended for your SLURM cluster to be terminated to ensure that the accounting DB is no longer being updated.

 

 

 

Connect to MariaDB and check the size of the SLURM accounting DB:

 

 

 

mariadbname=jmslurmmariadbeus #update with your specific db name
mariadbusername=themorey #update with your specific db user name
mysqlsh --uri ${mariadbusername}%40${mariadbname}@${mariadbname}.mariadb.database.azure.com:3306 

 

 

 

OPTIONAL - to check the size of the DB:

 

 

 

SELECT table_schema AS "Database",  
ROUND(SUM(data_length + index_length) / 1024 / 1024, 2) AS "Size (MB)"  
FROM information_schema.TABLES  
GROUP BY table_schema; 

 

 

 

Sample Output:

 

 

 

 

Check free space on VM to determine where backup can be stored:

 

 

 

[iCODE]df -h[/iCODE]

 

 

 

The output of the df -h command shows that we have enough space for us to dump our backup on our local disk:

 

 

 

 

Create a directory for your MariaDB backup. Note, the target directory for the backup must be empty:

 

 

 

sudo mkdir -p /backup/mysql/mariadb_backup/ 
sudo chown -R $(whoami) /backup 

 

 

 

Log back in to MariaDB using mysqlsh:

 

 

 

mariadbname=jmslurmmariadbeus #update with your specific db name
mariadbusername=themorey #update with your specific db user name
mysqlsh --uri ${mariadbusername}%40${mariadbname}@${mariadbname}.mariadb.database.azure.com:3306 

 

 

 

Switch to Javascript mode on the mysql shell:

 

 

 

[iCODE]\js [/iCODE]

 

 

 

Run the dumpUtil to take a full backup:

 

 

 

[iCODE]util.dumpInstance("/backup/mysqlsh/mariadb_backup",{threads: 16, showProgress: true, users: false}) [/iCODE]

 

 

 

Sample Output:

 

 

 

 

After the dumpUtil command is complete, you can exit out of the MySQL shell ( \q; ) and view the backup on your VM by listing the contents of the backup directory. The backup consists of a series of JSON files that contain the schema and data of the instance.

 

 

[HEADING=2] [/HEADING]

[HEADING=2]Step 3: Restore the backup to your Azure Database for MySQL Flexible Server[/HEADING]

 

Now that you have a backup of your accounting data from MariaDB, you can restore it to your Azure Database for MySQL Flexible Server.

 

 

 

To do this, first login to your Azure Database for MySQL Flexible Server using MySQL shell. Note, the syntax to establish the connection is slightly different than the one used to connect to MariaDB:

 

 

 

mysqldbname=jmmysqlfrommariadb #update with your specific db name
mysqldbusername=themorey #update with your specific db user name
mysqlsh --uri ${mysqldbusername}@${mariadbname}.mysqldb.database.azure.com:3306 

 

 

 

List the databases in your MySQL server to confirm that the slurm_acct_db does not exist:

 

 

 

[iCODE]show databases;[/iCODE]

 

 

 

Sample Output:

 

 

 

 

Switch to javascript mode and run the loadDump utility to import the MariaDB dump files:

 

 

 

\js
util.loadDump("/backup/mysqlsh/mariadb_backup", {threads: 16, showProgress: true, ignoreVersion: true}) 

 

 

 

Sample Output:

 

 

[HEADING=2] [/HEADING]

[HEADING=2]Step 4: Create SLURM user in Azure Database for MySQL and grant privileges[/HEADING]

 

When we used the util.loadDump() function to restore the data from the MariaDB backup, we only restored the SLURM accounting database and not the user accounts. This means that the SLURM user account that was used to access the database in MariaDB does not exist in the Azure Database for MySQL instance.

 

 

 

To fix this, we need to switch back to SQL mode, create a new user account with the same name and password as the SLURM user in MariaDB, and grant privileges:

 

 

 

\sql
create user slurm@'%'; 
ALTER USER slurm IDENTIFIED BY 'P@ssw0rd!@#'; 
grant usage on *.* to slurm@'%'; 
grant all privileges on slurm_acct_db.* to slurm@'%'; 
flush privileges; 
SHOW GRANTS FOR 'slurm'@'%'; 

 

 

 

Sample Output:

 

 

[HEADING=2] [/HEADING]

[HEADING=2]Step 5: Update your CycleCloud cluster configuration for Slurm Job Accounting[/HEADING]

 

After creating the user and granting the privileges, we are now ready to connect our cluster to the new MySQL server instance. To do this, we need to navigate to the advanced settings of the CycleCloud SLURM cluster and update following details:

 

• Slurm DBD URL = URL for MySQL Server instance
  
• Slurm DBD User = SLURM accounting DB user
  
• Slurm DBD Password = SLURM accounting DB password
  
• SSL Cert URL = https://dl.cacerts.digicert.com/DigiCertGlobalRootCA.crt.pem
  

 

Sample Output:

 

 

 

 

After updating the required details for your SLURM Cluster, save the configuration and start the cluster.

 

 

 

 

Once the cluster is operational, run "sacct" to verify that you can view the historical information about the jobs that have been ran in the cluster before the migration took place:

 

 

 

 

NOTE: Slurm accounting command (sacct) defaults to the same day. You may need to expand the search criteria to see jobs older than current day. For example, “sacct -S 060124“ will show jobs starting from 6/1/2024 until the current day.

 

 

 

[HEADING=2]SUMMARY[/HEADING]

 

The impending retirement date for MariaDB does not require you to abandon your Slurm accounting history. An Azure MySQL Flexible Server can be the solution moving forward while also loading the historical data from MariaDB.

 

 

 

[HEADING=2]REFERENCES[/HEADING]

 

 

Setting up SLURM Job Accounting with Azure CycleCloud and Azure Database for MySQL Flexible Server - Microsoft Community Hub

 

Migrating from Azure Database for MariaDB to Azure Database for MySQL - Microsoft Community Hub

 

Continue reading...