Using Azure online disk expansion for SAP

[Guest RobertBiro]

For SAP workloads, most systems utilize managed disk storage for SAP software and the database. Until now, resizing the disks required a downtime as the VM had to be stopped and deallocated. However, a new feature recently generally available in Azure allows online expansion of data disks, without any interruption.

 

Resize without downtime | Azure Docs

 

Generally available: Live resize for Premium SSD and Standard SSD Disk Storage | Azure Updates

 

 

 

Simply put, you can increase size of the data disks in Azure. You then inform the operating system of the VM about the changes and you perform few online commands within the OS to enlarge your filesystems, giving you applications more space without any interruption in operations. Note that a size reduction of disks is not possible, online nor offline, only an increase of size is possible. For online disk expansion, consider performing this activity at a time when your SAP system is less loaded, just as a precaution.

 

 

 

In this blog post we will discuss the implications of online disk expand feature for SAP workloads and best practices to follow. Note - this blog post deals with PremiumSSD_v1 storage and does not cover PremiumSSD_v2 where online disk expansion functionality is currently not yet available.

 

 

 

Resizing disks in Azure – how do I do it?

 

 

 

 

Azure documentation lists the steps required to enlarge data disks – for Windows here and for Linux here. The documentations also contains details how to register your Azure subscription for the feature, a quick onetime activity. Observe the limitations of the online expansion, as documented.

 

 

 

In essence you select the disk to be resized in portal (or CLI, ARM and API) – the disk itself or selecting through the VM -> Disks pane – and select the new, larger size. Once this completes – few minutes of wait time for request to finish – your disk is larger, and you can carry on with activities within the operating system.

 

Note that currently OS disks cannot be enlarged online and require deallocation of VM. Also expansion is currently possible only upto 4TiB, crossing the 4TiB boundary requires downtime. But other than this for SAP systems there are no practical limitations as the feature works also with M family of VMs and write accelerated disks.

 

 

 

Figure 1 - Example of a disk resize via Azure Portal

 

 

 

With premium storage v1, each disk size has its own minimal performance specifications – how many IOPS or how much I/O throughput its limited to or can burst to. See the details of each disk in Azure portal when you resize, or in this table. So, a new, larger disk will likely provide not just more capacity – your primary goal, but also more performance.

 

 

 

If capacity is not your goal but more performance only – you can leverage disk performance tiers to keep the same capacity but more performance, particularly when the performance need is only temporary – like an SAP system upgrade, migration or month-end peak. Performance tier change is also an online activity and documented here. After a performance tier change, as opposed to capacity change, there is no further action in OS, Azure simply raises your IOPS/throughput limits. And you can return back to your original performance tier matching your disk capacity after a cooldown period of 12 hours. Similar to performance tiers, on-demand bursting is an option to look into for additional performance.

 

 

 

Identifying disks inside the VM

 

 

 

 

Once the disks are enlarged within Azure – online or offline – they need to be increased within the OS as well to utilize the larger disk. To identify the disks within the OS, the LUN number from Azure is needed. If using the portal, you can see this on the VM’s disk blade.

 

 

 

Figure 2 - Disk LUNs visible on the VM's disk blade

 

 

 

Within the OS on the VM – we cover Linux in this blog post – you can see the disk mapping from LUN to the usual Linux device names sdc, sdd etc in /dev/disk/azure/scsi1 directory.

 

ls -l /dev/disk/azure/scsi1

 

total 0

 

lrwxrwxrwx 1 root root 12 Dec 21 08:53 lun0 -> ../../../sdc

 

lrwxrwxrwx 1 root root 12 Dec 21 08:53 lun1 -> ../../../sdd

 

lrwxrwxrwx 1 root root 12 Dec 21 08:53 lun2 -> ../../../sde

 

lrwxrwxrwx 1 root root 12 Dec 21 08:53 lun3 -> ../../../sdf

 

lrwxrwxrwx 1 root root 12 Dec 21 08:53 lun4 -> ../../../sdg

 

lrwxrwxrwx 1 root root 12 Dec 21 08:53 lun5 -> ../../../sdh

 

lrwxrwxrwx 1 root root 12 Dec 21 08:53 lun6 -> ../../../sdi

 

lrwxrwxrwx 1 root root 12 Dec 21 08:53 lun7 -> ../../../sdj

 

lrwxrwxrwx 1 root root 12 Dec 21 08:53 lun8 -> ../../../sdk

 

 

 

LVM volume extension

 

 

 

 

Typically, LVM is used for both the OS filesystems such as root, tmp as well as SAP and database filesystems. The disks (sdc, sdd, ...) are used as LVM physical devices.

 

If you are NOT using LVM and your filesystem is directly on the physical disk, you only have to rescan the scsi bus (see next code section) and resize the filesystem as described below. If the filesystem is on a partition, you will need to resize the partition with tools like parted/gparted. LVM usage is strongly encouraged even for simple single-disk filesystems on Linux.

 

 

 

All LVM commands are online and without interruption for your applications, although still preferable to run them during a quieter load on the SAP environment. For our LVM example, let’s assume we expanded lun8 which maps as per example above to the /dev/sdk disk. To expand the physical device we call

 

 

 

echo 1 | sudo tee /sys/class/block/sdk/device/rescan

 

sudo pvresize /dev/disk/azure/scsi1/lun8

 

 

 

Our first command above is to notify the OS to rescan lun8/sdk disk, as this will then allow the OS to see the size change. Other tools such as rescan-scsi-bus.sh script by SUSE or other actions on /sys/class/... objects are also possible. Example output from dmesg utility that OS after the rescan picks up the larger capacity:

 

sudo dmesg |tail -5

 

[ 472.032503] sd 5:0:0:8: [storvsc] Add. Sense: Capacity data has changed

 

[ 472.036368] sd 5:0:0:8: Capacity data has changed

 

[ 734.817639] sd 5:0:0:8: [sdk] 268435456 512-byte logical blocks: (137 GB/128 GiB)

 

[ 734.824609] sd 5:0:0:8: [sdk] 4096-byte physical blocks

 

[ 734.828808] sdk: detected capacity change from 68719476736 to 137438953472

 

 

 

A second command then resizes the physical volume. To confirm, we can run pvs or pvdisplay

 

sudo pvdisplay /dev/disk/azure/scsi1/lun8 -C

 

PV VG Fmt Attr PSize PFree

 

/dev/sdk vg_hana_usrsap lvm2 a-- 128.00g 64.00g

 

In this example we resized a P6 (64GiB) to a P10 (128GiB) disk in Azure earlier and thus have now 64GiB unallocated, free on this disk as shown above, the other 64GiB were already allocated/used.

 

 

 

Next for LVM is to extend logical volume and filesystem – the volume group uses the physical volumes (just enlarged) and doesn’t need any changes. For this we use lvextend, example:

 

sudo lvextend /dev/mapper/vg_hana_usrsap-lv_hana_usrsap -r -l +100%FREE

 

Size of logical volume vg_hana_usrsap/lv_hana_usrsap changed from 64.00 GiB (16383 extents) to 128.00 GiB (32767 extents).

 

Logical volume vg_hana_usrsap/lv_hana_usrsap successfully resized.

 

meta-data=/dev/mapper/vg_hana_usrsap-lv_hana_usrsap isize=512 agcount=4, agsize=4194048 blks

 

= sectsz=4096 attr=2, projid32bit=1

 

= crc=1 finobt=0 spinodes=0 rmapbt=0

 

= reflink=0

 

data = bsize=4096 blocks=16776192, imaxpct=25

 

= sunit=0 swidth=0 blks

 

naming =version 2 bsize=4096 ascii-ci=0 ftype=1

 

log =internal bsize=4096 blocks=8191, version=2

 

= sectsz=4096 sunit=1 blks, lazy-count=1

 

realtime =none extsz=4096 blocks=0, rtextents=0

 

data blocks changed from 16776192 to 33553408

 

In our example we specify to use all available free space for the extension, you can specify different values, e.g. if you want to extend by absolute number of GiB simply use -L +<size>g, like -L +5g to extend by 5GiB. See lvextend man page for details.

 

Executing lvextend with -r switch, there is no need to call filesystem extension, its already executed as part of the lvextend call. If you want to separate logical volume extension and file system extension, remove -r and run respective file system command such as xfs_growfs following lvextend.

 

 

 

Multi-disk striped LVM logical volumes

 

 

 

 

So far, we have handled situation of a single disk, like that of /usr/sap or /hana/shared for example.

 

What if you want to extend the space on the database data disks, like Oracle’s sapdata or HANA /hana/data? In Azure typically these filesystems are spread on multiple managed disks, stripped equally over them via LVM.

 

For Azure disk resource, simply enlarge the managed disks one by one to the desired size. Once done, within the OS again rescan the scsi bus to inform the OS of larger size and extend the physical volumes. For multi-striped volumes, ensure you keep equal new size for all disks within the volume group as otherwise performance could be impaired. Do not add new disks to the stripeset either. This means expansion is always a multiple of x2 increase.

 

 

 

Example below, /hana/data consists of 4 disks of 256GiB each, LUN 0-3 or sdc/sdd/sde/sdf disks. To extend use four rescan commands and extend commands or shortened form where either character from [cdef] or [0123] is the trailing character of the four disks.

 

echo 1 | sudo tee /sys/class/block/sd[cdef]/device/rescan

 

sudo pvresize /dev/disk/azure/scsi1/lun[0123]

 

 

 

We can verify all 4 disks were extended

 

sudo dmesg | grep 'detected capacity change'

 

[734.280680] sdc: detected capacity change from 274877906944 to 549755813888

 

[734.305356] sdd: detected capacity change from 274877906944 to 549755813888

 

[734.328573] sde: detected capacity change from 274877906944 to 549755813888

 

[734.348957] sdf: detected capacity change from 274877906944 to 549755813888

 

Or with

 

sudo pvs

 

 

 

Only one logical volume is here, regardless if multiple physical volumes form the volume group or just one. The command is identical as the example with /usr/sap earlier. The -r switch calls filesystem resize, immediately after so space is usable after this.

 

sudo lvextend /dev/mapper/vg_hana_data-lv_hana_data -r -l +100%FREE

 

Using stripesize of last segment 256.00 KiB

 

Size of logical volume vg_hana_data/lv_hana_data changed from 1023.98 GiB (262140 extents) to 2.00 TiB (524284 extents).

 

Logical volume vg_hana_data/lv_hana_data successfully resized.

 

 

 

Now that we have covered Azure and LVM, we can move to the filesystem. (if using software raid instead of LVM, see section below).

 

 

 

Handling filesystems

 

 

 

 

When using LVM, switch -r for lvextend automatically calls the respective filesystem extension and there is no action needed, the filesystem is extended and ready to use.

 

 

 

However, if you are not using LVM but disks/partitions directly (why?) you will need to call respective filesystem resize. We will cover xfs and ext4 as they are most common and supported filesystems for SAP.

 

 

 

XFS

 

 

 

 

With an enlarged disk/partition (or logical volume volume if not called with -r switch for lvextend earlier), you can simply run xfs_growfs on your filesystem. This is, same as Azure disk resize and LVM operations, fully online and without interruption for your applications.

 

As an example, lets resize the filesystem of earlier expanded lun8/vg_hana_usrsap-lv_hana_usrsap.

 

sudo xfs_growfs /usr/sap

 

meta-data=/dev/mapper/vg_hana_usrsap-lv_hana_usrsap isize=512 agcount=4, agsize=4194048 blks

 

= sectsz=4096 attr=2, projid32bit=1

 

= crc=1 finobt=0 spinodes=0 rmapbt=0

 

= reflink=0

 

data = bsize=4096 blocks=16776192, imaxpct=25

 

= sunit=0 swidth=0 blks

 

naming =version 2 bsize=4096 ascii-ci=0 ftype=1

 

log =internal bsize=4096 blocks=8191, version=2

 

= sectsz=4096 sunit=1 blks, lazy-count=1

 

realtime =none extsz=4096 blocks=0, rtextents=0

 

data blocks changed from 16776192 to 33553408

 

 

 

df -h /usr/sap

 

Filesystem Size Used Avail Use% Mounted on

 

/dev/mapper/vg_hana_usrsap-lv_hana_usrsap 128G 240M 128G 1% /usr/sap

 

And you are done, more capacity is available to you, without any application downtime.

 

 

 

Ext4

 

 

 

 

Ext4 filesystem also allows easy online expansion. The procedure is similar the one from xfs. Once the disk/partition/logical volume is enlarged, you can online execute resize2fs to enlarge the filesystem to the new size. One difference to xfs is that resize2fs expects the device and not the mounted filesystem as an argument. As an example below, lun9/sapmnt was increased from 32 to 64GiB

 

sudo resize2fs /dev/mapper/vg_sapmnt-lv_sapmnt

 

resize2fs 1.43.8 (1-Jan-2018)

 

Filesystem at /dev/mapper/vg_sapmnt-lv_sapmnt is mounted on /sapmnt; on-line resizing required

 

old_desc_blocks = 4, new_desc_blocks = 8

 

The filesystem on /dev/mapper/vg_sapmnt-lv_sapmnt is now 16776192 (4k) blocks long.

 

 

 

df -h /sapmnt

 

Filesystem Size Used Avail Use% Mounted on

 

/dev/mapper/vg_sapmnt-lv_sapmnt 63G 52M 60G 1% /sapmnt

 

Once done, more capacity available for applications.

 

 

 

What if I’m using Azure Disk Encryption lvm-on-crypt?

 

 

 

 

Should you be using Azure Disk Encryption – an in-OS encryption – instead of recommended SSE-CMK encryption for SAP environments, you can extend disk, physical volume, before extending the lvm and filesystem, you execute cryptsetup resize for the disk(s) you want the resize, as documented here.

 

Example to resize lun7/sdj:

 

sudo cryptsetup resize /dev/sdj

 

Follow then with lvm logical volume extension and filesystem extension as described earlier, for details see linked documentation.

 

With SSE-CMK encryption as opposed to ADE there are no additional steps needed as encryption is entirely transparent to the OS (lvm/filesystem).

 

 

 

Linux software raid/mdadm instead of LVM

 

 

 

 

If you are using Linux software raid, mdadm, instead of LVM, here are the steps needed to expand the raid array. First the partitions underlying the raid need to be enlarged, here using parted:

 

sudo parted /dev/sdd

 

GNU Parted 3.1

 

Using /dev/sdd

 

Welcome to GNU Parted! Type 'help' to view a list of commands.

 

(parted) unit GiB

 

(parted) print free

 

Model: Msft Virtual Disk (scsi)

 

Disk /dev/sdd: 512GiB

 

Sector size (logical/physical): 512B/4096B

 

Partition Table: msdos

 

Disk Flags:

 

 

 

Number Start End Size Type File system Flags

 

0.00GiB 0.00GiB 0.00GiB Free Space

 

1 0.00GiB 256GiB 256GiB primary raid, type=fd

 

256GiB 512GiB 256GiB Free Space

 

 

 

(parted) resize

 

Partition number? 1

 

End? [256GiB]? 511GiB

 

(parted) quit

 

Repeat the same for other disks, if multiple.

 

 

 

While growing the raid with new, additional disks can be done online, growing a raid to larger disks with raid0 raid-type used in Azure is not possible. The following example lists how this can be accomplished, however the filesystem – and thus the application using it – must be stopped and unmounted.

 

 

 

sudo umount /hana/data

 

sudo mdadm --stop /dev/md/mdhanadata

 

 

 

sudo mdadm --assemble --update=devicesize /dev/md/mdhanadata \

 

/dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1

 

 

 

sudo mount /hana/data

 

sudo xfs_growfs /hana/data

 

Unfortunately, with mdadm the no-downtime, online resize of Azure disks becomes something that does requires application downtime, to expand the software raid and later filesystems.

 

 

 

Closure

 

 

 

 

Online disk expansion – and online performance tier change – are a big quality of live improvements for SAP on Azure customers. Within a few minutes the disk(s) can be expanded, and OS actions are completed almost instantaneous afterwards, giving you tools to quickly react to changed needs without the need for any downtime at all. As with any changes to your production systems, ensure restore and recoverability in case of any unforeseen problems.

 

Continue reading...