SAP NetWeaver HA deployment with File Share running on Windows failover to DR Region using ASR

[Guest dennispadia]

Overview

 

 

 

 

The high availability of SAP NetWeaver systems on Windows can be achieved by using Windows Failover Clustering. The resources configured in the cluster are considered highly available, if the node that host the resources are up. But the Windows cluster generally requires more than half of the nodes to be running to avoid a split-brain scenario. To achieve this objective, quorum is required to be set in the Windows failover cluster. Quorum is designed to prevent split-brains scenarios. There are different quorum options available for SAP NetWeaver on Azure VMs.

 

 

 

Setting up high availability of SAP workload on Azure protect application from infrastructure maintenance or failure within region. But it doesn’t provide protection from widespread regional disaster. For Disaster Recovery (DR), protection of application running on Azure VMs can by enabled by replicating components using Azure Site Recovery (ASR) to another Azure region. In this article, we will talk about achieving high availability configuration on DR region using ASR, when SAP ASCS/ERS with file share running on Windows is configured in productive region.

 

 

 

NOTE: The article only focuses on the SAP NetWeaver HA deployment with file share cluster. If your system is setup using “cluster shared disk” configuration, this article is not applicable.

 

 

 

IMPORTANT NOTE:

 

 

 

• The example shown in this article is exercised on the below OS version, cluster share and quorum option -
  
  • SAP ASCS/ERS OS version: Windows Server 2022 Datacenter
    
  • Quorum: Cloud witness
    
  • Cluster share: File share (SMB on Azure Files)
    

  [*]Depending on the type of file share storage used for SAP workload, you have to adopt an appropriate method to replicate the storage data to the DR site. In this example, SMB on Azure Files is used and its DR setup can be achieved by configuring separate SMB on Azure Files on DR region and copy the data periodically using robocopy.

  NOTE: In case you are using SMB on Azure NetApp Files (ANF), the DR setup can be achieved using cross-region replication. ANF is not available in all regions. Refer Azure Products by Region to see if ANF is available in your DR region.

  [*]Failover of other dependent services like DNS or Active directory is not covered in this article.

  [*]To replicate VMs using ASR for DR setup, review supported regions.

  [*]SMB Azure file shares are available in every Azure region, including all public and sovereign regions. Premium SMB file shares, which is recommended for SAP workload are available in a subset of regions.

  [*]ASR doesn’t replicate Azure load balancer that is used as virtual IP for the SAP ASCS/ERS cluster configuration in source site. In the DR site, you need to create load balancer manually beforehand or at the time of the failover event.

  [*]The cloud witness is configured using Azure blob storage, so you need to configure a separate storage account in DR region beforehand or at the time of the failover event.

  [*]If you configured your SAP system using Windows DFS-N to support flexible SAPMNT share creation for SMB-based file share, then some of the post-steps like editing UNC path at multiple areas won’t be required. You can directly modify SMB endpoint in DFS management console. But this article covers SAP system installed without DFS-N namespace for SMB storage volumes.

  [*]The procedure described here has not been tested with different OS releases. So, it might need some additional works based on your implementations or with future OS releases. So, make sure you test and document the entire procedure thoroughly in your environment.

SAP ASCS/ERS with SMB on Azure Files Disaster Recovery architecture

 

 

 

 

In the figure below, SAP ASCS/ERS with SMB on Azure files high availability cluster is configured in the primary region. The cluster uses cloud witness as quorum option. To establish DR for the setup, Azure site recovery (ASR) is used to replicate the SAP ASCS and SAP ERS VMs across site. For file share, separate SMB on Azure Files is created on DR region and data is copied periodically using robocopy.

 

 

 

NOTE: You can also leverage SMB on Azure NetApp Files (ANF) for SAP ASCS/ERS cluster. But in this blog, details on SMB on ANF is not covered.

 

 

 

SAP ASCS/ERS with SMB on Azure Files DR Architecture

 

 

 

 

 

As described in the example, to achieve HA setup in the DR site for SAP ASCS/ERS we need to make sure that all components that are part of the solution are replicated.

 

 

 

Components	DR setup
SAP ASCS/ERS VMs	Replicate VMs using Azure Site Recovery
SMB on Azure premium files	Create separate SMB on Azure premium Files. Copy data using robocopy.
Storage used for Cloud Witness	Create a separate storage in DR region.
Load balancer used for cluster virtual IP.	Create a separate Load balancer in DR region.

 

 

 

Disaster Recovery (DR) site preparation

 

 

 

 

To achieve similar highly available setup of SAP ASCS/ERS on the DR site, you need to make sure that all the components are replicated and available in the event of failover.

 

 

 

Configure ASR for SAP ASCS/ERS and Application server VMs

 

 

 

 

1. Deploy Resource Group, Virtual Network, Subnet and Recovery Service Vault in secondary region. For more information on networking in Azure VM disaster recovery, refer to prepare networking for Azure VM disaster recovery.
   
2. Follow the instructions in Tutorial to set up disaster recovery for Azure VMs document to configure ASR for SAP ASCS and ERS VMs.
   
3. On enabling Azure site recovery for VM to setup DR, the OS and local data disk that are attached to VMs get replicated to DR site. During replication, the VM disk writes are sent to a cache storage account in the source region. Data is sent from there to the target region, and recovery points are generated from the data. When you fail over a VM during DR, a recovery point is used to restore the VM in the target region.
   
4. After the VMs are replicated, the status will turn into “Protected” and the replication health will be “Healthy”.
   Replicated VMs in Recovery Service Vault
   

 

 

 

Configure robocopy to replicate SMB on Azure files data to DR Region

 

 

 

 

Robocopy, short for “Robust File Copy”, is a command-line utility in Windows for copying files and directories from one location to another. It is available as standard feature in Windows sines Windows Server 2008. So, you do not have to install this tool separately on your Windows server from where you will be executing the command.

 

 

 

1. To copy the data of SMB on Azure premium files to DR region, you need to create a separate SMB on Azure premium files on the DR region.
   
2. Follow the same post steps (like Active directory integration, assign required access and roles) on the new SMB on Azure files that you have followed in Primary region.
   
3. Before you can use Robocopy, you need to make sure the SMB on Azure files is accessible. The easiest way is to mount the share as local network drive to the Windows servers from where you are planning on using for Robocopy.
   NOTE: You can have UNC path as your source and target location in robocopy command. It is not necessary to have SMB on Azure files mounted at all times on server from where you are executing command.
   
4. You can execute Robocopy from any Windows servers, but make sure it can access both source and target SMB on Azure files.
   
5. Execute Robocopy command. Below screenshot is just an example, you must make sure to set appropriate copy option with the command as per your requirement.
   Example: Robocopy command to replicate data between two SMB on Azure Files storage
   

 

NOTE: You can schedule a job to execute Robocopy command periodically to copy the content of SMB on Azure Files.

 

 

 

Configure cloud witness for SAP ASCS/ERS in DR Site

 

 

 

 

Tip: Based on your DR strategy, you can either execute this step when you are preparing your DR site like setting up ASR or you can execute at the time of failover event.

 

 

 

• Create an Azure storage account on DR site to use as cloud witness.
   
  

  Region	Storage account name
  Primary	ts3clusteastn
  Disaster Recovery	ts3drwestwitness

  
  

 

 

 

Configure Standard Load Balancer for SAP ASCS/ERS in DR Site

 

 

 

 

Tip: Based on your DR strategy, you can either execute this step when you are preparing your DR site like setting up ASR or you can execute at the time of failover event.

 

 

 

Deploy standard load balancer on the DR site, similar to the one you have deployed in primary region. If you are creating load balancer beforehand on the DR site, you won’t be able to assign VMs to the backend pool. You can create the backend pool as empty which allows you to define the load balancing rules, but you cannot assign the DR VMs to the backend pool. Also, keep following points in mind -

 

 

• Keep the probe port in DR load balancer same as primary.
  
• When VMs without public IP addresses are placed in the backend pool of internal (no public IP address) Standard Azure load balancer, there will be no outbound internet connectivity, unless additional configuration is performed to allow routing to public end points. For details on how to achieve outbound connectivity see public endpoint connectivity for Virtual Machines using Azure Standard Load Balancer in SAP high-ava....
   
  

  Site	Frontend IP
  Primary Region - ASCS	10.19.0.97
  Primary Region - ERS	10.19.0.105
  DR Region - ASCS	10.5.0.9
  DR Region - ERS	10.5.0.10

  
  

 

 

 

Disaster Recovery (DR) failover event

 

 

 

 

[A] - Applicable to SAP ASCS Node, - Applicable to SAP ERS Node, [C] - Applicable to SAP Dialog Nodes.

 

 

 

In case of a production DR failover event, the following procedure needs to be followed for SAP ASCS/ERS and SAP Application server. If you are using different Azure services with your SAP system, you many need to adjust your procedure accordingly. The failover procedure described here assumes that the system running on primary region in not reachable or unavailable. So, when initiating failover to DR region the system on primary region will remain down at all times.

 

 

 

1. Perform the failover of SAP ASCS/ERS and all application server VMs that are configured in ASR to the DR region. For more details on how to failover, refer Tutorial to fail over Azure VMs to a secondary region for disaster recovery with Azure Site Recovery document.
   NOTE: Use of Azure site recovery for databases isn’t recommended. For more details on the DR recommendation for databases, refer to SAP database servers DR guidelines.
   
2. After the failover is completed, the status of the replicated items in the recovery service vault would be like below -
   Failover completion status in Recovery Service Vault
   
3. Update the IP address of VMs in DNS or in host files (if maintained). In this example, update the IP address for SAP ASCS/ERS, and all application servers. The ASCS/ERS server name registered in cluster is also maintained in DNS. So, you need to update the IP address of ASCS/ERS server name in DNS or in host files as well.
    
   
   
   Component
   
   Primary region IP address
   
   DR region IP address
   
   ASCS Instance
   
   10.19.0.103
   
   10.5.0.8
   
   ERS Instance
   
   10.19.0.104
   
   10.5.0.5
   
   Dialog Instance
   
   10.19.0.106
   
   10.5.0.7
   
   ASCS Virtual Server Name (cluster resource)
   
   10.19.0.97
   
   10.5.0.9
   
   ERS Virtual Server Name (cluster resource)
   
   10.19.0.105
   
   10.5.0.10
   

   
    
   NOTE: The ASCS/ERS virtual Server Name IP address needs to be updated in DNS and in cluster configuration. Check point 9 for information on how to update Server Name resource IP address in cluster.
   

4. [A] Login to SAP ASCS and SAP ERS server and update the UNC path maintained in the symbolic link to the SMB on Azure files on DR region (ts3smbdrwest.file.core.windows.net)
    
   #Execute below command in PowerShell with Administrator
   New-Item -Type SymbolicLink -Path "F:\usr\sap\TS3\SYS" -Target "\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS" -Force
   New-Item -Type SymbolicLink -Path "F:\usr\sap\TS3\ASCS00\data" -Target "\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\ASCS00\data" -Force
   New-Item -Type SymbolicLink -Path "F:\usr\sap\TS3\ASCS00\log" -Target "\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\ASCS00\log" -Force
   New-Item -Type SymbolicLink -Path "F:\usr\sap\TS3\ASCS00\sec" -Target "\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\ASCS00\sec" -Force
   
5. Change the value of SAPGLOBALHOST and SAPTRANSHOST to DR storage account endpoint (ts3smbdrwest.file.core.windows.net) on DEFAULT, ASCS, ERS and all APP server profiles.
    
   # SAPGLOBALHOST and SAPTRANSHOST value updated to DR storage endpoint in DEFAULT and instances profile
   PS F:\usr\sap\TS3\SYS\profile> findstr /i /r /c:"SAPGLOBALHOST" /c:"SAPTRANSHOST" TS3* DEFAULT.PFL
   TS3_ASCS00_ts3ascseast:SAPGLOBALHOST = ts3smbdrwest.file.core.windows.net
   TS3_D00_mswinapp55:SAPGLOBALHOST = ts3smbdrwest.file.core.windows.net
   TS3_ERS10_ts3erseast:SAPGLOBALHOST = ts3smbdrwest.file.core.windows.net
   DEFAULT.PFL:SAPGLOBALHOST = ts3smbdrwest.file.core.windows.net
   DEFAULT.PFL:SAPTRANSHOST = ts3smbdrwest.file.core.windows.net
   DEFAULT.PFL:DIR_TRANS = \\$(SAPTRANSHOST)\sapmnt\trans
   
6. [A] [C] SAP services (SAPSID_XX) configured in services.msc has executable path defined with profile value set as UNC path. It needs to be updated with the DR storage endpoint UNC path.
    
   # Example to edit BINARY_PATH_NAME of SAPTS3_00 maintained in services.msc.
   # Primary region SMB on Azure files endpoint: ts3smbeast.file.core.windows.net. It need to be changed to secondary region SMB on Azure files endpoint: ts3smbdrwest.file.core.windows.net
   C:\Windows\system32>sc qc SAPTS3_00
    
   SERVICE_NAME: SAPTS3_00
   TYPE : 10 WIN32_OWN_PROCESS
   START_TYPE : 3 DEMAND_START
   ERROR_CONTROL : 1 NORMAL
   BINARY_PATH_NAME : "F:\usr\sap\TS3\ASCS00\exe\sapstartsrv.exe" pf="\\ts3smbeast.file.core.windows.net\sapmnt\TS3\SYS\profile\TS3_ASCS00_ts3ascseast"
   LOAD_ORDER_GROUP :
   TAG : 0
   DISPLAY_NAME : SAPTS3_00
   DEPENDENCIES : RPCSS
   : LanmanServer
   SERVICE_START_NAME : SAPCONTOSO\SAPServiceTS3
    
   # Command to change BINARY_PATH_NAME of ASCS, ERS, PAS service in respective server where it is created
   sc config SAPTS3_00 binPath= "\"F:\usr\sap\TS3\ASCS00\exe\sapstartsrv.exe\" pf=\"\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\profile\TS3_ASCS00_ts3ascseast\"
   sc config SAPTS3_10 binPath= "\"F:\usr\sap\TS3\ERS10\exe\sapstartsrv.exe\" pf=\"\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\profile\TS3_ERS10_ts3erseast\"
   sc config SAPTS3_00 binPath= "\"F:\usr\sap\TS3\D00\exe\sapstartsrv.exe\" pf=\"\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\profile\TS3_D00_mswinapp55\"
    
   # Example: After changing BINARY_PATH_NAME
   C:\Windows\system32>sc qc SAPTS3_00
    
   SERVICE_NAME: SAPTS3_00
   TYPE : 10 WIN32_OWN_PROCESS
   START_TYPE : 3 DEMAND_START
   ERROR_CONTROL : 1 NORMAL
   BINARY_PATH_NAME : "F:\usr\sap\TS3\ASCS00\exe\sapstartsrv.exe" pf="\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\profile\TS3_ASCS00_ts3ascseast"
   LOAD_ORDER_GROUP :
   TAG : 0
   DISPLAY_NAME : SAPTS3_00
   DEPENDENCIES : RPCSS
   : LanmanServer
   SERVICE_START_NAME : SAPCONTOSO\SAPServiceTS3
   
7. [A] [C] Change the UNC path value maintained in Path, RSEC_SSFS_DATAPATH, RSEC_SSFS_KEYPATH and SAPEXE environment variable to DR storage endpoint.
    
   # Example showing the value of Path, RSEC* and SAPEXE env variable updated with DR storage endpoint
   PS C:\Windows\system32> Get-ChildItem -Path Env: | Where-Object -Property Value -CLike "*ts3smbdrwest*"
    
   Name Value
   ---- -----
   Path C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files (x86)\dotnet\;C:\Users\ts3adm\AppData\Local\Microsoft\WindowsApps;\\ts3smbdrwest.file.core.windows.net\sapmnt\T...
   RSEC_SSFS_DATAPATH \\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\global\security\rsecssfs\data
   RSEC_SSFS_KEYPATH \\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\global\security\rsecssfs\key
   SAPEXE \\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\exe\uc\NTAMD64
   
8. If you have created standard load balancer on the DR region beforehand with empty backend pool. Add ASCS/ERS VMs in the backend pool.
   Backend pool configuration in DR load balancer
   
9. [A] Update the IP address for ASCS and ERS server name resource configured in cluster to the Frontend IP configured in the load balancer (the one provisioned in disaster recovery region).
   ASCS server name - IP change in clusterERS server name - IP change in cluster
   
10. [A] Change quorum to the cloud witness storage account created on disaster recovery region.
    Update cloud witness by changing storage name in quorum settingsUpdate cloud witness with DR storage account name and secret key
    
11. [A] Start ASCS and ERS cluster role in failover cluster manager.
    Start ASCS and ERS cluster role
    
12. [C] Update the user store in all dialog instance servers with correct database hostname that is running on DR region. Check SAP Note 1852017 to get more insight on how to update hdbuserstore on Windows.
    
13. [C] Start all dialog instances.
    Status of all instances
    

Failback to former primary region

 

 

Once the service on primary region is available, and you have scheduled failback of your production landscape back to the primary region.

 

 

 

1. Re-protect failed over Azure VMs to the primary region. Refer to the document for more details.
   
2. Schedule or run Robocopy command to copy data from SMB on Azure files on Disaster Region to Primary Region.
   
3. On the event of failure, follow the same post steps described above.
   

 

Continue reading...