SAP Applications and Microsoft Defender for Linux

[Guest Cameron_MSFT_SAP_PM]

What is “Defender”?

 

 

This blog is about “Microsoft Defender for Endpoint” for Linux, hereafter referred to as MDE. The term “Defender” is used across multiple products and technologies.

 

An overview of Microsoft 365 Defender is illustrated here What is Microsoft 365 Defender? | Microsoft Learn

 

 

 

The typical audience for this blog is SAP Basis administrators and consultants. Enterprise Security is a specialist role and the activities described in this blog should be planned in conjunction with the Security Administrators. The objective of this blog is to provide a basic understanding of MDE on Linux and how to operate, check and troubleshoot problems on SAP VMs running MDE. It is generally recommended that the Enterprise Security Team coordinate with the SAP team and jointly design the MDE configuration, exclusions and scheduling.

 

 

 

Before continuing it is strongly recommended to watch the video in the link below. Microsoft Defender for Endpoint (MDE) is one component in the set of Defender solutions and in turn has multiple subcomponents Microsoft Defender for Endpoint | Microsoft Docs

 

 

 

This blog is focusing on two subcomponents: Next-generation protection (AntiVirus) and Endpoint detection and response (EDR). Next-generation protection is an AntiVirus (AV) product similar to AV solutions for Windows environments. Endpoint detection and response (EDR) detects and can block suspicious activity and system calls.

 

Microsoft Defender for Endpoint Subcomponents

 

[attachment=29931:name] Core Defender Vulnerability Management	[attachment=29932:name] Attack surface reduction	[attachment=29933:name] Next-generation protection	[attachment=29934:name] Endpoint detection and response	[attachment=29935:name] Automated investigation and remediation	[attachment=29936:name] Microsoft Threat Experts
Centralized configuration and administration, APIs
Microsoft 365 Defender

 

 

How is Microsoft Defender for Endpoint on Linux Deployed?

 

 

MDE for Linux may be deployed to VMs running SAP applications automatically in a subscription if Microsoft Defender for Cloud is activated. The SAP administrators and/or infrastructure team may not be aware MDE for Linux will be automatically deployed as a VM Extension. SAP administrators may observe that MDE is not installed when a new VM is first created, but after some time the following extension can be seen in the “Extensions + applications” blade in the Azure Portal.

 

 

 

 

Microsoft Defender for Cloud may be activated for the subscription containing SAP resources and MDE for Windows and Linux may be deployed by default. Further information can be found here: Using Microsoft Defender for Endpoint in Microsoft Defender for Cloud to protect native, on-premises, and AWS machines. | Microsoft Learn

 

 

 

MDE for Linux can also be deployed manually via tools such as yum and zypper, or via ansible, chef and puppet

 

Microsoft Defender for Endpoint on Linux | Microsoft Learn

 

SAP administrators and consultants should check with the Enterprise Security team for details about which deployment mode is used for the Azure subscription running SAP VMs.

 

 

 

Prerequisites & Default Deployment Configuration

 

 

Prerequisites for deploying MDE for Linux on SAP VMs:

 

1. MDE version 101.88.48 or higher must be deployed. Do not use lower releases
   
2. MDE for Linux supports all the Linux releases used by SAP applications
   
3. MDE for Linux requires connectivity to from VMs to update AV Definitions
   
4. MDE for Linux requires some crontab (or other task scheduler) entries to schedule scans, log rotation and MDE updates. Enterprise Security team will normally manage these entries How to schedule an update of the Microsoft Defender for Endpoint (Linux) | Microsoft Learn
   

 

Internet connectivity can be confirmed with the command below

 

 

This means that the AV component of MDE will not intercept IO calls. There is no IO interception and no scheduled AV scanning therefore MDE for Linux will not cause IO performance degradation on SAP DBMS or Application servers.

 

Note: if MDE for Linux is deployed by methods other Azure Extension the AntiVirus functionality may be enabled.

 

 

 

The command mdatp health will output the value for real_time_protection_enabled when MDE is deployed as an Azure Extension.

 

 

The Linux crontab is typically used to schedule MDE AV scan and log rotation tasks

 

How to schedule scans with Microsoft Defender for Endpoint (Linux) | Microsoft Learn

 

 

 

EDR functionality will be Active whenever MDE for Linux is installed. There is no way to disable EDR functionality through command line or configuration.

 

See the section “Checklist for Troubleshooting Problems on SAP VMs Running MDE on Linux” for more information on troubleshooting EDR

 

 

 

How to Check Defender Deployment and Configuration

 

 

It is recommended to check the installation and configuration of MDE with the command mdatp health

 

 

 

SAP Application and DBMS servers MDE configuration should be similar the screenshot below. The key parameters are:

 

1. healthy = true
   
2. release_ring = Production. Pre-release and insider rings should not be used with SAP Applications
   
3. real_time_protection_enabled = false. This prevents realtime IO interception
   
4. automatic_definition_update_enabled = true
   
5. definition_status = “up_to_date”. Run a manual update if another value is seen
   
6. edr_early_preview_enabled = “disabled”. Do not enable on SAP systems. This may lead to system instability
   
7. conflicting_applications = [ ]. Other AV or security software installed on a VM
   
8. MDE engine_version = must be 101.88.48 or higher otherwise issues with NFS or sudo may occur in some cases
   

 

 

 

 

This article has some useful hints on troubleshooting installation issues for MDE:

 

Troubleshoot installation issues for Microsoft Defender for Endpoint on Linux | Microsoft Docs

 

 

 

How to Setup MDE AntiVirus Exclusions

 

 

It is generally recommended to enable real_time_protection_enabled = true after identifying the relevant DBMS and SAP exclusions. This provides the optimal protection while at the same time avoiding performance problems.

 

 

 

This article details how to configure AV exclusions for processes, files and folders per individual VM

 

Set up exclusions for Microsoft Defender Antivirus scans | Microsoft Learn

 

 

 

SAP administrators should contact the Enterprise Security Team to discuss how to configure AV exclusions for all SAP VMs in an Azure Resource Group or Subscription.

 

 

 

Warning: If real time scanning is enabled on MDE releases lower than 101.88.48 sudo may be blocked. It is strongly recommended to update to the latest version of MDE and verify the MDE release before enabling real time scanning.

 

 

 

It is recommended to exclude:

 

1. DBMS data files, log files and temp files, including disks containing backup files
   
2. The entire contents of the SAPMNT directory
   
3. The entire contents of the SAPLOC directory
   
4. The entire contents of the TRANS directory
   
5. The entire contents of directories for standalone engines such as TREX
   

 

Note: It is recommended to have database files on the separate mountpoint with read and write permissions only (exec permission to mount point).

 

Hana systems should exclude /hana/data, /hana/log and /hana/shared – see Note 1730930.

 

Oracle ASM systems do not need exclusions as MDE cannot read ASM disks.

 

Common mistakes to avoid when defining exclusions | Microsoft Learn

 

 

 

Recommended SAP OSS Notes

 

2248916 - Which files and directories should be excluded from an antivirus scan for SAP BusinessObjects Business Intelligence Platform products in Linux/Unix? - SAP ONE Support Launchpad

 

1984459 - Which files and directories should be excluded from an antivirus scan for SAP Data Services - SAP ONE Support Launchpad

 

2808515 - Installing security software on SAP servers running on Linux - SAP ONE Support Launchpad

 

1730930 - Using antivirus software in an SAP HANA appliance - SAP ONE Support Launchpad

 

1730997 - Unrecommended versions of antivirus software - SAP ONE Support Launchpad

 

 

 

 

Note: MDE for Linux folder exclusions are not recursive, meaning “/usr/sap/trans” exclusion does not include subfolders such as the “cofiles” or “data” subdirectories under “/usr/sap/trans”. Subfolders need to be specifically added.

 

 

 

After configuring exclusions it is possible to test with the EICAR test file. The EICAR test file can be placed in a temporary location to confirm MDE AV is functioning correctly

 

Configure and validate exclusions based on extension, name, or location | Microsoft Learn

 

 

 

In the example below the standard EICAR test file is downloaded with wget and a scan is run manually.

 

 

Threats can be listed with the command mdatp threat list and then the file(s) removed with the command below

 

 

 

Checklist for Troubleshooting Problems on SAP VMs Running MDE on Linux

 

 

If there are performance, stability or installation problems on an SAP VM running MDE for Linux it is recommended to follow the checklist below:

 

1. Run mdatp health and confirm all settings are set
   
   1. 
      
   2. healthy = true
      
   3. release_ring = Production
      
   4. real_time_protection_enabled = false (or true with appropriate exclusions configured)
      
   5. automatic_definition_update_enabled = true
      
   6. definition_status = “up_to_date”
      
   7. edr_early_preview_enabled = “disabled”
      
   8. conflicting_applications = [ ]
      

[*]Run zypper, yum or dnf to update mdatp. Deploy updates for Microsoft Defender for Endpoint on Linux | Microsoft Learn

[*]Run mdatp definitions update to update AV definitions

[*]Run mdatp connectivity test. If there are any connectivity issues follow the procedure Troubleshoot cloud connectivity issues for Microsoft Defender for Endpoint on Linux | Microsoft Learn

[*]Confirm that process, file and folder exclusions are appropriately configured with mdatp exclusion list

[*]Behavior Monitoring is disabled. It can be enabled or disabled via managed config. Use the command: mdatp config behavior-monitoring --value disabled

[*]Microsoft support may request to create a managed config file in the path /etc/opt/microsoft/mdatp/managed/mdatp_managed.json

 

This file can be configured with additional debugging/support options

 

Restart mdatp service with the command sudo service mdatp restart

 

1. Review the MDE on Linux logs for unusual events or warnings. Log files are located under "/var/log/microsoft/mdatp"
   

 

After completing the above checklist try to reproduce the problem. If the problem reproduces and MDE is a possible cause then follow the procedure below to open a support case.

 

 

 

How to Collect Logs & Open Support Cases

 

 

In rare cases MDE for Linux may impact performance or stability of an SAP VM. If this is suspected follow this checklist:

 

 

 

1. Download the Client Analyzer tool https://aka.ms/XMDEClientAnalyzer
   

 

Run the client analyzer on macOS or Linux | Microsoft Docs

 

1. Increase logging level if required Microsoft Defender for Endpoint on Linux resources | Microsoft Learn
   
2. Collect any other logs
   
3. Create a support request via the Defender 365 Portal
   

 

MDE for Linux support cases should be opened by the Enterprise Security Team via the Defender 365 Portal and not via the typical Azure Portal page. The support message should mention “Defender on Linux performance issues”

 

 

 

 

 

 

How to Uninstall Defender

 

 

In rare cases it may be necessary to uninstall MDE to isolate a problem. SAP support may also occasionally request that a problem is reproduced without any AntiVirus or security software installed.

 

MDE for Linux can be uninstalled using yum, zypper or dnf. Microsoft Defender for Endpoint on Linux resources | Microsoft Learn

 

 

 

Another option is to use the installer script Deploy Microsoft Defender for Endpoint on Linux manually | Microsoft Learn

 

 

 

Troubleshooting Steps

 

 

The complete list of log and config files for MDE is:

 

"/var/log/microsoft/mdatp"

 

"/var/opt/microsoft/mdatp"

 

"/etc/opt/microsoft/mdatp"

 

"/etc/opt/microsoft/mdatp/managed"

 

"/var/opt/microsoft/mdatp/crash"

 

 

 

AV or EDR events such as finding a Virus are logged into the Defender 365 Portal.

 

The deployment and health status of a subscription is also visible within the Defender 365 Portal, an example is illustrated below

 

 

 

 

 

 

 

 

It is generally recommended to install nmon and activate sysstat (SAR) on SAP servers. These tools are useful for determining if MDE for Linux or other processes are causing high CPU or disk utilization.

 

1. Unfortunately NMON is not available in some repositories such as zypper, dnf, yum and must be downloaded nmon for Linux | Main / HomePage
   
2. NMON also has the ability to record to a log file that can be analyzed in Excel. Execute the command nmon -f -s1 -c600 (this will record every 1 second for 600 count, or 10 minutes). The log file can then be analyzed in nmon analyzer Excel Macro nmon for Linux | Site / Nmon-Analyser (sourceforge.net)
   
3. sysstat or SAR may or may not be installed and activated by default. Suse gallery images may have SAR running by default. Check the directory /var/log/sa. If the directory does not exist or does not contain recent sarXX files then follow the steps below
   
4. KSAR is a graphical tool that presents historical system performance information in a simple and easy to interpret way. This tool requires a runtime JVM Releases · vlsi/ksar (download the latest pre-release version)
   

 

 

 

If sysstat needs to be installed follow the steps below:

 

# sudo yum install sysstat

 

# sudo service sysstat restart

 

Redirecting to /bin/systemctl restart sysstat.service

 

The /var/log/sa/sarXX files can be copied onto a Windows PC with sftp

 

sftp -i <keyfilename>.pem azureuser@<xx.xx.xx.xx>

 

get /var/log/sa/sar<XX>

 

 

 

On a Windows PC run this command and open the SAR file "Java -jar C:\sap_media\ksar.jar"

 

KSAR shows long term trends and NMON is a realtime tool.

 

When reviewing KSAR graphs problems with AV software may be indicated by high “Waiting I/O” times.

 

 

 

 

Example of NMON logging with analysis in Excel via Nmon-Analyzer macros. CPU shown in blue line and IOPS in pink.

 

 

Useful Commands & Links

 

 

 

 

During manual zypper installation on Suse an error “Nothing provides ‘policycoreutils’”

 

Troubleshoot installation issues for Microsoft Defender for Endpoint on Linux | Microsoft Docs

 

 

There are several command-line commands that can control the operation of mdatp. To turn off real-time protection, you can use the command:

 

mdatp config real-time-protection --value disabled

 

This command will tell mdatp to retrieve the latest definitions from the cloud:

 

mdatp definitions update

 

 

 

This command will test whether mdatp can connect to the cloud-based endpoints via the network:

 

mdatp connectivity test

 

 

 

These commands will update the mdatp software if needed:

 

yum update mdatp

 

zypper update mdatp

 

 

 

Since mdatp runs as a linux system service, you can control mdatp using the service command, eg:

 

service mdatp status

 

 

 

sudo mdatp diagnostic create (this command creates a diagnostic file that can be uploaded to Microsoft support)

 

 

 

In /hana/shared there are a very large number of binaries. These binaries should be excluded from Real Time Scanning. A loop like the below can be used:

 

#!/bin/bash

 

for f in /hana/shared/<put in full path here>/exe/hdb*; do

 

mdatp exclusion process add –name “%f”

 

done

 

 

 

Useful Links

 

 

Microsoft Endpoint Manager does not support Linux at this time Manage Microsoft Defender for Endpoint configuration settings on devices with Microsoft Endpoint Manager | Microsoft Learn

 

 

 

Microsoft Defender for Endpoint Linux - Configuration and Operation Command List - Microsoft Tech Community

 

 

 

Deploying Microsoft Defender for Endpoint on Linux Servers. - Microsoft Tech Community

 

 

 

Troubleshoot cloud connectivity issues for Microsoft Defender for Endpoint on Linux | Microsoft Docs

 

Troubleshoot performance issues for Microsoft Defender for Endpoint on Linux | Microsoft Docs

 

 

 

SAP Notes

 

 

2808515 - Installing security software on SAP servers running on Linux - SAP ONE Support Launchpad

 

784391 - SAP support terms and 3rd-party Linux kernel drivers - SAP ONE Support Launchpad

 

1494278 - NW-VSI: Summary of Virus Scan Adapter´s for SAP integration - SAP ONE Support Launchpad

 

666568 - Using the EICAR anti-virus test file - SAP ONE Support Launchpad

 

 

 

Useful blogs

 

 

yongrhee – Yong Rhee’s blog (wordpress.com)

 

 

 

 

 

Thanks to Anjan Banerjee, Rahul Tibdewal and Ankit Garg

 

Continue reading...