Leveraging Azure native tooling to hunt Kubernetes security issues

[Guest singhabhi]

[HEADING=1] [/HEADING]

[HEADING=1]Introduction[/HEADING]

 

 

 

Container binary drift refers to the phenomenon where a running container deviates from its original image over time. This can happen due to various reasons, such as manual updates, automated processes, or security vulnerabilities. Essentially, the container starts to differ from the static snapshot it was created from, leading to potential inconsistencies and security risks.

 

 

 

When thinking of container image drifts, it is important to understand the following:

 

• Security Risks: Image drift can introduce security risks, as the container may run software or processes that were not part of the original image. This can create a security blind spot, as traditional image scanning may not detect these changes
  
• Detection: Detecting image drift involves monitoring the container for changes that deviate from the original image. This can be done using tools that compare the running container's state with its original image.
  
• Prevention: To prevent image drift, it is recommended to implement image immutability, regularly update base images, and use image scanning tools. Monitoring and alerting for image drift can also help in identifying and addressing any deviations.
  

 

In this 3-part series we will look at the:

 

 

 

Part 1: Newest detection “binary drift” and how you can expand the capability using Microsoft XDR Portal Microsoft Defender portal - Microsoft Defender XDR. We will also look what you get as result of native integration between Defender for Cloud and Microsoft XDR. We will also showcase why this integration is advantageous for your SOC teams

 

 

 

Part 2: Further expanding on the integration capabilities, we will demonstrate how you can automate your hunts using Custom Detection Rules Create and manage custom detection rules in Microsoft Defender XDR - Microsoft Defender XDR. Reducing operational burden and allowing you to proactively detect Kubernetes security issues. Wherever applicable, we will also suggest an alternative way to perform the detection

 

 

 

Part 3: Bringing AI to your advantage, we will show how you can leverage Security Copilot both in Defender for Cloud and XDR portal for Kubernetes security use cases.

 

 

 

Note: To keep the discussion contained, we will assume that your container workloads are running on Azure Kubernetes Services (AKS) and that your AKS cluster leverages Azure’s RBAC (Use Microsoft Entra ID and Kubernetes RBAC for clusters - Azure Kubernetes Service). We also assume that you are using Azure Container Registry (ACR) for storing images (About registries, repositories, images, and artifacts - Azure Container Registry)

 

 

 

[HEADING=1]Capabilities needed to detect the drift[/HEADING]

 

 

 

Let’s discuss what you will need to set up in your environment to detect and triage the drift. (Remember not all drifts might be malicious, it might very well be a user or pipeline error)

 

 

 

[HEADING=1]Setting up Defender for Containers[/HEADING]

 

 

 

We assume that you have already enabled Defender for Containers if not please follow the directions listed here Configure Microsoft Defender for Containers components - Microsoft Defender for Cloud

 

We will set up the Defender for Containers to detect binary drifts. The feature is available for the Azure (AKS), Amazon (EKS), and Google (GKE) clouds.

 

 

 

To detect the drift you will need to set up Drift Policies (Binary drift detection (preview) - Microsoft Defender for Cloud). These policies define what you want or do not to alert on. You can create exclusions by setting higher priority rules for specific scopes or clusters, images, pods, Kubernetes labels, or namespaces.

 

In the sample rule below

 

[ATTACH type=full" alt="singhabhi_0-1723493997768.png]62704[/ATTACH]

 

 

 

Fig. Binary drift detection rule

 

1. Scope description: Human understandable description of where you are trying to detect the binary drift
   
2. Cloud Scope: Refers to Azure, AWS, or GCP where the rule applies. If you expand a cloud provider, you can select specific subscription. If you don't select the entire cloud provider, new subscriptions added to the cloud provider won't be included in the rule.
   
3. Resource Scope: Here you can narrow the scope to a specific object - Container name, Image name, Namespace, Pod labels, Pod name, or Cluster name.
   
4. Allow list of processes: List of processes that will not trigger an alert on the given Resource Scope
   

 

Also, note that there each rule has a priority and they are evaluated in ascending order. There is a default rule to ignore the binary drift detection

 

[ATTACH type=full" alt="singhabhi_1-1723493997771.png]62705[/ATTACH]

 

Fig. Default rule

 

 

 

[HEADING=1]Pre-requisites for generating Alerts[/HEADING]

 

Once you set up the rule it will be deployed on the Kubernetes nodes using Defender for Container’s enhanced sensor Overview of Container security in Microsoft Defender for Containers - Microsoft Defender for Cloud. You can check the Defender for Container Settings Microsoft Azure<SUB_ID_WITH _BINARY_DRIFT_DETECTION>/defendersPlan/Containers

 

 

 

[ATTACH type=full" alt="singhabhi_2-1723493997777.png]62706[/ATTACH]

 

 

 

Fig. Defender Sensor needs to be enabled

 

 

 

With Binary Drift detection rules set and the Defender Sensor enabled, you are all set to detect the binaries that are executing but did not originate from the original image.

 

 

 

[HEADING=1]Reviewing the alerts (a case study)[/HEADING]

 

You can see the alerts in the “Security Alerts” pane, like so

 

[ATTACH type=full" alt="singhabhi_4-1723494046740.png]62707[/ATTACH]

 

 

 

 

 

Fig. Binary Drift alert

 

 

 

For example, the image that the container is running does not come with wget

 

An attacker probably got hold of this container and downloaded this utility to import some tools.

 

 

 

The alert gives you information about where the activity happened like the object namespace, image, cluster, etc.

 

 

 

This might or might not be enough information for you to act. Say, if you want to identify “how” this drift came to be for example, did a user logged on to container and downloaded the said binary. To supplement the information provided by the alert we can then use Defender XDR portal (Microsoft Defender portal - Microsoft Defender XDR)

 

 

 

[HEADING=1]Summary[/HEADING]

 

 

 

This article showed you how to leverage binary drift detection and in the next article we will focus on how you can use XDR Portal to build more context around this alert and conduct hunts.

 

 

 

We will also share some queries that can serve as starter [Part 2 ].

 

Continue reading...