V
Vasavi_Pasula
Introduction
In cloud-native Kubernetes environments, Containers are often treated as immutable resources, meaning they shouldn’t change after deployment. Immutable containers minimize the attack surface because they do not allow modifications during runtime. This limits the potential for attackers to make unauthorized changes, install malware, or create backdoors within a running container.
Container drift refers to unintended or unauthorized manual changes, updates, patches, or other modifications made during its runtime. When containers drift, they may incorporate untested and unverified changes, such as software updates, configuration modifications, or new libraries. These changes can introduce new vulnerabilities that were not present in the original, vetted container image. Drift might introduce changes that grant elevated privileges to processes or users within the container, which can be exploited to gain broader access to the system or network. Changes caused by drift can alter or disable security monitoring tools within the container, making it harder to detect and respond to security incidents promptly.
Microsoft Defender for Containers introduces the binary drift detection feature in public preview, to detect execution of files in a running container drifting from original Container Image which was scanned, tested, and validated. It's available for the Azure (AKS) V1.29, Amazon (EKS), and Google (GKE) clouds.
Defender for Containers Binary Drift Detection helps organizations:
Early Detection of Breaches: Drift detection serves as an early warning system for potential security breaches. If an attacker compromises a container and makes unauthorized changes, drift detection can immediately alert security teams, enabling them to respond quickly and mitigate the impact.
Monitor for Insider Activity: Drift detection helps mitigate insider threats by monitoring for unauthorized changes that could indicate malicious activity by an insider. This includes unauthorized changes to configurations, deployment scripts, or access controls within containers.
Reduce Human Error: Human error is a common cause of security breaches. Drift detection reduces the risk of human error by ensuring that any unintended changes made by administrators or developers are quickly detected and corrected.
Ensure Compliance with Security Standards: Many regulatory standards require organizations to maintain secure configurations and prevent unauthorized changes. Drift detection helps ensure compliance by continuously monitoring and documenting the state of containers, providing evidence that configurations remain consistent with regulatory requirements.
Prerequisites to enable Binary drift detection:
Configure Binary Drift Detection
Security Admins can configure drift detection policies at Azure Subscription, AWS Connector or GCP Connector and on Resources at Cluster level, Name space, Pod, or individual container level.
For details on how to configure drift detection Rules, refer : Binary drift detection (preview) - Microsoft Defender for Cloud | Microsoft Learn
Rules are evaluated in ascending order of priority. First rule 1 is evaluated, if it's a match the evaluation stops. If no match is found, the next rule is evaluated. If there's no match for any rule, the out of the box Default binary drift rule with default Ignore drift detection applies.
Best practices for Drift Detection:
Test / Simulate a binary drift alert
To test the binary drift feature and generate alerts (only in situations you defined in the binary drift policy that you’d like to get an alerts) you can execute any binary process in the container (not part of the original image). You can also use this script to create binary drift scenario:
kubectl run ubuntu-pod --image=ubuntu --restart=Never -- /bin/bash -c “cp /bin/echo /bin/echod; /bin/echod This is a binary drift execution”
Below you can observe the drift detection alert generated in a threat scenario:
Click on Open Logs to further examine the activities performed on this resource around the time of the alert. The attempt to list the Cluster admin credentials succeeded.
The alert also indicates there are 42 more alerts on the affected resource
This incident indicates that suspicious activity has been detected on the Kubernetes cluster. Multiple alerts from different Defender for Cloud plans have been triggered on the same cluster, which increases the fidelity of malicious activity. The suspicious activity might indicate that a threat actor has gained unauthorized access to your environment and is attempting to compromise it.
Advanced Hunting with XDR
Security teams can now access Defender for Cloud alerts and incidents within the Microsoft Defender portal, get the complete picture of an attack, including suspicious and malicious events that happen in their cloud environment, through immediate correlations of alerts and incidents.
By combining drift detection data with other security event information, SOC teams can build a more comprehensive understanding of potential incidents. A multi-stage incident involving multiple alerts can be observed in the XDR portal.
The alert evidence pane shows there has been suspicious activity with “ubuntu-pod”
The SOC team can further investigate the commands executed on the affected pod, and the user who executed the commands using the below query:
CloudAuditEvents
| where Timestamp > ago(1d)
| where DataSource == "Azure Kubernetes Service"
| where OperationName == "create"
| where RawEventData.ObjectRef.resource == "pods" and RawEventData.ResponseStatus.code == 101
| where RawEventData.ObjectRef.namespace == "kube-system"
| where RawEventData.ObjectRef.subresource == "exec"
| where RawEventData.ResponseStatus.code == 101
| extend RequestURI = tostring(RawEventData.RequestURI)
| extend PodName = tostring(RawEventData.ObjectRef.name)
| extend PodNamespace = tostring(RawEventData.ObjectRef.namespace)
| extend Username = tostring(RawEventData.User.username)
| where PodName == "ubuntu-pod"
| extend Commands = extract_all(@"command=([^\&]*)", RequestURI)
| extend ParsedCommand = url_decode(strcat_array(Commands, " "))
| project Timestamp, AzureResourceId , OperationName, IPAddress, UserAgent, PodName, PodNamespace, Username, ParsedCommand
For more information on how to Investigate suspicious Kubernetes (Kubeaudit) control plane activities in XDR advanced hunting refer: Kubeaudit events in advanced hunting - Microsoft Defender for Cloud | Microsoft Learn
SOC team can assign incidents from the Manage incident pane for mitigating the attack
Kubernetes Cluster administrators can configure automated workflows to handle common drift scenarios, such as reverting unauthorized changes, notifying relevant teams, or trigger response actions automatically.
In cloud-native Kubernetes environments, Containers are often treated as immutable resources, meaning they shouldn’t change after deployment. Immutable containers minimize the attack surface because they do not allow modifications during runtime. This limits the potential for attackers to make unauthorized changes, install malware, or create backdoors within a running container.
Container drift refers to unintended or unauthorized manual changes, updates, patches, or other modifications made during its runtime. When containers drift, they may incorporate untested and unverified changes, such as software updates, configuration modifications, or new libraries. These changes can introduce new vulnerabilities that were not present in the original, vetted container image. Drift might introduce changes that grant elevated privileges to processes or users within the container, which can be exploited to gain broader access to the system or network. Changes caused by drift can alter or disable security monitoring tools within the container, making it harder to detect and respond to security incidents promptly.
Microsoft Defender for Containers introduces the binary drift detection feature in public preview, to detect execution of files in a running container drifting from original Container Image which was scanned, tested, and validated. It's available for the Azure (AKS) V1.29, Amazon (EKS), and Google (GKE) clouds.
Defender for Containers Binary Drift Detection helps organizations:
Early Detection of Breaches: Drift detection serves as an early warning system for potential security breaches. If an attacker compromises a container and makes unauthorized changes, drift detection can immediately alert security teams, enabling them to respond quickly and mitigate the impact.
Monitor for Insider Activity: Drift detection helps mitigate insider threats by monitoring for unauthorized changes that could indicate malicious activity by an insider. This includes unauthorized changes to configurations, deployment scripts, or access controls within containers.
Reduce Human Error: Human error is a common cause of security breaches. Drift detection reduces the risk of human error by ensuring that any unintended changes made by administrators or developers are quickly detected and corrected.
Ensure Compliance with Security Standards: Many regulatory standards require organizations to maintain secure configurations and prevent unauthorized changes. Drift detection helps ensure compliance by continuously monitoring and documenting the state of containers, providing evidence that configurations remain consistent with regulatory requirements.
Prerequisites to enable Binary drift detection:
- Defender for Containers plan should be enabled on Azure subscription, AWS Connector, GCP Connector. For more details refer Configure Microsoft Defender for Containers components - Microsoft Defender for Cloud | Microsoft Learn
- Defender sensor must be enabled.
- Security Admin or higher permissions on the tenant to create and modify drift policies
Configure Binary Drift Detection
Security Admins can configure drift detection policies at Azure Subscription, AWS Connector or GCP Connector and on Resources at Cluster level, Name space, Pod, or individual container level.
For details on how to configure drift detection Rules, refer : Binary drift detection (preview) - Microsoft Defender for Cloud | Microsoft Learn
Rules are evaluated in ascending order of priority. First rule 1 is evaluated, if it's a match the evaluation stops. If no match is found, the next rule is evaluated. If there's no match for any rule, the out of the box Default binary drift rule with default Ignore drift detection applies.
Best practices for Drift Detection:
- Kubernetes Administrators should ensure that all container images are regularly updated and patched to include the latest security fixes.
- Detecting drift at the cluster level helps prevent unauthorized changes that could compromise the security and stability of the entire cluster. For example, an attacker gaining access to the Kubernetes API server might change cluster-wide settings to escalate privileges or disable security features.
- In multi-tenant environments, where different teams or customers share the same Kubernetes cluster but operate within their own namespaces, organizations can apply drift detection at namespace level monitoring only the areas of the cluster that are relevant to particular applications or teams.
- In development or testing environments, developers might need to make ad-hoc changes to containers to test new features, configurations, or debug issues, without the overhead of redeploying containers. Apply the ruleset only to the specific labelled Kubernetes pods.
- During scheduled maintenance windows, organizations might need to apply emergency patches or make quick operational changes directly to running containers to address critical security vulnerabilities or fix urgent issues. In this scenario, modify the rule action to Ignore Drift detection to avoid false positives.
- Allow list for processes - Organizations might define specific processes like monitoring agents, logging agents to be exempt from drift detection to avoid false positives.
Test / Simulate a binary drift alert
To test the binary drift feature and generate alerts (only in situations you defined in the binary drift policy that you’d like to get an alerts) you can execute any binary process in the container (not part of the original image). You can also use this script to create binary drift scenario:
kubectl run ubuntu-pod --image=ubuntu --restart=Never -- /bin/bash -c “cp /bin/echo /bin/echod; /bin/echod This is a binary drift execution”
Below you can observe the drift detection alert generated in a threat scenario:
Click on Open Logs to further examine the activities performed on this resource around the time of the alert. The attempt to list the Cluster admin credentials succeeded.
The alert also indicates there are 42 more alerts on the affected resource
This incident indicates that suspicious activity has been detected on the Kubernetes cluster. Multiple alerts from different Defender for Cloud plans have been triggered on the same cluster, which increases the fidelity of malicious activity. The suspicious activity might indicate that a threat actor has gained unauthorized access to your environment and is attempting to compromise it.
Advanced Hunting with XDR
Security teams can now access Defender for Cloud alerts and incidents within the Microsoft Defender portal, get the complete picture of an attack, including suspicious and malicious events that happen in their cloud environment, through immediate correlations of alerts and incidents.
By combining drift detection data with other security event information, SOC teams can build a more comprehensive understanding of potential incidents. A multi-stage incident involving multiple alerts can be observed in the XDR portal.
The alert evidence pane shows there has been suspicious activity with “ubuntu-pod”
The SOC team can further investigate the commands executed on the affected pod, and the user who executed the commands using the below query:
CloudAuditEvents
| where Timestamp > ago(1d)
| where DataSource == "Azure Kubernetes Service"
| where OperationName == "create"
| where RawEventData.ObjectRef.resource == "pods" and RawEventData.ResponseStatus.code == 101
| where RawEventData.ObjectRef.namespace == "kube-system"
| where RawEventData.ObjectRef.subresource == "exec"
| where RawEventData.ResponseStatus.code == 101
| extend RequestURI = tostring(RawEventData.RequestURI)
| extend PodName = tostring(RawEventData.ObjectRef.name)
| extend PodNamespace = tostring(RawEventData.ObjectRef.namespace)
| extend Username = tostring(RawEventData.User.username)
| where PodName == "ubuntu-pod"
| extend Commands = extract_all(@"command=([^\&]*)", RequestURI)
| extend ParsedCommand = url_decode(strcat_array(Commands, " "))
| project Timestamp, AzureResourceId , OperationName, IPAddress, UserAgent, PodName, PodNamespace, Username, ParsedCommand
For more information on how to Investigate suspicious Kubernetes (Kubeaudit) control plane activities in XDR advanced hunting refer: Kubeaudit events in advanced hunting - Microsoft Defender for Cloud | Microsoft Learn
SOC team can assign incidents from the Manage incident pane for mitigating the attack
Kubernetes Cluster administrators can configure automated workflows to handle common drift scenarios, such as reverting unauthorized changes, notifying relevant teams, or trigger response actions automatically.
Additional Resources
You can also use the resources below to learn more about these capabilities:
- Binary drift detection in Defender for Containers (Video)
- Binary drift detection (preview) - Microsoft Defender for Cloud | Microsoft Learn
- Kubeaudit events in advanced hunting - Microsoft Defender for Cloud | Microsoft Learn
- Container security architecture - Microsoft Defender for Cloud | Microsoft Learn
Reviewers
Eyal Gur, Principal Product Manager, Defender for Cloud
Continue reading...