Using Defender XDR Portal to hunt for Kubernetes security issues

[Guest singhabhi]

As we saw in previous article, the binary drift alert gives you information about where the activity happened like the object namespace, image, cluster, etc.

 

 

 

This might or might not be enough information for you to act. Say, if you want to identify “how” this drift came to be for example, did a user logged on to container and downloaded the said binary. To supplement the information provided by the alert we can then use Defender XDR portal (Microsoft Defender portal - Microsoft Defender XDR)

 

 

 

[HEADING=1]Harnessing the power of the Microsoft Ecosystem[/HEADING]

 

 

 

If you are an E5 customer, your security teams most likely are very familiar with Advance Hunting on security.microsoft.com portal. Here we will extend that hunting capability to add context to your Kubernetes alerts. This is a huge time saver and cost advantage for you as you don’t need to teach your Red Team or SOC analysts Level 400 Kubernetes concepts. To jump start your Kubernetes hunting, you can leverage the developer knowledge of your Platform teams to provide most common Kubernetes actions like exec (access to a container), debug (access to node). The hunting team can then leverage these in the hunting queries in a data structure and format they already know using KQL.

 

[HEADING=2]Enhancing the hunt using XDR portal[/HEADING]

 

 

 

Defender for Cloud sends incidents and alerts to the Defender portal (Microsoft Defender for Cloud in the Microsoft Defender portal - Microsoft Defender XDR).

 

 

 

Similarly, Microsoft Sentinel also send the data to Defender portal (Microsoft Defender XDR integration with Microsoft Sentinel)

 

 

 

Since your SOC and Red Teams are already proficient in using XDR portal, Kubernetes hunts can now easily become part of their playbook.

 

 

 

By looking at the Alerts and Incidents in the XDR portal you can see the birds eye view of what, who, and where. This will equip you to further narrow down your search in Hunting Queries.

 

 

 

[ATTACH type=full" alt="singhabhi_0-1723555491129.png]62708[/ATTACH]

 

Fig. Attack Story for Binary Drift

 

 

 

The Evidence and Response tab shows all the relevant evidence. Most likely, you will create your hunting queries using these fields.

 

 

 

[ATTACH type=full" alt="singhabhi_1-1723555491155.png]62709[/ATTACH]

 

 

 

Fig. Kubernetes objects related to the incident

 

 

 

This integration allows us to run Advance Hunting queries using CloudAuditEvents table that has Defender for Cloud data.

 

The query below looks for exec in a Pod named ubuntu (assumption here being that this pod is also running a container ubuntu where the drift happened and alert generated)

 

 

 

 

 

 

 

CloudAuditEvents
| where DataSource == "Azure Kubernetes Service"
| where OperationName == "create"
| where RawEventData.ObjectRef.resource == "pods" and RawEventData.ResponseStatus.code == 101  
| where RawEventData.ObjectRef.subresource == "exec"
| where RawEventData.ResponseStatus.code == 101
| extend RequestURI = tostring(RawEventData.RequestURI)
| extend PodName = tostring(RawEventData.ObjectRef.name)
| extend PodNamespace = tostring(RawEventData.ObjectRef.namespace)
| extend Username = tostring(RawEventData.User.username)
| where PodName startswith "ubuntu"
| extend Commands =  extract_all(@"command=([^\&]*)", RequestURI)
| extend ParsedCommand = url_decode(strcat_array(Commands, " "))
| project Timestamp, AzureResourceId , OperationName, IPAddress, UserAgent, PodName, PodNamespace,  Username, ParsedCommand

 

 

 

 

 

 

 

The query above will produce the following, here you can see who executed the command, when was the command executed and many other fields that are helpful for the context

 

 

 

[ATTACH type=full" alt="singhabhi_2-1723555491161.png]62710[/ATTACH]

 

Fig. Query output

 

 

 

Another scenario you would want to look for are activities performed by a managed identity that might have been related to an alert.

 

 

 

 

 

 

 

CloudAuditEvents
| where RawEventData.principaloid == <managed identity id>

 

 

 

 

 

 

 

 

 

This query will provide you the actions that this managed identity performed. These might be enumerating key vaults, storage accounts, etc. As before by reviewing the output such as timestamp, geo etc. you can make a determination if this action is malicious.

 

 

 

You can then look at the activities that were performed during this timeframe, like so,

 

 

 

 

 

 

 

CloudAuditEvents
| where AzureResourceId == <AKS Cluster ID>
| where TimeGenerated > datetime(<start time>) and TimeGenerated < datetime(<end time>)
| where OperationName == "create"
| where UserAgent has “kubectl”

 

 

 

 

 

 

 

If the attacked conducted any “exec” during this time frame you will be able to see it under “ObjectRef” and “RequestURI” will show you the exact command that was executed.

 

 

 

Important to note that Query results are presented in your local time zone as per settings. Kusto filters, however, work in UTC.

 

 

 

[HEADING=2]Automating the process and Extending the out of the box detections[/HEADING]

 

 

 

Now that you have queries defined that provide you additional context behind the drifts, you can convert them into a custom detection rule that you can run at a defined frequency.

 

 

 

For example, say you want to get alerted on privileged pods:

 

 

 

These pods run with an elevated set of privileges required to do their job, but that could conceivably be used as a jumping off point to gain escalated privileges.

 

 

 

Note: You can also use Azure Policy built-in definitions to prevent this:

 

 

 

Microsoft Azure

 

 

 

 

 

 

 

CloudAuditEvents
| where Timestamp > ago(1d)
| where DataSource == "Azure Kubernetes Service"
| where OperationName == "create"
| where RawEventData.ObjectRef.resource == "pods" and isnull(RawEventData.ObjectRef.subresource)
| where RawEventData.ResponseStatus.code startswith "20"
| extend PodName = RawEventData.RequestObject.metadata.name
| extend PodNamespace = RawEventData.ObjectRef.namespace
| mv-expand Container = RawEventData.RequestObject.spec.containers
| extend ContainerName = Container.name
| where Container.securityContext.privileged == "true"
| extend Username = RawEventData.User.username
| extend DeviceId = RawEventData.AzureResourceId
| project Timestamp, ReportId, DeviceId , OperationName, IPAddress, UserAgent, PodName, PodNamespace, ContainerName, Username

 

 

 

 

 

 

 

Note that we will be using "Timestamp, ReportId, DeviceId" to turn the query into a custom detection rule. This again can be a starter pattern for your rules.

 

 

 

To create a custom detection/rule you will need to include the columns mentioned under Create and manage custom detection rules in Microsoft Defender XDR - Microsoft Defender XDR

 

 

 

You can now easily convert this query into a custom detection rule

 

[ATTACH type=full" alt="singhabhi_3-1723555491170.png]62711[/ATTACH]

 

Fig. Creating custom detection rule

 

 

 

Complete the fields that will show up in Alerts

 

 

 

[ATTACH type=full" alt="singhabhi_4-1723555491181.png]62712[/ATTACH]

 

Fig. Privileged pod alert details

 

 

 

Select the impacted object in our case the DeviceId (aka AzureResourceId)

 

 

 

[ATTACH type=full" alt="singhabhi_5-1723555491183.png]62713[/ATTACH]

 

 

 

Fig. Identity impacted by the alert

 

 

 

Choose the action

 

 

 

[ATTACH type=full" alt="singhabhi_6-1723555491185.png]62714[/ATTACH]

 

Fig. Action to be taken upon alert

 

 

 

Lastly, create your alert

 

 

 

[ATTACH type=full" alt="singhabhi_7-1723555491189.png]62715[/ATTACH]

 

Fig. Alert created

 

 

 

Once created the rule will appear in your custom detection rules in XDR portal

 

[ATTACH type=full" alt="singhabhi_8-1723555491191.png]62716[/ATTACH]

 

Fig. Custom detection rules

 

[HEADING=1] [/HEADING]

[HEADING=1]Call to action[/HEADING]

 

 

 

As you saw in this two-part series that unlike many third party solutions, you can maximize your existing investment in Microsoft’s security ecosystem to conduct deeper hunts using the tools that you already know.

 

 

 

Our suggestion is to:

 

 

 

1. Engage your Platform Engineering team to identify most risky Kubernetes actions that are relevant to your environment. For example, if you are using distroless images exec won’t make much sense
   
2. Provide your SOC and Red Team the XDR Portal and Defender for Cloud integration documentation so they can start leveraging the starter queries and customize to your environment
   
3. Review your third-party Kubernetes security solutions to see if the value you are getting for features that are not available in Native is worth the investment. In our discussions with dozens of customers, Native proves to be most direct, cost efficient, easier to use, and manage in the longer term
   
4. Moreover, if you have not thought about going Native-First in your security strategy reevaluate the choice and talk to your Security Seller about the latest roadmap of Native Security Services
   
5. Remember, a lot of on-premises security challenges were because of reliance on “best of breed”. That strategy does not translate well in Cloud. Share the following Native First security resources with your decision makers
   

• Native-First Cloud Security Approach
  
• Unleashing the Power of Microsoft Defender for Cloud – Unique Capabilities for Robust Protection
  

 

Continue reading...