Data Collection Rules Creation Impacting Sentinel UEBA ML Model

[Guest mlopinto]

We all know for a SOC to be successful; it is imperative for the SOC engineer and SOC analyst to be in constant communication about new connectors and their intended configurations. In some organizations role and responsibilities are separated whereas in others, one individual may play both parts. If separated, they must ensure both team members understand how data sources are connected, the configuration that makes up that connection, how the data is formatted after ingestion and even where that data lands so they can make use of it. In order to ensure all of the required data is inspected by User Entity Behavior Analytics (UEBA) machine learning model, the AMA deployment strategy needs to be discussed and established upfront.

 

 

 

As organizations continue to transition from the MMA to the AMA, understanding how Data Collection Rules (DCRs) are fully leveraged to filter, transform, and land data into different log analytic workspaces (LAW) can help them build a strategy for optimized logging. For more information on DCRs, please visit: Data Collection Rules in Azure Monitor - Azure Monitor | Microsoft Docs. As a reminder, the AMA gives customers increased flexibility over what the MMA allowed. AMA includes enhanced capabilities such as filtering by event ID to only capture what is required rather than a category class of EventIDs, which helps organizations gather only what they find as a value add. See this link for full details on AMA migration and a matrix of differences between the two agent's functionality as of today: Migrate to the Azure Monitor agent (AMA) from the Log Analytics agent (MMA/OMS) for Microsoft Sentinel | Microsoft Docs.

 

 

 

Data Collection Rules can be created in two different places within the Azure Portal, and each will impact the table into which the logs are routed in the Log Analytics Workspace (LAW), as well as and a downstream function, such as, in Microsoft Sentinel. DCRs can be created via the Azure Monitor blade or within the Sentinel connector called “Windows Security Events via AMA”.

 

• Azure Monitor: DCR collection creation via Azure Monitor will land the data in the “Events” table.
  
• Microsoft Sentinel: In the “Windows Security Event via AMA” blade of the Sentinel data connectors, you are given the capability to create a DCR rule. These events will land in the SecurityEvents table in the underlying LAW.
  

 

One important caveat, if you decide to move the logs to the “Events” table via the Azure Monitor DCR creation, you will lose out on the UEBA model benefiting for enrichment. This is important because the Events table WILL NOT be inspected by UEBA. The UEBA model is designed to inspect the SecurityEvents table to enrich its ML Model.

 

 

 

If you have no plans to leverage the UEBA model in Microsoft Sentinel, then this should not impact you; however, it is highly recommended you do as a core component of Microsoft Sentinel. The SecurityEvents logging from host is one of many inputs that train our ML model‘s behavioral baselines for users, devices, and other entities. UEBA will then look at data from these sources to find anomalies and glean insights outside of the normal behavior, and subsequently bubble those up for analyst triage review. For more information on UEBA, see here: Microsoft Sentinel UEBA reference | Microsoft Docs

 

 

 

Below details the process to create DCR rules through the Microsoft Sentinel and Azure Monitor portals, each of which creates a different data stream, routing the data to either the SecurityEvents or Events table.

 

 

 

 

• Click on “+Create data collection rule”
  

 

 

• Enter the “Rule Name”, “Subscription” and “Resource Group”
  

 

 

• Click on “+Add resource(s)
  

 

 

• Select the resources to capture the events from
  

• Whether a specific host, Subscription or Management Group is selected, this list will not update if there are changes made to the collection after this selection
  
• Click “Apply”
  

 

 

• Select the “Events” to collect
  

 

 

• Select “Review and Create” and then “Create”
  

 

 

• The rule should then be created
  

 

 

 

 

Azure Monitor DCR Policy for Microsoft Sentinel Creation

 

Browsing to the DCR rules within Azure Monitor, the new Policy can be found

 

 

 

 

When selecting the rule “Sentinel DCR” (the one just created) and browsing to the “Data Sources” blade you’ll see there are no exposed values.

 

 

Browsing back to the overview tab and selecting the “JSON” view, the filter can be found.

 

Note: Notice the “Tag” createdBy = “Sentinel” and that the data will be stream to the LAW table “Microsoft-SecurityEvent”.

 

 

 

 

If you are looking for instructions on how to modify an existing DCR you already build so proper logs can be routed to the SecurityEvent table, please see the procedures under the next section.

 

 

 

Conclusion:

 

Make sure your teams who are giving the authority to create and modify DCR rules understand the downstream impacts this could present. Monitor the creation of DCR rules and ensure necessary coverage and oversight are being handled through alerting so a certain authority can interrogate the design. Make sure it is explicitly identified in policy of the implications of not forwarding these logs to the SecurityEvents table, which will limit full use and functionality of the UEBA ML model and SOC efficiency.

 

 

 

Contributions:

 

A special thanks to my colleagues and peers who helped me prepare and review this content for this blog: @Beth_Bischoff and @Paul Bergson.

 

 

 

----------------------------------------------------------------------------------------------------------

 

How to Modify an Existing DCR:

 

You can modify a DCR rule as explained Tutorial - Editing Data Collection Rules - Azure Monitor | Microsoft Docs

 

 

 

To make it simple for the purpose of re-routing the DCR to the SecurityEvent table, I'll summarize the above post for this specific purpose.

 

 

 

Step 1: Obtain the DCR “ResourceID” and name a “FilePath”:

 

Your resource ID can be found in the JSON view on the DCR Overview page:

 

 

 

 

 

 

 

 

 

 

 

I've decided to name my "<FilePath>" "temp.dcr". Feel free to choose your own name but choose something unique to that DCR for future purposes.

 

 

 

Step 2: Execute the following commands to retrieve DCR content and save it to a file.

 

(Replace <ResourceId> with DCR ResourceID and <FilePath> with the name of the file to store DCR.)

 

 

 

Modify the first two lines of the script and then execute within PS:

 

 

 

$ResourceId = “<ResourceId>” # Resource ID of the DCR to edit

 

$FilePath = “<FilePath>” # Store DCR content in this file

 

$DCR = Invoke-AzRestMethod -Path ("$ResourceId"+"?api-version=2021-09-01-preview") -Method GET

 

$DCR.Content | ConvertFrom-Json | ConvertTo-Json -Depth 20 | Out-File -FilePath $FilePath

 

 

 

 

 

 

Step 3: Open the JSON file to modify it: 

 

Use the command below to open the JSON file above the command line window.

 

 

 

code "<FilePath>"

 

 

 

 

Step 4: Modify “Streams” and “Destinations” JSON fields:

 

 

 

1. Modify each of the "streams" to "Microsoft-SecurityEvent”
   
2. Change the initial name and destination to "DataCollectionEvent"
   

 

*In order to ensure completeness in applying changes to all fields within the JSON, you can use the CTRL+F function to locate all “streams” and “destinations” fields.

 

 

 

 

 

 

 

Don't forget to save the modified file by entering this Powershell CLI command:

 

 

 

$Output = Read-Host "Apply changes to DCR (Y/N)?"

 

(using "Y" to apply changes)

 

 

 

 

 

 

Step 5: Now put the file back, modifying the top two lines again for your ResourceId and FilePath:

 

$ResourceId = “<ResourceId>” # Resource ID of the DCR to edit

 

$FilePath = “FilePath” # Store DCR content in this file

 

$DCRContent = Get-Content $FilePath -Raw

 

Invoke-AzRestMethod -Path ("$ResourceId"+"?api-version=2021-09-01-preview") -Method PUT -Payload $DCRContent

 

 

 

You should receive a 200 Status Code indicating the changes were applied to the file successfully. You will see a similar screenshot as this:

 

 

 

 

Step 6: Removing the Temp File:

 

 

 

From here you can remove your temporary JSON file that you modified by running:

 

 

 

Remove-Item $FilePath

 

Continue reading...