Architecture Guidance: How to ingest GCP Firewall\VPC logs into Microsoft Sentinel

Firstly, I would like to thank Benjamin Kovacevic and Yael Bergman for their help with this article.

While the existing Sentinel GCP Pub/Sub Audit Logs connector documented here provides a way to ingest GCP platform audit logs, ingesting GCP Firewall logs or VPS logs remains a needed capability.

In this blog post I will show a simple way to ingest Google Cloud GCP Firewall logs or VPS logs.

In order to do this ingestion, I have utilized the Log Ingestion API with PowerShell to accomplish this.

Note: The code provided in this post is just a sample provided AS-IS. Further code optimization and additions could be added as required.

Following is the summary of tasks that are required in order to accomplish this from a high level:

1- Create Microsoft Entra application

2- Create data collection endpoint

3- Create new table in Log Analytics workspace

4- Create a new service account in the GCP project with the needed assigned GCP IAM role.

5- Create a new GCP PUBSUB topic and a new GCP PULL typed PUBSUB subscription.

6- Construct a JWT header and acquire a JWT token

7- Pull the PUBSUB messages from the PUBSUB REST API

8- Send a message ack back and ingest the message content into Sentinel

Detailed Steps

1- Create Microsoft Entra application

Working with Log Ingestion API requires to register a new App in Entra ID and note down the TenantId, AppId, Secret.

Steps to create a new App registration is documented here for reference.

2- Create data collection endpoint

The DCE is required in order to receive the incoming data stream. Required steps to create a new DCE is documented here.

3- Create new table in Log Analytics workspace

Before you can send data to the workspace, you need to create the custom table where the data will be sent. Required steps to create a new table is documented here.

What is remaining up to this point is to take note of the DCR ImmutableId and also assign role Monitoring Metrics Publisher to DCR as described in details in same document referenced above.

Note that the new table schema has to match exactly every ingested property's. For instance I have used following column names when creating the new table schema

TimeGenerated

publishTime

messageId

insertId

dest_ip

dest_port

protocol

src_ip

src_port

disposition

project_id

region

vm_name

zone

direction

ip_protocol

priority

reference

project_id

subnetwork_name

subnetwork_id

vpc_name

logName

receiveTimestamp

Logtype

timestamp

4- Create a new service account in the GCP project with the needed assigned GCP IAM role.

5- Create a new GCP PUBSUB topic and a new GCP PULL typed PUBSUB subscription.

Here for both #4 and #5 we move to do some GCP side configurations. As a start we need to create a new service account which we are going to impersonate and this service account has to be assigned the correct IAM roles to enable it to pull and ack messages from the PUBSUB.

Steps to create a new service account and a new role in GCP in addition to creating a new PUBSUB topic and log sink are explained in details in this document.

Note: Here also note that it's recommended to create a filter to only have Firewall\VPC logs in this subscription while creating the sink.

6- Construct a JWT header and acquire a JWT token

Following this GCP authentication documentation here, we could construct a JWT header that we can use to acquire an access token.

7- Pull the PUBSUB messages from the PUBSUB REST API

To accomplish this step we could use the standard Method: projects.subscriptions.pull as documented here. Note that the ReceivedMessage in the response body contains an object called PubsubMessage where the data field is a Base64 encoded (PubsubMessage) string that will need to be decoded in the code before ingestion in Sentinel.

8- Send a message ack back and ingest the message content into Sentinel

Here an ack is recommended to be sent back in order to have that message removed from the PUBSUB subscription queue.

Now to the code work

• Putting some needed parameters upfront
  

$appId = "xxxxxxxxxxxxxxxxxxxxxxxxxx"

$appSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxx"

$tenantId = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

$ingestionuri = "https://xxxxxxxxxxxxxxxxx.westeurope-1.ingest.monitor.azure.com"

$CertFile = "C:\xxxx\sentinel.p12"

$CertPassword = "notasecret"

$Project = "xxxxxxxx"

$ServiceAccountName = "sentinelserviceaccount"

$ServiceAccount = "sentinelserviceaccount@xxxxxxxxxxxxxx.iam.gserviceaccount.com"

$Scope = "https://www.googleapis.com/auth/pubsub"

$ExpirationSeconds = 3600

• Function that constructs JWT header and acquire access token
  

function CreateJWT {

Write-Host "Attempting to obtain JWT access token"

# import certificate

$Certificate = [system.Security.Cryptography.X509Certificates.X509Certificate2]::new($CertFile,$CertPassword,[system.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::Exportable)

$RSACryptoServiceProvider = New-Object System.Security.Cryptography.RSACryptoServiceProvider

$RSACryptoServiceProvider.ImportParameters($Certificate.PrivateKey.ExportParameters($true))

# create JWT Header

$JwtHeader = '{"alg":"RS256","typ":"JWT"}'

$JwtHeaderBase64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($JwtHeader))

$JwtHeaderBase64UrlEncoded = $JwtHeaderBase64 -replace "/","_" -replace "\+","-" -replace "=", ""

# create JWT Claim Set

$Now = (Get-Date).ToUniversalTime()

$NowUnixTimestamp = [Math]::Floor([decimal](Get-Date -Date $Now -UFormat "%s"))

$ExpirationUnixTimestamp = $NowUnixTimestamp + $ExpirationSeconds

$JwtClaimSet = @"

{

"iss":"$ServiceAccount",

"scope":"$Scope",

"aud":"https://oauth2.googleapis.com/token",

"exp":$ExpirationUnixTimestamp,

"iat":$NowUnixTimestamp

}

"@

$JwtClaimSetBase64 = [Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes($JwtClaimSet))

$JwtClaimSetBase64UrlEncoded = $JwtClaimSetBase64 -replace "/","_" -replace "\+","-" -replace "=", ""

# calculate Signature

$StringToSign = $JwtHeaderBase64UrlEncoded + "." + $JwtClaimSetBase64UrlEncoded

$SHA256 = [system.Security.Cryptography.SHA256]::Create()

$Hash = $SHA256.ComputeHash([Text.Encoding]::UTF8.GetBytes($StringToSign))

$SignatureBase64 = [Convert]::ToBase64String($RSACryptoServiceProvider.SignData([system.Text.Encoding]::UTF8.GetBytes($StringToSign),"SHA256"))

$SignatureBase64UrlEncoded = $SignatureBase64 -replace "/","_" -replace "\+","-" -replace "=", ""

# create JWT

$Jwt = $JwtHeaderBase64UrlEncoded + "." + $JwtClaimSetBase64UrlEncoded + "." + $SignatureBase64UrlEncoded

# send JWT request for oauth access token

$Body = "grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=$Jwt"

$AccessToken = Invoke-RestMethod -Method Post -Uri "https://oauth2.googleapis.com/token" -ContentType "application/x-www-form-urlencoded" -Body $Body | Select-Object -ExpandProperty access_token

Write-Host "Access token has been obtained successfully" #; "-"*50

Write-Host " "

return $AccessToken

}

• Function that pulls messages from PUBSUB
  

function pull-messages {

Write-Host "Attempting to pull PUBSUB messages"

$Pulluri = "https://pubsub.googleapis.com/v1/projects/sentinel-403712/subscriptions/sentinel-sub:pull"

$PullBody = @{

maxMessages = "100"

}

$pulledmessages = Invoke-RestMethod -Method Post -Uri $Pulluri -Headers @{"Authorization"="Bearer $JWTToken"} -Body ($PullBody | ConvertTo-Json) -ContentType 'application/json'

Write-Host "PUBSUB Messages have been obtained successfully"

write-host " "

return $pulledmessages

}

• Function that sends back an acknowledgement
  

function ackmsg {

param (

[string]$ackidss

)

$AckBody = @{

"ackIds" = $ackidss

}

$ackuri = "https://pubsub.googleapis.com/v1/projects/xxxxxxxxxx/subscriptions/xxxxxxxxxxx:acknowledge"

$autoack = Invoke-RestMethod -Method Post -Uri $ackuri -Headers @{"Authorization"="Bearer $JWTToken"} -Body ($AckBody | ConvertTo-Json) -ContentType 'application/json'

}

• Function that do the actual ingestion into the new table
  

Note that more data processing is done in order to decode the base64 encoded data and also to do some string manipulation to have all properties ready in the right shape for the ingestion.

function ingestmsg{

param (

[PSCustomObject]$mmm

)

# $mmm

$mmm = $mmm -replace ";", ""

$mmm = $mmm -replace "}", ""

$msgobjj = $mmm | ConvertFrom-String -PropertyNames data, attributes,messageId,publishTime

$msgobjj.data = $msgobjj.data -replace "@{data=", ""

$msgobjj.attributes = $msgobjj.attributes -replace "attributes=", ""

$msgobjj.messageId = $msgobjj.messageId -replace "messageId=", ""

$msgobjj.publishTime = $msgobjj.publishTime -replace "publishTime=" , ""

$msgobjj.data = [system.Text.Encoding]::UTF8.GetString([system.Convert]::FromBase64String($msgobjj.data))

$TimeGenerated = Get-Date ([datetime]::UtcNow) -Format O

$data = $msgobjj.data | ConvertFrom-Json

$publishTime = $msgobjj.publishTime

$messageId = $msgobjj.messageId

$InsertId = $data.InsertId

$dest_ip = $data.jsonPayload.connection.dest_ip

$dest_port= $data.jsonPayload.connection.dest_port

$protocol = $data.jsonPayload.connection.protocol

$src_ip = $data.jsonPayload.connection.src_ip

$src_port = $data.jsonPayload.connection.src_port

$disposition = $data.jsonPayload.disposition

$project_id = $data.jsonPayload.instance.project_id

$region = $data.jsonPayload.instance.region

$vm_name = $data.jsonPayload.instance.vm_name

$zone = $data.jsonPayload.instance.zone

$direction = $data.jsonPayload.rule_details.direction

$ip_protocol = $data.jsonPayload.rule_details.ip_port_info.ip_protocol

$priority = $data.jsonPayload.rule_details.priority

$reference = $data.jsonPayload.rule_details.reference

$project_id = $data.jsonPayload.instance.project_id

$subnetwork_name = $data.resource.labels.subnetwork_name

$subnetwork_id = $data.resource.labels.subnetwork_id

$vpc_name = $data.jsonPayload.vpc.vpc_name

$logName = $data.logName

$timestamp = $data.timestamp

$type = $data.resource.type

$receiveTimestamp = $data.receiveTimestamp

$staticData = @"

[

{

"TimeGenerated": "$TimeGenerated",

"publishTime": "$publishTime",

"messageId": "$messageId",

"insertId": "$InsertId",

"dest_ip": "$dest_ip",

"dest_port": "$dest_port",

"protocol": "$protocol",

"src_ip": "$src_ip",

"src_port": "$src_port",

"disposition":"$disposition",

"project_id":"$project_id",

"region":"$region",

"vm_name":"$vm_name",

"zone":"$zone",

"direction":"$direction",

"ip_protocol":"$ip_protocol",

"priority":"$priority",

"reference":"$reference",

"project_id":"$project_id",

"subnetwork_name":"$subnetwork_name",

"subnetwork_id":"$subnetwork_id",

"vpc_name":"$vpc_name",

"logName":"$logName",

"receiveTimestamp":"$dtimestamp",

"Logtype":"$resource.type",

"timestamp":"$receiveTimestamp"

}

]

"@;

Add-Type -AssemblyName System.Web

$appId = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

$appSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

$tenantId = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

$scope= [system.Web.HttpUtility]::UrlEncode("https://monitor.azure.com//.default")

$body = "client_id=$appId&scope=$scope&client_secret=$appSecret&grant_type=client_credentials";

$headers = @{"Content-Type"="application/x-www-form-urlencoded"};

$uri = "Sign in to your account"

$bearerToken = (Invoke-RestMethod -Uri $uri -Method "Post" -Body $body -Headers $headers).access_token

$dceEndpoint = "https://xxxxxxxxxxxxxxxxxx.westeurope-1.ingest.monitor.azure.com" #the endpoint property of the Data Collection Endpoint object

$dcrImmutableId = "dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxx" #the immutableId property of the DCR object

$streamName = "Custom-GcpFWLogs_CL" #name of the stream in the DCR that represents the destination table

$body = $staticData;

$headers = @{"Authorization"="Bearer $bearerToken";"Content-Type"="application/json"};

$urii = "$dceEndpoint/dataCollectionRules/$dcrImmutableId/streams/$($streamName)?api-version=2021-11-01-preview"

$uploadResponse = Invoke-RestMethod -Uri $urii -Method "Post" -Body $body -Headers $headers

}

• The actual code start running from here
  

$JWTToken = CreateJWT

$messages = pull-messages

$msgcount = ($messages.receivedMessages).count

Write-Host ""

write-host "Message count is:" $msgcount; ""

$msg = $messages.receivedMessages.message

if($msgcount -eq 1 ){

$ack = $messages.receivedMessages.ackid

write-host "Attempting to ack message by calling ack function"

ackmsg -ackidss $ack

Write-Host "message has been acked successfully"

$msg = $messages.receivedMessages.message

write-host "Attempting to ingest message by calling ingest function"

ingestmsg $msg

Write-Host "message has been ingested successfully"

}

<# Action when all if and elseif conditions are false #>

if($msgcount -eq 0 )

{

Write-Host "no new messages"

}

if($msgcount -gt 1 )

{

for ($i=1; $i -lt $msgcount; $i++)

{

$acks = $messages.receivedMessages.ackid

$acked = $acks[$i]

write-host "Attempting to ack message" $i "by calling ack function"

ackmsg -ackidss $acked

Write-Host "message" $i "has been acked successfully"

$msgs = $messages.receivedMessages.message

$msgg = $msgs[$i]

write-host "Attempting to ingest message" $i "by calling ingest function"

ingestmsg $msgg

Write-Host "message" $i "has been ingested successfully"

}

}

How data appears in Sentinel:

[ATTACH=full]55507[/ATTACH]

Final notes:

• This is a sample of code on how it could be done. This could be created in a form of Azure FunctionApp with a scheduled recurrence for instance.
  
• Some error handling could also be added for better monitoring of this ingestion pipeline.
  

Continue reading...