Troubleshoot Amazon Web Services S3 connector issues

[Guest Yael_Bergman]

The Amazon Web Services S3 connector allows you to ingest AWS service logs, collected in AWS S3 buckets, to Microsoft Sentinel. The types we currently support are AWS CloudTrail, VPC Flow Logs, and AWS GuardDuty. This article describes common methods for verifying and troubleshooting an Amazon Web Services S3 connector for Microsoft Sentinel. Learn how to connect Microsoft Sentinel to Amazon Web Services to ingest AWS service log data.

 

 

 

Issue: Microsoft Sentinel doesn’t receive data from the Amazon Web Services S3 connector or one of its data types

 

The logs for the AWS S3 connector (or one of its data types) aren’t visible in the Microsoft Sentinel workspace for more than 30 minutes after the connector was connected.

 

Notes

 

• It can take around 20-30 minutes from the moment the connector is connected until data is ingested into the workspace.
  

• The connector's connection status indicates that a collection rule exists; it doesn't indicate that data was ingested. If the status of the Amazon Web Services S3 connector is green, this means that there is a collection rule for one of the data types, but there is still no data.
  

 

 

 

Possible Causes

 

In this section, we cover these causes:

 

• The AWS S3 connector permissions policies are not set properly.
  
• The data is not ingested to the S3 bucket in AWS.
  
• The SQS (Amazon Simple Queue Service) in the AWS cloud does not receive notifications from the S3 bucket.
  
• The data cannot be read from the SQS/S3 in the AWS cloud. With GuardDuty logs, the issue is usually caused by wrong KMS permissions.
  

 

Troubleshoot

 

Review the next sections to identify and solve common issues.

 

Check that the AWS S3 connector permissions policies are set properly

 

This issue is mainly caused by incorrect permissions in the AWS environment. You need permissions policies to deploy the AWS S3 data connector. Learn about required permissions. - Azure-Sentinel/AwsRequiredPolicies.md at master · Azure/Azure-Sentinel (github.com)

 

Check that the relevant data exists in the S3 bucket

 

In AWS, open the S3 bucket, search for the relevant folder according to the required logs, and check if there are any logs inside the folder:

 

 

 

 

 

 

If the data does not exist, there’s an issue with the AWS configuration. Configure an AWS service to export logs to an S3 bucket.

 

Verify that the data arrived from S3 to the SQS:

 

In AWS, open the relevant SQS.

 

In the Monitoring tab, you should see traffic in the Number Of Messages Sent widget.

 

 

 

 

 

 

 

 

If there is no traffic in the SQS, there is an AWS configuration problem:

 

• Make sure that the event notifications definition for the SQS includes the correct data filters (prefix and suffix). To see the event notifications, in the S3 bucket, select the Properties tab, and locate the Event notifications section. If you can’t see this section, create it. Make sure that the SQS has the relevant policies to get the data from the S3 bucket. The SQS must contain this policy in the Access policy tab.
   
  

 

Verify that data is read from the SQS

 

1. In AWS, open the relevant SQS.
   

 

In the Monitoring tab, you should see traffic in the Number Of Messages Deleted and Number Of Messages Received widgets.

 

 

 

 

 

 

Note that one spike of data is not enough, as shown in the screenshot below. Wait until there is enough data, as shown in the screenshot above, and then check for issues.

 

 

 

 

 

 

 

 

2. If at least one of the widgets is empty, check the health logs by running this query:

 

 

 

 

 

 

 

SentinelHealth

| where TimeGenerated > ago(1d)

| where SentinelResourceKind in ('AmazonWebServicesCloudTrail', 'AmazonWebServicesS3')

| where OperationName == 'Data fetch failure summary'

| mv-expand TypeOfFailureDuringHour = ExtendedProperties["FailureSummary"]

| extend StatusCode = TypeOfFailureDuringHour["StatusCode"]

| extend StatusMessage = TypeOfFailureDuringHour["StatusMessage"]

| project SentinelResourceKind, SentinelResourceName, StatusCode, StatusMessage, SentinelResourceId, TypeOfFailureDuringHour, ExtendedProperties

 

 

 

 

 

 

 

 

 

3. Make sure that the health feature is enabled:

 

 

 

 

 

 

 

 

 

SentinelHealth

| take 20

 

 

 

 

 

 

 

4. If the health feature isn’t enabled,

enable it

 

 

Issue: Data from the AWS S3 connector (or one of its data types) is seen in Microsoft Sentinel with a delay of more than 30 minutes

 

 

 

Possible Causes

 

This issue usually happens because Microsoft can’t read files in the S3 folder, either because the files are encrypted, or the files are in the wrong format. In these cases, many retries eventually cause ingestion delay.

 

 

 

Troubleshoot

 

 

1. Check if the logs are fully or partially encrypted by the Key Management Service (KMS). In this case, make sure that Microsoft Sentinel has permission for this KMS to decrypt the files. Review the required KMS permissions for the GuardDuty and CloudTrail logs.
   
2. To verify that the events notification from S3 to the SQS is defined properly, check that:
   

• The notification is defined from the specific folder that includes the logs, and not from the main folder that contains the bucket.
  
• The notification is defined with the .gz suffix.
  

 

For example:

 

 

 

 

3. Verify that there are no errors in the health logs by running this query:

 

 

 

 

 

 

 

SentinelHealth

| where TimeGenerated between (ago(startTime)..ago(endTime))

| where SentinelResourceKind == "AmazonWebServicesS3"

| where Status != "Success"

| distinct TimeGenerated, OperationName, SentinelResourceName, Status, Description

 

 

 

 

 

 

 

4. Make sure that the health feature is enabled:

 

 

 

 

 

 

 

SentinelHealth

| take 20

 

 

 

 

 

 

 

5. If t

he health feature is

n’t enabled,

enable it

.

 

 

 

Note: thank you Danielle Ohfeld Zats and Sara Gamzu for co-writing this blog

 

Continue reading...