E2E Bootstrap Solution for Malicious File Scanning Using Microsoft Defender for Storage in Azure

[Srinivas_Nalla]

This blog post elucidates one of the architectural patterns that can be employed for efficiently monitoring the malware scan status while utilizing Microsoft Defender for storage malware scanning.


Real-world Scenario:



In a typical complex web application, file uploads are a common occurrence across various application scenarios (for instance an application responsible for handling employee payroll may have one module accepting proof of identity documents from employees for personal information updates and other module handling employee reimbursement requests based on the uploaded expense receipts).



Let's consider a scenario that requires file upload use-cases, and each use-case needs its individual storage account to manage its file uploads. Enabling Microsoft Defender for Storage’s add-on Malware Scanning on the scenario specific storage account can lead to inadvertent invocation of malicious files as developers may have direct access to them for troubleshooting the issues or it may not be the most cost-effective approach for handling scanning across each storage account container.



To address this challenge, we can create a solitary storage account container that remains separate from the storage containers specific to individual application scenarios. We can refer to this as a "Demilitarized Zone (DMZ) Storage Account Container", which acts as the frontline defence for processing file uploads originating from different application scenarios. Access to the DMZ container can be highly restricted, ensuring that only malware-free files proceed to the scenario-level storage account containers. At this stage, specific scenario-level processing can be conducted via middleware functions. In the event of malicious file detection within the DMZ container, our malware file processing functions can either delete the file on the spot or transfer it to a Quarantine Storage Account Container, which is accessible exclusively to security analysts.


In-Depth Architectural Pattern:



Drawing inspiration from the above use case, we've developed an efficient pattern that revolutionizes the conventional approach to accessing file scan status results via blob index tags. We introduce a Push-based architecture, which promptly notifies connected clients of file scan status. Additionally, it addresses the challenge of invoking supporting APIs to display other functionalities, such as delete or download options, upon successful file uploads to the destination system.



This architectural design leverages the capabilities of various Azure services:

1. Microsoft Defender for Cloud-Enabled Storage Account (DMZ Storage Account Container): This Storage Account Container is where the application uploads multiple scenario specific files.

1. Azure SignalR Service: To implement a push-based event notification mechanism, we utilize the Azure SignalR service. The Azure SignalR Hub efficiently manages concurrent connections from thousands to millions of clients and delivers events to actively connected clients at least once. During the file upload process, we connect to the Azure SignalR Hub, and the Azure SignalR service establishes a connection to its server using supported protocols like WebSocket, Server Sent Events or Long Polling, depending on underlying browser compatibility. This enables us to promptly display file scan status to users without requiring page refreshes or prolonged waiting, as opposed to periodically monitoring the presence of file scan index tags on blobs after a file scan.

Read more about Azure SignalR Service here and please note that using Azure SignalR incurs additional cost depending on the chosen plan. Check more on Azure SignalR Pricing here.

1. Azure Event Grid Topic: We employ an Event Grid topic to capture file scan status results from Microsoft Defender for Cloud Storage Malware Scanning, which is enabled at the storage account. The Event Grid Topic offers a wide range of subscription capabilities for multiple channels and Azure services to consume event statuses. In our pattern, we integrate and subscribe this to Azure Functions. Event Grid Topic created can be configured to receive events by navigating to Storage Account > Microsoft Defender for Cloud > Settings > Enable Event Grid > Select the newly created event grid topic from the list. Check the detailed tutorial here for manual creation. We will soon have bicep template ready to automate event grid creation process and configuring it to Defender for Cloud Storage Malware settings, check the bicep template updates here.

Read more about Azure Event Grid Service here and please note that using Azure Event Grid incurs additional cost depending on the chosen plan. Check more on Azure Event Grid Service Pricing here.

1. Azure App Configuration: Azure App Configuration serves as the repository for configuration information necessary for the smooth operation of this architectural pattern. It also stores the configuration required for moving files from the DMZ storage account container to application scenario specific storage account containers once file status becomes available.

Read more about Azure App Configuration Service here and please note that using Azure App Configuration Service incurs additional cost depending on the chosen plan. Check more on Azure App Configuration Service pricing here.

1. Azure Key Vault: For secure storage and retrieval of secrets, we rely on Azure Key Vault and reference them in Azure App Configuration.

Read more about Azure Key Vault Service here and please note that using Azure Key Vault service incurs additional cost depending on the chosen plan. Check more on Azure Key Vault Service pricing here.





Architectural Diagram: visio link



File Upload Scanning Architecture Using Microsoft Defender for Storage





Let's delve into the architectural workflow:

1. The user initiates the file upload process through the application, sending the file to the DMZ storage account container. Simultaneously, in the background, the application establishes a connection to the Azure SignalR Hub to monitor file scan statuses.
2. Microsoft Defender for Storage’s Malware Scanning, enabled at the DMZ Storage Account, scans for malware in blobs – near-real time - and transmits the scan status to the configured Event Grid Topic.
3. The Event Grid Topic has a subscription with an Event Grid Trigger called "Generic File Status Checker Azure Function". This function relays the file scan status to the SignalR Hub and takes appropriate actions. If the scan results indicate a non-malicious status “No threats found”, the file is moved from the DMZ storage container to the storage container specific to the application scenario. In the event of a malicious scan result “Malicious”, the file is deleted from the DMZ storage container, and a failure status is promptly relayed to the SignalR Hub.
4. Once the scan status becomes available in the SignalR Hub, the connected user, who uploaded the file for a specific scenario, is promptly notified about the file scan result. This notification occurs efficiently without necessitating a screen refresh or causing any undue delays.
5. After the file is transferred to the application’s scenario-specific container, developers can create scenario-specific middleware functions to process the files according to their scenario requirements. Subsequently, they can transmit the success or failure status of the scenario-specific processing to the SignalR Hub.
6. Once the connecting client receives the status of the scenario-specific processing, the application can invoke supporting APIs to enable more detailed functionalities, such as deletion and efficient downloading of the processed content, all without the need for a screen refresh.

Key Generic Components:



The generic bootstrap solution has following components to facilitate the end-to-end experience from file upload process within multiple scenarios of an application to file scan status communication to connected client (Web UI)

The below generic components are available for consumption at Microsoft Opensource GitHub Project



Bicep Infra files: Spin up required resources responsible for solution consumption in seconds. Click Here to view Bicep Code



SignalR Negotiate Azure function: Facilitates secured way of establishing connection with SignalR instance in a serverless methodology by exchanging connection string and short lived authentication code. Click Here to view Function Code



SignalRWrapper NPM Package: Code NPM Package Link

• The Generic NPM package takes care of establishing connection handshake process with Azure SignalR Service.
• Registers event listeners on interested topics.
• Clients can configure event handlers responsible for processing file malware status
• Connection clean-up.

Generic File Scan Status Checker Azure function : Click Here to view Function Code

• File scan status is sent to Event grid which will trigger the File Scan status checker function.
• File Scan Status Checker function sends the scan status to SignalR hub.
• and the status checker function moves the file to respective scenario specific container if the status is non-malicious result.
• or else if the status is Malicious then the file is deleted from the DMZ container and appropriate status is sent to SignalR hub.
  

Detailed Documentation related to code consumption can be found here.





Authors:

Srinivas Nalla, Software Engineer 2, Microsoft Digital

Ajith Kumar Rai, Software Engineer 2, Microsoft Digital

Deepika Somagari, Software Engineer 2, Microsoft Digital



Reviewers:

Arieh Bibliowicz, Principal Software Engineer, Microsoft Defender for Cloud

Ashish Mathur, Principal Software Engineer, Microsoft Digital

Fernanda Vela, Product Manager 2, Microsoft Defender for Cloud CXE

Continue reading...