Attack Simulation Training: Using machine learning to drive more effective simulations

[Guest VipulPandey]

Attack Simulation Training (AST) is an advanced tool for reducing the risk of phishing across an organization that measures behavior change and automates the deployment of an integrated security awareness training program across an organization. It allows security teams to run intelligent simulations, consume actionable insights and remediate risk with hyper-targeted training designed to change behavior, and then measure behavioral progress against that benchmark through repeated simulations.

 

 

 

Predicted Compromise Rate for payloads

 

One of the most crucial elements in running a phishing simulation is the right selection of the payload to drive the right user behavior. For organizations which are focused on improving end user resilience, the selection of the right quality of payload is important. If you are tracking only click-through as a quality metric, then over time there is an incentive to decrease the click rate by selecting easier-to-spot phish payloads. Eventually, it makes it less likely that the end user will change their behavior when the real event does come along.

 

 

 

To combat the tendency to use low-click-rate payloads and to maximize educational returns, we have created a new piece of metadata for every global payload in AST called predicted compromise rate (PCR).

 

 

 

The predicted compromise rate in Attack Simulation Training is built using historical data across Microsoft 365 that predicts the percentage of people who will be compromised by this payload (users compromised / total number of users who receive the simulation. We are excited to share that we have introduced a new intelligent mechanism using information like payload content, compromise rates (aggregated and anonymized) and payload metadata to predict a more accurate potential compromise rate when the payload is used within a simulation.

 

We encourage AST admins to use PCR to normalize actual user click through behavior for any given simulation and payload against another simulation by comparing the delta of actual to predicted instead of just the flat actual click rate.

 

 

 

We will be updating the value of existing global payloads to infer from the model, as well as make it available for an on-demand evaluation of your custom payloads. As part of the payload authoring experience, you can create your content (duplicate from existing templates or create from scratch) and use the predicted compromise rate model to determine the efficacy of your payload. For best results on rate calculation, it is recommended to use detailed information as part of payload authoring experience.

 

 

 

You can also review the overall performance of your organization by measuring the difference between the predicted compromise rate and the actual compromise rate across simulations using the Training Efficacy report.

 

 

 

Note: In the initial rollout, the on-demand experience would be available for English language payloads and will gradually rollout for remaining languages soon.

 

 

 

Example of on-demand PCR calculation for a payload

 

 

 

 

 

 

 

 

Recommendations from Microsoft Defender for Office 365

 

In addition to the machine learning integration for payload efficacy, we have recently introduced payload recommendations from campaigns targeting your organization. The recommendation feature offers organizations the ability to proactively defend against these attacks by providing a deeper understanding of the types of payloads being used in their specific environment.

 

The recommendations are built on campaign information from Microsoft Defender for Office 365 and are refined using inputs like Predicted Compromise Rate. It allows administrators to use payloads targeted and most frequently used within their organization. This information can then be utilized to create more effective phish simulations, helping to prepare employees and raise awareness of the latest tactics used by attackers.

 

 

 

 

 

 

With more intelligence infused within the simulation experience, you can now take advantage of predicted compromise rate, payload recommendations, to drive more effective phishing simulations, with the goal towards improving your user and organization resilience.

 

 

 

Want to learn more about Attack Simulation Training?

 

Get started with the available documentation today and you can read more details about new features in Attack Simulation Training.

 

If you have other questions or feedback about Microsoft Defender for Office 365, engage with the community and Microsoft experts in the Defender for Office 365 forum.

 

Continue reading...