G
glzbcrt
Introduction
Last year Microsoft introduced resource specific structured logging in Azure Firewall, it is well documented here. This year it was announced the general availability as described here.
In this post I will dive deep and show the expected cost optimization of this new structured logging, and what is causing this saving. I will use a sample record of a network rule log to explain it. Please note that this saving applies only if the sink is Log Analytics.
Log Analytics
Log Analytics is a widely used logging solution from Microsoft and its main cost driver is the ingestion, measured in GB. Every ingested network rule, application rule, NAT rule will be logged, if configured, and have some bytes charged against your subscription based on your price sheet.
You can find the amount of charged bytes for a record using the hidden column _BilledSize. The query below show how to extend the output to include it, and can be used on any table. You can find more information about this column here.
<table>
| extend _BilledSize
Let's now take a look at how much bytes every option will ingest.
Legacy Categories
In the image below you can see that this simple network rule log hit will cost around 389 bytes distributed over 15 columns.
All the network connection details are stored inside a single string column called msg_s. There are a lot of other columns present which account for the most part of the _BilledSize.
For instance, to search all requests made by a specific source IP address we must first parse the msg_s column to extract this IP address, which will need some CPU cycles to manipulate the strings, apply regular expressions, and other operations. For one record the amount of CPU cycles is so low that doesn't matter, but when you have millions or even billions of records it will take a lot of time just to parse this string and this time will add to the total runtime of the query.
New Categories in Resource Specific Table
Using the new categories and saving to resource specific tables is the most effective way of storing firewall logs.
The use of these new tables allowed the Azure Firewall team to tune and exclude around 8 columns which were not needed. Comparing this table with the previous one the number of columns went from 21 to 13. Columns like ResourceId, Category, ResourceGroup, SubscriptionId, ResourceProvider, Resource, ResourceType, and SourceSystem, all long strings, were completely removed.
The savings were huge, from 389 bytes per record to only 64 bytes!
Conclusion
To conclude this post below you can find a summary of the _BilledSize for the destination tables used to log network rules.
The numbers don't lie, using the new categories and storing the data in the resource specific tables expect a cost saving of Log Analytics ingestion of around 80% for Azure Firewall.
From the field with these small changes a customer was able to reduce their Log Analytics ingestion from 28 GB per day to 3 GB per day without losing any information. Just with simples changes in the way the data was logged.
Configure your log settings like shown below to benefit from this.
The only downside you might face is your queries will need to be changed, but the cost and performance optimizations we have seen are worth.
If this change doesn't help you optimize your Azure Firewall logging cost you can try the approach described here.
Hope it helps you optimize your costs and understand the reason behind the saving!
Continue reading...
Last year Microsoft introduced resource specific structured logging in Azure Firewall, it is well documented here. This year it was announced the general availability as described here.
In this post I will dive deep and show the expected cost optimization of this new structured logging, and what is causing this saving. I will use a sample record of a network rule log to explain it. Please note that this saving applies only if the sink is Log Analytics.
Log Analytics
Log Analytics is a widely used logging solution from Microsoft and its main cost driver is the ingestion, measured in GB. Every ingested network rule, application rule, NAT rule will be logged, if configured, and have some bytes charged against your subscription based on your price sheet.
You can find the amount of charged bytes for a record using the hidden column _BilledSize. The query below show how to extend the output to include it, and can be used on any table. You can find more information about this column here.
<table>
| extend _BilledSize
Let's now take a look at how much bytes every option will ingest.
Legacy Categories
In the image below you can see that this simple network rule log hit will cost around 389 bytes distributed over 15 columns.
All the network connection details are stored inside a single string column called msg_s. There are a lot of other columns present which account for the most part of the _BilledSize.
For instance, to search all requests made by a specific source IP address we must first parse the msg_s column to extract this IP address, which will need some CPU cycles to manipulate the strings, apply regular expressions, and other operations. For one record the amount of CPU cycles is so low that doesn't matter, but when you have millions or even billions of records it will take a lot of time just to parse this string and this time will add to the total runtime of the query.
New Categories in Resource Specific Table
Using the new categories and saving to resource specific tables is the most effective way of storing firewall logs.
The use of these new tables allowed the Azure Firewall team to tune and exclude around 8 columns which were not needed. Comparing this table with the previous one the number of columns went from 21 to 13. Columns like ResourceId, Category, ResourceGroup, SubscriptionId, ResourceProvider, Resource, ResourceType, and SourceSystem, all long strings, were completely removed.
The savings were huge, from 389 bytes per record to only 64 bytes!
Conclusion
To conclude this post below you can find a summary of the _BilledSize for the destination tables used to log network rules.
Destination Table | _BilledSize | Saving |
| Azure Diagnostics | 389 | 0% |
| Resource Specific | 64 | 83% |
The numbers don't lie, using the new categories and storing the data in the resource specific tables expect a cost saving of Log Analytics ingestion of around 80% for Azure Firewall.
From the field with these small changes a customer was able to reduce their Log Analytics ingestion from 28 GB per day to 3 GB per day without losing any information. Just with simples changes in the way the data was logged.
Configure your log settings like shown below to benefit from this.
The only downside you might face is your queries will need to be changed, but the cost and performance optimizations we have seen are worth.
If this change doesn't help you optimize your Azure Firewall logging cost you can try the approach described here.
Hope it helps you optimize your costs and understand the reason behind the saving!
Continue reading...