General availability: Get more value from your logs with Azure Monitor advanced features

[Guest AdiBiran]

We are excited to announce the general availability of Azure Monitor Logs’ new set of features, which let you gain more value from your logs at a lower cost. These features were made available in public preview in early 2022 and have already been successfully adopted by a large set of customers, from small companies to very large, complex organizations. The features we’re announcing today provide significant value based on feedback from our preview customers. We’re very happy to introduce them alongside enhanced Azure portal experiences that will help you better manage and control your log data.

 

 

 

These features were developed together with Microsoft Sentinel, to support security analysts and allow access to any data, over any timeframe, and provide the most comprehensive and innovative threat-hunting solution in the market. For more information on new Microsoft Sentinel announcements, see the Sentinel article.

 

 

 

Cost-effective solution for high-volume verbose logs

 

 

Basic Logs is a new flavor of logs that enables a lower-cost collection of high-volume verbose logs that you use for debugging and troubleshooting, but not for analytics and alerts. This data, which might have been historically stored outside of Azure Monitor Logs, can now be available inside your Log Analytics workspace, enabling one solution for all your log data.

 

Basic Logs is supported for a subset of Azure tables, as well as all custom tables created with or migrated to the Data Collection Rule (DCR)-based logs ingestion API.

 

We’ve enhanced the Azure Portal and made it easier to set up and consume Basic Logs: you can now use the Table Management blade in your Log Analytics workspace to set the table plan to Basic Logs or back to Analytics, and run search queries on Basic Log tables from the Logs blade.

 

For more information, see Log Analytics workspace overview.

 

 

Low-cost long-term storage of your log data

 

 

Log Archive is an in-place solution to store your data for long-term retention of up to seven years at a cost-effective price point. This lets you store all your data in Azure Monitor Logs, without having to manage an external data store for archival purposes, and query or import data in and out of Azure Monitor Logs. You can access archived data by running a search job or restoring it for a limited time for investigation, as detailed below.

 

We’ve enhanced the Azure Portal experience to support archiving by enabling you to set the archive period per table using the Table Management blade in your Log Analytics workspace.

 

For more information, see Configure data retention and archive in Azure Monitor Logs.

 

 

Search through large volumes of log data

 

 

To enable searching over your expanding data sets, including Basic Logs and archived data, we’re announcing the general availability of a new tool that searches through petabytes of log data: the search job. A search job can run from a few minutes to hours, scanning log data and fetching the relevant records into a new persistent search job results table. The search job results table supports the full set of analytics capabilities to enable further analysis and investigation of these records.

 

We’ve enhanced the Azure Portal experience to support search jobs by enabling you to run a search job from the Logs blade in your Log Analytics workspace.

 

For more information, see Search jobs in Azure Monitor.

 

 

Investigate archived logs

 

 

Restore is another tool for investigating your archived data. Unlike the search job, which accesses data based on specific criteria, restore makes a given time range of the data in a table available for high-performance queries. Restore is a powerful operation, with a relatively high cost, so it should be used in extreme cases when you need direct access to your archived data with the full interactive range of analytics capabilities.

 

For more information, see Restore logs in Azure Monitor.

 

 

 

Pricing Model

 

 

We’ve adjusted our pricing model to support Basic Logs, archive, search jobs and restore:

 

Basic Logs charges are based on the amount of data you ingest. You’re also billed for any search queries you run on Basic Logs data.

 

Archived logs charges are based on the amount of data that you store in the archive, per month.

 

Search job charges are based on the amount of data you scan during job execution, plus the amount of data the search job ingests into the search results table.

 

Restore charges are based on the amount of data you restore and how long you retain the restored data.

 

You are not charged for querying data in search job results table or restored logs.

 

 

 

For complete pricing details, see Pricing - Azure Monitor.

 

Note: Billing for querying Basic Logs data, search jobs, and restored logs is not yet enabled and will begin in early 2023. Advance notice will be provided before billing starts.

 

Continue reading...