Demystifying Log Ingestion API

absharan · Oct 7, 2024

In this blog, I’m going to delve into ingesting and transforming application logs to log analytics workspace using Log Ingestion API technique. Before we jump into the details, let’s explore the different types of ingestion techniques based on the application log types.

If applications and services logs information to text files instead of standard logging services such as Windows Event log or Syslog, Custom Text Logs can be leveraged to ingest such text file logs in log analytics workspace.
If applications and services logs information to JSON files instead of standard logging services such as Windows Event log or Syslog, Custom JSON Logs can be leveraged to ingest such text file logs in log analytics workspace.
If an application understands how to send data to an API then you can leverage Log Ingestion API to send the data to Log Analytics Workspace.

Custom Text/JSON Logs are out of scope for this blog, I might write a separate blog dedicated to these techniques later. In this blog, my focus will be on streaming data to log analytics workspace using Log Ingestion API and transforming the data for optimal usage.

Note: This blog aims to demonstrate how to ingest logs using the log ingestion API. To keep things straightforward, I’ll refer to our public documentation.

Overview of Log Ingestion API

The Logs Ingestion API in Azure Monitor lets you send data to a Log Analytics workspace using either a REST API call or client libraries. The API allows you to send data to supported Azure tables or to custom tables that you create.

Data can be sent to the Logs Ingestion API from any application that can make a REST API call. This may be a custom application that you create, or it may be an application or agent that understands how to send data to the API.

Note: The data sent by your application to the API must be formatted in JSON and match the structure expected by the DCR. It doesn't necessarily need to match the structure of the target table because the DCR can include a transformation to convert the data to match the table's structure.

Below diagram shows the flow of data from an application to data ingestion pipeline where transformation is applied, the transformed data is then ingested to the log analytics workspace.

Advantages of Log Ingestion API

Since we’ve covered the overview of Log Ingestion API, let’s understand why you should consider Log Ingestion API over traditional Data Collector API.

Since it requires a DCR-based custom table to ingest the data; Log Ingestion API supports transformations, which enable you to modify the data before it's ingested into the destination table, including filtering and data manipulation.
You can send the data to multiple destinations.
Enables you to manage the destination table schema, including column names, and whether to add new columns to the destination table when the source data schema changes.
HTTP Data Collector API is getting deprecated and it's replaced by Log Ingestion API so it's a future proof solution.

Prerequisites for Log Ingestion API

There are certain prerequisites for configuring Log Ingestion API which are as follows:

App registration and secret: The application registration is for the authentication of the API call. It must be granted permissions to the DCR. The API call includes the Application (client) ID and Directory (tenant) ID of the application and the Value of an application secret.
Data collection endpoint (DCE): We will need endpoint URI of the DCE in our configuration. The endpoint URI in a data collection endpoint (DCE) is a connection where data sources send collected data for processing and ingestion into Azure Monitor.
Table in Log Analytics workspace: The table in the Log Analytics workspace must exist before you can send data to it. You can use one of the supported Azure tables or create a custom table using any of the available methods.
Data collection rule (DCR): Azure Monitor uses the Data collection rule (DCR) to understand the structure of the incoming data and what to do with it.

Since the “What” and “Why” part is covered, let’s understand the “How” part.

Let’s look at how to send data to Azure Monitor using Log Ingestion API. We have detailed public documentation on this topic where we have covered approach based on configuration with Azure Portal and ARM template.

I will be leveraging our public documentation in this blog. Let’s get started:

Create App Registration and Secret

Browse to Entra ID > App Registrations > Create a new registration as shown below.

We will need Application (client) ID and the Directory (tenant) ID at later stage.
Now we need to generate an application client secret, which is similar to creating a password to use with a username. Select Certificates & secrets > New client secret. Give the secret a name to identify its purpose and select an Expires duration.

Ensure that you record the Value of secret because you can't recover it after you move away from this page.

Create Data Collection Endpoint

Browse to Data Collection Endpoint and create a new Data Collection Endpoint as shown below:

Copy the endpoint URI from the DCE, we will need it at later stage.

Create new DCR-based custom table in Log Analytics workspace

To avoid any duplication in the blog, I would recommend going through our public documentation for this step. The steps are very well articulated in our documentation for creating a DCR-based custom table and Data Collection Rule.
In my lab, the custom table name is "CustomLogDemo_CL" which is attached to "DEMO-DCR-UI" Data Collection Rule.

I have also applied some parsing and filtering to get better visibility of the data.
Browse to the Data Collection Rule created for this exercise and copy the immutable ID, we will need it at later stage. An immutable ID is an unique identifier for the data collection rule.

Assign Permission to DCR

The final step is to give the application permission to use the DCR. Any application that uses the correct application ID and application key can now send data to the new DCE and DCR.
Browse to Data Collection Rule > IAM > Add role assignment > Select Monitoring Metrics Publisher > Select the application that we created.

Select Review and assign the permission.
Follow the instructions in our public documentation to generate the sample data.
Once sample data is generated, send the sample data to Azure Monitor by following our public documentation.

Let’s check if it really works

Considering all the configurations are correct, the logs should appear in the log analytics workspace as shown below:

Transforming the data

Now that we are successfully sending data to DCE-based custom table using Log Ingestion API, let’s implement some transformation.

You can refer my blogs on transformation techniques for better understanding:

I will directly jump into the transformation which I’m going to implement in my custom table.

In my use case, I’m going to split the data into 2 tables, CustomLogDemo_CL and Split_CustomLogDemo_CL. Split_CustomLogDemo_CL table will have only those rows where the ResponseCode is 200, rest of the data will be ingested into CustomLogDemo_CL table.

Detailed steps as follows:

Browse to DCR > Choose the DCR created for this exercise > Automation > Export Template > Deploy > Edit Template.
As per below template, I’ve created 2 streams for this operation.
- One stream is sending all the data except where the response code is 200 to CustomLogDemo_CL table.
- Another stream is sending the data with response code 200 to Split_CustomLogDemo_CL table.

Let’s validate if it works as expected.

Browse to Log Analytics Workspace > Logs and check both the tables as shown below:

As we can see, we have split the incoming data as desired. You can optimize the table as per your needs with the help of transformation.

I would like to thank my colleague Kaustubh Dwivedi @Kausd for sharing his expertise and co-authoring this blog along with me.

References:

Demystifying Log Ingestion API

absharan