Private Preview of Kafka Input and Output with Azure Stream Analytics

Imagine getting real-time insights into customer transactions with sub-second latency to respond to customer behavior immediately, detecting anomalies associated with your machines running at your business, or building real-time dashboards with Power Bi. These scenarios and more are what Azure Stream Analytics offers to users.

To learn more about Azure Stream Analytics, please visit: Introduction to Azure Stream Analytics | Microsoft Learn

Now we have taken it a step further and added another scenario that brings a whole new opportunity to take advantage of the many benefits of real-time analytics that Azure Stream Analytics offers: Kafka. We are announcing the Private Preview of the first open-source connectors from Azure Stream Analytics.

What is Kafka?

Kafka is a distributed streaming platform used to publish and subscribe to streams of records. Kafka is designed to allow your apps to process records as they occur. It is an open-source system developed by the Apache Software Foundation written in Java and Scala.

The following are the major use cases:

• Messaging
  
• Website Activity Tracking
  
• Metrics
  
• Log Aggregation
  
• Stream Processing
  

Kafka with Azure Stream Analytics

Historically, users could use Azure Stream Analytics by using workarounds such as MirrorMaker with Event Hubs or the Kafka extensions for Azure Functions, then building a streaming scenario with Azure Stream Analytics. These solutions may work for some designs, but they could be better as they introduce more complexity and performance or business compliance issues.

Now you can configure an Azure Stream Analytics job to get access to your Kafka clusters and connect directly to stream data without worrying about additional complexity or business compliance issues.

Migration to Cloud with Azure Stream Analytics

A beneficial scenario you can apply is migrating your data from on-prem to the cloud. With the numerous benefits the cloud offers, businesses now face the challenge of reliably migrating to the cloud. Azure Stream Analytics shines in this scenario. Azure Stream Analytics is a job service, so you do not have to spend time managing clusters, and you do not have to worry about downtime with a 99.9% SLA (service level agreements) at the job level. Billing is also done at the job level making startup costs low (one Streaming Unit), but scalable (up to 396 Streaming Units). It is much more cost-effective to run a few Stream Analytics jobs than to run and maintain a cluster.

Connectivity

One challenge of using Kafka is securely giving access to external resources. You can solve this problem by using VNET/SWIFT. Configuring your Azure Stream Analytics jobs to use VNET (Virtual Network) allows you to securely connect to your databases inside a virtual network or behind a firewall. Please visit the documentation: Run your Azure Stream Analytics job inside your Azure Virtual Network.

Security

As data analysis explodes in the modern era, it has become imperative to shield against various malicious attacks and ensure businesses can protect intellectual property and customers.

Using Kafka with Azure Stream Analytics allows you to take advantage of the many built-in security advantages Azure offers. You can also take advantage of multiple types of authentications to your Kafka clusters. These include:

1. mTLS or SSL/TLS – encryption and authentication (recommended). 
   
2. SASL_SSL – authentication with an additional protection layer to avoid interceptions. 
   
3. SASL_PLAINTEXT – standard authentication with username and password. 
   
4. None – No authentication or encryption. Recommended only for testing. 
   

Key Vault Integration 

To offer the highest guarantee for storing secrets, Azure Stream Analytics integrates seamlessly with Azure Key Vault to access stored secrets needed for authentication and encryption when using mTLS or SASL_SSL security protocols. Your Azure Stream Analytics job connects to Azure Key Vault using managed identity to ensure a secure connection and avoid the exfiltration of secrets.

Note: To use mTLS or SASL_SSL security protocols, you must have Azure Key Vault and managed identity configured for your Azure Stream Analytics job. 

You can store the certificates as Key Vault certificates or Key Vault secrets. Private keys are stored in PEM format. Please visit the following to learn how to upload secrets to Azure Key Vault: Store a multiline secret in Azure Key Vault | Microsoft Learn 

Limitation:

The minimum version of Kafka you can configure Azure Stream Analytics to connect to is version 0.10.

Conclusion:

We are excited to announce the Private Preview of Kafka adapters to connect directly to your Kafka clusters to Kafka. With this release, you can directly connect an Azure Stream Analytics job to Kafka clusters to ingest and output data. The Kafka adapters by Azure Stream Analytics are managed by Microsoft’s Azure Stream Analytics team, allowing it to meet business compliance standards without managing extra infrastructure. The Kafka adapters are backward compatible and support versions starting from version 0.10 with the latest client release.

If you want to participate in the private preview, please complete this form.

Continue reading...