Azure Stream Analytics releases slew of improvements at Ignite 2022: Output to Delta Lake and more!

[Guest ajeta21]

Azure Stream Analytics releases slew of improvements at Ignite 2022: Output to Delta Lake, No code editor GA and much more!

 

Today we are excited to announce numerous new capabilities that unlock new stream processing patterns that work with your modern lakehouses. We are announcing native support of Delta Lake output, no code editor GA, improved development & troubleshooting experience and much more! We are making it easier for you to take on more stream processing scenarios easily without having to write a single line of code by providing 7 pre-defined templates in no code editor.

 

Below is the complete list of announcements for Ignite 2022. The product is continuously updated so you always get the latest version and can access all these new features automatically.

 

 

 

No code editor (General Availability)

 

We are thrilled to announce General Availability of Stream Analytics no code editor. With Stream Analytics no-code editor, you can easily develop a Stream Analytics job to process your real time data in your event hubs without any code for streaming ETL, ingestion, enrichment, and materializing data to data stores. It also provides 7 pre-defined templates to help you get started and get your stream processing steps completed within minutes.

 

Now, it is generally available with several new capabilities including:

 

1. Support two new outputs: Event Hubs and Azure Data Explorer.
   
2. Support built-in functions of three categories for data manipulation in “Manage fields”: Date Time function, String function, and Mathematical function.
   
3. Add three new scenario templates under Event Hubs – Process data.
   

 

You can learn more about the no-code stream processing experience and get started right away.

 

No code editor

 

 

 

 

 

 

 

 

 

Native support of Delta Lake output (Public Preview)

 

Delta Lake is a popular format that enterprises use to build their lakehouse as it adds reliability, quality, and performance to data lakes. We are thrilled to announce native support of delta lake output in Stream Analytics which allows you to directly write streaming data to your delta lake tables without writing a single line of code. Your job can be configured to write to either a new or a pre-created Delta table in an Azure Data Lake Storage Gen 2 account. This output is optimized for high-speed ingestion to delta tables in append mode and also provides exactly-once semantics which guarantees that no data is lost or duplicated. Ingesting real-time data streams from Azure Event Hubs into Delta tables allows you to perform ad-hoc interactive or batch analytics.

 

More details are available in this doc: Azure Stream Analytics - Writing to Delta Lake table

 

Delta lake output

 

 

 

Improved development and troubleshooting with Physical Job diagram (Public Preview)

 

Stream Analytics physical job diagram visualizes your job’s key metrics in diagram format and table format: CPU utilization, memory utilization, Input Events, Partition IDs, Backlogged Input events, and Watermark delay. You can check the job’s historical metrics data with different types of metrics, filters, and splitters. With the diagram, it helps you quickly identify if your job is running in parallel with the diagram and locate the bottlenecks (hottest streaming node that impacts your job’s performance) by ordering the streaming nodes with selecting one metric from CPU/memory Utilization, Input Events, Backlogged Input Events, and Watermark delay. And then take corrective actions.

 

More details are available in this doc: Stream Analytics job diagram (preview) in Azure portal - Azure Stream Analytics

 

Job diagram

 

 

 

Job diagram simulator in VS code (Public Preview)

 

One way to improve the performance of an Azure Stream Analytics job is to leverage parallelism in query. The job diagram simulator in VS Code provides a capability to simulate a job running topology with different Streaming Units and highlight the query steps that are not partitioned correctly. Then you can modify your query based on the suggestions provided to make it a parallel job.

 

More details are available in this doc: Optimize query’s performance using job diagram simulator in VS Code

 

Job diagram simulator

 

 

 

End-to-end exactly once processing with ADLS Gen2 output (Public Preview)

 

Stream Analytics now supports end-to-end exactly once semantics when reading any streaming input and writing to Azure Data Lake Storage Gen2. Your jobs now guarantee no data loss and no duplicates being produced as output. Stream Analytics now guarantees faster recovery and also guarantees exactly once output across user-initiated restarts from last output time, preventing duplicates from being produced. This greatly simplifies your streaming pipeline by not having to monitor, implement, and troubleshoot deduplication logic.

 

More details are available in this doc: Blob storage and Azure Data Lake Gen2 output from Azure Stream Analytics

 

Exactly once

 

 

 

Improved recovery during runtime version upgrades (General Availability)

 

Stream Analytics dramatically simplifies infrastructure management where you do not have to worry about runtime versions, security patches and upgrades as these are all automatically taken care for you behind the scenes. Your jobs are always running with the most updated runtime version which keeps improving over time. When we ship new versions of the Stream Analytics runtime, we automatically upgrade your jobs which triggers an internal restart to pick up the latest version. We have made significant improvements to the checkpoint restoration mechanism to allow for rapid recovery. Even if your job has lot of complex stateful logic, it will instantly resume from where it last stopped before the upgrade.

 

 

 

Azure Data Explorer (ADX) output (General Availability)

 

You can now use Azure Data Explorer as a native output in Azure Stream Analytics. This enables you to tackle scenarios such as pre-processing or analyzing event streams on the fly before writing it to the target destination in Azure Data Explorer.

 

More details are available in this doc: Azure Data Explorer output from Azure Stream Analytics

 

 

 

User-Assigned Managed Identity (General Availability)

 

Use user-assigned managed identities to authenticate your job's inputs and outputs, eliminate the need to manage credentials in code and allow Azure to manage this for you. With Managed Identity, take advantage of fully automated authentication by eliminating the limitations of user-based authentication.

 

More details are available in this doc:

 

User -Assigned Managed Identity - User-assigned managed identities for Azure Stream Analytics

 

 

 

Enhanced troubleshooting experience with improved errors and logging (General Availability)

 

We have enhanced the developer experience with significant improvements to messaging and logging for query errors.

 

ASA users can now get more actionable information so that they can quickly resolve what happened, why, and how to get to a better place. All error messages are self-contained, descriptive, and actionable to improve user productivity.

 

More details are available in this blog: Azure Stream Analytics Enhances Developer Experience with Improved Error Messaging and Logging - Microsoft Tech Community

 

 

 

 

SQL DB output connector improvements

 

We are continuously working on improving query development experience in portal. In the previous release, you read how you can configure your Azure Stream Analytics job to write to a table in your Azure SQL Database that hasn't yet been created. You can also see schema mismatch detection if you want to write output of Stream Analytics query to an existing SQL table.

 

What if you want to alter the existing SQL table based on incoming/transformed schema? You can now do that in your Stream Analytics job itself!

 

More details are available in this doc: Write to Azure SQL Database table from your Azure Stream Analytics jobs | Microsoft Learn

 

 

 

 

The Azure Stream Analytics team is highly committed to listening to your feedback. We welcome you to join the conversation and make your voice heard via submitting ideas and feedback. You can stay up-to-date on the latest announcements by following us on Twitter @AzureStreaming. If you have any questions or run into any issues accessing any of these new improvements, you can also reach out to us at askasa@microsoft.com.

 

Continue reading...