Simplifying Migration to Fabric Real-Time Intelligence for Power BI Real Time Reports

vashriva · Oct 31, 2024

Simplifying Migration to Fabric Real-Time Intelligence for Power BI Real Time Reports

Power BI with real-time streaming has been the preferred solution for users to visualize streaming data. Real-time streaming in Power BI is being retired. We recommend users to start planning the migration of their data processing pipeline to Fabric Real-Time Intelligence.

Microsoft Fabric Real-Time Intelligence is a part of the Microsoft Fabric platform. It enables businesses to ingest, process, analyze, and gain insights from real-time data streams. It provides an end-to-end solution for integrating real-time data pipelines, offering advanced analytics and data visualization in one place.

This document outlines the patterns and best practices as you explore Fabric Real-Time Intelligence and the Azure ecosystem.

1. Real-Time data flow architecture

Figure 1.1 below is a general data flow pattern for a real time data analysis and visualization on Azure.

Figure1.1

On the left side, we have common real time data ingestion service like Azure Event hub, IoTHub etc. Next, we have azure stream analytics which is a fully managed stream processing engine that is designed to analyze and process large volumes of streaming data with sub-millisecond latencies. You will also find some event driven applications using Power Automate or Logic Apps for simple event processing. The processed events are then sent to streaming datasets in Power BI and to some persistent storage layer.

2. Insights into Fabric Real-Time Intelligence

Fabric Real-Time intelligence is a powerful service that empowers everyone in your organization to extract insights and visualize their data in motion.

2.1 Ingestion & Processing

Eventstream in the Microsoft Fabric Real-Time Intelligence lets you bring real-time events into Fabric, transform them, and then route them to various destinations without writing any code. Event streams provide you with multiple source connectors to fetch event data from the various sources.

If you want to connect your own application with an eventstream, you can add a custom endpoint or a custom app as a source. This is described in detail in the following section.

In Eventstream, the event processor editor is a no-code experience that allows you to drag and drop to design the event data processing logic. Here is a link to learn more about event processor editor Event processor editor. You can then route the transformed data to various destinations.

For real-time reporting experience we would recommend using Fabric Eventhouse KQL database. Eventhouses provide a solution for handling and analyzing large volumes of data, particularly in scenarios requiring real-time analytics and exploration.

Figure 1.2

2.2 Deliver & Visualize

You can build real time reports using auto page refresh feature of Power BI with sources that support direct query. Power BI direct query is supported by Fabric Eventhouse KQL database. Use this link to learn more about automatic page refresh feature in PowerBI.

Direct query feature can be used with either of the selected Eventstream destination, Eventhouse or Lakehouse. However, Eventhouse KQL database offers robust solution for managing and analyzing substantial volumes of real-time data.

Eventhouse is designed to scale efficiently, ensuring effective performance and resource use. This design is beneficial in situations where timely insights are important. Specifically built for time-based, streaming events with features like auto indexing and partitioning based on ingestion time.

3. Azure Stream Analytics

Azure Stream Analytics PowerBI output sends the transformed data to PowerBI to build rich visualizations of analysis results. Migrating Azure Stream Analytics (ASA) jobs that utilize the Power BI output connector involves few key considerations to ensure a seamless transition and maintain real-time data visualization capabilities.

3.1 Ingestion & Processing

For users leveraging Stream Analytics with Power BI output connector in their architecture, who are unable to migrate their solution to Fabric, alternative patterns can be explored following Power BI output connector retirement.

With the processing and data analytics logic implemented in Azure Stream Analytics, you can:

Route Stream Analytics output to Fabric.
Switch to other Stream Analytics connectors that support direct query mode in Power BI.

To push Stream Analytics output data to Fabric you can use Eventstream custom endpoint connector. Here is a link to learn how to add a custom endpoint or a custom app as a source to Eventstream. Add a custom endpoint or custom app source to an eventstream

Once Eventstream custom endpoint setup is complete you should have an event hub namespace and connection details available in the Eventstream. Now back in Azure Stream Analytics you can select Event Hub output connector and add the custom With this setup your existing steam analytics job can now publish data to Fabric Eventstream. Next you can add Eventhouse as your destination in Eventstream and follow the same pattern described above.

Figure 1.3

The Power BI direct query feature is compatible with various ASA and Fabric Eventstream output destinations. Here is a list of other ASA and Eventstream output connectors you can use to build reports in PowerBI with direct query.

Other Azure Stream Analytics output connectors:

SQL Server
PostgreSQL
SynapseSQL
Azure Data Explorer

Other Eventstream destination:

Lakehouse

3.2 Deliver & Visualize

In Power BI you will select Get data with KQL database or one of the sources mentioned above to design your visualization and reports.

You can choose other sources with import mode if you do not plan to use PowerBI's auto page refresh feature.

4. Fabric, Real-Time dashboard

For the use cases where you are doing schedule refresh and looking for very short interval auto-refresh experience you can also explore Fabric Real-Time Dashboard. You can natively export Kusto Query Language (KQL) queries to a dashboard as visuals and later modify their underlying queries and visual formatting as needed. In addition to ease of data exploration, this fully integrated dashboard experience provides improved query and visualization performance.

Figure 1.4

5. Call to action

If you have questions on migration recommendations, please reach out to RTISupport.
For any questions on Azure stream analytics please contact askasa@microsoft.com

Simplifying Migration to Fabric Real-Time Intelligence for Power BI Real Time Reports

vashriva