Bring your own Event Hubs to programmatically interact with Microsoft Purview

[Guest Naga_Yenamandra]

Microsoft Purview Data Map is based on Apache Atlas and extends full support for Apache Atlas APIs as well as Atlas Kafka topics. In order to support Kafka functionality, Purview provisioned a managed event hub namespace and two managed event hubs 'atlas_hook' and 'atlas_notification' to send and receive messages from Purview. However, this approach had multiple limitations listed below:

 

 

 

• 1. You cannot modify the configuration of these managed event hubs. For example, every managed event hub is configured with 1 TU and as a customer, you don't have the ability to increase TU's. The same applies to other configurations like retention period, consumer group, etc.
  
• 2. You cannot choose the Kafka topics name used to interact with Purview. Since it is a managed event hub, you cannot name these topics as per your organization's standards.
  
• 3. You cannot use multiple topics to interact with Purview Data Map. The managed event hub provides only one topic to publish messages and one topic to consume messages from Microsoft Purview.
  

 

In order to address the above limitations, we are excited to announce that Microsoft Purview now supports customers to bring their own Event Hubs and associate them with Microsoft Purview's data map to send and receive messages. This means that as a customer you will be managing these Event Hubs instead of Purview managing them on your behalf. Since customers are managing these Event Hubs, they get to decide the configuration as well as name these topics as per their organization standards. Also, multiple event hubs can be associated with the Purview account thus allowing customers to send messages to Purview account from multiple topics.

 

 

 

 

 

 

BYO Event Hub is now GA and available to all customers in all Purview-supported regions. To use BYO Event Hubs during account creation see our documentation here and to use it on an existing account see our documentation here.

 

Continue reading...