Azure Data Factory February 2023 Monthly Update

[Guest Noelle_Li]

Welcome to Azure Data Factory’s February monthly update! Here we’ll share the latest updates on what’s new in Azure Data Factory. You can find all our updates here.

 

 

 

We’ll also be hosting our monthly livestream next week on March 14h, 2023! Join us to see some live demos and to ask us your ADF questions!

 

 

 

Join the livestream

.

 

 

 

 

 

Table of Contents

 

Data movement

 

• Anonymous authentication type supported for Azure Blob storage [Learn more]
  
• Updated SAP template to easily move SAP data to ADLSGen2 in Delta format [Learn more]
  

Monitoring

 

• Container monitoring view available in default ADF studio [Learn more]
  

Orchestration

 

• Set pipeline output value (Public preview) [Learn more]
  
• Managed Airflow (Public preview) [Learn more]
  

Developer Productivity

 

• Dark theme support added (Public preview) [Learn more]
  

 

 

 

 

 

Data movement

 

Anonymous authentication type supported for Azure Blob storage

 

 

Now, users can use the Service Principal authentication type for Azure Blob Storage! This allows you to use Azure role-based access control (Azure RBAC) to grant permissions to a security principal, which may be a user, group, or application service principal, who can then authenticate with Azure AD to authorize a request.

 

 

 

 

 

 

To learn more, read Copy and transform data in Azure Blob Storage - Azure Data Factory & Azure Synapse | Microsoft Learn.

 

 

 

Updated SAP template to easily move SAP data to ADLSGen2 in Delta format

 

 

A new SAP template has been added to the Azure Data Factory Template Gallery. This template helps users quickly get started to replicate multiple tables from SAP to ADLSGen2 in Delta format using a key partition.

 

 

 

 

 

 

For more information, read SAP Templates - Azure Data Factory | Microsoft Learn.

 

 

 

Monitoring

 

Container monitoring view available in default ADF studio

 

 

When monitoring your pipeline run, you now have the option to enable the Container view, providing a consolidated view of the activities that ran in the output of your pipeline debug run and in the detailed monitoring view found in the monitoring tab. This view, previously accessible through the ADF Preview Experience, is now generally available in the ADF studio.

 

 

 

 

 

 

To learn more, read Managing Azure Data Factory studio preview experience - Azure Data Factory | Microsoft Learn.

 

 

Orchestration

 

Set pipeline output value (Public Preview)

 

 

When building complex workflows in the cloud with Azure Data Factory and Azure Synapse Pipelines, a very common pattern is to separate different workflow branches into child pipelines. Now, we have expanded the Set Variable activity to allow users to set a new system variable, called Pipeline Return Value. This allows you to customize your pipeline return value when communicating between child and parent pipelines.

 

 

 

 

 

 

To learn more about this new system variable, read Setting a pipeline return value with UI.

 

 

 

Managed Airflow

 

 

The capability to run Apache Airflow DAGs (Directed Acyclic Graph) within ADF is now in public preview, adding a key Open-Source integration that provides extensibility for orchestrating python-based workflows at scale on Azure. ADF Managed Airflow provides a managed orchestration service for Apache Airflow, simplifying the creation and management of Airflow environments while providing a native integration of Apache Airflow with Azure Active Directory for a single sign-on (SSO) and a more secure solution.

 

 

 

 

 

 

To learn more about Managed Airflow, read Introducing Managed Airflow in Azure Data Factory (microsoft.com).

 

 

 

Developer Productivity

 

Dark theme support added (Public preview)

 

 

As we spend more time on our screens, it has become increasingly important to make accommodations for light sensitivity and help reduce eye strain. With the availability of themes, ADF now offers a new dark color scheme for the Data Factory studio to help improve productivity.

 

 

 

 

 

 

To learn more about the dark theme in ADF, read Introducing Dark Mode for ADF Studio (microsoft.com).

 

 

 

We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in our blogs or livestreams. We love hearing your feedback!

 

Continue reading...