Azure Data Factory January 2023 Monthly Update

[Guest Noelle_Li]

Welcome to Azure Data Factory’s January monthly update! Here we’ll share the latest updates on what’s new in Azure Data Factory. You can find all our updates here.

 

 

 

We’ll also be hosting our monthly livestream next week on February 16th, 2023! Join us to see some live demos and to ask us your ADF questions!

 

 

 

Join the livestream

.

 

 

 

 

 

 

Table of Contents

 

Data flow

 

• Flowlets now supports schema drift [Learn more]
  
• SQL CDC incremental extract now supports numeric columns [Learn more]
  

Data movement

 

• New top-level CDC resource - native CDC configuration in 3 simple steps [Learn more]
  

Orchestration

 

• Orchestrate Synapse notebooks and Synapse spark job definitions natively [Learn more]
  

Region expansion

 

• Continued region expansion – China North 3 now supported [Learn more]
  
• Continued region expansion – Switzerland West now supported [Learn more]
  

SQL Server Integration Services (SSIS)

 

• Express virtual network injection for SSIS now generally available [Learn more]
  

Developer Productivity

 

• Added support for favoriting resources on home page
  
• Save column width for Pipeline monitoring
  
• Monitor Filter Updates for faster searches
  
• Directly access Pipeline Template Gallery through Azure portal [Learn more]
  

 

 

 

Data flow

 

Flowlets now supports schema drift

 

 

Allow schema drift is now supported for flowlets. Flowlets are reusuable activities that can be created from scratch or from an existing mapping dataflow. Now with schema drift support, users can use their flowlets for datasets that have changing source columns, allowing for more flexible flowlet use across many mapping dataflows.

 

 

 

 

 

 

For more information about schema drift, read Schema drift in mapping data flow - Azure Data Factory & Azure Synapse | Microsoft Learn.

 

 

 

SQL CDC incremental extract now supports numeric columns

 

 

Enabling incremental extract in SQL sources in mapping dataflows allows for users to only process rows that have changed since the last time that pipeline was executed. Previously when enabling incremental extract, users were only allowed to select a date/time column as a watermark. Now, supported incremental column types include date/time and numeric columns.

 

 

 

For more information about change data capture, read Change Data Capture - Azure Data Factory & Azure Synapse | Microsoft Learn.

 

 

 

Data movement

 

New top-level CDC resource for easier native CDC configuration

 

 

One of the most difficult aspects to building enterprise-ready data integration pipelines is capturing just the latest changed data from disparate sources quickly. Change Data Capture (CDC) is a set of technologies and techniques that enables data engineers to discover changed records as they occur at the source. We are excited to share that a new CDC top-level resource is now available in Public Preview that will allow you to automatically read changes from your data sources in 3 simple steps!

 

 

 

 

 

 

For more information, read Announcing the Public Preview of a new CDC top-level resource in ADF (microsoft.com).

 

 

 

Orchestration

 

Orchestrate Synapse notebooks and Synapse spark job definitions natively

 

 

Azure Data Factory now supports orchestrating Synapse notebooks and Synapse spark job definitions (SJD) natively. This will help users who have invested in both ADF and Synapse Spark without having to switch to Synapse pipelines to orchestrate Synapse notebooks and spark job definitions. This also allows users more flexibility to use Spark SQL and PySpark to perform data transformations, from simple data cleaning tasks to more complex data processing tasks.

 

 

 

 

 

 

To learn more, read Orchestrate and operationalize Synapse Notebooks and Spark Job Definitions from Azure Data Factory (microsoft.com).

 

 

Region expansion

 

Continued region expansion – China North 3 now supported

 

 

Azure Data Factory is now available in China North 3. You can co-locate your ETL workflow in this new region if you are utilizing the region for storing and managing your modern data warehouse.

 

 

 

 

To learn more about supported regions, read Azure Products by Region | Microsoft Azure.

 

 

 

Continued region expansion – Switzerland West now supported

 

 

Azure Data Factory is now available in Switzerland West. You can co-locate your ETL workflow in this new region if you are utilizing the region for storing and managing your modern data warehouse.

 

 

 

 

 

 

To learn more about supported regions, read Azure Products by Region | Microsoft Azure.

 

 

SQL Server Integration Services (SSIS)

 

Express virtual network injection for SSIS is now generally available

 

 

Express virtual network injection for SSIS is now Generally Available. This method of injecting your SSIS integration runtime into a virtual network allows for your SSIS integration runtime provisions and starts faster. Inbound traffic is also not needed to meet Enterprise security compliance requirements.

 

 

 

To learn more about express virtual network injection for SSIS, read General Availability of Express Virtual Network injection for SSIS in Azure Data Factory.

 

 

 

Developer Productivity

 

Added support for favoriting resources on home page

 

 

In ADF, users can now pin their favorite resources on the ADF home page. This will allow for users to quickly and easily access most frequently used resources.

 

 

 

 

 

 

Save column width for Pipeline monitoring

 

 

When users change the width of columns in Pipeline Monitoring, the changes will now be saved. This allows for users to focus on monitoring without having to constantly update their monitoring view.

 

 

 

Monitor Filter updates for faster searches

 

 

In Pipeline Monitoring, improvements have been made to support faster filtering so that users can more easily find what they’re searching for. Users can also use autocomplete when searching through triggered pipeline runs to make searching quicker.

 

 

 

Directly access Pipeline Template Gallery through Azure portal

 

 

In the Azure portal, users can now directly access the Pipeline template gallery when launching their data factory. This will allow for easy access to the Azure Data Factory studio and help users quickly create pipelines from predefined pipeline scenarios.

 

 

 

 

To learn more about this, readDirectly Launch ADF Pipeline Template Gallery through the Azure Portal (microsoft.com).

 

 

 

We hope that you found this helpful! Let us know in the comments if there's anything else you'd like to see in our blogs or livestreams. We love hearing your feedback!

 

Continue reading...