Pipeline Logic 3: Error Handling and Try Catch

[Guest ChenyeCharlieZhu]

Series Overview

 

 

Orchestration allows conditional logic and enables users to take different paths based upon outcomes of a previous activity. Building upon the concepts of conditional paths, ADF and Synapse Pipelines allow users to build versatile and resilient workflows that can handle unexpected errors that work smoothly in auto-pilot mode.

 

 

 

This is an ongoing series that gradually levels up and helps you build even more complicated logic to handle more scenarios. We will walk through examples for some common use cases, and help you to build functional and useful workflows.

 

 

 

Please review the first installment in the series: Part 1: Error Handling and Best Effort Step and Part 2: OR

 

 

 

Error Handling and Try Catch

 

 

Error handling is a very common scenario in data engineering pipelines. From time to time, activities will fail, but we don't want to fail the whole pipeline due to a single activity failure.

 

 

 

We call this logic: Try-Catch, and we have streamlined the implementation for this common use case.

 

 

 

 

 

 

• Add first activity
  
• Add error handling to the UponFailure path
  
• Add second activity, but don't connect to the first activity
  
• Connect both UponFailure and UponSkip paths from the error handling activity to the second activity.
   
  

 

To learn more, read Pipeline failure and error message - Azure Data Factory | Microsoft Learn.

 

 

 

We hope that you have found this blog to be helpful! If you have any questions or feedback, please post them in the comments below.

 

Continue reading...