Introducing Azure Data Factory Community Templates

[Guest Noelle_Li]

Today, we’re excited to announce that Community Templates are now available, giving our community members an opportunity to contribute to our existing template gallery.

 

 

 

Templates are predefined Azure Data Factory pipelines that allow you to get started quickly with Data Factory. Templates are especially useful when you're new to Azure Data Factory or Azure Synapse pipelines and want to get started quickly as they reduce development time for building data integration projects and improve developer productivity. A wide variety of best practices are also infused in templates that cater to different scenarios that you can build on top of. And if you have Git enabled, templates help create shareable pipelines for better collaboration within the same factory.

 

 

 

To see our current Community Templates and Guidelines, please read Azure-DataFactory/community templates at main · Azure/Azure-DataFactory (github.com).

 

 

 

 

 

 

 

 

NOTE: Community template submissions will be reviewed by the team. If your submission, does not meet our guidelines or quality checks, we will not merge your template into the gallery.

 

 

 

You will need to follow 3 steps to successfully submit a community template: create a template, test the template, and submit it.

 

 

 

Create a template

 

 

All pipeline templates should follow ARM template syntax. There are 2 ways for you to do this.

 

 

 

Create a template from an existing pipeline

 

 

You can create a template from a new or existing pipeline in your data factory. First, navigate to the Author tab to create a new pipeline or select an existing pipeline.

 

 

 

Once your pipeline is ready, click the Actions button (3 dots next to the pipeline configuration panel) to find a list of actions. Select Export template to download a zip folder. This folder will contain your pipeline JSON file and the manifest.json file.

 

 

 

 

 

NOTE: Before exporting your template, make sure to scrub your pipeline of personal information. If you are using parameters, please set the values to Default or an example format. Additionally, add descriptions to your pipeline to better inform users of what the pipeline is doing.

 

 

 

In the manifest.json file, please make sure to add a property called “contributorType” and set it to “Community”. You can also update the “author” property with the name of the contributor. Additional properties to modify include “annotations”, “services”, and “categories”. Updating these properties will add filters so users can more easily find a template. Since the template gallery already has many categories, tags (annotations), and services, please try to use what already exists in the template gallery, unless what you need does not currently exist.

 

 

 

 

 

 

You must also add a “documentation” property In the manifest.json file and set it to a documentation link. This will show in the template and help instruct users on how to properly use the template. Please include your documentation in our repository in the documentation folder found within the community templates folder.

 

 

 

 

 

 

NOTE: Your manifest.json file MUST include the following or it will not be merged:

 

• a property called “contributorType” that is set to “Community”

  
  

• a property called “author” that is set to the name or group name of the template contributor

  
  

• a property called "documentation" that is set to a documentation link of how to use the template

  
  

 

 

 

Code a template

 

 

You can code a template on your own. Please make sure to include your pipeline JSON file and a corresponding manifest.json file.

 

 

 

Test your template

 

 

 

 

Before submitting your template, make sure that the template is looks and behaves correctly. To complete this testing, you will need to import your template to see if the pipeline can be successfully created from it.

 

 

 

In the Author section, click the + symbol under Factory Resources to open a submenu. Here you will see a list of options.

 

 

 

 

 

 

Click Pipeline and choose Import from pipeline template.

 

 

 

 

 

 

You will then be prompted to select a local template file.

 

 

 

NOTE: The template file that you import must be in zip format.

 

 

 

Once the template file is imported successfully, you will be taken to a new pane. This pane should show one or more input elements for linked services, which you configured in the template file. Inputs can include source and sink linked services. You should also be able to view your tags, services, and documentation, if it was included.

 

 

 

Use the drop-downs to select a linked services for each input.

 

 

 

 

 

 

 

 

If the template is not usable, an error hint will show in the top of the pane and you will not be able to continue.

 

 

 

 

 

 

Once everything is configured properly, click the Use this template button. This will automatically navigate you back to the authoring blade.

 

 

 

 

 

 

If the template is correct, then a pipeline will be created from it automatically.

 

 

 

 

 

 

Please take the time to review this newly created pipeline to ensure that your template is descriptive and does not contain any personal or sensitive information.

 

 

 

Submit your template

 

 

 

 

Once you’ve created and tested your template, you can submit your template(s) to Azure-DataFactory. Community templates should be added to the community templates folder. The repository has samples and existing community templates for reference.

 

 

 

When you’re ready, create a new pull request and our team will review it! Once everything is verified, it will be deployed to the Azure Data Factory Template Gallery.

 

 

 

We can't wait to see what you come up with!

 

 

Continue reading...