Granular Billing for Azure Data Factory

[Guest ChenyeCharlieZhu]

Overview

 

 

By default, Azure Data Factory reports lump sum charges for billing, meaning that at the factory level, we add up charges across all pipelines within a factory, and tell you how much you have spent on these pipelines. In many cases, these aggregate numbers should suffice. But in others, these numbers lack the clarity and transparency that we thrive to provide customers. For instance, if you are running data pipelines for multiple teams, you may want to determine the cost for each pipeline, for proper book-keeping and/or charge backs.

 

 

 

Now, Azure Data Factory will help you with this endeavor, with built-in per pipeline detailed billing view. Moreover, we built the feature on top of the Azure Billing and Cost Analysis platform, allowing you to stay with the cost and budget management tool that you are familiar with to identify spending trends and spot where overspending might have occurred.

 

 

 

Billing Report Behaviors

 

 

By opting in to the feature, you will have a separate line item for each of your pipelines. Charges associated with a pipeline will be grouped together under the pipeline name, giving you a clear view of the cost of operations. You also have the chance to get the aggregate view for your factories, as you can filter the charges by factory name in the Azure billing report.

 

 

 

 

 

 

NOTE: There will be one entry for each pipeline in your factory. Please be particularly aware if you have excessive amount of pipelines in the factory, as it may significantly lengthen your billing report.

 

 

 

How to Opt-in

 

 

You need to opt-in to this feature for every factory you want detailed billing for. To turn on the per pipeline detailed billing feature, (1) go to Azure Data Factory portal; (2) under the Manage tab, select Factory settings in the General section; (3) select Show billing report (preview) by pipeline; and (4) publish the change.

 

 

 

 

 

 

This setting is not included in the exported ARM templates from your factory, meaning that Continuous Integration and Delivery (CI/CD) will not overwrite billing behaviors for the factory. This neat trick allows you to set different billing behaviors for development, test, and production environments, even when they share the same pipeline definitions.

 

 

 

Known Limitations

 

 

 

 

1. Only Azure Data Factory billings will be included

 

Azure Data Factory runs on Azure infrastructure that accrues costs when you deploy new resources. It's important to understand that other extra infrastructure costs might accrue. For instance, when you move data across availability zones, bandwidth charges will apply. These charges will not be included in the per pipeline billing reports.

 

2. Certain Charges are inherently shared at factory level

 

These charges will file under a fallback line item for your factory:

 

• Data Factory Operations charges, including Read/Write and Monitoring
  
• Charges to host SQL Server Integration (Azure-SSIS) nodes
  

 

3. Dataflows with Time-to-Live setting

 

 

For now, Dataflows running on Azure Integration runtime with a Time-to-Live (TTL) setting will file under a fallback line item for your factory. We are iterating to improve the experience for our users.

 

 

 

A Special Note to Private Preview Customers

 

 

For all customers that previously onboarded to the feature's private preview whitelist: we will continue to honor our commitment and allow detailed billing for your factories. However, for the time being, you may notice some discrepancies in the setting: even when the factory setting states billing by factory, you may see detailed billing in your billing report.

 

 

 

We recommend the following steps:

 

1. Turn on detailed billings for factories you want detailed billing for
   
2. Select by factory billing for all other factories
   
3. Once ready with steps 1 and 2, contact your account manager to confirm you are ready for GA
   

 

We will gradually turn off the feature whitelist, as we progress to Public Preview and GA stages. Going forward, your factory settings will be the single source of control for the detailed billing behavior. Thank you so much for your understanding!

 

Continue reading...