F
flanakin
Whether you consider yourself a FinOps practitioner, someone who’s enthusiastic about driving cloud efficiency and maximizing the value you get from the cloud, or were just asked to look at ways to reduce cost, the FinOps toolkit has something for you. In August, we added support for Power BI reports on top of Cost Management exports without needing to deploy FinOps hubs; expanded optimization options in workbooks; improved optimization, security, and resiliency in Azure Optimization Engine; a new FOCUS article to help compare with actual/amortized data; and many smaller fixes and improvements across the board.
And if you missed our update last month, check out the July update, too!
In case you haven’t heard, the FinOps toolkit is an open-source collection of tools and resources that help you learn, adopt, and implement FinOps in the Microsoft Cloud. The foundation of the toolkit is the Implementing FinOps guide that helps you get started with FinOps whether you’re using native tools in the Azure portal, looking for ways to automate and extend those tools, or if you’re looking to build your own FinOps tools and reports. To learn more about the toolkit, how to provide feedback, or how to contribute, see the FinOps toolkit site.
As people move from the actual and amortized cost datasets to the FinOps Open Cost and Usage Specification (FOCUS), we often get questions about how one column maps to another. We offer scenario-based documentation to help you update existing reports to use FOCUS and also how to convert actual/amortized data to FOCUS, but everyone once in a while we get a question about how to compare a specific number in FOCUS to the actual or amortized cost datasets. To facilitate this, we’ve introduced a new article to help you compare FOCUS data to either actual or amortized cost data you might be seeing in the Azure portal or other tools.
The document offers one table for comparing with actual cost data and another for comparing with amortized cost data. Most columns are the same, except when it gets into the differences between purchases and refunds vs. effective costs after commitment discount purchases have been amortized.
To learn more, check out the full article, Validate FOCUS data.
Screenshot of the article that helps compare FOCUS cost data with actual and amortized cost data
One of the initial goals of the FinOps toolkit was to offer a scalable solution for cost reporting. This is partially why FinOps hubs were created. Of course, FinOps hubs have a broad vision that goes beyond cost reporting, which not everyone needs. For that reason, we’ve continued to include Power BI reports that work with the Cost Management connector. But the connector is currently in maintenance support only and is no longer receiving updates due to intrinsic limitations in how the connector worked. Going forward, the new recommendation for reporting on costs in Power BI is to export data to a storage account and leverage the Azure Data Lake Storage Gen2 connector. And with the ability to overwrite old month-to-date reports, Cost Management exports now provide everything you need to report on costs directly on top of raw, Cost Management exports. With that, one of the man features we included in the FinOps toolkit 0.5 release was support for Power BI reports connected to raw exports in storage without the need to use FinOps hubs.
While there are still a few reasons to use FinOps hubs, like managed exports, multi-tenant support, and a rich roadmap covering many FinOps capabilities, most organizations will find connecting Power BI reports directly to exported datasets in storage to be the simplest option for getting up and running in Power BI.
To connect Power BI reports to storage, simply copy the Data Lake storage endpoint from the Azure portal and paste it into the “Exports storage URL” parameter in Power BI. Add the target container and folder path to streamline data refreshes and close and apply changes.
Please note that uncompressed exports have additional size limitations that may cause timeouts or errors during data refresh. We recommend using compressed parquet exports when available for the best performance.
If you’re familiar with FinOps hubs, you may notice there are 2 storage parameters: one for hubs and one for exports. The intent of this is to support FinOps hubs for cost details and exports for other datasets, which aren’t supported by FinOps hubs yet. You can use both as needed. Simply set the hubs storage URL to the ingestion container and point the exports storage URL to container where your other exports are published. The only important point to remember is that raw exports MUST overwrite previous exports to ensure data is not duplicated. If you notice more cost and usage than expected, confirm the “Overwrite data” setting is on in export properties and enable it, if it isn’t.
With the additional support of raw exports covering all Cost Management datasets – namely reservation recommendations, which are included in the Rate optimization report – we’ve removed the Cost Management connector for Power BI. In order to see reservation recommendations, you will need to create an export and set the appropriate export storage URL in Power BI.
Beyond support for raw exports in Power BI, the following other updates were also made to the Power BI reports:
Let us know what you’d like to see next. Power BI reports are one of the most used tools in the FinOps toolkit, but also one we hear the least about. We’d love to know what we can do to streamline your reporting, deployment, and overall management of not only Power BI reports, but related FinOps solutions.
Every month we look for ways to improve the FinOps workbooks to make it easier for you to optimize and govern your cloud environment. In August, we added new queries and made usability improvements across both workbooks.
In the Optimization workbook:
In the Governance workbook:
We evaluate changes to our workbooks every month. Let us know what you’d like to see next!
In August, we improved recommendations, security, and flexibility in Azure Optimization Engine (AOE).
First, we improved underutilized premium disk recommendations to offer more accurate and actionable insights by updating the meter name, which changed in cost data, and including support for ZRS disks, which offer improved performance and reliability. This change will help you balance resiliency and maximize the value you get out of your cloud solutions.
Next up, given how critical security is for any cloud solution, we’ve updated the default security settings to enable Microsoft Entra ID authentication only for the SQL database. This update removes passwords from the initial deployment, making it easier and more secure.
Lastly, we also expanded support for adding additional Entra ID tenants (or directories), added an option to ingest Microsoft Customer Agreement (MCA) billing profiles, and resolved a bug related to MCA subscription cost data. To configure additional tenants, use the new Register-MultitenantAutomationSchedules PowerShell script.
We hope these improvements will help you maximize efficiency, security, and resiliency in your cloud solutions. We’re always looking for new opportunities like this, so please do let us know what you’d like to see next!
Many small improvements and bug fixes go into each release, so covering everything in detail can be a lot to take in. But I do want to call out a few other small things that you may be interested in.
In FinOps hubs:
In the FinOps toolkit PowerShell module:
In open data:
To see more details, refer to the FinOps toolkit changelog.
Here are a few of the things we’re looking at in the coming months:
To learn more, check out the FinOps toolkit roadmap, and please let us know if there’s anything you’d like to see in a future release. Whether you’re using native products, automating and extending those products, or using custom solutions, we’re here to help make FinOps easier to adopt and implement.
Continue reading...
And if you missed our update last month, check out the July update, too!
New to FinOps toolkit?
In case you haven’t heard, the FinOps toolkit is an open-source collection of tools and resources that help you learn, adopt, and implement FinOps in the Microsoft Cloud. The foundation of the toolkit is the Implementing FinOps guide that helps you get started with FinOps whether you’re using native tools in the Azure portal, looking for ways to automate and extend those tools, or if you’re looking to build your own FinOps tools and reports. To learn more about the toolkit, how to provide feedback, or how to contribute, see the FinOps toolkit site.
Comparing FOCUS and actual/amortized data
As people move from the actual and amortized cost datasets to the FinOps Open Cost and Usage Specification (FOCUS), we often get questions about how one column maps to another. We offer scenario-based documentation to help you update existing reports to use FOCUS and also how to convert actual/amortized data to FOCUS, but everyone once in a while we get a question about how to compare a specific number in FOCUS to the actual or amortized cost datasets. To facilitate this, we’ve introduced a new article to help you compare FOCUS data to either actual or amortized cost data you might be seeing in the Azure portal or other tools.
The document offers one table for comparing with actual cost data and another for comparing with amortized cost data. Most columns are the same, except when it gets into the differences between purchases and refunds vs. effective costs after commitment discount purchases have been amortized.
To learn more, check out the full article, Validate FOCUS data.
Screenshot of the article that helps compare FOCUS cost data with actual and amortized cost dataWhat’s new in Power BI
One of the initial goals of the FinOps toolkit was to offer a scalable solution for cost reporting. This is partially why FinOps hubs were created. Of course, FinOps hubs have a broad vision that goes beyond cost reporting, which not everyone needs. For that reason, we’ve continued to include Power BI reports that work with the Cost Management connector. But the connector is currently in maintenance support only and is no longer receiving updates due to intrinsic limitations in how the connector worked. Going forward, the new recommendation for reporting on costs in Power BI is to export data to a storage account and leverage the Azure Data Lake Storage Gen2 connector. And with the ability to overwrite old month-to-date reports, Cost Management exports now provide everything you need to report on costs directly on top of raw, Cost Management exports. With that, one of the man features we included in the FinOps toolkit 0.5 release was support for Power BI reports connected to raw exports in storage without the need to use FinOps hubs.
While there are still a few reasons to use FinOps hubs, like managed exports, multi-tenant support, and a rich roadmap covering many FinOps capabilities, most organizations will find connecting Power BI reports directly to exported datasets in storage to be the simplest option for getting up and running in Power BI.
To connect Power BI reports to storage, simply copy the Data Lake storage endpoint from the Azure portal and paste it into the “Exports storage URL” parameter in Power BI. Add the target container and folder path to streamline data refreshes and close and apply changes.
Please note that uncompressed exports have additional size limitations that may cause timeouts or errors during data refresh. We recommend using compressed parquet exports when available for the best performance.
If you’re familiar with FinOps hubs, you may notice there are 2 storage parameters: one for hubs and one for exports. The intent of this is to support FinOps hubs for cost details and exports for other datasets, which aren’t supported by FinOps hubs yet. You can use both as needed. Simply set the hubs storage URL to the ingestion container and point the exports storage URL to container where your other exports are published. The only important point to remember is that raw exports MUST overwrite previous exports to ensure data is not duplicated. If you notice more cost and usage than expected, confirm the “Overwrite data” setting is on in export properties and enable it, if it isn’t.
With the additional support of raw exports covering all Cost Management datasets – namely reservation recommendations, which are included in the Rate optimization report – we’ve removed the Cost Management connector for Power BI. In order to see reservation recommendations, you will need to create an export and set the appropriate export storage URL in Power BI.
Beyond support for raw exports in Power BI, the following other updates were also made to the Power BI reports:
- List cost and contracted cost numbers were updated to facilitate partial (not complete) cost savings calculations. It’s important to note this is not a complete cost savings. With support for the pricesheet dataset, we will include a full backfill of missing price/cost data in a future release.
- Minor updates to the Cost summary DQ page to identify rows where a unique ID cannot be identifies and rows where the billing and pricing currencies are different.
- Merged shared and single reservation recommendations pages into a consolidated Reservation recommendations page in the Rate optimization report. Note this requires reservation recommendations to be exported from Cost Management to see data.
- Fixed a bug where Cost Management exports showed committed usage as “Standard” pricing category, which can cause previous reports to be miscategorized. To fix the underlying data, please re-export any FOCUS 1.0 datasets that were exported before August 1, 2024.
Let us know what you’d like to see next. Power BI reports are one of the most used tools in the FinOps toolkit, but also one we hear the least about. We’d love to know what we can do to streamline your reporting, deployment, and overall management of not only Power BI reports, but related FinOps solutions.
What’s new in FinOps workbooks
Every month we look for ways to improve the FinOps workbooks to make it easier for you to optimize and govern your cloud environment. In August, we added new queries and made usability improvements across both workbooks.
In the Optimization workbook:
- New: Identify VMs by processor architecture type.
- New: Find SQL elastic pool instances with no databases.
- New: Detect premium disks for VMs that aren’t running.
- New: Export
- Updated: Identify reservation break-event point.
- Updated: Azure Hybrid Benefit core count for Virtual Machine Scale Sets was corrected.
- Updated: The idle disks query now ignores disks used by AKS pods.
- Updated: Excluded block blob storage accounts from the list of v1 storage accounts.
- Updated: Removed the management group filter to simplify subscription filtering.
In the Governance workbook:
- Updated: Removed the management group filter to simplify subscription filtering.
We evaluate changes to our workbooks every month. Let us know what you’d like to see next!
What's new in Azure Optimization Engine
In August, we improved recommendations, security, and flexibility in Azure Optimization Engine (AOE).
First, we improved underutilized premium disk recommendations to offer more accurate and actionable insights by updating the meter name, which changed in cost data, and including support for ZRS disks, which offer improved performance and reliability. This change will help you balance resiliency and maximize the value you get out of your cloud solutions.
Next up, given how critical security is for any cloud solution, we’ve updated the default security settings to enable Microsoft Entra ID authentication only for the SQL database. This update removes passwords from the initial deployment, making it easier and more secure.
Lastly, we also expanded support for adding additional Entra ID tenants (or directories), added an option to ingest Microsoft Customer Agreement (MCA) billing profiles, and resolved a bug related to MCA subscription cost data. To configure additional tenants, use the new Register-MultitenantAutomationSchedules PowerShell script.
We hope these improvements will help you maximize efficiency, security, and resiliency in your cloud solutions. We’re always looking for new opportunities like this, so please do let us know what you’d like to see next!
Other new and noteworthy updates
Many small improvements and bug fixes go into each release, so covering everything in detail can be a lot to take in. But I do want to call out a few other small things that you may be interested in.
In FinOps hubs:
- Expanded the cost estimate documentation to call out Power BI pricing and include a link to the Pricing Calculator.
- Various minor bug fixes and improvements.
In the FinOps toolkit PowerShell module:
- Added more filtering options in Get-FinOpsCostExport.
- Various minor bug fixes and improvements.
In open data:
- 16 new region values were added to map back to existing regions.
- 4 new pricing units were added and 1 was updated.
- 48 new resource types were added and 32 were updated.
- 20 new service mappings were added and 8 were updated.
To see more details, refer to the FinOps toolkit changelog.
What’s next
Here are a few of the things we’re looking at in the coming months:
- Power BI reports will expand to more FinOps capabilities – specifically, you’ll see new Governance and Workload optimization reports coming soon.
- FinOps hubs will add support for more datasets, private endpoints, and large datasets in Azure Data Explorer.
- FinOps workbooks will continue to get recurring updates, expand to more FinOps capabilities, and add cost from FinOps hubs.
- Azure Optimization Engine will continue to receive small updates as we plan out the next major release of the tool.
- Each release, we’ll try to pick at least one of the highest voted issues (based on
votes) to continue to evolve based on your feedback, so keep the feedback coming!
To learn more, check out the FinOps toolkit roadmap, and please let us know if there’s anything you’d like to see in a future release. Whether you’re using native products, automating and extending those products, or using custom solutions, we’re here to help make FinOps easier to adopt and implement.
Continue reading...