F
flanakin
Whether you consider yourself a FinOps practitioner, someone who’s enthusiastic about driving cloud efficiency and maximizing the value you get from the cloud, or were just asked to look at ways to reduce cost, the FinOps toolkit has something for you. This month, we’re excited to share support for FOCUS 1.0, updated tools and resources aligned to the FinOps Framework 2024 updates, a new (but familiar) tool for cloud optimization recommendations, FinOps hubs managed exports and cross-tenant support, and a slew of other small improvements and bug fixes. There really is something for everyone!
In case you haven’t heard, the FinOps toolkit is an open-source collection of tools and resources that help you learn, adopt, and implement FinOps in the Microsoft Cloud. The foundation of the toolkit is the Implementing FinOps guide that helps you get started with FinOps whether you’re using native tools in the Azure portal, looking for ways to automate and extend those tools, or if you’re looking to build your own FinOps tools and reports. To learn more about the toolkit, how to provide feedback, or how to contribute, see the FinOps toolkit site.
Anyone who’s been deeply engaged in the FinOps community and following major FinOps news and events knows about the FinOps Open Cost and Usage Specification (FOCUS). And for those who don’t, FOCUS is a provider-agnostic schema for cost and usage data. We’ve been driving FOCUS alongside customers, partners, and our industry peers to define a data format that can become the new “language” of FinOps, helping FinOps practitioners achieve their goals easier than ever, backed by the FinOps Framework and accompanying resources, like the KPI and use case libraries.
FOCUS 1.0 was announced at FinOps X 2024 in June and we were happy to co-announce support for FOCUS 1.0 in Cost Management exports at the same time. Now, with the July release of FinOps toolkit 0.4, we’re happy to share that all Power BI reports and FinOps hubs have both been updated to FOCUS 1.0!
If you’re using the Cost Management connector, you can easily switch to the newest version of the Cost Management connector report, which works for all EA, MCA, and MPA billing accounts/profiles, to see all your cost and usage data aligned to the FOCUS 1.0 schema. Please note the underlying APIs and API constraints for the connector are still the same, so if you’re looking for more, please switch to using raw exports, FinOps hubs, or consider ingesting data into Microsoft Fabric.
If you’re using FinOps hubs – or looking to level-up your reporting from the connector – you can opt to leave your existing 1.0 preview data as-is and leverage the new Power BI reports, which transform 1.0 preview to FOCUS 1.0, or you can create new FOCUS 1.0 exports (and consider even back-filling data) to get the latest data. It’s always best to re-export your data using the latest schema because it reduces processing time in Power BI, but you can make the decision that works best for you. (Just remember to delete the old exports if you create a new FOCUS 1.0 export!) And if you’re automating export creation with PowerShell, you can also use the New-FinOpsCostExport command to create FOCUS 1.0 exports.
Lastly, sample data files were updated for FOCUS 1.0 and new metadata files were added for both FOCUS 1.0 and FOCUS 1.0 preview to describe the dataset in a programmatic way based on the FOCUS 1.0 specification.
To start leveraging these new reports, check out the FinOps toolkit Power BI reports or learn about how you can deploy FinOps hubs today. To learn more about FOCUS and it’s benefits – which range from general usability, improved learnability, and transferability of skills to cost savings from reduced dataset size and compute costs compared to processing native actual and amortized datasets – check out the FOCUS introduction.
In April 2023, the FinOps Foundation announced some changes to the FinOps Framework – the start of a series of yearly updates we can expect going forward. We saw a few things in this release:
I won’t go into the details of all the changes, but if you’re interested, check out Key 2024 Changes to the FinOps Framework: Evolving the Framework with the Practice of FinOps. Within the toolkit, you’ll find an updated Implementing FinOps guide with all the new and updated capabilities, including the supplemental content, like What is FinOps? and How to conduct a FinOps iteration.
Each capability was updated to include new and fresh guidance, including referencing new tools and solutions that have been released since our last major update. If you’re getting started with any capability, make sure you check out the Implementing FinOps guide to see how it can help you jump start your efforts!
And to add to that, we also renamed the “Commitment discounts” report in Power Bi to “Rate optimization” to align with the FinOps Framework changes. We envision dedicated reports for each FinOps capability, so let us know what you’d like to see by sharing your feedback in GitHub.
Over the years, some organizations tell us they’re looking for more configurability to Azure Advisor recommendations. Maybe they want to cover new scenarios or maybe they just want to increase the look-back period for existing recommendations. Whatever the case, this isn’t possible today with Azure Advisor, so customers have been looking to build their own solution. The Azure Optimization Engine (AOE) has been around since May 2020 as an open-source project run by Helder Pinto and we’re proud to add Helder to the FinOps toolkit family as he officially moved AOE into the FinOps toolkit!
There’s a ton to talk about with Azure Optimization Engine, so instead of trying to cram it all in here, I’ll direct you to several blog posts Helder’s published:
And of course, you can also learn more from the Azure Optimization Engine documentation. Stay tuned as we evolve the next generation of Azure Optimization Engine. And if you want to be part of the conversation, join us at The evolution of Azure Optimization Engine within the FinOps toolkit.
FinOps hubs are data pipeline solutions that help you kickstart custom reporting efforts when you need more than what you can get in the Azure portal today. Instead of starting from nothing, you can deploy a FinOps hub instance and leverage pre-built Power BI reports to get up and running quickly. This month, you’ll find a few interesting updates with FinOps hubs that we’ll continue to build onto based on your feedback.
First up is FOCUS 1.0 support. I already talked about the fact that we added support for FOCUS 1.0 but I’d like to expand on that a little. FinOps hubs 0.4 now supports both FOCUS 1.0 preview or FOCUS 1.0. Data that is ingested will remain in storage in the schema it was ingested in, but the Power BI reports in 0.4 will all align to the FOCUS 1.0 schema. If you prefer to keep using FOCUS 1.0 preview, stick with the FinOps toolkit 0.3 reports. 0.3 and 0.4 reports are interchangeable and can work with either version of FinOps hubs, so feel free to try the new reports with an existing FinOps hubs 0.3 deployment or upgrade to get the other new features and keep using your existing reports without a concern. The only reason you really need to use the newer reports is if you want to start exporting FOCUS 1.0 data.
As part of adding support for FOCUS 1.0, we also streamlined the data ingestion pipeline to prepare for ingesting other datasets, which we’ll add in an upcoming release. In the meantime, if you’d like to add support for additional export types, you can create a custom mapping file and save it to config/schemas/<dataset-type>_<dataset-version>.json in hub storage. Use the existing FocusCost_1.0.json as a template. Keep in mind some datasets have different schemas for EA and MCA, which cannot be identified via these attributes and may cause an issue if you have both account types. We will add all datasets in a future release and account for the EA and MCA differences by aligning to FOCUS.
Next is managed exports. Managed exports are like managed disks for virtual machines. You grant the hub access to create exports on any scopes you want to manage cost for and it’ll create and run exports for you. You can grant access to any EA or MCA subscription or resource group, or any EA billing scope. MCA billing scopes will also be available in the future when Cost Management adds support for service principal access.
Speaking of data ingestion, we also added a feature called remote hubs, which enables you to ingest data from a hub instance in another directory (or tenant). To configure remote hubs, simply deploy a hub instance in the directory where you want to pull data from, create a SAS token for that hub storage account, and specify a SAS token to ingest data across directories. And when Cost Management exports support FOCUS data in other clouds, this will also enable those interested in consolidating data across Azure Commercial, Azure Government, or Azure China clouds to view all data in a single tool. Let us know what you’d like to see in this regard and we’ll share updates as capabilities evolve.
Lastly, I’d like to call out some improvements to the FinOps hubs and Power BI troubleshooting guide. We created a dedicated page for common errors and introduced a new step-by-step troubleshooting walkthrough to help validate your hub instance and Power BI setup. You’ll also find a new Ingestion errors page in the Data ingestion report, which can also help identify common issues. Oh, and I suppose I should also call out that the Commitment discounts report was renamed to Rate optimization to align to the FinOps Framework 2024 updates, which I mentioned earlier.
Many small improvements and bug fixes go into each release, so covering everything in detail can be a lot to take in. But I do want to call out a few other small things that you may be interested in.
In Power BI reports:
In Azure Monitor workbooks:
In the FinOps toolkit PowerShell module:
In open data files:
Looking forward, we’re planning to shift to smaller, more frequent updates. We published our goals for 2024 in our roadmap but here are a few highlights:
And of course, you’ll continue to see updates for the Cost optimization workbook and open data files as usual. So stay engaged and let us know what works well and how things could be improved for you. Whether you’re using native products, automating and extending those products, or using custom solutions, we’re here to help make FinOps easier to adopt and implement.
Continue reading...
New to FinOps toolkit?
In case you haven’t heard, the FinOps toolkit is an open-source collection of tools and resources that help you learn, adopt, and implement FinOps in the Microsoft Cloud. The foundation of the toolkit is the Implementing FinOps guide that helps you get started with FinOps whether you’re using native tools in the Azure portal, looking for ways to automate and extend those tools, or if you’re looking to build your own FinOps tools and reports. To learn more about the toolkit, how to provide feedback, or how to contribute, see the FinOps toolkit site.
Support for FOCUS 1.0
Anyone who’s been deeply engaged in the FinOps community and following major FinOps news and events knows about the FinOps Open Cost and Usage Specification (FOCUS). And for those who don’t, FOCUS is a provider-agnostic schema for cost and usage data. We’ve been driving FOCUS alongside customers, partners, and our industry peers to define a data format that can become the new “language” of FinOps, helping FinOps practitioners achieve their goals easier than ever, backed by the FinOps Framework and accompanying resources, like the KPI and use case libraries.
FOCUS 1.0 was announced at FinOps X 2024 in June and we were happy to co-announce support for FOCUS 1.0 in Cost Management exports at the same time. Now, with the July release of FinOps toolkit 0.4, we’re happy to share that all Power BI reports and FinOps hubs have both been updated to FOCUS 1.0!
If you’re using the Cost Management connector, you can easily switch to the newest version of the Cost Management connector report, which works for all EA, MCA, and MPA billing accounts/profiles, to see all your cost and usage data aligned to the FOCUS 1.0 schema. Please note the underlying APIs and API constraints for the connector are still the same, so if you’re looking for more, please switch to using raw exports, FinOps hubs, or consider ingesting data into Microsoft Fabric.
If you’re using FinOps hubs – or looking to level-up your reporting from the connector – you can opt to leave your existing 1.0 preview data as-is and leverage the new Power BI reports, which transform 1.0 preview to FOCUS 1.0, or you can create new FOCUS 1.0 exports (and consider even back-filling data) to get the latest data. It’s always best to re-export your data using the latest schema because it reduces processing time in Power BI, but you can make the decision that works best for you. (Just remember to delete the old exports if you create a new FOCUS 1.0 export!) And if you’re automating export creation with PowerShell, you can also use the New-FinOpsCostExport command to create FOCUS 1.0 exports.
Lastly, sample data files were updated for FOCUS 1.0 and new metadata files were added for both FOCUS 1.0 and FOCUS 1.0 preview to describe the dataset in a programmatic way based on the FOCUS 1.0 specification.
To start leveraging these new reports, check out the FinOps toolkit Power BI reports or learn about how you can deploy FinOps hubs today. To learn more about FOCUS and it’s benefits – which range from general usability, improved learnability, and transferability of skills to cost savings from reduced dataset size and compute costs compared to processing native actual and amortized datasets – check out the FOCUS introduction.
What’s new in the FinOps Framework
In April 2023, the FinOps Foundation announced some changes to the FinOps Framework – the start of a series of yearly updates we can expect going forward. We saw a few things in this release:
- New definition of FinOps.
- Expanded set of personas.
- Simplified domains with clearer lines.
- Cleaned up list of FinOps capabilities with some joined and some new.
I won’t go into the details of all the changes, but if you’re interested, check out Key 2024 Changes to the FinOps Framework: Evolving the Framework with the Practice of FinOps. Within the toolkit, you’ll find an updated Implementing FinOps guide with all the new and updated capabilities, including the supplemental content, like What is FinOps? and How to conduct a FinOps iteration.
Each capability was updated to include new and fresh guidance, including referencing new tools and solutions that have been released since our last major update. If you’re getting started with any capability, make sure you check out the Implementing FinOps guide to see how it can help you jump start your efforts!
And to add to that, we also renamed the “Commitment discounts” report in Power Bi to “Rate optimization” to align with the FinOps Framework changes. We envision dedicated reports for each FinOps capability, so let us know what you’d like to see by sharing your feedback in GitHub.
Optimize your cloud environment with Azure Optimization Engine
Over the years, some organizations tell us they’re looking for more configurability to Azure Advisor recommendations. Maybe they want to cover new scenarios or maybe they just want to increase the look-back period for existing recommendations. Whatever the case, this isn’t possible today with Azure Advisor, so customers have been looking to build their own solution. The Azure Optimization Engine (AOE) has been around since May 2020 as an open-source project run by Helder Pinto and we’re proud to add Helder to the FinOps toolkit family as he officially moved AOE into the FinOps toolkit!
There’s a ton to talk about with Azure Optimization Engine, so instead of trying to cram it all in here, I’ll direct you to several blog posts Helder’s published:
- Augmenting Azure Advisor Cost Recommendations for Automated Continuous Optimization – Part 1
- Augmenting Azure Advisor Cost Recommendations for Automated Continuous Optimization – Part 2
- Augmenting Azure Advisor Cost Recommendations for Automated Continuous Optimization – Part 3
- Automating Continuous Optimization with the Azure Optimization Engine
- Get Azure Reservations and Savings Plans Insights with the Azure Optimization Engine
And of course, you can also learn more from the Azure Optimization Engine documentation. Stay tuned as we evolve the next generation of Azure Optimization Engine. And if you want to be part of the conversation, join us at The evolution of Azure Optimization Engine within the FinOps toolkit.
What's new in FinOps hubs
FinOps hubs are data pipeline solutions that help you kickstart custom reporting efforts when you need more than what you can get in the Azure portal today. Instead of starting from nothing, you can deploy a FinOps hub instance and leverage pre-built Power BI reports to get up and running quickly. This month, you’ll find a few interesting updates with FinOps hubs that we’ll continue to build onto based on your feedback.
First up is FOCUS 1.0 support. I already talked about the fact that we added support for FOCUS 1.0 but I’d like to expand on that a little. FinOps hubs 0.4 now supports both FOCUS 1.0 preview or FOCUS 1.0. Data that is ingested will remain in storage in the schema it was ingested in, but the Power BI reports in 0.4 will all align to the FOCUS 1.0 schema. If you prefer to keep using FOCUS 1.0 preview, stick with the FinOps toolkit 0.3 reports. 0.3 and 0.4 reports are interchangeable and can work with either version of FinOps hubs, so feel free to try the new reports with an existing FinOps hubs 0.3 deployment or upgrade to get the other new features and keep using your existing reports without a concern. The only reason you really need to use the newer reports is if you want to start exporting FOCUS 1.0 data.
As part of adding support for FOCUS 1.0, we also streamlined the data ingestion pipeline to prepare for ingesting other datasets, which we’ll add in an upcoming release. In the meantime, if you’d like to add support for additional export types, you can create a custom mapping file and save it to config/schemas/<dataset-type>_<dataset-version>.json in hub storage. Use the existing FocusCost_1.0.json as a template. Keep in mind some datasets have different schemas for EA and MCA, which cannot be identified via these attributes and may cause an issue if you have both account types. We will add all datasets in a future release and account for the EA and MCA differences by aligning to FOCUS.
Next is managed exports. Managed exports are like managed disks for virtual machines. You grant the hub access to create exports on any scopes you want to manage cost for and it’ll create and run exports for you. You can grant access to any EA or MCA subscription or resource group, or any EA billing scope. MCA billing scopes will also be available in the future when Cost Management adds support for service principal access.
Speaking of data ingestion, we also added a feature called remote hubs, which enables you to ingest data from a hub instance in another directory (or tenant). To configure remote hubs, simply deploy a hub instance in the directory where you want to pull data from, create a SAS token for that hub storage account, and specify a SAS token to ingest data across directories. And when Cost Management exports support FOCUS data in other clouds, this will also enable those interested in consolidating data across Azure Commercial, Azure Government, or Azure China clouds to view all data in a single tool. Let us know what you’d like to see in this regard and we’ll share updates as capabilities evolve.
Lastly, I’d like to call out some improvements to the FinOps hubs and Power BI troubleshooting guide. We created a dedicated page for common errors and introduced a new step-by-step troubleshooting walkthrough to help validate your hub instance and Power BI setup. You’ll also find a new Ingestion errors page in the Data ingestion report, which can also help identify common issues. Oh, and I suppose I should also call out that the Commitment discounts report was renamed to Rate optimization to align to the FinOps Framework 2024 updates, which I mentioned earlier.
Other new and noteworthy updates
Many small improvements and bug fixes go into each release, so covering everything in detail can be a lot to take in. But I do want to call out a few other small things that you may be interested in.
In Power BI reports:
- Use the new x_IncrementalRefreshDate column when configuring incremental refresh in Power BI.
- View resource count and cost per resource in the Inventory page of the Cost summary report.
- Simplify tag parsing with a new default “{}” value for tags to facilitate tag expansion.
- Load data faster with less memory and fewer errors with simpler, optimized queries.
In Azure Monitor workbooks:
- Get the latest updates from the Azure Advisor . For July, this includes:
- New reservations recommendations with break-even point.
- Redesigned sustainability tab.
- Removed dynamic IPs from the public IP addresses query to ensure more precise results.
- Excluded free tier web apps to focus on top services.
- Monitor managed disks in the Governance workbook.
In the FinOps toolkit PowerShell module:
- Track progress and retry throttling errors in Start-FinOpsCostExport.
- Removed support for Windows PowerShell due to inconsistencies with PowerShell 7.
In open data files:
- All open data files were updated to include the latest pricing units, services, resources, etc.
- All dataset examples were updated to recent month to align with the new FOCUS 1.0 dataset.
What’s next
Looking forward, we’re planning to shift to smaller, more frequent updates. We published our goals for 2024 in our roadmap but here are a few highlights:
- Power BI reports will expand to more FinOps capabilities and add support for raw exports without requiring the Cost Management connector or FinOps hubs.
- Azure Monitor workbooks will continue to get recurring updates, expand to more FinOps capabilities, and add cost from FinOps hubs.
- FinOps hubs will add support for more datasets, private endpoints, and large datasets in Azure Data Explorer.
- Azure Optimization Engine will continue to receive small updates as we plan out the next major release of the tool.
- Help and support experiences will continue to evolve across all tools.
- Each release, we’ll try to pick at least one of the highest voted issues (based on
votes) to continue to evolve based on your feedback, so keep the feedback coming!
And of course, you’ll continue to see updates for the Cost optimization workbook and open data files as usual. So stay engaged and let us know what works well and how things could be improved for you. Whether you’re using native products, automating and extending those products, or using custom solutions, we’re here to help make FinOps easier to adopt and implement.
Continue reading...