New storage providers for Azure Durable Functions now generally available

[Guest lily-ma]

We’re excited to announce that support for the new Azure Durable Functions storage providers, Netherite and Microsoft SQL Server (MSSQL), is now generally available! Some of you may have learned about these new storage providers during the annual Build conference and/or from this preview announcement blog post.

 

 

 

Durable Functions enables you to write long-running, reliable, event-driven, and stateful logic on the serverless Azure Functions platform using everyday imperative code. We believe the new storage providers will enable a host of new scenarios for apps using Durable Functions because they allow apps to run at a higher scale, with greater price-performance efficiency, and with more portability compared to the default option Azure Storage.

 

 

 

Why we developed the new storage providers

 

Durable Functions requires a storage provider, the default of which is Azure Storage, to persist orchestration and entity states. Azure Storage is the lowest-cost and most convenient option because it requires no additional configuration. As such, it will remain the default storage provider for Durable Functions and will be under active development just like the new providers. Azure Storage, however, does have some notable limitations that prevent it from enabling certain scenarios for apps using Durable Functions:

 

• Azure Storage has limits on the number of transactions per second for a storage account, limiting the maximum scalability of a Durable Function app.
  
• Azure Storage has strict data size limits for queue messages and Azure Table entities, requiring slow and expensive workarounds when handling large payloads.
  
• Azure Storage costs can be hard to predict since they are per-transaction and have very limited support for batching.
  
• Azure Storage can’t easily support certain enterprise business continuity requirements, such as backup/restore and disaster recovery without data loss.
  
• Azure Storage can’t be used outside of the Azure cloud.
  

 

These limitations motivated investments in the new storage providers.

 

 

 

Netherite for maximum orchestration throughput

 

Netherite may sound familiar to you if you’re a Minecraft fan. It refers to a rare material that is more durable than diamond and is resistant to fire and lava. The Netherite storage provider aspires to have similar qualities in the context of Durable Functions.

 

 

 

To illustrate Netherite’s throughput performance, we show the results of an example benchmark called Hello5. This benchmark performs many independent orchestrations concurrently, each of which calls 5 activities in sequence. We made the activities do very little CPU work so we can measure how efficient orchestration progress is persisted to storage. Read more about this benchmark.

 

 

 

Single-Node Throughput	Multi-Node Throughput
[attachment=30357:name]	[attachment=30358:name]

 

 

 

Key observations for the Hello5 benchmark:

 

• On a single worker node, Netherite is about 2x faster than Azure Storage on EP1 (1 vcore) and about 4x faster than Azure Storage on EP3 (4 vcores).
  
• When scaling out, Netherite throughput remains roughly proportional to the number of vcores, while Azure Storage maxes out near 120 orchestrations/sec.
  
• On an EP3 plan, a single Netherite node reaches about the same throughput as 12 Azure Storage nodes, at about 1/12th the cost.
  
• On an EP3 plan with 12 nodes, Netherite outperforms Azure Storage by over an order of magnitude.
  

 

 

 

For more information and numbers, see the article on multi-node throughputs in the documentation.

 

 

 

Note: Results may vary due to many factors. We’re simply presenting the observations of a benchmark we did and are not claiming to guarantee specific numbers. We encourage you to run these tests yourself and tweak them for your purposes.

 

 

 

Microsoft SQL for maximum control and portability

 

The Microsoft SQL (MSSQL) storage provider for Durable Functions was designed to fulfill enterprise needs, including the ability to decouple from the Azure cloud.

 

 

 

MSSQL can run anywhere, including on-premises servers, Edge devices, Linux Docker containers, on the Azure SQL Database serverless tier, and even on competitor cloud providers like AWS and GCP. With the support of this storage provider, you can run Durable Functions anywhere that Azure Functions can run, including your own Azure Arc-enabled Kubernetes clusters. In fact, the Azure Functions Core Tools and the Azure Arc App Service extension have been updated to support automatically configuring Durable Function apps on a Kubernetes cluster with the MSSQL KEDA scaler for elastic scale-out.

 

 

 

 

 

 

In addition to the above, the MSSQL provider offers the following benefits:

 

• Data portability: Because data is stored in a single database, you can easily backup the data and migrate it in a new server or service as necessary.
  
• Data control: You have full control over the database, logins, and have direct access to the runtime data, making it easy to protect and secure as necessary. MSSQL also has great support for encryption and business continuity, ensuring that your apps can meet enterprise compliance requirements.
  
• Multitenancy: Multiple applications can share the same database in a way that isolates the data between each app using low-privilege SQL login credentials.
  
• 3rd party app integrations: MSSQL comes with a set of stored procedures, SQL functions, and views that allow you to easily integrate Durable orchestrations and entities into your existing SQL-based applications.
  

 

 

 

How to choose?

 

Here’s an image that highlights the key strengths of each storage provider:

 

 

 

 

 

 

For a comprehensive list of tradeoffs between the three providers, see “Comparing storage providers”.

 

 

 

Try it out!

 

It’s easy to configure Durable Functions to use the Netherite or MSSQL storage provider for local development or for apps running on Azure. See the following quickstarts for instructions:

 

• Configure storage provider - Netherite
  
• Configure storage provider - MSSQL
  

 

 

Resources

 

Detailed documentation on architecture, scaling, performance, and more can be found on our github.io pages for Netherite and MSSQL.

 

 

 

Development of these new storage providers happens in the open just like other features of Durable Functions. You can find their source code in the durabletask-netherite and durabletask-mssql GitHub repos. You’re welcome to submit PRs if you’d like to contribute to the development of these new providers, as well as to the default Azure Storage provider.

 

 

 

If you have suggestions, questions, and/or bugs to report, you can file them as issues in the azure-functions-durable-extension repo.

 

 

 

Thank you for your interest in Azure Durable Functions!

 

Continue reading...