N
novenera
In collaboration with Xuhong Liu, Hila Menashe Hadady, and Wagner Silveira
Logic App Standard is a cloud-based service that allows you to create and run automated workflows to integrate apps, data, services, and systems. Load testing is the process of putting demand on a system and measuring its response to ensure it can handle high usage and traffic. Performing load tests on your Logic App Standard workflows is essential to identify performance bottlenecks, ensure scalability, and maintain reliable operation under peak loads.
Let us first revisit how Logic App runtime (running on top of Function runtime) works under the covers to better understand how each component behaves and the strategies to take for performance optimization. Below is a simplified diagram of the Logic App runtime in action (as explained in greater detail in this blog post: Azure Logic Apps Running Anywhere – Runtime Deep Dive (microsoft.com)), moreover, how are these components being monitored using Logic App Standard built-in metrics:
The tools you need for load testing depend on the type of trigger your Logic App uses (e.g., HTTP based trigger or event based such as Azure Service Bus based trigger or Azure Event Hub based trigger).
For HTTP-based triggers:
You can write your own JMeter scripts or use Azure Load Testing service. For example, you can create URL-based tests in Azure Load Testing service.
Add POST HTTP request to the test plan by getting the HTTP trigger invocation URL of the workflow.
Configure the load configuration such as number of test engines, load pattern, ramp-up time, etc. Do note the number of virtual user minutes you would incur based on these settings.
Add the application components you would like to monitor during the load test run:
Run the load test. Assuming there are no failures during the run, once the test completes, you can view the client-side and server-side metrics over the duration of the test:
Under server-side metrics, you can filter the services you want to observe, in this example I am filtering by Logic App, the compute where its hosted, as well as the storage account it uses.
This view reveals that there are spikes in compute CPU and memory percentage during the load test run, which you can investigate further in the next section. If this is your first run, you can use this as baseline values and compare this to the results from succeeding load tests to ascertain improvements to the optimization applied to the app or compute.
You can either use Azure Load Testing custom plugins to simulate sending events to downstream services like Event Hub or Azure Service Bus (Azure Load Testing with custom plugins to simulate device behaviors - Azure Architecture Center ) or write your own custom application to fire off the events.
Always define your performance targets/load profile. Are you measuring throughput (X workflows completed per minute), latency (Y ms workflow run completed), response time, CPU usage? If this is the first time running performance test, capture baseline performance targets observed, iteratively run test whenever there are optimizations done in code or compute configuration, and then compare values to check if there have been any improvements.
In the rest of this blog, we will use throughput as a metric (X workflows completed per minute) to determine and evaluate the outcome of load tests. We will demonstrate how to monitor the load test, analyze potential issues if the outcome does not meet our goals, and how to improve the results based on the collected metrics.
For instance, if you use both Event Hub and a Logic App Standard with event hub trigger, you can check the built-in metrics such us “incoming messages” and "outgoing messages" for Event hub throughput. To evaluate the throughput of the upstream service, monitor metrics to get the number of messages going out of Event Hub every minute.
Event Hub metrics reference: Event Hub Message Metrics (https://learn.microsoft.com/en-us/azure/event-hubs/monitor-event-hubs-reference#message-metrics)
You can analyze trends of Logic App Standard built-in workflow metrics captured during and after your load test. Among them, “Workflow Runs Completed Count” or “Workflow Runs Started Count” can be used to get information around the number of concurrent runs your logic apps.
For example, if the target of our load test is the throughput, you can use “Workflow Runs Completed Count” per minute as the metric to evaluate the test results.
Note: Remember to adjust the "Time Granularity" setting from its default of 5 minutes to 1 minute at the top right when you are looking to gauge the number of concurrent runs each minute using these metrics.
Workflow Job Execution Delay start to trend upwards or stay high for some time…
This could be an indication of resource contention at the compute level. Check the CPU and memory usage if trending upwards.
Note: You can also use App Service Diagnostics tool to cross validate that CPU usage is being a blocker
If we see CPU or Memory being a blocker (e.g., above 80%), then the next step is to check the autoscaling setup (e.g., target-based scaling or runtime autoscaling, max burst instance, prewarmed instances) and further configure them to accommodate the traffic. In addition, we can also consider increasing the number of workflows in the sample Logic App (Packing more workflows into Logic App Standard ).
You can also check the scaling behavior via Application Insights (attached to this Logic App), using the query below. The sample diagram indicates that the scaling did not happen and go beyond the “Always Ready Instance”, hence there are some issues with the Logic App scaling.
When you see scaling not happen, ensure the default “runtime scaling” is not disabled. You can consider remove target-based scaling (if you enabled in the past).
Logic Apps Standard has access to the App Service Diagnostics Tools, which gives you access to tools that helps identify availability and performance issues such as running High CPU analysis, memory analysis, HTTP 4xx Errors, Logic App Down or Reporting Errors.
Go to Availability and Performance -> High CPU analysis blade to get more information on what is causing the CPU to shoot up. In this example, it is observed to occur across all worker instances despite having scaled out.
You can also analyze the CPU Drill Down report to see Top 3 Instances CPU Usage and the app or process level CPU for each instance.
It is highly recommended optimize your Logic App code by implementing patterns and practices for performance and scalability, some of these include:
See reference blogs for additional strategies and best practices for optimizing Logic Apps workflows and design:
If Storage Account (SA) is the bottleneck - You can configure to shard across multiple storage accounts to distribute the load.
Tip: if you want to run your load test at a clean state (no old backlog of runs), create new storage accounts and update the app settings appropriately, before each round of load test.
Throughput - Enable concurrency to process multiple tasks simultaneously.
Concurrency support for Service Bus built-in connector in Logic Apps Standard - Microsoft Community Hub
Delete hanging instances before starting new round of Logic App Load test:
Ensure there are no workflow instances that are stuck and not completing. This can be done by editing host.json with parameters such as Jobs.CleanupJobPartitionPrefixes or Jobs.SuspendedJobPartitionPartitionPrefixes.
For more details, see the official documentation: Edit runtime and environment settings for Standard logic apps - Azure Logic Apps | Microsoft Learn)
Reference document: Premium Plan Settings (https://learn.microsoft.com/en-ca/azure/azure-functions/functions-premium-plan?tabs=portal#plan-and-sku-settings%3E)
During your load test, monitor the "online servers" metric to evaluate the scalability of your Logic App.
You can drill down into "failures" to see detailed error messages if you encounter unexpected failures during your tests.
An alternative way of observing Logic App performance is by looking at the Insights blade (currently in Public Preview), which is a built-in dashboard containing some high-level insights from these metrics.
Go to Monitoring -> Insights blade which opens the Overview tab. This view will give you a breakdown of all the workflow run, job and trigger count by completion status (Succeeded, Failed or Skipped):
The Workflows tab presents sum of succeeded runs, jobs and triggers broken down by workflows, and another one for failed runs, jobs, and triggers.
Compute tab displays compute-related metrics like instance count, average memory working set, CPU, and memory percentage…
… as well as Workflow Job Count and Workflow Job Execution delay:
Continue reading...
Overview
Logic App Standard is a cloud-based service that allows you to create and run automated workflows to integrate apps, data, services, and systems. Load testing is the process of putting demand on a system and measuring its response to ensure it can handle high usage and traffic. Performing load tests on your Logic App Standard workflows is essential to identify performance bottlenecks, ensure scalability, and maintain reliable operation under peak loads.
Revisiting Logic App Internals
Let us first revisit how Logic App runtime (running on top of Function runtime) works under the covers to better understand how each component behaves and the strategies to take for performance optimization. Below is a simplified diagram of the Logic App runtime in action (as explained in greater detail in this blog post: Azure Logic Apps Running Anywhere – Runtime Deep Dive (microsoft.com)), moreover, how are these components being monitored using Logic App Standard built-in metrics:
- Workflow Job Execution Delay: The amount of time between when a job was scheduled to run and when the job actually ran.
- Workflow Job Execution Duration: The amount of time a job took to complete its execution.
- Workflow Runs Completed Count: The number of workflow executions completed, regardless of status.
- Workflow Runs Dispatched Count: The number of previously queued requests that are now processed.
- Workflow Runs Started Count: The number of workflows started, regardless of outcome status.
- Workflow Triggers Completed Count: The number of triggers completed, regardless of outcome.
- Workflow Action Completed Count: The number of actions completed, regardless of status.
Tools for Testing
The tools you need for load testing depend on the type of trigger your Logic App uses (e.g., HTTP based trigger or event based such as Azure Service Bus based trigger or Azure Event Hub based trigger).
For HTTP-based triggers:
You can write your own JMeter scripts or use Azure Load Testing service. For example, you can create URL-based tests in Azure Load Testing service.
Add POST HTTP request to the test plan by getting the HTTP trigger invocation URL of the workflow.
Configure the load configuration such as number of test engines, load pattern, ramp-up time, etc. Do note the number of virtual user minutes you would incur based on these settings.
Add the application components you would like to monitor during the load test run:
Run the load test. Assuming there are no failures during the run, once the test completes, you can view the client-side and server-side metrics over the duration of the test:
Under server-side metrics, you can filter the services you want to observe, in this example I am filtering by Logic App, the compute where its hosted, as well as the storage account it uses.
This view reveals that there are spikes in compute CPU and memory percentage during the load test run, which you can investigate further in the next section. If this is your first run, you can use this as baseline values and compare this to the results from succeeding load tests to ascertain improvements to the optimization applied to the app or compute.
Tips for using Azure Load Test in this scenario:
- Ensure that your JMeter scripts simulate realistic user behavior and cover all critical workflows.
- Use Azure Load Test to manage, execute, and analyze your load tests efficiently. It can integrate with your existing CI/CD pipeline to automate load testing.
For Event-based triggers
You can either use Azure Load Testing custom plugins to simulate sending events to downstream services like Event Hub or Azure Service Bus (Azure Load Testing with custom plugins to simulate device behaviors - Azure Architecture Center ) or write your own custom application to fire off the events.
Always define your performance targets/load profile. Are you measuring throughput (X workflows completed per minute), latency (Y ms workflow run completed), response time, CPU usage? If this is the first time running performance test, capture baseline performance targets observed, iteratively run test whenever there are optimizations done in code or compute configuration, and then compare values to check if there have been any improvements.
In the rest of this blog, we will use throughput as a metric (X workflows completed per minute) to determine and evaluate the outcome of load tests. We will demonstrate how to monitor the load test, analyze potential issues if the outcome does not meet our goals, and how to improve the results based on the collected metrics.
How to Monitor and Analyze the Test Results
Check the built-in metrics of the upstream event producer
For instance, if you use both Event Hub and a Logic App Standard with event hub trigger, you can check the built-in metrics such us “incoming messages” and "outgoing messages" for Event hub throughput. To evaluate the throughput of the upstream service, monitor metrics to get the number of messages going out of Event Hub every minute.
Event Hub metrics reference: Event Hub Message Metrics (https://learn.microsoft.com/en-us/azure/event-hubs/monitor-event-hubs-reference#message-metrics)
- When the “outgoing messages” numbers/min meet your required throughput, the next step is to check the Logic App performance, see if it can pick up and consume that volume of messages.
- If you see incoming messages rate significantly higher than outgoing messages, and the outgoing messages is rate not meeting the throughput you want, check and the performance of your Logic App to further analyze the reasons leading to the results that messages cannot be processed immediately.
Check Logic App Standard built-in workflow metrics and compute (App Service Plan) metrics
You can analyze trends of Logic App Standard built-in workflow metrics captured during and after your load test. Among them, “Workflow Runs Completed Count” or “Workflow Runs Started Count” can be used to get information around the number of concurrent runs your logic apps.
For example, if the target of our load test is the throughput, you can use “Workflow Runs Completed Count” per minute as the metric to evaluate the test results.
Note: Remember to adjust the "Time Granularity" setting from its default of 5 minutes to 1 minute at the top right when you are looking to gauge the number of concurrent runs each minute using these metrics.
Workflow Job Execution Delay start to trend upwards or stay high for some time…
This could be an indication of resource contention at the compute level. Check the CPU and memory usage if trending upwards.
Note: You can also use App Service Diagnostics tool to cross validate that CPU usage is being a blocker
If we see CPU or Memory being a blocker (e.g., above 80%), then the next step is to check the autoscaling setup (e.g., target-based scaling or runtime autoscaling, max burst instance, prewarmed instances) and further configure them to accommodate the traffic. In addition, we can also consider increasing the number of workflows in the sample Logic App (Packing more workflows into Logic App Standard ).
You can also check the scaling behavior via Application Insights (attached to this Logic App), using the query below. The sample diagram indicates that the scaling did not happen and go beyond the “Always Ready Instance”, hence there are some issues with the Logic App scaling.
When you see scaling not happen, ensure the default “runtime scaling” is not disabled. You can consider remove target-based scaling (if you enabled in the past).
Check App Service Diagnostics
Logic Apps Standard has access to the App Service Diagnostics Tools, which gives you access to tools that helps identify availability and performance issues such as running High CPU analysis, memory analysis, HTTP 4xx Errors, Logic App Down or Reporting Errors.
Go to Availability and Performance -> High CPU analysis blade to get more information on what is causing the CPU to shoot up. In this example, it is observed to occur across all worker instances despite having scaled out.
You can also analyze the CPU Drill Down report to see Top 3 Instances CPU Usage and the app or process level CPU for each instance.
Additional recommendations to monitor and improve Logic App Performance based on Load Test Results
Refactor code for performance and scalability
It is highly recommended optimize your Logic App code by implementing patterns and practices for performance and scalability, some of these include:
- Breaking a single workflow into two or more workflows
- For Stateful workflows – consider debatching workflows by using the SplitOn technique instead of looping thru the items using For-each
- For Stateless workflows – you can use For-each for stateless workflows only due to performance improvements
See reference blogs for additional strategies and best practices for optimizing Logic Apps workflows and design:
- Using Inline Code instead of a Foreach Loop for better performance in Logic Apps (microsoft.com)
- Improved For-each loop performance in Stateless Workflows - Microsoft Community Hub
- Scaling Logic App Standard – High Throughput Batched Invoice Processing System (microsoft.com)
- Scaling Logic Apps Standard – Sustained Event Processing System - Microsoft Community Hub
- Logic Apps Standard Performance Benchmark - Burst workloads - Microsoft Community Hub
- Logic Apps Standard Hosting & Performance Tips - Microsoft Community Hub
Scale storage account
If Storage Account (SA) is the bottleneck - You can configure to shard across multiple storage accounts to distribute the load.
Tip: if you want to run your load test at a clean state (no old backlog of runs), create new storage accounts and update the app settings appropriately, before each round of load test.
Throughput - Enable concurrency to process multiple tasks simultaneously.
Concurrency support for Service Bus built-in connector in Logic Apps Standard - Microsoft Community Hub
Delete hanging instances before starting new round of Logic App Load test:
Ensure there are no workflow instances that are stuck and not completing. This can be done by editing host.json with parameters such as Jobs.CleanupJobPartitionPrefixes or Jobs.SuspendedJobPartitionPartitionPrefixes.
For more details, see the official documentation: Edit runtime and environment settings for Standard logic apps - Azure Logic Apps | Microsoft Learn)
Reference document: Premium Plan Settings (https://learn.microsoft.com/en-ca/azure/azure-functions/functions-premium-plan?tabs=portal#plan-and-sku-settings%3E)
Check Application Insights attached to the Logic App
During your load test, monitor the "online servers" metric to evaluate the scalability of your Logic App.
You can drill down into "failures" to see detailed error messages if you encounter unexpected failures during your tests.
Observe using Logic App Insights (Public Preview)
An alternative way of observing Logic App performance is by looking at the Insights blade (currently in Public Preview), which is a built-in dashboard containing some high-level insights from these metrics.
Go to Monitoring -> Insights blade which opens the Overview tab. This view will give you a breakdown of all the workflow run, job and trigger count by completion status (Succeeded, Failed or Skipped):
The Workflows tab presents sum of succeeded runs, jobs and triggers broken down by workflows, and another one for failed runs, jobs, and triggers.
Compute tab displays compute-related metrics like instance count, average memory working set, CPU, and memory percentage…
… as well as Workflow Job Count and Workflow Job Execution delay:
Continue reading...