The challenge of Cloud PC performance testing

[Guest Ron Martinsen]

Let's dive into why relative performance data for Windows 365 helps you compare different configurations and choose the one that suits your workload best.

 

Many years ago, I built the first large scale performance lab for the series of products that would become Office 97 for Windows and Office 98 Macintosh Edition. This process taught me a lifetime of lessons about the challenges of collecting performance data. I would continue to acquire new learnings during my experiences in SQL Server, Internet Explorer, Windows, and more.

 

After 28+ years at Microsoft, my key takeaway after being involved in countless performance studies—internally and externally with our partners—is that it's nearly impossible to get repeatable data in an environment that reflects the reality of what users will be using. The reason for this is the overwhelming number of variables involved, some of which include:

 

• The operating system that is paging memory in and out of storage in a performance-optimized but non-deterministic way
  
• Network latency due to variable traffic loads
  
• Disk controller caching
  
• Memory optimization that is dependent on the startup order of applications
  

 

The net result is that when you run the same test repeatedly under identical conditions, you get results that might not be an exact match each time. In fact, the combination of these variables listed can lead to occasional wild outliers in the results. Increased sample sizes help, but it can still be challenging to make valid comparisons.

 

Why performance testing is often done in unrealistic environments

 

 

When there are multiple variables impacting performance, the way to get reproducible numbers is to remove as many variables as possible. In other words, testing is done in a perfectly sterile environment devoid of countless variables that may skew results. This leads to the practice of product makers displaying the best possible numbers, even though they are not applicable to real world usage.

 

Instead of focusing on the absolute numbers, it's more productive to look at the relative differences between products. For example, a storage device listing 7000 Mbps is indeed faster than the one labelled 4000 Mbps, even though you may never see more than 250 Mbps in your own testing. The takeaway here is that the relative differences – not the absolutes - are what really matter when deciding which solution is right for you.

 

Why performance testing in the cloud is even harder

 

 

Imagine if you take all these variables and complexities and put them into an environment where the host compute is running on a cluster and serving a variable number of active Cloud PCs, which then, in turn, could host their own virtualization workloads (e.g., Windows Subsystem for Linux or Windows Subsystem for Android running on your Cloud PC via nested virtualization). At that point, the number of variables impacting performance are overwhelming, and isolation is impossible. Added to that is optimization in every part of the software and hardware components in the system, designed to maximize performance, and you begin to understand why two test runs never have identical results—how could they?

 

Azure shares performance data that's painstakingly collected to help admins configure their own custom virtual machines, as you can see in the following examples:

 

• Virtual machine and disk performance
  
• Azure Premium Storage: Design for high performance
  
• Azure Write Accelerator
  
• Managed disk bursting
  

Windows 365 simplifies administration; increases benchmarking variables

 

 

One of the benefits of Windows 365 is that it allows users to access their Cloud PCs from any supported device type and location. However, this also means that the underlying hardware of Cloud PCs may vary over time as Microsoft updates and upgrades its infrastructure. Therefore, it is not possible to provide exact performance metrics for each Cloud PC configuration as they may differ slightly depending on hardware configuration. Instead, Microsoft's goal is to ensure that all Cloud PCs meet expected service level agreements and provide a consistent user experience across different configurations.

 

Windows 365 is a software-as-a-service (SaaS) offering that provides Cloud PCs at a predictable per-user, per-month price. The hardware configuration for Cloud PCs may change over time to optimize performance and efficiency. Additionally, Windows 365 does not guarantee any specific hardware for its customers. If you want to choose your own cloud configuration based on performance data, you can use Azure Virtual Desktop instead (see Session host virtual machine sizing guidelines). Azure Virtual Desktop and Windows 365 have different use cases, and some organizations may benefit from using both solutions.

 

Where are the performance numbers for Windows 365?

 

 

One might wonder why Microsoft does not publish specific numbers for each Cloud PC configuration, given the various factors that affect performance. However, such numbers would not be very useful or reliable, since a Cloud PC is not a fixed hardware device but a dynamic SaaS offering that Microsoft continuously evolves. Rather than focusing on precise numbers that may vary depending on the situation, we provide relative performance charts for different Cloud PC sizes. The charts, based on internal testing, help you compare the relative differences between our most popular configurations. This way, you can choose the best Cloud PC configuration for your needs and trust Microsoft to deliver consistent and optimal performance.

 

Next steps

 

• Read Relative performance for different Cloud PC sizes.
  
• Review the Windows 365 roadmap and submit feature requests.
  
• Improve your Windows 365 knowledge with Windows in the Cloud.
  
• View the Planning guide for Windows 365.
  
• Explore Test Base for Microsoft 365 for your application validation needs.
  

 

Continue the conversation. Find best practices. Bookmark the Windows 365 Tech Community and follow us @MSWindowsITPro on Twitter. Looking for support? Visit Windows on Microsoft Q&A.

 

Continue reading...