C
cynthiatreger
This article examines the data flow and performance benefits of Microsoft Azure's ExpressRoute and ExpressRoute FastPath features. It outlines the default asymmetric data routing and the enhancements achieved through FastPath. Key updates and constraints for FastPath, as well as IP address limits and monitoring metrics, are also discussed.
The data flow between On-Premises and Azure using ExpressRoute is asymmetric by design.
The traffic from On-Premises to Azure transits via the ExpressRoute Gateway, but the return traffic (Azure to On-Premises) bypasses the ExpressRoute Gateway and is forwarded directly to the MSEEs.
If multiple ExpressRoute Circuits are advertising identical On-Premises routes and are connected to the same ExpressRoute Gateway, Azure to On-Premises traffic is distributed (ECMP) across the different ExpressRoute Circuits available unless traffic engineering is configured to prioritise one path over the others.
Enabling ExpressRoute FastPath allows On-Premises to Azure traffic to bypass the ExpressRoute Gateway, providing improved data path performances.
FastPath on ExpressRoute Direct circuit connections now also honors:
This results in reduced latency and the capability to exceed the ExpressRoute Gateway's maximum throughput limit of 10 Gbps with the UltraPerformance or ErGW3AZ Gateway SKU, or the 40 Gbps limit of the upcoming ExpressRoute Scalable Gateway SKU.
Current constraints and limitations:
As per documentation, the following limits apply for the number of FastPath IPs. The limit is applied per ExpressRoute provider circuit (in the "Service Provider model") or per ExpressRoute Direct port (when using the "Direct model"):
Azure Monitor offers metrics for ExpressRoute Direct resources, which includes the ability to track the number of configured FastPath routes at the port level.
Key takeaway: FastPath is configured per connection, while the FastPath IP limit is per Service Provider circuit or per ExpressRoute Direct port (= an overall limit for all the FastPath-enabled connections terminating on that port).
Continue reading...
Data flow with ExpressRoute
The data flow between On-Premises and Azure using ExpressRoute is asymmetric by design.
The traffic from On-Premises to Azure transits via the ExpressRoute Gateway, but the return traffic (Azure to On-Premises) bypasses the ExpressRoute Gateway and is forwarded directly to the MSEEs.
If multiple ExpressRoute Circuits are advertising identical On-Premises routes and are connected to the same ExpressRoute Gateway, Azure to On-Premises traffic is distributed (ECMP) across the different ExpressRoute Circuits available unless traffic engineering is configured to prioritise one path over the others.
Data flow with ExpressRoute FastPath enabled
Enabling ExpressRoute FastPath allows On-Premises to Azure traffic to bypass the ExpressRoute Gateway, providing improved data path performances.
FastPath on ExpressRoute Direct circuit connections now also honors:
- UDRs on the Gateway subnet, configured to route traffic through an AzFW in the hub (or a 3P FW NVA) for inspection: traffic coming from On-Premises and matching a UDR on the Gateway subnet will be sent directly to the AzFW or 3P NVA, bypassing the ExpressRoute Gateway.
- VNet peering within the same region: traffic coming from On-Premises will be sent directly to VMs in spoke VNets, bypassing the ExpressRoute Gateway (global VNet peering is not supported).
This results in reduced latency and the capability to exceed the ExpressRoute Gateway's maximum throughput limit of 10 Gbps with the UltraPerformance or ErGW3AZ Gateway SKU, or the 40 Gbps limit of the upcoming ExpressRoute Scalable Gateway SKU.
Current constraints and limitations:
- IP address limit: see section below.
- Private Endpoint/Private Link: limited GA support, for 100 Gbps ExpressRoute Direct circuits only. Not supported with ExpressRoute partner circuits.
About the FastPath IP address limit
As per documentation, the following limits apply for the number of FastPath IPs. The limit is applied per ExpressRoute provider circuit (in the "Service Provider model") or per ExpressRoute Direct port (when using the "Direct model"):
| ExpressRoute SKU | Bandwidth | FastPath IP limit |
|---|---|---|
| ExpressRoute Direct | 100 Gbps | 200k |
| ExpressRoute Direct | 10 Gbps | 100k |
| ExpressRoute provider circuit | =< 10 Gbps | 25k |
- When FastPath is enabled, this limit represents the maximum number of Azure endpoints in the VNet environment that can bypass the ExpressRoute Gateway. Whenever a new IP address is added or removed, the FastPath IPs are automatically reprogrammed to reflect that change, keeping the FastPath IP list up to date.
- The limit applies to customer private IPs ; for example, a VM with 3 NICs requires 3 FastPath routes, each counting towards the limit (note: FastPath is configured per connection, not per circuit).
- If the FastPath IP limit is exceeded, traffic will not get dropped but will instead be routed according to the default behaviour via the ExpressRoute Gateway.
Azure Monitor offers metrics for ExpressRoute Direct resources, which includes the ability to track the number of configured FastPath routes at the port level.
Key takeaway: FastPath is configured per connection, while the FastPath IP limit is per Service Provider circuit or per ExpressRoute Direct port (= an overall limit for all the FastPath-enabled connections terminating on that port).
Continue reading...