How Azure Monitor's Implementation of Private Link Differs from Other Services

[Guest mtbmsft]

Azure Monitor implements Private Link DNS resolution differently from other Azure services. This article details the more common pattern for DNS resolution to support Private Endpoints, walks through an example from the client perspective using a Storage Account, then describes how Azure Monitor differs and how this may impact your Private Link architecture.

 

 

 

How most Azure services implement Private Link

 

 

For those Azure PaaS services where your resource has a globally-unique endpoint, such as Azure Storage, the platform adds a resourcename.privatelink.servicedomain… CNAME DNS record to your resource name when you enable a Private Endpoint on the resource.

 

 

 

From this point forward, if a client submits a DNS query for the resource FQDN, they will receive the resourcename.privatelink.servicedomain… CNAME in their response from Azure public DNS which their DNS server will attempt to resolve to an IP address.

 

 

 

If the DNS server resolving the request has a defined privatelink.servicedomain… zone (either via a conditional forwarder or a local forward lookup DNS zone) and the zone has an 'A' record for the requested name, the client will resolve the Private Endpoint IP. If the PrivateLink zone does not have a record for the requested name, the client will be unable to resolve an IP for the resource and be unable to connect.

 

 

 

If the DNS server responding to the client’s query has no privatelink.servicedomain… zone, then the publicly accessible privatelink.servicedomain… zone hosted by Microsoft DNS is used to resolve the query, which resolves to the public endpoint IP of the service.

 

 

 

Therefore, enabling Private Endpoints alone is **not enough to block access to PaaS services from the Internet**. You must also configure the service firewall to block requests from the public endpoint!

 

 

 

For the services implementing Private Link as described above, privatelink.servicedomain… DNS zone can be created and accessible to the DNS client, while still permitting a mix of both PaaS services with public endpoints and Private Endpoints to coexist in the environment.

 

 

 

Demo of typical Private Link DNS resolution

 

 

The following is a demonstration of the above concepts using the blob endpoint on a Storage Account named 'myprivateendpointdemo'. DNS queries are using the Azure provided DNS IP of 168.63.129.16 for simplicity, but the same could the accomplished with a custom DNS server with conditional forwarders or a server-level forwarder to 168.63.129.16. The VNET where the demo client resides has a Private DNS zone ‘privatelink.blob.core.windows.net’ linked to it.

 

 

 

After creating the Storage Account and making no other configuration, running a DNS query with 'nslookup' returns the records below--a CNAME record (alias), which resolves to the storage backend IP hosting this Storage Account

 

 

 

 

Next, I add a Private Endpoint to the blob service on the Storage Account:

 

 

 

 

Now, querying the same name when no A resource record has been created in the privatelink.blob.core.windows.net DNS zone (there is no address in the non-authoritative answer so the client connection will fail):

 

 

 

 

Altering my 'nslookup' configuration to show CNAME records reveals a 'myprivateendpointdemo.privatelink.blob.core.windows.net' in the response. Since my DNS server has a conditional forwarder defined for this zone, it will not attempt to resolve this CNAME via Azure public DNS:

 

 

 

 

Next, I create a DNS 'A' record in my 'privatelink.blob.core.windows.net' Private DNS Zone:

 

 

 

 

Running the same query a third time gives me my Private Endpoint IP address:

 

 

 

 

 

 

If I run the same query from a client off my VNET and corporate network, I see the *privatelink* CNAME in the response, but since my ISP’s DNS server has no ‘privatelink.blob.core.windows.net’ DNS zone overriding Microsoft’s public zone with that name, the public zone is used to resolve the query which returns the public endpoint IP:

 

 

 

 

A summary of the DNS resolution for blob endpoints in each scenario:

 

When Private Endpoints are not configured	myprivateendpointdemo.blob.core.windows.net -> blob.bzn10prdstr02a.store.core.windows.net -> 20.150.88.164
With Private Endpoints configured and empty private 'privatelink.blob.core.windows.net' zone in DNS resolution path	myprivateendpointdemo.blob.core.windows.net -> myprivateendpointdemo.privatelink.blob.core.windows.net -> _NO NAME FOUND_
With Private Endpoints configured, a private 'privatelink.blob.core.windows.net' DNS zone, and a 'myprivateendpointdemo' 'A' record	myprivateendpointdemo.blob.core.windows.net -> myprivateendpointdemo.privatelink.blob.core.windows.net -> 10.0.10.4
With Private Endpoints configured, but no ‘privatelink.blob.core.windows.net’ DNS zone overriding the public ‘privatelink.blob.core.windows.net’ zone (same for a public internet client)	myprivateendpointdemo.blob.core.windows.net -> myprivateendpointdemo.privatelink.blob.core.windows.net -> blob.bzn10prdstr02a.store.core.windows.net -> 20.150.88.164

 

 

 

How Azure Monitor implements Private Link DNS names

 

The Azure Monitor team needed to take a different approach to their implementation of Private Link because Azure Monitor endpoints use the same names across all customers. This means that they could not do what we saw above where a *privatelink* CNAME is added to the service DNS records only after the service is enabled for Private Endpoints. Instead, they need to always return a *privatelink* CNAME so that they can support customers using Private Link and public endpoints via the same name.

 

 

 

For example, the Log Analytics query DNS name is 'api.loganalytics.io'. Running 'nslookup' for this name returns the following results--note the 'api.privatelink.monitor.azure.com' name:

 

 

 

 

As described in the previous section, the DNS client will attempt to resolve the CNAME alias 'api.privatelink.monitor.azure.com'. If this zone is defined, either directly on the client's configured DNS server or on another server to which the original server forwards queries (such as Azure DNS), the query response will either be an 'A' record in this zone, or a 'name not found' if there isn't one. If the zone is not defined (and thereby overriding the public DNS resolution), the public DNS zone ‘api.privatelink.monitor.com’ will be used to resolve a public endpoint IP via Traffic Manager.

 

 

 

To summarize Azure Monitor DNS resolution in each scenario:

 

Default path (uses public ‘privatelink.monitor.azure.com’ zone)	api.loganalytics.io -> api.monitor.azure.com -> api.privatelink.monitor.azure.com -> (Traffic Manager) -> 40.67.122.3 (or other public IP)
With an empty 'privatelink.monitor.azure.com' zone in DNS resolution path	api.loganalytics.io -> api.monitor.azure.com -> api.privatelink.monitor.azure.com -> _NO NAME FOUND_
With a 'api.privatelink.monitor.azure.com' zone and an AMPLS 'A' record	api.loganalytics.io -> api.monitor.azure.com -> api.privatelink.monitor.azure.com -> 10.0.10.5

 

 

 

How this affects Private Link deployment designs

 

Azure Monitor's approach to Private Link works as intended but interferes with a common Private Link deployment architecture where customers pre-create Private DNS Zones for every privatelink.servicedomain… namespace and link all these DNS zones to their VNETs. Usually, these customers are referencing this article and list of Private DNS zones: Azure Private Endpoint DNS configuration | Microsoft Docs.

 

 

 

The problem that these customers experience is that the 'privatelink.monitor.azure.com' zone effectively "blackholes" their client's DNS queries to endpoints such as 'api.loganalytics.io' because the zone exists but contains no records! Unless and until these customers deploy an Azure Monitor Private Link Scope, they will be unable to query Log Analytics from clients within their network with DNS queries sent to this empty zone.

 

 

 

In this scenario, customers see errors such as the ones below:

 

Querying Log Analytics via PowerShell:

 

Invoke-AzOperationalInsightsQuery -WorkspaceId $workspaceResourceId -Query "AzureActivity | sample 1"

 

>> Invoke-AzOperationalInsightsQuery : An error occurred while sending the request.

 

 

 

Opening the Log Analytics log pane in the Portal:

 

Failed to retrieve schema. Please try to refresh the page.

We have experienced a connection issue while retrieving data. This is usually an indication that the network is down or a firewall or browser extension (such as an ad blocker) is mistakenly preventing access. Connection Error

 

To address this issue, there are a couple options:

 

• Deploy an Azure Monitor Private Link Scope and add all of your Log Analytics workspace to the scope, in which case the 'api.privatelink.monitor.azure.com' CNAME will resolve to a Private Link IP address.
  
• Delete or unlink the 'privatelink.monitor.azure.com' Private DNS zone from your VNET (until you are ready to deploy AMPLS)
  

 

 

 

Summary

 

To summarize, if you create and link a privatelink.servicedomain… DNS zone for a service type, and enable a Private Endpoint on your resource, your PrivateLink DNS zone must contain an 'A' record for your resource name. For Azure Monitor, there is no step to enable Private Endpoints for your Log Analytics workspace--if you create and link a privatelink.servicedomain… DNS zone for 'privatelink.monitor.azure.com', you'll be unable to access your Log Analytics workspace data unless you've enabled AMPLS.

 

Continue reading...