T
theringe
In my previous blog: Simple Self-Troubleshooting Steps for Function App Not Seeing Triggers, I've mentioned a simple self-check method for Synctriggers. However, more and more enterprises are leaning towards using more secure network architectures. Due to the involvement of many different components, it is easy for minor configuration issues to cause the entire app to be unusable. Therefore, in this article, I will delve into more detailed content, specifically discussing NSG, UDR, DNS, and Storage Account.
TOC
NSG rules
UDR and Firewall
DNS
Storage Account
NSG rules
Please refer to this network architecture diagram first. Most enterprises want the Function App to run in a completely private network environment, hence this setup.
For outbound traffic, since the Function App is set up with VNET integration, NSG (i.e., Network Security Group) can be used to restrict which protocols and ports are allowed, and UDR (i.e., User Defined Route / Route Table) can be used to control which routes different source and destination traffic should take.
For inbound traffic, since the Function App is set up with a Private Endpoint, NSG can be used to restrict which protocols and ports are allowed to be accepted.
Firstly, in Azure's network architecture, an invocation passes through various resources in the following order: outbound NSG > outbound UDR > inbound NSG.
Let's take a look at the configuration of each sample component.
1) In the outbound NSG "NSG-SNET-FUNCTION-APP" rules are matched in order of ascending priority. Therefore, this synctriggers invocation (i.e., TCP port 443) will match the first rule "Rule_1"
2) In the outbound route table "RT-SNET-FUNCTION-APP" rules are matched based on the specificity of the target address, with more specific addresses taking precedence. In this example, this invocation will match the only one rule, directing all the traffic to firewall for additional processing.
3) In the inbound NSG "NSG-SNET-PE" the invocation will finally match "Rule_2" such that the traffic will be blocked.
The solution is to add a rule in 'NSG-SNET-PE' that allows TCP port 443, such as 'Rule_3'.
If you are unsure whether the relevant traffic complies with the expected NSG rules during transmission, you can temporarily record these behaviors by setting up 'NSG flow logs'
Step 1. Please separately go to your each target NSG on the Azure Portal.
Step 2. Select "NSG flow logs" and click "Create".
Step 3. Choose "Network security group" and then select "Select target Resource" as "Network security group".
Step 4. Choose the Storage Account where the logs will be stored as blobs. After confirming, click "Review + Create" and then "Create".
Step 5. After creation, wait 5 minutes for it to take effect.
Step 6. After creation, try to reproduce synctriggers connection. Please record the detailed operation time in UTC+0 format.
Step 7. After reproduction, please wait 10 minutes to ensure all logs are written to the corresponding Storage Account before exporting them.
Step 8. Go to the corresponding Storage Account, select "Containers", then "insights-logs-networksecuritygroupflowevent".
Step 9. Click on "Switch to Access key".
Step 10. This step is a bit troublesome, as you need to obtain log files from different folders based on different conditions (NSG name/time). You can refer to the picture; below is the folder structure where you might need to obtain files.
Step 11. Use a JSON parser to open this file to verify the requests, it's an example.
UDR and Firewall
The primary purpose of setting up UDR is to route outbound traffic from the VNET through a Firewall for execution and logging. In this example, all traffic from the Function App will be executed through the Firewall.
As different enterprises may use different brands of Firewalls, the log formats may vary. Generally, as shown in the diagram, it appears that all traffic from the Function App to its Private Endpoint is being blocked by the Firewall rules.
The solution is to modify the 'default-block' rule or add a new rule to override it, allowing all TCP port 443 traffic from 10.10.1.0/24 to 10.10.2.0/24
DNS
Most enterprises also want to maintain their own internal DNS Server. In this example, when the Function App sends a request to its Private Endpoint, the following process occurs:
1) my-function-app tries to resolve its Private Endpoint using the Custom DNS under VNET-FUNCTION-APP.
2) Since VNET-FUNCTION-APP is configured to route all traffic through the Firewall, the actual requester is the Firewall.
3) The Firewall uses the Azure DNS under VNET-FIREWALL to resolve its Private Endpoint.
4) Since the privatelink.azurewebsites.net Private DNS Zone does not have a Virtual network link with VNET-FIREWALL, the Azure DNS under VNET-FIREWALL cannot resolve my-function-app.azurewebsites.net using the recursive resolver function.
5) The request fails due to the resolution failure.
There are two solutions to this issue:
Solution 1: Modify the DNS Server in VNET-FIREWALL to 10.11.1.1, 10.11.1.2, which is the custom DNS Server.
Solution 2: Create a virtual network link for privatelink.azurewebsites.net to connect to the VNET-FIREWALL.
Assuming we use Solution 1, the process will continue as follows:
3) The Firewall uses the custom DNS Server under VNET-FIREWALL to resolve its Private Endpoint.
4) The Custom DNS Server can add an A record for my-function-app.azurewebsites.net pointing to 10.10.2.1, or implement a conditional forwarder as described in this article (Azure DNS Private Resolver - Azure Example Scenarios), forwarding unknown domains to Azure DNS for resolution using the recursive resolver function.
5) After the resolution is complete and the Private Endpoint IP is obtained, the actual request will be initiated.
Storage Account
The Synctriggers behavior saves a file named 'last' under the Storage blob to record the execution details using a hash.
Therefore, before each execution, the Function App communicates with the Storage Account to ensure that Synctriggers can continue to execute. Consequently, the network between the Function App and the Storage Account is a key consideration for Synctriggers. Please refer to the following architecture diagram.
In this diagram, we need to ensure:
1) The Private Endpoints for both blob and file services of the Storage are enabled.
2) The corresponding Private DNS Zones 'privatelink.blob.core.windows.net' and 'privatelink.file.core.windows.net' have established virtual network links with the relevant VNET.
3) The NSG needs to set up rules to allow or receive requests on TCP ports 443/445.
4) The Firewall also needs to set up rules to allow requests on TCP ports 443/445 from specific sources or destinations.
Continue reading...
TOC
NSG rules
UDR and Firewall
DNS
Storage Account
NSG rules
Please refer to this network architecture diagram first. Most enterprises want the Function App to run in a completely private network environment, hence this setup.
For outbound traffic, since the Function App is set up with VNET integration, NSG (i.e., Network Security Group) can be used to restrict which protocols and ports are allowed, and UDR (i.e., User Defined Route / Route Table) can be used to control which routes different source and destination traffic should take.
For inbound traffic, since the Function App is set up with a Private Endpoint, NSG can be used to restrict which protocols and ports are allowed to be accepted.
Firstly, in Azure's network architecture, an invocation passes through various resources in the following order: outbound NSG > outbound UDR > inbound NSG.
Let's take a look at the configuration of each sample component.
1) In the outbound NSG "NSG-SNET-FUNCTION-APP" rules are matched in order of ascending priority. Therefore, this synctriggers invocation (i.e., TCP port 443) will match the first rule "Rule_1"
| Name | Access | Direction | Protocol | Priority | Source Address Prefix | Source Port Range | Destination Address Prefix | Destination Port Range |
| Rule_1 | Allow | Outbound | * | 100 | VirtualNetwork | * | VirtualNetwork | * |
| Rule_2 | Deny | Outbound | * | 200 | * | * | * | * |
2) In the outbound route table "RT-SNET-FUNCTION-APP" rules are matched based on the specificity of the target address, with more specific addresses taking precedence. In this example, this invocation will match the only one rule, directing all the traffic to firewall for additional processing.
3) In the inbound NSG "NSG-SNET-PE" the invocation will finally match "Rule_2" such that the traffic will be blocked.
| Name | Access | Direction | Protocol | Priority | Source Address Prefix | Source Port Range | Destination Address Prefix | Destination Port Range |
| Rule_1 | Deny | Inbound | Udp | 100 | * | * | * | * |
| Rule_2 | Deny | Inbound | Tcp | 200 | * | * | * | * |
The solution is to add a rule in 'NSG-SNET-PE' that allows TCP port 443, such as 'Rule_3'.
| Name | Access | Direction | Protocol | Priority | Source Address Prefix | Source Port Range | Destination Address Prefix | Destination Port Range |
| Rule_3 | Allow | Inbound | TCP | 50 | VirtualNetwork | * | VirtualNetwork | 443 |
| Rule_1 | Deny | Inbound | Udp | 100 | * | * | * | * |
| Rule_2 | Deny | Inbound | Tcp | 200 | * | * | * | * |
If you are unsure whether the relevant traffic complies with the expected NSG rules during transmission, you can temporarily record these behaviors by setting up 'NSG flow logs'
Step 1. Please separately go to your each target NSG on the Azure Portal.
Step 2. Select "NSG flow logs" and click "Create".
Step 3. Choose "Network security group" and then select "Select target Resource" as "Network security group".
Step 4. Choose the Storage Account where the logs will be stored as blobs. After confirming, click "Review + Create" and then "Create".
Step 5. After creation, wait 5 minutes for it to take effect.
Step 6. After creation, try to reproduce synctriggers connection. Please record the detailed operation time in UTC+0 format.
Step 7. After reproduction, please wait 10 minutes to ensure all logs are written to the corresponding Storage Account before exporting them.
Step 8. Go to the corresponding Storage Account, select "Containers", then "insights-logs-networksecuritygroupflowevent".
Step 9. Click on "Switch to Access key".
Step 10. This step is a bit troublesome, as you need to obtain log files from different folders based on different conditions (NSG name/time). You can refer to the picture; below is the folder structure where you might need to obtain files.
Step 11. Use a JSON parser to open this file to verify the requests, it's an example.
UDR and Firewall
The primary purpose of setting up UDR is to route outbound traffic from the VNET through a Firewall for execution and logging. In this example, all traffic from the Function App will be executed through the Firewall.
As different enterprises may use different brands of Firewalls, the log formats may vary. Generally, as shown in the diagram, it appears that all traffic from the Function App to its Private Endpoint is being blocked by the Firewall rules.
The solution is to modify the 'default-block' rule or add a new rule to override it, allowing all TCP port 443 traffic from 10.10.1.0/24 to 10.10.2.0/24
DNS
Most enterprises also want to maintain their own internal DNS Server. In this example, when the Function App sends a request to its Private Endpoint, the following process occurs:
1) my-function-app tries to resolve its Private Endpoint using the Custom DNS under VNET-FUNCTION-APP.
2) Since VNET-FUNCTION-APP is configured to route all traffic through the Firewall, the actual requester is the Firewall.
3) The Firewall uses the Azure DNS under VNET-FIREWALL to resolve its Private Endpoint.
4) Since the privatelink.azurewebsites.net Private DNS Zone does not have a Virtual network link with VNET-FIREWALL, the Azure DNS under VNET-FIREWALL cannot resolve my-function-app.azurewebsites.net using the recursive resolver function.
5) The request fails due to the resolution failure.
There are two solutions to this issue:
Solution 1: Modify the DNS Server in VNET-FIREWALL to 10.11.1.1, 10.11.1.2, which is the custom DNS Server.
Solution 2: Create a virtual network link for privatelink.azurewebsites.net to connect to the VNET-FIREWALL.
Assuming we use Solution 1, the process will continue as follows:
3) The Firewall uses the custom DNS Server under VNET-FIREWALL to resolve its Private Endpoint.
4) The Custom DNS Server can add an A record for my-function-app.azurewebsites.net pointing to 10.10.2.1, or implement a conditional forwarder as described in this article (Azure DNS Private Resolver - Azure Example Scenarios), forwarding unknown domains to Azure DNS for resolution using the recursive resolver function.
5) After the resolution is complete and the Private Endpoint IP is obtained, the actual request will be initiated.
Storage Account
The Synctriggers behavior saves a file named 'last' under the Storage blob to record the execution details using a hash.
Therefore, before each execution, the Function App communicates with the Storage Account to ensure that Synctriggers can continue to execute. Consequently, the network between the Function App and the Storage Account is a key consideration for Synctriggers. Please refer to the following architecture diagram.
In this diagram, we need to ensure:
1) The Private Endpoints for both blob and file services of the Storage are enabled.
2) The corresponding Private DNS Zones 'privatelink.blob.core.windows.net' and 'privatelink.file.core.windows.net' have established virtual network links with the relevant VNET.
3) The NSG needs to set up rules to allow or receive requests on TCP ports 443/445.
4) The Firewall also needs to set up rules to allow requests on TCP ports 443/445 from specific sources or destinations.
Continue reading...