Investigate Azure Spring Apps Networking Issue with the new Connect feature

[Guest Hanli_Ren]

In A Simple Network Connection Test Tool for Azure Spring Cloud we demonstrated a way to build a very simple App for DNS resolve and TCP ping tests in Azure Spring Apps.

 

 

 

Azure Spring Apps published a new Connect Feature that allows you access the shell environment inside your application instances to do advanced troubleshooting.

 

 

 

This blog aimed to describe how to directly access the shell environment inside your application instances to investigate common networking issues.

 

 

 

Step 1: Connect to an app instance for troubleshooting

 

Step 2: Investigate Azure Spring Apps Networking Issue in the Console

 

Step 1: Connect to an app instance for troubleshooting

 

 

Reference Doc: Connect to an app instance for troubleshooting

 

 

1. Before connecting to an app instance, you must be granted the role Azure Spring Apps Connect Role to your Azure Spring Apps service.

Detailed guide can be found in Assign Azure Spring Apps Connect Role

 

• Using Azure Portal
  Azure Spring Apps -> Access control (IAM) -> Add role assignment
  

 

 

 

 

• Use Azure CLI
  

 

 

 

az role assignment create --role 'Azure Spring Apps Connect Role' --scope '/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.AppPlatform/Spring/xxx' --assignee xxx@xxx.com

 

 

 

 

 

 

 

2. Connect to app instance

Important Note:

If your Azure Spring Apps is created with VNET injection, by default, the shell environment can only be accessed from inside the VNET.

If you want to connect the console from outside the VNET, please enable "Log streaming on public network".

 

 

 

 

• Using Azure Portal to connect to app instance
  Azure Spring Apps -> Apps, select the App you want to connect to
  

 

 

 

 

Go to

Console

Select the

App instance

and

Start up shell

, then click

Connect

.

 

 

 

 

• Using Azure CLI to connect to app instance
  

 

 

 

az spring app connect \

--subscription <subscription-id> \

--service <your-service-instance> \

--resource-group <your-resource-group> \

--name <app-name> \

--deployment <deployment-name> \

--instance <instance_name> \

--shell-cmd /bin/bash

 

 

 

 

 

 

Note: --deployment, --instance and --shell-cmd are optional arguments.

 

 

 

 

Step 2: Investigate Azure Spring Apps Networking Issue in the Console

 

 

After we Connect to the shell environment inside your application instances, we can use preinstalled commands to investigate the following common networking issues:

 

 

 

DNS resolve failure

 

Connection Timeout Issue

 

SSL handshake issue

 

Monitor established connections status

 

 

 

DNS resolve failure

 

 

It is a common issue that our customer DNS server is not correctly configured to forward DNS requests to upstream public DNS server. In this case we can use nslookup command to run name resolver test.

 

For example:

Test nslookup with default DNS server

 

Note: If you are using VNET injection for your Azure Spring Apps, the platform will pick your DNS server configured in your VNET as the default DNS server.

 

 

 

nslookup mcr.microsoft.com

 

 

 

 

 

 

Test nslookup with Azure DNS server

 

 

 

nslookup mcr.microsoft.com 168.63.129.16

 

 

 

 

 

 

 

 

Test nslookup with Custom Specified DNS server

 

 

 

nslookup mcr.microsoft.com 10.0.0.123

 

 

 

 

 

 

 

 

Connection Timeout Issue

 

 

It is a common issue that an App running inside Azure Spring Apps encounter connection issue to a remote target.

We can use "ncat" command to test connection in TCP/UDP level.

 

For example:

Test TCP connection

 

 

 

nc -vz mcr.microsoft.com 443

 

 

 

 

 

 

 

 

Test UDP connection

 

 

 

nc -vz -u ntp.ubuntu.com 123

 

 

 

 

 

 

 

 

SSL handshake issue

 

 

It is also common to encounter SSL handshake issue.

We can use "curl" command to run SSL test.

 

For example:

Curl Failed because of SSL failures

 

 

 

curl https://self-signed.badssl.com/

 

 

 

 

 

 

 

 

Test connection without security check

 

 

 

curl --insecure https://self-signed.badssl.com/

 

 

 

 

 

 

 

 

Use “curl -iv” to do further investigation

 

 

 

curl -iv https://self-signed.badssl.com/

 

 

 

 

 

 

 

 

Monitor established connections status

 

 

We can also use netstat command to monitor established connections status.

 

 

 

netstat -anp

 

 

 

 

 

 

Continue reading...