Configure remote write to send data from ARO to Azure managed Prometheus using AAD authentication.

[UmarMohamedUsman]

This blog describes how to configure remote-write to send data from Prometheus running in your Azure Red Hat OpenShift (ARO) cluster to Azure managed Prometheus using Azure Active Directory authentication. Then query and view the data via Azure managed Prometheus and Grafana.



Currently ARO doesn't support Managed Identity, hence we are using Azure Active Directory Service Principal approach. Managed Identity support for ARO is coming soon. One of the major drawbacks with Service Principal approach is that the client secret will have an expiry date.



Credit

Huge thanks to Vishwanath Narasimhan & Rakshith Padmanabha from Microsoft and Paul Czarkowski from Red Hat for providing guidance to accomplish this.



Cluster configurations

This article applies to the following cluster configurations:

• Azure Red Hat OpenShift cluster

Prerequisites

• Azure Red Hat OpenShift cluster up and running. Prometheus comes pre-installed and configured for Azure Red Hat OpenShift 4.x clusters.

High level steps

• Create an Azure Active Directory application
• Create an Azure Monitor Workspace (i.e. Azure managed Prometheus, data from Azure managed Prometheus is stored in Azure Monitor Workspace)
• Create an Azure managed Grafana and connect the Azure Monitor workspace to the Grafana
• Assign Monitoring Metrics Publisher role on the data collection rule to the AAD App (i.e. to the Service Principal)
• Create Secret object in ARO
• Update ARO's cluster monitoring config
• View the metric data in Prometheus and Grafana

Create Azure Active Directory application

Follow the procedure at Register an application with Azure AD and create a service principal to register an application for Prometheus remote-write and create a service principal. Also get the Tenant ID.





# Get Tenant ID
TENANT_ID=$(az account get-access-token --query tenant --output tsv)

# Create a Service Principal and get the Client Secret
SERVICE_PRINCIPAL_CLIENT_SECRET="$(az ad sp create-for-rbac --name umarm-$AROCLUSTER --query 'password' -otsv)"

# Create a Service Principal Client ID
SERVICE_PRINCIPAL_CLIENT_ID="$(az ad sp list --display-name umarm-$AROCLUSTER --query '[0].appId' -otsv)"






Create an Azure Monitor Workspace

Follow the procedure at Create an Azure Monitor Workspace to create Azure Monitor Workspace. This is the way of setting up Azure managed Prometheus and data from Azure managed Prometheus is stored in Azure Monitor Workspace.



Create an Azure Managed Grafana and connect the Azure Monitor workspace to the Grafana

Follow the procedure at Create an Azure Managed Grafana to create Azure Managed Grafana instance.

Next follow the procedure at Link a Grafana workspace to connect the Azure Monitor workspace to the Azure managed Grafana instance.



Assign Monitoring Metrics Publisher role the AAD App (i.e. to the Service Principal)

Follow the procedure at Assign Monitoring Metrics Publisher role to the Service Principal to assign Monitoring Metrics Publisher role on the data collection rule associated with your Azure Monitor workspace to the Service Principal.



Create Secret object in ARO

Create a secret object in ARO using the Service Principal client ID and client secret values from above.



cat << EOF | oc apply -f -
apiVersion: v1
kind: Secret
metadata:
name: oauth2-credentials
stringData:
id: "${SERVICE_PRINCIPAL_CLIENT_ID}"
secret: "${SERVICE_PRINCIPAL_CLIENT_SECRET}"
EOF





Update ARO's cluster monitoring config

Update ARO's cluster monitoring config to remote write on the Azure managed Prometheus.

To edit cluster-monitoring-config file run the following command.





oc edit -n openshift-monitoring cm cluster-monitoring-config





Edit this config file in editor like Vim with following configuration.

Make sure to replace {{INGESTION-URL}} value below with Metrics ingestion endpoint from the Overview page for the Azure Monitor workspace.





data:
config.yaml: |
prometheusK8s:
remoteWrite:
- url: "{{INGESTION-URL}}"
oauth2:
clientId:
secret:
name: oauth2-credentials
key: id
clientSecret:
name: oauth2-credentials
key: secret
tokenUrl: "Sign in to your account"
scopes:
- "https://monitor.azure.com/.default"






View the metric data in Prometheus and Grafana

You can either view your clusters CPU/Memory usage or assuming you are running an application in ARO already, you can start viewing your application resources metrics in Prometheus and Grafana.



Querying metrics in Prometheus

Go to Prometheus explorer in Azure Monitor Workspace and run the following PromQL queries.

• The following query retrieves the current memory usage in bytes for the specified pod's container.

container_memory_usage_bytes{pod="<pod_name>", namespace="<namespace>", container="<container_name>"}

For example:​

container_memory_usage_bytes{pod="mypod-azure-private-file", namespace="default", container="mypod-azure-private-file-container"}

​

• This query retrieves the total CPU usage in seconds for the specified pod's container.

container_cpu_usage_seconds_total{pod="<pod_name>", namespace="<namespace>", container="<container_name>"}



For example:

container_cpu_usage_seconds_total{pod="mypod-azure-private-file", namespace="default", container="mypod-azure-private-file-container"}

​

Viewing metrics in Grafana

To view your clusters consolidated CPU, Memory, etc. usage go to Dashboards in Grafana and select Azure Managed Prometheus. Then under Compute Resources section, select Kubernetes / Compute Resources / Cluster.

​

   

To view one of your Pod's memory or CPU usage, go to Dashboards in Grafana and select Azure Managed Prometheus. Then under Compute Resources section, select Kubernetes / Compute Resources / Nodes (Pods).

​

Select the necessary nodes from node dropdown and enter your Pod name under the Filters. Now you can view CPU and Memory Usage using Grafana dashboard for your Pod.

​

Continue reading...