Enable and Operate Open Service Mesh in AKS hybrid cluster

[Guest EkeleAsonye]

A number of organizations adopting microservices architectures have looked to service meshes as one of the tools that help these organizations in enforcing enterprise-wide policies and rules for traffic, security, reliability, and monitoring. These are generally categorized into Security, Reliability and Observability. While a networking solution other than service mesh can provide some of these features, it won’t do it without disrupting the application code or existing infrastructure. Service Mesh provides an independent infrastructure layer that can be managed by a separate team and is least intrusive. It deploys a sidecar container to pods by design, which makes it very powerful for monitoring your services within a kubernetes cluster.

 

 

 

Open Service Mesh (OSM) is a lightweight, extensible service mesh that is utilized to secure communication of services running in a Kubernetes environment. To achieve interoperability between service meshes, a common interface standard SMI (Service Mesh Interface) has been defined by CNCF. OSM is Microsoft's implementation of the Service Mesh Interface in an actual service mesh. It is easy to install, maintain and operate. It is also effortless to troubleshoot and less complicated to configure with SMI.

 

 

 

The Open Service Mesh extension is a managed service mesh for Arc-enabled Kubernetes clusters that is lightweight and extensible. It can be configured with Service Mesh Interface APIs, works by injecting envoy proxy as a sidecar to each application instance, and brings a new Azure Portal experience for onboarding.

 

Some of the use cases you can solve with OSM are:

 

• Service to Service communication over mutual TLS authentication
  
• Access control for service-to-service communication
  
• Traffic management for canary deployments
  
• Observability for services
  
• Traffic management capabilities like auto retries, rate limiting, circuit breaking
  

• Multi-cluster communication over mTLS.
  

 

With the OSM extension being enabled through Azure Arc, OSM can be applied to your on-premises AKS hybrid environment to manage communication and secure your cloud-native workload.

 

In this article, the OSM functionalities were applied to the communication between the bookbuyer and bookstore sample applications. First, traffic access between the bookbuyer and bookstore services was disabled, but only got enabled by applying an SMI traffic policy. Additionally, the traffic-splitting capability was also implemented by creating a second bookstore service and then weighted traffic from the bookbuyer to bookstore-v1 and bookstore-v2 respectively.

 

 

 

Setting up OSM

 

The latest OSM version (v1.2.0) used for this article requires a kubernetes cluster running v1.22.9 or higher and a Kubernetes command-line tool. It also requires a workstation that is capable of executing bash commands/scripts and has the OSM code repo available locally.

 

For this work, the setup was done in an Arc-enabled Aks cluster with WSL-2 (windows subsystem for linux) enabled. To setup your own Aks hybrid cluster, see Use PowerShell to set up Kubernetes on Azure Stack HCI and Windows Server clusters - AKS-HCI | Microsoft Learn. This should have Az CLI running, but you can check using az version.

 

To get things going, bring up your cluster.

 

 

 

PS C:\Users\ekele> Get-AksHciCluster

 

Status : {ProvisioningState, Details}

 

ProvisioningState : Deployed

 

KubernetesVersion : v1.23.8

 

PackageVersion : v1.23.8

 

NodePools : linuxpool

 

WindowsNodeCount : 0

 

LinuxNodeCount : 3

 

ControlPlaneNodeCount : 3

 

ControlPlaneVmSize : Standard_A2_v2

 

AutoScalerEnabled : False

 

AutoScalerProfile :

 

LoadBalancer : {VMSize, Count, Sku}

 

Name : <ClusterName>

 

 

 

The next step is to enable OSM kubernetes extension on the AKS hybrid cluster after connecting the cluster to Azure Arc.

 

Note: In order to run some bash commands for validation purposes, the rest of the cmdlets used in this article were run in Git bash terminal on Visual Studio Code with Az cli installed. Also, ensure to download the OSM cli to be able to run osm commands. To download the latest OSM cli, see Releases · openservicemesh/osm (github.com)

 

 

 

ea@SA18n30-3 MINGW64 ~ (master)

 

$ az k8s-extension create --cluster-name <ClusterName> --resource-group <ResourceGroup> --cluster-type connectedClusters --extension-type Microsoft.openservicemesh --scope cluster --name osm

 

{

 

"aksAssignedIdentity": null,

 

"autoUpgradeMinorVersion": true,

 

"configurationProtectedSettings": {},

 

"configurationSettings": {},

 

"customLocationSettings": null,

 

"errorInfo": null,

 

"extensionType": "microsoft.openservicemesh",

 

"id": "/subscriptions/5807bea4-cd17-4b81-b4d8-d4e94caab82d/resourceGroups/<ResourceGroup>/providers/Microsoft.Kubernetes/connectedClusters/<ClusterName>/providers/Microsoft.KubernetesConfiguration/extensions/osm",

 

"identity": {

 

"principalId": "0c8ed1d6-358f-4f6a-8342-61d10e1a9571",

 

"tenantId": null,

 

"type": "SystemAssigned"

 

},

 

"name": "osm",

 

"packageUri": null,

 

"provisioningState": "Succeeded",

 

"releaseTrain": "Stable",

 

"resourceGroup": "<ResourceGroup>",

 

"scope": {

 

"cluster": {

 

"releaseNamespace": "arc-osm-system"

 

},

 

"type": "Microsoft.KubernetesConfiguration/extensions",

 

"version": "1.1.1-1"

 

}

 

 

 

Verify that the OSM k8s extension was installed correctly by running

 

ea@SA18n30-3 MINGW64 ~ (master)

 

$ az k8s-extension show --cluster-type connectedClusters --cluster-name <ClusterName>

 

--resource-group <ResourceGroup> --name osm

 

 

 

Get the credentials of your cluster to be sure you can run kubectl using

 

ea@SA18n30-3 MINGW64 ~ (master)

 

$ kubectl config current-context

 

<ClusterName>-admin@<ClusterName>

 

 

 

You can further verify that the OSM components are running on your cluster by checking the deployment, pods, and services in arc-osm-system namespace.

 

ea@SA18n30-3 MINGW64 ~ (master)

 

$ kubectl get deployment -n arc-osm-system

 

NAME READY UP-TO-DATE AVAILABLE AGE

 

osm-bootstrap 1/1 1 1 86m

 

osm-controller 1/1 1 1 86m

 

osm-injector 1/1 1 1 86m

 

osm-metrics-agent 1/1 1 1 86m

 

 

 

ea@SA18n30-3 MINGW64 ~ (master)

 

$ kubectl get pods -n arc-osm-system

 

NAME READY STATUS RESTARTS AGE

 

osm-bootstrap-58b8986b87-wfp6g 1/1 Running 0 88m

 

osm-controller-768df46759-n29ps 1/1 Running 0 88m

 

osm-injector-6ddd574779-wl52s 1/1 Running 0 88m

 

osm-metrics-agent-78c846d57b-v9vtd 4/4 Running 1 (87m ago) 88m

 

 

 

ea@SA18n30-3 MINGW64 ~ (master)

 

$ kubectl get services -n arc-osm-system

 

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

 

osm-bootstrap ClusterIP 10.99.80.35 <none> 9443/TCP,9095/TCP 88m

 

osm-controller ClusterIP 10.111.193.40 <none> 15128/TCP,9092/TCP,9091/TCP 88m

 

osm-injector ClusterIP 10.100.99.169 <none> 9090/TCP 88m

 

osm-validator ClusterIP 10.108.48.217 <none> 9093/TCP 88m

 

 

 

OSM controller configuration

 

OSM deploys a MeshConfig resource osm-mesh-config as a part of its control plane in arc-osm-system namespace. The MeshConfig provides the mesh owner or operator the ability to update some of the mesh configurations based on need. The default values of the MeshConfig can be viewed using the following command.

 

$ kubectl describe meshconfig osm-mesh-config -n arc-osm-system

 

Name: osm-mesh-config

 

Namespace: arc-osm-system

 

Labels: <none>

 

Annotations: <none>

 

API Version: config.openservicemesh.io/v1alpha2

 

Kind: MeshConfig

 

Metadata:

 

Creation Timestamp: 2022-10-17T22:56:58Z

 

Generation: 1

 

Managed Fields:

 

API Version: config.openservicemesh.io/v1alpha2

 

Fields Type: FieldsV1

 

fieldsV1:

 

f:metadata:

 

f:annotations:

 

{ …

 

Traffic:

 

Enable Egress: false

 

Enable Permissive Traffic Policy Mode: true

 

… }

 

It can be observed that the permissive traffic policy mode of the output is set at true. With permissive traffic policy mode enabled in your OSM mesh:

 

• The SMI traffic policy enforcement is bypassed.
  
• OSM automatically discovers services that are a part of the service mesh.
  
• OSM programs traffic policy rules on each Envoy proxy sidecar to be able to communicate with these services.
  

 

It is important to note that values in the MeshConfig osm-mesh-config are persisted across upgrades. Changes to osm-mesh-config can be made using the kubectl patch command. In a subsequent step for this sample deployment, the permissive traffic policy mode was changed to false through this command.

 

 

 

Deploy Bookstore Applications

 

After validating the checks, the sample applications manifest (Bookbuyer, Bookstore, Bookwarehouse) is deployed that will generate the services on which OSM will be applied to.

 

Kubectl create -f https://raw.githubusercontent.com/openservicemesh/osm/release-v0.8/docs/example/manifests/apps/bookbuyer.yaml

 

Kubectl create -f https://raw.githubusercontent.com/openservicemesh/osm/release-v0.8/docs/example/manifests/apps/bookstore.yaml

 

Kubectl create -f https://raw.githubusercontent.com/openservicemesh/osm/release-v0.8/docs/example/manifests/apps/bookwarehouse.yaml

 

 

 

Validate Application by port-forwarding the Kubernetes service

 

The components of the bookstore namespace can be checked to see the pods and services running using ‘kubectl get all -n bookstore’. Afterwards, the logs of the bookbuyer pod could be traced to ensure it is communicating with the bookstore service.

 

To query that bookbuyer is reaching bookstore service, run the following commands to trail the bookbuyer logs.

 

POD="$(kubectl get pods -n bookbuyer  --show-labels --selector app=bookbuyer --no-headers | grep -v 'Terminating' | awk '{print $1}' | head -n1)"

 

Kubectl logs “${POD}” -n bookbuyer -c bookbuyer --tail=100 -f

 

{ … Fetching http://bookstore.bookstore:14001/books-bought

 

Request Headers: map[Client-App:[bookbuyer] User-Agent:[Go-http-client/1.1]]

 

Identity: bookstore-v1

 

Booksbought: 5258

 

Server: n/a

 

Date: Tue, 18 Oct 2022 04:42:48 GMT

 

Status: 200 OK

 

MAESTRO! THIS TEST SUCCEEDED!

 

 

 

Fetching http://bookstore.bookstore:14001/buy-a-book/new

 

Request Headers: map[]

 

Identity: bookstore-v1

 

Booksbought: 5259

 

Server: n/a

 

Date: Tue, 18 Oct 2022 04:42:49 GMT

 

Status: 200 OK

 

MAESTRO! THIS TEST SUCCEEDED!

 

 

 

… }

 

The status should show that everything is okay and that it is able to fetch http bookstore-v1 data.

 

 

 

Onboard Bookstore Apps namespaces for OSM

 

In order to start using OSM capabilities, the application namespaces need to be onboarded to the service mesh so that OSM can manage these namespaces. Run the command

 

$ osm namespace add bookstore bookbuyer bookwarehouse

 

Namespace [bookstore] successfully added to mesh [osm]

 

Namespace [bookbuyer] successfully added to mesh [osm]

 

Namespace [bookwarehouse] successfully added to mesh [osm]

 

 

 

Enable proxy sidecar injection for Bookstore Apps

 

The sidecar containers can be enabled using the ‘kubectl rollout restart’ command. You can check the pod details before and after to know that the sidecar proxies were correctly deployed.

 

$ kubectl get pods -n bookbuyer

 

NAME READY STATUS RESTARTS AGE

 

bookbuyer-56d6bcc7f8-gvzn8 1/1 Running 0 86m

 

 

 

ea@SA18n30-3 MINGW64 ~ (master)

 

$ kubectl rollout restart deployment bookbuyer -n bookbuyer

 

deployment.apps/bookbuyer restarted

 

 

 

ea@SA18n30-3 MINGW64 ~ (master)

 

$ kubectl rollout restart deployment bookstore -n bookstore

 

deployment.apps/bookstore restarted

 

 

 

ea@SA18n30-3 MINGW64 ~ (master)

 

$ kubectl rollout restart deployment bookwarehouse -n bookwarehouse

 

deployment.apps/bookwarehouse restarted

 

 

 

By checking the pod details of the bookbuyer after restart, the ‘Ready’ state shows that there are 2 out of 2 pods running.

 

$ kubectl get pods -n bookbuyer

 

NAME READY STATUS RESTARTS AGE

 

bookbuyer-6ccd656dbf-7c28x 2/2 Running 0 115s

 

To get further information, the ‘kubectl describe pod’ command will show an envoy running alongside the bookbuyer container.

 

POD="$(kubectl get pods -n bookbuyer  --show-labels --selector app=bookbuyer --no-headers | grep -v 'Terminating' | awk '{print $1}' | head -n1)"

 

Kubectl describe pod $POD -n bookbuyer

 

 

 

Normal Started 4m29s kubelet Started container bookbuyer

 

Normal Pulling 4m29s kubelet Pulling image "mcr.microsoft.com/oss/envoyproxy/envoy:v1.19.3"

 

Normal Pulled 4m26s kubelet Successfully pulled image "mcr.microsoft.com/oss/envoyproxy/envoy:v1.19.3" in 3.413382681s

 

Normal Created 4m26s kubelet Created container envoy

 

Normal Started 4m26s kubelet Started container envoy

 

 

 

Verify Bookstore App functioning with sidecar injection

 

Kubectl logs "${POD}" -n bookbuyer -c bookbuyer --tail=100 -f

 

…

 

Fetching http://bookstore.bookstore:14001/books-bought

 

Request Headers: map[Client-App:[bookbuyer] User-Agent:[Go-http-client/1.1]]

 

Identity: bookstore-v1

 

Booksbought: 415

 

Server: envoy

 

Date: Tue, 18 Oct 2022 04:51:14 GMT

 

Status: 200 OK

 

MAESTRO! THIS TEST SUCCEEDED!

 

 

 

Fetching http://bookstore.bookstore:14001/buy-a-book/new

 

Request Headers: map[]

 

Identity: bookstore-v1

 

Booksbought: 416

 

Server: envoy

 

Date: Tue, 18 Oct 2022 04:51:15 GMT

 

Status: 200 OK

 

MAESTRO! THIS TEST SUCCEEDED!

 

…

 

Disable OSM Permissive Mode

 

If the permissive traffic policy mode is disabled, communication between the bookbuyer and bookstore will cease. This can be done using the following command

 

$ kubectl patch meshconfig osm-mesh-config -n arc-osm-system -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":false}}}' --type=merge

 

meshconfig.config.openservicemesh.io/osm-mesh-config patched

 

 

 

Verify that the bookbuyer can no longer reach the bookstore

 

Kubectl logs "${POD}" -n bookbuyer -c bookbuyer --tail=100 -f

 

{ …

 

Fetching http://bookstore.bookstore:14001/books-bought

 

Request Headers: map[Client-App:[bookbuyer] User-Agent:[Go-http-client/1.1]]

 

Error fetching http://bookstore.bookstore:14001/books-bought: Get "http://bookstore.bookstore:14001/books-bought": dial tcp 10.99.49.182:14001: connect: connection

 

refused

 

ERROR: response code for "http://bookstore.bookstore:14001/books-bought" is 0; expected 200

 

Fetching http://bookstore.bookstore:14001/buy-a-book/new

 

Request Headers: map[]

 

Error fetching http://bookstore.bookstore:14001/buy-a-book/new: Get "http://bookstore.bookstore:14001/buy-a-book/new": dial tcp 10.99.49.182:14001: connect: connection refused

 

ERROR: response code for "http://bookstore.bookstore:14001/buy-a-book/new" is 0;

 

expected 200

 

… }

 

 

 

Deploy SMI Traffic Access Policies

 

To restore traffic access between the bookbuyer and bookstore, run the SMI traffic policy that will enable communication between bookbuyer and bookstore.

 

$ Kubectl apply -f 01-deploy-smi-access-policy.yaml

 

traffictarget.access.smi-spec.io/bookbuyer-access-bookstore created

 

httproutegroup.specs.smi-spec.io/bookstore-service-routes created

 

traffictarget.access.smi-spec.io/bookstore-access-bookwarehouse created

 

httproutegroup.specs.smi-spec.io/bookwarehouse-service-routes created

 

 

 

If the bookbuyer logs are checked again, traffic between bookbuyer and bookstore are restored.

 

$ Kubectl logs "${POD}" -n bookbuyer -c bookbuyer --tail=100 -f

 

{ … Fetching http://bookstore.bookstore:14001/books-bought

 

Request Headers: map[Client-App:[bookbuyer] User-Agent:[Go-http-client/1.1]]

 

Identity: bookstore-v1

 

Booksbought: 552

 

Server: envoy

 

Date: Tue, 18 Oct 2022 04:56:07 GMT

 

Status: 200 OK

 

MAESTRO! THIS TEST SUCCEEDED!

 

 

 

Fetching http://bookstore.bookstore:14001/buy-a-book/new

 

Request Headers: map[]

 

Identity: bookstore-v1

 

Booksbought: 553

 

Server: envoy

 

Date: Tue, 18 Oct 2022 04:56:07 GMT

 

Status: 200 OK

 

MAESTRO! THIS TEST SUCCEEDED!

 

…}

 

 

 

Deploy bookstore v2 service

 

To display the traffic split functionality of OSM, an additional bookstore v2 service was deployed to allow the bookbuyer reach either bookstore-v1 or bookstore-v2. The deployed bookstore-v2 service would have an SMI traffic policy created as well.

 

$ Kubectl apply -f 02-deploy-bookstore-v2.yaml

 

service/bookstore-v2 created

 

serviceaccount/bookstore-v2 created

 

deployment.apps/bookstore-v2 created

 

traffictarget.access.smi-spec.io/bookbuyer-access-bookstore-v2 created

 

 

 

Implement a SMI Traffic Split

 

The SMI traffic split is applied to enable the bookbuyer to communicate with the bookstore and the bookstore-v2 service at weighted measures. Here, the weight was shared at 25% for bookstore and 75% for bookstore-v2 service.

 

To see the content of the trafficsplit manifest run

 

$ cat 03-deploy-smi-trafficsplit.yaml

 

apiVersion: split.smi-spec.io/v1alpha2

 

kind: TrafficSplit

 

metadata:

 

name: bookstore-split

 

namespace: bookstore

 

spec:

 

service: bookstore.bookstore

 

backends:

 

- service: bookstore

 

weight: 25

 

- service: bookstore-v2

 

weight: 75

 

 

 

Apply the manifest by running

 

$ Kubectl apply -f 03-deploy-smi-trafficsplit.yaml

 

trafficsplit.split.smi-spec.io/bookstore-split created

 

Trail the identity of the bookbuyer logs and see that traffic is split with 75% going to bookstore-v2 and 25% going to bookstore-v1. i.e.

 

$ Kubectl logs "${POD}" -n bookbuyer -c bookbuyer --tail=100 -f | grep 'Identity:'

 

Identity: bookstore-v2

 

Identity: bookstore-v2

 

Identity: bookstore-v2

 

Identity: bookstore-v2

 

Identity: bookstore-v1

 

Identity: bookstore-v1

 

Identity: bookstore-v2

 

Identity: bookstore-v2

 

Identity: bookstore-v1

 

Identity: bookstore-v2

 

Identity: bookstore-v2

 

Identity: bookstore-v2

 

 

 

To Uninstall OSM from your AKS hybrid cluster

 

az k8s-extension delete --cluster-type connectedClusters --cluster-name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --name osm

 

 

 

Summary

 

In this article, we see the capabilities of OSM been applied to a sample bookstore application and its usefulness in securing your Kubernetes cluster in a variety of ways like Ingress traffic management via policies to permit/deny external access into the mesh, Sidecar proxy injection using the envoy network proxy to ensure services participate in the mesh and obey policies that have been defined, Service-to-service access control via Layer 7 (http) access policies to allow authorized clients and disallow unauthorized clients, and Traffic splitting for canary and blue-green deployments.

 

In a subsequent article, the observability benefits would be demonstrated to provide metrics, tracing, and logs. This would also enable visualizing application traffic flows, errors, latency and other metrics in tools like Grafana and Jaeger.

 

 

 

Further reading

 

Azure Arc-enabled Open Service Mesh - Azure Arc | Microsoft Learn

 

Continue reading...