Guest AndreDewes_MSFT Posted January 27, 2023 Posted January 27, 2023 Introduction The default option to have a scalable self-hosted pool in Azure Devops is use Azure virtual machine scale sets. However, if you have an ecosystem that is running mostly in containers and Kubernetes, you might also think about using containers instead and leverage KEDA scalable triggers to spin up more agents when the demand increases and lower them when the demand decreases. Compared to Azure VM scale sets, the Kubernetes instances are much faster to create (given you have available resources in the cluster and don't need to create new nodes). The purpose for this article is to quickly demonstrate a working solution for this scenario. It is not the purpose to show how to work with KEDA because we have so many other articles in TechCommunity already for this topic. The source code for this exercise is here (and a Portuguese version of this article). Pre-requisites Azure Kubernetes cluster Azure Container Registry Integrated Azure Container Registry with AKS KEDA installed in the AKS cluster Azure DevOps organization Azure DevOps configured to have at least 5 parallel jobs for self-hosted private projects Docker Kubectl Azure CLI Setting up Azure Devops Creating an agent pool After you have your organization created in Azure Devops, let's create a new agent pool. For that, go to Organization settings -> Agent pools -> Add pool. In the pool type, select "Self-hosted" and give it a name like "keda-pool". Then, check the box "Grant access permissions to all pipelines" and click "Create". After the agent pool is created, we need to get its id. One way to get the agent pool id is to access this URL: https://dev.azure.com/**your-organization-name**/_apis/distributedtask/pools?poolname=keda-pool. The response contains a JSON with its id (in my case, its "18"): Generating a PAT (Personal Access Token) in Azure Devops It is necessary to have an Azure Devops token to be able to register our agents in the agent pool. Follow ">these steps to create a PAT. Don't forget to save your token because we will need it later. Building a Docker image for the agent and publishing it to the container registry Creating a Dockerfile Create a file named "Dockerfile" with the following contents: FROM ubuntu:20.04 RUN DEBIAN_FRONTEND=noninteractive apt-get update RUN DEBIAN_FRONTEND=noninteractive apt-get upgrade -y RUN DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends \ apt-transport-https \ apt-utils \ ca-certificates \ curl \ git \ iputils-ping \ jq \ lsb-release \ software-properties-common RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash # Can be 'linux-x64', 'linux-arm64', 'linux-arm', 'rhel.6-x64'. ENV TARGETARCH=linux-x64 WORKDIR /azp COPY ./start.sh . RUN chmod +x start.sh ENTRYPOINT [ "./start.sh" ] In the same folder where you created your Dockerfile, add another file named "start.sh" with the following contents: #!/bin/bash set -e if [ -z "$AZP_URL" ]; then echo 1>&2 "error: missing AZP_URL environment variable" exit 1 fi if [ -z "$AZP_TOKEN_FILE" ]; then if [ -z "$AZP_TOKEN" ]; then echo 1>&2 "error: missing AZP_TOKEN environment variable" exit 1 fi AZP_TOKEN_FILE=/azp/.token echo -n $AZP_TOKEN > "$AZP_TOKEN_FILE" fi unset AZP_TOKEN if [ -n "$AZP_WORK" ]; then mkdir -p "$AZP_WORK" fi export AGENT_ALLOW_RUNASROOT="1" cleanup() { if [ -e config.sh ]; then print_header "Cleanup. Removing Azure Pipelines agent..." # If the agent has some running jobs, the configuration removal process will fail. # So, give it some time to finish the job. while true; do ./config.sh remove --unattended --auth PAT --token $(cat "$AZP_TOKEN_FILE") && break echo "Retrying in 30 seconds..." sleep 30 done fi } print_header() { lightcyan='\033[1;36m' nocolor='\033[0m' echo -e "${lightcyan}$1${nocolor}" } # Let the agent ignore the token env variables export VSO_AGENT_IGNORE=AZP_TOKEN,AZP_TOKEN_FILE print_header "1. Determining matching Azure Pipelines agent..." AZP_AGENT_PACKAGES=$(curl -LsS \ -u user:$(cat "$AZP_TOKEN_FILE") \ -H 'Accept:application/json;' \ "$AZP_URL/_apis/distributedtask/packages/agent?platform=$TARGETARCH&top=1") AZP_AGENT_PACKAGE_LATEST_URL=$(echo "$AZP_AGENT_PACKAGES" | jq -r '.value[0].downloadUrl') if [ -z "$AZP_AGENT_PACKAGE_LATEST_URL" -o "$AZP_AGENT_PACKAGE_LATEST_URL" == "null" ]; then echo 1>&2 "error: could not determine a matching Azure Pipelines agent" echo 1>&2 "check that account '$AZP_URL' is correct and the token is valid for that account" exit 1 fi print_header "2. Downloading and extracting Azure Pipelines agent..." curl -LsS $AZP_AGENT_PACKAGE_LATEST_URL | tar -xz & wait $! source ./env.sh print_header "3. Configuring Azure Pipelines agent..." ./config.sh --unattended \ --agent "${AZP_AGENT_NAME:-$(hostname)}" \ --url "$AZP_URL" \ --auth PAT \ --token $(cat "$AZP_TOKEN_FILE") \ --pool "${AZP_POOL:-Default}" \ --work "${AZP_WORK:-_work}" \ --replace \ --acceptTeeEula & wait $! print_header "4. Running Azure Pipelines agent..." if ! grep -q "template" <<< "$AZP_AGENT_NAME"; then echo "Cleanup Traps Enabled" trap 'cleanup; exit 0' EXIT trap 'cleanup; exit 130' INT trap 'cleanup; exit 143' TERM fi chmod +x ./run-docker.sh # To be aware of TERM and INT signals call run.sh # Running it with the --once flag at the end will shut down the agent after the build is executed ./run-docker.sh "$@" --once & wait $! Just be aware that you need to set the EOL (end of line) character for these files to Unix/Linux format LF. You can do that by using a text/code editor like Visual Studio Code or Notepad++ for instance. Building the image and publishing to ACR Open a terminal window in the folder where you have Dockerfile and start.sh and execute the following commands: # Build the image docker build -t azure-devops-agent:latest . # Login to ACR docker login -u <user> -p <password> <your ACR address> # Tag the image docker tag azure-devops-agent:latest <your ACR address>/azure-devops-agent:latest # Send image to ACR docker push <your ACR address>/azure-devops-agent:latest Setting up Kubernetes Creating Kubernetes YML manifests for the Azure DevOps Agent Take the PAT generated previously and convert it to base64. Add a file named "deployment.yml" with the following content: apiVersion: v1 kind: Secret metadata: name: azdevops data: AZP_TOKEN: '<your base64 PAT>' #replace --- apiVersion: apps/v1 kind: Deployment metadata: name: azdevops-deployment labels: app: azdevops-agent spec: replicas: 1 selector: matchLabels: app: azdevops-agent template: metadata: labels: app: azdevops-agent spec: containers: - name: azdevops-agent image: <your ACR address>/azure-devops-agent:latest #replace env: - name: AZP_URL value: "Azure DevOps Services | Microsoft Azure<your organization name>" #replace - name: AZP_POOL value: "keda-pool" - name: AZP_TOKEN valueFrom: secretKeyRef: name: azdevops key: AZP_TOKEN Deploying to Kubernetes Let's install the AKS CLI, login to the cluster e run the deploy commands: # Installing AKS CLI az aks install-cli # Logging in az aks get-credentials --resource-group <myResourceGroup> --name <myAKSCluster> # Create a namespace kubectl create namespace azdevops # Deploying to AKS kubectl apply -f deployment.yml -n azdevops # Check if the pod is up and running kubectl get pods -n azdevops # You should see an output similar to: NAME READY STATUS RESTARTS AGE azdevops-deployment-86b9c58cbd-nm45l 1/1 Running 0 13s In the Azure DevOps portal, check if your new agent is correctly showing up in the agent pool: Creating a pipeline Add a new project to your Azure DevOps organization: doc Then, create a new pipeline: doc In the end of the pipeline creation wizard, change the pool name do "keda-pool" and save/run it: Go back to the Agent Pool page and check if your new agent is running the pipeline: Adding KEDA to dynamically provision new agent instances Great, so far you have a single agent. That means if you try to build multiple pipelines at the same time with it, they will queue up and run sequentially. Let's leverage KEDA to automatically provision a new instance of the agent for each pipeline run... with the maximum of 5 instances. Create a file named "scale-object.yml" with the following content: apiVersion: v1 kind: Secret metadata: name: pipeline-auth data: personalAccessToken: '<your base64 PAT>' #replace --- apiVersion: keda.sh/v1alpha1 kind: TriggerAuthentication metadata: name: pipeline-trigger-auth spec: secretTargetRef: - parameter: personalAccessToken name: pipeline-auth key: personalAccessToken --- apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: azure-pipelines-scaledobject spec: scaleTargetRef: name: azdevops-deployment minReplicaCount: 1 maxReplicaCount: 5 #Maximum number of parallel instances triggers: - type: azure-pipelines metadata: poolID: "<Agent Pool Id>" #Replace with your agent pool ID organizationURLFromEnv: "AZP_URL" authenticationRef: name: pipeline-trigger-auth And then deploy the manifest to Kubernetes: kubectl apply -f scale-object.yml -n azdevops # Verify if it is created kubectl get scaledobject -n azdevops # You should see something similar to NAME SCALETARGETKIND SCALETARGETNAME MIN MAX TRIGGERS AUTHENTICATION READY ACTIVE FALLBACK AGE azure-pipelines-scaledobject apps/v1.Deployment azdevops-deployment 1 5 azure-pipelines pipeline-trigger-auth True False False 73s Now, run your new pipeline many times (without waiting for the previous run to complete) and check how the number of agents increase in the agent pool: In this case, I ran the pipeline 4 times and I got 4 agents in Kubernetes to handle the deployments. If these agents do not get a new job within 5 minutes (the default KEDA scaling down timeout), the pods will be killed in Kubernetes until we are down to 1 single agent again. An important consideration Since the DevOps agent is already a container running in Kubernetes, by default you can't use it to build another container in the pipeline context (unless you use something like Kaniko). My recommendation for this scenario is that you leverage the ACR build task and build your container directly in your Azure Container Registry instead of trying to build it in the pipeline. Conclusion Using KEDA is a fast and cost efficient way to scale your Azure DevOps pipelines if you are comfortable working with containers and Kubernetes. Just be aware this tutorial covered a basic scenario. KEDA provides other options to configure the scaling behavior as you can see below: triggers: - type: azure-pipelines metadata: # Optional: Name of the pool in Azure DevOps poolName: "{agentPoolName}" # Optional: Learn more in 'How to determine your pool ID' poolID: "{agentPoolId}" # Optional: Azure DevOps organization URL, can use TriggerAuthentication as well organizationURLFromEnv: "AZP_URL" # Optional: Azure DevOps Personal Access Token, can use TriggerAuthentication as well personalAccessTokenFromEnv: "AZP_TOKEN" # Optional: Target queue length targetPipelinesQueueLength: "1" # Default 1 activationTargetPipelinesQueueLength: "5" # Default 0 # Optional: Parent template to read demands from parent: "{parent ADO agent name}" # Optional: Demands string to read demands from ScaledObject demands: "{demands}" authenticationRef: name: pipeline-trigger-auth References Example for ScaledObject Azure Pipelines scaler Docker Agents in Azure Devops Continue reading... Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.