Protect production branches from having secrets through an Azure DevOps branch policy

[Guest Jamesdld23]

Introduction

 

 

The objective of this article is to share a strategy that has been implemented for a client that asked how we can protect its production branches from having secrets in code.

 

 

 

After few discussions we have highlighted the following requirements:

 

 

 

1. The solution should be simple to implement: no more than one script to maintain and we didn’t want to have some sort of regex file to teach the script what is a secret.
   
2. A comprehensive monitoring dashboard should be available to reference all our repositories, pointing out the ones that have secrets and ideally what kind of secrets have been founded (ex: SQL connection strings, Certificates with private keys, etc..).
   
3. The solution should alert the developer when he is trying to add a secret in the git Organisation.
   
4. Developers should have a solution to suppress false positives in order not to block production deliveries.
   
5. Any new repository should be automatically onboarded into this scanning and blocking strategy.
   
6. The solution should work on Azure DevOps but also on GitHub.
   
7. Ideally, the solution should be free.
   

 

 

 

Solution

 

 

At this point, you can certainly have serious doubts that a solution can meet all these requirements.

 

 

 

We did actually find one solution that answers all these requirements, the solution is a combination of the following Azure DevOps features.

 

• An Azure DevOps build validation policy to automate code scanning when a pull request is done on any production branch of the Azure DevOps project.
  
• The Azure DevOps extension: Microsoft Security DevOps to scan secret through the Credscan tool that is maintained by Microsoft.
  
• The Azure DevOps extension: SARIF SAST Scans Tab to display the analysis results under the “Scans” tab of each Azure DevOps pipelines.
  

 

 

 

A build validation policy queues a new build when a new PR is created or changes are pushed to an existing PR that targets the branch. The build policy evaluates the build results to determine whether the PR can be completed.

 

 

 

As illustrated in the following diagram our build validation policy will be unique across our repositories and its scan tasks will be located in one yaml file: “secret-scanning.yaml”.

 

 

 

 

 

 

 

 

 

 

Prerequisite

 

• Add the following Azure DevOps extension: Microsoft Security DevOps.
  
• Add the following Azure DevOps extension: SARIF SAST Scans Tab.
  
• Create a “commons” git repository.
  
• Disable the option “Protect access to repositories in YAML pipelines” to allow the Azure DevOps Build Service to pull remote repositories. You can find this option by navigating to the project settings page under “Pipeline” > “Settings”.
  

 

 

 

Script

 

 

Create a Yaml script file “secret-scanning.yaml” in the git repository “commons” and commit the following code.

 

 

 

This code is the Yaml description of an Azure DevOps pipeline, it is composed of the following actions:

 

• “trigger: none” disables Continuous Integration triggers. This pipeline will actually be triggered by a build validation policy we will create in the next chapter.
  
• Both variables SOURCE_REPOSITORY_URI & SOURCE_BRANCH are retrieving Azure DevOps system variables that belong to the repository that will initiate the pull request. We need them to retrieve the source code that will be scanned.
  
• “checkout, clean and persistCredentials” will avoid keeping the repository that contains our build validation script (the “commons” repository), our objective is not to scan this repository but the one that will initiate the pull request. We will also persist the Build Service credentials in order to clone the remote repository.
  
• The task named “enableGitCommands” will enable the next tasks to run Git commands.
  
• The task named “gitClone” will clone the repository that initiated the pull request (the remote repository).
  
• The latest task will perform the secrets scanning with the Azure DevOps extension Microsoft Security DevOps.
  

 

 

 

 

 

 

 

 

 

# This is a pipeline that can be used as an Azure DevOps build validation pipeline to prevent secrets from being in the code.

# Reference: Protect production branches from having secrets through an Azure DevOps branch policy

# Prerequisite:

# - Disable the option "Protect access to repositories in YAML pipelines" to allow the Azure DevOps Build Service to pull remote repositories. You can find this option by navigating to the project settings page under "Pipeline" > "Settings".

# - Grant version control permissions to the build service, cf Run Git commands in a script - Azure Pipelines

 

trigger: none

 

pool:

vmImage: 'ubuntu-latest'

 

variables:

GIT_USER_EMAIL: "you@example.com"

GIT_USER_NAME: "Your Name"

SOURCE_REPOSITORY_URI: $[variables['System.PullRequest.SourceRepositoryURI']] #Azure DevOps System variable, cf Predefined variables - Azure Pipelines

SOURCE_BRANCH: $[variables['System.PullRequest.SourceBranch']] #Azure DevOps System variable

 

steps:

- checkout: self

clean: true

persistCredentials: true

 

- script: |

git config --global user.email "$(GIT_USER_EMAIL)"

git config --global user.name "$(GIT_USER_NAME)"

displayName: 'Enable scripts to run Git commands, cf Run Git commands in a script - Azure Pipelines'

name: enableGitCommands

 

- script: |

#Variable

SOURCE_REPOSITORY_URI=$(SOURCE_REPOSITORY_URI)

REPOSITORY_NAME=$(echo ${SOURCE_REPOSITORY_URI//*\/})

BRANCH_NAME=$(echo ${SOURCE_BRANCH//refs\/heads\/})

SOURCE_REPOSITORY_URI_WITHOUT_USER=$(cut -d @ -f 2 <<< "$SOURCE_REPOSITORY_URI")

SOURCE_REPOSITORY_URI_SUFFIX=$(echo ${SOURCE_REPOSITORY_URI_WITHOUT_USER//https:\/\/})

SOURCE_REPOSITORY_URI="https://$SYSTEM_ACCESSTOKEN@$SOURCE_REPOSITORY_URI_SUFFIX"

 

#Action

echo "Cleaning the local folder"

ls -lart

rm -rf .git ; rm -rf .gitignore ; rm -rf .dockerignore ; rm -rf ./*

echo "git clone $SOURCE_REPOSITORY_URI ."

git clone $SOURCE_REPOSITORY_URI .

git checkout $BRANCH_NAME

displayName: 'Clone the source repository'

name: gitClone

env:

SYSTEM_ACCESSTOKEN: $(System.AccessToken)

 

- task: MicrosoftSecurityDevOps@1

displayName: 'Secrets scanning with Microsoft Security DevOps, cf Configurer l’extension Microsoft Security DevOps Azure DevOps'

inputs:

categories: 'secrets'

break: true

 

 

 

 

 

 

 

 

 

Pipeline

 

 

We need to publish the pipeline in order to use it in our build validation policy.

 

1. From Azure DevOps, navigate to “Pipelines” and click on the icon “New pipeline”.
   
2. Select “Azure Repos Git”.
   
3. Select the repository “commons”.
   
4. Select “Existing Azure Pipelines YAML file”.
   
5. Select the existing YAML file “secret-scanning.yaml”, click on the icon “Continue” and then select “Save”.
   
6. Click on the “More actions”, select “Rename/move” and under “Name” fill in “Secret scanning with Microsoft Security DevOps”, then click on the “Save” icon.
   

 

 

 

 

 

 

 

 

 

 

Build validation policy

 

 

We will now create a build validation policy to make sure a secret scan is performed whenever a pull request is performed on the production branch “main”.

 

1. Go to the project settings page and navigate under“Settings” > “Repositories” > “Policies”.
   
2. Select the + button next to “Build validation”.
   
3. Select “Protect the default branch of each repository” and click on “Create”.
   

 

 

 

 

 

 

4. Fill out the “Set build policy” form to use the pipeline we created in the previous step: “Secret scanning with Microsoft Security DevOps” and click on the “Save” icon.

 

 

 

 

 

 

 

 

 

 

Demonstration

 

 

The following screenshot illustrates a pull request that has been blocked by the build validation policy because secrets have been founded in the pull request source repository branch.

 

 

 

 

 

 

 

 

The following screenshot illustrates the “Scans” tab of the Azure DevOps pipeline, it prints the secret that have been founded revealing its path and its type, here it is an API Key.

 

 

 

 

 

 

 

 

The following screenshot is an Azure DevOps Workbook, it gives a summary of all secrets that have been founded on your organisation. This workbook is available if you go through this setup: “Quickstart: Connect your Azure DevOps repositories to Microsoft Defender for Cloud”.

 

 

 

Note: Microsoft Defender for DevOps is still in preview, changes should be expected while Microsoft improve the product.

 

 

 

 

 

 

 

 

 

 

Conclusion

 

 

Branch policies help teams protect their important branches of development. Policies enforce your team’s code quality and change management standards. This article was focused on analyzing secrets, but note that the Microsoft Security DevOps extension can be used to perform more general code analysis.

 

 

 

See You in the Cloud

 

Jamesdld

 

Continue reading...