How to build an environment when your Azure ML workspace is behind a VNet

[Guest hristinajilova]

Build environment for your experiment when your Azure ML workspace is behind a Vnet

 

 

 

 

We see a common request from our customers to create a ML pipeline using an Azure ML workspace behind a VNet.

 

Creating and building environment in a workspace behind a VNet has some specifics and causes questions. The reason is that you cannot build the Docker Image directly on ACR, when the ACR is behind a virtual network

 

 

 

In this article you will find the steps that will help you to build your custom environment when you Azure ML workspace is behind a VNet.

 

 

Example files: AzureML/CreateEnvBehindVnet at main · HristinaJilova/AzureML (github.com)

 

1. Prepare the Docker Image Context

 

• Create a folder for you docker context called 'DockerContext' (or choose another name)
  
• In 'DockerContext' folder create a 'requirements.txt' file that contains your packages:
  

 

 

 

 

 

 

 

openpyxl==3.0.10

pyod==1.0.4

kaleido==0.2.1

plotly==5.10.0

pandas==1.1.5

numpy==1.21.6

seaborn==0.11.2

pathlib2==2.3.7.post1

matplotlib==3.2.1

argparse

 

 

 

 

 

 

 

Note: It’s a good practice to fix the versions for your packages. You can find the version with command: pip freeze | grep 'the package name'

 

• Create a Dockerfile:
  

 

 

 

 

 

 

 

#FROM mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04

FROM python:3.8

 

# python installs

COPY requirements.txt .

RUN pip install -r requirements.txt

 

# set command

CMD ["bash"]

 

 

 

 

 

 

 

 

 

Option 1:

 

Configure an AzureML compute cluster to build environments

 

 

 

1. Create an Azure Machine Learning compute cluster (Only a CPU SKU is supported). This cluster will be used to build the docker images when ACR is behind a VNet. For more information, see Create a compute cluster.
   
2. Use the az ml workspace update command to set a build compute:
   

 

az ml workspace update --name myworkspace --resource-group myresourcegroup --image-build-compute mycomputecluster

 

For reference: Enable Azure Container Registry (ACR)

 

 

 

1. Register the environment:
   

• Create an ‘environment.yml’ as in the example file.
  
• Open the terminal in the folder where ‘environment.yml’ is located and run the command:
  

 

az ml environment create -f environment.yml

 

 

 

1. Point your job/component environment to the registered environment:
   

 

environment: azureml:mynewenv@latest

 

 

 

1. Run your job and a ‘prepare_image’ job will start to build your image.
   

 

 

 

Option 2:

 

Build Image locally and push it to ACR with commands:

 

 

 

1. Open terminal in your 'DockerContext' folder
   
2. Login to your ACR:
   

 

docker login your-acr-name.azurecr.io

 

Note: You will find the username (it's your-acr-name) and the password in the Access keys section of your ACR.

 

3. Build your docker file locally:

 

 

docker image build -t your-acr-name.azurecr.io/repo/mynewenv:v1 .

 

 

 

1. Push your image to ACR:
   

 

docker push your-acr-name.azurecr.io/repo/mynewenv:v1

 

 

 

1. Point your job/component environment to the new image:
   

 

image: your-acr-name.azurecr.io/repo/ mynewenv:v1

 

 

 

Useful links:

 

Enterprise security and governance - Azure Machine Learning | Microsoft Learn

 

Secure workspace resources using virtual networks (VNets) - Azure Machine Learning | Microsoft Learn

 

Secure an Azure Machine Learning workspace with virtual networks - Azure Machine Learning | Microsoft Learn

 

About Azure Machine Learning environments - Azure Machine Learning | Microsoft Learn

 

azureml-examples/sdk/python/assets/environment at main · Azure/azureml-examples (github.com)

 

azureml-examples/cli/assets/environment at main Azure/azureml-examples (github.com)

 

chboudry/aml-secure-terraform (github.com)

 

Continue reading...