New Custom and Managed Python images in Azure Data Explorer

[adieldar]

Azure Data Explorer (ADX) supports running Python code embedded in KQL query using the python() plugin . The plugin runtime is hosted in a sandbox, an isolated and secured environment hosted on ADX existing compute nodes. This sandbox contains the language engine as well as common mathematical and scientific packages. The plugin extends KQL native functionalities with a huge archive of OSS packages, enabling ADX users to run advanced algorithms, such as machine learning, statistical tests, time series analysis and many more, as part of the KQL query.



We are constantly working to improve the Python plugin capabilities, and today we introduce new managed Python images as well as the option to fully customize the Python image to include your required Python packages:

• Managed images are Python environment that are built and maintained by the Kusto team, containing specific Python engine and a set of packages. We are adding today Python 3.11.7 and Python 3.11.7 DL (containing torch & tensorflow), both images contain up to date Python engine 3.11.7 and packages (you can review the full contents of these images by clicking the respective links).
• Custom images let you create your specific Python images in case you need additional packages or different versions of the Python engine and/or packages. You can create a custom image from scratch by specifying the Python engine and supplying a requirements text file containing full list of packages, or by selecting a base existing image and supplying a minimal requirements file containing only the additional packages to install on top of the base image.

​

Defining a custom Python image from ADX portal​

For further information see Create a custom image - Azure Data Explorer | Microsoft Learn and Python plugin packages - Kusto | Microsoft Learn



You are welcome to try it and share your feedback!

Continue reading...