azure - How to define a custom environment in ML Studio with customized Python-version? - Stack Overflow

admin管理员组

文章数量:1023187

Unfortunately I'm stuck on a development in ML Studio for which I can't seem to find a solution. The task is to create an environment from a notebook code via Python SDK V2. The environment should be defined via a requirements.txt and additionally with a customized Python version (3.11). The parent image of the new docker file is 'mcr.microsoft/azureml/curated/minimal-py311-inference:14'.

Numerous attempts have unfortunately failed. There is the possibility via:

from azureml.core import Environment
from azureml.core.conda_dependencies import CondaDependencies
conda = CondaDependencies()
# add conda packages
conda.add_conda_package('torchvision')
# add pip packages
conda.add_pip_package('pyyaml')
conda.set_python_version('3.11')
# create environment
env = Environment('test-1') 
env.python.conda_dependencies = conda
env.docker.base_image = 'mcr.microsoft/azureml/curated/minimal-py311-inference:14'

The created environment can then be registered in the workspace.

However, it is not possible to capture the current environment via pip freeze > requirement.txt and integrate it into the conda yaml file for the new environement.

from azureml.core import Environment
pip freeze > requirements.txt
env = Environment.from_pip_requirements('test-2', './requirements.txt')
env.docker.base_image = 'mcr.microsoft/azureml/curated/minimal-py311-inference:14'

Although the parent image (minimal-py311-inferece) comes with Python 3.11, the conda-yaml-file of the test-2 contains the Python version=3.8.21 as depedency.

Where does this Python version come from and how can I customize it?

Thank you very much and have a nice evening

Unfortunately I'm stuck on a development in ML Studio for which I can't seem to find a solution. The task is to create an environment from a notebook code via Python SDK V2. The environment should be defined via a requirements.txt and additionally with a customized Python version (3.11). The parent image of the new docker file is 'mcr.microsoft/azureml/curated/minimal-py311-inference:14'.

Numerous attempts have unfortunately failed. There is the possibility via:

from azureml.core import Environment
from azureml.core.conda_dependencies import CondaDependencies
conda = CondaDependencies()
# add conda packages
conda.add_conda_package('torchvision')
# add pip packages
conda.add_pip_package('pyyaml')
conda.set_python_version('3.11')
# create environment
env = Environment('test-1') 
env.python.conda_dependencies = conda
env.docker.base_image = 'mcr.microsoft/azureml/curated/minimal-py311-inference:14'

The created environment can then be registered in the workspace.

However, it is not possible to capture the current environment via pip freeze > requirement.txt and integrate it into the conda yaml file for the new environement.

from azureml.core import Environment
pip freeze > requirements.txt
env = Environment.from_pip_requirements('test-2', './requirements.txt')
env.docker.base_image = 'mcr.microsoft/azureml/curated/minimal-py311-inference:14'

Although the parent image (minimal-py311-inferece) comes with Python 3.11, the conda-yaml-file of the test-2 contains the Python version=3.8.21 as depedency.

Where does this Python version come from and how can I customize it?

Thank you very much and have a nice evening

• azure
• environment
• ml-studio

Share Improve this question Follow asked Nov 18, 2024 at 19:37 meisterkaiomeisterkaio 522 bronze badges 3

• you want to use sdk v1 or v2? – JayashankarGS Commented Nov 19, 2024 at 7:04
• Hi, I want to use SDK V2. – meisterkaio Commented Nov 19, 2024 at 8:04
• Check the below solution. @meisterkaio – JayashankarGS Commented Nov 25, 2024 at 4:26

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

In sdk v2 you can use create environment using build context providing Dockerfile and requirements.txt file.

Create Dockerfile with below content.

FROM mcr.microsoft/azureml/curated/minimal-py311-inference:14

# python installs
COPY requirements.txt .
RUN pip install -r requirements.txt && rm requirements.txt

# set command
CMD ["bash"]

Next, create a requirements.txt correctly from a proper environment.

And run below code.

# import required libraries
from azure.ai.ml import MLClient
from azure.ai.ml.entities import Environment, BuildContext
from azure.identity import DefaultAzureCredential

try:
    credential = DefaultAzureCredential()
except Exception as ex:
    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work
    credential = InteractiveBrowserCredential()

ml_client = MLClient(credential, subscription_id, resource_group, workspace)

env_docker_context = Environment(
    build=BuildContext(path="docker-contexts/python-and-pip"),
    name="docker-context-example",
    description="Environment created from a Docker context.",
)
ml_client.environments.create_or_update(env_docker_context)

Here, i kept my Dockerfile and requirements.txt file in docker-contexts/python-and-pip folder.

Output:

Refer more about this here.

Unfortunately I'm stuck on a development in ML Studio for which I can't seem to find a solution. The task is to create an environment from a notebook code via Python SDK V2. The environment should be defined via a requirements.txt and additionally with a customized Python version (3.11). The parent image of the new docker file is 'mcr.microsoft/azureml/curated/minimal-py311-inference:14'.

Numerous attempts have unfortunately failed. There is the possibility via:

from azureml.core import Environment
from azureml.core.conda_dependencies import CondaDependencies
conda = CondaDependencies()
# add conda packages
conda.add_conda_package('torchvision')
# add pip packages
conda.add_pip_package('pyyaml')
conda.set_python_version('3.11')
# create environment
env = Environment('test-1') 
env.python.conda_dependencies = conda
env.docker.base_image = 'mcr.microsoft/azureml/curated/minimal-py311-inference:14'

The created environment can then be registered in the workspace.

However, it is not possible to capture the current environment via pip freeze > requirement.txt and integrate it into the conda yaml file for the new environement.

from azureml.core import Environment
pip freeze > requirements.txt
env = Environment.from_pip_requirements('test-2', './requirements.txt')
env.docker.base_image = 'mcr.microsoft/azureml/curated/minimal-py311-inference:14'

Although the parent image (minimal-py311-inferece) comes with Python 3.11, the conda-yaml-file of the test-2 contains the Python version=3.8.21 as depedency.

Where does this Python version come from and how can I customize it?

Thank you very much and have a nice evening

Unfortunately I'm stuck on a development in ML Studio for which I can't seem to find a solution. The task is to create an environment from a notebook code via Python SDK V2. The environment should be defined via a requirements.txt and additionally with a customized Python version (3.11). The parent image of the new docker file is 'mcr.microsoft/azureml/curated/minimal-py311-inference:14'.

Numerous attempts have unfortunately failed. There is the possibility via:

from azureml.core import Environment
from azureml.core.conda_dependencies import CondaDependencies
conda = CondaDependencies()
# add conda packages
conda.add_conda_package('torchvision')
# add pip packages
conda.add_pip_package('pyyaml')
conda.set_python_version('3.11')
# create environment
env = Environment('test-1') 
env.python.conda_dependencies = conda
env.docker.base_image = 'mcr.microsoft/azureml/curated/minimal-py311-inference:14'

The created environment can then be registered in the workspace.

However, it is not possible to capture the current environment via pip freeze > requirement.txt and integrate it into the conda yaml file for the new environement.

from azureml.core import Environment
pip freeze > requirements.txt
env = Environment.from_pip_requirements('test-2', './requirements.txt')
env.docker.base_image = 'mcr.microsoft/azureml/curated/minimal-py311-inference:14'

Although the parent image (minimal-py311-inferece) comes with Python 3.11, the conda-yaml-file of the test-2 contains the Python version=3.8.21 as depedency.

Where does this Python version come from and how can I customize it?

Thank you very much and have a nice evening

• azure
• environment
• ml-studio

Share Improve this question Follow asked Nov 18, 2024 at 19:37 meisterkaiomeisterkaio 522 bronze badges 3

• you want to use sdk v1 or v2? – JayashankarGS Commented Nov 19, 2024 at 7:04
• Hi, I want to use SDK V2. – meisterkaio Commented Nov 19, 2024 at 8:04
• Check the below solution. @meisterkaio – JayashankarGS Commented Nov 25, 2024 at 4:26

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

In sdk v2 you can use create environment using build context providing Dockerfile and requirements.txt file.

Create Dockerfile with below content.

FROM mcr.microsoft/azureml/curated/minimal-py311-inference:14

# python installs
COPY requirements.txt .
RUN pip install -r requirements.txt && rm requirements.txt

# set command
CMD ["bash"]

Next, create a requirements.txt correctly from a proper environment.

And run below code.

# import required libraries
from azure.ai.ml import MLClient
from azure.ai.ml.entities import Environment, BuildContext
from azure.identity import DefaultAzureCredential

try:
    credential = DefaultAzureCredential()
except Exception as ex:
    # Fall back to InteractiveBrowserCredential in case DefaultAzureCredential not work
    credential = InteractiveBrowserCredential()

ml_client = MLClient(credential, subscription_id, resource_group, workspace)

env_docker_context = Environment(
    build=BuildContext(path="docker-contexts/python-and-pip"),
    name="docker-context-example",
    description="Environment created from a Docker context.",
)
ml_client.environments.create_or_update(env_docker_context)

Here, i kept my Dockerfile and requirements.txt file in docker-contexts/python-and-pip folder.

Output:

Refer more about this here.

本文标签： azureHow to define a custom environment in ML Studio with customized PythonversionStack Overflow