Trying to run a custom base image for creating endpoint in Azure Machine Learning (unable to start container: "runsvdir": not found in $PATH: unknown)

Question

I'm trying to do inference deployment on Azure ML from a model that we have in a custom image. The model can perform inference locally within the custom image container with input and output written within the container. The ultimate goal is to have it working on Azure ML so that clients can pass data to it and then model perform the inference and output the results somehow inside the Azure ML workspace (or outside).

For this type of inference deployment, Azure ML documentation suggests using "compute clusters" and creating endpoints (add a model.pkl and an environment that could be a base image stored on azure registry repository).

Now a question is if we can utilize a custom base image in Azure ML? The documentation seems to suggest this, but all information ultimately points to the necessity of using FROM mcr.microsoft.com/azureml/… Is that the case?

For instance, all of the examples on these pages: https://github.com/Azure/AzureML-Containers https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-custom-container?view=azureml-api-2&viewFallbackFrom=azureml-api-1&tabs=cli

Specifically, there seem to be stuff happening behind the scene in the suggested mcr base image such as (azureml-app, azureml-logger, azureml-util) and lack of them in our custom image causes error (shown below) when running the endpoint. So if custom images are allowed, another question is, what do we need to add to a custom image to make it work for inference deployment?

Using an image other than the mcr base image results in such error:

*Instance status:
SystemSetup: Succeeded
UserContainerImagePull: Succeeded
ModelDownload: Succeeded
UserContainerStart: InProgress

Container events:
Kind: Pod, Name: Pulling, Type: Normal, Time: 2024-02-17T00:35:18.551821Z, Message: Start pulling container image
Kind: Pod, Name: Downloading, Type: Normal, Time: 2024-02-17T00:35:19.599858Z, Message: Start downloading models
Kind: Pod, Name: Pulled, Type: Normal, Time: 2024-02-17T00:35:47.018588Z, Message: Container image is pulled successfully
Kind: Pod, Name: Downloaded, Type: Normal, Time: 2024-02-17T00:35:47.018588Z, Message: Models are downloaded successfully
Kind: Pod, Name: Created, Type: Normal, Time: 2024-02-17T00:35:47.25383Z, Message: Created container inference-server
Kind: Pod, Name: Failed, Type: Warning, Time: 2024-02-17T00:35:47.55194Z, Message: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "runsvdir": executable file not found in $PATH: unknown*

At the end of the day, I want to know if it is a realistic usecase for using Azure ML to have all the model and artifacts within a custom image, and have it running on Azure ML via an endpoint. Do you have any thoughts?

Answer 1

Yes, you need a base image to create a custom image with your commands.

Next, whatever you are not having in the custom image can be installed using a conda YAML file or requirements text file.

To create an endpoint for your model in a custom image, you can follow the steps below:

1. Create an environment with a custom image.
2. Then, use the model from the registry and create a real-time endpoint.

Use a Dockerfile to create a custom image.

Refer to this Stack Overflow solution to create an environment using a Dockerfile. In this solution, a conda_dependencies.yaml and requirements.txt are provided, which is where you need to add your missing packages. Add your custom commands in the Dockerfile.

After creating the environment, deploy it to a real-time endpoint by following this Stack Overflow solution from step 7 onwards.

This should help you deploy your model to an endpoint with a custom image.