Run gpu docker images as agent in jenkins

Question

I have installed jenkins using the following documentaion to run with Dind : https://www.jenkins.io/doc/book/installing/docker/ on a machine with a GPU

Both the containers here are running with --gpus all parameter.

My requirement in Jenkins is that I need to handle gpu workloads in the pipelines and jobs and hence I wanted to try out a simple example just so as to check if the agent as docker image gets access to the gpu. Here's the pipeline :

pipeline {
    agent none
    options {
        skipStagesAfterUnstable()
    }
    stages {
        stage('Build') {
            agent {
                docker {
                    image 'nvidia/cuda:11.6.2-base-ubuntu20.04'
                }
            }
            steps {
                sh 'nvidia-smi'
            }
        }
    }
}

when I run this pipeline I receive this error in the pipeline logs :

[Pipeline] {
[Pipeline] sh
+ nvidia-smi
/var/jenkins_home/workspace/demo-bit@tmp/durable-95bc9af6/script.sh: 1: nvidia-smi: not found

it probably is because, we need to provide the gpu parameter for the agent, but how do I provide it within the pipeline?

Answer 1

Yes, you are right.

If you don't pass right ARG during docker execution you can't use gpu and nvidia-smi command will not found. There are several ways how to do it, below I show one of them:

- app.inside("--gpus all")

Full Jenkinsfile demo:

 def app

 pipeline {
      agent { label 'GPU' }
      stages {
        stage('RUN with GPU') { 
          steps { 
            script{
              app = docker.image('nvidia/cuda:11.6.2-base-ubuntu20.04')
              app.inside("--gpus all") {
                  sh "nvidia-smi"
              }
            }
          }
        }  
      }
    }

links:

• https://echorand.me/posts/jenkins-docker-workflow-inside/
• https://www.jenkins.io/doc/book/pipeline/docker/