Certificate issue for launching kubernetes pods for Jenkins (hosted outside of that kubernetes cluster)

Question

I have been trying to configure jenkins kubernetes cloud agents on my existing jenkins setup (which is hosted outside k8s cluster). My jenkins is hosted in Google Cloud Platform in a Windows VM. It is exposed to internet and we have a ssl certification added to it.

We are trying to add cloud agents using Jenkins Kubernetes Plugin. The connection to kubernetes works (i have verified this by Test Connection. Also the pods container is added to cluster when my job tries to start). I am adding the following configuration in my pod template - the pod container starts in my kubernetes engine.

Problem - the job does not run and keeps creating new pods and deleting old pods. Need some help in the right direction. I have scoured the web to find if anyone had a similar issue or setup. Appears everyone is hosting jenkins in k8s along with cloud agents.

I think the problem is because our jenkins is outside our kubernetes cluster.

I am using the default base image for now just to see and check that the job runs. Job just has a echo command in Build shell step. Images I have tried

• jnlp-slave-with-java-build-tools
• jenkins/inbound-agent
• jenkins/agent

Authentication to my kubernetes cluster is using a service account json which has for all all access. I am using Freestyle job

Error in google stackdriver logs -

SEVERE: Failed to connect to https://bflow.br.iq/tcpSlaveAgentListener/: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target`

`java.io.IOException: Failed to connect to https://bflow.br.iq/tcpSlaveAgentListener/: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:214)
    at hudson.remoting.Engine.innerRun(Engine.java:689)
    at hudson.remoting.Engine.run(Engine.java:514)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:324)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:267)
    at sun.security.ssl.TransportContext.fatal(TransportContext.java:262)
    at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
    at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
    at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
    at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
    at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
    at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
    at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
    at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:167)
    at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:211)
    ... 2 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:456)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:323)
    at sun.security.validator.Validator.validate(Validator.java:271)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:315)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:223)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129)
    at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638)
    ... 16 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:451)
    ... 22 more

Answer 1

Setup Details:

• Jenkins master Running on VM as a docker container.
• Jenkins running on a private network.
• Kubernetes plugin installed.
• Jenkins namespace, serviceAccount, Role, and RoleBinding created in Kubernetes cluster.
• Under Managed Nodes and Clouds > Configure Clouds > Kubernetes Configured with the required details.
  • Kubernetes URL
  • Kubernetes server certificate key
  • Kubernetes Namespace
  • Credentials
  • WebSocket

This problem is coming because the end-user must be using the Jenkins with HTTPS as a self-signed certificate. So when the Kubernetes plugin tries to launch the base jenkins-inbound-agent container it does not identify the master Jenkins cert. hence the unable to find valid certification path to requested target error.

Solution: - To fix this issue import the master Jenkins certificate into the java truststore cacerts of the jenkins-inbound-agent. This means having a custom jenkins-inbound-agent hosted on the private registry.

1. Get The master Jenkins certificate.

    $ openssl s_client -connect jenkins.my.domain.net:443 -showcerts > jenkins.crt
    depth=0 C = IN, ST = , L = Delhi, O = domain, OU = IT Operations, CN = *.jenkins.my.domain.net
    verify error:num=20:unable to get local issuer certificate
    verify return:1
    depth=0 C = IN, ST = , L = Delhi, O = domain, OU = IT Operations, CN = *.jenkins.my.domain.net
    verify error:num=21:unable to verify the first certificate
    verify return:1
    $ ls -lrth jenkins.crt
    -rw-rw-r--. 1 jenkins jenkins 3.3K Oct 20 09:52 jenkins.crt
    $

2. Inject this self-signed cert into the Jenkins-inbound-agent java cacerts.

Add below to your Dockerfile, Before entry point script.

COPY jenkins.crt /tmp/jenkins.crt
RUN keytool -import -trustcacerts -keystore /opt/java/openjdk/jre/lib/security/cacerts -storepass ******* -noprompt -alias jenkins-master -file /tmp/jenkins.crt \
     && rm -Rf /tmp/jenkins.crt

3. Build custom docker image of jenkins-inbound-agent

$ docker build . -t myregistery.company.net:5000/company/jenkins-agent:latest
$ docker push myregistery.company.net:5000/company/jenkins-agent:latest

4. Pull the custom jenkins-agent from private registry.

• In Manage Jenkins > Manage Nodes and Clouds > Configure Clouds > Kubernetes Cloud details
• Advanced.. > Defaults Provider Template Name > set value default-java
• Expand Pod templates
  • Set Name to default-java
• Expand Container tempalates
  • Set Name to jnlp
  • Set Docker Image to myregistery.company.net:5000/company/jenkins-agent:latest
  • Set Always Pull Image to True
  • Set Allocate pseudo-TTY to True

After this, Create one sample Pipeline Project and use the below code to test run jenkins-inbound-agent on the Kubernetes cluster on the fly.

pipeline {
  agent {
    kubernetes {
      yaml '''
        apiVersion: v1
        kind: Pod
        '''
    }
  }
  stages {
    stage('Run') {
      steps {
            sh 'date'
            sh 'ls -lrth'
        
      }
    }
  }
}