Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50801

Cannot provision Slave Pod: mountPath: Required value

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Component/s: kubernetes-plugin
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      We are running the standard Jenkins container (lts) on Kubernetes and it is provisioning build slaves on the same cluster. After upgrading to Kubernetes Plugin v1.5.1 (from v1.3.2), Jenkins was no longer able to provision any build slaves and the Jenkins logs showed many errors like:

       

      {{Error in provisioning; agent=KubernetesSlave name: builder-lfc8x, template=PodTemplate{, name='builder', namespace='jenkins', label='builder', nodeSelector='', nodeUsageMode=NORMAL, workspaceVolume=org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume@68a57ce6, volumes=[org.csanchez.jenkins.plugins.kubernetes.volumes.EmptyDirVolume@8e436e0, HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock], HostPathVolume [mountPath=/usr/bin/docker, hostPath=/usr/bin/docker]], containers=[ContainerTemplate
      {name='git-auth', image='us.gcr.io/myproject/git-auth:v1', workingDir='', command='', args='', resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@4f8c0bfd}
      , ContainerTemplate
      {name='jnlp', image='us.gcr.io/myproject/jnlp-linux-node:v1', workingDir='/workspace/', command='', args='', resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@26954299}
      , ContainerTemplate
      {name='builder', image='us.gcr.io/myproject/builder:v8', workingDir='/workspace/', command='cat', args='', ttyEnabled=true, resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@6d1df705}
      ]}}}
       io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default/api/v1/namespaces/jenkins/pods. Message: Pod "builder-lfc8x" is invalid: spec.containers[0].volumeMounts[3].mountPath: Required value. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.containers[0].volumeMounts[3].mountPath, message=Required value, reason=FieldValueRequired, additionalProperties={})], group=null, kind=Pod, name=builder-lfc8x, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "builder-lfc8x" is invalid: spec.containers[0].volumeMounts[3].mountPath: Required value, metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
       {{ at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:472)}}
       {{ at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:411)}}
       {{ at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)}}
       {{ at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)}}
       {{ at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:227)}}
       {{ at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:756)}}
       {{ at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:334)}}
       {{ at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:105)}}
       {{ at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:288)}}
       {{ at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)}}
       {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
       {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
       {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
       {{ at java.lang.Thread.run(Thread.java:748)}}{{}}
       
      

      This seems like it is probably related to https://issues.jenkins-ci.org/browse/JENKINS-50525, but since that fix is in 1.5.1, perhaps we are experiencing something slightly different.

      We have reverted the Kubernetes plugin back to 1.3.2 while we figure this out.

       

        Attachments

          Issue Links

            Activity

            Hide
            csanchez Carlos Sanchez added a comment -

            pipeline or global config? do you have an example?

            Show
            csanchez Carlos Sanchez added a comment - pipeline or global config? do you have an example?
            Hide
            rwehner Robert Wehner added a comment -

            The jobs failing in this way are all pipelines (we don't have any non-pipeline jobs to compare to). Here's one example:

             

            node('builder') { 
            stage('Clean Workspace') { 
            cleanWs() 
            } 
            stage('Clone source repo') { 
            checkout scm 
            } 
            stage('Run Tests') { 
            container('builder') { 
            setGerritReview unsuccessfulMessage: "Tests failed" 
            sh 'make test' 
            } 
            } 
            stage('Run Package Build') { 
            container('builder') { 
            setGerritReview unsuccessfulMessage: "Build failed" 
            sh """ 
            export BUILD_NUMBER="${env.BUILD_NUMBER}" 
            export WORKSPACE="${env.WORKSPACE}" 
            make build 
            """ 
            } 
            } 
            }
             
            

            The "builder" node is a PodTemplate defined in the UI. It inherits another UI-defined PodTemplate called "k8s-linux". Here's the cloud section from the config.xml:

            <clouds>
             <org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin="kubernetes@1.5.1">
             <name>kubernetes</name>
             <defaultsProviderTemplate></defaultsProviderTemplate>
             <templates>
             <org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
             <inheritFrom></inheritFrom>
             <name>k8s-linux</name>
             <namespace></namespace>
             <privileged>false</privileged>
             <alwaysPullImage>false</alwaysPullImage>
             <instanceCap>2147483647</instanceCap>
             <slaveConnectTimeout>100</slaveConnectTimeout>
             <idleMinutes>5</idleMinutes>
             <activeDeadlineSeconds>0</activeDeadlineSeconds>
             <label>k8s linux</label>
             <nodeSelector></nodeSelector>
             <nodeUsageMode>NORMAL</nodeUsageMode>
             <customWorkspaceVolumeEnabled>false</customWorkspaceVolumeEnabled>
             <workspaceVolume class="org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume">
             <memory>false</memory>
             </workspaceVolume>
             <volumes>
             <org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume>
             <mountPath>/usr/bin/docker</mountPath>
             <hostPath>/usr/bin/docker</hostPath>
             </org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume>
             <org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume>
             <mountPath>/var/run/docker.sock</mountPath>
             <hostPath>/var/run/docker.sock</hostPath>
             </org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume>
             <org.csanchez.jenkins.plugins.kubernetes.volumes.EmptyDirVolume>
             <mountPath>/var/lib/gob-auth</mountPath>
             <memory>true</memory>
             </org.csanchez.jenkins.plugins.kubernetes.volumes.EmptyDirVolume>
             </volumes>
             <containers>
             <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
             <name>jnlp</name>
             <image>us.gcr.io/myproject/jnlp-linux-node:v1</image>
             <privileged>false</privileged>
             <alwaysPullImage>false</alwaysPullImage>
             <workingDir>/workspace/</workingDir>
             <command></command>
             <args></args>
             <ttyEnabled>false</ttyEnabled>
             <resourceRequestCpu></resourceRequestCpu>
             <resourceRequestMemory></resourceRequestMemory>
             <resourceLimitCpu></resourceLimitCpu>
             <resourceLimitMemory></resourceLimitMemory>
             <envVars/>
             <ports/>
             <livenessProbe>
             <execArgs></execArgs>
             <timeoutSeconds>0</timeoutSeconds>
             <initialDelaySeconds>0</initialDelaySeconds>
             <failureThreshold>0</failureThreshold>
             <periodSeconds>0</periodSeconds>
             <successThreshold>0</successThreshold>
             </livenessProbe>
             </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
             <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
             <name>git-auth</name>
             <image>us.gcr.io/myproject/git-auth:v1</image>
             <privileged>false</privileged>
             <alwaysPullImage>false</alwaysPullImage>
             <workingDir></workingDir>
             <command></command>
             <args></args>
             <ttyEnabled>false</ttyEnabled>
             <resourceRequestCpu></resourceRequestCpu>
             <resourceRequestMemory></resourceRequestMemory>
             <resourceLimitCpu></resourceLimitCpu>
             <resourceLimitMemory></resourceLimitMemory>
             <envVars/>
             <ports/>
             <livenessProbe>
             <execArgs></execArgs>
             <timeoutSeconds>0</timeoutSeconds>
             <initialDelaySeconds>0</initialDelaySeconds>
             <failureThreshold>0</failureThreshold>
             <periodSeconds>0</periodSeconds>
             <successThreshold>0</successThreshold>
             </livenessProbe>
             </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
             </containers>
             <envVars/>
             <annotations/>
             <imagePullSecrets/>
             </org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
             <org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
             <inheritFrom>k8s-linux</inheritFrom>
             <name>builder</name>
             <namespace>jenkins</namespace>
             <privileged>false</privileged>
             <alwaysPullImage>false</alwaysPullImage>
             <instanceCap>2147483647</instanceCap>
             <slaveConnectTimeout>100</slaveConnectTimeout>
             <idleMinutes>1</idleMinutes>
             <activeDeadlineSeconds>0</activeDeadlineSeconds>
             <label>builder</label>
             <nodeSelector></nodeSelector>
             <nodeUsageMode>NORMAL</nodeUsageMode>
             <customWorkspaceVolumeEnabled>false</customWorkspaceVolumeEnabled>
             <workspaceVolume class="org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume">
             <memory>false</memory>
             </workspaceVolume>
             <volumes/>
             <containers>
             <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
             <name>builder</name>
             <image>us.gcr.io/myproject/builder:v8</image>
             <privileged>false</privileged>
             <alwaysPullImage>false</alwaysPullImage>
             <workingDir>/workspace/</workingDir>
             <command>cat</command>
             <args></args>
             <ttyEnabled>true</ttyEnabled>
             <resourceRequestCpu></resourceRequestCpu>
             <resourceRequestMemory></resourceRequestMemory>
             <resourceLimitCpu></resourceLimitCpu>
             <resourceLimitMemory></resourceLimitMemory>
             <envVars/>
             <ports/>
             <livenessProbe>
             <execArgs></execArgs>
             <timeoutSeconds>0</timeoutSeconds>
             <initialDelaySeconds>0</initialDelaySeconds>
             <failureThreshold>0</failureThreshold>
             <periodSeconds>0</periodSeconds>
             <successThreshold>0</successThreshold>
             </livenessProbe>
             </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
             </containers>
             <envVars/>
             <annotations/>
             <imagePullSecrets/>
             </org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
             </templates>
             <serverUrl>https://kubernetes.default</serverUrl>
             <skipTlsVerify>false</skipTlsVerify>
             <namespace>jenkins</namespace>
             <jenkinsUrl>http://jenkins-ui.jenkins.svc.cluster.local:8080</jenkinsUrl>
             <jenkinsTunnel>jenkins-discovery.jenkins.svc.cluster.local:50000</jenkinsTunnel>
             <credentialsId>jenkins</credentialsId>
             <containerCap>10</containerCap>
             <retentionTimeout>5</retentionTimeout>
             <connectTimeout>0</connectTimeout>
             <readTimeout>0</readTimeout>
             <labels class="com.google.common.collect.ImmutableMap">
             <entry>
             <string>jenkins</string>
             <string>slave</string>
             </entry>
             </labels>
             <maxRequestsPerHost>32</maxRequestsPerHost>
             </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud>
             </clouds>
            Show
            rwehner Robert Wehner added a comment - The jobs failing in this way are all pipelines (we don't have any non-pipeline jobs to compare to). Here's one example:   node( 'builder' ) { stage( 'Clean Workspace' ) { cleanWs() } stage( 'Clone source repo' ) { checkout scm } stage( 'Run Tests' ) { container( 'builder' ) { setGerritReview unsuccessfulMessage: "Tests failed" sh 'make test' } } stage( 'Run Package Build' ) { container( 'builder' ) { setGerritReview unsuccessfulMessage: "Build failed" sh """ export BUILD_NUMBER= "${env.BUILD_NUMBER}" export WORKSPACE= "${env.WORKSPACE}" make build """ } } }   The "builder" node is a PodTemplate defined in the UI. It inherits another UI-defined PodTemplate called "k8s-linux". Here's the cloud section from the config.xml: <clouds> <org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin= "kubernetes@1.5.1" > <name>kubernetes</name> <defaultsProviderTemplate></defaultsProviderTemplate> <templates> <org.csanchez.jenkins.plugins.kubernetes.PodTemplate> <inheritFrom></inheritFrom> <name>k8s-linux</name> <namespace></namespace> <privileged> false </privileged> <alwaysPullImage> false </alwaysPullImage> <instanceCap>2147483647</instanceCap> <slaveConnectTimeout>100</slaveConnectTimeout> <idleMinutes>5</idleMinutes> <activeDeadlineSeconds>0</activeDeadlineSeconds> <label>k8s linux</label> <nodeSelector></nodeSelector> <nodeUsageMode>NORMAL</nodeUsageMode> <customWorkspaceVolumeEnabled> false </customWorkspaceVolumeEnabled> <workspaceVolume class= "org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume" > <memory> false </memory> </workspaceVolume> <volumes> <org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume> <mountPath>/usr/bin/docker</mountPath> <hostPath>/usr/bin/docker</hostPath> </org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume> <org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume> <mountPath>/ var /run/docker.sock</mountPath> <hostPath>/ var /run/docker.sock</hostPath> </org.csanchez.jenkins.plugins.kubernetes.volumes.HostPathVolume> <org.csanchez.jenkins.plugins.kubernetes.volumes.EmptyDirVolume> <mountPath>/ var /lib/gob-auth</mountPath> <memory> true </memory> </org.csanchez.jenkins.plugins.kubernetes.volumes.EmptyDirVolume> </volumes> <containers> <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> <name>jnlp</name> <image>us.gcr.io/myproject/jnlp-linux-node:v1</image> <privileged> false </privileged> <alwaysPullImage> false </alwaysPullImage> <workingDir>/workspace/</workingDir> <command></command> <args></args> <ttyEnabled> false </ttyEnabled> <resourceRequestCpu></resourceRequestCpu> <resourceRequestMemory></resourceRequestMemory> <resourceLimitCpu></resourceLimitCpu> <resourceLimitMemory></resourceLimitMemory> <envVars/> <ports/> <livenessProbe> <execArgs></execArgs> <timeoutSeconds>0</timeoutSeconds> <initialDelaySeconds>0</initialDelaySeconds> <failureThreshold>0</failureThreshold> <periodSeconds>0</periodSeconds> <successThreshold>0</successThreshold> </livenessProbe> </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> <name>git-auth</name> <image>us.gcr.io/myproject/git-auth:v1</image> <privileged> false </privileged> <alwaysPullImage> false </alwaysPullImage> <workingDir></workingDir> <command></command> <args></args> <ttyEnabled> false </ttyEnabled> <resourceRequestCpu></resourceRequestCpu> <resourceRequestMemory></resourceRequestMemory> <resourceLimitCpu></resourceLimitCpu> <resourceLimitMemory></resourceLimitMemory> <envVars/> <ports/> <livenessProbe> <execArgs></execArgs> <timeoutSeconds>0</timeoutSeconds> <initialDelaySeconds>0</initialDelaySeconds> <failureThreshold>0</failureThreshold> <periodSeconds>0</periodSeconds> <successThreshold>0</successThreshold> </livenessProbe> </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> </containers> <envVars/> <annotations/> <imagePullSecrets/> </org.csanchez.jenkins.plugins.kubernetes.PodTemplate> <org.csanchez.jenkins.plugins.kubernetes.PodTemplate> <inheritFrom>k8s-linux</inheritFrom> <name>builder</name> <namespace>jenkins</namespace> <privileged> false </privileged> <alwaysPullImage> false </alwaysPullImage> <instanceCap>2147483647</instanceCap> <slaveConnectTimeout>100</slaveConnectTimeout> <idleMinutes>1</idleMinutes> <activeDeadlineSeconds>0</activeDeadlineSeconds> <label>builder</label> <nodeSelector></nodeSelector> <nodeUsageMode>NORMAL</nodeUsageMode> <customWorkspaceVolumeEnabled> false </customWorkspaceVolumeEnabled> <workspaceVolume class= "org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume" > <memory> false </memory> </workspaceVolume> <volumes/> <containers> <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> <name>builder</name> <image>us.gcr.io/myproject/builder:v8</image> <privileged> false </privileged> <alwaysPullImage> false </alwaysPullImage> <workingDir>/workspace/</workingDir> <command>cat</command> <args></args> <ttyEnabled> true </ttyEnabled> <resourceRequestCpu></resourceRequestCpu> <resourceRequestMemory></resourceRequestMemory> <resourceLimitCpu></resourceLimitCpu> <resourceLimitMemory></resourceLimitMemory> <envVars/> <ports/> <livenessProbe> <execArgs></execArgs> <timeoutSeconds>0</timeoutSeconds> <initialDelaySeconds>0</initialDelaySeconds> <failureThreshold>0</failureThreshold> <periodSeconds>0</periodSeconds> <successThreshold>0</successThreshold> </livenessProbe> </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate> </containers> <envVars/> <annotations/> <imagePullSecrets/> </org.csanchez.jenkins.plugins.kubernetes.PodTemplate> </templates> <serverUrl>https: //kubernetes. default </serverUrl> <skipTlsVerify> false </skipTlsVerify> <namespace>jenkins</namespace> <jenkinsUrl>http: //jenkins-ui.jenkins.svc.cluster.local:8080</jenkinsUrl> <jenkinsTunnel>jenkins-discovery.jenkins.svc.cluster.local:50000</jenkinsTunnel> <credentialsId>jenkins</credentialsId> <containerCap>10</containerCap> <retentionTimeout>5</retentionTimeout> <connectTimeout>0</connectTimeout> <readTimeout>0</readTimeout> <labels class= "com.google.common.collect.ImmutableMap" > <entry> <string>jenkins</string> <string>slave</string> </entry> </labels> <maxRequestsPerHost>32</maxRequestsPerHost> </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud> </clouds>

              People

              • Assignee:
                csanchez Carlos Sanchez
                Reporter:
                rwehner Robert Wehner
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: