Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49200

Dynamic podTemplate not available when resuming pipeline after restart

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • kubernetes-plugin
    • None
    • Jenkins 2.89.3
      Kubernetes plugin 1.1.3

      Steps to reproduce:

      1. Create Pipeline Job

      podTemplate(label: "mynode", containers: [
          containerTemplate(
              name: 'jnlp',
              image: 'jenkinsci/jnlp-slave:3.16-1',
              args: '${computer.jnlpmac} ${computer.name}',   
              command: '',
              privileged: false,
              resourceRequestCpu: '50m',
              resourceLimitCpu: '500m',
              resourceRequestMemory: '256Mi',
              resourceLimitMemory: '1Gi'        
          ),]
          
          )
      {
         stage ('Stage1'){     
           node("mynode") {
             sh "sleep 5"
             sh "echo 'work done!'"
           }     
         }
         
         def choice = input message: 'Ready?', parameters: [
                          [$class: 'BooleanParameterDefinition', defaultValue: false, description: 'Ready to continue?', name: 'Ready']]
      
         stage ('Stage 2'){
           node("mynode") {
             sh "echo 'sleeping 10 s'"
             sh "sleep 10"
             sh "echo 'awake...'"
           }
         }
       
      }

      2. Start this sample pipeline and wait until pipeline is waiting in input step
      3. Restart Jenkins master
      4. When Jenkins is available again answer input step in build log and see that jenkins fails to continue pipeline because it cannot schedule a pod due to missing pod template

      Starting the pipeline creates a dynamic podTemplate in kubernetes cloud configuration. You can see this when you edit the system config via UI. This configuration is not persistent, after Jenkins restart it is lost and resuming the pipeline obviously does not recreate this dynamic podTemplate configuration. Then the resumed build waits forever with this message:

      Still waiting to schedule task
      Waiting for next available executor

       

      A workaround for this problem is to create a podTemplate in Jenkins Kubernetes cloud config and use a Pipeline without podTemplate definition:

         stage ('Stage1'){     
           node("mygeneralnode") {
             sh "sleep 5"
             sh "echo 'work done!'"
           }     
         }
         
         def choice = input message: 'Ready?', parameters: [
                          [$class: 'BooleanParameterDefinition', defaultValue: false, description: 'Ready to continue?', name: 'Ready']]
      
         stage ('Stage 2'){
           node("mygeneralnode") {
             sh "echo 'sleeping 10 s'"
             sh "sleep 10"
             sh "echo 'awake...'"
           }
         }

            csanchez Carlos Sanchez
            dyrssen Kai Dyrssen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: