Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53621

Executors in-progress block container provisioning after configuration changes

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: docker-plugin
    • Labels:
      None
    • Environment:
      CentOS 6.8,
      Jenkins 2.141
      Docker-Plugin 1.1.5
    • Similar Issues:

      Description

      Changing Cloud configuration while there are existing containers seems to really screw things up.  I changed the docker host URI (to a different host) and changed the Container Cap.

      As a result, on the new host, I had a disagreement between the Container Cap and the actual container count.  For example, I had the following combination at one point:

      • Containers running: 1
      • Container Cap: 7 (I kept increasing it, but it only helped temporarily)
      • Log shows: 
        Sep 17, 2018 7:16:35 AM INFO com.nirima.jenkins.plugins.docker.DockerCloud provision
        Not provisioning additional slaves for container&&centos&&mem32; we have 3 executors being started already

      This did not change in 3 days (Jenkins just kept printing this in the log), so I hacked my way into clearing the count of containers in progress in a jenkins script console: (I have only one Docker cloud)

      Jenkins.getInstance().clouds.get(com.nirima.jenkins.plugins.docker.DockerCloud).CONTAINERS_IN_PROGRESS.clear()
      

      And now I'm getting the right count of provisioned containers!

      Incidentally, the content of "CONTAINERS_IN_PROGRESS" was a map with two keys (old and new Docker host), new one was set to 3 and the old one was at 7...

       

        Attachments

          Activity

          Hide
          pjdarton pjdarton added a comment -

          I'd seen anecdotal reports of this kind of issue before ... but none mentioning "Changing Cloud configuration while there are existing containers seems to really screw things up" - I think you've found the underlying cause there

          That information makes this kind of bug a lot easier to fix...

          Show
          pjdarton pjdarton added a comment - I'd seen anecdotal reports of this kind of issue before ... but none mentioning "Changing Cloud configuration while there are existing containers seems to really screw things up" - I think you've found the underlying cause there That information makes this kind of bug a lot easier to fix...

            People

            • Assignee:
              pjdarton pjdarton
              Reporter:
              akom Alexander Komarov
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: