Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28403

OpenStack-Clouds-Plugin (single-use slave has already been used and is pending removal for 10 Mins.)

    Details

    • Similar Issues:

      Description

      When the Job finishes building.
      The slave status becomes "Single-use slave has already been used and is pending removal."
      but the VM is still stale.
      Force deleting it switches to "Slave is pending removal.", until it eventually
      gets deleted. if i do not interfere by force deleting the VM, it gets deleted after 10 Mins.
      The VM should get deleted instantly after the job finishes.

        Attachments

        1. 10.png
          10.png
          11 kB
        2. 2.png
          2.png
          11 kB
        3. 3.png
          3.png
          3 kB
        4. 3.png
          3.png
          20 kB
        5. 4.png
          4.png
          17 kB
        6. 5.png
          5.png
          11 kB
        7. 9.png
          9.png
          21 kB

          Activity

          jenkinsguru Jenkins Guru created issue -
          jenkinsguru Jenkins Guru made changes -
          Field Original Value New Value
          Summary SSH-slaves fails because Windows VM did not start yet on OpenStack weird behaviour openstack 2 instances are created instead of one and only one gets deleted instantly, the other takes 30 minutes
          Attachment 2.png [ 29792 ]
          Attachment 3.png [ 29793 ]
          Description When Open-stack cloud plugin dynamically provisions a VM, it uses SSH-slaves plugins to install the slave agent. Similar to this case https://issues.jenkins-ci.org/browse/JENKINS-20921
           but i used Cygwin as suggested by arazauci.
          The problem is that it takes a windows VM a little bit more time to start the SSH daemon, so the ssh-slaves plugin fails to start the SSH connection since windows did not start just yet.
          5 seconds later when the VM launches, i can connect the slave node. However, i would have to do this manually to trigger a job every time.
          Can we add an a time out for the ssh slaves configuration in openstack provisioning ?
          When Open-stack cloud plugin dynamically provisions a VM, it uses SSH-slaves plugins to install the slave agent. Similar to this case https://issues.jenkins-ci.org/browse/JENKINS-20921
           but i used Cygwin as suggested by arazauci.
          Jenkins eventually manages to ssh to the Windows machine and install the agent. The weird behavior occurs the second after it Jenkins node shows "Slave successfully connected and online", another instances gets created and deleted instantly, while the one that runs the job stays running stale. OpenStack have already deleted it but the machine is actually still running under Xen Hypervisor. Jenkins still sees the VM and eventually it gets deleted. i tried checking\Unchecking single use slave and i had the same behavior.
          Why are 2 instances created instead of 1 ?
          why does it keep the VM online for so long

          Priority Major [ 3 ] Blocker [ 1 ]
          jenkinsguru Jenkins Guru made changes -
          Summary weird behaviour openstack 2 instances are created instead of one and only one gets deleted instantly, the other takes 30 minutes Weird behavior openstack-plugin
          jenkinsguru Jenkins Guru made changes -
          Description When Open-stack cloud plugin dynamically provisions a VM, it uses SSH-slaves plugins to install the slave agent. Similar to this case https://issues.jenkins-ci.org/browse/JENKINS-20921
           but i used Cygwin as suggested by arazauci.
          Jenkins eventually manages to ssh to the Windows machine and install the agent. The weird behavior occurs the second after it Jenkins node shows "Slave successfully connected and online", another instances gets created and deleted instantly, while the one that runs the job stays running stale. OpenStack have already deleted it but the machine is actually still running under Xen Hypervisor. Jenkins still sees the VM and eventually it gets deleted. i tried checking\Unchecking single use slave and i had the same behavior.
          Why are 2 instances created instead of 1 ?
          why does it keep the VM online for so long

          When Open-stack cloud plugin dynamically provisions a VM, it uses SSH-slaves plugins to install the slave agent. Similar to this case https://issues.jenkins-ci.org/browse/JENKINS-20921
           but i used Cygwin as suggested by arazauci.
          Jenkins eventually manages to ssh to the Windows machine and install the agent. The weird behavior occurs the second after it Jenkins node shows "Slave successfully connected and online", another instances gets created and deleted instantly, while the one that runs the job stays running stale. Jenkins still sees the VM, I have to (delete slave) manually. i tried checking\Unchecking single use slave and i had the same behavior.
          Why are 2 instances created instead of 1 ?
          why does it keep the second one online?

          Hide
          jenkinsguru Jenkins Guru added a comment -

          openstack-cloud-plugin

          Show
          jenkinsguru Jenkins Guru added a comment - openstack-cloud-plugin
          jenkinsguru Jenkins Guru made changes -
          Component/s jclouds-plugin [ 16529 ]
          Component/s ssh-slaves-plugin [ 15578 ]
          jenkinsguru Jenkins Guru made changes -
          Summary Weird behavior openstack-plugin OpenStack-Clouds-Plugin takes forever to delete single use slaves
          Attachment 3.png [ 29795 ]
          Description When Open-stack cloud plugin dynamically provisions a VM, it uses SSH-slaves plugins to install the slave agent. Similar to this case https://issues.jenkins-ci.org/browse/JENKINS-20921
           but i used Cygwin as suggested by arazauci.
          Jenkins eventually manages to ssh to the Windows machine and install the agent. The weird behavior occurs the second after it Jenkins node shows "Slave successfully connected and online", another instances gets created and deleted instantly, while the one that runs the job stays running stale. Jenkins still sees the VM, I have to (delete slave) manually. i tried checking\Unchecking single use slave and i had the same behavior.
          Why are 2 instances created instead of 1 ?
          why does it keep the second one online?

          When the Job finishes building.
          The slave status becomes "Single-use slave has already been used and is pending removal."
          but the VM is still stale.
          Force deleting it switches to "Slave is pending removal.", until it eventually
          gets deleted. The VM should get deleted instantly after the job finishes.
          jenkinsguru Jenkins Guru made changes -
          Summary OpenStack-Clouds-Plugin takes forever to delete single use slaves OpenStack-Clouds-Plugin (single-use slave has already been used and is pending removal)
          jenkinsguru Jenkins Guru made changes -
          Attachment 5.png [ 29796 ]
          Attachment 4.png [ 29797 ]
          jenkinsguru Jenkins Guru made changes -
          Description When the Job finishes building.
          The slave status becomes "Single-use slave has already been used and is pending removal."
          but the VM is still stale.
          Force deleting it switches to "Slave is pending removal.", until it eventually
          gets deleted. The VM should get deleted instantly after the job finishes.
          When the Job finishes building.
          The slave status becomes "Single-use slave has already been used and is pending removal."
          but the VM is still stale.
          Force deleting it switches to "Slave is pending removal.", until it eventually
          gets deleted. if i do not interfere by force deleting the VM, it gets deleted after 10 Mins.
          The VM should get deleted instantly after the job finishes.
          jenkinsguru Jenkins Guru made changes -
          Priority Blocker [ 1 ] Minor [ 4 ]
          Hide
          nikolay_borisenko Nikolay Borisenko added a comment - - edited

          Removing nodes also takes about 10 mins. I don't use "Single Use" option.

          Show
          nikolay_borisenko Nikolay Borisenko added a comment - - edited Removing nodes also takes about 10 mins. I don't use "Single Use" option.
          Hide
          jenkinsguru Jenkins Guru added a comment -

          It gets deleted eventually but It takes 10 minutes to delete the VM

          Show
          jenkinsguru Jenkins Guru added a comment - It gets deleted eventually but It takes 10 minutes to delete the VM
          jenkinsguru Jenkins Guru made changes -
          Attachment 9.png [ 29800 ]
          Hide
          jenkinsguru Jenkins Guru added a comment -

          I tried running the job many times, i have to wait for 10 minutes for the VM to be deleted from OpenStack and Xen

          Show
          jenkinsguru Jenkins Guru added a comment - I tried running the job many times, i have to wait for 10 minutes for the VM to be deleted from OpenStack and Xen
          Hide
          jenkinsguru Jenkins Guru added a comment -

          How long should it take to the node offline and delete it ?

          Show
          jenkinsguru Jenkins Guru added a comment - How long should it take to the node offline and delete it ?
          jenkinsguru Jenkins Guru made changes -
          Summary OpenStack-Clouds-Plugin (single-use slave has already been used and is pending removal) OpenStack-Clouds-Plugin (single-use slave has already been used and is pending removal for 10 Mins.)
          Hide
          jenkinsguru Jenkins Guru added a comment -

          I tried using Single Use Slave Plugin to try and speed up the deletion process.
          However, it still takes 10 minutes.

          Show
          jenkinsguru Jenkins Guru added a comment - I tried using Single Use Slave Plugin to try and speed up the deletion process. However, it still takes 10 minutes.
          jenkinsguru Jenkins Guru made changes -
          Attachment 10.png [ 29802 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 163242 ] JNJira + In-Review [ 181149 ]
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Fritz Elfert
          Path:
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsComputer.java
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsOneOffSlave.java
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsRetentionStrategy.java
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsSlave.java
          http://jenkins-ci.org/commit/jclouds-plugin/f141bc7633987e691856e4c20af6dd262e27f9c1
          Log:
          Fixing JENKINS-28403 and JENKINS-27471 (WIP)

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Fritz Elfert Path: jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsComputer.java jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsOneOffSlave.java jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsRetentionStrategy.java jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsSlave.java http://jenkins-ci.org/commit/jclouds-plugin/f141bc7633987e691856e4c20af6dd262e27f9c1 Log: Fixing JENKINS-28403 and JENKINS-27471 (WIP)
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Fritz Elfert
          Path:
          jclouds-plugin/pom.xml
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/InstancesToRun.java
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsBuildWrapper.java
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsCloud.java
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsOneOffSlave.java
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsStartupHandler.java
          jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/internal/TerminateNodes.java
          jclouds-plugin/src/main/resources/jenkins/plugins/jclouds/compute/InstancesToRun/config.jelly
          jclouds-plugin/src/main/resources/jenkins/plugins/jclouds/compute/JCloudsBuildWrapper/help-instancesToRun.html
          jclouds-plugin/src/main/resources/jenkins/plugins/jclouds/compute/JCloudsBuildWrapper/help.html
          jclouds-plugin/src/main/resources/jenkins/plugins/jclouds/compute/JCloudsOneOffSlave/help.html
          jclouds-plugin/src/test/java/jenkins/plugins/jclouds/compute/internal/TerminateNodesTest.java
          http://jenkins-ci.org/commit/jclouds-plugin/3bc1200cba16f54c9bcad01cec6402bf29b2e6c6
          Log:
          Fixing JENKINS-28403 and JENKINS-27471 (WIP)

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Fritz Elfert Path: jclouds-plugin/pom.xml jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/InstancesToRun.java jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsBuildWrapper.java jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsCloud.java jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsOneOffSlave.java jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/JCloudsStartupHandler.java jclouds-plugin/src/main/java/jenkins/plugins/jclouds/compute/internal/TerminateNodes.java jclouds-plugin/src/main/resources/jenkins/plugins/jclouds/compute/InstancesToRun/config.jelly jclouds-plugin/src/main/resources/jenkins/plugins/jclouds/compute/JCloudsBuildWrapper/help-instancesToRun.html jclouds-plugin/src/main/resources/jenkins/plugins/jclouds/compute/JCloudsBuildWrapper/help.html jclouds-plugin/src/main/resources/jenkins/plugins/jclouds/compute/JCloudsOneOffSlave/help.html jclouds-plugin/src/test/java/jenkins/plugins/jclouds/compute/internal/TerminateNodesTest.java http://jenkins-ci.org/commit/jclouds-plugin/3bc1200cba16f54c9bcad01cec6402bf29b2e6c6 Log: Fixing JENKINS-28403 and JENKINS-27471 (WIP)
          Hide
          felfert Fritz Elfert added a comment -

          Should be fixed now in git.
          Single-use JClouds-slaves ar now usually removed much faster.
          Please note however, that removal is postponed until they are completely idle, which might not be obvious if they are executing FlyWeight tasks.

          Show
          felfert Fritz Elfert added a comment - Should be fixed now in git. Single-use JClouds-slaves ar now usually removed much faster. Please note however, that removal is postponed until they are completely idle, which might not be obvious if they are executing FlyWeight tasks.
          felfert Fritz Elfert made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              kohsuke Kohsuke Kawaguchi
              Reporter:
              jenkinsguru Jenkins Guru
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: