Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52159

Jenkins Queue DoS when QueueItemSorter extension throws runtime Exception

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      Jenkins 2.130

      Description

      We just upgraded to 2.129. Our jenkins is configured to start nodes as needed; release nodes as upon idle; and run a job on master which turns nodes off if they aren't in use.

      We had the accelerated-build-now-plugin (version 1.0.1) plugin enabled.

      When our normal or pipeline jobs start, they try to pick any node other than master. Jenkins starts the node, it appears in the list of available nodes, but Jenkins doesn't manage to start a build on the node:

      SEVERE: Timer task hudson.model.Queue$MaintainTask@2041414e failed
      java.lang.ClassCastException: org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask cannot be cast to hudson.model.AbstractProject
              at org.terracotta.jenkins.plugins.acceleratedbuildnow.AcceleratedBuildNowComparator.compare(AcceleratedBuildNowComparator.java:21)
              at org.terracotta.jenkins.plugins.acceleratedbuildnow.AcceleratedBuildNowComparator.compare(AcceleratedBuildNowComparator.java:11)
              at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
              at java.util.TimSort.sort(TimSort.java:220)
              at java.util.Arrays.sort(Arrays.java:1512)
              at java.util.ArrayList.sort(ArrayList.java:1462)
              at java.util.Collections.sort(Collections.java:175)
              at org.terracotta.jenkins.plugins.acceleratedbuildnow.AcceleratedBuildNowSorter.sortBuildableItems(AcceleratedBuildNowSorter.java:30)
              at hudson.model.Queue.maintain(Queue.java:1571)
              at hudson.model.Queue$MaintainTask.doRun(Queue.java:2862)
              at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748) 

      Instead, the jobs sit waiting, and eventually the computers being idle are released by jenkins and then reaped by the job on master.

      Disabling this plugin results in our system working as it used to work.

        Attachments

          Issue Links

            Activity

            Hide
            jsoref Josh Soref added a comment -

            I think that a try/catch block here:

            https://github.com/jenkinsci/jenkins/blob/d4737b2b60438ce0e3cb7a8a6f26675f7c880914/core/src/main/java/hudson/model/Queue.java#L1571

            would protect our jobs from getting lost, but I'm not sure if people would accept that.

            Note, per: https://github.com/Terracotta-OSS/accelerated-build-now-plugin/issues/3#issuecomment-374920842, this plugin appears to be mostly unmaintained.

            Show
            jsoref Josh Soref added a comment - I think that a try/catch block here: https://github.com/jenkinsci/jenkins/blob/d4737b2b60438ce0e3cb7a8a6f26675f7c880914/core/src/main/java/hudson/model/Queue.java#L1571 would protect our jobs from getting lost, but I'm not sure if people would accept that. Note, per:  https://github.com/Terracotta-OSS/accelerated-build-now-plugin/issues/3#issuecomment-374920842 , this plugin appears to be mostly unmaintained.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I'd guess accelerated-build-now-plugin is just incompatible with Jenkins Pipeline.
            Whatever was the configuration before the upgrade, this code is just a bug: https://github.com/jenkinsci/accelerated-build-now-plugin/blob/master/src/main/java/org/terracotta/jenkins/plugins/acceleratedbuildnow/AcceleratedBuildNowComparator.java#L19-L21 . Unchecked casts.

            The plugin also has releases being done outside Jenkins org. It may confuse contributors, but URL in POM is correct.

            Daniel Beck Anthony Dahanne maybe should we just remove the fork in Jenkins GitHub

            Show
            oleg_nenashev Oleg Nenashev added a comment - I'd guess accelerated-build-now-plugin is just incompatible with Jenkins Pipeline. Whatever was the configuration before the upgrade, this code is just a bug: https://github.com/jenkinsci/accelerated-build-now-plugin/blob/master/src/main/java/org/terracotta/jenkins/plugins/acceleratedbuildnow/AcceleratedBuildNowComparator.java#L19-L21 . Unchecked casts. The plugin also has releases being done outside Jenkins org. It may confuse contributors, but URL in POM is correct. Daniel Beck Anthony Dahanne maybe should we just remove the fork in Jenkins GitHub
            Hide
            jsoref Josh Soref added a comment -

            It's possible that someone installed this plugin last week and we only just now hit the rough edge. Anyway. We can split this into two bugs if necessary. (And fix the priority, I agree it doesn't seem like a regression it just apparently triggered coincidentally.)

            Is it possible that the sorting hook only triggers w/ jenkins launch whereas the rest of the plugin can be used w/ an install w/o restart?

            Show
            jsoref Josh Soref added a comment - It's possible that someone installed this plugin last week and we only just now hit the rough edge. Anyway. We can split this into two bugs if necessary. (And fix the priority, I agree it doesn't seem like a regression it just apparently triggered coincidentally.) Is it possible that the sorting hook only triggers w/ jenkins launch whereas the rest of the plugin can be used w/ an install w/o restart?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Josh Soref I kindly ask to create a separate issue for the Jenkins core robustness.
            Otherwise people may be confused when we release the core and maybe backport the fix.
            With all patches on the core side, this issue will be still there

            Show
            oleg_nenashev Oleg Nenashev added a comment - Josh Soref I kindly ask to create a separate issue for the Jenkins core robustness. Otherwise people may be confused when we release the core and maybe backport the fix. With all patches on the core side, this issue will be still there
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Detached the plugin bug to JENKINS-52353, because the issue has been already changelogged

            Show
            oleg_nenashev Oleg Nenashev added a comment - Detached the plugin bug to  JENKINS-52353 , because the issue has been already changelogged
            Hide
            anthonydahanne Anthony Dahanne added a comment -

            hello, sorry I have not been pro active with those issues.

            This plugin has been un maintained for a while.

            If I can help making it "officially" not maintained, please let me know. Or if I need to first ask the community if there's someone interested in maintaining it.

            Show
            anthonydahanne Anthony Dahanne added a comment - hello, sorry I have not been pro active with those issues. This plugin has been un maintained for a while. If I can help making it "officially" not maintained, please let me know. Or if I need to first ask the community if there's someone interested in maintaining it.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Anthony Dahanne no special actions needed, you confirmation is enough.

            I have marked the plugin for adoption. Thanks!

            Show
            oleg_nenashev Oleg Nenashev added a comment - Anthony Dahanne no special actions needed, you confirmation is enough. I have marked the plugin for adoption. Thanks!
            Hide
            anthonydahanne Anthony Dahanne added a comment -

            yes I confirm it's fine ! Adoption is a good idea, I can see you updated the page : https://wiki.jenkins.io/display/JENKINS/Accelerated+Build+Now+Plugin

            Thanks Oleg Nenashev !

            Show
            anthonydahanne Anthony Dahanne added a comment - yes I confirm it's fine ! Adoption is a good idea, I can see you updated the page : https://wiki.jenkins.io/display/JENKINS/Accelerated+Build+Now+Plugin Thanks Oleg Nenashev !

              People

              • Assignee:
                anthonydahanne Anthony Dahanne
                Reporter:
                jsoref Josh Soref
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: