Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49006

ConcurrentModificationException in getThrottledPipelineRunsForCategory

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      I started two builds of a pipeline job which uses the throttle step against a category. The category is configured to permit a maximum of one build at a time. The second build hit the throttled node step, waited for somewhere around 30–40 minutes, and then started executing anyway, despite the first build not having exited the throttled node step yet. I found this in my Jenkins log:

      SEVERE: Timer task hudson.model.Queue$MaintainTask@63d577c failed
      java.util.ConcurrentModificationException
      	at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
      	at java.util.TreeMap$EntryIterator.next(TreeMap.java:1247)
      	at java.util.TreeMap$EntryIterator.next(TreeMap.java:1242)
      	at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getThrottledPipelineRunsForCategory(ThrottleJobProperty.java:316)
      	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.throttleCheckForCategoriesOnNode(ThrottleQueueTaskDispatcher.java:132)
      	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canTakeImpl(ThrottleQueueTaskDispatcher.java:101)
      	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canTake(ThrottleQueueTaskDispatcher.java:55)
      	at hudson.model.queue.QueueTaskDispatcher.canTake(QueueTaskDispatcher.java:102)
      	at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
      	at hudson.model.Queue.maintain(Queue.java:1597)
      	at hudson.model.Queue$MaintainTask.doRun(Queue.java:2833)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
      	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Activity

          Hide
          sk_nano Sascha Kiedrowski added a comment -

          We're having the same problem here.

          OS: Windows 2012 R2, Windows 2016, Jenkins 2.89.3, TCB 2.0.1

          Show
          sk_nano Sascha Kiedrowski added a comment - We're having the same problem here. OS: Windows 2012 R2, Windows 2016, Jenkins 2.89.3, TCB 2.0.1
          Hide
          sk_nano Sascha Kiedrowski added a comment -

          I've created a pull request to fix this issue:

          https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/54

          Show
          sk_nano Sascha Kiedrowski added a comment - I've created a pull request to fix this issue: https://github.com/jenkinsci/throttle-concurrent-builds-plugin/pull/54
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Unfortunately I was unable to find time to work on plugin over last year. I decided to step down as maintainer and to mark the plugin for adoption.

          If anybody is interested, let me know

          Show
          oleg_nenashev Oleg Nenashev added a comment - Unfortunately I was unable to find time to work on plugin over last year. I decided to step down as maintainer and to mark the plugin for adoption. If anybody is interested, let me know
          Hide
          medianick Nick Jones added a comment -

          We were bitten by this today; a PR build (on a multibranch pipeline job) started freezing – appearing to still be in progress in the UI, but not assigned to any executor, and never progressing past the node step, with this in the Jenkins system log:

          Apr 23, 2019 7:36:18 AM WARNING jenkins.util.AtmostOneTaskExecutor$1 call
          null
          java.util.ConcurrentModificationException
          	at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
          	at java.util.TreeMap$EntryIterator.next(TreeMap.java:1247)
          	at java.util.TreeMap$EntryIterator.next(TreeMap.java:1242)
          	at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getThrottledPipelineRunsForCategory(ThrottleJobProperty.java:316)
          	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.throttleCheckForCategoriesAllNodes(ThrottleQueueTaskDispatcher.java:283)
          	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRunImpl(ThrottleQueueTaskDispatcher.java:256)
          	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:218)
          	at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:176)
          	at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1197)
          	at hudson.model.Queue.maintain(Queue.java:1554)
          	at hudson.model.Queue$1.call(Queue.java:325)
          	at hudson.model.Queue$1.call(Queue.java:322)
          	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108)
          	at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98)
          	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112)
          	at java.lang.Thread.run(Thread.java:748)
          

          I tried aborting and restarting it multiple times but it consistently behaved the same way; the only way we got it to build again was by commenting out the throttle block entirely. It would be marginally better if it had simply failed the build, but far better if this ConcurrentModificationException were not happening at all. Incidentially, this PR was a switch to use Scripted Pipeline due to JENKINS-45140, since Declarative Syntax didn't seem to throttle at all.

          Show
          medianick Nick Jones added a comment - We were bitten by this today; a PR build (on a multibranch pipeline job) started freezing – appearing to still be in progress in the UI, but not assigned to any executor, and never progressing past the node step, with this in the Jenkins system log: Apr 23, 2019 7:36:18 AM WARNING jenkins.util.AtmostOneTaskExecutor$1 call null java.util.ConcurrentModificationException at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211) at java.util.TreeMap$EntryIterator.next(TreeMap.java:1247) at java.util.TreeMap$EntryIterator.next(TreeMap.java:1242) at hudson.plugins.throttleconcurrents.ThrottleJobProperty.getThrottledPipelineRunsForCategory(ThrottleJobProperty.java:316) at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.throttleCheckForCategoriesAllNodes(ThrottleQueueTaskDispatcher.java:283) at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRunImpl(ThrottleQueueTaskDispatcher.java:256) at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:218) at hudson.plugins.throttleconcurrents.ThrottleQueueTaskDispatcher.canRun(ThrottleQueueTaskDispatcher.java:176) at hudson.model.Queue.getCauseOfBlockageForItem(Queue.java:1197) at hudson.model.Queue.maintain(Queue.java:1554) at hudson.model.Queue$1.call(Queue.java:325) at hudson.model.Queue$1.call(Queue.java:322) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108) at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112) at java.lang.Thread.run(Thread.java:748) I tried aborting and restarting it multiple times but it consistently behaved the same way; the only way we got it to build again was by commenting out the throttle block entirely. It would be marginally better if it had simply failed the build, but far better if this ConcurrentModificationException were not happening at all. Incidentially, this PR was a switch to use Scripted Pipeline due to JENKINS-45140 , since Declarative Syntax didn't seem to throttle at all.

            People

            • Assignee:
              Unassigned
              Reporter:
              haw777 Christopher Head
            • Votes:
              5 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: