Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21034

Jenkins Startup Deadlock - QueueSorter.installDefaultQueueSorter and Queue.init

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core, prioritysorter-plugin
    • Labels:
      None
    • Environment:
      Jenkins 1.544 / Jenkins Priority Sorter Plugin 2.2
    • Similar Issues:

      Description

      Found one Java-level deadlock:
      =============================
      "Handling GET /jenkins/view/Portal/ : RequestHandlerThread[#12]":
        waiting to lock monitor 0x000000001a449d08 (object 0x0000000419d757e8, a hudson.ExtensionList$Lock),
        which is held by "QueueSorter.installDefaultQueueSorter"
      "QueueSorter.installDefaultQueueSorter":
        waiting to lock monitor 0x000000001ae56c48 (object 0x000000041895e0f0, a hudson.model.Queue),
        which is held by "Queue.init"
      "Queue.init":
        waiting to lock monitor 0x000000001a449d08 (object 0x0000000419d757e8, a hudson.ExtensionList$Lock),
        which is held by "QueueSorter.installDefaultQueueSorter"
      

        Attachments

          Activity

          Hide
          rvanoo Rob van Oostrum added a comment - - edited

          Ran into this as well. Seems unrelated to the upgrade. It's simply that the queue that's persisted when the Jenkins service shuts down causes a deadlock when Jenkins re-initializes. The quick fix for me was to rename the file (this is on Ubuntu) from queue.xml into something like queue.xml.keep (or you could just remove it altogether).

          I've had this happen both at Jenkins upgrades, and simply restarting Jenkins after plugin updates, since 1.542 or so. Once in this deadlock, downgrading as far as 1.540 didn't do the trick. Ended up upgrading back to 1.544 and removing queue.xml before restarting.

          Show
          rvanoo Rob van Oostrum added a comment - - edited Ran into this as well. Seems unrelated to the upgrade. It's simply that the queue that's persisted when the Jenkins service shuts down causes a deadlock when Jenkins re-initializes. The quick fix for me was to rename the file (this is on Ubuntu) from queue.xml into something like queue.xml.keep (or you could just remove it altogether). I've had this happen both at Jenkins upgrades, and simply restarting Jenkins after plugin updates, since 1.542 or so. Once in this deadlock, downgrading as far as 1.540 didn't do the trick. Ended up upgrading back to 1.544 and removing queue.xml before restarting.
          Hide
          emsa23 Magnus Sandberg added a comment -

          Does this happen on a "normal" restart or only directly after the upgrade?

          Does this only happen after a upgrade of Priority Sorter or in other situations as well?

          Did you have a huge queue at restart or does this happen with any queue size?

          Show
          emsa23 Magnus Sandberg added a comment - Does this happen on a "normal" restart or only directly after the upgrade? Does this only happen after a upgrade of Priority Sorter or in other situations as well? Did you have a huge queue at restart or does this happen with any queue size?
          Hide
          rvanoo Rob van Oostrum added a comment -

          @Magnus:

          • I've had this happen after plugin upgrades as well. Yesterday I restarted after upgrading the JDK on Jenkins' server and it deadlocked. Once it deadlocks, restarting again just results in another deadlock.
          • see previous answer
          • I believe my queue had only 2 items in it when I restarted yesterday. It may have been a couple more than that, but not a huge number by any stretch.
          Show
          rvanoo Rob van Oostrum added a comment - @Magnus: I've had this happen after plugin upgrades as well. Yesterday I restarted after upgrading the JDK on Jenkins' server and it deadlocked. Once it deadlocks, restarting again just results in another deadlock. see previous answer I believe my queue had only 2 items in it when I restarted yesterday. It may have been a couple more than that, but not a huge number by any stretch.
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          In my opinion, the code needs some correction even if it works properly in previous Jenkins versions.
          Queue.init() initializes QueueSorter extensions, so it is quite strange to retrieve data from it at this stage...

          I see the following fix:

          • Remove queue handling logic from AdvancedQueueSorter constructor
          • Move it to a separate method an call on the first sortBuildableItems() call
          • It also makes sense to call this method for new mode only (just don't do anything for legacy)
          Show
          oleg_nenashev Oleg Nenashev added a comment - In my opinion, the code needs some correction even if it works properly in previous Jenkins versions. Queue.init() initializes QueueSorter extensions, so it is quite strange to retrieve data from it at this stage... I see the following fix: Remove queue handling logic from AdvancedQueueSorter constructor Move it to a separate method an call on the first sortBuildableItems() call It also makes sense to call this method for new mode only (just don't do anything for legacy)
          Hide
          emsa23 Magnus Sandberg added a comment -

          Yes this is actually already fixed for the 2.3 where the init is done with Initializer (after.JOB_LOADED)(I though that was already in 2.2 but is is not)

          https://github.com/emsa23/priority-sorter-plugin/commit/a2447a7e371869cb4ada8ed76a7114b3eb165ee8

          @Oleg do you think that fix will do it?

          Show
          emsa23 Magnus Sandberg added a comment - Yes this is actually already fixed for the 2.3 where the init is done with Initializer (after.JOB_LOADED)(I though that was already in 2.2 but is is not) https://github.com/emsa23/priority-sorter-plugin/commit/a2447a7e371869cb4ada8ed76a7114b3eb165ee8 @Oleg do you think that fix will do it?
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Most probably, it will help with the issue. BTW, any test cases will be appreciated

          Is it guaranteed that initializer's method call will be finished before starting of scheduling?
          If no, it makes sense to somehow synchronize methods.

          Show
          oleg_nenashev Oleg Nenashev added a comment - Most probably, it will help with the issue. BTW, any test cases will be appreciated Is it guaranteed that initializer's method call will be finished before starting of scheduling? If no, it makes sense to somehow synchronize methods.
          Hide
          emsa23 Magnus Sandberg added a comment -

          I've added some test cases in the 2.3 pull - testing the queue behaviour is not that easy though.

          Regarding the order I'm not sure - one option could be to use before=COMPLETED rather than after=JOB_LOADED but I'm not sure.

          I have restarted Jenkins with items in the queue like a gazillion times and I have never seen this so it is a bit tricky for me to to verify this.

          I reckon that it cannot be wore than it is so I'm considering releasing 2.3 with the current 2.3 pull (https://github.com/jenkinsci/priority-sorter-plugin/pull/10)

          Show
          emsa23 Magnus Sandberg added a comment - I've added some test cases in the 2.3 pull - testing the queue behaviour is not that easy though. Regarding the order I'm not sure - one option could be to use before=COMPLETED rather than after=JOB_LOADED but I'm not sure. I have restarted Jenkins with items in the queue like a gazillion times and I have never seen this so it is a bit tricky for me to to verify this. I reckon that it cannot be wore than it is so I'm considering releasing 2.3 with the current 2.3 pull ( https://github.com/jenkinsci/priority-sorter-plugin/pull/10 )
          Hide
          emsa23 Magnus Sandberg added a comment -

          Hopefully fixed in 2.3

          The startup sequence is different from 2.2 so if this still occurs please provide new log-file.

          Show
          emsa23 Magnus Sandberg added a comment - Hopefully fixed in 2.3 The startup sequence is different from 2.2 so if this still occurs please provide new log-file.

            People

            • Assignee:
              emsa23 Magnus Sandberg
              Reporter:
              gcummings Geoff Cummings
            • Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: