Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52636

Gerrit triggered jobs getting delayed

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: gerrit-trigger-plugin
    • Environment:
    • Similar Issues:

      Description

      We are experiencing a delay in Gerrit triggered Jobs in our Jenkins Jobs.

      attached is the stack trace of the blocking threads.

      Jul 18, 2018 5:07:57 PM com.sonymobile.tools.gerrit.gerritevents.GerritHandler checkQueueSize
      WARNING: The Gerrit incoming events queue contains 28095 items! Something might be stuck, or your system can't process the commands fast enough. Try to increase the number of receiving worker threads. Current thread-pool size: 30

       

      Jul 18, 2018 6:54:37 PM com.sonymobile.tools.gerrit.gerritevents.GerritJsonEventFactory getEvent
      FINE: Constructor with JSONObject as parameter missing, trying default constructor.
      java.lang.NoSuchMethodException: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated.<init>(net.sf.json.JSONObject)
      at java.lang.Class.getConstructor0(Class.java:3082)
      at java.lang.Class.getConstructor(Class.java:1825)
      at com.sonymobile.tools.gerrit.gerritevents.GerritJsonEventFactory.getEvent(GerritJsonEventFactory.java:69)
      at com.sonymobile.tools.gerrit.gerritevents.workers.AbstractJsonObjectWork.perform(AbstractJsonObjectWork.java:69)
      at com.sonymobile.tools.gerrit.gerritevents.workers.StreamEventsStringWork.perform(StreamEventsStringWork.java:67)
      at com.sonymobile.tools.gerrit.gerritevents.workers.EventThread.run(EventThread.java:66)
      at com.sonyericsson.hudson.plugins.gerrit.trigger.SystemEventThread.run(SystemEventThread.java:66)

       

       

       

        Attachments

          Activity

          Hide
          dsoprea Dustin Oprea added a comment -

          Any status on this? There's a chance that my team just encountered this.

           

          Show
          dsoprea Dustin Oprea added a comment - Any status on this? There's a chance that my team just encountered this.  
          Hide
          darrien Darrien Glasser added a comment -

          It doesn't look like there's been any action on the PR yet although it mostly looks good to go from here.

          In the meantime, I have a patched version we're using at the company I'm at. You're free to use while we wait: https://github.com/DarrienG/gerrit-trigger-plugin/releases/tag/2.31.0-uninterested

          It's basically HEAD from the official repo + ignores irrelevant events. We have probably 3000+ builds a day and haven't seen any issues with it while we wait. Jenkins was unusable for us otherwise.

          Show
          darrien Darrien Glasser added a comment - It doesn't look like there's been any action on the PR yet although it mostly looks good to go from here. In the meantime, I have a patched version we're using at the company I'm at. You're free to use while we wait: https://github.com/DarrienG/gerrit-trigger-plugin/releases/tag/2.31.0-uninterested It's basically HEAD from the official repo + ignores irrelevant events. We have probably 3000+ builds a day and haven't seen any issues with it while we wait. Jenkins was unusable for us otherwise.
          Hide
          dsoprea Dustin Oprea added a comment - - edited

          We tried bumping to 2.29.0 during a system upgrade, and the queue started wildly accumulating without actually starting any builds. We reverted the plugin to 2.27.5 and things appear to be rolling again.

          Show
          dsoprea Dustin Oprea added a comment - - edited We tried bumping to 2.29.0 during a system upgrade, and the queue started wildly accumulating without actually starting any builds. We reverted the plugin to 2.27.5 and things appear to be rolling again.
          Hide
          lemonjia Jia Jia added a comment -

          Any status on this? There's also a chance that my team just encountered this recently.

          Show
          lemonjia Jia Jia added a comment - Any status on this? There's also a chance that my team just encountered this recently.
          Hide
          ezcorch Christoffer Cortes Sjöwall added a comment -

          Recently 2.30.0 was released that include two changes meant to reduce queue load. One reduces the disk writing when playback is enabled and the other gives you the ability to filter out unnecessary gerrit messages from the main settings panel under advanced. Some may still be experiencing delays and queue build ups though.

          Show
          ezcorch Christoffer Cortes Sjöwall added a comment - Recently 2.30.0 was released that include two changes meant to reduce queue load. One reduces the disk writing when playback is enabled and the other gives you the ability to filter out unnecessary gerrit messages from the main settings panel under advanced. Some may still be experiencing delays and queue build ups though.

            People

            • Assignee:
              scoheb Scott Hebert
              Reporter:
              eattsma Amit Sharma
            • Votes:
              12 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

              • Created:
                Updated: