Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59215

gerrit trigger stuck and can not trigger jobs

    Details

    • Similar Issues:

      Description

      After upgrade gerrit trigger to 2.30, sometimes gerrit trigger can not trigger jenkins jobs, but the jenkins account can receive the gerrit stream events at the same time. We find a lot of WARNING logs in jenkins log, 

      The Gerrit incoming events queue contains 6389 items! Something might be stuck, or your system can't process the commands fast enough. Try to increase the number of receiving worker threads. Current thead-pool size: 10.

       

      By the way, we set gerrit-trigger only listening to change-merged and comment-added event types. Even we restart jenkins server, gerrit-trigger seems still stuck.

       

        Attachments

          Issue Links

            Activity

            Hide
            elliliu1984 Li Liu added a comment -

            Exactly the same problem has found on Jenkins 2.176.2 and gerrit trigger plugin 2.30. Downgraded to 2.27.7 helped.

            Show
            elliliu1984 Li Liu added a comment - Exactly the same problem has found on Jenkins 2.176.2 and gerrit trigger plugin 2.30. Downgraded to 2.27.7 helped.
            Hide
            tho74 Thorsten Möllers added a comment - - edited

            I found the same problem and found a way to reproduce. For me it appears when I have a

            • disabled job at jenkins start time
            • configured with dynamic trigger

            If the job is enabled and jenkins start time, I don't see the problem, even if I disable it later. If the job is disabled at start time, the problem occurs and still stays even if I enable the job later. If I remove the dynamic trigger configuration, the problem disappears.

            I can reproduce the problem with 2.29.0 and 2.30.0, didn't see it with 2.27.X

            Show
            tho74 Thorsten Möllers added a comment - - edited I found the same problem and found a way to reproduce. For me it appears when I have a disabled job at jenkins start time configured with dynamic trigger If the job is enabled and jenkins start time, I don't see the problem, even if I disable it later. If the job is disabled at start time, the problem occurs and still stays even if I enable the job later. If I remove the dynamic trigger configuration, the problem disappears. I can reproduce the problem with 2.29.0 and 2.30.0, didn't see it with 2.27.X
            Hide
            sascht Charles Tudor added a comment -

            Ran into the same problem on 2.29 and 2.30.  Went back to 2.28 and things are working as expected.

            Show
            sascht Charles Tudor added a comment - Ran into the same problem on 2.29 and 2.30.  Went back to 2.28 and things are working as expected.
            Hide
            georgbremer Georg Bremer added a comment - - edited

            At least the issue described by having a dynamic trigger job disabled on startup is caused by https://github.com/jenkinsci/gerrit-trigger-plugin/pull/350

            The executors are waiting for the project to be ready, i.e. the dynamic trigger to be loaded. However, the the GerritTriggerTimerTask does a couple of checks (including job.isBuildable()) before updateTriggerConfigURL. This means the latch introduced in that pull request is never released and the executors are blocked forever.

             

            Possible fix: https://github.com/jenkinsci/gerrit-trigger-plugin/pull/403

            Show
            georgbremer Georg Bremer added a comment - - edited At least the issue described by having a dynamic trigger job disabled on startup is caused by  https://github.com/jenkinsci/gerrit-trigger-plugin/pull/350 The executors are waiting for the project to be ready, i.e. the dynamic trigger to be loaded. However, the the GerritTriggerTimerTask does a couple of checks (including job.isBuildable()) before updateTriggerConfigURL. This means the latch introduced in that pull request is never released and the executors are blocked forever.   Possible fix:  https://github.com/jenkinsci/gerrit-trigger-plugin/pull/403
            Hide
            neonman Max P added a comment -

            This is the same as JENKINS-56528?

            Show
            neonman Max P added a comment - This is the same as JENKINS-56528 ?
            Hide
            georgbremer Georg Bremer added a comment -

            From the description (despite being a bit vague about the actual error message) it looks exactly like this issue, it also matches what we have seen.

            Show
            georgbremer Georg Bremer added a comment - From the description (despite being a bit vague about the actual error message) it looks exactly like this issue, it also matches what we have seen.
            Hide
            georgbremer Georg Bremer added a comment -

            Symptoms match JENKINS-56528, closing this as duplicate since the other one is older.

            Show
            georgbremer Georg Bremer added a comment - Symptoms match JENKINS-56528 , closing this as duplicate since the other one is older.

              People

              • Assignee:
                rsandell rsandell
                Reporter:
                stackoverflow BO WANG
              • Votes:
                7 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: