Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51057

EventDispatcher and ConcurrentLinkedQueue ate my JVM

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: sse-gateway-plugin
    • Environment:
    • Similar Issues:

      Description

      We started running out of memory in our JVM (Xmx 8G) and when looking at Melody's memory (heap) histogram (JENKINS_URL/monitoring?part=heaphisto) the top two items were:

       

      Class Size (Kb) % size Instances % instances Source
      org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$Retry 2,890,399 44 92,492,793 43  
      java.util.concurrent.ConcurrentLinkedQueue$Node 2,167,981 33 92,500,553 43  

       
      77% (and growing as we were researching the problem) of the memory was being used by these two items.

      I have two support bundles from this time and an .hprof as well.

      I can either screen share with someone or if you can tell me how to analyze these files I would be happy to.

        Attachments

          Issue Links

            Activity

            Hide
            danmordechai Dan Mordechai added a comment - - edited

            Álvaro Iradier Thank you for this script! I will definitely try it as we are facing the same issue on our 2.107.2 Jenkins.

            Some things to point out:

            1. I had to change the line 
              def dispatcher = channelSubscriber.this$0
              

              to

              def dispatcher = channelSubscriber.eventDispatcher
              

                     2. Your script will only work for SSE gatway plugin version > 1.15 as the SSEChannelSubscriber inner class in EventDispatcher.java will not have the eventDispatcher member any more.

            Show
            danmordechai Dan Mordechai added a comment - - edited Álvaro Iradier  Thank you for this script! I will definitely try it as we are facing the same issue on our 2.107.2 Jenkins. Some things to point out: I had to change the line  def dispatcher = channelSubscriber. this $0 to def dispatcher = channelSubscriber.eventDispatcher          2. Your script will only work for SSE gatway plugin version > 1.15 as the SSEChannelSubscriber inner class in EventDispatcher.java will not have the eventDispatcher member any more.
            Hide
            raihaan Raihaan Shouhell added a comment -

            A fix got merged to master that definitely helped our masters

            Show
            raihaan Raihaan Shouhell added a comment - A fix got merged to master that definitely helped our masters
            Show
            olamy Olivier Lamy added a comment - fixed with  https://github.com/jenkinsci/sse-gateway-plugin/commit/9cfe9d9c4d9e284ee7d099d4e734b3677ee677f9
            Hide
            andreasgk Andreas Galek added a comment -

            Having installed the newest plugin's version 1.19 I was facing error messages like this:

             
            Aug 09, 2019 6:36:46 PM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
            SEVERE: A thread (EventDispatcher.retryProcessor/37063) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
            java.lang.OutOfMemoryError: unable to create new native thread
                at java.lang.Thread.start0(Native Method)
                at java.lang.Thread.start(Thread.java:717)
                at java.util.Timer.<init>(Timer.java:160)
                at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing(EventDispatcher.java:296)
                at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.processRetries(EventDispatcher.java:437)
                at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$1.run(EventDispatcher.java:299)
                at java.util.TimerThread.mainLoop(Timer.java:555)
                at java.util.TimerThread.run(Timer.java:505)
             
             
            Is it the same problem as the one you fixed in this Jira ticket?
             

            Show
            andreasgk Andreas Galek added a comment - Having installed the newest plugin's version 1.19 I was facing error messages like this:   Aug 09, 2019 6:36:46 PM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException SEVERE: A thread (EventDispatcher.retryProcessor/37063) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code. java.lang.OutOfMemoryError: unable to create new native thread     at java.lang.Thread.start0(Native Method)     at java.lang.Thread.start(Thread.java:717)     at java.util.Timer.<init>(Timer.java:160)     at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.scheduleRetryQueueProcessing(EventDispatcher.java:296)     at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher.processRetries(EventDispatcher.java:437)     at org.jenkinsci.plugins.ssegateway.sse.EventDispatcher$1.run(EventDispatcher.java:299)     at java.util.TimerThread.mainLoop(Timer.java:555)     at java.util.TimerThread.run(Timer.java:505)     Is it the same problem as the one you fixed in this Jira ticket?  
            Hide
            olamy Olivier Lamy added a comment -

            Andreas Galek side effect bug of the fix  please read here https://issues.jenkins-ci.org/browse/JENKINS-58684

            Show
            olamy Olivier Lamy added a comment - Andreas Galek side effect bug of the fix   please read here  https://issues.jenkins-ci.org/browse/JENKINS-58684

              People

              • Assignee:
                olamy Olivier Lamy
                Reporter:
                docwhat Christian Höltje
              • Votes:
                16 Vote for this issue
                Watchers:
                24 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: