Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54603

Memory leak in remoting causes Jenkins to crash

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Not A Defect
    • Component/s: remoting
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      Some of our jobs rely on an external slave. This has been working for a while without any issues. Recently, the number of jobs that run daily has been increased. This is when our problems started. After a while Jenkins has consumed all the memory available to the VM, and it locks up as a result.

      The log is full of this exception:

      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: SEVERE: This command is created here
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: Nov 12, 2018 5:03:53 PM hudson.remoting.Channel$1 handle
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: SEVERE: Failed to execute command Pipe.Flush(-1) (channel PLTSTSRV001)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: java.util.concurrent.ExecutionException: Invalid object ID -1 iota=1723
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.ExportTable.get(ExportTable.java:397)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.Channel.getExportedObject(Channel.java:780)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.ProxyOutputStream$Flush.execute(ProxyOutputStream.java:307)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.Channel$1.handle(Channel.java:565)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:85)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: Caused by: java.lang.Exception: Object appears to be deallocated at lease before Mon Nov 12 16:39:51 CET 2018
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:474)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: ... 5 more

      I think this is what's causing Jenkins to leak.

        Attachments

        1. heap-histogram.txt
          15 kB
        2. Jenkins Job Output
          27 kB
        3. Pipeline.txt
          7 kB

          Issue Links

            Activity

            Hide
            smokeythebandit Benjamin Martens added a comment - - edited

            Hey guys thank you for your response! I've upgraded the 'Pipeline: API' plugin from version 2.32 to 2.33. Usually the server crashes within 24 hours of its last restart, I will keep monitoring it and see if the issue is resolved.

            Jeff Thompson It is the master that runs out of memory. I cannot really pin down an event that is causing it to run out of memory. I did notice that when I increased the memory allocated to the VM from 1024MB to 2048MB it took longer for the server to crash, confirming that its probably a memory leak.

            I've attached the log of one of the jobs. I had to remove some of the output because it contains sensitive information. The information I removed is generated by a python script that runs automated tests for our software.

            Included are the pipeline script and the output it generated for the job that I suspect is causing this issue.

            Jenkins Job Output

            Pipeline.txt

            Show
            smokeythebandit Benjamin Martens added a comment - - edited Hey guys thank you for your response! I've upgraded the 'Pipeline: API' plugin from version 2.32 to 2.33. Usually the server crashes within 24 hours of its last restart, I will keep monitoring it and see if the issue is resolved. Jeff Thompson It is the master that runs out of memory. I cannot really pin down an event that is causing it to run out of memory. I did notice that when I increased the memory allocated to the VM from 1024MB to 2048MB it took longer for the server to crash, confirming that its probably a memory leak. I've attached the log of one of the jobs. I had to remove some of the output because it contains sensitive information. The information I removed is generated by a python script that runs automated tests for our software. Included are the pipeline script and the output it generated for the job that I suspect is causing this issue. Jenkins Job Output Pipeline.txt
            Hide
            jglick Jesse Glick added a comment -

            Benjamin Martens the build log is unlikely to be useful in diagnosing a memory leak. The bare minimum would be a heap histogram. This is most easily collected by installing the Support Core plugin, then selecting the Master Heap Histogram diagnostic when getting a Support bundle. You can attach that individual nodes/master/heap-histogram.txt, or send the bundle to one of us privately, or select the system option to anonymize support bundles and then attach the whole bundle here (always best to give the contents a manual review to look for sensitive information).

            Show
            jglick Jesse Glick added a comment - Benjamin Martens the build log is unlikely to be useful in diagnosing a memory leak. The bare minimum would be a heap histogram. This is most easily collected by installing the Support Core plugin, then selecting the Master Heap Histogram diagnostic when getting a Support bundle. You can attach that individual nodes/master/heap-histogram.txt , or send the bundle to one of us privately, or select the system option to anonymize support bundles and then attach the whole bundle here (always best to give the contents a manual review to look for sensitive information).
            Hide
            smokeythebandit Benjamin Martens added a comment - - edited

            http://example.com/I installed this plugin and ran a couple of jobs. I'm not sure how to interpret the heap histogram, but I'm starting to suspect this plugin: https://wiki.jenkins.io/display/JENKINS/Test+Results+Analyzer+Plugin

            Anyway, here is the heap histogram.

            heap-histogram.txt

            Show
            smokeythebandit Benjamin Martens added a comment - - edited http://example.com/ I installed this plugin and ran a couple of jobs. I'm not sure how to interpret the heap histogram, but I'm starting to suspect this plugin:  https://wiki.jenkins.io/display/JENKINS/Test+Results+Analyzer+Plugin Anyway, here is the heap histogram. heap-histogram.txt
            Hide
            jglick Jesse Glick added a comment -

            Indeed. If this plugin is not critical for your workflow, try disabling it for a while.

            Show
            jglick Jesse Glick added a comment - Indeed. If this plugin is not critical for your workflow, try disabling it for a while.
            Hide
            smokeythebandit Benjamin Martens added a comment -

            It is important for some developers in our company. I think it's best that they open a new ticket and figure it out with the maintainer of the plugin.

            Jeff Thompson Jesse Glick Thank you for your help!

             

            Show
            smokeythebandit Benjamin Martens added a comment - It is important for some developers in our company. I think it's best that they open a new ticket and figure it out with the maintainer of the plugin. Jeff Thompson Jesse Glick Thank you for your help!  

              People

              • Assignee:
                smokeythebandit Benjamin Martens
                Reporter:
                smokeythebandit Benjamin Martens
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: