Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54603

Memory leak in remoting causes Jenkins to crash

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Not A Defect
    • Component/s: remoting
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      Some of our jobs rely on an external slave. This has been working for a while without any issues. Recently, the number of jobs that run daily has been increased. This is when our problems started. After a while Jenkins has consumed all the memory available to the VM, and it locks up as a result.

      The log is full of this exception:

      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: SEVERE: This command is created here
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: Nov 12, 2018 5:03:53 PM hudson.remoting.Channel$1 handle
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: SEVERE: Failed to execute command Pipe.Flush(-1) (channel PLTSTSRV001)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: java.util.concurrent.ExecutionException: Invalid object ID -1 iota=1723
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.ExportTable.get(ExportTable.java:397)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.Channel.getExportedObject(Channel.java:780)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.ProxyOutputStream$Flush.execute(ProxyOutputStream.java:307)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.Channel$1.handle(Channel.java:565)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:85)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: Caused by: java.lang.Exception: Object appears to be deallocated at lease before Mon Nov 12 16:39:51 CET 2018
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:474)
      Nov 12 17:03:53 mfa-sp.microflown.loc jenkins[46451]: ... 5 more

      I think this is what's causing Jenkins to leak.

        Attachments

        1. heap-histogram.txt
          15 kB
        2. Jenkins Job Output
          27 kB
        3. Pipeline.txt
          7 kB

          Issue Links

            Activity

            smokeythebandit Benjamin Martens created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Link This issue relates to JENKINS-54566 [ JENKINS-54566 ]
            smokeythebandit Benjamin Martens made changes -
            Attachment Jenkins Job Output [ 45217 ]
            smokeythebandit Benjamin Martens made changes -
            Attachment Jenkins Job Output [ 45218 ]
            smokeythebandit Benjamin Martens made changes -
            Attachment Pipeline.txt [ 45219 ]
            smokeythebandit Benjamin Martens made changes -
            Attachment Jenkins Job Output [ 45218 ]
            smokeythebandit Benjamin Martens made changes -
            Attachment Jenkins Job Output [ 45217 ]
            smokeythebandit Benjamin Martens made changes -
            Attachment Jenkins Job Output [ 45220 ]
            smokeythebandit Benjamin Martens made changes -
            Attachment heap-histogram.txt [ 45225 ]
            smokeythebandit Benjamin Martens made changes -
            Assignee Jeff Thompson [ jthompson ] Benjamin Martens [ smokeythebandit ]
            smokeythebandit Benjamin Martens made changes -
            Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
            Resolution Fixed [ 1 ]
            smokeythebandit Benjamin Martens made changes -
            Status Fixed but Unreleased [ 10203 ] Closed [ 6 ]
            Resolution Fixed [ 1 ] Not A Defect [ 7 ]

              People

              • Assignee:
                smokeythebandit Benjamin Martens
                Reporter:
                smokeythebandit Benjamin Martens
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: