Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65771

Memory leak with gerrit-trigger plugin installed

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • gerrit-trigger-plugin
    • None
    • Jenkins LTS 2.277.2
      gerrit-trigger version 2.34.0 (latest)
    • 2.35.0-beta-1

      Since upgrading a large number (100s) of Jenkins instances to LTS v2.277.2, and the associated plugin updates required for the UI changes, I have observed a number of these instances are leak memory, and in one case it only lasts about 36 hours before exhausting the heap (in that case it is 6GB).

      a heapdump (e.g. jmap -dump:live,file=jenkins-20210603-0939.hprof) showed that there were a large number (1000s) of hudson.remoting.Channel classes retained in a list by the class org.jvnet.hudson.test.ChannelShutdownListener. Doing a grep in the installed plugins folder, the only plugin that referenced this class was the gerrit-trigger plugin:

      plugins$ grep -iRH "org.jvnet.hudson.test.ChannelShutdownListener" *
      Binary file gerrit-trigger/WEB-INF/lib/jenkins-test-harness-2.56.jar matches
       
      I found it curious that a test class was being included in a production plugin, so I do not know why this jenkins-test-harness is bundled into the plugin.

      Uninstalling (or disabling) the gerrit-trigger plugin (and in some cases the sonar-gerrit plugin as well, which has gerrit-trigger as a dependency) and restarting caused this ChannelShutdownListener to no longer be loaded (as evidenced by the output of jmap -histo:live <pid> | grep org.jvnet.hudson.test.ChannelShutdownListener), and the memory profile has remained flat over the past 24 hours.

      I have attached a screenshot of the leak suspect from MAT https://www.eclipse.org/mat/
       
      The jenkins-test-harness is pulled in here in the pom: https://github.com/jenkinsci/gerrit-trigger-plugin/blob/master/pom.xml#L280-L283, which was brought in with this commit in February: https://github.com/jenkinsci/gerrit-trigger-plugin/commit/04fd6aff727c0104344c91c72dda372c23cac4ba

      The release notes suggest that 2.34.0 was the release that brought in in the change as part of https://issues.jenkins.io/browse/JENKINS-53250

      I don't understand the interaction of this test-harness library and the running system, but it seems to be injecting itself in and causing the leak. If it's just a test harness it should probably be a dev dependency and not included in the main plugin, perhaps it came in by mistake?
       
      Note: It does not appear to matter whether the gerrit plugin in used or not (and in the worst case I mentioned before, it wasn't used at all), so we are working to uninstall it where it isn't needed.

        1. jenkins_heap_dump_leak_suspects.png
          90 kB
          Radoslaw Moszczynski
        2. leak-suspects-from-mat.png
          147 kB
          Andy Taylor

            rsandell rsandell
            fraz3alpha Andy Taylor
            Votes:
            3 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: