Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29664

Jenkins java heap full of Strings -> Running out of Memory on Large Jenkins Deploy

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: perforce-plugin
    • Environment:
    • Similar Issues:

      Description

      We run a Jenkins master server with over 3500 jobs defined. We've been running an 8GB heap size for some time, we average arorund 4-6GB used and collections tend to happen reasonably. We recently updated to Jenkins 1.618 (from 1.594) and in the past week our master server has hung in garbage collection twice. Oddly upon restart, it will sometimes immediately hang again.

      We took a heap dump of todays failure to see if we had a reference leak or misbehaving plugin and couldn't find anything obvious. However, we have a huge amount of memory allocated to retaining strings. Java.lang.string/char consume %70 of our memory. While many of the strings seem to be job descriptions, recent changelists etc (things found on the job details page) there is a huge number of strings dedicated to what looks like errors and stack traces I'd normally expect to find in a log (see the attached file for a few examples associated with: org.apache.commons.jelly.JellyTagException).

      I'm not sure if this is normal behavior or not (never analyzed these part of Jenkins before). We're still trying to figure out what retains these strings. Given that we think this may be "normal" we've bumped our heap to 12GB's to see if we've just gotten so big we need more memory.

      I've also attached the monitoring graphs (as images) of relevant monitors. I can, if it's useful, upload the 8GB heap dump somewhere.

        Attachments

        1. jenkins.st.4
          1.05 MB
        2. jenkinserr.txt
          14 kB
        3. p4changelist.png
          p4changelist.png
          64 kB
        4. unnamed.png
          unnamed.png
          142 kB
        5. usedMemory.png
          usedMemory.png
          43 kB
        6. usedMemory1week.png
          usedMemory1week.png
          31 kB
        7. usedPhysicalMemorySize.png
          usedPhysicalMemorySize.png
          25 kB
        8. usedPhysicalMemorySize1week.png
          usedPhysicalMemorySize1week.png
          26 kB

          Activity

          Hide
          maxfields2000 Maxfield Stewart added a comment -

          Adding more details as we dig deeper. We've attributed a huge (almost 3GB's) amount of strings to the Perforce plugin, in particular a consistent set of repeating strings. This error repeated 326K per object, there are 23 objects.

          "//depot/lol/Releases/5.14/code/External/sdks/macosx/10.7/System/Library/PrivateFrameworks/GraphicsAp Warning! Only first 100 elements are retrieved.Please increase the Maximum number of content display from View->Options Menu to view more data"

          I've attached a shot from IBM Heap Analyzer.

          Show
          maxfields2000 Maxfield Stewart added a comment - Adding more details as we dig deeper. We've attributed a huge (almost 3GB's) amount of strings to the Perforce plugin, in particular a consistent set of repeating strings. This error repeated 326K per object, there are 23 objects. "//depot/lol/Releases/5.14/code/External/sdks/macosx/10.7/System/Library/PrivateFrameworks/GraphicsAp Warning! Only first 100 elements are retrieved.Please increase the Maximum number of content display from View->Options Menu to view more data" I've attached a shot from IBM Heap Analyzer.
          Hide
          maxfields2000 Maxfield Stewart added a comment -

          Also attaching a stack trace of threads, there are several request handlers in BLOCKED state. Hard to tell if this is a symptom or a cause.

          Show
          maxfields2000 Maxfield Stewart added a comment - Also attaching a stack trace of threads, there are several request handlers in BLOCKED state. Hard to tell if this is a symptom or a cause.
          Hide
          maxfields2000 Maxfield Stewart added a comment -

          We believe we found the issue, you should be able to close this ticket.

          On our jenkins instance we have a particular job type that runs 102 executors. This problem is intermittant because that job uses the P4 plugin and can pick up particularly large changsets. In this case, we had a changeset that was 75 MB's. Over 102 executors (each job has it's own copy of the changeset) that needs approximately 6.5 GB's of RAM. When this happens, we run out of memory in Jenkins (configured for an 6GB heap).

          It would be nice if the Perforce plugin didn't keep the entire change set in memory, but for now, we've bumped our heap size to accomodate while we consider our options (we actually plan to run far more executors than this).

          Show
          maxfields2000 Maxfield Stewart added a comment - We believe we found the issue, you should be able to close this ticket. On our jenkins instance we have a particular job type that runs 102 executors. This problem is intermittant because that job uses the P4 plugin and can pick up particularly large changsets. In this case, we had a changeset that was 75 MB's. Over 102 executors (each job has it's own copy of the changeset) that needs approximately 6.5 GB's of RAM. When this happens, we run out of memory in Jenkins (configured for an 6GB heap). It would be nice if the Perforce plugin didn't keep the entire change set in memory, but for now, we've bumped our heap size to accomodate while we consider our options (we actually plan to run far more executors than this).
          Hide
          danielbeck Daniel Beck added a comment -

          Assigning to P4 to get some feedback from Paul. Maybe he considers this to be a bug?

          Show
          danielbeck Daniel Beck added a comment - Assigning to P4 to get some feedback from Paul. Maybe he considers this to be a bug?
          Hide
          maxfields2000 Maxfield Stewart added a comment -

          I don't think it's a bug. There's a feature in the P4 plugin, one of the advanced options is to select a file limit, which is how we are resolving this issue. These change sets have 326,000 files in them, given that we don't need the file list feature on the jenkins details pages, we just set that to like, 50, and we stop consuming so much memory maintaining these lists.

          What is interesting is that there are two objects that have the same changeset string according to my heap analyzer and that means you consume twice the memory to host the list:

          • java.util.StringTokenizer
          • java.lang.StringBuilder

          Are two objects held by the "hudson.modelExecutor" object. The Tokenizer has a java.lang.String object, and the String Builder obviously has a char[] array of the string.

          Show
          maxfields2000 Maxfield Stewart added a comment - I don't think it's a bug. There's a feature in the P4 plugin, one of the advanced options is to select a file limit, which is how we are resolving this issue. These change sets have 326,000 files in them, given that we don't need the file list feature on the jenkins details pages, we just set that to like, 50, and we stop consuming so much memory maintaining these lists. What is interesting is that there are two objects that have the same changeset string according to my heap analyzer and that means you consume twice the memory to host the list: java.util.StringTokenizer java.lang.StringBuilder Are two objects held by the "hudson.modelExecutor" object. The Tokenizer has a java.lang.String object, and the String Builder obviously has a char[] array of the string.
          Hide
          p4paul Paul Allen added a comment -

          Looking at your plugin list only the `perforce` plugin is listed not the `p4` plugin. Is this true? If so please reassign.

          Daniel Beck

          Show
          p4paul Paul Allen added a comment - Looking at your plugin list only the `perforce` plugin is listed not the `p4` plugin. Is this true? If so please reassign. Daniel Beck
          Hide
          danielbeck Daniel Beck added a comment -

          Sorry about that Paul Allen, didn't see the plugins list, only the user comments referring to P4 Plugin…

          Show
          danielbeck Daniel Beck added a comment - Sorry about that Paul Allen , didn't see the plugins list, only the user comments referring to P4 Plugin…

            People

            • Assignee:
              Unassigned
              Reporter:
              maxfields2000 Maxfield Stewart
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: