Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38914

hudson.model.TextParameterValue heavily bloats memory use and size of build.xml by loading all builds TextParameterValue in memorey

    Details

    • Similar Issues:

      Description

      Hi,

      Recently we upgraded from 1.596.3 to 1.651.3 and observing that the later version is consuming lot of heap as soon as it comes up.

      WE have got many jobs with huge build history.. in terms of 4000 builds. Each builds's build.xml is in terms of 500k.

      Whenever the instance is restarted the heap starts to grow and finally the instance crashed with OEM.

      The heap dump shows that values defined under hudson.model.TextParameterValue of each build is loaded into memory!

      Refer screenshots: heap-screenshot-0.png & heap-screenshot-1.png

      Here is the contents of build.xml of one such build. refer screenshot build.xml-screenshot.png

      Is it this commit SECURITY-170 Store initial parameters list for later use which is loading all values?

        Attachments

        1. build.xml-screenshot.png
          37 kB
          Dilip Mahadevappa
        2. heap-screenshot-0.png
          48 kB
          Dilip Mahadevappa
        3. heap-screenshot-1.png
          243 kB
          Dilip Mahadevappa
        4. lazy_BuildReference.MODE.JPG
          116 kB
          Dilip Mahadevappa
        5. lazy_BuildReference.MODE-2.JPG
          52 kB
          Dilip Mahadevappa

          Activity

          Hide
          danielbeck Daniel Beck added a comment -

          Could you clarify, are you using extremely long parameter values for your builds?

          Show
          danielbeck Daniel Beck added a comment - Could you clarify, are you using extremely long parameter values for your builds?
          Hide
          dilipm79 Dilip Mahadevappa added a comment -

          Daniel Beckham Thanks for checking. We did some more investigation and here are our findings.

          We have a got internal Test Framework which is kicked in with every builds and it is generating large amount of Environment variables. One of the Environment variable is having a large value as seen in heap-screenshot-1.png ;

          We have got some 3k jobs like this with each build.xml ~ 500k.

          Noting that there is a major code changes between 1.596.3 and 1.609.1, we did some testing by setting jenkins.model.lazy.BuildReference.MODE to none and weak.

          With Core 1.651.3
          lazy_BuildReference.MODE.JPG

          With core 2.19.1 ( jenkins.model.lazy.BuildReference.MODE unset)
          lazy_BuildReference.MODE-2.JPG

          As we see in the graph, there is significant amount of GC with jenkins.model.lazy.BuildReference.MODE set to weak in core 1.651.3

          Looks like core 2.19.1 with default jenkins.model.lazy.BuildReference.MODE (i.e. soft) does perform well with huge build history.

          Any insights in very much appreciated.

          Show
          dilipm79 Dilip Mahadevappa added a comment - Daniel Beckham Thanks for checking. We did some more investigation and here are our findings. We have a got internal Test Framework which is kicked in with every builds and it is generating large amount of Environment variables. One of the Environment variable is having a large value as seen in heap-screenshot-1.png ; We have got some 3k jobs like this with each build.xml ~ 500k. Noting that there is a major code changes between 1.596.3 and 1.609.1, we did some testing by setting jenkins.model.lazy.BuildReference.MODE to none and weak . With Core 1.651.3 lazy_BuildReference.MODE.JPG With core 2.19.1 ( jenkins.model.lazy.BuildReference.MODE unset) lazy_BuildReference.MODE-2.JPG As we see in the graph, there is significant amount of GC with jenkins.model.lazy.BuildReference.MODE set to weak in core 1.651.3 Looks like core 2.19.1 with default jenkins.model.lazy.BuildReference.MODE (i.e. soft) does perform well with huge build history. Any insights in very much appreciated.
          Hide
          danielbeck Daniel Beck added a comment -

          1.633 introduced build history pagination and search (JENKINS-26554), but a bug resulted in all build records for a project getting loaded when the widget was shown. That was only fixed in 2.17.

          This issue needs to be reproduced on current Jenkins releases to be relevant.

          Show
          danielbeck Daniel Beck added a comment - 1.633 introduced build history pagination and search ( JENKINS-26554 ), but a bug resulted in all build records for a project getting loaded when the widget was shown. That was only fixed in 2.17. This issue needs to be reproduced on current Jenkins releases to be relevant.

            People

            • Assignee:
              Unassigned
              Reporter:
              dilipm79 Dilip Mahadevappa
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: