Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19022

GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Hello everyone.

      Months ago, we've noticed a bug/issue with the GIT plug-in. Previously, it was only a minor nuisance but now, it causes each build that we start to use up ~3MB of main memory and ~5MB of disk space in the build.xml.

      The issue is due to the following behaviour of the GIT plug-in:
      For every build that has the GIT SCM defined, it retrieves the list of branches in the remote repository. For each branch, it retrieves the last build in Jenkins that was run against this branch.

      This information is then stored in the Build object in form of the "BuildData" field. This means, that the full list of all branches, plus their last builds is stored in each and every build – thus using up main memory and using up disk space in the "build.xml" file allocated for the build.

      It uses this information to populate a page for the build with the association of branches to builds:
      http://<SERVER>/job/<JOBNAME>/<BUILD-ID>/git/?

      For normal repositories, this data is relatively small, as only a limited number of unmerged branches exist. Unfortunately, we use GIT in an automated manner, where thousands of tags and branches are spawned without merging back into the mainline.

      This means that each build saves several hundred to thousand pointless key-value pairs for GIT branches and Jenkins builds that serve no purpose whatsoever.

      In our case, this means – as outlined above – we waste 3MB of RAM per build and 5 MB of disk space. With 10k builds per day, you can imagine that this is quite a predicament.

      As a workaround, we've written a Jenkins job that removes the tags contained in "<hudson.plugins.git.util.BuildData>" in the "build.xml". This cuts down its size from 5MB down to 16kB (~0.156MB). This of course also greatly boosts the speed of deserealizing the builds from disk.

      Our request would be: Either remove the collections/deserialization of this (from our POV) pointless data, or make its generation optional via a configuration option.

      Best regards,
      Martin Schröder
      Intel Mobile Communications GmbH

        Attachments

          Issue Links

            Activity

            Hide
            jkd Jim D added a comment -

            Thanks for the update!

            Show
            jkd Jim D added a comment - Thanks for the update!
            Hide
            markewaite Mark Waite added a comment -

            Jim D this issue won't be fixed in 4.0.0. The incompatibilities from the BuildDetails change were too great for the community. The accidental release of git plugin 4.0.0-rc to the production update centers showed incompatibilities that I had missed in my testing and that others had missed in their testing.

            BuildData will be the same bloated memory user in 4.0.0 that it is in 3.x.

            Show
            markewaite Mark Waite added a comment - Jim D this issue won't be fixed in 4.0.0. The incompatibilities from the BuildDetails change were too great for the community. The accidental release of git plugin 4.0.0-rc to the production update centers showed incompatibilities that I had missed in my testing and that others had missed in their testing. BuildData will be the same bloated memory user in 4.0.0 that it is in 3.x.
            Hide
            jkd Jim D added a comment -

            Mark Waite, I'm glad I found the comments in this issue, thank you!  When the final 4.0.0 is released with the changes for this issue, will it make sure to handle both BuildDetails and BuildData for previous build runs?  We updated to 4.0.0-rc some time back, and I just came across this thread recently trying to resolve a BuildData issue, and rolled back to 3.9.3.  At this point, Git plugin is working and creating BuildData for each build run, but all those old builds done with 4.0.0-rc don't have any kind of "Git Build Data" in the Jenkins UI anymore, and no BuildData or BuildDetails when retrieved programmatically from WorkflowRun.getAllActions.  I think it must be that 3.9.3 isn't able to read the new BuildDetails object info.  Will 4.0.0 be able to read both, from both 3.9.X builds and 4.0.0 builds?  Thanks again.

             

            Show
            jkd Jim D added a comment - Mark Waite , I'm glad I found the comments in this issue, thank you!  When the final 4.0.0 is released with the changes for this issue, will it make sure to handle both BuildDetails and BuildData for previous build runs?  We updated to 4.0.0-rc some time back, and I just came across this thread recently trying to resolve a BuildData issue, and rolled back to 3.9.3.  At this point, Git plugin is working and creating BuildData for each build run, but all those old builds done with 4.0.0-rc don't have any kind of "Git Build Data" in the Jenkins UI anymore, and no BuildData or BuildDetails when retrieved programmatically from WorkflowRun.getAllActions.  I think it must be that 3.9.3 isn't able to read the new BuildDetails object info.  Will 4.0.0 be able to read both, from both 3.9.X builds and 4.0.0 builds?  Thanks again.  
            Hide
            jekeller Jacob Keller added a comment -

            tzafrir, the gitlab plugin probably just needs to be updated to look for BuildDetails instead of BuildData.

            As for the rebuilding issue, that is likely due to the fact that the new plugin no longer maintains history of "I built this already" if the build which did it was deleted.

            If the build hasn't been deleted it's plausible there is a bug in how we look up the old data again....

            It may be worth more seriously considering the XmlFile approach.

            Show
            jekeller Jacob Keller added a comment - tzafrir , the gitlab plugin probably just needs to be updated to look for BuildDetails instead of BuildData. As for the rebuilding issue, that is likely due to the fact that the new plugin no longer maintains history of "I built this already" if the build which did it was deleted. If the build hasn't been deleted it's plausible there is a bug in how we look up the old data again.... It may be worth more seriously considering the XmlFile approach.
            Hide
            ilendir Stefan Hengelein added a comment -

            Thanks for your quick responses!

            Connor Tumbleson The downgrade to the previous version (3.9.1) fixed the issue for now. Just took some time to identify the culprit.

            Mark Waite No problem mate, mistakes happen. I'll have a closer look on monday so hopefully I can help to provide steps to reproduce the issue!

             

            Show
            ilendir Stefan Hengelein added a comment - Thanks for your quick responses! Connor Tumbleson The downgrade to the previous version (3.9.1) fixed the issue for now. Just took some time to identify the culprit. Mark Waite No problem mate, mistakes happen. I'll have a closer look on monday so hopefully I can help to provide steps to reproduce the issue!  

              People

              • Assignee:
                Unassigned
                Reporter:
                mhschroe Martin Schröder
              • Votes:
                28 Vote for this issue
                Watchers:
                70 Start watching this issue

                Dates

                • Created:
                  Updated: