Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19022

GIT Plugin (any version) heavily bloats memory use and size of build.xml with "BuildData" fields

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Hello everyone.

      Months ago, we've noticed a bug/issue with the GIT plug-in. Previously, it was only a minor nuisance but now, it causes each build that we start to use up ~3MB of main memory and ~5MB of disk space in the build.xml.

      The issue is due to the following behaviour of the GIT plug-in:
      For every build that has the GIT SCM defined, it retrieves the list of branches in the remote repository. For each branch, it retrieves the last build in Jenkins that was run against this branch.

      This information is then stored in the Build object in form of the "BuildData" field. This means, that the full list of all branches, plus their last builds is stored in each and every build – thus using up main memory and using up disk space in the "build.xml" file allocated for the build.

      It uses this information to populate a page for the build with the association of branches to builds:
      http://<SERVER>/job/<JOBNAME>/<BUILD-ID>/git/?

      For normal repositories, this data is relatively small, as only a limited number of unmerged branches exist. Unfortunately, we use GIT in an automated manner, where thousands of tags and branches are spawned without merging back into the mainline.

      This means that each build saves several hundred to thousand pointless key-value pairs for GIT branches and Jenkins builds that serve no purpose whatsoever.

      In our case, this means – as outlined above – we waste 3MB of RAM per build and 5 MB of disk space. With 10k builds per day, you can imagine that this is quite a predicament.

      As a workaround, we've written a Jenkins job that removes the tags contained in "<hudson.plugins.git.util.BuildData>" in the "build.xml". This cuts down its size from 5MB down to 16kB (~0.156MB). This of course also greatly boosts the speed of deserealizing the builds from disk.

      Our request would be: Either remove the collections/deserialization of this (from our POV) pointless data, or make its generation optional via a configuration option.

      Best regards,
      Martin Schröder
      Intel Mobile Communications GmbH

        Attachments

          Issue Links

            Activity

            Hide
            jekeller Jacob Keller added a comment -

            tzafrir, the gitlab plugin probably just needs to be updated to look for BuildDetails instead of BuildData.

            As for the rebuilding issue, that is likely due to the fact that the new plugin no longer maintains history of "I built this already" if the build which did it was deleted.

            If the build hasn't been deleted it's plausible there is a bug in how we look up the old data again....

            It may be worth more seriously considering the XmlFile approach.

            Show
            jekeller Jacob Keller added a comment - tzafrir , the gitlab plugin probably just needs to be updated to look for BuildDetails instead of BuildData. As for the rebuilding issue, that is likely due to the fact that the new plugin no longer maintains history of "I built this already" if the build which did it was deleted. If the build hasn't been deleted it's plausible there is a bug in how we look up the old data again.... It may be worth more seriously considering the XmlFile approach.
            Hide
            ilendir Stefan Hengelein added a comment -

            Thanks for your quick responses!

            Connor Tumbleson The downgrade to the previous version (3.9.1) fixed the issue for now. Just took some time to identify the culprit.

            Mark Waite No problem mate, mistakes happen. I'll have a closer look on monday so hopefully I can help to provide steps to reproduce the issue!

             

            Show
            ilendir Stefan Hengelein added a comment - Thanks for your quick responses! Connor Tumbleson The downgrade to the previous version (3.9.1) fixed the issue for now. Just took some time to identify the culprit. Mark Waite No problem mate, mistakes happen. I'll have a closer look on monday so hopefully I can help to provide steps to reproduce the issue!  
            Hide
            tzafrir11 tzafrir added a comment -

            The buildData is also required for gitlab plugin.
            gitlab plugin pipeline step "gitlabCommitStatus" isn't working anymore with 4.0.0-rc

             It does not find any BuildData object.  Had to downgrade to 3.9.3 where all was working fine.

            Show
            tzafrir11 tzafrir added a comment - The buildData is also required for gitlab plugin. gitlab plugin pipeline step "gitlabCommitStatus" isn't working anymore with 4.0.0-rc  It does not find any BuildData object.  Had to downgrade to 3.9.3 where all was working fine.
            Hide
            ibotpeaches Connor Tumbleson added a comment -

            Thanks Mark Waite - we reverted 3.9.2, then re-upgraded to 3.9.3. Unfortunately our builds were in quite a messed up shape. New commits to a branch were triggering all branches that still lived on the remote. We tried deleting the repos from the filesystem of Jenkins, hoping a new clone would resolve that - It did not. For other googlers who stumble upon this page, we had to restore a server backup and lose about 2 days of history on builds. We lost minor configurations, but it was way better than having our system bloated with thousands of builds and building nearly everything on each commit.

            Mistakes happen mate, no worries and I'll try and file a bug to keep these bugs organized.

            Show
            ibotpeaches Connor Tumbleson added a comment - Thanks Mark Waite - we reverted 3.9.2, then re-upgraded to 3.9.3. Unfortunately our builds were in quite a messed up shape. New commits to a branch were triggering all branches that still lived on the remote. We tried deleting the repos from the filesystem of Jenkins, hoping a new clone would resolve that - It did not. For other googlers who stumble upon this page, we had to restore a server backup and lose about 2 days of history on builds. We lost minor configurations, but it was way better than having our system bloated with thousands of builds and building nearly everything on each commit. Mistakes happen mate, no worries and I'll try and file a bug to keep these bugs organized.
            Hide
            markewaite Mark Waite added a comment -

            You should downgrade to git plugin 3.9.2 then install git plugin 3.9.3 which includes a fix for an agent / tool interaction issue that was first introduced in git plugin 3.9.2.

            The git plugin 4.0.0-rc is only a release candidate, not a production ready release. I made a mistake when I chose the version number. I assumed that the "-rc" suffix would deliver the plugin only from the experimental update center and not from the general update center. I was wrong. I apologize sincerely for my mistake.

            The issues detected with git plugin 4.0.0-rc indicate that it was not as close to release as my testing indicated. More testing and more fixes will be needed before git plugin 4.0.0 will be released for general availability.

            You can help assure that the problems you've found are fixed in git plugin 4.0.0-rc by providing a bug report which contains numbered steps that describe how someone else can see the same bug that you are seeing. The descriptions in the comments of this specific bug report seem to indicate at least two different bugs which are outside of this bug report and should be reported separately so that they can be tracked separately to resolution.

            Show
            markewaite Mark Waite added a comment - You should downgrade to git plugin 3.9.2 then install git plugin 3.9.3 which includes a fix for an agent / tool interaction issue that was first introduced in git plugin 3.9.2. The git plugin 4.0.0-rc is only a release candidate, not a production ready release. I made a mistake when I chose the version number. I assumed that the "-rc" suffix would deliver the plugin only from the experimental update center and not from the general update center. I was wrong. I apologize sincerely for my mistake. The issues detected with git plugin 4.0.0-rc indicate that it was not as close to release as my testing indicated. More testing and more fixes will be needed before git plugin 4.0.0 will be released for general availability. You can help assure that the problems you've found are fixed in git plugin 4.0.0-rc by providing a bug report which contains numbered steps that describe how someone else can see the same bug that you are seeing. The descriptions in the comments of this specific bug report seem to indicate at least two different bugs which are outside of this bug report and should be reported separately so that they can be tracked separately to resolution.

              People

              • Assignee:
                Unassigned
                Reporter:
                mhschroe Martin Schröder
              • Votes:
                27 Vote for this issue
                Watchers:
                63 Start watching this issue

                Dates

                • Created:
                  Updated: