Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44729

Optimize git clones via shared reference repo for multi branch builds of same repository

    Details

    • Similar Issues:

      Description

      We love the new pipeline multi branch (and github org) plugin models, but have a general issue of bloated disk usage and slow clones for some of our more active and larger repositories. Specifically we have an active repo that has a lot of open PRs at any given time and a few long-running mainline/maintenance branches.

      Every time a new branch or PR is opened, we generate a new clone on at least one build node for the new build (and typically as the PR gets more commits and the build nodes get assigned for subsequent builds differently it multiples to more clones on other machines).

      Given that these branches and PRs are all for the same repository, it'd be great to automatically have it silently share a common reference repo on each build node. Something like automatically incorporating https://github.com/orrc/git-webhook-proxy into the workflow.

      Otherwise with a single repo having ~70 open PRs at any given time and 4 main branches, we end up with ~74 clones of the *same* repo on each build node. That's a significant unnecessary disk bloat and network bandwidth hog. As a first level, just having a full clone to use as a reference per-repo per-agent would be a big help.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            Up to the Git plugin. IIRC there was an open PR which does essentially this.

            Show
            jglick Jesse Glick added a comment - Up to the Git plugin. IIRC there was an open PR which does essentially this.
            Hide
            seadub Chris Williams added a comment -

            This looks to be the open PR related to this: https://github.com/jenkinsci/git-plugin/pull/502

            Show
            seadub Chris Williams added a comment - This looks to be the open PR related to this: https://github.com/jenkinsci/git-plugin/pull/502
            Hide
            tr1z Tristan Lewis added a comment -

            Any update on the PR that Chris Williams linked? My team would love to have the git plugin handle managing reference repos on each node for us to keep disk usage down as well as speed up clones of newly-created branches.

            Show
            tr1z Tristan Lewis added a comment - Any update on the PR that Chris Williams linked? My team would love to have the git plugin handle managing reference repos on each node for us to keep disk usage down as well as speed up clones of newly-created branches.
            Hide
            bitwiseman Liam Newman added a comment -

            No update yet, but I'm starting to do some work in this area.  I'll provide a more detailed update shortly. 

            Show
            bitwiseman Liam Newman added a comment - No update yet, but I'm starting to do some work in this area.  I'll provide a more detailed update shortly. 
            Hide
            bitwiseman Liam Newman added a comment -

            Tristan Lewis Chris Williams
            The submitted PR has been postponed by Mark Waite. If you are interested in making it happen, please say so and we'll figure out a way to loop you with folks working on the git plugin.

            Show
            bitwiseman Liam Newman added a comment - Tristan Lewis Chris Williams The submitted PR has been postponed by Mark Waite . If you are interested in making it happen, please say so and we'll figure out a way to loop you with folks working on the git plugin.
            Hide
            tr1z Tristan Lewis added a comment -

            Liam Newman - I've hackily enabled this functionality to some degree via a shared pipeline plugin, but we've considered it a workaround until it was supported in the plugin itself. So yes, we're still interested in getting the PR moved forward and eventually integrated if possible. Let me know if I can do anything to help on that front. Thanks!

            Show
            tr1z Tristan Lewis added a comment - Liam Newman - I've hackily enabled this functionality to some degree via a shared pipeline plugin, but we've considered it a workaround until it was supported in the plugin itself. So yes, we're still interested in getting the PR moved forward and eventually integrated if possible. Let me know if I can do anything to help on that front. Thanks!
            Hide
            bitwiseman Liam Newman added a comment -

            Tristan Lewis
            Is that shared library available for people to look at?
            Also, Jenkins is participatory. I'd be happy to help if you want to revive that PR - grab the source and continue working on it. You should check with the git-plugin maintainers first before sinking effort into it though. Note, you might be able to use the environment variable expansion from JENKINS-43894 to make this work.

            Show
            bitwiseman Liam Newman added a comment - Tristan Lewis Is that shared library available for people to look at? Also, Jenkins is participatory. I'd be happy to help if you want to revive that PR - grab the source and continue working on it. You should check with the git-plugin maintainers first before sinking effort into it though. Note, you might be able to use the environment variable expansion from JENKINS-43894 to make this work.

              People

              • Assignee:
                Unassigned
                Reporter:
                seadub Chris Williams
              • Votes:
                9 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated: