Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42351

Shallow clone on master when running job across slaves

    Details

    • Similar Issues:

      Description

      Right now we use Jenkinsfiles for all of our branches and we run these jobs across slaves.
      Our master does no jobs, it's meant to just coordinate the slaves.

      However because each branch is built, that means the master clones our entire repository per branch. This takes up a lot of disk space.

      Is there a way to do a shallow clone on the master so that it only knows about the branch that it's building? It doesnt make sense to clone the whole repo per branch when the branch is a subset of the repo itself.

      https://paste.fedoraproject.org/paste/bmZqN6-Mciz-oPhGuf1aKV5M1UNdIGYhyRLivL9gydE=

      183M    /var/lib/jenkins/workspace/redact-EKFU2VZFRWU4KXB4654FWURV2Q4AOPB2UXMBLUPEMX77NXWGZGDA@script
      175M    /var/lib/jenkins/workspace/lth_4636-product-to-service-R7DOH3YG2S3PCNRVQ4OHWJEQLMV4WUVPZJY6NXITQX7BCI63RAQQ@script
      732K    /var/lib/jenkins/workspace/redact-2DDF7H6TH77GDLANNM7J5JNMKMBXUZQGZUBSBQW7HKRZZDTOISYQ@script
      175M    /var/lib/jenkins/workspace/play-0-if-no-cases-for-year-YVL7UHVREPETGHMM6SUR7ZYDXOAYPWSLHTW24NKWJALUS5LGUXUQ@script
      6.2G    /var/lib/jenkins/
      

      every branch is 175+MB on the master because each branch is a git clone. I'd like to only get the context of a branch on the master as well as the slave.

      Also, on the slave I'm unsure how to do a shallow clone via Jenkinsfile DSL without specifying all of the other information, such as User credentials, credentials Id, etc.

      I'd like to just have

      node('ec2') {
        stage('Checkout') {
          checkout scm //something about adding shallow clone here
        }
      }
      

        Attachments

          Issue Links

            Activity

            Hide
            markewaite Mark Waite added a comment - - edited

            This seems like more of a request for help than a description of a bug report. There are only a very few people watching bug reports to the Jenkins git plugin. Please submit your questions to the Jenkins user mailing list so that it can be discussed there.

            The shallow clone setting is passed from the multi-branch pipeline parent to the jobs which are created for it.

            In your case, you may get the most benefit by using a reference copy of your large repository on the master node (and possibly on all the slave nodes). A reference copy can dramatically reduce disc use and time to clone a repository.

            Refer to the first 5-10 minutes of this youtube video for hints and recommendations for managing large git repositories with the Jenkins git plugin.

            Show
            markewaite Mark Waite added a comment - - edited This seems like more of a request for help than a description of a bug report. There are only a very few people watching bug reports to the Jenkins git plugin. Please submit your questions to the Jenkins user mailing list so that it can be discussed there. The shallow clone setting is passed from the multi-branch pipeline parent to the jobs which are created for it. In your case, you may get the most benefit by using a reference copy of your large repository on the master node (and possibly on all the slave nodes). A reference copy can dramatically reduce disc use and time to clone a repository. Refer to the first 5-10 minutes of this youtube video for hints and recommendations for managing large git repositories with the Jenkins git plugin.
            Hide
            myoung34 marc young added a comment -

            This isn't really a request for help, it definitely seems like a bug that I'm scanning an organization for repositories, to build branches, but per branch i'm cloning the repository.

            I'm not using pipelines, I'm using the new SCM 2.0 stuff, specifically the github-branches-api code.

            Per the screen shot I dont have access to anything SCM-related to set shallow clone on the master, so this feels like a bug. Time isn't an issue and neither is the slave, those come up/down ad-hoc so i'm not worried about the repository at that level. Only as to why the master clones the entire repo per branch and why I cannot set shallow at that level. By the time the Jenkinsfile has been compiled and run, the master has already cloned the entire repo for that branch: https://paste.fedoraproject.org/paste/bmZqN6-Mciz-oPhGuf1aKV5M1UNdIGYhyRLivL9gydE=

            Show
            myoung34 marc young added a comment - This isn't really a request for help, it definitely seems like a bug that I'm scanning an organization for repositories, to build branches, but per branch i'm cloning the repository. I'm not using pipelines, I'm using the new SCM 2.0 stuff, specifically the github-branches-api code. Per the screen shot I dont have access to anything SCM-related to set shallow clone on the master, so this feels like a bug. Time isn't an issue and neither is the slave, those come up/down ad-hoc so i'm not worried about the repository at that level. Only as to why the master clones the entire repo per branch and why I cannot set shallow at that level. By the time the Jenkinsfile has been compiled and run, the master has already cloned the entire repo for that branch: https://paste.fedoraproject.org/paste/bmZqN6-Mciz-oPhGuf1aKV5M1UNdIGYhyRLivL9gydE=
            Hide
            jglick Jesse Glick added a comment -

            GitSCMSource definitely is designed to keep one master clone per repository, not per branch. If that is not working then there is a bug. Steps to reproduce from scratch?

            Show
            jglick Jesse Glick added a comment - GitSCMSource definitely is designed to keep one master clone per repository , not per branch . If that is not working then there is a bug. Steps to reproduce from scratch?
            Hide
            jglick Jesse Glick added a comment -

            Oh, never mind, you are not talking about the branch indexing clone. You are talking about the @script workspace.

            Show
            jglick Jesse Glick added a comment - Oh, never mind, you are not talking about the branch indexing clone. You are talking about the @script workspace.

              People

              • Assignee:
                Unassigned
                Reporter:
                myoung34 marc young
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: