Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33273

Optimize Jenkinsfile loading and branch detection

    Details

    • Similar Issues:

      Description

      Currently if use the Git branch source, your repository gets cloned three times:

      1. Once to determine the head revision of the branch.
      2. Once to load Jenkinsfile.
      3. Once when you checkout scm as part of your build. (Normally once. Could be multiple times, or even zero.)

      This could be a performance issue for large repositories. What we would rather do in the second step is use SCMFileSystem to locate Jenkinsfile. Unfortunately this is not currently implemented in any of the current scm-api clients (git, subversion, mercurial, github-branch-source).

      One of the first two clones would still be necessary for Git since the remote protocol functionality here is limited to git-ls-remote, which could yield the head revision needed for step one (which is currently implemented using a local cache of the repository), but not the contents of Jenkinsfile (which, unlike for Literate, we need to have before the build starts). The Git plugin could be enhanced to avoid needing a cache for the first step, but this would not help workflow-multibranch at all since it would wind up needing it for an implementation of SCMFileSystem in the second step anyway.

      For Mercurial, the situation is somewhat similar in that the wire protocol does not support remote file access. It might in principle support remote head revision calculation, but not using the standard hg binary, which this plugin is based on. So the Mercurial plugin when used as a branch source just requests that you enabling the "caching" feature, which existed long before this, and which maintains a local clone of the repository. That could serve as the implementation of SCMFileSystem as well as providing the tip revision. Unlike with the Git plugin, the Mercurial plugin will actually reuse this cache during a regular workspace checkout, which has some advantages—for example, your slave need not be able to make a direct connection to the Mercurial server, so long as the master can—but may also be undesirable for performance, since it uses the Jenkins remoting channel. Probably the Mercurial plugin needs an intermediate mode, whereby caches are maintained on both master and slave, yet the slave cache is synchronized directly with the remote server rather than with the master. Or whereby the master maintains a cache, but the slave does not use caching at all.

      For Subversion the situation is simpler since the wire protocol (and, as far as I know, SVNKit) supports both remote head revision calculation and remote file retrieval. Therefore the only actual checkout would be in the user workspace.

      Note that in principle the first two steps could be collapsed: check out the SCM including Jenkinsfile to the master's jobname@script workspace, as with pre-multibranch CpsScmFlowDefinition, and then inspect the checkout after the fact to find its revision somehow for SCMBinder: for example, using git rev-parse HEAD, or hg log -r . --template '{node}\n', or svnversion .. Unfortunately scm-api offers no generic way of doing this; you need to call build with an SCMRevision but that revision can only be gotten by the repository inspection. So if this approach is to be taken, a new API would need to be introduced and implemented in the major SCM plugins. Anyway this approach is less desirable for the case of a massive working copy.

        Attachments

          Issue Links

            Activity

            jglick Jesse Glick created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-33272 [ JENKINS-33272 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-34140 [ JENKINS-34140 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-34596 [ JENKINS-34596 ]
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-32400 [ JENKINS-32400 ]
            jglick Jesse Glick made changes -
            Epic Link JENKINS-35386 [ 171179 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 169168 ] JNJira + In-Review [ 183384 ]
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-31155 [ JENKINS-31155 ]
            abayer Andrew Bayer made changes -
            Component/s pipeline-general [ 21692 ]
            abayer Andrew Bayer made changes -
            Component/s workflow-plugin [ 18820 ]
            jglick Jesse Glick made changes -
            Component/s workflow-multibranch-plugin [ 21465 ]
            Component/s pipeline [ 21692 ]
            seanf Sean Flanigan made changes -
            Link This issue is related to JENKINS-32246 [ JENKINS-32246 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-37345 [ JENKINS-37345 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-39548 [ JENKINS-39548 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-39703 [ JENKINS-39703 ]
            jglick Jesse Glick made changes -
            Link This issue depends on JENKINS-39355 [ JENKINS-39355 ]
            jamesdumay James Dumay made changes -
            Labels api multibranch api blueocean multibranch
            stephenconnolly Stephen Connolly made changes -
            Link This issue is duplicated by JENKINS-38212 [ JENKINS-38212 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-multibranch PR 47 (Web Link)" [ 15264 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "git PR 467 (Web Link)" [ 15265 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-cps PR 97 (Web Link)" [ 15266 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "github-branch-source PR 104 (Web Link)" [ 15267 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-40340 [ JENKINS-40340 ]
            hrmpw Patrick Wolf made changes -
            Link This issue relates to JENKINS-42080 [ JENKINS-42080 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-42351 [ JENKINS-42351 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-38659 [ JENKINS-38659 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-multibranch PR 47 (Web Link)" [ 15264 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 49 (Web Link)" [ 15506 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 57 (Web Link)" [ 15507 ]
            cdenneen Chris Denneen made changes -
            Link This issue is related to JENKINS-35282 [ JENKINS-35282 ]
            jglick Jesse Glick made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-42406 [ JENKINS-42406 ]
            navaati Léo Gillot-Lamure made changes -
            Comment [ Hi.

            We are badly affected by this problem (our git repo is 2GB) and it is still present for us even though we're on an up to date Jenkins.

            Here is a transcript of the first build for a PR:
            {noformat}
            Branch event
            21:46:11 Connecting to https://api.github.com using navaati/****** (navaati github superpower token)
            Checking out git https://github.com/H3IO/HyperCube.git https://github.com/H3IO/HyperCube.git to read Jenkinsfile
            Cloning the remote Git repository
            Cloning repository https://github.com/H3IO/HyperCube.git
             > git init /var/lib/jenkins/workspace/H3IO_HyperCube_PR-2456-QRWOPV3KS7P4NT653VHEA3YTIWGDI2HVFN6AWXB4RVJQQDYPR3UA@script # timeout=10
            Fetching upstream changes from https://github.com/H3IO/HyperCube.git
             > git --version # timeout=10
            using GIT_ASKPASS to set credentials navaati github superpower token
             > git fetch --tags --progress https://github.com/H3IO/HyperCube.git +refs/heads/*:refs/remotes/origin/*
             > git config remote.origin.url https://github.com/H3IO/HyperCube.git # timeout=10
             > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
             > git config remote.origin.url https://github.com/H3IO/HyperCube.git # timeout=10
            Fetching upstream changes from https://github.com/H3IO/HyperCube.git
            using GIT_ASKPASS to set credentials navaati github superpower token
             > git fetch --tags --progress https://github.com/H3IO/HyperCube.git +refs/heads/*:refs/remotes/origin/*
             > git config remote.origin1.url https://github.com/H3IO/HyperCube.git # timeout=10
            Fetching upstream changes from https://github.com/H3IO/HyperCube.git
            using GIT_ASKPASS to set credentials navaati github superpower token
             > git fetch --tags --progress https://github.com/H3IO/HyperCube.git +refs/pull/*/head:refs/remotes/origin/pr/*
            Merging master commit eb08b54b49658077f34f3a5e71010ae9ccbbf879 into PR head commit a674225cb3cbd2dc4a126996a50d73262d5132f4
             > git config core.sparsecheckout # timeout=10
             > git checkout -f a674225cb3cbd2dc4a126996a50d73262d5132f4
             > git merge eb08b54b49658077f34f3a5e71010ae9ccbbf879 # timeout=10
             > git rev-parse HEAD^{commit} # timeout=10
            Merge succeeded, producing a674225cb3cbd2dc4a126996a50d73262d5132f4
            Checking out Revision a674225cb3cbd2dc4a126996a50d73262d5132f4 (PR-2456)
             > git config core.sparsecheckout # timeout=10
             > git checkout -f a674225cb3cbd2dc4a126996a50d73262d5132f4{noformat}

            As you can see a full fetch is still done, in a new directory for each PR.
            Do I need to do something in particular for this fix to take effect ?

            Regards,
            Léo Gillot-Lamure. ]
            navaati Léo Gillot-Lamure made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            jglick Jesse Glick made changes -
            Status Reopened [ 4 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            maxpowa Max Gurela made changes -
            Link This issue is related to JENKINS-43194 [ JENKINS-43194 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-39608 [ JENKINS-39608 ]
            seadub Chris Williams made changes -
            Link This issue relates to JENKINS-44729 [ JENKINS-44729 ]
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-42957 [ JENKINS-42957 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-42817 [ JENKINS-42817 ]
            behrens Florian Behrens made changes -
            Link This issue is related to JENKINS-42518 [ JENKINS-42518 ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal OSS-628 (Web Link)" [ 18860 ]
            jmalopoy Jesús Malo Poyatos made changes -
            Link This issue is blocking JENKINS-48420 [ JENKINS-48420 ]
            kpop kpop made changes -
            Link This issue blocks JENKINS-50490 [ JENKINS-50490 ]
            kpop kpop made changes -
            Link This issue blocks JENKINS-50490 [ JENKINS-50490 ]
            kpop kpop made changes -
            Link This issue is blocking JENKINS-50490 [ JENKINS-50490 ]
            sberube Steve Berube made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            sberube Steve Berube made changes -
            Comment [ This appears to still be an issue when using the job type 'Github Organization' using a multi-branch job type it downloads the jenkinsfile directly. Using a Github Organization job type exhibits the original behavior described in this defect. ]
            jglick Jesse Glick made changes -
            Status Reopened [ 4 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            abayer Andrew Bayer made changes -
            Link This issue is duplicated by JENKINS-53988 [ JENKINS-53988 ]
            abayer Andrew Bayer made changes -
            Link This issue is duplicated by JENKINS-53988 [ JENKINS-53988 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                59 Vote for this issue
                Watchers:
                89 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: