Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38973

Pipeline initial checkout of Jenkinsfile not honoring git timeout

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: git-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.19.1
      Git client plugin 2.0.0
      Git plugin 3.0.0
      Pipeline: Multibranch: 2.9
    • Similar Issues:

      Description

      I am having an issue cloning a large repo for use in a multi-branch pipeline project.

      My goal is to run pipeline jobs against certain branches within my repository. However, the initial fetch seems to timeout. As you can see from the attached log, the issue appears to arise from a 10 minute timeout issue. However, if you see the below screen shots, I have all of my timeouts set to 180minutes.

      In the attached tickets there were known issues with the 10 minute git timeout that were fixed by allowing users access to overrides as I have shown in the attached images. It appears as though these overrides are not being honored for some reason within the MultibranchPipeline project.

      I have used the settings described here on a smaller repo and it was able to properly index the branches.

      As a work around, I also tried only using the overrides as advanced settings for checkout/fetch behavior. This did not work either, unfortunately.

        Attachments

          Issue Links

            Activity

            Hide
            erik_hakansson Erik Hakansson added a comment -

            We have this issue too. Always timeout on indexing.

            Show
            erik_hakansson Erik Hakansson added a comment - We have this issue too. Always timeout on indexing.
            Hide
            erik_hakansson Erik Hakansson added a comment -

            This pull request to Git client plugin makes the default timeout configurable and should solve this issue.

            Show
            erik_hakansson Erik Hakansson added a comment - This pull request to Git client plugin makes the default timeout configurable and should solve this issue.
            Hide
            orkenstein Ruslan Khamidullin added a comment -

            Huge blocker. Needs to be fixed.
            Any available workarounds?

            Show
            orkenstein Ruslan Khamidullin added a comment - Huge blocker. Needs to be fixed. Any available workarounds?
            Hide
            tisi Tino Siegmund added a comment -

            Same problem here. Tryed to manually pre clone the repository does not help.

            Show
            tisi Tino Siegmund added a comment - Same problem here. Tryed to manually pre clone the repository does not help.
            Hide
            ironsean Sean MacKay added a comment -

            This is suddenly an issue for me. I'm not sure the timeout is the culprit, as it used to clone fine, but now some problem with the git fetch does cause the timeout to expire.

            Additionally, this doesn't read as a failure on the git fetch for the multibranch pipeline project, it just stays locked on this git fetch process preventing other jobs from running. Perhaps this is another issue that should have a new bug opened?

            Show
            ironsean Sean MacKay added a comment - This is suddenly an issue for me. I'm not sure the timeout is the culprit, as it used to clone fine, but now some problem with the git fetch does cause the timeout to expire. Additionally, this doesn't read as a failure on the git fetch for the multibranch pipeline project, it just stays locked on this git fetch process preventing other jobs from running. Perhaps this is another issue that should have a new bug opened?
            Hide
            markewaite Mark Waite added a comment -

            An alternative might be to pass -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=37 as a Java command line option to Jenkins. That will set the Jenkins global git timeout to 37 minutes instead of its default of 10 minutes.

            I don't know why the pipeline initial clone ignores the timeout setting, but I've confirmed that it does ignore it.

            I confirmed it is ignored with the following steps:

            1. Create a multi-branch pipeline job with Git as the SCM source
            2. Use https://github.com/torvalds/linux.git as the repository location
            3. Add the "Additional Behaviour" for "Advanced clone options"
            4. Set the timeout to a small value (1 or 2)
            5. Click the "Scan Multibranch Pipeline Now" link
            6. Monitor the progress of the clone, note that it will run longer than the assigned timeout value. Other cases honor the timeout, but apparently not that case
            Show
            markewaite Mark Waite added a comment - An alternative might be to pass -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=37 as a Java command line option to Jenkins. That will set the Jenkins global git timeout to 37 minutes instead of its default of 10 minutes. I don't know why the pipeline initial clone ignores the timeout setting, but I've confirmed that it does ignore it. I confirmed it is ignored with the following steps: Create a multi-branch pipeline job with Git as the SCM source Use https://github.com/torvalds/linux.git as the repository location Add the "Additional Behaviour" for "Advanced clone options" Set the timeout to a small value (1 or 2) Click the "Scan Multibranch Pipeline Now" link Monitor the progress of the clone, note that it will run longer than the assigned timeout value. Other cases honor the timeout, but apparently not that case
            Hide
            hardi249 hardi249 added a comment -

            -Dorg.jenkinsci.plugins.gitclient.Git.timeOut does nothing. The initial checkout/fetch for each triggered branch job still defaults to 10 minutes.

            Show
            hardi249 hardi249 added a comment - -Dorg.jenkinsci.plugins.gitclient.Git.timeOut does nothing. The initial checkout/fetch for each triggered branch job still defaults to 10 minutes.
            Hide
            lozjo John Lozano added a comment -

            +1 on this issue.

            Passing the org.jenkinsci.plugins.gitclient.Git.timeOut in as JAVA_OPTS, using setProperty() from the script console, and manually configuring the clone behavior on a multi-branch pipeline job all fail to modify the timeout used for the initial checkout/fetch when indexing branches. 

            I really want to try out this plugin in order to simplify my current build strategy, but I don't want to resort to splitting up my repository.

            Using TFS/Git as repository provider.

            Show
            lozjo John Lozano added a comment - +1 on this issue. Passing the org.jenkinsci.plugins.gitclient.Git.timeOut in as JAVA_OPTS, using setProperty() from the script console, and manually configuring the clone behavior on a multi-branch pipeline job all fail to modify the timeout used for the initial checkout/fetch when indexing branches.  I really want to try out this plugin in order to simplify my current build strategy, but I don't want to resort to splitting up my repository. Using TFS/Git as repository provider.
            Hide
            markewaite Mark Waite added a comment -

            John Lozano have you considered using a reference repository as a way to significantly reduce the clone time and the disc use of your large git repository?  Refer to my Jenkins World 2017 presentation for an overview of the options for better handling large repositories in Jenkins.

            Show
            markewaite Mark Waite added a comment - John Lozano have you considered using a reference repository as a way to significantly reduce the clone time and the disc use of your large git repository?  Refer to my Jenkins World 2017 presentation for an overview of the options for better handling large repositories in Jenkins.
            Hide
            g_doc2080 guillaume docquier added a comment -

            +1 this issue.

            I have a local reference repository on the disk and, but it is ignored. Just like the timeout option.

            See attachments.

            would LOVE to have a fix or workaround for this. I tried digging into the code but I am a total newbie

            Wouldn't the solution be to add a configurable timeout and reference repository to the 'Branch Discovery' feature ?

             

            Show
            g_doc2080 guillaume docquier added a comment - +1 this issue. I have a local reference repository on the disk and, but it is ignored. Just like the timeout option. See attachments. would LOVE to have a fix or workaround for this. I tried digging into the code but I am a total newbie Wouldn't the solution be to add a configurable timeout and reference repository to the 'Branch Discovery' feature ?  
            Hide
            aseg Antonio Asenjo added a comment -

            Hi, 

            guillaume docquier please check this link: https://stackoverflow.com/a/37133464 

            Show
            aseg Antonio Asenjo added a comment - Hi,  guillaume docquier  please check this link:  https://stackoverflow.com/a/37133464  
            Hide
            lostinberlin Steve Boardwell added a comment -

            Hi,

            I second guillaume docquier's comment. Our git repository is close to 5GB in size and the local git repository is not being used at all when running "Branch Indexing".

            Antonio Asenjo: thanks for the link but setting the timeout option higher is not really a long term solution since we have a large number of pipelines with all require branch indexing meaning the 5GB will be duplicated many times on the master.

            I too would like to see a solution for this. I will have a look into the code when I have time but I don't know when this will be.

             

            Show
            lostinberlin Steve Boardwell added a comment - Hi, I second guillaume docquier 's comment. Our git repository is close to 5GB in size and the local git repository is not being used at all when running "Branch Indexing". Antonio Asenjo : thanks for the link but setting the timeout option higher is not really a long term solution since we have a large number of pipelines with all require branch indexing meaning the 5GB will be duplicated many times on the master. I too would like to see a solution for this. I will have a look into the code when I have time but I don't know when this will be.  
            Hide
            sseb Sebastien Hoareau added a comment -

            Not exactly the same, but we have a similar problem with the shallow clone option: the plugin doesn't seem to take into account either (no --depth=xx on the git fetch command line)

            Show
            sseb Sebastien Hoareau added a comment - Not exactly the same, but we have a similar problem with the shallow clone option: the plugin doesn't seem to take into account either (no --depth=xx on the git fetch command line)
            Hide
            rohit_gagan Rohit Gagan added a comment -

            Any resolution to this issue. Still facing the same problem

            Show
            rohit_gagan Rohit Gagan added a comment - Any resolution to this issue. Still facing the same problem
            Hide
            markewaite Mark Waite added a comment - - edited

            No Rohit Gagan, the best work around is still the stackoverflow article referenced by Antonio Asenjo. Since branch indexing happens on the master and uses a known cache, it should create only a single copy of the large repository, then it will reuse that for additional operations.

            Show
            markewaite Mark Waite added a comment - - edited No Rohit Gagan , the best work around is still the stackoverflow article referenced by Antonio Asenjo . Since branch indexing happens on the master and uses a known cache, it should create only a single copy of the large repository, then it will reuse that for additional operations.
            Hide
            rohit_gagan Rohit Gagan added a comment -

            Mark Waite. Tried adding that, this still takes timeout as 10 min

             

            Started
            [Wed May 22 11:55:15 EDT 2019] Starting branch indexing...
            > git --version # timeout=10

            Show
            rohit_gagan Rohit Gagan added a comment - Mark Waite . Tried adding that, this still takes timeout as 10 min   Started [Wed May 22 11:55:15 EDT 2019] Starting branch indexing... > git --version # timeout=10
            Hide
            rohit_gagan Rohit Gagan added a comment -

            Mark Waite only option i see is change in git plugin code then to increase the default timeout. What you suggest?

             

            Show
            rohit_gagan Rohit Gagan added a comment - Mark Waite only option i see is change in git plugin code then to increase the default timeout. What you suggest?  
            Hide
            rohit_gagan Rohit Gagan added a comment -

            Mark Waite. Thanks. This works. i was adding the parameter at the wrong end

            Show
            rohit_gagan Rohit Gagan added a comment - Mark Waite . Thanks. This works. i was adding the parameter at the wrong end

              People

              • Assignee:
                Unassigned
                Reporter:
                magic David Johnson
              • Votes:
                19 Vote for this issue
                Watchers:
                27 Start watching this issue

                Dates

                • Created:
                  Updated: