Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43106

pipeline job hangs forever at checkout GitSCM - after git rev-list (sporadic)

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Duplicate
    • Component/s: core, git-plugin
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      Sometimes a pipeline job hangs around checkout. 

      Lately this seems to be after git rev-list. 

       

       > git rev-list --no-walk db25209b84b12cca5214227a7999fe280c82017e # timeout=10

      It will hang here until job is killed. A restart is fine - and it only happens occasionally. 

      This has been more commonly reported since 2018 February/January, a recent update. 

       

      A support bundle of this when it was happening: support_2018-02-07_22.54.25.zip

      IT was happening on slave-05 at the time (can see in its thread dump a few waiting things). Timeout does not work, it will wait indefinitely. 

       

       


      My pipeline job is getting sporadically hung on checkout step.

      Our git repo is relatively small and clone should not take more than 1 min.

      The problem occurring really often once a couple builds. Once it's occurring, the only way that I know to get rid of this state so far is, is disconnect the slave and re-connect it back.

      Build log:

      Started by user <USER>
      Checking out git <REPO> to read Jenkinsfile.bootstrap
      > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
      > git config remote.origin.url <REPO> # timeout=10
      Fetching upstream changes from <REPO>
      > git --version # timeout=10
      using GIT_SSH to set credentials jenkinsslave SSH key
      > git fetch --tags --progress <REPO> +refs/heads/:refs/remotes/origin/
      > git rev-parse refs/remotes/origin/devops_jenkins^{commit} # timeout=10
      > git rev-parse refs/remotes/origin/origin/devops_jenkins^{commit} # timeout=10
      Checking out Revision 2fd998fbe8b23f814b57d8c6eb7b46004268eedc (refs/remotes/origin/devops_jenkins)
      > git config core.sparsecheckout # timeout=10
      > git checkout -f 2fd998fbe8b23f814b57d8c6eb7b46004268eedc
      > git rev-list 2fd998fbe8b23f814b57d8c6eb7b46004268eedc # timeout=10
      [Pipeline] timestamps
      [Pipeline] {
      [Pipeline] node
      00:00:01.249 Running on <SLAVE> in <WORKSPACE>
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Load Jenkinsfile)
      [Pipeline] echo
      00:00:01.267 Branch: origin/devops_jenkins Repo: <REPO>
      [Pipeline] checkout
      00:03:27.565 > git rev-parse -is-inside-work-tree # timeout=10 <- This and further lines are being printed only after slave disconnect.
      00:03:27.566 Fetching changes from the remote Git repository
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [Pipeline] // timestamps
      [Pipeline] End of Pipeline
      hudson.remoting.RemotingSystemException: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:272)
      at com.sun.proxy.$Proxy98.setRemoteUrl(Unknown Source)
      at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
      at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:794)
      at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1070)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1101)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
      at hudson.security.ACL.impersonate(ACL.java:221)
      at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
      at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:191)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:256)
      ... 16 more
      Caused by: hudson.remoting.Channel$OrderlyShutdown
      at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1121)
      at hudson.remoting.Channel$1.handle(Channel.java:526)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
      Caused by: Command close created at
      at hudson.remoting.Command.<init>(Command.java:59)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1115)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1113)
      at hudson.remoting.Channel.close(Channel.java:1273)
      at hudson.remoting.Channel.close(Channel.java:1255)
      at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1120)
      ... 2 more
      Finished: FAILURE

        Attachments

        1. Capture.PNG
          Capture.PNG
          96 kB
        2. config.xml
          2 kB
        3. config.xml
          5 kB
        4. Jenkinsfile
          2 kB
        5. support_2017-10-06_04.20.32.zip
          4.47 MB
        6. support_2017-10-11_21.17.12.zip
          2.65 MB
        7. support_2018-02-07_22.54.25.zip
          1.12 MB
        8. threadDump.txt
          68 kB

          Issue Links

            Activity

            Hide
            markewaite Mark Waite added a comment - - edited

            Vincent Robert I don't think it is enough to disable the Jira plugin. I believe that you need to downgrade the Jira plugin from 2.5.1 to 2.5.0 or earlier. There was a discussion of "blacklisting" Jira plugin 2.5.1 and that has been implemented about a week ago in the Jenkins update center.

            Refer to JENKINS-48357 for the details of the types of damage done to critical libraries by the 2.5.1 release.

            Show
            markewaite Mark Waite added a comment - - edited Vincent Robert I don't think it is enough to disable the Jira plugin. I believe that you need to downgrade the Jira plugin from 2.5.1 to 2.5.0 or earlier. There was a discussion of "blacklisting" Jira plugin 2.5.1 and that has been implemented about a week ago in the Jenkins update center . Refer to JENKINS-48357 for the details of the types of damage done to critical libraries by the 2.5.1 release.
            Hide
            vrobert78 Vincent Robert added a comment -

            Mark Waite , since disabling the three plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean), hangs have gone. Everything has running fine yesterday.

            Show
            vrobert78 Vincent Robert added a comment - Mark Waite , since disabling the three plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean), hangs have gone. Everything has running fine yesterday.
            Hide
            vrobert78 Vincent Robert added a comment -

            We have just completly uninstall plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean) and then reinstall them.

            Indeed, versions has changed:

            • Blue Ocean 1.4.1
            • JIRA Integration for Blue Ocean 1.4.1
            • JIRA Plugin 2.5 (was 2.5.1)

            We hope this will be as fine as yesterday.

            Show
            vrobert78 Vincent Robert added a comment - We have just completly uninstall plugins (JIRA plugin, JIRA Integration for Blue Ocean and Blue Ocean) and then reinstall them. Indeed, versions has changed: Blue Ocean 1.4.1 JIRA Integration for Blue Ocean 1.4.1 JIRA Plugin 2.5 (was 2.5.1) We hope this will be as fine as yesterday.
            Hide
            markewaite Mark Waite added a comment - - edited

            Vincent Robert thanks very much for your investigation and for sharing your results. I'd like to close this bug as a duplicate of JENKINS-48357. JENKINS-48357 will remain open until the fix is released in the Jira plugin. Is that OK?

            Show
            markewaite Mark Waite added a comment - - edited Vincent Robert thanks very much for your investigation and for sharing your results. I'd like to close this bug as a duplicate of JENKINS-48357 . JENKINS-48357 will remain open until the fix is released in the Jira plugin. Is that OK?
            Hide
            vgaidarji Veaceslav Gaidarji added a comment -
            Show
            vgaidarji Veaceslav Gaidarji added a comment - Sam Van Oort ping

              People

              • Assignee:
                vrobert78 Vincent Robert
                Reporter:
                levt Lev Tartakovsky
              • Votes:
                8 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: