Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38441

Commit notifications stop triggering job after error accessing git server

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • None
    • jenkins 2.7.4
      git-plugin 2.6.0

      We have a simple pipeline job that checks out source from git (stash) and builds it. This uses "Stash Webhook to Jenkins" commit notifications which should include a SHA1 hash. It generally works fine but occasionally the server will return http code 503 (log below) when checking out. After this, the job stops responding to commit notifications. While we have multiple nodes, this job is only running on one specific node (for performance reasons).

      Running the job manually gets it out of this state but this should not be necessary. I believe that Jenkins should be resilient to transient failures like this.

      > git -c core.askpass=true fetch --tags --progress https://stash/product.git +refs/heads/*:refs/remotes/origin/* # timeout=30
       > git config --local --remove-section credential # timeout=10
      ERROR: Error fetching remote repo 'origin'
      hudson.plugins.git.GitException: Failed to fetch from https://stash/product.git
      	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:799)
      	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1055)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1086)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:52)
      	at hudson.security.ACL.impersonate(ACL.java:213)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:49)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress https://stash/product.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
      stdout: 
      stderr: fatal: unable to access 'https://stash/product.git/': Received HTTP code 503 from proxy after CONNECT
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1740)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1476)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:63)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:314)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:153)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
      	at hudson.remoting.Request$2.run(Request.java:332)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at hudson.remoting.Engine$1$1.run(Engine.java:85)
      	at java.lang.Thread.run(Thread.java:745)
      	at ......remote call to jagent-lin6(Native Method)
      	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
      	at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
      	at hudson.remoting.Channel.call(Channel.java:781)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
      	at sun.reflect.GeneratedMethodAccessor572.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
      	at com.sun.proxy.$Proxy85.execute(Unknown Source)
      	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:797)
      	... 13 more

      Our pipeline script looks something like this...:

      node('specific_node') {
          stage('Check out') {
              checkout scm: [$class: 'GitSCM',
                        branches: [[name: "*/master"]],
                        doGenerateSubmoduleConfigurations: false,
                        extensions: [
                            [$class: 'CleanCheckout'],
                            [$class: 'CloneOption', reference: '', shallow: false, timeout: 30]
                        ],
                        submoduleCfg: [],
                        userRemoteConfigs: [[credentialsId: 'abcd1234',
                            url: 'https://git/product.git']]
                  ]
          }
      }
      

            Unassigned Unassigned
            rg Russell Gallop
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: