Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59317

Spurious SCM change with a failed git call (e.g. network error)

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None

      Our internal build farm's connection to the internet has its flaws, ups and downs, and sometimes during the downs we see gems like this: an effectively failed SCM poll ("Couldn't get remote head revision") is interpreted as a "Changes found" situation, and causes a build "Started by an SCM change", as the (failed) build's Polling Log says:

      ````
      Polling Log

      This page captures the polling log that triggered this build.

      Started on Sep 7, 2019 6:15:15 AM
      Using strategy: Specific revision
      [poll] Last Built Revision: Revision dee1deac2b8bc60039ea3cd2907afa708a967486 (release/IPM_Infra-1.4)
      using credential 955c217d-8133-41c2-9235-9ce3936ddca6
      > git --version # timeout=10
      using GIT_ASKPASS to set credentials 42ity-ci GH Token as password
      Setting http proxy: thunderbolt.roz.lab.etn.com:8081
      > git ls-remote -h https://github.com/42ity/fty-outage.git # timeout=10
      Unexpected ls-remote output line ''
      [poll] Couldn't get remote head revision
      Done. Took 23 sec
      Changes found
      ````

      In this particular situation (outage) as a result, this build fails quickly during checkout of sources with
      ```
      ...
      Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/42ity/fty-outage.git +refs/heads/release/IPM_Infra-1.4:refs/remotes/origin/release/IPM_Infra-1.4" returned status code 128:
      stdout:
      stderr: fatal: unable to access 'https://github.com/42ity/fty-outage.git/': The requested URL returned error: 503
      ...
      ```
      which is a reasonable outcome of a network outage (our proxy failed to fetch => 503)... however when the network comes back, the last-tried (green, long ago) revision is the same as the current HEAD, so no need to rebuild, as the branch's Git polling log says:
      ````
      Git Polling Log

      Started on Sep 11, 2019 1:59:13 PM
      Using strategy: Specific revision
      [poll] Last Built Revision: Revision dee1deac2b8bc60039ea3cd2907afa708a967486 (release/IPM_Infra-1.4)
      using credential 955c217d-8133-41c2-9235-9ce3936ddca6
      > git --version # timeout=10
      using GIT_ASKPASS to set credentials 42ity-ci GH Token as password
      Setting http proxy: thunderbolt.roz.lab.etn.com:8081
      > git ls-remote -h https://github.com/42ity/fty-outage.git # timeout=10
      ````

      So we end up with with a latest-known state of a branch being "failed", more prominently so on still-tracked old releases that are in fact untouched for months and years (and won't be, unless a need for some hotfix arises).

      As a workaround, I have to go to such jobs and manually rebuild them, so they do not pollute our logs and job lists with red lights, and do not trigger alarms we've made.

      I believe the undetermined HEAD revision should either not cause a build, or if as a failsafe we want to try one - be remembered as such (null?) and when the network comes back, cause a re-rebuild with a definitive commit, even if same as tried before, so the latest-build status converges to a valid up-to-date verdict.

            Unassigned Unassigned
            jimklimov Jim Klimov
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: