Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27093

Spurious gits scm poll change detection

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
      git-plugin 2.3.5
      git-client-plugin 1.16.1
      jenkins 1.580.3
    • Similar Issues:

      Description

      We were running jenkins 1.580.2, git-plugin 2.3.3, git-client-plugin 1.15.0, before upgrading to jenkins 1.580.3, git-plugin 2.3.5, git-client-plugin 1.16.1.

      After the upgrade, some our SCM poll jobs started to behave erratically, as shown by following poll log:

      Started on Feb 23, 2015 1:33:00 PM
      Using strategy: Default
      [poll] Last Built Revision: Revision a74f1d1204a5c892466b52ac68ee6443c1e459d7 (refs/remotes/origin/linux-3.14.y)
       > /usr/bin/git --version # timeout=10
       > /usr/bin/git -c core.askpass=true ls-remote -h git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git # timeout=10
      [poll] Latest remote head revision on origin/linux-3.14.y is: a74f1d1204a5c892466b52ac68ee6443c1e459d7
      Done. Took 0.69 sec
      Changes found
      

      So, as the log above shows, "Last Built Revision" and "Latest remote head revision" are the same, and yet "Changes found". And this "Changes found" happened at each poll point (every 5 mins in our case), with build triggering, building that revision, and then doing it over again.

      After closer look at by job owners, it turned out that after upgrade, this job started to use optimized workspace-less polling (using git ls-remote), whereas previously, it used workspace-ful polling.

      Looking at https://github.com/jenkinsci/git-plugin/blob/master/src/main/java/hudson/plugins/git/GitSCM.java#L590 , turns out that while poll log talks about "Last Built Revision" and "Latest remote head revision", the actual logic for detecting whether change has happened is different. So, "Latest remote head revision" is taken, then some kind of data structure is queried for a build number associated with that change, and if there's no such build, it is triggered. And in our case, apparently exactly this querying part what failed, because otherwise revision values were ok.

      Summing up: 2.3.5/1.16.1 appear to have made WS-less polling optimization more aggressive, which I don't really see noted in changelog. That's not bad on its own, but apparently that mode has some bugs. In this particular case, the problems can be avoided by adding an extra stop-gap check of "Last Built Revision" and "Latest remote head revision" being equal - if they're, then there're for sure no changes, regardless of presence of detailed revision-to-build mapping.

      I would recommend plugin maintainers to add such a condition, to make plugin more robust.

      Thanks!

        Attachments

          Activity

          Hide
          knormoyle Kevin Normoyle added a comment -

          Hi Mohamed.
          As I noted above, the problem I had disappeared as I was messing with things.

          what sort of system do you have jenkins running on?
          I reinstalled jenkins on my ubuntu system with apt-get install jenkins --reinstall
          that may have had a correlation

          I can show some things from my current jobs.
          Git client plugin version 1.16.1
          Git plugin version 2.3.5
          Github API plugin 1.63
          Github plugin 1.11
          Jenkins 1.607

          I have a String Parameter on the build called BRANCH

          Under Advanced in Source Code Management, the Refspec is
          +refs/heads/${BRANCH}:refs/remotes/origin/${BRANCH}
          Branches to Build (Branch Specifier) is
          ${BRANCH}

          I'm using Poll SCM

          it currently is working, with Git Polling Log giving results like this when no change is needed
          Started on Apr 6, 2015 3:55:00 PM
          Using strategy: Default
          [poll] Last Built Revision: Revision 4a8e9987b4d2ca94a966abe88a590809e3109e4c (origin/cliffc_jenkins)
          > git ls-remote -h https://github.com/h2oai/h2o-dev.git # timeout=10
          [poll] Latest remote head revision on origin/cliffc_jenkins is: 4a8e9987b4d2ca94a966abe88a590809e3109e4c - already built by 1032
          Done. Took 3.5 sec
          No changes

          Show
          knormoyle Kevin Normoyle added a comment - Hi Mohamed. As I noted above, the problem I had disappeared as I was messing with things. what sort of system do you have jenkins running on? I reinstalled jenkins on my ubuntu system with apt-get install jenkins --reinstall that may have had a correlation I can show some things from my current jobs. Git client plugin version 1.16.1 Git plugin version 2.3.5 Github API plugin 1.63 Github plugin 1.11 Jenkins 1.607 I have a String Parameter on the build called BRANCH Under Advanced in Source Code Management, the Refspec is +refs/heads/${BRANCH}:refs/remotes/origin/${BRANCH} Branches to Build (Branch Specifier) is ${BRANCH} I'm using Poll SCM it currently is working, with Git Polling Log giving results like this when no change is needed Started on Apr 6, 2015 3:55:00 PM Using strategy: Default [poll] Last Built Revision: Revision 4a8e9987b4d2ca94a966abe88a590809e3109e4c (origin/cliffc_jenkins) > git ls-remote -h https://github.com/h2oai/h2o-dev.git # timeout=10 [poll] Latest remote head revision on origin/cliffc_jenkins is: 4a8e9987b4d2ca94a966abe88a590809e3109e4c - already built by 1032 Done. Took 3.5 sec No changes
          Hide
          mgalaleldin Mohamed Galal El-Din added a comment -

          Hi Kevin,

          I'm using Ubuntu 14.04. And originally installed Jenkins using the apt-get method as explained here

          My Jenkins server version is 1.607. And my git repository is hosted on BitBucket.

          My issue seems to be a different than this one. I'll file a separate JIRA.

          Show
          mgalaleldin Mohamed Galal El-Din added a comment - Hi Kevin, I'm using Ubuntu 14.04. And originally installed Jenkins using the apt-get method as explained here My Jenkins server version is 1.607. And my git repository is hosted on BitBucket. My issue seems to be a different than this one. I'll file a separate JIRA.
          Hide
          mgalaleldin Mohamed Galal El-Din added a comment -

          I've reported JENKINS-27769 to track my issue. Thanks Kevin.

          Show
          mgalaleldin Mohamed Galal El-Din added a comment - I've reported JENKINS-27769 to track my issue. Thanks Kevin.
          Hide
          markewaite Mark Waite added a comment -

          The git plugin 2.3.6 pre-release and git client plugin 1.18.0 pre-release are being tested in hopes of releasing new versions before the end of June. If you're willing to assist with the testing, please download and install a pre-release build of the git client plugin and the git plugin. Problems detected in the pre-release should be e-mailed to Mark Waite and Nicolas De Loof.

          I wrote some test ideas if you would like suggestions of areas that need testing. The git plugin supports many different use cases and its automated tests only evaluate a very few of those use cases.

          Show
          markewaite Mark Waite added a comment - The git plugin 2.3.6 pre-release and git client plugin 1.18.0 pre-release are being tested in hopes of releasing new versions before the end of June. If you're willing to assist with the testing, please download and install a pre-release build of the git client plugin and the git plugin . Problems detected in the pre-release should be e-mailed to Mark Waite and Nicolas De Loof . I wrote some test ideas if you would like suggestions of areas that need testing. The git plugin supports many different use cases and its automated tests only evaluate a very few of those use cases.
          Hide
          markewaite Mark Waite added a comment -

          Fix included in git plugin 2.4.0 released 18 July 2015

          Show
          markewaite Mark Waite added a comment - Fix included in git plugin 2.4.0 released 18 July 2015

            People

            • Assignee:
              ndeloof Nicolas De Loof
              Reporter:
              pfalcon Paul Sokolovsky
            • Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: