Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55714

Git plugin scm polling trigger not working on first build

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • Jenkins v2.148
      Git plugin v3.9.1

      We have an odd problem with Git SCM polling on some of our jenkins projects. The projects in question only seem to be affected when they are built for the first time. These projects are of type "pipeline" and they always have multiple Git repositories being checked out.

      The build configurations are somewhat complex, but here are the variables that I think are contributing to the problem. First off, we have multiple stages for these builds, each of which checks out several of the same git repos. Further, to ensure that each stage checks out the same copy / revision of these repos we first check out the head revision from a given refspec on the first stage and then cache the revision / hash from that checkout, and then use that hash when checking out subsequent copies. In pseudo-code, the mechanics look something like this:

      def cached_rev = '' 
      stage ("stage 1") {
           node {
               checkout (scm: [class: '$GitSCM', branches: 'master'...)
               cached_rev = sh(returnStdout: true, script: 'git rev-parse HEAD').trim()
               // do some build stuff
           }
      }
      stage ("stage 2") {
           node {
               checkout (scm: [class: '$GitSCM', branches: cached_rev...])
               // do more build stuff
           }
      } 

      The mechanics of this configuration work as expected at built time (ie: the correct / same revisions get checked out for each stage), however the polling mechanics don't work. For some reason changes to repos configured in this way always detect changes in them which causes builds to run over and over and over again, in perpetuity.

      One other factor that may play a role here is that we also dynamically set the `changelog` attribute on each SCM checkout. We set the flag to `true` in the first stage, and then to `false` on all subsequent stages. We do this to try and prevent the change logs from these shared repositories from showing up many times in duplicate on the build summary page (ie: if we have 5 stages configured like this, all sharing the same repo configuration, then changes to this repo typically show up 5 times in the build summary page, which is annoying at best).

      One other interesting piece of the puzzle here as well is that we have tried setting the `poll: false` option on these common Git repos as well to try and circumvent this strange pollng behavior. However, interestingly enough, this seems to have no effect. It is almost like the poll flag is being ignored in certain cases or perhaps confusing the Git plugin in certain situations. 

      Another interesting piece of the puzzle is that the problematic behavior only seems to happen when we build a project for the first time. I've noticed the message "no polling baseline" several times in the polling logs for these builds, which seems to suggest to me that perhaps the Git plugin is doing something different when a job is first built, maybe because it has no pre-existing workspace to use as a reference for pulling Git revisions. It seems a little odd to me that Jenkins would be using the workspace for polling data seeing as how the workspace may exist on any number of build agents attached to the same build farm, some of which may be more recent than others. Also, should the build agent used for the most recent build go offline then that could introduce instability in the logic as well. I'm not 100% convinced this specific message contributes to the problem I'm reporting here. I'm just mentioning it in case it's relevant in some way.

      Finally, we've tried several things to "work around" this problem in our production environment. Killing duplicate builds that get triggered yields no effect. The next SCM trigger operation still picks up changes from earlier builds and flags them as "new". We've tried killing newly triggered builds from the queue before they start building, but subsequent SCM triggers still kick in. The only effective workaround we've found is if we disable the jobs that exhibit the problem (ie: using Groovy API calls), leaving the job disabled until the next build trigger kicks in, and then re-enable the job, the SCM triggers seem to correct themselves and the problem ceases to exist. Subsequent build operations proceed as normal.

      Any help anyone may have that could explain this problem or how to fix it would be greatly appreciated. 

       

            Unassigned Unassigned
            leedega Kevin Phillips
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: