Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50191

Multibranch Pipeline tends to keep rebuilding same commits

    Details

    • Similar Issues:

      Description

      We have a number of Pipeline jobs set up with automatic generation through Organization folders (both Stash and Github) and MultiBranch Pipelines generated from those (and finally PR/branch leaf jobs wherever Jenkinsfile's are found). Our default pipeline template includes an SCM polling setup (ranging from H/2 to H/15) because we don't have webhook ability due to corporate firewalling.

      Quite often I see builds of the same commit IDs running in parallel, or even the later build starting hours after an earlier one has finished (even if successfully). They often have different causes, such as "Started by an SCM change" vs. "Branch indexing".

      If a branch or PR has errors building (e.g. bad timing - new commit noticed in the original branch for a PR, but PR merged or closed during the half a minute or so while a build is preparing, so code can not be checked out - but MBP re-scanning did not yet happen to disable the job), then we can have hundreds of such failed rebuild attempts.

      During FOSDEM-2018 discussion with Andrew Bayer and Mark Waite (I'm not sure if we logged a ticket at that time) we concluded that such behavior is a bug, and that Jenkins is supposed to track in some matter which source commits it had processed (or even began processing - has queued or is building) and should avoid such automated duplicate re-runs of the same workload, regardless of whether the workspace (on an agent or master) is persistent or removed after each build.

      This does not happen often (e.g. every time), but sufficiently frequently to be noticeable and annoying (and sometimes expensive - e.g. if this kicks off the avalanche of product build and re-testing that costs several hours of blocked resources here and there).

        Attachments

          Activity

          Hide
          danielbeck Daniel Beck added a comment -

          Checkout needs to have finished for a build to be able to make subsequent SCM operations consider the commit already handled. Is this the problem? Huge checkouts, or perhaps even checkouts that aren't happening at the start of the build?

          Show
          danielbeck Daniel Beck added a comment - Checkout needs to have finished for a build to be able to make subsequent SCM operations consider the commit already handled. Is this the problem? Huge checkouts, or perhaps even checkouts that aren't happening at the start of the build?
          Hide
          apgray Andrew Gray added a comment - - edited

          Seeing the same behaviour on Windows and latest LTS release 2.138.2 (however have seen the same behaviour since at least 2.107.x).

          Our situation:

          Multibranch pipeline (via a pollSCM(* * * * *) in the Jenkinsfile) will detect an SCM change (size of commit is irrelevant) and kick off a build.  Within a minute another build will be kicked off under the auspices of "Branch Indexing" (via the "Scan Multibranch Pipeline Triggers" setting).

          This second duplicate "Branch Indexing build" is buggy behaviour.  The pollSCM renders the branch Indexing build redundant.  Messy workaround is to manually cancel second duplicate build.

          Show
          apgray Andrew Gray added a comment - - edited Seeing the same behaviour on Windows and latest LTS release 2.138.2 (however have seen the same behaviour since at least 2.107.x). Our situation: Multibranch pipeline (via a pollSCM(* * * * *) in the Jenkinsfile) will detect an SCM change (size of commit is irrelevant) and kick off a build.  Within a minute another build will be kicked off under the auspices of "Branch Indexing" (via the "Scan Multibranch Pipeline Triggers" setting). This second duplicate "Branch Indexing build" is buggy behaviour.  The pollSCM renders the branch Indexing build redundant.  Messy workaround is to manually cancel second duplicate build.
          Hide
          nikhilnanal nikhil nanal added a comment -

          I am facing the same issue with a multibranch pipeline (Jenkins 2.164, git hub branch source plugin 2.5.3, git plugin 3.10.0, scm -api 2.4.1). I am using scm polling enabled via the job configuration. If a PR build fails then it gets restarted arbitrarily during a subsequent polling interval even though there was no new change or pull request opened. These restarted PR builds hog my executors and if  a subsequent new PR comes in it has to wait  for the existing builds to complete or I have to manually cancel them. This is a serious problem for us. 

          I am using "Merging the pull request with current target branch revision" option in my builds. could this be happening if a PR failed but other PRs passed which caused the target revision to be updated and hence the failed PR also gets retriggered?

          Show
          nikhilnanal nikhil nanal added a comment - I am facing the same issue with a multibranch pipeline (Jenkins 2.164, git hub branch source plugin 2.5.3, git plugin 3.10.0, scm -api 2.4.1). I am using scm polling enabled via the job configuration. If a PR build fails then it gets restarted arbitrarily during a subsequent polling interval even though there was no new change or pull request opened. These restarted PR builds hog my executors and if  a subsequent new PR comes in it has to wait  for the existing builds to complete or I have to manually cancel them. This is a serious problem for us.  I am using "Merging the pull request with current target branch revision" option in my builds. could this be happening if a PR failed but other PRs passed which caused the target revision to be updated and hence the failed PR also gets retriggered?
          Hide
          jimklimov Jim Klimov added a comment -

          For the bit about PR-merge builds, yes this is expected to re-run (once per seen target branch update though!) and part of the design idea about such ephemeral merge branches.

          Show
          jimklimov Jim Klimov added a comment - For the bit about PR-merge builds, yes this is expected to re-run (once per seen target branch update though!) and part of the design idea about such ephemeral merge branches.

            People

            • Assignee:
              Unassigned
              Reporter:
              jimklimov Jim Klimov
            • Votes:
              6 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated: