Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61845

Poll SCM enqueue a build when a build for the same commit is already running

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: git-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      We have a job with "Poll SCM" running on `* * * * *` (every minute). We need to use "Included Regions" so the local polling strategy is used. The build takes about 30 minutes. It correctly ignores changes in the steady state but when a commit comes in, it correctly notices a change saying something like:

      [poll] Last Built Revision: Revision 50fc7882752856ac86b0838fdf878897e17955cd (refs/remotes/origin/master)
      Done. Took 6 ms
      Changes found
      

      and starts the build. But then 1 minute later, it notices the same thing and starts another build. These new builds continue to be started until one of them finishes and then the queueing stops.

      I've tried all sorts of combinations to try and get new jobs to not start with the exact same SHA but to no avail. The closest I can get is to let only 1 concurrent build go at a time, but that still leaves 1 job in the queue and we end up double building everything.

      Is there a way to have the SCM polling take into account existing builds?

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          I'm not aware of any way to have git plugin polling account for running builds. As far as I know, it accounts for completed builds, but not for running builds.

          Best choice is to not poll. Refer to Kohsuke's blog post for a much less resource intensive solution that avoids polling. Even better is to use web hooks from the git provider. GitHub, Bitbucket, GitLab, and Gitea all provide web hooks.

          Second best choice is to poll no more often than the build duration (assuming you do not want concurrent builds running).

          Third best choice is to disable concurrent builds. That is the default with Freestyle jobs and can be enabled with declarative pipeline using disableConcurrentBuilds option and can be enabled with scripted pipeline using the disableConcurrentBuilds property.

          Show
          markewaite Mark Waite added a comment - I'm not aware of any way to have git plugin polling account for running builds. As far as I know, it accounts for completed builds, but not for running builds. Best choice is to not poll. Refer to Kohsuke's blog post for a much less resource intensive solution that avoids polling. Even better is to use web hooks from the git provider. GitHub, Bitbucket, GitLab, and Gitea all provide web hooks. Second best choice is to poll no more often than the build duration (assuming you do not want concurrent builds running). Third best choice is to disable concurrent builds. That is the default with Freestyle jobs and can be enabled with declarative pipeline using disableConcurrentBuilds option and can be enabled with scripted pipeline using the disableConcurrentBuilds property .
          Hide
          ptarjan Paul Tarjan added a comment - - edited

          Thanks Mark.

          I would very much love to stop polling but our security policy doesn't allow our Jenkins instance to be hittable by GitHub.

          If I increase the polling time to 30 minutes, then my average build time would go up by 15 minutes because after someone pushes it has to wait a while before it checks again. And even with that, we'll be rebuilding if the job takes 31 minutes.

          For your last option I've tried that but it still seems to enqueue another build even thought it won't build. Do I have something setup wrong is that expected behavior? I've tried both "Do not allow concurrent builds" and "Throttle Concurrent Builds" with a "Maximum Total Concurrent Builds" == 1.

          Show
          ptarjan Paul Tarjan added a comment - - edited Thanks Mark. I would very much love to stop polling but our security policy doesn't allow our Jenkins instance to be hittable by GitHub. If I increase the polling time to 30 minutes, then my average build time would go up by 15 minutes because after someone pushes it has to wait a while before it checks again. And even with that, we'll be rebuilding if the job takes 31 minutes. For your last option I've tried that but it still seems to enqueue another build even thought it won't build. Do I have something setup wrong is that expected behavior? I've tried both "Do not allow concurrent builds" and "Throttle Concurrent Builds" with a "Maximum Total Concurrent Builds" == 1.
          Hide
          markewaite Mark Waite added a comment -

          I don't know of a way to avoid that second build. Today, when the git plugin finds a commit on the remote that matches a branch it is tracking, then a build is scheduled for that commit if the commit has not already been built.

          If developers have network access to the Jenkins server, then you might experiment with placing a pre-push hook inside each developer repository. That pre-push hook would notify the Jenkins server that a push is happening. I've not tried it myself, but I have used post-receive hooks on the server quite successfully.

          Show
          markewaite Mark Waite added a comment - I don't know of a way to avoid that second build. Today, when the git plugin finds a commit on the remote that matches a branch it is tracking, then a build is scheduled for that commit if the commit has not already been built. If developers have network access to the Jenkins server, then you might experiment with placing a pre-push hook inside each developer repository. That pre-push hook would notify the Jenkins server that a push is happening. I've not tried it myself, but I have used post-receive hooks on the server quite successfully.
          Hide
          mwinter69 Markus Winter added a comment -

          As a workaround you could use a small poller job that just triggers the original job and immediately terminates.

          Show
          mwinter69 Markus Winter added a comment - As a workaround you could use a small poller job that just triggers the original job and immediately terminates.

            People

            • Assignee:
              Unassigned
              Reporter:
              ptarjan Paul Tarjan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: