Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50886

Declarative pipeline with pollSCM triggers 2 builds

    Details

    • Similar Issues:

      Description

      I have a multibranch declarative pipeline with a pollSCM trigger to poll every 2 minutes. Whenever there is a change to the branch in BitBucket it triggers correctly but about 2 minutes later it triggers again. I'm not sure if this is a bug or some configuration error on my behalf.

       

      I've attached a screenshot of the duplicate builds and poll log for one of the duplicate builds. Below is the top of my declarative pipeline for reference to how I am setting it up.

      pipeline {
         agent none
         environment {
            APPVERSION = '0.1.0'
            NUGET_API_KEY = credentials('build-server-nuget-id-readonly')
         }
         triggers {
             // poll repo every 2 minute for changes
             pollSCM('*/2 * * * *')
         }
         options {
             // add timestamps to output
             timestamps()
             overrideIndexTriggers(false)
             buildDiscarder(logRotator(numToKeepStr: '10'))
             skipStagesAfterUnstable()
             durabilityHint('PERFORMANCE_OPTIMIZED')
         }
      ...
      }

       

        Attachments

          Activity

          jonathank Jonathan Kuleff created issue -
          jonathank Jonathan Kuleff made changes -
          Field Original Value New Value
          Labels bitbucket declarative pipeline, pollscm
          jonathank Jonathan Kuleff made changes -
          Description I have a multibranch declarative pipeline with a pollSCM trigger to poll every 2 minutes. Whenever there is a change to the branch in BitBucket it triggers correctly but about 2 minutes later it triggers again. I'm not sure if this is a bug or some configuration error on my behalf.

           

          I've attached a screenshot of the duplicate builds and poll log for one of the duplicate builds. Below is the top of me declarative pipeline for reference to how I am setting it up.
          {code:java}
          pipeline {
             agent none
             environment {
                APPVERSION = '0.1.0'
                NUGET_API_KEY = credentials('build-server-nuget-id-readonly')
             }
             triggers {
                 // poll repo every 2 minute for changes
                 pollSCM('*/2 * * * *')
             }
             options {
                 // add timestamps to output
                 timestamps()
                 overrideIndexTriggers(false)
                 buildDiscarder(logRotator(numToKeepStr: '10'))
                 skipStagesAfterUnstable()
                 durabilityHint('PERFORMANCE_OPTIMIZED')
             }
          ...
          }{code}
           
          I have a multibranch declarative pipeline with a pollSCM trigger to poll every 2 minutes. Whenever there is a change to the branch in BitBucket it triggers correctly but about 2 minutes later it triggers again. I'm not sure if this is a bug or some configuration error on my behalf.

           

          I've attached a screenshot of the duplicate builds and poll log for one of the duplicate builds. Below is the top of my declarative pipeline for reference to how I am setting it up.
          {code:java}
          pipeline {
             agent none
             environment {
                APPVERSION = '0.1.0'
                NUGET_API_KEY = credentials('build-server-nuget-id-readonly')
             }
             triggers {
                 // poll repo every 2 minute for changes
                 pollSCM('*/2 * * * *')
             }
             options {
                 // add timestamps to output
                 timestamps()
                 overrideIndexTriggers(false)
                 buildDiscarder(logRotator(numToKeepStr: '10'))
                 skipStagesAfterUnstable()
                 durabilityHint('PERFORMANCE_OPTIMIZED')
             }
          ...
          }{code}
           
          jonathank Jonathan Kuleff made changes -
          Priority Minor [ 4 ] Major [ 3 ]
          Hide
          abayer Andrew Bayer added a comment -

          So this seems likely to be the git plugin, but not sure. Mark Waite, you seen anything like this?

          Show
          abayer Andrew Bayer added a comment - So this seems likely to be the git plugin, but not sure. Mark Waite , you seen anything like this?
          abayer Andrew Bayer made changes -
          Component/s git-plugin [ 15543 ]
          Component/s bitbucket-plugin [ 18755 ]
          Component/s pipeline [ 21692 ]
          Component/s pollscm-plugin [ 17336 ]
          Assignee Vincent Latombe [ vlatombe ]
          Hide
          markewaite Mark Waite added a comment -

          @abayer since the polling interval is every 2 minutes, I assume the build must complete in much less than 2 minutes. If the build does not complete in 2 minutes, it seems there might be a risk that the same changes detected earlier might be detected again.

          I don't recall a similar bug being reported, though there are several open bugs related to polling scenarios.

          Show
          markewaite Mark Waite added a comment - @abayer since the polling interval is every 2 minutes, I assume the build must complete in much less than 2 minutes. If the build does not complete in 2 minutes, it seems there might be a risk that the same changes detected earlier might be detected again. I don't recall a similar bug being reported, though there are several open bugs related to polling scenarios.
          Hide
          jonathank Jonathan Kuleff added a comment -

          Mark Waite Andrew Bayer the build does not complete in less then 2 minutes.

          Excuse me if I have misunderstood how this feature is meant to work but I am assuming that when a scan runs and it detects a new commit, it triggers a build for the commit and records the commit hash as being triggered/seen. 2 minutes later it scans again if no new commits nothing happens, however if there is a new commit is triggers for that commit.

          From Mark's comment it sounds like the build has to be finished before the next poll.

           

          If there is any other information I can provide to get to the bottom of this, please let me know.

          Show
          jonathank Jonathan Kuleff added a comment - Mark Waite Andrew Bayer the build does not complete in less then 2 minutes. Excuse me if I have misunderstood how this feature is meant to work but I am assuming that when a scan runs and it detects a new commit, it triggers a build for the commit and records the commit hash as being triggered/seen. 2 minutes later it scans again if no new commits nothing happens, however if there is a new commit is triggers for that commit. From Mark's comment it sounds like the build has to be finished before the next poll.   If there is any other information I can provide to get to the bottom of this, please let me know.
          Hide
          markewaite Mark Waite added a comment -

          Jonathan Kuleff I can't duplicate the problem you're reporting.

          Steps I took trying to duplicate the problem:

          1. Created a JENKINS-50886 branch in my jenkins-bugs repository so that the job will be created in my multi-branch Pipelines
          2. Define an ant script that sleeps for MINUTES_TO_WAIT=13 + 30 seconds
          3. Define a scripted Pipeline in the Jenkinsfile that defines polling interval as MINUTES_TO_WAIT
          4. Scan the repository to detect the new branch
          5. Confirm that a single build is run when a change is detected and that no additional builds are run until the next polling interval detects changes

          I assumed that the difference between scripted and declarative is not relevant in this case.

          I assumed that a multi-branch Pipeline was a sufficient approximation for your case. Are you using a Pipeline (without multi-branch) or are you using multi-branch Pipeline?

          I was able to cause two concurrent builds of the same commit if I used curl to the Jenkins notifyCommit URL shortly after the commit happened. Since you're using polling, I assume you're not using notifyCommit and I assume you're not using webhooks. If you are using notifyCommit or webhooks, then you don't need polling.

          Show
          markewaite Mark Waite added a comment - Jonathan Kuleff I can't duplicate the problem you're reporting. Steps I took trying to duplicate the problem: Created a JENKINS-50886 branch in my jenkins-bugs repository so that the job will be created in my multi-branch Pipelines Define an ant script that sleeps for MINUTES_TO_WAIT=13 + 30 seconds Define a scripted Pipeline in the Jenkinsfile that defines polling interval as MINUTES_TO_WAIT Scan the repository to detect the new branch Confirm that a single build is run when a change is detected and that no additional builds are run until the next polling interval detects changes I assumed that the difference between scripted and declarative is not relevant in this case. I assumed that a multi-branch Pipeline was a sufficient approximation for your case. Are you using a Pipeline (without multi-branch) or are you using multi-branch Pipeline? I was able to cause two concurrent builds of the same commit if I used curl to the Jenkins notifyCommit URL shortly after the commit happened. Since you're using polling, I assume you're not using notifyCommit and I assume you're not using webhooks. If you are using notifyCommit or webhooks, then you don't need polling.
          Hide
          jonathank Jonathan Kuleff added a comment -

          Mark Waite thanks for looking at that. I'll try to reproduce with a simplified jenkinsfile for you.

           

          Here is what I did to try reproduce it:

          1. Created Bitbucket as a private repo
          2. Created a `feature/JENKINS-50886` branch and added the Jenkinsfile (declarative pipeline) and README.md (nothing special)
          3. Created a new pipeline in BlueOcean
          4. Jumped out to classic and turned on `Scan Multibranch Pipeline Triggers -> Periodically if not otherwise run` with an interval of 5 minutes, so new branches are picked up automatically

           

          This is were I have hit a road block, for some reason polling is not picking up changes, even though there are some.

           

          Poll Log
          
          Started on May 12, 2018 3:48:31 PM
          Done. Took 0 ms
          No changes

           

           

          Scan Multibranch Pipeline Log
          
          Started by timer
          [Sat May 12 15:49:00 GMT 2018] Starting branch indexing...
          Connecting to https://bitbucket.org using builder/****** (Bitbucket server credentials)
          Repository type: Git
          Looking up repo/jenkins-bug for branches
          Checking branch feature/JENKINS-50886 from repo/jenkins-bug
                ‘Jenkinsfile’ found
              Met criteria
          Changes detected: feature/JENKINS-50886 (dc08795440200d86ee6dfe67b9f1bcefa44f2e40 → 5dbf835960f9b4a599fed8b4dd2903795e334194)
          Scheduled build for branch: feature/JENKINS-50886
          Checking branch master from repo/jenkins-bug
                ‘Jenkinsfile’ found
              Met criteria
          No changes detected: master (still at 5883611d9e07b7595051a9aca1227920b42e747d)
          
            2 branches were processed
          Looking up repo/jenkins-bug for pull requests
          
            0 pull requests were processed
          [Sat May 12 15:49:26 GMT 2018] Finished branch indexing. Indexing took 25 sec
          Finished: SUCCESS

           

          Any idea why polling is not working or recommendations on where to look? One thing I noticed is that this new pipeline has `Polling Log` while existing have `Git Polling Log`

          Show
          jonathank Jonathan Kuleff added a comment - Mark Waite thanks for looking at that. I'll try to reproduce with a simplified jenkinsfile for you.   Here is what I did to try reproduce it: Created Bitbucket as a private repo Created a `feature/ JENKINS-50886 ` branch and added the Jenkinsfile (declarative pipeline) and README.md (nothing special) Created a new pipeline in BlueOcean Jumped out to classic and turned on `Scan Multibranch Pipeline Triggers -> Periodically if not otherwise run` with an interval of 5 minutes, so new branches are picked up automatically   This is were I have hit a road block, for some reason polling is not picking up changes, even though there are some.   Poll Log Started on May 12, 2018 3:48:31 PM Done. Took 0 ms No changes     Scan Multibranch Pipeline Log Started by timer [Sat May 12 15:49:00 GMT 2018] Starting branch indexing... Connecting to https://bitbucket.org using builder/****** (Bitbucket server credentials) Repository type: Git Looking up repo/jenkins-bug for branches Checking branch feature/JENKINS-50886 from repo/jenkins-bug ‘Jenkinsfile’ found Met criteria Changes detected: feature/JENKINS-50886 (dc08795440200d86ee6dfe67b9f1bcefa44f2e40 → 5dbf835960f9b4a599fed8b4dd2903795e334194) Scheduled build for branch: feature/JENKINS-50886 Checking branch master from repo/jenkins-bug ‘Jenkinsfile’ found Met criteria No changes detected: master (still at 5883611d9e07b7595051a9aca1227920b42e747d) 2 branches were processed Looking up repo/jenkins-bug for pull requests 0 pull requests were processed [Sat May 12 15:49:26 GMT 2018] Finished branch indexing. Indexing took 25 sec Finished: SUCCESS   Any idea why polling is not working or recommendations on where to look? One thing I noticed is that this new pipeline has `Polling Log` while existing have `Git Polling Log`
          Hide
          markewaite Mark Waite added a comment -

          Jonathan Kuleff you may be encountering a bug that polling may not always correctly detect branches of the form parent/child. The plugin has some awkward spots trying to handle the difference between origin/master and feature/new-thing. Try without the "feature", or try fully qualifying the branch name as origin/feature/JENKINS-50886.

          Show
          markewaite Mark Waite added a comment - Jonathan Kuleff you may be encountering a bug that polling may not always correctly detect branches of the form parent/child . The plugin has some awkward spots trying to handle the difference between origin/master and feature/new-thing . Try without the "feature", or try fully qualifying the branch name as origin/feature/ JENKINS-50886 .
          Hide
          jonathank Jonathan Kuleff added a comment -

          Mark Waite this has mostly fixed itself up. We are using AWS EFS to back our container cluster that Jenkins is running on and it turns out it didn't have enough IOPS credits, a recently feature that allows us to provision IOPS has seems to fix this up for us bar the occasional double build.

          Show
          jonathank Jonathan Kuleff added a comment - Mark Waite this has mostly fixed itself up. We are using AWS EFS to back our container cluster that Jenkins is running on and it turns out it didn't have enough IOPS credits, a recently feature that allows us to provision IOPS has seems to fix this up for us bar the occasional double build.
          markewaite Mark Waite made changes -
          Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
          Resolution Not A Defect [ 7 ]
          markewaite Mark Waite made changes -
          Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
          Hide
          ascheman Gerd Aschemann added a comment - - edited

          I still encounter this problem every now and then. On a public Jenkins you can see it occasionally here: https://dev.dukecon.org/jenkins/job/dukecon_pwa/job/develop/, the respective Jenkinsfile is here: https://github.com/dukecon/dukecon_pwa/blob/develop/Jenkinsfile with 

          options { 
            disableConcurrentBuilds() 
            buildDiscarder(logRotator( numToKeepStr: '5', daysToKeepStr: '5'))
          }
          
          triggers { 
            pollSCM('* * * * *')
          }
           
          Show
          ascheman Gerd Aschemann added a comment - - edited I still encounter this problem every now and then. On a public Jenkins you can see it occasionally here:  https://dev.dukecon.org/jenkins/job/dukecon_pwa/job/develop/ , the respective Jenkinsfile is here:  https://github.com/dukecon/dukecon_pwa/blob/develop/Jenkinsfile  with  options { disableConcurrentBuilds() buildDiscarder(logRotator( numToKeepStr: '5' , daysToKeepStr: '5' )) } triggers { pollSCM( '* * * * *' ) }
          ascheman Gerd Aschemann made changes -
          Resolution Not A Defect [ 7 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          priyatham Priyatham Tadikonda added a comment -

          I encounter this issue too. Do we have a solution for this yet? I'm using Multibranch Pipeline project and Github repostitory. We are polling every 5minutes but the build takes more than 30mins to complete.Duplicate Builds are being triggered with the same  commit ID.

          Please help me.

           

          Thank you.

          Show
          priyatham Priyatham Tadikonda added a comment - I encounter this issue too. Do we have a solution for this yet? I'm using Multibranch Pipeline project and Github repostitory. We are polling every 5minutes but the build takes more than 30mins to complete.Duplicate Builds are being triggered with the same  commit ID. Please help me.   Thank you.
          Hide
          markewaite Mark Waite added a comment -

          Priyatham Tadikonda no solution in progress and no solution proposed. We recommend that webhooks are better than polling (faster response, lighter load on the git server). See "polling must die" by Kohsuke Kawaguchi for more details. If polling is truly the only way to achieve your goal, then you might also consider setting the polling interval to be greater than the average build time.

          Show
          markewaite Mark Waite added a comment - Priyatham Tadikonda no solution in progress and no solution proposed. We recommend that webhooks are better than polling (faster response, lighter load on the git server). See "polling must die" by Kohsuke Kawaguchi for more details. If polling is truly the only way to achieve your goal, then you might also consider setting the polling interval to be greater than the average build time.
          markewaite Mark Waite made changes -
          Status Reopened [ 4 ] Open [ 1 ]
          Hide
          ganthore Mark Austin added a comment - - edited

          I'm seeing this happen with one of my teams using the multi-branch pipeline setup. Two different people are able to trigger the problem rather easily, but I can't for the life of me reproduce it, so this leads me to question if local browser configs/plugins are contributing to the problem.

          Now my use case is a little different since I don't even want polling, but I've noticed that PollSCM is switched on by default with any multibranch pipeline I setup.

          Show
          ganthore Mark Austin added a comment - - edited I'm seeing this happen with one of my teams using the multi-branch pipeline setup. Two different people are able to trigger the problem rather easily, but I can't for the life of me reproduce it, so this leads me to question if local browser configs/plugins are contributing to the problem. Now my use case is a little different since I don't even want polling, but I've noticed that PollSCM is switched on by default with any multibranch pipeline I setup.

            People

            • Assignee:
              Unassigned
              Reporter:
              jonathank Jonathan Kuleff
            • Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: