Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36195

Pipeline polling ignores special polling rules

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The git plugin currently offers several special rules on polling:

      1. Polling ignores commits from certain users
      2. Polling ignores commits from certain paths (Included/Excluded regions)
      3. Polling ignores commits with certain messages
      4. Branch specifier

      For example:
      checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'UserExclusion', excludedUsers: 'dvader'], [$class: 'PathRestriction', excludedRegions: 'One/.*', includedRegions: 'Test/.*'], [$class: 'MessageExclusion', excludedMessage: '.\\[maven-release-plugin
      ].
      ']], gitTool: 'Default', submoduleCfg: [], userRemoteConfigs: [[credentialsId: '...', url: 'ssh://git@bitbucket.org/test.git']]])

      If a commit is committed to bitbucket.org/test.git, regardless of the user, location, branch or message, the step that includes the checkout will be run.

      Similar issue has been reported in:

        Attachments

          Issue Links

            Activity

            dvader Darth Vader created issue -
            dvader Darth Vader made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-35988 [ JENKINS-35988 ]
            markewaite Mark Waite made changes -
            Assignee Mark Waite [ markewaite ]
            Hide
            markewaite Mark Waite added a comment -

            Since the git plugin doesn't read the contents of Jenkinsfile when performing polling, I'm not sure how it would honor the polling settings inside the Jenkinsfile in the repository. It seems like it would be even more challenging in a repository with more than one branch, since the polling settings might differ between branches within the same repository.

            I wonder if Jesse Glick has any ideas how to handle this case and JENKINS-35988 (and others like it).

            Show
            markewaite Mark Waite added a comment - Since the git plugin doesn't read the contents of Jenkinsfile when performing polling, I'm not sure how it would honor the polling settings inside the Jenkinsfile in the repository. It seems like it would be even more challenging in a repository with more than one branch, since the polling settings might differ between branches within the same repository. I wonder if Jesse Glick has any ideas how to handle this case and JENKINS-35988 (and others like it).
            Hide
            azry daryl piffre added a comment -

            I'm having the same problem as well.
            Poll still triggers after setting excluded messages/users
            It's kinda important to us and probably others because we use maven-release-plugin, as shown in the example in the snippet generator.

            Show
            azry daryl piffre added a comment - I'm having the same problem as well. Poll still triggers after setting excluded messages/users It's kinda important to us and probably others because we use maven-release-plugin, as shown in the example in the snippet generator.
            dvader Darth Vader made changes -
            Assignee Jesse Glick [ jglick ]
            Hide
            dvader Darth Vader added a comment -

            Jesse Glick Any comment on this issue? I can't imagine being the only one with this problem. This is really the only thing preventing us to move to a pipeline/multi-branch workflow. I know that pipelines are the future of Jenkins so we really want to make the switch so we can benefit from all the recent improvements in Jenkins.

            Show
            dvader Darth Vader added a comment - Jesse Glick Any comment on this issue? I can't imagine being the only one with this problem. This is really the only thing preventing us to move to a pipeline/multi-branch workflow. I know that pipelines are the future of Jenkins so we really want to make the switch so we can benefit from all the recent improvements in Jenkins.
            Hide
            prestonj Preston Jennings added a comment -

            +1

            For me, it's the polling ignores that are configured in the job itself and not the Jenkinsfile.

            Show
            prestonj Preston Jennings added a comment - +1 For me, it's the polling ignores that are configured in the job itself and not the Jenkinsfile.
            Hide
            markewaite Mark Waite added a comment -

            Preston Jennings you're describing a condition which has a very different root case compared to the conditions which cause this bug report. Please find an existing bug report which describes cases where a job ignores polling exclusion rules which are defined within the job, or submit a new bug.

            Show
            markewaite Mark Waite added a comment - Preston Jennings you're describing a condition which has a very different root case compared to the conditions which cause this bug report. Please find an existing bug report which describes cases where a job ignores polling exclusion rules which are defined within the job, or submit a new bug.
            Hide
            prestonj Preston Jennings added a comment -

            Mark Waite Understood. I've created a separate defect as I couldn't find anything specific to my case: https://issues.jenkins-ci.org/browse/JENKINS-36836

            Show
            prestonj Preston Jennings added a comment - Mark Waite Understood. I've created a separate defect as I couldn't find anything specific to my case: https://issues.jenkins-ci.org/browse/JENKINS-36836
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 172856 ] JNJira + In-Review [ 184814 ]
            abayer Andrew Bayer made changes -
            Component/s pipeline-general [ 21692 ]
            abayer Andrew Bayer made changes -
            Component/s workflow-plugin [ 18820 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-36836 [ JENKINS-36836 ]
            Hide
            jglick Jesse Glick added a comment -

            I'm not sure how it would honor the polling settings inside the Jenkinsfile in the repository.

            This is automatic, based on the checkout done in the last build.

            Not sure offhand, should work, since Pipeline supports workspace-based polling, but perhaps there is some bug, Git-specific or not, and missing test coverage.

            Generally speaking, polling exclusions are discouraged for Git since the plugin does not currently support them without reverting to workspace-based polling, which is always bad. AbstractGitSCMSource does define a caching system which could be used to fix that, but non-multibranch parts of the Git plugin do not currently take advantage of it.

            Show
            jglick Jesse Glick added a comment - I'm not sure how it would honor the polling settings inside the Jenkinsfile in the repository. This is automatic, based on the checkout done in the last build. Not sure offhand, should work, since Pipeline supports workspace-based polling , but perhaps there is some bug, Git-specific or not, and missing test coverage. Generally speaking, polling exclusions are discouraged for Git since the plugin does not currently support them without reverting to workspace-based polling, which is always bad. AbstractGitSCMSource does define a caching system which could be used to fix that, but non-multibranch parts of the Git plugin do not currently take advantage of it.
            jglick Jesse Glick made changes -
            Component/s workflow-job-plugin [ 21716 ]
            Component/s workflow-scm-step-plugin [ 21717 ]
            Component/s pipeline [ 21692 ]
            Hide
            jglick Jesse Glick added a comment -

            Not to be confused with JENKINS-35988: multibranch indexing does not use SCM.compareRemoteRevisionWith at all.

            Show
            jglick Jesse Glick added a comment - Not to be confused with JENKINS-35988 : multibranch indexing does not use SCM.compareRemoteRevisionWith at all.
            Hide
            dvader Darth Vader added a comment -

            Jesse Glick Thanks for chiming in. I'm not sure I understand all the comments other than it should work. I use polling but in a sneaky way: I use Bitbucket server with an add-on that triggers Jenkins job git polling. The job is set to poll but on an infinite schedule (never). So I get job triggering only when the git repository is updated but with all the git polling rules being observed. I use the git caching too to speed up the data transfers (I fetch updates to the repository once daily using a Jenkins job).
            Mark Waite Jesse has chimed in. Any comments on his remarks?

            Show
            dvader Darth Vader added a comment - Jesse Glick Thanks for chiming in. I'm not sure I understand all the comments other than it should work. I use polling but in a sneaky way: I use Bitbucket server with an add-on that triggers Jenkins job git polling. The job is set to poll but on an infinite schedule (never). So I get job triggering only when the git repository is updated but with all the git polling rules being observed. I use the git caching too to speed up the data transfers (I fetch updates to the repository once daily using a Jenkins job). Mark Waite Jesse has chimed in. Any comments on his remarks?
            Hide
            markewaite Mark Waite added a comment -

            No comments from me without a detailed investigation. There are higher priority bugs in the git plugin which will get my attention before this bug, and several pull requests which will get my attention before this bug. My apologies, but I won't do a detailed investigation of this bug report for a long time.

            If it is crucial to you, you're encouraged to fork the git plugin source code, write a test which shows the failure, write the fix, and then submit the pull request.

            Show
            markewaite Mark Waite added a comment - No comments from me without a detailed investigation. There are higher priority bugs in the git plugin which will get my attention before this bug, and several pull requests which will get my attention before this bug. My apologies, but I won't do a detailed investigation of this bug report for a long time. If it is crucial to you, you're encouraged to fork the git plugin source code, write a test which shows the failure, write the fix, and then submit the pull request.
            Hide
            mtraynham Matt Traynham added a comment - - edited

            Mark Waite, this has been a fairly bothersome bug for my team. Most of our pipelines are set up to do maven releases, which can be a bit chatty with Git commit/push; one for the release version commit, a second for the dependency version bump. The two extra concurrent builds can create heavy load on the machine, sometimes causing the release build to fail which can leave our SCM and Nexus repos in a weird limbo state.

            There are two potential workarounds we've identified to getting this down to a single extra build. Either we restrict the maven push til the very end or attempt to filter out webhook calls from our repo service. Both are a bit of a hack, and we were trying to gauge this bug, ultimately filtering out certain commits with particular messages/users.

            Could you make a recommendation on any other potential solutions, or point me in the right direction of which projects/code would likely need to change to get this fixed? We are using multi-branch for this, so that may complicate our position a bit further as mentioned above, it doesn't use workspace polling.

            Show
            mtraynham Matt Traynham added a comment - - edited Mark Waite , this has been a fairly bothersome bug for my team. Most of our pipelines are set up to do maven releases, which can be a bit chatty with Git commit/push; one for the release version commit, a second for the dependency version bump. The two extra concurrent builds can create heavy load on the machine, sometimes causing the release build to fail which can leave our SCM and Nexus repos in a weird limbo state. There are two potential workarounds we've identified to getting this down to a single extra build. Either we restrict the maven push til the very end or attempt to filter out webhook calls from our repo service. Both are a bit of a hack, and we were trying to gauge this bug, ultimately filtering out certain commits with particular messages/users. Could you make a recommendation on any other potential solutions, or point me in the right direction of which projects/code would likely need to change to get this fixed? We are using multi-branch for this, so that may complicate our position a bit further as mentioned above, it doesn't use workspace polling.
            Hide
            markewaite Mark Waite added a comment -

            Matt Traynham the git-plugin is the most likely place that needs the change. The AbstractGitSCMSource class is the one that is used by the pipeline code for its git operations. That seems like the first place to start looking. There are various guides to getting started as a Jenkins plugin developer, including plugin tutorial and Extending Jenkins. The jenkins-dev mailing list is also a good place for help.

            Show
            markewaite Mark Waite added a comment - Matt Traynham the git-plugin is the most likely place that needs the change. The AbstractGitSCMSource class is the one that is used by the pipeline code for its git operations. That seems like the first place to start looking. There are various guides to getting started as a Jenkins plugin developer, including plugin tutorial and Extending Jenkins . The jenkins-dev mailing list is also a good place for help.
            Hide
            jglick Jesse Glick added a comment -

            Most of our pipelines are set up to do maven releases, which can be a bit chatty with Git commit/push; one for the release version commit, a second for the dependency version bump. The two extra concurrent builds can create heavy load

            Ultimately the Maven release plugin is not really compatible with CI workflows.

            We are using multi-branch for this, so that may complicate our position

            This bug is about regular polling, which is supposed to support polling rules. Multibranch scanning definitely does not support polling rules. See JENKINS-35988.

            Show
            jglick Jesse Glick added a comment - Most of our pipelines are set up to do maven releases, which can be a bit chatty with Git commit/push; one for the release version commit, a second for the dependency version bump. The two extra concurrent builds can create heavy load Ultimately the Maven release plugin is not really compatible with CI workflows. We are using multi-branch for this, so that may complicate our position This bug is about regular polling, which is supposed to support polling rules. Multibranch scanning definitely does not support polling rules. See JENKINS-35988 .
            Hide
            vorobievalex Alexander Vorobiev added a comment -

            Can this functionality be moved to later stages?
            So some steps can be skipped in case there are no appropriate changes as described in JENKINS-37978.

            Show
            vorobievalex Alexander Vorobiev added a comment - Can this functionality be moved to later stages? So some steps can be skipped in case there are no appropriate changes as described in JENKINS-37978 .
            Hide
            jlehtnie Jukka Lehtniemi added a comment - - edited

            At least Included/Excluded regions actually works ok if you also specify Disable Remote Poll, e.g.:

            checkout( scm: [ ... ,
                extensions: [[$class: 'DisableRemotePoll'], [$class: 'PathRestriction', excludedRegions: '', includedRegions: 'foo/.*']], ...
            ])
            
            Show
            jlehtnie Jukka Lehtniemi added a comment - - edited At least Included/Excluded regions actually works ok if you also specify Disable Remote Poll , e.g.: checkout( scm: [ ... , extensions: [[$class: 'DisableRemotePoll' ], [$class: 'PathRestriction' , excludedRegions: '', includedRegions: ' foo/.*']], ... ])
            Hide
            killdash9 Michael Andrews added a comment -

            Yes. polling REGION inclusion/exclusion is working. The USER inclusion/exclusion is not working. We too have the issue of automatic releases where the release is tagged and the version is bumped by Jenkins. This results in a builds after the push. In Jenkins 1.x this was easily remedied using the user exclusions. Hoping a fix is in the works.

            Show
            killdash9 Michael Andrews added a comment - Yes. polling REGION inclusion/exclusion is working. The USER inclusion/exclusion is not working. We too have the issue of automatic releases where the release is tagged and the version is bumped by Jenkins. This results in a builds after the push. In Jenkins 1.x this was easily remedied using the user exclusions. Hoping a fix is in the works.
            Hide
            markewaite Mark Waite added a comment -

            No fix is in the works as far as I know. This particular use case is quite far down my priority list (behind large file support, submodule bug fixes, and authentication issues).

            Show
            markewaite Mark Waite added a comment - No fix is in the works as far as I know. This particular use case is quite far down my priority list (behind large file support, submodule bug fixes, and authentication issues).
            Hide
            killdash9 Michael Andrews added a comment -

            Mark Waite Ok. Thanks for the update.

            Show
            killdash9 Michael Andrews added a comment - Mark Waite Ok. Thanks for the update.
            Hide
            vanta Krzysztof Wolny added a comment -

            Can we also make it possible to pass properties (like excludedUsers) from git step to checkout step, as currently git step allows only for url, branch and credentialsId. It would make life a lot easier, as we wouldn't have to switch from git to checkout (which is less readable) just to pass some property.

            Show
            vanta Krzysztof Wolny added a comment - Can we also make it possible to pass properties (like excludedUsers ) from git step to checkout step, as currently git step allows only for url , branch and credentialsId . It would make life a lot easier, as we wouldn't have to switch from git to checkout (which is less readable) just to pass some property.
            Hide
            hafizullah Hafizullah Nikben added a comment -

            Desperately need this feature working

            Show
            hafizullah Hafizullah Nikben added a comment - Desperately need this feature working
            Hide
            jglick Jesse Glick added a comment -

            Krzysztof Wolny the git step is only for the most simplistic use cases. Use checkout.

            Show
            jglick Jesse Glick added a comment - Krzysztof Wolny the git step is only for the most simplistic use cases. Use checkout .
            Hide
            jglick Jesse Glick added a comment -

            To be clear, from the description I am assuming this is a bug not a feature. Would require someone to take time to do the analysis. Offhand I have no idea where the problem lies.

            Show
            jglick Jesse Glick added a comment - To be clear, from the description I am assuming this is a bug not a feature. Would require someone to take time to do the analysis. Offhand I have no idea where the problem lies.
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            Hide
            vanta Krzysztof Wolny added a comment - - edited

            Jesse Glick I know that, and I think it should be easy to pass properties to "parent". But that's the point of using git - simplicity. If you go this way you proposed further, we can remove git - and use checkout, right?

            Show
            vanta Krzysztof Wolny added a comment - - edited Jesse Glick I know that, and I think it should be easy to pass properties to "parent". But that's the point of using git - simplicity. If you go this way you proposed further, we can remove git - and use checkout , right?
            Hide
            wimederveen Wim Ederveen added a comment -

            I have an even trickier problem. We use standardized jenkinsfiles from a seperate repository to facilitate 200+ java projects. And these Jenkinsfiles are fed with parameters in the job, with properties like gitUrl and branchName. These values are used in the jenkinsfiles in a checkout step. So the 'pipeline script from SCM' bit in the job is not the git repository that contains the code. It does pick up commit changes in the parameterized git repository, but exclusions are not honored.
            So we have git repository 'A' containing jenkinsfiles. These are used in the pipelinescript from SCM bit in the Job.
            And we have git repository 'B' containing sourcecode to be built. The git url of this repository is fed to the jenkinsfile from repository 'A'
            The problem now is that we have an SCM trigger on the pipeline job to run every night, but the [maven-release-plugin] triggers itself because of version update commits. So we want to exclude either the jenkins user commits or commits containing [maven-release-plugin].
            I've tried adding exclusions to the pipeline script from SCM additional behaviours as well as in the jenkinsfiles checkout step.
            We really need this for our pipeline implementation

            Show
            wimederveen Wim Ederveen added a comment - I have an even trickier problem. We use standardized jenkinsfiles from a seperate repository to facilitate 200+ java projects. And these Jenkinsfiles are fed with parameters in the job, with properties like gitUrl and branchName. These values are used in the jenkinsfiles in a checkout step. So the 'pipeline script from SCM' bit in the job is not the git repository that contains the code. It does pick up commit changes in the parameterized git repository, but exclusions are not honored. So we have git repository 'A' containing jenkinsfiles. These are used in the pipelinescript from SCM bit in the Job. And we have git repository 'B' containing sourcecode to be built. The git url of this repository is fed to the jenkinsfile from repository 'A' The problem now is that we have an SCM trigger on the pipeline job to run every night, but the [maven-release-plugin] triggers itself because of version update commits. So we want to exclude either the jenkins user commits or commits containing [maven-release-plugin] . I've tried adding exclusions to the pipeline script from SCM additional behaviours as well as in the jenkinsfiles checkout step. We really need this for our pipeline implementation
            Hide
            wimederveen Wim Ederveen added a comment -

            Just wanted to share our workaround for the polling issue.
            We came up with a solution using the groovy script. It is not perfect but it helps us by preventing unwanted releases triggered by the maven-release-plugin.
            In one of the first stages we check the "git log" for the comments of the last day using:
            def gitComments = sh(returnStdout: true, script: "git log --pretty=format:\"%cd  %h  %s\" --date=short --since=\"last day\"")
            We then parse this list to see if it only contains [maven-release-plugin] commits. If so we stop the Jenkins job using the following lines:
            currentBuild.result = "NOT_BUILT"
            error "Pipeline not built. No new user commits found"

            It is a bit ugly but the job status shows up as NOT_BUILT and that is enough for us.
             

            Show
            wimederveen Wim Ederveen added a comment - Just wanted to share our workaround for the polling issue. We came up with a solution using the groovy script. It is not perfect but it helps us by preventing unwanted releases triggered by the maven-release-plugin. In one of the first stages we check the "git log" for the comments of the last day using: def gitComments = sh(returnStdout: true, script: "git log --pretty=format:\"%cd  %h  %s\" --date=short --since=\"last day\"") We then parse this list to see if it only contains [maven-release-plugin]  commits. If so we stop the Jenkins job using the following lines: currentBuild.result = "NOT_BUILT" error "Pipeline not built. No new user commits found" It is a bit ugly but the job status shows up as NOT_BUILT and that is enough for us.  
            Hide
            pyrocks Mor L added a comment - - edited

            It's a bummer to have something given to you by the Pipeline Syntax page, only to figure out it is not working properly after some trial and error.

            If it's not working properly - Pipeline Syntax page should not allow to configure it, and if someone tries to - it should at least generate a warning during runtime (as long as it is a known limitation and not an undiscovered bug).

            Show
            pyrocks Mor L added a comment - - edited It's a bummer to have something given to you by the Pipeline Syntax page, only to figure out it is not working properly after some trial and error. If it's not working properly - Pipeline Syntax page should not allow to configure it, and if someone tries to - it should at least generate a warning during runtime (as long as it is a known limitation and not an undiscovered bug).
            Hide
            markewaite Mark Waite added a comment -

            Mor L I believe a set of changes currently being prepared (or at least discussed) may remove the "ill-advised" additional behaviours from the pipeline syntax (like the "Checkout to subdirectory" and "Ignore polling ..." and others)

            Show
            markewaite Mark Waite added a comment - Mor L I believe a set of changes currently being prepared (or at least discussed) may remove the "ill-advised" additional behaviours from the pipeline syntax (like the "Checkout to subdirectory" and "Ignore polling ..." and others)
            Hide
            pyrocks Mor L added a comment -

            Mark Waite When that happens, the question that remains is whether this issue would be considered a bug or RFE... I prefer to see it as a bug.

            We work around this by checking the commit authors post checkout and abort the build if only the build user committed. Soon we will switch to post-receive hook and have no need for this but its still worth doing.

            Show
            pyrocks Mor L added a comment - Mark Waite When that happens, the question that remains is whether this issue would be considered a bug or RFE... I prefer to see it as a bug. We work around this by checking the commit authors post checkout and abort the build if only the build user committed. Soon we will switch to post-receive hook and have no need for this but its still worth doing.
            Hide
            markewaite Mark Waite added a comment -

            I consider this a request for enhancement, rather than a bug. The git plugin special polling rules have been consciously chosen to not apply in the pipeline case. They are not a general purpose technique across all the SCM systems supported in pipeline.

            Show
            markewaite Mark Waite added a comment - I consider this a request for enhancement, rather than a bug. The git plugin special polling rules have been consciously chosen to not apply in the pipeline case. They are not a general purpose technique across all the SCM systems supported in pipeline.
            Hide
            demee Slawomir Demichowicz added a comment -

            Mark Waite : I understand the decision made, but it's not obvious from UI. UI clearly shows those options, and it's confusing why they are not working. 

            Show
            demee Slawomir Demichowicz added a comment - Mark Waite : I understand the decision made, but it's not obvious from UI. UI clearly shows those options, and it's confusing why they are not working. 
            Hide
            markewaite Mark Waite added a comment -

            Slawomir Demichowicz the UI confusion is resolved or will be resolved with the release of git plugin 3.4.0 and the SCMNavigator improvements from Stephen Connolly. I believe this still should remain open as an enhancement request for those users who want pipeline to be able to ignore certain users or certain regions.

            Show
            markewaite Mark Waite added a comment - Slawomir Demichowicz the UI confusion is resolved or will be resolved with the release of git plugin 3.4.0 and the SCMNavigator improvements from Stephen Connolly . I believe this still should remain open as an enhancement request for those users who want pipeline to be able to ignore certain users or certain regions.
            Hide
            stephenconnolly Stephen Connolly added a comment -

            I think this would be best implemented as a Filtering trait if people want to ignore branches from certain users.

            If people want the branches to remain but ignore specific commits, the best bet IMHO is to have the pipeline abort in a defined way from the Jenkinsfile parse so that Jenkins is happy that the commit has been built and everyone else is happy that the build did nothing

            Show
            stephenconnolly Stephen Connolly added a comment - I think this would be best implemented as a Filtering trait if people want to ignore branches from certain users. If people want the branches to remain but ignore specific commits, the best bet IMHO is to have the pipeline abort in a defined way from the Jenkinsfile parse so that Jenkins is happy that the commit has been built and everyone else is happy that the build did nothing
            demee Slawomir Demichowicz made changes -
            Attachment image-2017-07-18-16-01-33-798.png [ 38933 ]
            demee Slawomir Demichowicz made changes -
            Attachment image-2017-07-18-16-02-28-808.png [ 38934 ]
            demee Slawomir Demichowicz made changes -
            Attachment image-2017-07-18-16-03-09-364.png [ 38935 ]
            markewaite Mark Waite made changes -
            Issue Type Bug [ 1 ] Improvement [ 4 ]
            demee Slawomir Demichowicz made changes -
            Attachment image-2017-07-18-16-01-33-798.png [ 38933 ]
            demee Slawomir Demichowicz made changes -
            Attachment image-2017-07-18-16-02-28-808.png [ 38934 ]
            Hide
            demee Slawomir Demichowicz added a comment -

            Not sure if this is the place to post this, but it may be useful to someone.

            I have it implement this way (outside of `step` block) 

             

             

            if (stash.isJenkinsLastAuthor() && !params.SKIP_CHANGELOG_CHECK) {
                echo "No more new changes, stopping pipeline"
                currentBuild.result = "SUCCESS"
                return
            }
            
            

             

            setting SKIP_CHANGELOG_CHECK - this allows me to force build ( in case I do want the pipeline to run regardless of who was the last one to commit )

            Then it looks like this in the UI

            Show
            demee Slawomir Demichowicz added a comment - Not sure if this is the place to post this, but it may be useful to someone. I have it implement this way (outside of `step` block)      if (stash.isJenkinsLastAuthor() && !params.SKIP_CHANGELOG_CHECK) { echo "No more new changes, stopping pipeline" currentBuild.result = "SUCCESS" return }   setting SKIP_CHANGELOG_CHECK - this allows me to force build ( in case I do want the pipeline to run regardless of who was the last one to commit ) Then it looks like this in the UI
            Hide
            jglick Jesse Glick added a comment -

            As I have noted several times here, there are two completely related cases.

            For a standalone Pipeline job, all SCM features including exclusions are supposed to work exactly as they do for freestyle projects. If they do not, that is a bug. I just do not know how to reproduce it.

            For a branch project, there is no support for commit exclusions, nor should any such be offered in the recently amended UX. If we want to add support, I think it should just work as people expect: branch changes with only insignificant changes should not trigger new builds.

            In either case, you always have the workaround of inspecting currentBuild.changeSets from a script and returning early if the changes look boring. JENKINS-27092 would allow you to do this with the error step so you could get the natural NOT_BUILT status (and not have to deal with a return statement, which does not work inside a library, closure, etc.).

            Show
            jglick Jesse Glick added a comment - As I have noted several times here, there are two completely related cases. For a standalone Pipeline job, all SCM features including exclusions are supposed to work exactly as they do for freestyle projects. If they do not, that is a bug. I just do not know how to reproduce it. For a branch project, there is no support for commit exclusions, nor should any such be offered in the recently amended UX. If we want to add support, I think it should just work as people expect: branch changes with only insignificant changes should not trigger new builds. In either case, you always have the workaround of inspecting currentBuild.changeSets from a script and returning early if the changes look boring.  JENKINS-27092 would allow you to do this with the error step so you could get the natural NOT_BUILT status (and not have to deal with a return statement, which does not work inside a library, closure, etc.).
            stephenconnolly Stephen Connolly made changes -
            Link This issue is related to JENKINS-44611 [ JENKINS-44611 ]
            markewaite Mark Waite made changes -
            Link This issue is duplicated by JENKINS-46059 [ JENKINS-46059 ]
            Hide
            phreakadelle Stephan Watermeyer added a comment - - edited

            +1 to have the "Polling ignores commits from certain users" option available for Multibranch Pipeline Plugins. I would like to ignore the commits from the Jenkins-User. 

            The Reason for that: We are doing java development and using Maven with the official maven-release-plugin. Thus the call of "mvn release:prepare release:perform" results in a commit on the current branch. And this commit results then in a new build.

            I can check for the workaround mentioned by Jesse, but at least i want to have at less scripting as possible in our Jenkinsfiles. So having the option in the Jenkins GIT-Plugin would be much helpful.

            //edit: The Option "Polling ignores commits from certain users" is available in the old plugin version 3.3.2 

            Show
            phreakadelle Stephan Watermeyer added a comment - - edited +1 to have the "Polling ignores commits from certain users" option available for Multibranch Pipeline Plugins. I would like to ignore the commits from the Jenkins-User.  The Reason for that: We are doing java development and using Maven with the official maven-release-plugin. Thus the call of "mvn release:prepare release:perform" results in a commit on the current branch. And this commit results then in a new build. I can check for the workaround mentioned by Jesse, but at least i want to have at less scripting as possible in our Jenkinsfiles. So having the option in the Jenkins GIT-Plugin would be much helpful. //edit: The Option "Polling ignores commits from certain users" is available in the old plugin version 3.3.2 
            Hide
            jglick Jesse Glick added a comment -

            Stephan Watermeyer please reread my above comments. You cannot use commit exclusions in multibranch Pipeline jobs. They are available in standalone jobs. This is by design. If you want a job doing Maven releases, it should be a standalone job, not a branch job.

            Show
            jglick Jesse Glick added a comment - Stephan Watermeyer please reread my above comments. You cannot use commit exclusions in multibranch Pipeline jobs . They are available in standalone jobs. This is by design. If you want a job doing Maven releases, it should be a standalone job, not a branch job.
            Hide
            phreakadelle Stephan Watermeyer added a comment - - edited

            Thanks Jesse Glick for your reply. Understood. 

            So we have to create two jobs per component, one job to make the ci-builds for each branch and another job to release the component. So i've to create/maintain a second Jenkinsfile in each component. Would this be the recommended way to go? That would be nearly the way how jenkins-plugins are developed but i am still unsure if Jenkins should be used in that way.

            Further: I am still thinking about an option to extend my Jenkinsfile to update the "scm-revision-hash.xml" file after a mvn release. My idea is to put in something in my pipeline like:

            ...
            stage('Release') {
              when {
                branch 'master'
              }
            
              steps {
                sh 'mvn release:prepare release:perform'
            
                // get the latest commit id
                sh '$LASTCOMMITID=git log -1 --pretty=format:"%H";'
            
                // Search and Replace the Hash in the scm-revision-hash.xml file with $LASTCOMMITID
                sh 'search/replace in jobs/<projectname>/<branchname>/scm-revision-hash.xml and replace hash with $LASTCOMMITID'
              }
            }
            ...

            What do you think? What would be a nice & clean way to update the scm-revision-hash.xml file? Or is this just crap?

            Show
            phreakadelle Stephan Watermeyer added a comment - - edited Thanks Jesse Glick for your reply. Understood.  So we have to create two jobs per component, one job to make the ci-builds for each branch and another job to release the component. So i've to create/maintain a second Jenkinsfile in each component. Would this be the recommended way to go ? That would be nearly the way how jenkins-plugins are developed but i am still unsure if Jenkins should be used in that way. Further: I am still thinking about an option to extend my Jenkinsfile to update the "scm-revision-hash.xml" file after a mvn release. My idea is to put in something in my pipeline like: ... stage( 'Release' ) { when { branch 'master' } steps { sh 'mvn release:prepare release:perform' // get the latest commit id sh '$LASTCOMMITID=git log -1 --pretty=format: "%H" ;' // Search and Replace the Hash in the scm-revision-hash.xml file with $LASTCOMMITID sh 'search/replace in jobs/<projectname>/<branchname>/scm-revision-hash.xml and replace hash with $LASTCOMMITID' } } ... What do you think? What would be a nice & clean way to update the scm-revision-hash.xml file? Or is this just crap?
            Hide
            jglick Jesse Glick added a comment -

            Stephan Watermeyer yes you should have a separate job definition for release builds, based on some Pipeline script from SCM, but not multibranch.

            I do not advise trying to touch scm-revision-hash.xml. This is not supported, and is also incompatible with a secured Jenkins instance.

            Show
            jglick Jesse Glick added a comment - Stephan Watermeyer yes you should have a separate job definition for release builds, based on some Pipeline script from SCM, but not multibranch. I do not advise trying to touch scm-revision-hash.xml . This is not supported, and is also incompatible with a secured Jenkins instance.
            Hide
            rswanso ron swanson added a comment -

            Good day.

            I have been watching this item for a while and thought I would chime in.

            We have a non-multi branch pipeline project that does the maven release steps and during the release process it will check the pom back in with updates to <version> and we encountered this issue where the polling exclusion doesn't work to ignore changes made by the build user, so we would get stuck in an infinite build loop until we killed the job.

            The work around I came up with was to use the 'ci-skip' feature of the GitLab CI project trigger and I modified our maven check-in comments to prepend '[ci-skip]' to any of the check-in commit messages performed by our maven release process.

            Jesse Glick   Let me dig up my test jobs and version info to be able to pass on some information about reproducing this.   I had encountered this a while ago and haven't tested it with recent versions,  so it will be good to make sure I have the details of the problem again.

            Thanks for all your work.

            Ron

            Show
            rswanso ron swanson added a comment - Good day. I have been watching this item for a while and thought I would chime in. We have a non-multi branch pipeline project that does the maven release steps and during the release process it will check the pom back in with updates to <version> and we encountered this issue where the polling exclusion doesn't work to ignore changes made by the build user, so we would get stuck in an infinite build loop until we killed the job. The work around I came up with was to use the 'ci-skip' feature of the GitLab CI project trigger and I modified our maven check-in comments to prepend ' [ci-skip] ' to any of the check-in commit messages performed by our maven release process. Jesse Glick    Let me dig up my test jobs and version info to be able to pass on some information about reproducing this.   I had encountered this a while ago and haven't tested it with recent versions,  so it will be good to make sure I have the details of the problem again. Thanks for all your work. Ron
            Hide
            jglick Jesse Glick added a comment -

            ron swanson it is expected that a Maven release will trigger a new CI build by default, since sources have indeed changed (even if only trivially). There should be no “loop” since a job with an SCM trigger should never be performing releases! A release job should have only a manual trigger.

            Please keep it to the user’s list. This issue is about a claimed bug which I do not know how to reproduce: that in a standalone, not multibranch, job using an SCM trigger and an SCM checkout which specifies polling exclusions using whatever SCM-specific configuration (reported here for Git), that ignorable commits nonetheless trigger builds. No other comments are relevant in this issue.

            Show
            jglick Jesse Glick added a comment - ron swanson it is expected that a Maven release will trigger a new CI build by default, since sources have indeed changed (even if only trivially). There should be no “loop” since a job with an SCM trigger should never be performing releases! A release job should have only a manual trigger. Please keep it to the user’s list. This issue is about a claimed bug which I do not know how to reproduce: that in a standalone, not multibranch , job using an SCM trigger and an SCM checkout which specifies polling exclusions using whatever SCM-specific configuration (reported here for Git), that ignorable commits nonetheless trigger builds. No other comments are relevant in this issue.
            Hide
            rswanso ron swanson added a comment -

            Jesse Glick    Sorry, i thought I was talking about what was relevant.  I have a non-multi branch project which attempts to use  excludedUsers to ignore checkins that were made by the user running the build.

            I DONT want to get into a discussion about what maven should or shouldn't be doing, but I think I am trying to stay on what I think the topic is, which to me is, the excludedUsers doesn't work in the scm checkout to ignore checkouts by specific users.

            Am I still off-base?

            Thanks

            Ron

            Show
            rswanso ron swanson added a comment - Jesse Glick     Sorry, i thought I was talking about what was relevant.  I have a non-multi branch project which attempts to use  excludedUsers to ignore checkins that were made by the user running the build. I DONT want to get into a discussion about what maven should or shouldn't be doing, but I think I am trying to stay on what I think the topic is, which to me is, the excludedUsers doesn't work in the scm checkout to ignore checkouts by specific users. Am I still off-base? Thanks Ron
            Hide
            jglick Jesse Glick added a comment -

            the excludedUsers doesn't work in the scm checkout to ignore checkouts by specific users

            If reproducible (using a minimal self-contained test case etc.), that is a bug.

            Show
            jglick Jesse Glick added a comment - the excludedUsers doesn't work in the scm checkout to ignore checkouts by specific users If reproducible (using a minimal self-contained test case etc.), that is a bug.
            Hide
            rswanso ron swanson added a comment -

            Jesse Glick   Thanks....  and I have to ask as I am new to this, do you mean a test case like this?

            https://github.com/jenkinsci/git-plugin/blob/4e38437ca3aed7b976b687a2e006e1987276437d/src/test/java/hudson/plugins/git/GitSCMTest.java#L483

             

            Thanks

            Ron

            Show
            rswanso ron swanson added a comment - Jesse Glick    Thanks....  and I have to ask as I am new to this, do you mean a test case like this? https://github.com/jenkinsci/git-plugin/blob/4e38437ca3aed7b976b687a2e006e1987276437d/src/test/java/hudson/plugins/git/GitSCMTest.java#L483   Thanks Ron
            Hide
            jglick Jesse Glick added a comment -

            Well an actual JUnit test case reproducing the bug (for bonus points, in a PR with an initial @Ignore annotation) is great, since then a developer can get straight to work analyzing and fixing it. But detailed manual steps suffice: install Jenkins 2.xx, install such-and-such plugins, create a job with the following script, run the following Git commands in a shell, whatever.

            AFAICT neither this nor this currently test polling exclusions explicitly. This call is where Pipeline passes control to the SCM to determine whether there are changes, implemented for example here.

            Show
            jglick Jesse Glick added a comment - Well an actual JUnit test case reproducing the bug (for bonus points, in a PR with an initial @Ignore annotation) is great, since then a developer can get straight to work analyzing and fixing it. But detailed manual steps suffice: install Jenkins 2.xx, install such-and-such plugins, create a job with the following script, run the following Git commands in a shell, whatever. AFAICT neither this  nor this  currently test polling exclusions explicitly. This call  is where Pipeline passes control to the SCM to determine whether there are changes, implemented for example here .
            Hide
            rswanso ron swanson added a comment -

            Thanks for that detailed response, that gives me some good stuff to look at.

            I have an environment that i just tested this out in, but i will have to recreate to make sure I get the details right and I can post it back.  Doing a PR is a bit out of my wheelhouse, but I do want to learn how to write Jenkins plugins, so this is good.

            Thanks

            Ron

            Show
            rswanso ron swanson added a comment - Thanks for that detailed response, that gives me some good stuff to look at. I have an environment that i just tested this out in, but i will have to recreate to make sure I get the details right and I can post it back.  Doing a PR is a bit out of my wheelhouse, but I do want to learn how to write Jenkins plugins, so this is good. Thanks Ron
            Hide
            nafiux Ignacio Ocampo added a comment -

            Same problem here, using:

            I have tried to ignore poll SCM with Excluded Users and Excluded Messages, none worked.

             

            Show
            nafiux Ignacio Ocampo added a comment - Same problem here, using: Jenkins ver. 2.60.3 https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin  v 2.5 I have tried to ignore poll SCM with Excluded Users  and Excluded Messages , none worked.  
            nafiux Ignacio Ocampo made changes -
            Issue Type Improvement [ 4 ] Bug [ 1 ]
            nafiux Ignacio Ocampo made changes -
            Issue Type Bug [ 1 ] Improvement [ 4 ]
            Hide
            rswanso ron swanson added a comment -

            I am at

            Jenkins ver  2.46.2

            https://plugins.jenkins.io/gitlab-plugin  1.4.7

            https://plugins.jenkins.io/workflow-scm-step 2.5

             

             My test case was:

            -Pipeline job that checked out single Jenkinsfile

            -"Poll SCM" was checked, but schedule was empty

            -"Triggered builds remotely (eg from scripts)"  and I set a Authentication token

            -Gitlab has this trigger for my job: http://myserver:8080/job/test.infinite.loop/build?token=mytoken

            -Tried the exclude logic from the job config in UI, as well as from in Jenkinsfile using "checkout scm".

            -My Jenkinsfile calls these maven commands to simulate a release: versions:set, versions:commit, scm:add and scm:checkin, which just updates the pom.xml file with the next version.

             

             

            I tested every combination I could think of.

            Setting different user.name at checkout time using additional behaviour, changing displayname as defined in /etc/passwd, adding multiple users to exclude list, trying logic at job level, and in pipeline using 'checkout SCM'.

            Something to note is Jenkins reports the job was triggered by the user  'jenkins' as that is the user Jenkins is running as, but the user that checks into GIT is what is defined as the display name from /etc/passwd file.  So I am not exactly sure what user it SHOULD be expecting to try to exclude, but I have tried to exclude all the different users, individually as well as in a multiple list.

             

            This is from  the "Changes" link in a particular build and shows that 'jenkins' did it, as 'jenkins' is the unix user that Jenkins is running under.

            *Commit f3b85jh3jh4k3j4h5kj3h45kj2k3j2bfc9c99727c566ba718c2 by *jenkins   [ci-skip] updating pom to 2017.08.427

             

            But in GITLAB, the checkin comment shows the user that is defined as the display name in the /etc/passwd file for the user.  I have also tried changing that displayname to match the actual username, so it is the same value on both sides, to no avail.

               [ci-skip] updating pom to 2017.08.427
               CMJenkins committed 2 minutes ago

             
            In my pom i use these GIT connection urls, so I have an SSH key setup between the user jenkins is running as, and the user we are connecting to GIT with, which is a different user, (and I've  tried to exclude that one too)
             
            <connection>scm:git:ssh://git@myGITLABserver/myGROUP/myPROJECT.git</connection><developerConnection>scm:git:ssh://git@myGITLABserver/myGROUP/myPROJECT.git</developerConnection>

             

            It doesn't mean much, but I have a similar setup using 1.X Jenkins interacting with GitHub and things work as they should.

             

             

            Thanks for everything.

            Ron

             

            Show
            rswanso ron swanson added a comment - I am at Jenkins ver  2.46.2 https://plugins.jenkins.io/gitlab-plugin   1.4.7 https://plugins.jenkins.io/workflow-scm-step 2.5    My test case was: -Pipeline job that checked out single Jenkinsfile -"Poll SCM" was checked, but schedule was empty -"Triggered builds remotely (eg from scripts)"  and I set a Authentication token -Gitlab has this trigger for my job: http://myserver:8080/job/test.infinite.loop/build?token=mytoken -Tried the exclude logic from the job config in UI, as well as from in Jenkinsfile using "checkout scm". -My Jenkinsfile calls these maven commands to simulate a release: versions:set, versions:commit, scm:add and scm:checkin, which just updates the pom.xml file with the next version.     I tested every combination I could think of. Setting different user.name at checkout time using additional behaviour, changing displayname as defined in /etc/passwd, adding multiple users to exclude list, trying logic at job level, and in pipeline using 'checkout SCM'. Something to note is Jenkins reports the job was triggered by the user  'jenkins' as that is the user Jenkins is running as, but the user that checks into GIT is what is defined as the display name from /etc/passwd file.  So I am not exactly sure what user it SHOULD be expecting to try to exclude, but I have tried to exclude all the different users, individually as well as in a multiple list.   This is from  the "Changes" link in a particular build and shows that 'jenkins' did it, as 'jenkins' is the unix user that Jenkins is running under. * Commit f3b85jh3jh4k3j4h5kj3h45kj2k3j2bfc9c99727c566ba718c2 by *jenkins    [ci-skip] updating pom to 2017.08.427   But in GITLAB, the checkin comment shows the user that is defined as the display name in the /etc/passwd file for the user.  I have also tried changing that displayname to match the actual username, so it is the same value on both sides, to no avail.    [ci-skip] updating pom to 2017.08.427    CMJenkins committed 2 minutes ago   In my pom i use these GIT connection urls, so I have an SSH key setup between the user jenkins is running as, and the user we are connecting to GIT with, which is a different user, (and I've  tried to exclude that one too)   <connection>scm:git:ssh://git@myGITLABserver/myGROUP/myPROJECT.git</connection><developerConnection>scm:git:ssh://git@myGITLABserver/myGROUP/myPROJECT.git</developerConnection>   It doesn't mean much, but I have a similar setup using 1.X Jenkins interacting with GitHub and things work as they should.     Thanks for everything. Ron  
            andrey_klyukin Andrey Klyukin made changes -
            Comment [ According to the comments above:

            [~jlehtnie]
            {quote}At least _Included/Excluded regions_ actually works ok if you also specify _Disable Remote Poll_, e.g. ...
            {quote}
            and

            [~killdash9]
            {quote}Yes. polling REGION inclusion/exclusion is working. The USER inclusion/exclusion is not working ...
            {quote}
            But REGION inclusion/exclusion is not working for me.

            Pipeline: 
            {code:java}
            pipeline {
                agent any

                stages {
                    stage("Checkout repo"){
                        steps {
                            checkout([
                                $class: 'GitSCM',
                                branches: [[name: '*/master']],
                                doGenerateSubmoduleConfigurations: false,
                                extensions: [[$class: 'DisableRemotePoll'], [$class: 'PathRestriction', excludedRegions: '', includedRegions: 'ggg/.*']], submoduleCfg: [], userRemoteConfigs: [[url: 'ssh://git@myrepo/testrepo.git&#39;]]])
                        }
                    }
                    stage("Next step"){
                        steps {
                            sh 'echo "Testing"'
                        }
                    }
                }
            }
            {code}
            "Poll SCM" parameter is checked

            When I make changes to the repository (not in "ggg/" directory) job is triggered, although it should not.

            [Jenkins ver. 2.92|https://jenkins.io/]

            Git plugin 3.6.4

            Can anyone help? ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal OSS-1919 (Web Link)" [ 18511 ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal OSS-1409 (Web Link)" [ 18708 ]
            Hide
            nsidhaye Nikhil Sidhaye added a comment - - edited

            Do we have any way to git with include / exclude path for polling in pipeline? Because from pipeline syntax, I could not able to add exclude / include path behavior. Is this functionality is missing in pipeline? Even in simple scripted pipeline? 

             

             

            Michael Andrews: Did you able to test this with pipeline with git? How did you configure it? Can you share your working pipeline syntax? 

            Show
            nsidhaye Nikhil Sidhaye added a comment - - edited Do we have any way to git with include / exclude path for polling in pipeline? Because from pipeline syntax, I could not able to add exclude / include path behavior. Is this functionality is missing in pipeline? Even in simple scripted pipeline?      Michael Andrews : Did you able to test this with pipeline with git? How did you configure it? Can you share your working pipeline syntax? 
            Hide
            killdash9 Michael Andrews added a comment - - edited

            Nikhil Sidhaye It is not missing. You just need to use the full SCM step, and then use the extensions attribute.

            checkout([
                $class: 'GitSCM',
                branches: [
                    [
                        name: gitBranch
                    ]
                ],
                doGenerateSubmoduleConfigurations: false,
                extensions: [
                    [
                        $class: 'UserExclusion',
                        excludedUsers: """${scmPollingUserExclusions}"""
                    ],
                    [
                        $class: 'PathRestriction',
                        excludedRegions: scmPollingPathExclusions, 
                        includedRegions: scmPollingPathInclusions
                    ]
                ],

                ...

             

            Where,  scmPollingUserExclusions,  scmPollingPathExclusions, and  scmPollingPathInclusions are pipeline parameters.

             

             

            Show
            killdash9 Michael Andrews added a comment - - edited Nikhil Sidhaye It is not missing. You just need to use the full SCM step, and then use the extensions attribute. checkout([     $class: 'GitSCM',     branches: [         [             name: gitBranch         ]     ],     doGenerateSubmoduleConfigurations: false,     extensions: [         [             $class: 'UserExclusion',             excludedUsers: """${scmPollingUserExclusions}"""         ],         [             $class: 'PathRestriction',             excludedRegions: scmPollingPathExclusions,              includedRegions: scmPollingPathInclusions         ]     ],     ...   Where,  scmPollingUserExclusions,  scmPollingPathExclusions, and  scmPollingPathInclusions are pipeline parameters.    
            scui Xiaohui Cui made changes -
            Attachment image-2018-06-28-18-09-30-343.png [ 43243 ]
            Hide
            scui Xiaohui Cui added a comment -

            Using v3.9.1 version of this plugin, I tried Michael Andrews suggestion about using pipeline parameters (scmPollingPathInclusions, "$scmPollingPathInclusions", "${scmPollingPathInclusions}", '$scmPollingPathInclusions'), and no luck so far. Also tried adding [$class: 'DisableRemotePoll'], job was still triggered on a region outside of the specified IncludedRegion.

            However, the region inclusion/exclusion works in freestyle jobs. Any plan to have this fixed soon? 

             

            Pipeline - does not work

            [$class: 'PathRestriction', [$class: 'PathRestriction', excludedRegions: '', includedRegions: 'src/.*']]

             

            Freestyle - works

             

            Show
            scui Xiaohui Cui added a comment - Using v3.9.1 version of this plugin, I tried Michael Andrews suggestion about using pipeline parameters (scmPollingPathInclusions, "$scmPollingPathInclusions", "${scmPollingPathInclusions}", '$scmPollingPathInclusions'), and no luck so far. Also tried adding [$class: 'DisableRemotePoll'] , job was still triggered on a region outside of the specified IncludedRegion. However, the region inclusion/exclusion works in freestyle jobs. Any plan to have this fixed soon?    Pipeline - does not work [$class: 'PathRestriction', [$class: 'PathRestriction', excludedRegions: '', includedRegions: 'src/.*'] ]   Freestyle - works  
            Hide
            markewaite Mark Waite added a comment -

            Xiaohui Cui no plans from me to fix it soon. There were discussions between Jesse Glick and Stephen Connolly of techniques that might work to add region exclusion to the SCM API and then allow that region exclusion to be used by the git plugin and other SCM plugins. However, there has been no implementation of those discussions and I don't expect any implementation of those discussions in the near future.

            Show
            markewaite Mark Waite added a comment - Xiaohui Cui no plans from me to fix it soon. There were discussions between Jesse Glick and Stephen Connolly of techniques that might work to add region exclusion to the SCM API and then allow that region exclusion to be used by the git plugin and other SCM plugins. However, there has been no implementation of those discussions and I don't expect any implementation of those discussions in the near future.
            Hide
            killdash9 Michael Andrews added a comment -

            You need to use regular expressions - not glob style pattern matching. E.g., ^foo\/bar\/.*$ for all files in the /foo/bar directory. 

            Show
            killdash9 Michael Andrews added a comment - You need to use regular expressions - not glob style pattern matching. E.g., ^foo\/bar\/.*$ for all files in the /foo/bar directory. 
            Hide
            killdash9 Michael Andrews added a comment -

            ...and be sure to escape the separator as shown in the previous comment.

            Show
            killdash9 Michael Andrews added a comment - ...and be sure to escape the separator as shown in the previous comment.
            scui Xiaohui Cui made changes -
            Attachment image-2018-06-29-14-45-17-341.png [ 43260 ]
            Hide
            scui Xiaohui Cui added a comment - - edited

            Mark Waite and Michael Andrews, thanks for the reply.

             

            FYI, I did more testing and seems like the regular expression does not make any difference. To simplify the paths to include, I limited to the README.md in the root folder. Again, the freestyle job was triggered correctly but pipeline job did not trigger correctly (as in changing to any file in the repo would trigger).

             

            Works:

             

            Does not work:

            [$class: 'PathRestriction', excludedRegions: '', includedRegions: 'README.md']

            [$class: 'PathRestriction', excludedRegions: '', includedRegions: "README.md"]

            [$class: 'PathRestriction', excludedRegions: '', includedRegions: '^README.md\$']

            [$class: 'PathRestriction', excludedRegions: '', includedRegions: "^README.md\$"]

             

            I even tried to use the pipeline syntax function on jenkins to make sure that I got the syntax correct. And I got exactly this line which did not work.

            [$class: 'PathRestriction', excludedRegions: '', includedRegions: 'README.md']

            No luck finding a workaround yet...

             

            Show
            scui Xiaohui Cui added a comment - - edited Mark Waite and Michael Andrews , thanks for the reply.   FYI, I did more testing and seems like the regular expression does not make any difference. To simplify the paths to include, I limited to the README.md in the root folder. Again, the freestyle job was triggered correctly but pipeline job did not trigger correctly (as in changing to any file in the repo would trigger).   Works:   Does not work: [$class: 'PathRestriction', excludedRegions: '', includedRegions: 'README.md'] [$class: 'PathRestriction', excludedRegions: '', includedRegions: "README.md"] [$class: 'PathRestriction', excludedRegions: '', includedRegions: '^README.md\$'] [$class: 'PathRestriction', excludedRegions: '', includedRegions: "^README.md\$"]   I even tried to use the pipeline syntax function on jenkins to make sure that I got the syntax correct. And I got exactly this line which did not work. [$class: 'PathRestriction', excludedRegions: '', includedRegions: 'README.md'] No luck finding a workaround yet...  
            Hide
            killdash9 Michael Andrews added a comment -

            Did you try escaping the dot?  

            README\.md

            or

            ^README\.md$

             

            M

            Show
            killdash9 Michael Andrews added a comment - Did you try escaping the dot?   README\.md or ^README\.md$   M
            Hide
            killdash9 Michael Andrews added a comment -

            You should use an online regex tester to make sure your expressions is correct before attempting to use the pattern in Jenkins.  https://regex101.com 

            Show
            killdash9 Michael Andrews added a comment - You should use an online regex tester to make sure your expressions is correct before attempting to use the pattern in Jenkins.   https://regex101.com  
            scui Xiaohui Cui made changes -
            Attachment image-2018-06-29-15-36-13-977.png [ 43261 ]
            Hide
            scui Xiaohui Cui added a comment - - edited

            Need to double escape the backslash. Just tried the following and all triggered by change to another file file in the repo.

            'README\\.md'
            "README\\.md"
            '^README\\.md$'
            '^README\\.md\$'
            

            Michael Andrews, and thanks for the reminder about the dot. But the dot in regex would match "." anyways?

            Show
            scui Xiaohui Cui added a comment - - edited Need to double escape the backslash. Just tried the following and all triggered by change to another file file in the repo. 'README\\.md' "README\\.md" '^README\\.md$' '^README\\.md\$' Michael Andrews , and thanks for the reminder about the dot. But the dot in regex would match "." anyways?
            shauli10 Shaul Lahav made changes -
            Assignee SHAUL LAHAV [ shauli10 ]
            Hide
            shauli10 Shaul Lahav added a comment - - edited

            What is causing the problem is that when the Git CLI client calls the "git log" command giving it a range of revisions to filter by (last build's most recent revision to the latest committed revision), all files in the repo/tree are returned instead of just the newly committed file(s).

            Example:
            I committed a file named "dir2/README8", and when Jenkins polled the SCM, it invoked:

            git log --full-history --no-abbrev --format=raw -M -m --raw 645221fb9fb8b752db7736d40c4c311e032482bb..d36267d8d88bdd085c894f35b1585a9f1ee9d755 # timeout=10
            

            Which resulted in the following output:

            commit d36267d8d88bdd085c894f35b1585a9f1ee9d755
            tree 6efc76859dbd9dee1afd6881f81d2c0aad574168
            parent 645221fb9fb8b752db7736d40c4c311e032482bb
            author Shaul Lahav <slahav@company.com> 1532602858 +0300
            committer Shaul Lahav <slahav@company.com> 1532602858 +0300
                new file8
            :000000 100644 0000000000000000000000000000000000000000 576c717174a2f49675f4b57429424e3d54cf2a07 A README.md
            :000000 100644 0000000000000000000000000000000000000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 A dir1/README2
            :000000 100644 0000000000000000000000000000000000000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 A dir2/README8

            Notice that other, unchanged files, also appear in the output.

            However, When I ran the same command not from Jenkins, I got:

            {{}}

             

            commit d36267d8d88bdd085c894f35b1585a9f1ee9d755
            tree 6efc76859dbd9dee1afd6881f81d2c0aad574168
            parent 645221fb9fb8b752db7736d40c4c311e032482bb
            author Shaul Lahav <slahav@company.com> 1532602858 +0300
            committer Shaul Lahav <slahav@company.com> 1532602858 +0300
                new file8
            :000000 100644 0000000000000000000000000000000000000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 A      dir2/README8
            

            {{}}
            Which is the expected result.

            This is turn causes the PathRestriction extension to iterate all these files instead of only the relevant ones, which means that if at least one of these files matches the "included" regions pattern (I made it so only the "dir1/.*" path is included), it will return PollingResult.SIGNIFICANT instead of PollingResult.NO_CHANGES.

            I verified that Jenkins is using the same git client I used in the command line, which is the one found in the path.
            I can't think of any explanation for this.
            Any ideas??

            Show
            shauli10 Shaul Lahav added a comment - - edited What is causing the problem is that when the Git CLI client calls the " git log " command giving it a range of revisions to filter by (last build's most recent revision to the latest committed revision), all files in the repo/tree are returned instead of just the newly committed file(s). Example: I committed a file named "dir2/README8", and when Jenkins polled the SCM, it invoked: git log --full-history --no-abbrev --format=raw -M -m --raw 645221fb9fb8b752db7736d40c4c311e032482bb..d36267d8d88bdd085c894f35b1585a9f1ee9d755 # timeout=10 Which resulted in the following output: commit d36267d8d88bdd085c894f35b1585a9f1ee9d755 tree 6efc76859dbd9dee1afd6881f81d2c0aad574168 parent 645221fb9fb8b752db7736d40c4c311e032482bb author Shaul Lahav <slahav@company.com> 1532602858 +0300 committer Shaul Lahav <slahav@company.com> 1532602858 +0300     new file8 :000000 100644 0000000000000000000000000000000000000000 576c717174a2f49675f4b57429424e3d54cf2a07 A README.md :000000 100644 0000000000000000000000000000000000000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 A dir1/README2 :000000 100644 0000000000000000000000000000000000000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 A dir2/README8 Notice that other, unchanged files, also appear in the output. However, When I ran the same command not from Jenkins, I got: {{}}   commit d36267d8d88bdd085c894f35b1585a9f1ee9d755 tree 6efc76859dbd9dee1afd6881f81d2c0aad574168 parent 645221fb9fb8b752db7736d40c4c311e032482bb author Shaul Lahav <slahav@company.com> 1532602858 +0300 committer Shaul Lahav <slahav@company.com> 1532602858 +0300     new file8 :000000 100644 0000000000000000000000000000000000000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 A      dir2/README8 {{}} Which is the expected result. This is turn causes the PathRestriction extension to iterate all these files instead of only the relevant ones, which means that if at least one of these files matches the "included" regions pattern (I made it so only the "dir1/.*" path is included), it will return PollingResult.SIGNIFICANT instead of PollingResult.NO_CHANGES. I verified that Jenkins is using the same git client I used in the command line, which is the one found in the path. I can't think of any explanation for this. Any ideas??
            allan_burdajewicz Allan BURDAJEWICZ made changes -
            Hide
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited

            I was able to reproduce this easily and also to find a workaround.

            Reproduce

            I have the repo "user/repoA" that have the following Jenkinsfile:

            node {
                deleteDir()
            
                dir("repoA") {
                    checkout scm: [$class: 'GitSCM', 
                        branches: [[name: 'master']], 
                        extensions: [[$class: 'UserExclusion', excludedUsers: 'jenkinsbot']], 
                        userRemoteConfigs: [[credentialsId: 'mycredentials', url: 'https://1q2w3e4r5t6y7u8i9o@github.example.com/user/repoA.git']]]
                }
                
                dir("repoB") {
                    checkout changelog: false, poll: false, scm: [$class: 'GitSCM', 
                        branches: [[name: 'master']], 
                        extensions: [[$class: 'IgnoreNotifyCommit']], 
                        userRemoteConfigs: [[credentialsId: 'mycredentials', url: 'https://1q2w3e4r5t6y7u8i9o@github.example.com/user/repoB.git']]
                    ]
                }
                
                dir('repoB') {
                    try {
                        sh "git checkout master"
                        sh "git config --global user.name 'jenkinsbot'"
                        sh "git config --global user.email 'jenkinsbot@example.com'"
                        sh "echo ${JOB_NAME}/${BUILD_NUMBER} > test.txt"
                        sh "git add ."
                        sh "git commit -m 'Automated: Updating test.txt during build'"
                        sh "git push"
                    } catch (err) {
                        echo "No change in test.txt"
                    }
                }
            }
            

            The Pipeline jobs looks like the following:

            Expected Behavior

            Achange to "repoA" and "repoB" should trigger polling.
            Polling should ignore "repoA" commits from the "jenkinsbot" user.
            Polling should ignore "repoB".

            Actual Behavior

            So a change to "repoA" and "repoB" should trigger polling.
            Polling should ignore "repoA" commits from the "jenkinsbot" user.
            Polling should ignore "repoB".

            In that particular case, because the user exclusion doe snot work the job builds in loop...

            Workaround

            The workaround is to set the User Exclusion in the Pipeline Definition SCM instead of inside the Jenkinsfile. Doing the following fix this scenario:

            Note: This is not required but in this case it makes sense anyway to use checkout scm in the Jenkinsfile.

            ****

            So it seems that the problem come from polling rules of the SCMs defined in checkout steps (vs. SCM definition in the Pipeline job). This workaround works in a scenario where the the commits that must be ignored are from the repository holding the Jenkinsfile. I have not tested a scenario where the polling rules are set to "repoB" with "poll: true".

            I am hoping this helps to find the root cause here.

            Show
            allan_burdajewicz Allan BURDAJEWICZ added a comment - - edited I was able to reproduce this easily and also to find a workaround. Reproduce I have the repo "user/repoA" that have the following Jenkinsfile: node { deleteDir() dir( "repoA" ) { checkout scm: [$class: 'GitSCM' , branches: [[name: 'master' ]], extensions: [[$class: 'UserExclusion' , excludedUsers: 'jenkinsbot' ]], userRemoteConfigs: [[credentialsId: 'mycredentials' , url: 'https: //1q2w3e4r5t6y7u8i9o@github.example.com/user/repoA.git' ]]] } dir( "repoB" ) { checkout changelog: false , poll: false , scm: [$class: 'GitSCM' , branches: [[name: 'master' ]], extensions: [[$class: 'IgnoreNotifyCommit' ]], userRemoteConfigs: [[credentialsId: 'mycredentials' , url: 'https: //1q2w3e4r5t6y7u8i9o@github.example.com/user/repoB.git' ]] ] } dir( 'repoB' ) { try { sh "git checkout master" sh "git config --global user.name 'jenkinsbot' " sh "git config --global user.email 'jenkinsbot@example.com' " sh "echo ${JOB_NAME}/${BUILD_NUMBER} > test.txt" sh "git add ." sh "git commit -m 'Automated: Updating test.txt during build' " sh "git push" } catch (err) { echo "No change in test.txt" } } } The Pipeline jobs looks like the following: Expected Behavior Achange to "repoA" and "repoB" should trigger polling. Polling should ignore "repoA" commits from the "jenkinsbot" user. Polling should ignore "repoB". Actual Behavior So a change to "repoA" and "repoB" should trigger polling. Polling should ignore "repoA" commits from the "jenkinsbot" user. Polling should ignore "repoB". In that particular case, because the user exclusion doe snot work the job builds in loop... Workaround The workaround is to set the User Exclusion in the Pipeline Definition SCM instead of inside the Jenkinsfile . Doing the following fix this scenario: Note: This is not required but in this case it makes sense anyway to use checkout scm in the Jenkinsfile. **** So it seems that the problem come from polling rules of the SCMs defined in checkout steps (vs. SCM definition in the Pipeline job). This workaround works in a scenario where the the commits that must be ignored are from the repository holding the Jenkinsfile. I have not tested a scenario where the polling rules are set to "repoB" with "poll: true". I am hoping this helps to find the root cause here.
            Hide
            ric Ric Featherstone added a comment -

            Are there plans to address this?

            Working with a scripted pipeline it's possible to define polling as follows

            properties(
                [ 
                    pipelineTriggers([
                        [$class: "SCMTrigger", scmpoll_spec: "H/5 * * * *"],
                    ]) 
                ]
            )
            

            It would be very handy to be able to specify, includes / excludes at this level

            Show
            ric Ric Featherstone added a comment - Are there plans to address this? Working with a scripted pipeline it's possible to define polling as follows properties( [ pipelineTriggers([ [$class: "SCMTrigger" , scmpoll_spec: "H/5 * * * *" ], ]) ] ) It would be very handy to be able to specify, includes / excludes at this level
            shauli10 Shaul Lahav made changes -
            Assignee Shaul Lahav [ shauli10 ] Mark Waite [ markewaite ]
            Hide
            markewaite Mark Waite added a comment -

            There have been some brief discussions about it, but there are no plans to address it.

            Show
            markewaite Mark Waite added a comment - There have been some brief discussions about it, but there are no plans to address it.
            markewaite Mark Waite made changes -
            Assignee Mark Waite [ markewaite ]

              People

              • Assignee:
                Unassigned
                Reporter:
                dvader Darth Vader
              • Votes:
                69 Vote for this issue
                Watchers:
                74 Start watching this issue

                Dates

                • Created:
                  Updated: