Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35988

Equivalent to polling commit exclusions for branch indexing

    Details

    • Type: New Feature
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: branch-api-plugin
    • Labels:
      None
    • Environment:
      Windows
    • Similar Issues:

      Description

      I have an organizational project configured which scans repositories for jenkinsfile. One of the repository (https://github.com/VirtoCommerce/vc-module-jenkinssample) has a following Jenkinsfile defined:

          node 
          {
          	stage 'Checkout'
                  checkout([
                      $class: 'GitSCM', 
                      branches: [[name: '*/master']], 
                      extensions: [[
          			    $class: 'PathRestriction', 
          			    excludedRegions: 'CommonAssemblyInfo\\.cs', 
          			    includedRegions: ''
          		    ]], 
                      userRemoteConfigs: [[
                          url: 'git@github.com:VirtoCommerce/vc-module-jenkinssample.git']]])
          }
      

      As you might see I have an excluded region defined, so changes made to CommonAssemblyInfo.cs don't trigger any builds. However "Branch Indexing" still forces the job to build when I commit changes to "CommonAssemblyInfo.cs". How can I prevent this?

      Btw, the above script works fine in a separate/individual Pipeline Job.

        Attachments

        1. multibranch_indexing_options.png
          multibranch_indexing_options.png
          182 kB
        2. non_multibranch_pipeline_filters.png
          non_multibranch_pipeline_filters.png
          147 kB
        3. screenshot-1.png
          screenshot-1.png
          2 kB
        4. screenshot-2.png
          screenshot-2.png
          95 kB
        5. workaround.txt
          0.9 kB

          Issue Links

            Activity

            woland Alexander Siniouguine created issue -
            woland Alexander Siniouguine made changes -
            Field Original Value New Value
            Description I have an organizational project configured which scans repositories for jenkinsfile. One of the repository (https://github.com/VirtoCommerce/vc-module-jenkinssample) has a following Jenkinsfile defined:

                node
                {
                 stage 'Checkout'
                        checkout([
                            $class: 'GitSCM',
                            branches: [[name: '*/master']],
                            extensions: [[
                 $class: 'PathRestriction',
                 excludedRegions: 'CommonAssemblyInfo\\.cs',
                 includedRegions: ''
                 ]],
                            userRemoteConfigs: [[
                                url: 'git@github.com:VirtoCommerce/vc-module-jenkinssample.git']]])
                }

            As you might see I have an excluded region defined, so changes made to CommonAssemblyInfo.cs don't trigger any builds. However "Branch Indexing" still forces the job to build when I commit changes to "CommonAssemblyInfo.cs". How can I prevent this?

            Btw, the above script works fine in a separate/individual Pipeline Job.
            I have an organizational project configured which scans repositories for jenkinsfile. One of the repository (https://github.com/VirtoCommerce/vc-module-jenkinssample) has a following Jenkinsfile defined:

            {{
                node
                {
                 stage 'Checkout'
                        checkout([
                            $class: 'GitSCM',
                            branches: [[name: '*/master']],
                            extensions: [[
                 $class: 'PathRestriction',
                 excludedRegions: 'CommonAssemblyInfo\\.cs',
                 includedRegions: ''
                 ]],
                            userRemoteConfigs: [[
                                url: 'git@github.com:VirtoCommerce/vc-module-jenkinssample.git']]])
                }
            }}
            As you might see I have an excluded region defined, so changes made to CommonAssemblyInfo.cs don't trigger any builds. However "Branch Indexing" still forces the job to build when I commit changes to "CommonAssemblyInfo.cs". How can I prevent this?

            Btw, the above script works fine in a separate/individual Pipeline Job.
            markewaite Mark Waite made changes -
            Assignee Mark Waite [ markewaite ]
            woland Alexander Siniouguine made changes -
            Component/s github-organization-folder-plugin [ 21427 ]
            Component/s git-plugin [ 15543 ]
            woland Alexander Siniouguine made changes -
            Component/s workflow-multibranch-plugin [ 21465 ]
            woland Alexander Siniouguine made changes -
            Component/s workflow-multibranch-plugin [ 21465 ]
            dvader Darth Vader made changes -
            Link This issue is related to JENKINS-36195 [ JENKINS-36195 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 172625 ] JNJira + In-Review [ 184698 ]
            jglick Jesse Glick made changes -
            Description I have an organizational project configured which scans repositories for jenkinsfile. One of the repository (https://github.com/VirtoCommerce/vc-module-jenkinssample) has a following Jenkinsfile defined:

            {{
                node
                {
                 stage 'Checkout'
                        checkout([
                            $class: 'GitSCM',
                            branches: [[name: '*/master']],
                            extensions: [[
                 $class: 'PathRestriction',
                 excludedRegions: 'CommonAssemblyInfo\\.cs',
                 includedRegions: ''
                 ]],
                            userRemoteConfigs: [[
                                url: 'git@github.com:VirtoCommerce/vc-module-jenkinssample.git']]])
                }
            }}
            As you might see I have an excluded region defined, so changes made to CommonAssemblyInfo.cs don't trigger any builds. However "Branch Indexing" still forces the job to build when I commit changes to "CommonAssemblyInfo.cs". How can I prevent this?

            Btw, the above script works fine in a separate/individual Pipeline Job.
            I have an organizational project configured which scans repositories for jenkinsfile. One of the repository (https://github.com/VirtoCommerce/vc-module-jenkinssample) has a following Jenkinsfile defined:

            {code}
                node
                {
                 stage 'Checkout'
                        checkout([
                            $class: 'GitSCM',
                            branches: [[name: '*/master']],
                            extensions: [[
                 $class: 'PathRestriction',
                 excludedRegions: 'CommonAssemblyInfo\\.cs',
                 includedRegions: ''
                 ]],
                            userRemoteConfigs: [[
                                url: 'git@github.com:VirtoCommerce/vc-module-jenkinssample.git']]])
                }
            {code}

            As you might see I have an excluded region defined, so changes made to CommonAssemblyInfo.cs don't trigger any builds. However "Branch Indexing" still forces the job to build when I commit changes to "CommonAssemblyInfo.cs". How can I prevent this?

            Btw, the above script works fine in a separate/individual Pipeline Job.
            jglick Jesse Glick made changes -
            Component/s branch-api-plugin [ 18621 ]
            Component/s github-organization-folder-plugin [ 21427 ]
            Component/s workflow-multibranch-plugin [ 21465 ]
            Hide
            jglick Jesse Glick added a comment -

            The Branch API does not support polling exclusions as such. It merely defines revisions. For a GitHub branch source to support these would be rather difficult, I think. Git polling generally does not even support this mode without using workspace-based polling, which is usually a bad idea.

            Show
            jglick Jesse Glick added a comment - The Branch API does not support polling exclusions as such. It merely defines revisions. For a GitHub branch source to support these would be rather difficult, I think. Git polling generally does not even support this mode without using workspace-based polling, which is usually a bad idea.
            jglick Jesse Glick made changes -
            Issue Type Improvement [ 4 ] New Feature [ 2 ]
            Component/s github-branch-source-plugin [ 20858 ]
            Component/s branch-api-plugin [ 18621 ]
            Hide
            vorobievalex Alexander Vorobiev added a comment -

            Can this functionality be moved to later stages?
            So the repository is fetched, but some steps can be skipped in case there are no appropriate changes as described in JENKINS-37978.

            Show
            vorobievalex Alexander Vorobiev added a comment - Can this functionality be moved to later stages? So the repository is fetched, but some steps can be skipped in case there are no appropriate changes as described in JENKINS-37978 .
            Hide
            jglick Jesse Glick added a comment -

            Alexander Vorobiev you can do that yourself already, using currentBuild.changeSets.

            Show
            jglick Jesse Glick added a comment - Alexander Vorobiev you can do that yourself already, using currentBuild.changeSets .
            jglick Jesse Glick made changes -
            Component/s branch-api-plugin [ 18621 ]
            Component/s github-branch-source-plugin [ 20858 ]
            Hide
            jglick Jesse Glick added a comment - - edited

            The existing SCM.compareRemoteRevisionWith implementing commit exclusions for traditional polling cannot be used from multibranch projects since it relies on SCMRevisionAction whereas we need to deal with SCMRevision. So we would need a new API in scm-api to provide metadata, similar to SCMFileSystem.changesSince. Then it would need to implemented in branch source plugins; for example

            Show
            jglick Jesse Glick added a comment - - edited The existing SCM.compareRemoteRevisionWith implementing commit exclusions for traditional polling cannot be used from multibranch projects since it relies on SCMRevisionAction whereas we need to deal with SCMRevision . So we would need a new API in scm-api to provide metadata, similar to SCMFileSystem.changesSince . Then it would need to implemented in branch source plugins; for example in git we could use the existing master repository cache to supply such metadata (no need for a workspace) in github-branch-source we could probably get that information from the GitHub API
            jglick Jesse Glick made changes -
            Summary Organization job "Branch Indexing" ignores excluded regions defined in jenkinsfile Equivalent to polling commit exclusions for branch indexing
            Hide
            maxfields2000 Maxfield Stewart added a comment -

            This is definitely a much needed feature for easier adoption. Being able to just not run a build at all if a commit message or particular user checked in helps resolve various builds that decide they need to check back into github (not the least being the pesky maven-release plugin) but likewise other more complex build flows that need to update meta-data in the source repo with build information.

            Workarounds are cumbersome and result in the build log being full of aborted builds, it also turns a lot of my users off because they could get this capability in existing job types so it stops them from using these awesome new Jenkins features.

            Show
            maxfields2000 Maxfield Stewart added a comment - This is definitely a much needed feature for easier adoption. Being able to just not run a build at all if a commit message or particular user checked in helps resolve various builds that decide they need to check back into github (not the least being the pesky maven-release plugin) but likewise other more complex build flows that need to update meta-data in the source repo with build information. Workarounds are cumbersome and result in the build log being full of aborted builds, it also turns a lot of my users off because they could get this capability in existing job types so it stops them from using these awesome new Jenkins features.
            krever Wojciech Pituła made changes -
            Attachment workaround.txt [ 38191 ]
            Hide
            krever Wojciech Pituła added a comment - - edited

            workaround.txt- This is a workaround we have used.

            Show
            krever Wojciech Pituła added a comment - - edited workaround.txt - This is a workaround we have used.
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-43842 [ JENKINS-43842 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-44371 [ JENKINS-44371 ]
            Hide
            uxian Gerard Krupa added a comment -

            I've been using a similar workaround but extended it to copy the result of the previous build (again, less than ideal since it's pretty clunky and requires some hairy APIs to be white-listed):

             

            currentBuild.setResult(currentBuild.rawBuild.getPreviousBuild()?.result?.toString())
            

            A slightly better workaround (though still not ideal), from my point of view, would be some way of causing the current build to be deleted as soon as it completes so that the UI doesn't get polluted with these alternating builds/non-builds.

             

            Show
            uxian Gerard Krupa added a comment - I've been using a similar workaround but extended it to copy the result of the previous build (again, less than ideal since it's pretty clunky and requires some hairy APIs to be white-listed):   currentBuild.setResult(currentBuild.rawBuild.getPreviousBuild()?.result?.toString()) A slightly better workaround (though still not ideal), from my point of view, would be some way of causing the current build to be deleted as soon as it completes so that the UI doesn't get polluted with these alternating builds/non-builds.  
            jglick Jesse Glick made changes -
            Remote Link This issue links to "CloudBees-internal mirror (Web Link)" [ 20187 ]
            Hide
            jglick Jesse Glick added a comment -

            The currently best workaround I know of is to inspect currentBuild.changeSets from your Jenkinsfile and exit early. JENKINS-27092 would make that a bit easier, as well as allowing visualizations such as Blue Ocean to automatically hide the build or fold it into an adjoining build with a normal completion status.

            Show
            jglick Jesse Glick added a comment - The currently best workaround I know of is to inspect currentBuild.changeSets from your Jenkinsfile and exit early.  JENKINS-27092 would make that a bit easier, as well as allowing visualizations such as Blue Ocean to automatically hide the build or fold it into an adjoining build with a normal completion status.
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-27092 [ JENKINS-27092 ]
            Hide
            jglick Jesse Glick added a comment -

            Some more discussion in git PR 573.

            Show
            jglick Jesse Glick added a comment - Some more discussion in git PR 573.
            jglick Jesse Glick made changes -
            Remote Link This issue links to "git PR 573 (Web Link)" [ 20188 ]
            Hide
            leedega Kevin Phillips added a comment -

            So, I just wanted to confirm a few things relating to this feature request.

            First, there seems to be 2 slightly different use cases relating to branch indexing as I see it. The first use case is when there is a multibranch pipeline job configured with branch indexing which then loads a Jenkinsfile containing a git() build step pointing to the same git repository as the Jenkinsfile itself but with customized polling options. The other use case is when there is a multibranch pipeline job configured with branch indexing but the polling options are configured directly in the multibranch pipeline job itself and not in the Jenkinsfiles being indexed. The second use case seems like a more focused and isolated (and perhaps simpler implement) variation of the first. This improvement seems to be focused on the former use case, but I think we need to consider the latter use case as well. Does this sound reasonable?

            In regards to the second use case I just mentioned, there currently doesn't seem to be any way to filter branch indexing triggers in any way even though similarly configured non-multibranch pipeline jobs provide several such options (ie: "Polling ignores commits with certain messages", "Polling ignores commits from certain users", etc.). If I am understanding correctly, this difference in behavior appears to be intentional due to some technical limitation in git. Could someone clarify what this technical limitation is exactly? Is it that the git toolset requires a workspace in order to perform this filtering? That doesn't seem to hold true for non-multibranch pipelines since the polling operation applies before the source repository is checked out (or at least it seems this way based on my crude understanding of the mechanics involved).

            Perhaps as a first step someone could look into adding support for basic branch indexing filters to multibranch pipeline jobs themselves and then perhaps look to extend that behavior to support further customizations from the indexed Jenkinsfiles.

            If you think these 2 slightly different use cases are independent enough to warrant their own feature requests I'd be happy to create a new one for the second use case and link it to this one instead.

            Show
            leedega Kevin Phillips added a comment - So, I just wanted to confirm a few things relating to this feature request. First, there seems to be 2 slightly different use cases relating to branch indexing as I see it. The first use case is when there is a multibranch pipeline job configured with branch indexing which then loads a Jenkinsfile containing a git() build step pointing to the same git repository as the Jenkinsfile itself but with customized polling options. The other use case is when there is a multibranch pipeline job configured with branch indexing but the polling options are configured directly in the multibranch pipeline job itself and not in the Jenkinsfiles being indexed. The second use case seems like a more focused and isolated (and perhaps simpler implement) variation of the first. This improvement seems to be focused on the former use case, but I think we need to consider the latter use case as well. Does this sound reasonable? In regards to the second use case I just mentioned, there currently doesn't seem to be any way to filter branch indexing triggers in any way even though similarly configured non-multibranch pipeline jobs provide several such options (ie: "Polling ignores commits with certain messages", "Polling ignores commits from certain users", etc.). If I am understanding correctly, this difference in behavior appears to be intentional due to some technical limitation in git. Could someone clarify what this technical limitation is exactly? Is it that the git toolset requires a workspace in order to perform this filtering? That doesn't seem to hold true for non-multibranch pipelines since the polling operation applies before the source repository is checked out (or at least it seems this way based on my crude understanding of the mechanics involved). Perhaps as a first step someone could look into adding support for basic branch indexing filters to multibranch pipeline jobs themselves and then perhaps look to extend that behavior to support further customizations from the indexed Jenkinsfiles. If you think these 2 slightly different use cases are independent enough to warrant their own feature requests I'd be happy to create a new one for the second use case and link it to this one instead.
            leedega Kevin Phillips made changes -
            leedega Kevin Phillips made changes -
            Attachment multibranch_indexing_options.png [ 41758 ]
            Hide
            leedega Kevin Phillips added a comment -

            Here are a couple of screen shots illustrating the discrepancy between the multibranch and non-multibranch pipeline jobs, which seems to have no obvious explanation:

            Show
            leedega Kevin Phillips added a comment - Here are a couple of screen shots illustrating the discrepancy between the multibranch and non-multibranch pipeline jobs, which seems to have no obvious explanation:
            Hide
            leedega Kevin Phillips added a comment - - edited

            I probably should also clarify that the more limited and isolated use case (#2) I mentioned in my earlier comment should ONLY apply a given filter when performing the branch indexing operation, at least as a first pass. So this would allow builds to be excluded ONLY when commits are made to the repository being indexed, not to any other repositories that may be checked out from the Jenkinsfiles that are detected. My hope would be that this would be easier to implement as well since the complexities involved with the polling mechanics should be greatly simplified. Then it could be treated as a future improvement to possibly extract nested Git configurations from the Jenkinsfiles if so desired.

            Show
            leedega Kevin Phillips added a comment - - edited I probably should also clarify that the more limited and isolated use case (#2) I mentioned in my earlier comment should ONLY apply a given filter when performing the branch indexing operation, at least as a first pass. So this would allow builds to be excluded ONLY when commits are made to the repository being indexed, not to any other repositories that may be checked out from the Jenkinsfiles that are detected. My hope would be that this would be easier to implement as well since the complexities involved with the polling mechanics should be greatly simplified. Then it could be treated as a future improvement to possibly extract nested Git configurations from the Jenkinsfiles if so desired.
            Hide
            acejam Joshua Noble added a comment -

            I'm also looking for a way to exclude/ignore commits from a specific Git author. For example, an automated user who bumps the package.json version of an npm module. If that automated user pushes back to the repository after a build, that push will trigger yet another build. Is there a way to ignore/filter out events from certain commit authors? From what I can tell, the webhook events from GitHub include commit author information when a push occurs.

            Show
            acejam Joshua Noble added a comment - I'm also looking for a way to exclude/ignore commits from a specific Git author. For example, an automated user who bumps the package.json version of an npm module. If that automated user pushes back to the repository after a build, that push will trigger yet another build. Is there a way to ignore/filter out events from certain commit authors? From what I can tell, the webhook events from GitHub include commit author information when a push occurs.
            Hide
            maxime_lem Maxime Lemanissier added a comment - - edited

            I'm having a similar issue using pipeline library.

            When i merge a change on the master branch of my jenkins pipeline library, all jobs using this library (more than 100...) are all rebuilt simultaneously, which fills Jenkins queue for hours everytime a change is made on the library...

            This default behavior is probably well intended, as if the pipeline changes, all jobs using should be rebuilt, but when having a lot of jobs, this becomes unpractical/unusable. I would prefer to wait for the next manual build or SCM change on the project repository (not on the lib one) to trigger a new build.

            Is there a way to not trigger builds due to changes on the pipeline library?

            If not, can i use a specific naming convention for the master branch of the pipeline library, and excluding it from polling as shared in previous comment? Will this exclusion also apply on pipeline library?

            Show
            maxime_lem Maxime Lemanissier added a comment - - edited I'm having a similar issue using pipeline library. When i merge a change on the master branch of my jenkins pipeline library, all jobs using this library (more than 100...) are all rebuilt simultaneously, which fills Jenkins queue for hours everytime a change is made on the library... This default behavior is probably well intended, as if the pipeline changes, all jobs using should be rebuilt, but when having a lot of jobs, this becomes unpractical/unusable. I would prefer to wait for the next manual build or SCM change on the project repository (not on the lib one) to trigger a new build. Is there a way to not trigger builds due to changes on the pipeline library? If not, can i use a specific naming convention for the master branch of the pipeline library, and excluding it from polling as shared in previous comment? Will this exclusion also apply on pipeline library?
            Hide
            natefaerber Nate Faerber added a comment -

            I believe this is the feature I've been looking for.  It makes sense with the introduction of custom Jenkinsfile path to allow us to ignore changes that occur outside of the Jenkinsfile path.  Even that might be too restrictive.  It would solve a lot of people's problems if we can just specific the subdirectory we want to poll.  This would allow folks to put their Jenkinsfiles in completely different paths.

            Show
            natefaerber Nate Faerber added a comment - I believe this is the feature I've been looking for.  It makes sense with the introduction of custom Jenkinsfile path to allow us to ignore changes that occur outside of the Jenkinsfile path.  Even that might be too restrictive.  It would solve a lot of people's problems if we can just specific the subdirectory we want to poll.  This would allow folks to put their Jenkinsfiles in completely different paths.
            nlowe_hyland Nathan Lowe made changes -
            Attachment screenshot-1.png [ 45330 ]
            nlowe_hyland Nathan Lowe made changes -
            Attachment screenshot-2.png [ 45331 ]
            Hide
            nlowe_hyland Nathan Lowe added a comment -

            We'd love to see this as well. The closest I can get now is a step that scripts the currentBuild.result to ABORTED and fail the build. This shows up as skipped in some places:

            But failed in others:

            And jenkins reports the build as failed to Bitbucket.

            Show
            nlowe_hyland Nathan Lowe added a comment - We'd love to see this as well. The closest I can get now is a step that scripts the currentBuild.result to ABORTED and fail the build. This shows up as skipped in some places: But failed in others: And jenkins reports the build as failed to Bitbucket.
            Hide
            webrat Andreas Sieferlinger added a comment -

            any update on this?

            Show
            webrat Andreas Sieferlinger added a comment - any update on this?
            Hide
            tknerr Torben Knerr added a comment -

            +1 to what Kevin Phillips mentioned above. "Polling ignores commits in certain paths" is available for "normal" Pipelines, just not for Multibranch Pipelines.

            Jesse Glick any chance the extension to scm-api you mentioned in https://issues.jenkins-ci.org/browse/JENKINS-35988?focusedCommentId=295310&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-295310 are on the roadmap?

            (in my case, it would even be for the bitbucket-branch-source-plugin....)

            Show
            tknerr Torben Knerr added a comment - +1 to what Kevin Phillips mentioned above. "Polling ignores commits in certain paths" is available for "normal" Pipelines, just not for Multibranch Pipelines. Jesse Glick any chance the extension to scm-api you mentioned in https://issues.jenkins-ci.org/browse/JENKINS-35988?focusedCommentId=295310&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-295310 are on the roadmap? (in my case, it would even be for the bitbucket-branch-source-plugin....)
            Show
            bitwiseman Liam Newman added a comment - - edited Joshua Noble Ignore by commiter: https://github.com/jenkinsci/ignore-committer-strategy-plugin Torben Knerr Andreas Sieferlinger Alexander Siniouguine Nathan Lowe Nate Faerber Kevin Phillips I think this does what is being requested here: https://wiki.jenkins.io/display/JENKINS/Multibranch+Build+Strategy+Extension+Plugin
            bitwiseman Liam Newman made changes -
            Assignee Liam Newman [ bitwiseman ]
            Hide
            bitwiseman Liam Newman added a comment -

            Resolving for now based on existing plugin. Please comment if not done.

            Show
            bitwiseman Liam Newman added a comment - Resolving for now based on existing plugin. Please comment if not done.
            bitwiseman Liam Newman made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            bryceman Bryce Schober added a comment -

            Liam Newman Note that the ignore-committer-strategy plugin doesn't support simple (non-email) subversion usernames, and at least superficially (at least in its UI wording) only supports Git. See also JENKINS-57474.

            Show
            bryceman Bryce Schober added a comment - Liam Newman Note that the ignore-committer-strategy plugin doesn't support simple (non-email) subversion usernames, and at least superficially (at least in its UI wording) only supports Git. See also JENKINS-57474 .
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-57164 [ JENKINS-57164 ]
            bitwiseman Liam Newman made changes -
            Link This issue is related to JENKINS-57474 [ JENKINS-57474 ]
            Hide
            bitwiseman Liam Newman added a comment -

            Bryce Schober
            Thanks for filing a new JIRA for that issue.

            Show
            bitwiseman Liam Newman added a comment - Bryce Schober Thanks for filing a new JIRA for that issue.

              People

              • Assignee:
                bitwiseman Liam Newman
                Reporter:
                woland Alexander Siniouguine
              • Votes:
                32 Vote for this issue
                Watchers:
                39 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: