Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54864

github multibranch builds fail to build with latest branch api update

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: branch-api-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.138.3 on Ubuntu
      git plugin 3.9.1 and branch api plugin 2.1.1 is broken.
      git plugin 3.9.0 and branch api plugin 2.0.21 is ok.
    • Similar Issues:
    • Released As:
      branch-api 2.5.4

      Description

      The latest updates broke github multibranch builds.

      The branches are being discovered, but all automaticdiscovery of all new changes on all branches results in the following typical log entries:

      Checking pull-requests...
       
      Checking pull request #3
      ‘Jenkinsfile’ found
      Met criteria
      Changes detected: PR-3 (null → 24d2f4cb5debd9b3f6f8c86383eb077be0dee0c4+517f061b1a7db844659ca98d1b61a7dcd0b6fb82)
      No automatic builds for PR-3

      These builds should have triggered.  Backing out to the last stable set of plugins fixes the problem.

        Attachments

          Issue Links

            Activity

            Hide
            jasoncrow Jason Crow added a comment - - edited

            Having the same problem PR's are detected but never built.  I've reverted and PR's are building again. However there are error messages showing:

             
            Dependency errors:

            Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to restore the functionality provided by these plugins.
            Basic Branch Build Strategies Plugin version 1.1.1Branch API Plugin v2.0.21 is older than required. To fix, install v2.1.0 or later.
            Go to plugin managerConfigure which of these warnings are shown
             
             Warnings have been published for the following currently installed components.Git plugin 3.9.0 Server-side request forgery

            Show
            jasoncrow Jason Crow added a comment - - edited Having the same problem PR's are detected but never built.  I've reverted and PR's are building again. However there are error messages showing:   Dependency errors: Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to restore the functionality provided by these plugins. Basic Branch Build Strategies Plugin version 1.1.1Branch API Plugin v2.0.21 is older than required. To fix, install v2.1.0 or later. Go to plugin managerConfigure which of these warnings are shown    Warnings have been published for the following currently installed components. Git plugin 3.9.0 Server-side request forgery
            Hide
            markewaite Mark Waite added a comment -

            It should be enough to revert branch API plugin 2.1.1.

            The changes between git plugin 3.9.0 and 3.9.1 added a requirement that POST must be used when checking the repository browser (SECURITY-810) and some minor automated testing changes.

            Show
            markewaite Mark Waite added a comment - It should be enough to revert branch API plugin 2.1.1. The changes between git plugin 3.9.0 and 3.9.1 added a requirement that POST must be used when checking the repository browser (SECURITY-810) and some minor automated testing changes.
            Hide
            jasoncrow Jason Crow added a comment -

            Cool, works with just the API plugin reverted.

            Show
            jasoncrow Jason Crow added a comment - Cool, works with just the API plugin reverted.
            Hide
            jasoncrow Jason Crow added a comment -

            Does anyone know what's going on with branch api plugin 2.1.1? We have continuous deployment pipeline that's dependent on PR builds to pass github checks and the PR's never build with this plugin version.

            Show
            jasoncrow Jason Crow added a comment - Does anyone know what's going on with branch api plugin 2.1.1? We have continuous deployment pipeline that's dependent on PR builds to pass github checks and the PR's never build with this plugin version.
            Hide
            markewaite Mark Waite added a comment -

            rsandell and Jesse Glick were involved in the most recent release of the branch API plugin, though the changes from branch api plugin 2.1.0 to 2.1.1 seem unlikely to have caused this change of behavior.

            Show
            markewaite Mark Waite added a comment - rsandell and Jesse Glick were involved in the most recent release of the branch API plugin, though the changes from branch api plugin 2.1.0 to 2.1.1 seem unlikely to have caused this change of behavior.
            Hide
            jglick Jesse Glick added a comment -

            Perhaps caused by JENKINS-47859 by Stephen Connolly. Just guessing from the plugin changelog.

            Show
            jglick Jesse Glick added a comment - Perhaps caused by JENKINS-47859 by Stephen Connolly . Just guessing from the plugin changelog.
            Hide
            jasoncrow Jason Crow added a comment -

            Mark Waite I don't think it was between 2.1.0 and 2.1.1, I had to roll back to 2.0.21 to return to a "semi-functioning" pipeline - still seeing unexplained issues with PR builds with discovery and status reporting back to github.

            Show
            jasoncrow Jason Crow added a comment - Mark Waite I don't think it was between 2.1.0 and 2.1.1, I had to roll back to 2.0.21 to return to a "semi-functioning" pipeline - still seeing unexplained issues with PR builds with discovery and status reporting back to github.
            Hide
            stephenconnolly Stephen Connolly added a comment -

            I suspect you had "Automatic branch project triggering" configured to something like master, PR-*

            This gets mapped to matching branches that have the set of names as automatic build, but doesn't include Pull requests in automatic builds. As a workaround, just add Pull Requests to the build strategies and pull requests should start building.

            If somebody is interested in fixing this you would need to:

            • Establish that all SCMSource implementations only use PR- as the prefix for pull requests... if there is an SCMSource implementation that uses something like CR- then we cannot fix this issue as we would not know if PR-* is intended to target branches that start with PR- or pull requests.
            • If we know that PR- is guaranteed to only apply to pull requests, then modify the migration so that it parses the "Automatic branch project triggering" value, pulls out the PR- ones and adds named branch build strategies for all others. If there is any PR- entries then add the change request build strategy
            Show
            stephenconnolly Stephen Connolly added a comment - I suspect you had "Automatic branch project triggering" configured to something like master, PR-* This gets mapped to matching branches that have the set of names as automatic build, but doesn't include Pull requests in automatic builds. As a workaround, just add Pull Requests to the build strategies and pull requests should start building. If somebody is interested in fixing this you would need to: Establish that all SCMSource implementations only use PR- as the prefix for pull requests... if there is an SCMSource implementation that uses something like CR- then we cannot fix this issue as we would not know if PR-* is intended to target branches that start with PR- or pull requests. If we know that PR- is guaranteed to only apply to pull requests, then modify the migration so that it parses the "Automatic branch project triggering" value, pulls out the PR- ones and adds named branch build strategies for all others. If there is any PR- entries then add the change request build strategy
            Hide
            jglick Jesse Glick added a comment - - edited

            AFAICT you need to install a new Basic Branch Build Strategies plugin to recover that functionality, and there should have been an administrative monitor displayed to alert you to this fact. (pay attention to what Stephen Connolly wrote instead)

            Show
            jglick Jesse Glick added a comment - - edited AFAICT you need to install a new Basic Branch Build Strategies plugin to recover that functionality, and there should have been an administrative monitor displayed to alert you to this fact. (pay attention to what Stephen Connolly wrote instead)
            Hide
            jasoncrow Jason Crow added a comment - - edited

            Stephen Connolly My Automatic branch project triggering is set to .*

            The Basic Branch Build Strategies plugin is installed, when I went through this upgrade the only way I got PR's to build automatically again was to rollback the branch-api-plugin to 2.0.21.  Hence the following message:

            Dependency errors:

            Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to restore the functionality provided by these plugins.

            Basic Branch Build Strategies Plugin version 1.1.1

            Branch API Plugin v2.0.21 is older than required. To fix, install v2.1.0 or later.

            Show
            jasoncrow Jason Crow added a comment - - edited Stephen Connolly My Automatic branch project triggering is set to .* The Basic Branch Build Strategies plugin is installed, when I went through this upgrade the only way I got PR's to build automatically again was to rollback the branch-api-plugin to 2.0.21.  Hence the following message: Dependency errors: Some plugins could not be loaded due to unsatisfied dependencies. Fix these issues and restart Jenkins to restore the functionality provided by these plugins. Basic Branch Build Strategies Plugin version 1.1.1 Branch API Plugin v2.0.21 is older than required. To fix, install v2.1.0 or later.
            Hide
            spmason Steve Mason added a comment -

            Just to add - we had this problem and adding the "Change requests" build strategy seems to have made our PRs build automatically again on the latest version of the plugin

            Show
            spmason Steve Mason added a comment - Just to add - we had this problem and adding the "Change requests" build strategy seems to have made our PRs build automatically again on the latest version of the plugin
            Hide
            jasoncrow Jason Crow added a comment -

            Thanks Steve Mason this fixes the issue. I think this was more a difference in plugin functionality than anything else. This issue could probably be closed.

            Show
            jasoncrow Jason Crow added a comment - Thanks Steve Mason this fixes the issue. I think this was more a difference in plugin functionality than anything else. This issue could probably be closed.
            Hide
            spmason Steve Mason added a comment -

            The only bug I can see is that this config should have been copied over from the previous version correctly

            Show
            spmason Steve Mason added a comment - The only bug I can see is that this config should have been copied over from the previous version correctly
            Hide
            stephenconnolly Stephen Connolly added a comment - - edited

            So if somebody can come up with an expression parser for the old configuration string that can reliably distinguish between:

            • PR-* means build all branches whos names begin with PR- such as PR-foobar and do not build pull requests because our SCMSource doesn't even have the concept.
            • PR-* means build all pull-requests but we don't want to accidentally build branches with names that begin with PR- because we just want to build the master branch.
            • CR-* means build all pull-requests because our SCMSource calls them Change Requests and uses CR- as the prefix

            I would be happy to merge.

            Until then, the least risk (given that building a branch can result in deploying things into production or expose secrets to a build that isn't trusted) is to err on the side of "if we don't know, do nothing".

            In any case, we could argue that the old `Automatic branch project triggering` has a bug of requiring you to know to add PR-* in order to build pull requests... just that people were relying on the bug!

            Show
            stephenconnolly Stephen Connolly added a comment - - edited So if somebody can come up with an expression parser for the old configuration string that can reliably distinguish between: PR-* means build all branches whos names begin with PR- such as PR-foobar and do not build pull requests because our SCMSource doesn't even have the concept. PR-* means build all pull-requests but we don't want to accidentally build branches with names that begin with PR- because we just want to build the master branch. CR-* means build all pull-requests because our SCMSource calls them Change Requests and uses CR- as the prefix I would be happy to merge. Until then, the least risk (given that building a branch can result in deploying things into production or expose secrets to a build that isn't trusted) is to err on the side of "if we don't know, do nothing". In any case, we could argue that the old `Automatic branch project triggering` has a bug of requiring you to know to add PR-* in order to build pull requests... just that people were relying on the bug!
            Hide
            stephenconnolly Stephen Connolly added a comment -

            And just in case somebody says "who'd call your branches PR-*?" suppose you are using JIRA and your JIRA project ID is PR... then it might make extreme sense to name all your branches by the JIRA ID that they are developing... so that JIRA ID PR-5612 is in the branch PR-5612... if you are using an SCMSource that doesn't have Change Requests... perhaps you rolled a custom one similar to https://github.com/stephenc/asf-gitpubsub-jenkins-plugin for your own in-house GIT hosting...

            You might not want to build any of the OPS ticket branches on your Jenkins, so you just set the automatic building to master,PR-*

            So if we are going to embed the logic to figure these things out into code... we should handle the above!

            Show
            stephenconnolly Stephen Connolly added a comment - And just in case somebody says "who'd call your branches PR-* ?" suppose you are using JIRA and your JIRA project ID is PR ... then it might make extreme sense to name all your branches by the JIRA ID that they are developing... so that JIRA ID PR-5612 is in the branch PR-5612 ... if you are using an SCMSource that doesn't have Change Requests... perhaps you rolled a custom one similar to https://github.com/stephenc/asf-gitpubsub-jenkins-plugin for your own in-house GIT hosting... You might not want to build any of the OPS ticket branches on your Jenkins, so you just set the automatic building to master,PR-* So if we are going to embed the logic to figure these things out into code... we should handle the above!
            Hide
            jtancer Jon Tancer added a comment -

            I can confirm Steve Mason's solution fixed our issue.

            For posterity:

            If you use GitHub Organizations with Jenkins and Jenkins is no longer building commits from PR branches, perform the following steps to resolve the issue

            1. Open the GitHub Organization folder
            2. On the left hand side, select Configure
            3. Scroll down to Build Strategies which is nested underneath the Projects section
            4. Press Add
            5. Select Change requests
            6. Scroll down to the bottom of the page and press Save

            {{}}{{}}You will now see the GitHub Organization being re-scanned and all your missing PR builds will begin to build.

             

            Show
            jtancer Jon Tancer added a comment - I can confirm Steve Mason 's solution fixed our issue. For posterity: If you use GitHub Organizations with Jenkins and Jenkins is no longer building commits from PR branches, perform the following steps to resolve the issue Open the GitHub Organization folder On the left hand side, select Configure Scroll down to Build Strategies  which is nested underneath the Projects section Press Add Select Change requests Scroll down to the bottom of the page and press  Save {{}}{{}}You will now see the GitHub Organization being re-scanned and all your missing PR builds will begin to build.  
            Hide
            alt_jmellor John Mellor added a comment -

            Can someone please detail the alleged workaround details?  I do not see the config sections noted by Steve Mason and Jon Tancer in either of my sites.  I see the note above explaining what I should alter in the github section config when I use the deprecated and replaced organization plugin, but not how to do this in an up-to-date system.  Is it possible to perform a workaround without using the do-not-use plugins?

             

            Show
            alt_jmellor John Mellor added a comment - Can someone please detail the alleged workaround details?  I do not see the config sections noted by Steve Mason and Jon Tancer in either of my sites.  I see the note above explaining what I should alter in the github section config when I use the deprecated and replaced organization plugin, but not how to do this in an up-to-date system.  Is it possible to perform a workaround without using the do-not-use plugins?  
            Hide
            markewaite Mark Waite added a comment -

            John Mellor did you review the comment from Stephen Connolly? He noted:

            Add Pull Requests to the build strategies and pull requests should start building.

            Show
            markewaite Mark Waite added a comment - John Mellor did you review the comment from Stephen Connolly ? He noted: Add Pull Requests to the build strategies and pull requests should start building.
            Hide
            alt_jmellor John Mellor added a comment -

            Sorry for being obtuse, but pull requests ARE in the build stategies already.  I do not see what I should have to add to all the jobs.  Typical job screenshot attached.  What needs to be added?

            Show
            alt_jmellor John Mellor added a comment - Sorry for being obtuse, but pull requests ARE in the build stategies already.  I do not see what I should have to add to all the jobs.  Typical job screenshot attached.  What needs to be added?
            Hide
            spmason Steve Mason added a comment - - edited

            John Mellor That section is "GitHub Organisation", you want to add the "Change requests" strategy under the "Build strategies" section. See this screenshot:

             

            Show
            spmason Steve Mason added a comment - - edited John Mellor That section is "GitHub Organisation", you want to add the "Change requests" strategy under the "Build strategies" section. See this screenshot:  
            Hide
            alt_jmellor John Mellor added a comment -

            Thanks Steve, but I still do not have enough context.  I do not have that section in the job configs like you do.  What plugin provides that configuration section?  Screenshot of what I see attached.

            Show
            alt_jmellor John Mellor added a comment - Thanks Steve, but I still do not have enough context.  I do not have that section in the job configs like you do.  What plugin provides that configuration section?  Screenshot of what I see attached.
            Hide
            aairey Andy Airey added a comment -

            I'm also facing this issue and also didn't see this option (using Bitbucket Branch Source Plugin).
            It only appears when you migrate to the "Named Branch Plugin", following this message from the Recommendations (the number with red background in the top bar of the UI in Jenkins):

            The "Automatic branch project triggering" option has been replaced with a "Named branch" build strategy

            Show
            aairey Andy Airey added a comment - I'm also facing this issue and also didn't see this option (using Bitbucket Branch Source Plugin). It only appears when you migrate to the "Named Branch Plugin", following this message from the Recommendations (the number with red background in the top bar of the UI in Jenkins): The "Automatic branch project triggering" option has been replaced with a "Named branch" build strategy
            Hide
            markewaite Mark Waite added a comment -

            Andy Airey for the benefit of others that may refer to this bug report, would you be willing to attach a screenshot of the Bitbucket Branch Source Plugin configuration setting?

            I realize that this bug report is about the GitHub multibranch, but many readers may find it and assume that Bitbucket and GitHub plugins have the same options and same configurations (which they do not).

            Show
            markewaite Mark Waite added a comment - Andy Airey  for the benefit of others that may refer to this bug report, would you be willing to attach a screenshot of the Bitbucket Branch Source Plugin configuration setting? I realize that this bug report is about the GitHub multibranch, but many readers may find it and assume that Bitbucket and GitHub plugins have the same options and same configurations (which they do not).
            Hide
            sebgagn Sebastien Gagnon added a comment -

            Hi,

            Same problem on my side.

            Any update on this issue ?

            Thanks

            Show
            sebgagn Sebastien Gagnon added a comment - Hi, Same problem on my side. Any update on this issue ? Thanks
            Hide
            markewaite Mark Waite added a comment -

            Sebastien Gagnon did you apply the instructions in the summary of the configuration change that is usually the best way to resolve the issue?

            Show
            markewaite Mark Waite added a comment - Sebastien Gagnon did you apply the instructions in the summary of the configuration change that is usually the best way to resolve the issue?
            Hide
            alt_jmellor John Mellor added a comment -

            Instead of suggesting that people apply the workaround, having to fix hundreds of thousands of jobs worldwide, how about just reverting the breaking change?  I alone have 3570 jobs to inspect and make a code change to because of this error.  How do I get back to the point of NOT having to make this change, and simply have the github functionality work as expected?

             

            Show
            alt_jmellor John Mellor added a comment - Instead of suggesting that people apply the workaround, having to fix hundreds of thousands of jobs worldwide, how about just reverting the breaking change?  I alone have 3570 jobs to inspect and make a code change to because of this error.  How do I get back to the point of NOT having to make this change, and simply have the github functionality work as expected?  
            Hide
            batmat Baptiste Mathus added a comment -

            John Mellor we understand your frustration, but please be more careful with your phrasing, and be respectful of people time here. This is not a customer support channel.

            Show
            batmat Baptiste Mathus added a comment - John Mellor we understand your frustration, but please be more careful with your phrasing, and be respectful of people time here. This is not a customer support channel.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Version 2.5.4 of Branch API plugin was just released. This version undeprecates the original property (and two others that were deprecated at the same time) and disables the automatic migration going forward (but does not try to reverse it automatically to avoid making things worse).

            Show
            dnusbaum Devin Nusbaum added a comment - Version 2.5.4 of Branch API plugin was just released. This version undeprecates the original property (and two others that were deprecated at the same time) and disables the automatic migration going forward (but does not try to reverse it automatically to avoid making things worse).

              People

              • Assignee:
                dnusbaum Devin Nusbaum
                Reporter:
                alt_jmellor John Mellor
              • Votes:
                8 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: