Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41121

GitHub Branch Source upgrade can cause a lot of rebuilds

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The gist of this is that due to some encoding change made in the SCM API 2.0 release, conventional URIs

      This was originally noticed by the Blue Ocean team in this pull request discussing integrating SCM API 2.0 changes. See also JENKINS-40299.

      The blog post discusses some of the data migrations which are supposed to take place, however the URI changing issue results in two bad behaviors:

      * URIs change for Multibranch-based Pipelines which use "special characters" in their names, such as improvement/JENKINS-41047 would previously be located at: https://ci.blueocean.io/job/blueocean/job/improvement%2FJENKINS-41047/. After these changes it would be found at https://ci.blueocean.io/job/blueocean/job/improvement-JENKINS-41047.d61e72/
      * Post-upgrade, rebooting the Jenkins installation results in *every* Multibranch Pipeline being retriggered. Such as this Pipeline despite no new changes being made.


      Because the upgrade to SCM API 2.0 is a one-direction-only change, I propose we blacklist the updated versions of the plugins referenced in this post before too many people upgrade a non-reversible upgrade.

      Incorrect analysis with lots of FUD.

      The renaming of jobs has nothing to do here.

      The issue here is that GitHub Branch Source 1.x passes a null id to the constructor of it's SCMSource: https://github.com/jenkinsci/github-branch-source-plugin/blob/1.10-stable/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMNavigator.java#L312

      This means that every time you perform a scan of the organization folder, all the SCMSource instances get a new ID.

      When the branch changes SCMSource instance (i.e. the ID is different) then the Branch API is supposed to trigger a rebuild (because the branch is from a different source, never mind that it may be the same revision, we have no way of knowing)

      In 2.x we have a consistent ID: https://github.com/jenkinsci/github-branch-source-plugin/blob/master/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMNavigator.java#L446

      The issue here is that GitHub Branch Source did not correctly ensure that the SCMSource from a Organization Folder had a consistent ID.

      Thus, every time you do a scan of an organization folder, with the old logic it should have been re-triggering a build... It wasn't because of other bugs... this leads to things like "I cannot re-trigger PRs", etc

      In fixing the problems in GitHub Branch Source, we get to a point where everything is behaving correctly, but the data for the previous branch origins is not current and thus the PRs have not been built from their current source.

        Attachments

          Issue Links

            Activity

            Hide
            kingdonb Kingdon Barrett added a comment -

            I can tell you that I did not download any plugins by hand, and only used the update center. And still wound up reading this issue.

            Echo everything that Morgan Goose said. I downgraded some plugins because that appeared to be the recommended path, and I only narrowly avoided deleting my settings after reading the same warnings that you saw about old data. Maybe I caught the upgrade moments before you blacklisted the 2.0.1 update? Not sure. The timing is about right though.

            Thanks for reposting the ZIP link. I see that you're planning a Monday release, I will wait for that, thanks for providing a clear(ish) path in the issue documentation. FWIW none of the plugins that complained at me about bad dependency versions, actually appeared to be malfunctioning, but I am not using Blue Ocean or advanced GitHub API plugins yet.

            Show
            kingdonb Kingdon Barrett added a comment - I can tell you that I did not download any plugins by hand, and only used the update center. And still wound up reading this issue. Echo everything that Morgan Goose said. I downgraded some plugins because that appeared to be the recommended path, and I only narrowly avoided deleting my settings after reading the same warnings that you saw about old data. Maybe I caught the upgrade moments before you blacklisted the 2.0.1 update? Not sure. The timing is about right though. Thanks for reposting the ZIP link. I see that you're planning a Monday release, I will wait for that, thanks for providing a clear(ish) path in the issue documentation. FWIW none of the plugins that complained at me about bad dependency versions, actually appeared to be malfunctioning, but I am not using Blue Ocean or advanced GitHub API plugins yet.
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Kingdon Barrett you have a different case from Morgan.

            Morgan said:

            I've already grabbed the two hpi files, and installed manually:

            Whereas you downloaded the plugins via the recommended path (and were just bit by the issues described in JENKINS-41661 which is where I am complaining that the UC does not do enough validation. The UC method does at least try to upgrade all the required dependencies... it should grey out plugins that cannot be installed IMHO)

            Kingdon, I recommend reading https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-41661+Post-Mortem for more details on the JENKINS-41661 issue that affected you.

            The plugins will all be released on Monday.

            Sorry for any inconvenience caused.

            Show
            stephenconnolly Stephen Connolly added a comment - Kingdon Barrett you have a different case from Morgan. Morgan said: I've already grabbed the two hpi files, and installed manually: github-organization-folder-1.6.hpi scm-api-2.0.2.hpi from https://repo.jenkins-ci.org/public/org/jenkins-ci/plugins/ Whereas you downloaded the plugins via the recommended path (and were just bit by the issues described in JENKINS-41661 which is where I am complaining that the UC does not do enough validation. The UC method does at least try to upgrade all the required dependencies... it should grey out plugins that cannot be installed IMHO) Kingdon, I recommend reading https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-41661+Post-Mortem for more details on the JENKINS-41661 issue that affected you. The plugins will all be released on Monday. Sorry for any inconvenience caused.
            Hide
            kingdonb Kingdon Barrett added a comment -

            Yeah, sounds like you're on top of things. Lesson learned: full backups immediately before every upgrade!

            No trouble at all, thanks for your contribution!

            Show
            kingdonb Kingdon Barrett added a comment - Yeah, sounds like you're on top of things. Lesson learned: full backups immediately before every upgrade! No trouble at all, thanks for your contribution!
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Kingdon Barrett

            This was a particularly troublesome set of upgrades. Due to the mistakes made in GHBS and BBBS (because of lack of documentation) we had to break data compatibility in a particularly complicated way that prevented the normal evolution mode in Jenkins.

            Hopefully we will not be doing that again any time in the near future

            Show
            stephenconnolly Stephen Connolly added a comment - Kingdon Barrett This was a particularly troublesome set of upgrades. Due to the mistakes made in GHBS and BBBS (because of lack of documentation) we had to break data compatibility in a particularly complicated way that prevented the normal evolution mode in Jenkins. Hopefully we will not be doing that again any time in the near future
            Hide
            morgan_goose Morgan Goose added a comment -

            Stephen Connolly I'd like to clarify, that I only installed those two plugins manually after multiple plugins failed to load saying that these versions were required. All of those complaining plugins were shown as having updates available and were updated in the suggested way (via the plugin update center web UI).

            Show
            morgan_goose Morgan Goose added a comment - Stephen Connolly I'd like to clarify, that I only installed those two plugins manually after multiple plugins failed to load saying that these versions were required. All of those complaining plugins were shown as having updates available and were updated in the suggested way (via the plugin update center web UI).

              People

              • Assignee:
                stephenconnolly Stephen Connolly
                Reporter:
                rtyler R. Tyler Croy
              • Votes:
                4 Vote for this issue
                Watchers:
                31 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: