Details

    • Similar Issues:

      Description

      Problems

      There are quite a few libraries that core bundles. Some of them, such as remoting, stapler, and so on, are heavily used by plugins and systems we build on top of Jenkins core, such as Blue Ocean.

      This situation creates a class of pain for those plugins; when they need a newer feature or a bug fix in those libraries, they have to wait for multiple months before the change propagates to a version of LTS.

      Solution

      If we can modify Jenkins core such that it allows any of its libraries to be shipped as plugins, then this would eliminate this class of pain altogether.

      Requirements

      • In plugin manifest, BO can specify the minimum core version that bundles one version of stapler, then the version of stapler it requires separately, and the latter is newer than the former
      • Jenkins admin can install a new remoting plugin in order to deploy a security fix without upgrading the whole Jenkins.
      • From users’ & administrators’ perspective, “stapler plugin” and “remoting” plugin looks no different from “git plugin” and “promoted builds plugin.”
      • In a running Jenkins JVM, there’s only one version of stapler or remoting active at any given moment.
      • Spend a reasonable engineering effort to make Jenkins robust in the face of administrators installing wrong set of plugins, old plugins, and other such situations that arise from our weak plugin manager, incompetent users, and so on. (ie, consider manual manipulation of the plugins directory)

      Acceptance criteria

      • The implementation should have enough logging on Success and non-success paths
      • The core components plugin should always require a restart when the replacement plugin is installed
      • Diagnostics things like /about or support core plugin should properly report the library versions (ideally with override references)
      • Reference implementation of Remoting plugin. Use-cases like Remoting.jar upgrade should work natively
      • Reference implementation of Stapler plugin, which should be used in another reference plugin (e.g. BlueOcean which was the original requestor)
      • Jenkins ATH, JTH and PCT should support the new mode
      • Enough test automation for the feature (Quality gate is TBD)
      • Upgrade use-cases are well tested (and automated if possible)
      • When the admin installs or updates a plugin which requires an updated core component, that core component should be included in the update.
      • When a plugin using jenkins-test-harness has a newer version of a pluggable core component in the test scope, the JenkinsRule tests should include the update pluggable core component when Jenkins starts.
      • If an updated core component lacks a classpath entry (class or resource) that was present in the original, that entry should no longer be present in the classpath.
      • In the particular case of remoting, there is slave.jar available for download. We would expect this download to be updated by a new version of the remoting pluggable core component.
      • Jenkins administrators are able to see that some core components have been upgraded or overridden in the UI.
      • If a plugin installed by the SetupWizard requires an updated core component, the SetupWizard should install this pluggable core component just as if you had installed it manually from the Available tab.
      • The administrator will be able to update core components using the 'install-plugin' CLI command.
      • A release of a pluggable core component should not be immediately available for installation in the update center.

      Follow-ups:

      • Support Core plugin should be able to report overrides

      Limitations/Clarifications

      • Updated pluggable core components are not shown in the Updates tab until the user installs that pluggable component for the first time.

      Engineering Design

      Today, jenkins.war puts core and all its dependencies in WEB-INF/lib and let servlet container classloader load them all. This results in a fully populated classloader by the time Jenkins core boots up, and it makes it impossible for Jenkins to programmatically control versions of libraries to use.
      We can change this if Jenkins creates its own classloader during bootstraps, and uses its own logic to decide what jar files to put in there; mix and match from jenkins.war and $JENKINS_HOME.
      More specifically, we move everything from WEB-INF/lib to WEB-INF/jars, so that core and its dependencies are not loaded by servlet container automatically. Then place small bootstrap code in WEB-INF/lib instead, which creates a classloader that loads core, then passes on execution to hudson.WebAppMain.
      In order to allow newer versions of components to be delivered as plugins, this bootstrap process will also pick up some files from $JENKINS_HOME/plugins. The likes of remoting/stapler plugins need to have a flag in them, so that the bootstrap process can put them in the main classloader, while others like git & promoted builds are treated the same as today.
      Bootstrap code needs to be small and with as little dependencies as possible, so core component plugins need to be different enough from regular plugins. At the same time, they need to look similar enough so that our update center and plugin manager can treat them equally. A boolean entry in MANIFEST.MF would be one example that meets those two goals.
      For a similar reason, Care has to be taken to avoid breaking the core catastrophically, in the face of inevitable user mistake. One such measure could be to compare versions bundled in WEB-INF/jars against versions supplied as plugins, and if the former is newer, use that and ignore the latter.
      No additional GUI is needed, as the existing plugin manager should be sufficient.

        Attachments

          Issue Links

            Activity

            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            docs/SUT-VERSIONS.md
            src/main/java/org/jenkinsci/test/acceptance/update_center/LocalOverrideUpdateCenterMetadataDecoratorImpl.java
            src/main/java/org/jenkinsci/test/acceptance/update_center/UpdateCenterMetadataDecorator.java
            src/test/java/plugins/RemotingPluginTest.java
            http://jenkins-ci.org/commit/acceptance-test-harness/71e56fa79f894d917341d96915a10ecc02e0f819
            Log:
            Revert "JENKINS-41196 demonstrate that core component override plugin can work in ATH"

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: docs/SUT-VERSIONS.md src/main/java/org/jenkinsci/test/acceptance/update_center/LocalOverrideUpdateCenterMetadataDecoratorImpl.java src/main/java/org/jenkinsci/test/acceptance/update_center/UpdateCenterMetadataDecorator.java src/test/java/plugins/RemotingPluginTest.java http://jenkins-ci.org/commit/acceptance-test-harness/71e56fa79f894d917341d96915a10ecc02e0f819 Log: Revert " JENKINS-41196 demonstrate that core component override plugin can work in ATH"
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            src/test/java/plugins/RemotingPluginTest.java
            http://jenkins-ci.org/commit/acceptance-test-harness/2ef2ee88b27764f09258eb9227bf0e9497e6ca49
            Log:
            Revert "JENKINS-41196 demonstrate the capability of loading core component jar"

            This reverts commit 673dc7b6946eebc923ae276f3f843ffde93966fe.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: src/test/java/plugins/RemotingPluginTest.java http://jenkins-ci.org/commit/acceptance-test-harness/2ef2ee88b27764f09258eb9227bf0e9497e6ca49 Log: Revert " JENKINS-41196 demonstrate the capability of loading core component jar" This reverts commit 673dc7b6946eebc923ae276f3f843ffde93966fe.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            src/test/java/plugins/RemotingPluginTest.java
            http://jenkins-ci.org/commit/acceptance-test-harness/8dbf55820c164e46ba6b900bba010e92fccfd3c8
            Log:
            JENKINS-41196 demonstrate the capability of loading core component jar

            Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/2ef2ee88b277^...8dbf55820c16

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: src/test/java/plugins/RemotingPluginTest.java http://jenkins-ci.org/commit/acceptance-test-harness/8dbf55820c164e46ba6b900bba010e92fccfd3c8 Log: JENKINS-41196 demonstrate the capability of loading core component jar Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/2ef2ee88b277 ^...8dbf55820c16
            Hide
            batmat Baptiste Mathus added a comment -

            Closing as per Jesse Glick recommendation in https://github.com/jenkinsci/jenkins/pull/2754#issuecomment-453667192.

            We'll reopen when/if this works gets rebooted. Most of the related PRs now also need anyway to get conflict resolution and didn't receive changes since 1+ year.

            Show
            batmat Baptiste Mathus added a comment - Closing as per Jesse Glick recommendation in https://github.com/jenkinsci/jenkins/pull/2754#issuecomment-453667192 . We'll reopen when/if this works gets rebooted. Most of the related PRs now also need anyway to get conflict resolution and didn't receive changes since 1+ year.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I believe this story is largely replaced by Custom WAR Packager anyway (https://github.com/jenkinsci/custom-war-packager). This tool allows quickly building Custom WARs so that users can pick up Remoting/Stapler updates for their own baselines if they do not want to upgrade due to various reasons. Not a 100% solution, but it definitely helps in this use-case

            Show
            oleg_nenashev Oleg Nenashev added a comment - I believe this story is largely replaced by Custom WAR Packager anyway ( https://github.com/jenkinsci/custom-war-packager ). This tool allows quickly building Custom WARs so that users can pick up Remoting/Stapler updates for their own baselines if they do not want to upgrade due to various reasons. Not a 100% solution, but it definitely helps in this use-case

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                kohsuke Kohsuke Kawaguchi
              • Votes:
                2 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: