Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48130

JNLP1-3 Channel Failure: RejectedExecutionException is not handled properly

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: remoting
    • Labels:
      None
    • Similar Issues:

      Description

      Background: I was analysing JIRA issues related to the NIOHub fatal channel termination causing massive disconnection of agents. It appears that the SingleLaneExecutor is not completely correctly used there...

      TL;DR: A single packet sent to the channel with pending shutdown may cause the termination of all remoting channels in JNLP1, JNLP2, CLI, and CLI2 protocols. JNLP4 does not seem to be affected.

      • When we receive the command in a particular NioTransport, message parts are are being submitted to its SingleLaneExecutor
      • If the executor service rejects the task (e.g. due to the pending shutdown), a runtime RejectedExecutionException is being thrown. E.g. here
      • This exception is not being caught on the NioTransport level and gets proparated to the top level of NioChannelHub#run()
      • NioChannelHub#run() catches the unhandled RuntimeException... and terminates the entire NioHub.

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I have found at least one support bundle where it happens, so I will treat it as a Major defect

            Show
            oleg_nenashev Oleg Nenashev added a comment - I have found at least one support bundle where it happens, so I will treat it as a Major defect
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/remoting/AtmostOneThreadExecutor.java
            src/main/java/hudson/remoting/JarCacheSupport.java
            src/main/java/hudson/remoting/SingleLaneExecutorService.java
            src/main/java/org/jenkinsci/remoting/nio/NioChannelHub.java
            src/main/java/org/jenkinsci/remoting/util/ExecutorServiceUtils.java
            src/test/java/hudson/remoting/PipeTest.java
            http://jenkins-ci.org/commit/remoting/eee8030c07f14b0228bc82392851385594efc2ad
            Log:
            JENKINS-48130 - Improve handling and diagnostics of RejectedExecutionException in the code (#156)

            • Introduce the reloable ExecutorServiceUtils#submitAsync() method
            • Rework existing callers of ExecutorService#submit() to ExecutorServiceUtils, process errors
            • Introduce the FatalRejectedExecutionException and proper handling in ExecutorServiceUtils
            • RejectedExecutionException in SingleLaneService should refer the base exec service
            • Polish the PipeTest implementation for a better diagnosability
            • Fix the double submission issue in the SingleLaneExecutor
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/remoting/AtmostOneThreadExecutor.java src/main/java/hudson/remoting/JarCacheSupport.java src/main/java/hudson/remoting/SingleLaneExecutorService.java src/main/java/org/jenkinsci/remoting/nio/NioChannelHub.java src/main/java/org/jenkinsci/remoting/util/ExecutorServiceUtils.java src/test/java/hudson/remoting/PipeTest.java http://jenkins-ci.org/commit/remoting/eee8030c07f14b0228bc82392851385594efc2ad Log: JENKINS-48130 - Improve handling and diagnostics of RejectedExecutionException in the code (#156) Introduce the reloable ExecutorServiceUtils#submitAsync() method Rework existing callers of ExecutorService#submit() to ExecutorServiceUtils, process errors Introduce the FatalRejectedExecutionException and proper handling in ExecutorServiceUtils RejectedExecutionException in SingleLaneService should refer the base exec service Polish the PipeTest implementation for a better diagnosability Fix the double submission issue in the SingleLaneExecutor
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/main/java/hudson/Launcher.java
            core/src/main/java/hudson/model/Computer.java
            core/src/main/java/hudson/slaves/ChannelPinger.java
            core/src/main/java/hudson/slaves/SlaveComputer.java
            core/src/main/java/jenkins/FilePathFilter.java
            core/src/main/java/jenkins/slaves/StandardOutputSwapper.java
            pom.xml
            test/src/test/java/hudson/bugs/JnlpAccessWithSecuredHudsonTest.java
            test/src/test/java/jenkins/security/Security218CliTest.java
            http://jenkins-ci.org/commit/jenkins/cb3990a4d6094260bea4571e7079fd0e3949047f
            Log:
            Update to Remoting 3.15 and Cleanup issues in Channel#current() usages (#3145)

            Pulls in fixes for: JENKINS-48133, JENKINS-48055, JENKINS-37566, JENKINS-48309, JENKINS-47965, JENKINS-48130, JENKINS-37670, JENKINS-37566, JENKINS-46724

            This change also adds some missing null/closing channel checks in the core.
            In some cases the change prevents spawning threads if the channel is in the invalid state.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/hudson/Launcher.java core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/slaves/ChannelPinger.java core/src/main/java/hudson/slaves/SlaveComputer.java core/src/main/java/jenkins/FilePathFilter.java core/src/main/java/jenkins/slaves/StandardOutputSwapper.java pom.xml test/src/test/java/hudson/bugs/JnlpAccessWithSecuredHudsonTest.java test/src/test/java/jenkins/security/Security218CliTest.java http://jenkins-ci.org/commit/jenkins/cb3990a4d6094260bea4571e7079fd0e3949047f Log: Update to Remoting 3.15 and Cleanup issues in Channel#current() usages (#3145) Pulls in fixes for: JENKINS-48133 , JENKINS-48055 , JENKINS-37566 , JENKINS-48309 , JENKINS-47965 , JENKINS-48130 , JENKINS-37670 , JENKINS-37566 , JENKINS-46724 This change also adds some missing null/closing channel checks in the core. In some cases the change prevents spawning threads if the channel is in the invalid state.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The change has been released in Remoting 3.15 and integrated towards Jenkins 2.98

            Show
            oleg_nenashev Oleg Nenashev added a comment - The change has been released in Remoting 3.15 and integrated towards Jenkins 2.98

              People

              • Assignee:
                oleg_nenashev Oleg Nenashev
                Reporter:
                oleg_nenashev Oleg Nenashev
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: