Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24050

All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Environment:
      Enterprise Linux 5.x master, Windows and Linux slaves of varying releases. Slaves are added and removed reasonably frequently in a way similar to the EC2Plugin (although others have reported with snapshot reverting and even with regular slaves)
    • Similar Issues:

      Description

      We have an issue where we get a CancelledKeyException and 100% of our slaves disconnect and no new new slaves can connect until a restart happens. The issue seems to happen randomly.

      See: https://issues.jenkins-ci.org/browse/JENKINS-22932?focusedCommentId=205983&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-205983#JENKINS-22932 and later for some more context.

      The full error message in the build is:
      FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Failed to abort
      hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Failed to abort
      at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
      at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
      at hudson.remoting.Request.call(Request.java:174)
      at hudson.remoting.Channel.call(Channel.java:739)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
      at com.sun.proxy.$Proxy83.join(Unknown Source)
      at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:956)
      at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
      at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:772)
      at hudson.model.Build$BuildExecution.build(Build.java:199)
      at hudson.model.Build$BuildExecution.doRun(Build.java:160)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:535)
      at hudson.model.Run.execute(Run.java:1732)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:234)
      Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Failed to abort
      at hudson.remoting.Request.abort(Request.java:299)
      at hudson.remoting.Channel.terminate(Channel.java:802)
      at hudson.remoting.Channel$2.terminate(Channel.java:483)
      at hudson.remoting.AbstractByteArrayCommandTransport$1.terminate(AbstractByteArrayCommandTransport.java:72)
      at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:195)
      at org.jenkinsci.remoting.nio.NioChannelHub.abortAll(NioChannelHub.java:618)
      at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:592)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)
      Caused by: java.io.IOException: Failed to abort
      ... 9 more
      Caused by: java.nio.channels.CancelledKeyException
      at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
      at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
      at java.nio.channels.SelectionKey.isReadable(SelectionKey.java:289)
      at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:513)
      ... 6 more

        Attachments

          Issue Links

            Activity

            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            pom.xml
            http://jenkins-ci.org/commit/jenkins/8fc609fe0952b285d5b26a59fd5ff4c29704d33d
            Log:
            [JENKINS-23471 JENKINS-24050]

            Integrated the fix in remoting to Jenkins 1.580.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/8fc609fe0952b285d5b26a59fd5ff4c29704d33d Log: [JENKINS-23471 JENKINS-24050] Integrated the fix in remoting to Jenkins 1.580.
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3685
            [JENKINS-23471 JENKINS-24050] (Revision 8fc609fe0952b285d5b26a59fd5ff4c29704d33d)

            Result = SUCCESS
            kohsuke : 8fc609fe0952b285d5b26a59fd5ff4c29704d33d
            Files :

            • changelog.html
            • pom.xml
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3685 [JENKINS-23471 JENKINS-24050] (Revision 8fc609fe0952b285d5b26a59fd5ff4c29704d33d) Result = SUCCESS kohsuke : 8fc609fe0952b285d5b26a59fd5ff4c29704d33d Files : changelog.html pom.xml
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            pom.xml
            http://jenkins-ci.org/commit/jenkins/9c82fc42eb08b89047c544aaa586291ad1485472
            Log:
            [JENKINS-23471 JENKINS-24050]

            Integrated the fix in remoting to Jenkins 1.580.

            (cherry picked from commit 8fc609fe0952b285d5b26a59fd5ff4c29704d33d)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/9c82fc42eb08b89047c544aaa586291ad1485472 Log: [JENKINS-23471 JENKINS-24050] Integrated the fix in remoting to Jenkins 1.580. (cherry picked from commit 8fc609fe0952b285d5b26a59fd5ff4c29704d33d)
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #4292
            [JENKINS-23471 JENKINS-24050] (Revision 9c82fc42eb08b89047c544aaa586291ad1485472)

            Result = UNSTABLE
            ogondza : 9c82fc42eb08b89047c544aaa586291ad1485472
            Files :

            • pom.xml
            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #4292 [JENKINS-23471 JENKINS-24050] (Revision 9c82fc42eb08b89047c544aaa586291ad1485472) Result = UNSTABLE ogondza : 9c82fc42eb08b89047c544aaa586291ad1485472 Files : pom.xml changelog.html
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            pom.xml
            http://jenkins-ci.org/commit/jenkins/91c5551d4c7682d4adba28fe591fa7772eee62e0
            Log:
            [JENKINS-23471 JENKINS-24050]

            Integrated the fix in remoting to Jenkins 1.580.

            (cherry picked from commit 8fc609fe0952b285d5b26a59fd5ff4c29704d33d)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/91c5551d4c7682d4adba28fe591fa7772eee62e0 Log: [JENKINS-23471 JENKINS-24050] Integrated the fix in remoting to Jenkins 1.580. (cherry picked from commit 8fc609fe0952b285d5b26a59fd5ff4c29704d33d)

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                kbrowder Kevin Browder
              • Votes:
                5 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: