Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24050

All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Environment:
      Enterprise Linux 5.x master, Windows and Linux slaves of varying releases. Slaves are added and removed reasonably frequently in a way similar to the EC2Plugin (although others have reported with snapshot reverting and even with regular slaves)
    • Similar Issues:

      Description

      We have an issue where we get a CancelledKeyException and 100% of our slaves disconnect and no new new slaves can connect until a restart happens. The issue seems to happen randomly.

      See: https://issues.jenkins-ci.org/browse/JENKINS-22932?focusedCommentId=205983&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-205983#JENKINS-22932 and later for some more context.

      The full error message in the build is:
      FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Failed to abort
      hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Failed to abort
      at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
      at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
      at hudson.remoting.Request.call(Request.java:174)
      at hudson.remoting.Channel.call(Channel.java:739)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
      at com.sun.proxy.$Proxy83.join(Unknown Source)
      at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:956)
      at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
      at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:772)
      at hudson.model.Build$BuildExecution.build(Build.java:199)
      at hudson.model.Build$BuildExecution.doRun(Build.java:160)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:535)
      at hudson.model.Run.execute(Run.java:1732)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:234)
      Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Failed to abort
      at hudson.remoting.Request.abort(Request.java:299)
      at hudson.remoting.Channel.terminate(Channel.java:802)
      at hudson.remoting.Channel$2.terminate(Channel.java:483)
      at hudson.remoting.AbstractByteArrayCommandTransport$1.terminate(AbstractByteArrayCommandTransport.java:72)
      at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:195)
      at org.jenkinsci.remoting.nio.NioChannelHub.abortAll(NioChannelHub.java:618)
      at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:592)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)
      Caused by: java.io.IOException: Failed to abort
      ... 9 more
      Caused by: java.nio.channels.CancelledKeyException
      at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
      at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
      at java.nio.channels.SelectionKey.isReadable(SelectionKey.java:289)
      at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:513)
      ... 6 more

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                kbrowder Kevin Browder
              • Votes:
                5 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: