We have an issue where we get a CancelledKeyException and 100% of our slaves disconnect and no new new slaves can connect until a restart happens. The issue seems to happen randomly.

See: https://issues.jenkins-ci.org/browse/JENKINS-22932?focusedCommentId=205983&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-205983#JENKINS-22932 and later for some more context.

The full error message in the build is:
FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Failed to abort
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Failed to abort
at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
at hudson.remoting.Request.call(Request.java:174)
at hudson.remoting.Channel.call(Channel.java:739)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
at com.sun.proxy.$Proxy83.join(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:956)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:137)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:772)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:535)
at hudson.model.Run.execute(Run.java:1732)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:234)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Failed to abort
at hudson.remoting.Request.abort(Request.java:299)
at hudson.remoting.Channel.terminate(Channel.java:802)
at hudson.remoting.Channel$2.terminate(Channel.java:483)
at hudson.remoting.AbstractByteArrayCommandTransport$1.terminate(AbstractByteArrayCommandTransport.java:72)
at org.jenkinsci.remoting.nio.NioChannelHub$NioTransport.abort(NioChannelHub.java:195)
at org.jenkinsci.remoting.nio.NioChannelHub.abortAll(NioChannelHub.java:618)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:592)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Failed to abort
... 9 more
Caused by: java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
at java.nio.channels.SelectionKey.isReadable(SelectionKey.java:289)
at org.jenkinsci.remoting.nio.NioChannelHub.run(NioChannelHub.java:513)
... 6 more