Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60280

Jobs failing sporadically with javax.crypto.BadPaddingException: Invalid TLS padding data

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Component/s: core, swarm-plugin
    • Labels:
      None
    • Environment:
    • Similar Issues:

      Description

      We are sporadically seeing jobs failing with this callstack shown in the log:

      javax.crypto.BadPaddingException: Invalid TLS padding data
      	at sun.security.ssl.CipherBox.removePadding(CipherBox.java:802)
      	at sun.security.ssl.CipherBox.decrypt(CipherBox.java:574)
      	at sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:200)
      	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:963)
      Caused: javax.net.ssl.SSLException: Invalid TLS padding data
      	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
      	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709)
      	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:970)
      	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896)
      	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766)
      	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:347)
      	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
      	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)
      	at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
      	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:160)
      	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from <server name>/<server ip>:54103 failed. The channel is closing down or has closed down
      	at hudson.remoting.Channel.call(Channel.java:950)
      	at hudson.FilePath.act(FilePath.java:1068)
      	at hudson.FilePath.act(FilePath.java:1057)
      	at hudson.FilePath.mkdirs(FilePath.java:1242)
      	at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:181)
      	at org.jenkinsci.plugins.durabletask.WindowsBatchScript$BatchController.<init>(WindowsBatchScript.java:93)
      	at org.jenkinsci.plugins.durabletask.WindowsBatchScript$BatchController.<init>(WindowsBatchScript.java:91)
      	at org.jenkinsci.plugins.durabletask.WindowsBatchScript.doLaunch(WindowsBatchScript.java:60)
      	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launchWithCookie(FileMonitoringTask.java:106)
      	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:99)
      	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:305)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:268)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:176)
      	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
      	at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1278)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1172)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:810)
      	at groovy.lang.GroovyObjectSupport.invokeMethod(GroovyObjectSupport.java:46)
      	at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1278)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1172)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
      	at buildscript.call(C:\Jenkins_master\workspace\jobs\NightlyBuild_SDK_QNX\builds\365\libs\pipeline-common\vars\buildscript.groovy:62)
      	at ___cps.transform___(Native Method)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor291.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
      	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
      	at sun.reflect.GeneratedMethodAccessor374.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor291.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor291.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      There are days when this is seen several times a day, and others when it is not seen at all. It happens on different Windows10 clients, but so far I have not seen it on any of our Linux clients. When restarting the failing job (with the same parameters) it usually runs fine.

        Attachments

          Activity

          Hide
          basil Basil Crow added a comment -

          Hey Stephan Vogt, I'm not sure what the root cause of this problem is. May I suggest a few things:

          • The stack trace you posted above is from the Jenkins master (server) side. It would be interesting to see if there are any errors on the Swarm (client) side. To do this, try starting the Swarm client with more verbose logging. This page provides an example of a verbose logging.properties file that logs as much as possible. Perhaps the additional logs will shed more light into what the Swarm client was doing at the time of the failure.
          • Are you running the Swarm clients with the same Java version as the master? The Java ecosystem these days is growing quite diverse, and even minor versions (e.g. 8u151 vs 8u202) can have subtle differences leading to serialization incompatibilities. Another potential source of incompatibility is using e.g. the Oracle JDK vs OpenJDK/AdoptOpenJDK or other open source Java runtimes.
          Show
          basil Basil Crow added a comment - Hey Stephan Vogt , I'm not sure what the root cause of this problem is. May I suggest a few things: The stack trace you posted above is from the Jenkins master (server) side. It would be interesting to see if there are any errors on the Swarm (client) side. To do this, try starting the Swarm client with more verbose logging. This page provides an example of a verbose logging.properties file that logs as much as possible. Perhaps the additional logs will shed more light into what the Swarm client was doing at the time of the failure. Are you running the Swarm clients with the same Java version as the master? The Java ecosystem these days is growing quite diverse, and even minor versions (e.g. 8u151 vs 8u202) can have subtle differences leading to serialization incompatibilities. Another potential source of incompatibility is using e.g. the Oracle JDK vs OpenJDK/AdoptOpenJDK or other open source Java runtimes.
          Hide
          stephanvogt Stephan Vogt added a comment -

          Hi, thanks for the suggestions. Especially the one about activating additional logging might come in handy.
          As for the Java versions: Both the master and the client were using the exact same Java version - every time I tried a new one I installed it on all machines at the same time.

          In the meantime however I have upgraded our environment to Windows 10, OpenJDK 11.0.6_10, and Jenkins master 2.204.1. So far I have not seen this issue again in several weeks of usage. I am not sure what part exactly fixed this issue for me, but with the new configuration it is no longer seen, so I think we can close this ticket.

          Show
          stephanvogt Stephan Vogt added a comment - Hi, thanks for the suggestions. Especially the one about activating additional logging might come in handy. As for the Java versions: Both the master and the client were using the exact same Java version - every time I tried a new one I installed it on all machines at the same time. In the meantime however I have upgraded our environment to Windows 10, OpenJDK 11.0.6_10, and Jenkins master 2.204.1. So far I have not seen this issue again in several weeks of usage. I am not sure what part exactly fixed this issue for me, but with the new configuration it is no longer seen, so I think we can close this ticket.
          Hide
          basil Basil Crow added a comment -

          I am not sure what part exactly fixed this issue for me, but with the new configuration it is no longer seen, so I think we can close this ticket.

          Done.

          Show
          basil Basil Crow added a comment - I am not sure what part exactly fixed this issue for me, but with the new configuration it is no longer seen, so I think we can close this ticket. Done.

            People

            • Assignee:
              Unassigned
              Reporter:
              stephanvogt Stephan Vogt
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: