Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58627

jenkins agent getting terminated with version 2.176.2

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: core, remoting
    • Labels:
      None
    • Environment:
      Windows 2012, Java 1.8.0_211, Jenkins 2.176.2
    • Similar Issues:

      Description

      Hi,

      Jenkins agent is geting terminated with new version 2.176.2 as explained below

       

      OS - windows 2012 R2 (both Master and agent)

      Java - 1.8.0_211 (both Mater and slave)

      Jenkins Version - 2.176.2

      Launch Method - Java Web start, JNLP(Launch by connecting it to the Master)

       

      Issue - I have a project in Jenkins which fails mainly because the agent terminates after 'Starting ServiceWrapper in the CLI mode' during execution.Finds a child process conhost.exe and failed due to Killing as fallback.

      This is happening only for a project/ pipeline, where I have python script which pings a internal URL (https://URL/ping), invokes the network emulator tool installed, adjust the network latency and also write to a file/ push to ELK. I have a internet proxy set up using .pac file. 

      It completes all the steps in project, at the end the agent terminates, marking the build as failure 

      Note - We also have another old version of jenkins 2.76 installed where the same project/ pipeline works with out any issue. Now we want to upgrade to latest jenkins we are seeing issue with agent getting terminated. I now have to close the network emulator tool every time and restart the jenkins service to bring it back online. 

      Error as obsreved in Jenkins-slave.wrapper.log:-

      2019-07-24 03:51:59,751 INFO  - Stopping jenkinsslave-e__jenkins
      2019-07-24 03:51:59,751 DEBUG - ProcessKill 6872
      2019-07-24 03:51:59,860 INFO  - Found child process: 6408 Name: conhost.exe
      2019-07-24 03:51:59,892 INFO  - Stopping process 6408
      2019-07-24 03:51:59,907 INFO  - Send SIGINT 6408
      2019-07-24 03:51:59,907 WARN  - SIGINT to 6408 failed - Killing as fallback
      2019-07-24 03:51:59,907 INFO  - Stopping process 6872
      2019-07-24 03:51:59,907 INFO  - Send SIGINT 6872
      2019-07-24 03:51:59,907 WARN  - SIGINT to 6872 failed - Killing as fallback
      2019-07-24 03:51:59,907 INFO  - Finished jenkinsslave-e__jenkins
      2019-07-24 03:51:59,907 DEBUG - Completed. Exit code is 0

       

      Error as obsreved in jenkins-slave.error.log:-
      Jul 24, 2019 3:57:23 AM hudson.remoting.jnlp.Main$CuiListener statusJul
      24, 2019 3:57:23 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Terminated
      Jul 24, 2019 3:57:23 AM hudson.util.ProcessTree getWARNING: Error while determining if vetoers existhudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException
      at hudson.remoting.Request.abort(Request.java:340) at hudson.remoting.Channel.terminate(Channel.java:1040)
      at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
      at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
      at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
      at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
      at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172)
      at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
      at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
      at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1800(BIONetworkLayer.java:48)
      at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:264)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
      at java.lang.Thread.run(Unknown Source) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection to jenkinsmater/IP address:33131
      at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
      at hudson.remoting.Request.call(Request.java:202) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
      at com.sun.proxy.$Proxy6.fetch3(Unknown Source)
      at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:209)
      at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source)
      at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(Unknown Source)
      at java.lang.ClassLoader.defineClass(Unknown Source) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:383) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:307)
      at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source)
      at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(Unknown Source)
      at java.lang.ClassLoader.defineClass(Unknown Source) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:383) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:307)
      at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source)
      at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(Unknown Source)
      at java.lang.ClassLoader.defineClass(Unknown Source) at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:383) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:307)
      at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source)
      at hudson.util.ProcessTree.get(ProcessTree.java:415)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
      at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask.run(Unknown Source) ... 4 moreCaused by: java.nio.channels.ClosedChannelException ... 7 more
      Jul 24, 2019 3:57:23 AM hudson.util.ProcessTree getWARNING: Failed to load winp. Reverting to the defaultjava.lang.NoClassDefFoundError: hudson/util/ProcessTree$WindowsOSProcessException
      at hudson.util.ProcessTree.get(ProcessTree.java:430)
      at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1103) at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1094)
      at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369)
      at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
      at java.lang.Thread.run(Unknown Source)Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree$WindowsOSProcessException at java.net.URLClassLoader.findClass(Unknown Source)
      at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:171) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) ... 12 more
      Jul 24, 2019 3:57:23 AM hudson.remoting.Request$2 runINFO: Failed to send back a reply to the request hudson.remoting.Request$2@626d990c: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@4d834d1e:JNLP4-connect connection to jenkinsmaster/IP Address:33131": channel is already closed

      Jul 24, 2019 3:57:33 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Performing onReconnect operation.

       

      Can someone please help with this issue ?

       

      Thanks,

      Chethan

        Attachments

          Activity

          Hide
          chethan_m Chethan Mahabaleswara added a comment -

          More information about the issue - 

           

          I'm having an issue while using Networl Connection Emulator that it just terminates the jenkins agent process as explained below: 

          Scenario 1 - I have jenkins agent set up in Windows 2012 machine where connection emulator is installed. I run a python script through jenkins pipeline to set the latency and open the connection emulator using python  function "subprocess.Popen(["schtasks","/Run","/TN","StartNetworkEmulator"]) ", which opens emulator from windows task scheduler as a last step. 
          Once the connection emulator is opened, it interferes with the jenkins agent process and terminates the jenkins agent and master communication. 
          I have to close the emulator and restart the jenkins agent windows service to make agent communicate with jenkins master 

          Scenario 2 - There are no jenkins jobs or pipeline running. Jenkins agent is active and connected to Master. 
          When I open and run Connection emulator manually by going C:\Program Files, the jenkins agent and master communication is broken. 

           

          Sometimes I see Linkage error and few times "Error while determining if vetoers exist"

          The error log observed in build and slave are attached. 

          Note - Same job or pipeline works with no issues with old jenkins version 2.76 having remoting version 3.11.  But we want to upgrade to new windows server where I installed latest jenkins version.

           

          Is this a bug with latest version of jenkins ?  Appreciate any help in fixing this issue

           

          Thanks,

          Chethan

          Show
          chethan_m Chethan Mahabaleswara added a comment - More information about the issue -    I'm having an issue while using Networl Connection Emulator that it just terminates the jenkins agent process as explained below:  Scenario 1 - I have jenkins agent set up in Windows 2012 machine where connection emulator is installed. I run a python script through jenkins pipeline to set the latency and open the connection emulator using python  function "subprocess.Popen( ["schtasks","/Run","/TN","StartNetworkEmulator"] ) ", which opens emulator from windows task scheduler as a last step.  Once the connection emulator is opened, it interferes with the jenkins agent process and terminates the jenkins agent and master communication.  I have to close the emulator and restart the jenkins agent windows service to make agent communicate with jenkins master  Scenario 2 - There are no jenkins jobs or pipeline running. Jenkins agent is active and connected to Master.  When I open and run Connection emulator manually by going C:\Program Files, the jenkins agent and master communication is broken.    Sometimes I see Linkage error and few times "Error while determining if vetoers exist" The error log observed in build and slave are attached.  Note - Same job or pipeline works with no issues with old jenkins version 2.76 having remoting version 3.11.  But we want to upgrade to new windows server where I installed latest jenkins version.   Is this a bug with latest version of jenkins ?  Appreciate any help in fixing this issue   Thanks, Chethan
          Hide
          chethan_m Chethan Mahabaleswara added a comment -

          Hello Oleg,

          I see the fix is released as part of jenskins  V 2.181, for Component  remoting [15489]. But when i tried with jenkins version 2.187, observed the same issue/ error

          I had also tried jenkins version 2.176.2, but I see fix is not released in this version. 

           

           

           

          Show
          chethan_m Chethan Mahabaleswara added a comment - Hello Oleg, I see the fix is released as part of jenskins  V 2.181, for Component  remoting [15489] . But when i tried with jenkins version 2.187, observed the same issue/ error I had also tried jenkins version 2.176.2, but I see fix is not released in this version.       

            People

            • Assignee:
              Unassigned
              Reporter:
              chethan_m Chethan Mahabaleswara
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: