Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56474

agent stuck in infinite loop at connect time on FreeBSD/arm with openjdk 8

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Cannot Reproduce
    • Component/s: remoting
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      Starting an agent (both via ssh and from the command line) on a FreeBSD-12/arm64 agent sends the slave in an infinite loop for over five minutes. If started over ssh, jenkins will try launching a second (and third and so on) copy of the agent to the same result.

      For some reason it did connect once, but I can't reproduce the correct behaviour. I've tried deleting and recreating the host, no changes.

      Attaching stdout, stderr, remoting log and two thread dumps taken a few minutes after startup.

      It would seem that the slave is looping forever in hudson.remoting.ClassFilter$RegExpClassFilter.add - maybe the fact that it's running on an interpreter-mode JVM makes it super inefficient at parsing regexes? Either way, it's been running at 100% CPU for over 15 minutes now and Jenkins still doesn't see the connection as up 

        Attachments

        1. err
          2 kB
        2. out
          0.1 kB
        3. remoting.log.0
          1 kB
        4. tdump.1
          13 kB
        5. tdump.2
          13 kB
        6. tdump.3
          13 kB

          Activity

          kinkie kinkie created issue -
          Hide
          jthompson Jeff Thompson added a comment -

          I'm not too surprised that class loading, filtering, and pattern matching would be slow on an interpreted-only JVM. Those can be some intensive operations. Commonly the agent and the operations it needs to perform can be resource intensive. If you're running into that much resource issues just getting it up and going, it may be difficult for it perform the builds, particularly if other plugins are configured for the job.

          I think your best approach is to figure out how to speed up your processes. If you can get away from an interpreted-only JVM, that would be a good step. Otherwise you may try throwing hardware or other resources at to get it to run faster.

          Show
          jthompson Jeff Thompson added a comment - I'm not too surprised that class loading, filtering, and pattern matching would be slow on an interpreted-only JVM. Those can be some intensive operations. Commonly the agent and the operations it needs to perform can be resource intensive. If you're running into that much resource issues just getting it up and going, it may be difficult for it perform the builds, particularly if other plugins are configured for the job. I think your best approach is to figure out how to speed up your processes. If you can get away from an interpreted-only JVM, that would be a good step. Otherwise you may try throwing hardware or other resources at to get it to run faster.
          Hide
          jthompson Jeff Thompson added a comment -

          Since there has been a lack of response for quite a while on this, I'm going to close this issue. If you have additional information to help move this along, feel free to provide it and re-open as needed.

          Show
          jthompson Jeff Thompson added a comment - Since there has been a lack of response for quite a while on this, I'm going to close this issue. If you have additional information to help move this along, feel free to provide it and re-open as needed.
          jthompson Jeff Thompson made changes -
          Field Original Value New Value
          Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
          Resolution Cannot Reproduce [ 5 ]
          jthompson Jeff Thompson made changes -
          Resolution Cannot Reproduce [ 5 ]
          Status Fixed but Unreleased [ 10203 ] Reopened [ 4 ]
          jthompson Jeff Thompson made changes -
          Status Reopened [ 4 ] Closed [ 6 ]
          Resolution Cannot Reproduce [ 5 ]
          Hide
          garga Renato Botelho added a comment -

          I believe I saw the same issue here.  Jenkins is running on a CentOS 7 box and I'm trying to configure a FreeBSD 13-CURRENT aarch64 as a builder node.  Using the default openjdk package (version 8.232.09) I got the same problem.  I decided then to go ahead and upgrade it to the latest version available on FreeBSD ports tree, which is 8.242.07.1.

          After that, it takes like 5 minutes to start, but after that it works.  It throws these messages on console:

           

          <===[JENKINS REMOTING CAPACITY]===>

          channel started

          <===[JENKINS REMOTING CAPACITY]===>

          channel started

          Remoting version: 3.36.1This is a Unix agentFeb 12, 2020 11:40:39 AM hudson.remoting.UserRequest performWARNING: LinkageError while performing UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@4c81d622java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/freebsd-aarch64/libjnidispatch.so) not found in resource path ([]) at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1032) at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988) at com.sun.jna.Native.<clinit>(Native.java:195) at hudson.util.jna.GNUCLibrary.<clinit>(GNUCLibrary.java:115) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.swap(StandardOutputSwapper.java:60) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:45) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:39) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
          Agent successfully connected and online

           

          Show
          garga Renato Botelho added a comment - I believe I saw the same issue here.  Jenkins is running on a CentOS 7 box and I'm trying to configure a FreeBSD 13-CURRENT aarch64 as a builder node.  Using the default openjdk package (version 8.232.09) I got the same problem.  I decided then to go ahead and upgrade it to the latest version available on FreeBSD ports tree, which is 8.242.07.1. After that, it takes like 5 minutes to start, but after that it works.  It throws these messages on console:   <=== [JENKINS REMOTING CAPACITY] ===> channel started <=== [JENKINS REMOTING CAPACITY] ===> channel started Remoting version: 3.36.1This is a Unix agentFeb 12, 2020 11:40:39 AM hudson.remoting.UserRequest performWARNING: LinkageError while performing UserRequest:jenkins.slaves.StandardOutputSwapper$ChannelSwapper@4c81d622java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/freebsd-aarch64/libjnidispatch.so) not found in resource path ([]) at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1032) at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988) at com.sun.jna.Native.<clinit>(Native.java:195) at hudson.util.jna.GNUCLibrary.<clinit>(GNUCLibrary.java:115) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.swap(StandardOutputSwapper.java:60) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:45) at jenkins.slaves.StandardOutputSwapper$ChannelSwapper.call(StandardOutputSwapper.java:39) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Agent successfully connected and online  

            People

            • Assignee:
              jthompson Jeff Thompson
              Reporter:
              kinkie kinkie
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: