Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19445

Jobs randomly stuck with "building remotely on slave-name" message

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Hi folks,

      Last week, I started to experience a weird issue with freezing/endless waiting of building jobs on slave machines. It first started with WinXP 32 bit slave, but I'm now experiencing the same problem also with WinXP 64 bit slave.

      The symptoms are like this...
      The job is started by SCM change or predefined time.
      Both slave machines are correctly started (the last line, in their respective logs, says "Slave successfully connected and online").
      Now the problem is, that the last line in console says "Building remotely on slave-name..." and there is just scrolling the progress circle. Nothing else happens. No GIT or SVN activity, no building, no testing, simply nothing and it stays like that until I kill the job manually.

      Sometimes, the problem happens with just one machine, and the other one runs OK. But today it struck both machines.

      Last week, when the problem started, I tried to update the Jenkins and both slaves with the most recent versions and latest plugins (except the "Git Client Plugin", because of a known issue with building history). Few hours all worked great, then the problem reappeared.

      Please see the attached files with logs and thread dumps from both slave machines.

      What struck me is the high number of threads started on both machines, which looks really weird? But I don't understand Jenkins that deep so I can't say for sure if it's normal or not?

      Any idea what to try or how to solve this behavior?

      Have a nice day,
      Pavel

        Attachments

        1. LB3D-MASTER_05_09_13.txt
          53 kB
        2. LB3D-MASTER.txt
          41 kB
        3. LB3D-Wxpp32sp3_05_09_13.txt
          6 kB
        4. LB3D-Wxpp32sp3.txt
          89 kB
        5. LB3D-Wxpp64sp2_05_09_13.txt
          7 kB
        6. LB3D-Wxpp64sp2.txt
          92 kB
        7. remoting-2.47.patch
          2 kB
        8. threadump.txt
          9 kB

          Issue Links

            Activity

            Hide
            bbva BBVA added a comment -

            Dave Taddei, can you confirm you are using the property hudson.remoting.RemoteClassLoader.force to force the load of com.sun.jna.Native. Also that you can see that the class is being loaded via the log of the slave.

            Show
            bbva BBVA added a comment - Dave Taddei, can you confirm you are using the property hudson.remoting.RemoteClassLoader.force to force the load of com.sun.jna.Native. Also that you can see that the class is being loaded via the log of the slave.
            Hide
            drtaddei Dave Taddei added a comment - - edited

            Hi BBVA, it does not appear so but I am unsure where to look. Should this be set in the JAVA_ARGS for the master, slave or both? I was also under the impression that had been added to release 2.47 so didn't set anything.

            Show
            drtaddei Dave Taddei added a comment - - edited Hi BBVA, it does not appear so but I am unsure where to look. Should this be set in the JAVA_ARGS for the master, slave or both? I was also under the impression that had been added to release 2.47 so didn't set anything.
            Hide
            bbva BBVA added a comment -

            Dave Taddei set JVM Options of the slave with: -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native

            Show
            bbva BBVA added a comment - Dave Taddei set JVM Options of the slave with: -Dhudson.remoting.RemoteClassLoader.force=com.sun.jna.Native
            Hide
            gregcovertsmith Greg Smith added a comment -

            We are using dynamic slaves, with the vcenter plugin. The vcenter plugin does not allow the specifications of any JVM options.

            At least, I do not believe it does. Could the above change be made permanent, or set as the default for all slaves some how?

            Show
            gregcovertsmith Greg Smith added a comment - We are using dynamic slaves, with the vcenter plugin. The vcenter plugin does not allow the specifications of any JVM options. At least, I do not believe it does. Could the above change be made permanent, or set as the default for all slaves some how?
            Hide
            jglick Jesse Glick added a comment -

            Greg Smith et al.—see the last comment I left in JENKINS-39179: I wrote a little plugin which might work around JNA deadlocks without needing individual configuration of static agents.

            Show
            jglick Jesse Glick added a comment - Greg Smith et al.—see the last comment I left in  JENKINS-39179 : I wrote a little plugin which might work around JNA deadlocks without needing individual configuration of static agents.

              People

              • Assignee:
                schristou Steven Christou
                Reporter:
                odklizec Pavel Kudrys
              • Votes:
                13 Vote for this issue
                Watchers:
                24 Start watching this issue

                Dates

                • Created:
                  Updated: