Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15682

archive artifacts hangs on ia64 slave due to JNA initialization error

    Details

    • Similar Issues:

      Description

      The archiving artifacts didn't work before jenkins 1.486 due to https://issues.jenkins-ci.org/browse/JENKINS-13202 I believe. The exception was the same.

      But since ~1.486 the build just hangs on artifacts archiving, and only the following can be found in logs:
      ===========
      INFO: test-ia64 #6 main build action completed: SUCCESS
      Oct 31, 2012 2:10:04 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
      SEVERE: I/O error in channel ca-build26
      java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2553)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
      at hudson.remoting.Command.readFrom(Command.java:90)
      at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
      ==================

      The build is marked as successful after manually build abortion, and an exception can be seen in the console.

      The job is simple: 'echo TEST > ${WORKSPACE}/1.txt', and archive 1.txt then.
      thead dump from master and slave is attached

        Attachments

        1. master.txt
          36 kB
        2. slave.txt
          4 kB

          Issue Links

            Activity

            Hide
            nnau Natalia Naumova added a comment -

            Looks like the problem occurs with connection to the slave.
            The archive artifacts hang because the connection between slave and master is terminated soon after slave restart:
            =====
            Failed to load native POSIX impl; falling back on Java impl. Stacktrace follows.
            java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
            at org.jruby.ext.posix.POSIXFactory.loadLibC(POSIXFactory.java:96)
            at org.jruby.ext.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:65)
            at org.jruby.ext.posix.POSIXFactory.getPOSIX(POSIXFactory.java:24)
            at hudson.os.PosixAPI.<clinit>(PosixAPI.java:41)
            at hudson.Util.resolveSymlink(Util.java:1074)
            at hudson.util.DirScanner$Glob.scan(DirScanner.java:121)
            at hudson.FilePath.writeToTar(FilePath.java:1827)
            at hudson.FilePath.access$1000(FilePath.java:166)
            at hudson.FilePath$36.invoke(FilePath.java:1768)
            at hudson.FilePath$36.invoke(FilePath.java:1765)
            at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2236)
            at hudson.remoting.UserRequest.perform(UserRequest.java:118)
            at hudson.remoting.UserRequest.perform(UserRequest.java:48)
            at hudson.remoting.Request$2.run(Request.java:326)
            at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
            at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            Nov 26, 2012 4:20:28 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run
            SEVERE: I/O error in channel channel
            java.io.StreamCorruptedException: invalid type code: 3B
            at java.io.ObjectInputStream.readObject0(Unknown Source)
            at java.io.ObjectInputStream.readObject(Unknown Source)
            at hudson.remoting.Command.readFrom(Command.java:90)
            at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
            at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
            channel stopped
            ERROR: Connection terminated
            ==========

            Show
            nnau Natalia Naumova added a comment - Looks like the problem occurs with connection to the slave. The archive artifacts hang because the connection between slave and master is terminated soon after slave restart: ===== Failed to load native POSIX impl; falling back on Java impl. Stacktrace follows. java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native at org.jruby.ext.posix.POSIXFactory.loadLibC(POSIXFactory.java:96) at org.jruby.ext.posix.POSIXFactory.loadLinuxPOSIX(POSIXFactory.java:65) at org.jruby.ext.posix.POSIXFactory.getPOSIX(POSIXFactory.java:24) at hudson.os.PosixAPI.<clinit>(PosixAPI.java:41) at hudson.Util.resolveSymlink(Util.java:1074) at hudson.util.DirScanner$Glob.scan(DirScanner.java:121) at hudson.FilePath.writeToTar(FilePath.java:1827) at hudson.FilePath.access$1000(FilePath.java:166) at hudson.FilePath$36.invoke(FilePath.java:1768) at hudson.FilePath$36.invoke(FilePath.java:1765) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2236) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Nov 26, 2012 4:20:28 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel channel java.io.StreamCorruptedException: invalid type code: 3B at java.io.ObjectInputStream.readObject0(Unknown Source) at java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.Command.readFrom(Command.java:90) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) channel stopped ERROR: Connection terminated ==========
            Hide
            jglick Jesse Glick added a comment -

            @nnau’s exception would not occur when using Java 7+.

            Show
            jglick Jesse Glick added a comment - @nnau’s exception would not occur when using Java 7+.
            Hide
            brevilo Oliver Bock added a comment -

            The problem is that Java 7+ isn't available for all platforms, e.g. OS X < 10.7.

            Show
            brevilo Oliver Bock added a comment - The problem is that Java 7+ isn't available for all platforms, e.g. OS X < 10.7.
            Hide
            jglick Jesse Glick added a comment -

            @brevilo true but this was filed on Linux, for which Java 7 is available on all platforms I know about.

            Show
            jglick Jesse Glick added a comment - @brevilo true but this was filed on Linux, for which Java 7 is available on all platforms I know about.
            Hide
            brevilo Oliver Bock added a comment - - edited

            @jglick Agreed. I'd love to use this workaround on OS X 10.4 as well. This (and/or the related issue) is a long standing issue that's really precluding us from doing actual CI right now.

            Show
            brevilo Oliver Bock added a comment - - edited @jglick Agreed. I'd love to use this workaround on OS X 10.4 as well. This (and/or the related issue ) is a long standing issue that's really precluding us from doing actual CI right now.
            Hide
            davehunt Dave Hunt added a comment -

            We had a similar issue to this (but with Ubuntu slaves) that was resolved by switching to Oracle's JDK from Open JDK.

            Show
            davehunt Dave Hunt added a comment - We had a similar issue to this (but with Ubuntu slaves) that was resolved by switching to Oracle's JDK from Open JDK.
            Hide
            baranov Valentin Baranov added a comment -

            We have the similar problem on Windows Server.
            Can you tell in which Jenkins version will the fix be available?

            Show
            baranov Valentin Baranov added a comment - We have the similar problem on Windows Server. Can you tell in which Jenkins version will the fix be available?
            Hide
            makr Matthias Kraft added a comment - - edited

            We are observing this issue with Jenkins 1.480.3 on RedHat Enterprise Linux Server and SUSE Linux Enterprise Server running on a Mainframe (s390x) using current IBM JDK 7 SR4.

            A very interesting observation is that on AIX and HP-UX - which also don't have an Oracle JDK - a message appears: "Failed to load native POSIX impl; falling back on Java impl. Unsupported OS." - and everything seems to work out.

            Show
            makr Matthias Kraft added a comment - - edited We are observing this issue with Jenkins 1.480.3 on RedHat Enterprise Linux Server and SUSE Linux Enterprise Server running on a Mainframe (s390x) using current IBM JDK 7 SR4. A very interesting observation is that on AIX and HP-UX - which also don't have an Oracle JDK - a message appears: "Failed to load native POSIX impl; falling back on Java impl. Unsupported OS." - and everything seems to work out.
            Hide
            makr Matthias Kraft added a comment -

            A possible solution might be lined out in these comments: http://jira.codehaus.org/browse/JRUBY-6712

            Show
            makr Matthias Kraft added a comment - A possible solution might be lined out in these comments: http://jira.codehaus.org/browse/JRUBY-6712
            Hide
            jglick Jesse Glick added a comment -

            @makr: newer versions of Jenkins on Java 7+ will not exhibit this error because they use a different code path.

            By the way anyone using proprietary or unusual platforms should not expect that Jenkins core developers will be able to reproduce their problems. Debug it yourself and file a pull request if you want a prompt fix.

            Show
            jglick Jesse Glick added a comment - @makr: newer versions of Jenkins on Java 7+ will not exhibit this error because they use a different code path. By the way anyone using proprietary or unusual platforms should not expect that Jenkins core developers will be able to reproduce their problems. Debug it yourself and file a pull request if you want a prompt fix.
            Hide
            brevilo Oliver Bock added a comment -

            By the way anyone using proprietary or unusual platforms should not expect that Jenkins core developers will be able to reproduce their problems. Debug it yourself and file a pull request if you want a prompt fix.

            @jglick: What if we'd provide access to an affected machine, either via ssh or direct access? Would some core developer be willing to tackle the problem in that case?

            Show
            brevilo Oliver Bock added a comment - By the way anyone using proprietary or unusual platforms should not expect that Jenkins core developers will be able to reproduce their problems. Debug it yourself and file a pull request if you want a prompt fix. @jglick: What if we'd provide access to an affected machine, either via ssh or direct access? Would some core developer be willing to tackle the problem in that case?
            Hide
            jglick Jesse Glick added a comment -

            @brevilo: yes this is a possibility. I cannot personally commit to finding the time for it at the moment, I am afraid.

            Show
            jglick Jesse Glick added a comment - @brevilo: yes this is a possibility. I cannot personally commit to finding the time for it at the moment, I am afraid.
            Hide
            d1morto Donald Morton added a comment -

            Let me see if I understand the flow of logic. It is trying to resolve a symlink, correct? Util.java -> resolveSymlink.

            First, it tries calling 'java.nio.file.Files' in Java7. If that isn't available, it tries to load this library:

            https://kenai.com/projects/jna-posix/sources/mercurial/show/src/org/jruby/ext/posix?rev=59

            If that fails, it tries to load this library:

            https://github.com/jnr/jnr-posix/tree/master/src/main/java/jnr/posix

            All three fail, possibly due to using a non-Oracle Java runtime? The key section in the logs being:

            java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
            at org.jruby.ext.posix.POSIXFactory.loadLibC(POSIXFactory.java:96)

            I suppose that makes this a duplicate of JENKINS-13202?

            Show
            d1morto Donald Morton added a comment - Let me see if I understand the flow of logic. It is trying to resolve a symlink, correct? Util.java -> resolveSymlink. First, it tries calling 'java.nio.file.Files' in Java7. If that isn't available, it tries to load this library: https://kenai.com/projects/jna-posix/sources/mercurial/show/src/org/jruby/ext/posix?rev=59 If that fails, it tries to load this library: https://github.com/jnr/jnr-posix/tree/master/src/main/java/jnr/posix All three fail, possibly due to using a non-Oracle Java runtime? The key section in the logs being: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native at org.jruby.ext.posix.POSIXFactory.loadLibC(POSIXFactory.java:96) I suppose that makes this a duplicate of JENKINS-13202 ?
            Hide
            rodrigc Craig Rodrigues added a comment - - edited

            Jenkins has been updated to use JNA 4.1: JENKINS-24521
            Please try again, since this fixes many native library issues.
            The fix will be in Jenkins 1.586 and higher.

            Show
            rodrigc Craig Rodrigues added a comment - - edited Jenkins has been updated to use JNA 4.1: JENKINS-24521 Please try again, since this fixes many native library issues. The fix will be in Jenkins 1.586 and higher.

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                nnau Natalia Naumova
              • Votes:
                9 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: