XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: unity3d-plugin
    • Labels:
      None
    • Environment:
      * Mac OS X
      * Slave node running on same machine as Jenkins (connected via SSH to localhost, same 'jenkins' user running everything)
      * Unity Cache server (running on same machine as Jenkins and Unity slave node)
    • Similar Issues:

      Description

      We get a pipe broken exception in the Jenkins Unity plugin about 50% of the time. The workspace is cleaned before every build, and compiled assets are fetched from a Unity Cache server running on the same machine.

      The error happens ALWAYS at the exact same step:
      [...]
      Refresh: trashing asset 2197dcffa7f1e4295a0f8e83fed9f5bf
      Refresh: trashing asset bca6eeff7c4924a429da9a1b3450b036
      Refresh: trashing asset 6b4a9fff255e844c999a4bcd6105d894
      Refresh: trashing asset d63cefffccda34624843a84caddbb852
      Failure on remote
      java.io.IOException: Pipe broken
      at java.io.PipedInputStream.read(PipedInputStream.java:322)
      at java.io.PipedInputStream.read(PipedInputStream.java:378)
      at java.io.InputStream.read(InputStream.java:101)
      at org.jenkinsci.plugins.unity3d.io.StreamCopyThread.run(StreamCopyThread.java:64)

      When it succeeds, the next line is ALWAYS
      Updating Assets - GUID: 00000000000000001000000000000000...
      done. [Time: 18.743246 ms]

      So IF the build fails, then it is ALWAYS right after the last asset being trashed. If it makes it to the next line (updating asset with GUID 00001000), then the build will ALWAYS succeed.

      I think that the frequency of the error is dependent on how large the project is, as we didn't get the error nearly as much in the beginning phases of the project. Now that the project is large we get the pipe broken error about 50% of the time. This is of course conjecture and the frequency of the error could be caused by something else entirely.

      Also, errors tend to happen multiple times in a row. Conversely, once it succeeds it tends to succeed for several consecutive builds.

      This is of course very much a showstopper for us, so I'd be extremely grateful if someone could have a look at this issue.

      Thanks!

      • Tommi Kiviniemi
        VP Technology, Co-founder
        Seriously Digital Entertainment Ltd.

        Attachments

          Issue Links

            Activity

            Hide
            lacostej lacostej added a comment -

            Thanks Charles for the logs.

            I don't know what's happening yet but this should help us a bit. I am surprised to not see more communication.

            What can I see:

            • Pipe.Chunks are seen up to May 21, 2015 9:26:15
            • followed by lots of hudson.remoting.UnexportCommand
            • a PingThread kicks in at May 21, 2015 9:27:40 AM
            • a PingThread kicks in at May 21, 2015 9:28:11 AM
            • nothing during 2 min
            • a Pipe.Chunk is sent
            • a Dead is received instead of an Ack

            Here's a patch to apply to your remoting.jar https://github.com/lacostej/remoting/commit/02969e1271443afc933dcb8e0d472c1dfc9e8856 that could help knowing why the Death. Maybe worth a shot ?

            This also should help a bit: https://wiki.jenkins-ci.org/display/JENKINS/Remoting+issue although Charles said he couldn't find any specific info in his slaves logs.

            Show
            lacostej lacostej added a comment - Thanks Charles for the logs. I don't know what's happening yet but this should help us a bit. I am surprised to not see more communication. What can I see: Pipe.Chunks are seen up to May 21, 2015 9:26:15 followed by lots of hudson.remoting.UnexportCommand a PingThread kicks in at May 21, 2015 9:27:40 AM a PingThread kicks in at May 21, 2015 9:28:11 AM nothing during 2 min a Pipe.Chunk is sent a Dead is received instead of an Ack Here's a patch to apply to your remoting.jar https://github.com/lacostej/remoting/commit/02969e1271443afc933dcb8e0d472c1dfc9e8856 that could help knowing why the Death. Maybe worth a shot ? This also should help a bit: https://wiki.jenkins-ci.org/display/JENKINS/Remoting+issue although Charles said he couldn't find any specific info in his slaves logs.
            Hide
            lacostej lacostej added a comment - - edited

            Good news, I managed to find at least one reproducible scenario in which this error happens. It required a distributed built setup. More info later!

            Show
            lacostej lacostej added a comment - - edited Good news, I managed to find at least one reproducible scenario in which this error happens. It required a distributed built setup. More info later!
            Hide
            lacostej lacostej added a comment -

            I know the root cause and I have a fix that seems to pass my tests.

            The root cause is when I wrote the plugin, I created my own piping mechanism because I was having issues with the Jenkins remoting one. When doing so, I went with using the remoting and java.io.Piped*Stream.

            See https://github.com/jenkinsci/unity3d-plugin/blob/b4d78b6fee8752c02eacb35c1b0416c55d9d76f5/src/main/java/org/jenkinsci/plugins/unity3d/io/Pipe.java#L35

            While the java.io.Piped*Stream classes work fine most of the time, they have some issues, including thread unfriendliness. See http://www.live-graph.org/javadoc/complete/org/LiveGraph/dataFile/read/PipedInputStream.html for some description.

            As stated in the Javadoc for the java.io.PipedInputStream: A pipe is said to be broken if a thread that was providing data bytes to the connected piped output stream is no longer alive.

            In the case of jenkins used in a distributed scenario, those threads are pooled (search threadPoolForRemoting in jenkins core). In particular, if the remote output isn't producing output for a while, this pooling seems to have more chance of happening. In this case, the writeSide thread is recycled/replaced. The next call to read detects the "pipe is broken".

            So we cannot use the java.io.Piped*Stream in conjunction with Jenkins.

            Now it seems that I can use the jenkins remoting FastPiped*Stream classes and everything work fine. So easy...

            Show
            lacostej lacostej added a comment - I know the root cause and I have a fix that seems to pass my tests. The root cause is when I wrote the plugin, I created my own piping mechanism because I was having issues with the Jenkins remoting one. When doing so, I went with using the remoting and java.io.Piped*Stream. See https://github.com/jenkinsci/unity3d-plugin/blob/b4d78b6fee8752c02eacb35c1b0416c55d9d76f5/src/main/java/org/jenkinsci/plugins/unity3d/io/Pipe.java#L35 While the java.io.Piped*Stream classes work fine most of the time, they have some issues, including thread unfriendliness. See http://www.live-graph.org/javadoc/complete/org/LiveGraph/dataFile/read/PipedInputStream.html for some description. As stated in the Javadoc for the java.io.PipedInputStream: A pipe is said to be broken if a thread that was providing data bytes to the connected piped output stream is no longer alive. In the case of jenkins used in a distributed scenario, those threads are pooled (search threadPoolForRemoting in jenkins core). In particular, if the remote output isn't producing output for a while, this pooling seems to have more chance of happening. In this case, the writeSide thread is recycled/replaced. The next call to read detects the "pipe is broken". So we cannot use the java.io.Piped*Stream in conjunction with Jenkins. Now it seems that I can use the jenkins remoting FastPiped*Stream classes and everything work fine. So easy...
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jerome Lacoste
            Path:
            src/main/java/org/jenkinsci/plugins/unity3d/io/Pipe.java
            http://jenkins-ci.org/commit/unity3d-plugin/dff4212ee1f3c9ec1ed54dda0f2788d0126d2fc1
            Log:
            [FIXED JENKINS-23958] use jenkins remoting piping streams for piping. java.io.Piped*Stream are not threads friendly and cause 'Pipe is broken' issue when jenkins pool the writing threads

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jerome Lacoste Path: src/main/java/org/jenkinsci/plugins/unity3d/io/Pipe.java http://jenkins-ci.org/commit/unity3d-plugin/dff4212ee1f3c9ec1ed54dda0f2788d0126d2fc1 Log: [FIXED JENKINS-23958] use jenkins remoting piping streams for piping. java.io.Piped*Stream are not threads friendly and cause 'Pipe is broken' issue when jenkins pool the writing threads
            Hide
            lacostej lacostej added a comment -

            Fixed in v0.8

            Show
            lacostej lacostej added a comment - Fixed in v0.8

              People

              • Assignee:
                lacostej lacostej
                Reporter:
                tommigun Tommi Kiviniemi
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: