Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-7871

"java.io.IOException: Bad file descriptor" when file copied from slave

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      Linux, 1.381, all plugins updated to latest and greatest.

      Description

      Just updated from 1.377 to 1.381.

      Large reporting jobs, where Cobertura output is large (many files around 1MB, totaling 18MB), will have issues while copying "coverage.xml" files from slave machine to main Hudson server.

      No error in logs on Slave machine, but we find the following error on the Hudson Server:


      Oct 21, 2010 4:40:03 AM hudson.remoting.ProxyOutputStream$Chunk$1 run
      WARNING: Failed to write to stream
      java.io.IOException: Bad file descriptor
      at java.io.FileOutputStream.writeBytes(Native Method)
      at java.io.FileOutputStream.write(FileOutputStream.java:247)
      at hudson.remoting.ProxyOutputStream$Chunk$1.run(ProxyOutputStream.java:185)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)


      The build actually fails then with following stack trace:


      Publishing Cobertura coverage report...
      Publishing Cobertura coverage results...
      FATAL: Unable to parse /home/hudson/hudson/server_installation/jobs/projectname/builds/2010-10-21_03-00-32/coverage35.xml
      hudson.util.IOException2: Cannot parse coverage results
      at hudson.plugins.cobertura.CoberturaCoverageParser.parse(CoberturaCoverageParser.java:89)
      at hudson.plugins.cobertura.CoberturaCoverageParser.parse(CoberturaCoverageParser.java:52)
      at hudson.plugins.cobertura.CoberturaPublisher.perform(CoberturaPublisher.java:262)
      at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:580)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:558)
      at hudson.model.Build$RunnerImpl.post2(Build.java:157)
      at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
      at hudson.model.Run.run(Run.java:1303)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:139)
      Caused by: org.xml.sax.SAXParseException: XML document structures must start and end within the same entity.
      at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:195)
      at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:174)
      at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:388)
      at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1411)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.endEntity(XMLDocumentFragmentScannerImpl.java:902)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.endEntity(XMLDocumentScannerImpl.java:605)
      at com.sun.org.apache.xerces.internal.impl.XMLEntityManager.endEntity(XMLEntityManager.java:1393)
      at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1763)
      at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipSpaces(XMLEntityScanner.java:1492)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanAttribute(XMLDocumentFragmentScannerImpl.java:1520)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1316)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2747)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:648)
      at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
      at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:807)
      at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
      at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107)
      at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205)
      at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522)
      at javax.xml.parsers.SAXParser.parse(SAXParser.java:395)
      at javax.xml.parsers.SAXParser.parse(SAXParser.java:198)
      at hudson.plugins.cobertura.CoberturaCoverageParser.parse(CoberturaCoverageParser.java:81)
      ... 12 more
      Email was triggered for: Failure
      Sending email for trigger: Failure
      Sending email to: xx@xx.com
      Finished: FAILURE


      Looking at the file "coverage35.xml" on the Hudson server, we see that the size is 983040 bytes. However, in the slave workspace, this file is 1023819 bytes.

      => As the file size 983040 is a multiple of 64k, and the difference between the original and copied is less than 64k, it is tempting to say that the error must be caused by some buffering issue and/or lack of a output "flush"

        Issue Links

          Activity

          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/src/main/java/hudson/FilePath.java
          core/src/test/java/hudson/FilePathTest.java
          remoting/src/main/java/hudson/remoting/Channel.java
          remoting/src/main/java/hudson/remoting/LocalChannel.java
          remoting/src/main/java/hudson/remoting/VirtualChannel.java
          http://jenkins-ci.org/commit/jenkins/341fd4bb056216eb3fd14add5d8c72ea83329498
          Log:
          [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo
          that results in missing tail bits when the master is heavily loaded.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/FilePath.java core/src/test/java/hudson/FilePathTest.java remoting/src/main/java/hudson/remoting/Channel.java remoting/src/main/java/hudson/remoting/LocalChannel.java remoting/src/main/java/hudson/remoting/VirtualChannel.java http://jenkins-ci.org/commit/jenkins/341fd4bb056216eb3fd14add5d8c72ea83329498 Log: [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo that results in missing tail bits when the master is heavily loaded.
          Hide
          dogfood dogfood added a comment -

          Integrated in jenkins_main_trunk #734
          [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo

          Kohsuke Kawaguchi : 957ffd30096a4309fc7da779d03086edb46e4fe3
          Files :

          • core/src/main/java/hudson/FilePath.java
          • remoting/src/main/java/hudson/remoting/LocalChannel.java
          • core/src/test/java/hudson/FilePathTest.java
          • remoting/src/main/java/hudson/remoting/VirtualChannel.java
          • remoting/src/main/java/hudson/remoting/Channel.java
          Show
          dogfood dogfood added a comment - Integrated in jenkins_main_trunk #734 [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo Kohsuke Kawaguchi : 957ffd30096a4309fc7da779d03086edb46e4fe3 Files : core/src/main/java/hudson/FilePath.java remoting/src/main/java/hudson/remoting/LocalChannel.java core/src/test/java/hudson/FilePathTest.java remoting/src/main/java/hudson/remoting/VirtualChannel.java remoting/src/main/java/hudson/remoting/Channel.java
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/src/main/java/hudson/FilePath.java
          core/src/test/java/hudson/FilePathTest.java
          remoting/src/main/java/hudson/remoting/Channel.java
          remoting/src/main/java/hudson/remoting/LocalChannel.java
          remoting/src/main/java/hudson/remoting/VirtualChannel.java
          http://jenkins-ci.org/commit/jenkins/957ffd30096a4309fc7da779d03086edb46e4fe3
          Log:
          [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo
          that results in missing tail bits when the master is heavily loaded.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/FilePath.java core/src/test/java/hudson/FilePathTest.java remoting/src/main/java/hudson/remoting/Channel.java remoting/src/main/java/hudson/remoting/LocalChannel.java remoting/src/main/java/hudson/remoting/VirtualChannel.java http://jenkins-ci.org/commit/jenkins/957ffd30096a4309fc7da779d03086edb46e4fe3 Log: [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo that results in missing tail bits when the master is heavily loaded.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/src/main/java/hudson/FilePath.java
          core/src/test/java/hudson/FilePathTest.java
          remoting/src/main/java/hudson/remoting/Channel.java
          remoting/src/main/java/hudson/remoting/LocalChannel.java
          remoting/src/main/java/hudson/remoting/VirtualChannel.java
          http://jenkins-ci.org/commit/jenkins/957ffd30096a4309fc7da779d03086edb46e4fe3
          Log:
          [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo
          that results in missing tail bits when the master is heavily loaded.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/src/main/java/hudson/FilePath.java core/src/test/java/hudson/FilePathTest.java remoting/src/main/java/hudson/remoting/Channel.java remoting/src/main/java/hudson/remoting/LocalChannel.java remoting/src/main/java/hudson/remoting/VirtualChannel.java http://jenkins-ci.org/commit/jenkins/957ffd30096a4309fc7da779d03086edb46e4fe3 Log: [FIXED JENKINS-7871] Fixed a race condition in FilePath.copyTo that results in missing tail bits when the master is heavily loaded.
          Hide
          pancake pancake added a comment -

          Oh, thank you for fixing that nasty bug we've been experiencing for several recent Jenkins releases!
          Which Jenkins version will the fix be included to?

          Show
          pancake pancake added a comment - Oh, thank you for fixing that nasty bug we've been experiencing for several recent Jenkins releases! Which Jenkins version will the fix be included to?

            People

            • Assignee:
              lacostej lacostej
              Reporter:
              terjestrand terjestrand
            • Votes:
              3 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: