Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29656

Windows save of program.dat fails

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Component/s: workflow-cps-plugin
    • Labels:
    • Environment:
      Operating System: Windows 7 64 bit
      Jenkins: 1.596.2 and 1.609.1
      Java: 1.7.0_15
    • Similar Issues:

      Description

      Hello everyone,

      We are facing a "program state save failed" error which prevents the workflow from running the next steps.

      Aug 24, 2015 1:18:27 PM org.jenkinsci.plugins.workflow.cps.CpsThreadGroup saveProgram
      WARNING: program state save failed
      java.io.IOException: rename C:\Jenkins\current\jobs\fo12_MainStream_R1X_development_path_Pilot_Workflow_003\builds\45\atomic7701129931116736150.tmp to C:\Jenkins\current\jobs\fo12_MainStream_R1X_development_path_Pilot_Workflow_003\builds\45\program.dat failed
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:350)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:328)
      	at org.jenkinsci.plugins.workflow.cps.CpsStepContext$3.onSuccess(CpsStepContext.java:482)
      	at org.jenkinsci.plugins.workflow.cps.CpsStepContext$3.onSuccess(CpsStepContext.java:478)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:546)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:32)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      

      This error can happen in different steps of the same Workflow, for example during creation of source-files from a model or during compilation.

      We did multiple tests to find-out possible reasons for this issue, by implementing a Workflow with these properties:

      • with and without stages,
      • with and without try-catch blocks,
      • using multiple node{} steps or a single one,
      • using multiple batch in a function or a single one,
      • adding sleep commands between the steps

      But the problem still persists. The Workflow is not stable in this case and we cannot use it for our project.

      We are using following environments and tools:

      • Windows 7 Professional 64-bit
      • Java: 1.7.0_15
      • Jenkins: 1.609.1
      • MKS Integrity Client 2009

      EDIT: I tried to find out why the program.dat file could not be renamed by using Process monitor to monitor the builds directory of the job.
      I got the following information corresponding to the time where the error occured:

      Date: 24.08.2015 13:18:27,8099332
      Thread: 8036
      Class: File System
      Operation: SetRenameInformationFile
      Result: NAME COLLISION
      Path: C:\Jenkins\current\jobs\fo12_MainStream_R1X_development_path_Pilot_Workflow_003\builds\45\atomic7701129931116736150.tmp

      ReplaceIfExists; False
      FileName. C:\Jenkins\current\jobs\fo12_MainStream_R1X_development_path_Pilot_Workflow_003\builds\45\program.dat

      Thank you in advance for your help.

        Attachments

        1. Error 2015-09-14.txt
          3 kB
        2. ProcessMonitor.PNG
          ProcessMonitor.PNG
          312 kB
        3. workflow-cps.hpi
          1.78 MB

          Issue Links

            Activity

            Hide
            cowwoc cowwoc added a comment -

            Jesse Glick Taking a cue from leon_j, I used Process Monitor to track down who was locking program.dat.

            In my case, CrashPlan (backup software) locks program.dat for 90 ms after the file is created. Unfortunately, Jenkins happens to try moving atomic*.tmp to program.dat during this time-frame and dies horribly.

            A sleep+retry loop would definitely solve the problem in my case.

            Show
            cowwoc cowwoc added a comment - Jesse Glick Taking a cue from leon_j, I used Process Monitor to track down who was locking program.dat. In my case, CrashPlan (backup software) locks program.dat for 90 ms after the file is created. Unfortunately, Jenkins happens to try moving atomic*.tmp to program.dat during this time-frame and dies horribly. A sleep+retry loop would definitely solve the problem in my case.
            Hide
            jglick Jesse Glick added a comment -

            Filed alternate PR.

            Show
            jglick Jesse Glick added a comment - Filed alternate PR.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/20c6a3036c8b0da7269951ff01947bc2e35ccde2
            Log:
            [FIXED JENKINS-29656] If saving program.dat fails only due to the final rename, throw up an exception but do not fail the build.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java http://jenkins-ci.org/commit/workflow-cps-plugin/20c6a3036c8b0da7269951ff01947bc2e35ccde2 Log: [FIXED JENKINS-29656] If saving program.dat fails only due to the final rename, throw up an exception but do not fail the build.
            Hide
            jglick Jesse Glick added a comment -

            Hope that fixed the issue, or at least reduced it to some occasional log warnings.

            Show
            jglick Jesse Glick added a comment - Hope that fixed the issue, or at least reduced it to some occasional log warnings.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/89ce0aa56f2f84fc60231fdba29108c4a810d72f
            Log:
            Merge pull request #96 from jglick/saveProgramIfPossible-JENKINS-29656

            JENKINS-29656 Better handle failure renaming program.dat

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java http://jenkins-ci.org/commit/workflow-cps-plugin/89ce0aa56f2f84fc60231fdba29108c4a810d72f Log: Merge pull request #96 from jglick/saveProgramIfPossible- JENKINS-29656 JENKINS-29656 Better handle failure renaming program.dat

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                leon_j leon_j
              • Votes:
                7 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: