Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55836

Unable to delete workspace due to locked git .pack file

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      In our Jenkins site, all builds which are run on a windows Jenkins client and which do not delete their working direktory prior to run the job will fail with a message like:

      [WS-CLEANUP] Deleting project workspace...
      [WS-CLEANUP] Deferred wipeout is used...
      ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Entw\Jenkins\workspace\Testautomatisierung\01_Smoketest\UFT\UFT_A_Smoketest_Master\.git\objects\pack\pack-5e75a4a1a37fced205964f0aeaadd0b304e779f8.pack'. Tried 3 times (of a maximum of 3) waiting 0,1 Sekunden between attempts.
      ERROR: Cannot delete workspace: Unable to delete 'C:\Entw\Jenkins\workspace\Testautomatisierung\01_Smoketest\UFT\UFT_A_Smoketest_Master\.git\objects\pack\pack-5e75a4a1a37fced205964f0aeaadd0b304e779f8.pack'. Tried 3 times (of a maximum of 3) waiting 0,1 Sekunden between attempts.
      Recording test results
      ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run? 
      For example, C:\Entw\Jenkins\workspace\Testautomatisierung\01_Smoketest\UFT\UFT_A_Smoketest_Master\Results29012019125534536.xml is 3 Minuten 34 Sekunden old
      
      Recording test results
      ERROR: Step ‘Publish Micro Focus tests result’ failed: Test reports were found but none of them are new. Did leafNodes run? 
      For example, C:\Entw\Jenkins\workspace\Testautomatisierung\01_Smoketest\UFT\UFT_A_Smoketest_Master\Results29012019125534536.xml is 3 Minuten 34 Sekunden old
      ...
      Sucessfully imported JUnit XML results from Results29012019125534536.xml
      Finished: FAILURE
      

        Attachments

          Activity

          Hide
          iplaman Idan Bidani added a comment -

          Assigned to hp-application-automation-tools-plugin.
          Workaround: Since I didn't need the HP plugin anymore I simply uninstalled it but for those which may still use it It could be worth trying to revert to 5.0 version(last known working version I used)
          For hp-application-automation-tools-plugin maintainers, it's recommended you either to place the repository manipulating calls inside try with resources, or switch (where feasible) to use calls to the git client plugin rather than calling directly to JGit.

          Show
          iplaman Idan Bidani added a comment - Assigned to hp-application-automation-tools-plugin. Workaround: Since I didn't need the HP plugin anymore I simply uninstalled it but for those which may still use it It could be worth trying to revert to 5.0 version(last known working version I used) For hp-application-automation-tools-plugin maintainers, it's recommended you either to place the repository manipulating calls inside try with resources, or switch (where feasible) to use calls to the git client plugin rather than calling directly to JGit.
          Hide
          bptacek Ben Ptacek added a comment -

          We also have a Windows node and we ran the fileleak and it is pointing to the same issue/plugin -

          com.microfocus.application.automation.tools.octane.model.processors.scm.GitSCMProcessor$FileContentCallable.invoke(GitSCMProcessor.java:128)

          We will report back if reverting helps.

          Show
          bptacek Ben Ptacek added a comment - We also have a Windows node and we ran the fileleak and it is pointing to the same issue/plugin - com.microfocus.application.automation.tools.octane.model.processors.scm.GitSCMProcessor$FileContentCallable.invoke(GitSCMProcessor.java:128) We will report back if reverting helps.
          Hide
          bptacek Ben Ptacek added a comment - - edited

          Okay, the revert could only go to 5.6.2 of  the Microfocus plugin and this did not help.  The issue is when you have the Delete Workspace selected (see pic below for where this option is set).  If I unselect it, stop the slave service, manually clean the workspace, start the service and run the job, then I am able to run the job successfully thereafter.

          Show
          bptacek Ben Ptacek added a comment - - edited Okay, the revert could only go to 5.6.2 of  the Microfocus plugin and this did not help.  The issue is when you have the Delete Workspace selected (see pic below for where this option is set).  If I unselect it, stop the slave service, manually clean the workspace, start the service and run the job, then I am able to run the job successfully thereafter.
          Hide
          csmith_sgas Christopher Smith added a comment - - edited

          **It turned out that we had used a different user than the user account the Jenkins slave service was using when troubleshooting/working with the Jenkins workspace directory. Despite both accounts being members of the Administrators group it seems everything needed to be owned/created/modified using the same account**

          Recently ran into issues with plugins and we did a update of all plugins. As soon as that was done this problem started occurring for us. Tried a bunch of workarounds so far and none of it seems to work. Adding the workaround Aaron suggested. Even manually deleting the contents of workspace myself doesn't seem to fix the issue either as the next time a job runs and goes to pull down a branch it fails due to permissions. My C:\Jenkins\workspace folder on the Slave node where these files should be being cleaned up is basically wide open right now with Everyone having full permission. I assume this is still related to some kind of lock....?

          Note: The below should says 

          */.git/** 
          

           but the second asterisk isn't coming through.

          */.git/* exclude and Apply pattern also on directories set in the Workspace Cleanup config in the job*
          [WS-CLEANUP] Deleting project workspace...
          ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
          ERROR: Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
          sent [C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\hpc_install20190320.log] to splunk in 1 events
          Finished: FAILURE

          */.git/* exclude, Apply pattern also on directories, and Disable deferred wipeout set in the Workspace Cleanup config in the job*
          [WS-CLEANUP] Deleting project workspace...
          [WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
          ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
          ERROR: Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
          sent [C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\hpc_install20190320.log] to splunk in 1 events
          Finished: FAILURE

          Show
          csmith_sgas Christopher Smith added a comment - - edited ** It turned out that we had used a different user than the user account the Jenkins slave service was using when troubleshooting/working with the Jenkins workspace directory. Despite both accounts being members of the Administrators group it seems everything needed to be owned/created/modified using the same account ** Recently ran into issues with plugins and we did a update of all plugins. As soon as that was done this problem started occurring for us. Tried a bunch of workarounds so far and none of it seems to work. Adding the workaround Aaron suggested. Even manually deleting the contents of workspace myself doesn't seem to fix the issue either as the next time a job runs and goes to pull down a branch it fails due to permissions. My C:\Jenkins\workspace folder on the Slave node where these files should be being cleaned up is basically wide open right now with Everyone having full permission. I assume this is still related to some kind of lock....? Note: The below should says  */.git/**   but the second asterisk isn't coming through. * /.git/ * exclude and Apply pattern also on directories set in the Workspace Cleanup config in the job* [WS-CLEANUP] Deleting project workspace... ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. ERROR: Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. sent [C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\hpc_install20190320.log] to splunk in 1 events Finished: FAILURE * /.git/ * exclude, Apply pattern also on directories, and Disable deferred wipeout set in the Workspace Cleanup config in the job* [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... ERROR: [WS-CLEANUP] Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. ERROR: Cannot delete workspace: Unable to delete 'C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\.git\FETCH_HEAD'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. sent [C:\Jenkins\workspace\LabD_Build_Automation\Validate_LabD_VCSA\hpc_install20190320.log] to splunk in 1 events Finished: FAILURE
          Hide
          markewaite Mark Waite added a comment -

          Christopher Smith you don't mention in your description if you are using the hp-automation-tools-plugin. If you are not using the hp-automation-tools-plugin, then you've found a different problem and it most likely has a different root cause. If you are not using the hp-automation-tools-plugin, please submit a separate bug report with the details of your environment and your best guess at the steps that will allow others to recreate the problem.

          Show
          markewaite Mark Waite added a comment - Christopher Smith you don't mention in your description if you are using the hp-automation-tools-plugin. If you are not using the hp-automation-tools-plugin, then you've found a different problem and it most likely has a different root cause. If you are not using the hp-automation-tools-plugin, please submit a separate bug report with the details of your environment and your best guess at the steps that will allow others to recreate the problem.

            People

            • Assignee:
              Unassigned
              Reporter:
              carsten_hilber Carsten Hilber
            • Votes:
              20 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated: