Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50025

On some Windows Build Agents, Batch Steps Hang

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Component/s: durable-task-plugin
    • Environment:
      Durable-task plugin v2.20, Jenkins core irrelevant, Windows Build Agent running batch steps (some agents, not others)
    • Similar Issues:

      Description

      Like the title said, two users reported in comments on JENKINS-34150 that they'd seen a recurrence of Windows batch steps hanging when upgrading durable task plugin from 1.18 to 1.20

      Notes: all initial tests passed in CI (including Windows), both with the durable-task plugin and with workflow-durable-task plugin upgraded to use v1.20 as a dependency.

      I initially identified a potential issue with the line terminator (\n vs \r\n) when running without returning a stdOut file. When testing with helpful user Steven Foster this seemed to initially solve the issue on one Jenkins instance... but not on another. Both were using Windows build agents from the same Windows image.

      Testing showed that the final command of the batch file (move "%s.tmp" "%s" with the appropriate escaping applied) seems to have failed. The jenkins-result.txt.tmp file existed but had not been copied to jenkins-result.txt).

      The users's jenkins-wrap.bat file is below (with their permission):

      @echo off
      cmd /c ""D:/jenkins/workspace/le-32QSIUVUUTA2U@tmp/durable-605bf002/jenkins-main.bat"" > "D:/jenkins/workspace/le-32QSIUVUUTA2U@tmp/durable-605bf002/jenkins-log.txt" 2>&1
      echo %ERRORLEVEL% > "D:/jenkins/workspace/le-32QSIUVUUTA2U@tmp/durable-605bf002/jenkins-result.txt.tmp"
      move "D:/jenkins/workspace/le-32QSIUVUUTA2U@tmp/durable-605bf002/jenkins-result.txt.tmp" "D:/jenkins/workspace/le-32QSIUVUUTA2U@tmp/durable-605bf002/jenkins-result.txt"
      

      Running this command permitted the batch step to complete successfuly as expected:

      move D:/jenkins/workspace/le-32QSIUVUUTA2U@tmp/durable-605bf002/jenkins-result.txt D:/jenkins/workspace/le-32QSIUVUUTA2U@tmp/durable-605bf002/jenkins-result.txt.tmp
      

      On the system that failed, it appears that running the wrapper script directly triggered a "the System cannot find the file specified" in workspace D:\jenkins\workspace\project\dttest@tmp\durable-6b266373 (again, path listed with the user's permission).

      It's possible that something like https://github.com/svanoort/durable-task-plugin/commit/38fc4709fb89da12b78e79d1292298de3d5ca24e#diff-ba4605d3233e8bd08bb06dd87253e5f8 might be needed here (double-quoting) but the simpler and safer solution is to revert the changes to the Batch file durable task – the check for a zero-length result file should protect against the issue that PR#66 resolved.

        Attachments

          Issue Links

            Activity

            Hide
            svanoort Sam Van Oort added a comment -

            This essentially duplicates the previous issue but I'm tracking separately because the root cause is different and we want to track the investigation.

            Show
            svanoort Sam Van Oort added a comment - This essentially duplicates the previous issue but I'm tracking separately because the root cause is different and we want to track the investigation.
            Hide
            svanoort Sam Van Oort added a comment -

            Released with v1.21

            Show
            svanoort Sam Van Oort added a comment - Released with v1.21
            Hide
            svanoort Sam Van Oort added a comment -

            James Nord Has suggested the root cause here might be antivirus blocking the terminal 'move' operation.

            Show
            svanoort Sam Van Oort added a comment - James Nord Has suggested the root cause here might be antivirus blocking the terminal 'move' operation.

              People

              • Assignee:
                svanoort Sam Van Oort
                Reporter:
                svanoort Sam Van Oort
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: