Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41854

Contextualize a fresh FilePath after an agent reconnection

    Details

    • Similar Issues:
    • Released As:
      workflow-durable-task-step 2.31, workflow-basic-steps 2.18

      Description

      PlaceholderExecutable does not bother listening for a closed connection as such, it just lets any nested step actually using the agent fail if the connection dies in the middle. Usually this is fine, but there is a corner case where it is probably wrong (unconfirmed): if an agent is disconnected and reconnected during the first sh in

      node {
        sh 'sleep 999'
        sh 'sleep 999'
      }
      

      then the second sh could fail since it would be using the old Channel, even when the first sh succeeds because the DurableTaskStep.Execution recomputes the FilePath after the ChannelClosedException. (But if Jenkins restarted during the first sh after the reconnection then it should work, since the FilePath would be reconstructed from a FilePathPickle.)

      The fix may be tricky since FilePath.channel is effectively final, so currently whatever workspace is passed from PlaceholderExecutable will be used for the duration of the block. Perhaps BodyInvoker.withContexts should support offering a Provider of contextual objects—in this case something that caches a FilePath so long as it is valid (!Channel.outClosed?), and otherwise falls back to FilePathUtils.find like FilePathPickle.

        Attachments

          Issue Links

            Activity

            jglick Jesse Glick created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Epic Link JENKINS-35399 [ 171192 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-40613 [ JENKINS-40613 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-47868 [ JENKINS-47868 ]
            Hide
            jglick Jesse Glick added a comment -

            Oleg Nenashev reminds me that fixing this would probably involve also acquiring a fresh WorkspaceList lock, after a new Computer becomes available and hence a new WorkspaceList.

            Show
            jglick Jesse Glick added a comment - Oleg Nenashev reminds me that fixing this would probably involve also acquiring a fresh WorkspaceList lock, after a new Computer becomes available and hence a new WorkspaceList .
            Hide
            basil Basil Crow added a comment -

            Is there any workaround for this issue? Is there any way for me to run my scripted pipeline on a node without using the problematic PlaceholderExecutable?

            Show
            basil Basil Crow added a comment - Is there any workaround for this issue? Is there any way for me to run my scripted pipeline on a node without using the problematic PlaceholderExecutable ?
            Hide
            jglick Jesse Glick added a comment -

            If the issue indeed exists in the described form, I would not expect there to be any general workaround. The build would just fail. Possibly you could retry the second sh. You could consolidate the scripts, if there were no crucial intervening steps of other kinds. You could run each sh in its own node, using stash and unstash as needed.

            Show
            jglick Jesse Glick added a comment - If the issue indeed exists in the described form, I would not expect there to be any general workaround. The build would just fail. Possibly you could retry the second sh . You could consolidate the scripts, if there were no crucial intervening steps of other kinds. You could run each sh in its own node , using stash and unstash as needed.
            basil Basil Crow made changes -
            Link This issue is duplicated by JENKINS-50504 [ JENKINS-50504 ]
            Hide
            basil Basil Crow added a comment -

            > If the issue indeed exists in the described form

            The issue does indeed exist in the described form, as my production experience shows (described in JENKINS-50504, which I have now marked as a duplicate of this bug). I can also confirm that restarting Jenkins fixes the problem as described above by reconstructing the FilePath from a FilePathPickle. However, between the time that I hit this bug and the time I restart Jenkins, in-use workspaces are handed out to new runs, causing both runs to fail. This is a huge inconvenience for my users.

            I have written a reproducible test case.

            Show
            basil Basil Crow added a comment - > If the issue indeed exists in the described form The issue does indeed exist in the described form, as my production experience shows (described in JENKINS-50504 , which I have now marked as a duplicate of this bug). I can also confirm that restarting Jenkins fixes the problem as described above by reconstructing the FilePath from a FilePathPickle . However, between the time that I hit this bug and the time I restart Jenkins, in-use workspaces are handed out to new runs, causing both runs to fail. This is a huge inconvenience for my users. I have written a reproducible test case .
            tom_ghyselinck Tom Ghyselinck made changes -
            Link This issue is related to JENKINS-49707 [ JENKINS-49707 ]
            vivek Vivek Pandey made changes -
            Labels robustness robustness triaged-2018-11
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-54643 [ JENKINS-54643 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-46067 [ JENKINS-46067 ]
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 101 (Web Link)" [ 22536 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-41791 [ JENKINS-41791 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-basic-steps #86 (Web Link)" [ 23036 ]
            Hide
            dnusbaum Devin Nusbaum added a comment - - edited

            Fixes for this issue have just been released in Pipeline Nodes and Processes Plugin version 2.31 and Pipeline Basic Steps Plugin version 2.18. You must update Pipeline Groovy Plugin to version 2.70 at the same time you update the other plugins.

            Show
            dnusbaum Devin Nusbaum added a comment - - edited Fixes for this issue have just been released in Pipeline Nodes and Processes Plugin version 2.31 and Pipeline Basic Steps Plugin version 2.18. You must update Pipeline Groovy Plugin to version 2.70 at the same time you update the other plugins.
            dnusbaum Devin Nusbaum made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Released As workflow-durable-task-step 2.31, workflow-basic-steps 2.17
            Hide
            jglick Jesse Glick added a comment -

            (Or you may update Pipeline: Groovy first, and then the other plugins later.)

            Show
            jglick Jesse Glick added a comment - (Or you may update Pipeline: Groovy first , and then the other plugins later .)
            Hide
            basil Basil Crow added a comment -

            Well done! Thank you for fixing this long-standing robustness issue. This is much appreciated!

            Show
            basil Basil Crow added a comment - Well done! Thank you for fixing this long-standing robustness issue. This is much appreciated!
            dnusbaum Devin Nusbaum made changes -
            Released As workflow-durable-task-step 2.31, workflow-basic-steps 2.17 workflow-durable-task-step 2.31, workflow-basic-steps 2.18
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-49651 [ JENKINS-49651 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-58900 [ JENKINS-58900 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                4 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: