Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60785

corrupt workspaces.txt file on agent, changes multibranch builds to use workspace directory with slash converted to %2F, causing errors

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: branch-api-plugin
    • Labels:
    • Environment:
      linux, multi-branch pipeline build with agents doing the building.
      Jenkins version 2.190.3 Branch API Plugin Version 2.5.5
    • Similar Issues:

      Description

      I'm facing the same problem with my multibranch pipeline project: "feature/branch" from a git repository is transformed to "feature%2Fbranch" which causes build to fail because the build process cannot find the files. Agents  aredoing the building. In my case interestingly this was only happening on 1 of three nodes (it took me hours to figure THAT out).

      After much searching what I found was that when I reviewed the node log for the failing node (Manage Jenkins -> Manage Nodes -> click on node in question -> Log, I found that the /home/jenkins/workspace/workspace.txt file was noted as being corrupt.  For whatever reason this meant that when the build was attempted on that node, the build directory was not the usual workspace directory (which has all branch name slashes (encoded as %2F) cases  replaced with underscores).

      In my case, I just created a new workspace.txt file and then disconnected and reconnected the node.  And... things started working again.

      So, if you're experiencing this, review the node logs.  

      As a suggestion for improving jenkins, I would suggest that if the workspaces.txt file is corrupt, the node should be taken offline.  At least then the admin has a clue that this is a node based issue, and that the node needs manual intervention for things to get back to normal.

      From Jesse Glick,

      "Of course better would be to automatically correct a corrupted file, and better still would be to figure out what corrupted it to begin with. What did the supposedly corrupt file contain?"

      I've attached the corrupted workspaces.txt file. It looks like it was cut off mid way through writing a branch name.  Perhaps variable overrun, or an uncerimonious shut down created this issue.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            an uncerimonious shut down

            It certainly looks like it, though the code is supposed to defend against that. Not sure what happened. Do you still have the system log that you found the warning in?

            At any rate, this code could try harder. Currently it falls back to the previous implementation (a reversible translation from job name to folder name), which was correct behavior so far as it goes but leads to trouble if you are using buggy build tools which cannot handle certain characters in the working directory path. Alternately it could discard the corrupted index and start fresh, though this introduces other risks: stale workspaces being permanently forgotten (consuming disk space); possible collision between similarly-named jobs trying to use the same (nonreversibly simplified) folder name.

            Show
            jglick Jesse Glick added a comment - an uncerimonious shut down It certainly looks like it, though the code is supposed to defend against that . Not sure what happened. Do you still have the system log that you found the warning in? At any rate, this code could try harder. Currently it falls back to the previous implementation (a reversible translation from job name to folder name), which was correct behavior so far as it goes but leads to trouble if you are using buggy build tools which cannot handle certain characters in the working directory path. Alternately it could discard the corrupted index and start fresh, though this introduces other risks: stale workspaces being permanently forgotten (consuming disk space); possible collision between similarly-named jobs trying to use the same (nonreversibly simplified) folder name.
            Hide
            jrichards Jeff Richards added a comment -

            Yeah, from my perspective, I would have been totally satisfied if the node was just marked offline. 

            The file is corrupt.  There is a problem. The problem is rare (any defense against partial writing of a file in code is admirable, but going to be partial (what happens in case of power outages). So marking the node offline would avoid an unexpected workspace name, alert that the node needs attention and the confusion that that causes.  Call in the humans and let them deal with it.

            Once I realized that my issue was related to a problem on a single node, it only took a few minutes to think to look at the log, which pointed me straight to the problem.  And while I didn't know quite what to do to fix it, I think that would be handled by an FAQ/Troubleshooting entry on the web for people to find.  Or better yet reference it in the log message that is displayed.

            Show
            jrichards Jeff Richards added a comment - Yeah, from my perspective, I would have been totally satisfied if the node was just marked offline.  The file is corrupt.  There is a problem. The problem is rare (any defense against partial writing of a file in code is admirable, but going to be partial (what happens in case of power outages). So marking the node offline would avoid an unexpected workspace name, alert that the node needs attention and the confusion that that causes.  Call in the humans and let them deal with it. Once I realized that my issue was related to a problem on a single node, it only took a few minutes to think to look at the log, which pointed me straight to the problem.  And while I didn't know quite what to do to fix it, I think that would be handled by an FAQ/Troubleshooting entry on the web for people to find.  Or better yet reference it in the log message that is displayed.

              People

              • Assignee:
                Unassigned
                Reporter:
                jrichards Jeff Richards
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: