Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11993

matrix flyweight and configuration builds have "overlapping" workspace directories

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I have a (few) matrix jobs on a distributed build environment, I'm using SCM(SVN) polling using "Emulate clean checkout by first deleting unversioned/ignored files, then 'svn update'" strategy.

      I just noticed that sometimes a random configuration job from my matrix job lose it's workspace, and has to do a clean checkout:
      "Checking out a fresh workspace because there's no workspace at /home/jenkins/workspace/php-src-5.4-matrix-build"

      After experiencing it a few times I also noticed, that it is correlated to the other issue, that I was seeing, namely that sometimes the flyweight build of my matrix job gets executed on a different node that I was pinning to using the matrix tie parent plugin.
      Of course executing that on a different node means that it has to do a clean checkout from SCM, and this is when I got enlightened:

      • the flyweight build will use $JENKINS_HOME/$JOB_NAME as the workspace directory
      • the configuration jobs will use the $JENKINS_HOME/$JOB_NAME/$AXES
        This means that if both the flyweight and one of the configuration jobs will be executed on the same node, they will interfere with each others workspaces.
        What I see to happen, is that the flyweight job will delete everything from $JENKINS_HOME/$JOB_NAME, and create a clean checkout there.
        Which in turn will force my configuration build to also do a clean checkout, as it won't find it's workspace.

      I created a bug report for the immediate problem (which made this issue visible for me): that the matrix tie parent plugin won't schedule the flyweight job to the tied label in some cases:
      https://issues.jenkins-ci.org/browse/JENKINS-11987

      The author of that plugin confirmed, that this could indeed happen, and marked the bug as "Not A Defect".

      Now I don't really see a way to prevent this issue from happening, as the matrix tie parent can only reduce this situation, but not prevent it, and I can't tell jenkins to use a different workspace path for the flyweight and the configuration job(different == one of them not containing the other one).

      Do you think that this could be fixed somehow, or do you know any workaround for this problem?

        Attachments

          Issue Links

            Activity

            Hide
            tyrael Ferenc Kovacs added a comment -

            I just created a much more simplified test scenarion:

            • create a matrix project using an svn repo and "Emulate clean checkout by first deleting unversioned/ignored files, then 'svn update'"
            • create a custom axis named test, with dummy values like "1 2 3"
            • execute the build, make sure you have changes (update to a previous revision manually or commit something)

            first time, you will get a checkout for all of your builds (flyweight + 3 configuration job)

            on the second(and every other) run, you will get the following for the flyweight build:
            Cleaning up /var/lib/jenkins/jobs/matrix-svn-checkout-bug/workspace/.
            Deleting /var/lib/jenkins/jobs/matrix-svn-checkout-bug/workspace/./test

            then the configuration builds will have their workspace removed (they were under /var/lib/jenkins/jobs/matrix-svn-checkout-bug/workspace/test) so a full checkout will happen.

            for now, I will change my "Check-out Strategy", as this is the easier fix, but here are some possible solutions:

            • if the matrixtieparent would work every time, that would prevent this from happening.
            • if jenkins would use different workspace directory for the flyweight and the configuration builds, that would also fix this.
            • the svn plugin could be aware of this edge case, and not removing the configuration build workspace from the flyweight build workspace.
            Show
            tyrael Ferenc Kovacs added a comment - I just created a much more simplified test scenarion: create a matrix project using an svn repo and "Emulate clean checkout by first deleting unversioned/ignored files, then 'svn update'" create a custom axis named test, with dummy values like "1 2 3" execute the build, make sure you have changes (update to a previous revision manually or commit something) first time, you will get a checkout for all of your builds (flyweight + 3 configuration job) on the second(and every other) run, you will get the following for the flyweight build: Cleaning up /var/lib/jenkins/jobs/matrix-svn-checkout-bug/workspace/. Deleting /var/lib/jenkins/jobs/matrix-svn-checkout-bug/workspace/./test then the configuration builds will have their workspace removed (they were under /var/lib/jenkins/jobs/matrix-svn-checkout-bug/workspace/test) so a full checkout will happen. for now, I will change my "Check-out Strategy", as this is the easier fix, but here are some possible solutions: if the matrixtieparent would work every time, that would prevent this from happening. if jenkins would use different workspace directory for the flyweight and the configuration builds, that would also fix this. the svn plugin could be aware of this edge case, and not removing the configuration build workspace from the flyweight build workspace.
            Hide
            kmart216 David Humeniuk added a comment -

            This is problem for me as well. Basically makes the emulate clean checkout strategy useless with matrix projects.

            Show
            kmart216 David Humeniuk added a comment - This is problem for me as well. Basically makes the emulate clean checkout strategy useless with matrix projects.
            Hide
            tyrael Ferenc Kovacs added a comment -

            for the record: I've just realized that the Exclude flyweight tasks plugin can be used to workaround this problem.

            Show
            tyrael Ferenc Kovacs added a comment - for the record: I've just realized that the Exclude flyweight tasks plugin can be used to workaround this problem.
            Hide
            kevinm Kevin Moody added a comment -

            Any updates on this issue? I believe that I might be seeing something similar. However, my matrix builds are using the git plugin, as the scm, and execute the "clean after checkout" option.

            Show
            kevinm Kevin Moody added a comment - Any updates on this issue? I believe that I might be seeing something similar. However, my matrix builds are using the git plugin, as the scm, and execute the "clean after checkout" option.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Use Throttle Concurrent Builds plugin (or its equivalents).
            The described issue is not a defect IMO

            Show
            oleg_nenashev Oleg Nenashev added a comment - Use Throttle Concurrent Builds plugin (or its equivalents). The described issue is not a defect IMO
            Hide
            tyrael Ferenc Kovacs added a comment -

            I don't have this problem anymore, as the project using this setup was eventually moved to git, but from a quick look into Throttle Concurrent Builds plugin, I'm not sure that this would solve the issue.
            AFAIR you don't have to enable concurrent execution for the matrix job to be able to bump into this issue, and I don't see any mention about flyweight builds at https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin and even if it would affect the execution of the flyweight jobs, this issue isn't about executing the jobs in concurrent, but executing the flyweight and the matrix configuration jobs on the same node (even in a serialized matter).
            could you ellaborate how would your suggestion work in the practice?
            I can see why could somebody argue that the svn plugin is doing what it was instructed to do (eg. wiping out any files/dirs from the checkout) but I think that it is rather unfortunate, that by default the checkout dir for the flyweight job to fetch the remote changes will use the same directory which the matrix configuration builds are residing in.

            Show
            tyrael Ferenc Kovacs added a comment - I don't have this problem anymore, as the project using this setup was eventually moved to git, but from a quick look into Throttle Concurrent Builds plugin, I'm not sure that this would solve the issue. AFAIR you don't have to enable concurrent execution for the matrix job to be able to bump into this issue, and I don't see any mention about flyweight builds at https://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin and even if it would affect the execution of the flyweight jobs, this issue isn't about executing the jobs in concurrent, but executing the flyweight and the matrix configuration jobs on the same node (even in a serialized matter). could you ellaborate how would your suggestion work in the practice? I can see why could somebody argue that the svn plugin is doing what it was instructed to do (eg. wiping out any files/dirs from the checkout) but I think that it is rather unfortunate, that by default the checkout dir for the flyweight job to fetch the remote changes will use the same directory which the matrix configuration builds are residing in.
            Hide
            danielbeck Daniel Beck added a comment -

            Well, this is kind of messed up to be honest.

            Simplest solution would be to have the parent build use a subdirectory as well, so cleaning that up would not interfere with child build workspaces.

            Show
            danielbeck Daniel Beck added a comment - Well, this is kind of messed up to be honest. Simplest solution would be to have the parent build use a subdirectory as well, so cleaning that up would not interfere with child build workspaces.
            Hide
            tlipcon Todd Lipcon added a comment -

            Still no update on this? It would be great if there were a way to specify the "custom workspace" separately for the parent from the child builds. "Throttle concurrent builds" is a hacky workaround at best. Another hack that might be helpful is a variable like "$IS_PARENT_BUILD" or some way to programatically generate the workspace patch via a template language or groovy snippet or somesuch.

            Show
            tlipcon Todd Lipcon added a comment - Still no update on this? It would be great if there were a way to specify the "custom workspace" separately for the parent from the child builds. "Throttle concurrent builds" is a hacky workaround at best. Another hack that might be helpful is a variable like "$IS_PARENT_BUILD" or some way to programatically generate the workspace patch via a template language or groovy snippet or somesuch.

              People

              • Assignee:
                kbertelson kbertelson
                Reporter:
                tyrael Ferenc Kovacs
              • Votes:
                3 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: