Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8346

Copy to slave incompatible with multi configuration project

    Details

    • Similar Issues:

      Description

      copy-to-slave does nothing when the job runs on the master. This is the wrong behavior if the job is a multi-configuration project, because these jobs run with a modified workspace, depending on the parameter of the current job.

      What the plugin should do when the job is running on the master, is to check if the file in the 'default' workspace is identical to the target file (it won't be in a multi-config build) and if not, copy it to the new workspace.

      For example, the job on master has this workspace:
      $ pwd
      .../hudson/jobs/bootstrap-toolchain/workspace
      $ ls
      buildconf config-env.yml config.yml my-bootstrap.sh TARGET

      where TARGET is the first category for the parametrization. The job will never run in this 'workspace' dir, but somewhere deep in the TARGET subdir. So copy-to-slave should copy the file from this workspace into the deep 'multi-config' workspace.

      For example, when my job runs, these variables are set:
      JOB_NAME=bootstrap-toolchain/TARGET=gnulinux,arch=amd64,branch=master
      WORKSPACE=.../hudson/jobs/bootstrap-toolchain/workspace/TARGET/gnulinux/arch/amd64/branch/master

      I hope this report is clear about the issue.

        Attachments

          Activity

          Hide
          rseguy Romain Seguy added a comment -

          Well, I can't say it's clear to me...

          One first thing: Does multi-configuration project == matrix project? If yes, then I'll need more details since I don't use this kind of jobs.

          Show
          rseguy Romain Seguy added a comment - Well, I can't say it's clear to me... One first thing: Does multi-configuration project == matrix project? If yes, then I'll need more details since I don't use this kind of jobs.
          Hide
          psoetens psoetens added a comment -

          Well, if you click on 'New Job' there will be a radio button for 'Build multi-configuration project'. I don't have an option for 'matrix project', although in the config page of the created project, it's called 'Configuration Matrix'.

          The point of a multi-config project is that it works in a deep subdir of the project's workspace. Somehow, the other tools, like the version control tools, know that they have to check-out to that directory, but your plugin doesn't even try, it just does the 'master' check instead of checking if source and target file are the same.

          What I'd like to be the behavior of this plugin is similar to this script (bash syntax):

          for slave in $nodes; do

          1. always copy if: slave is not master OR source and target file are not the same file.
            if [ $slave -neq "master" -o -not $HUDSON/jobs/$JOB_NAME/$file_to_copy -ef $HOME/$WORKSPACE/$file_to_copy ]; then
            scp $HUDSON/jobs/$JOB_NAME/$file_to_copy $slave:$WORKSPACE
            fi
            done

          So to use jobs/$JOB_NAME as source directory and to use $WORKSPACE as target directory.

          The JOB_NAME in the original post shows how it is actually encoded,so you'd need to strip everything after the last '/'.

          Peter

          Show
          psoetens psoetens added a comment - Well, if you click on 'New Job' there will be a radio button for 'Build multi-configuration project'. I don't have an option for 'matrix project', although in the config page of the created project, it's called 'Configuration Matrix'. The point of a multi-config project is that it works in a deep subdir of the project's workspace. Somehow, the other tools, like the version control tools, know that they have to check-out to that directory, but your plugin doesn't even try, it just does the 'master' check instead of checking if source and target file are the same. What I'd like to be the behavior of this plugin is similar to this script (bash syntax): for slave in $nodes; do always copy if: slave is not master OR source and target file are not the same file. if [ $slave -neq "master" -o -not $HUDSON/jobs/$JOB_NAME/$file_to_copy -ef $HOME/$WORKSPACE/$file_to_copy ]; then scp $HUDSON/jobs/$JOB_NAME/$file_to_copy $slave:$WORKSPACE fi done So to use jobs/$JOB_NAME as source directory and to use $WORKSPACE as target directory. The JOB_NAME in the original post shows how it is actually encoded,so you'd need to strip everything after the last '/'. Peter
          Hide
          rseguy Romain Seguy added a comment -

          Thanks for the precision, I think I'll have to dive into the way Hudson handles that.
          In fact, when you say

          the other tools, like the version control tools, know that they have to check-out to that directory, but your plugin doesn't even try, it just does the 'master' check instead of checking if source and target file are the same

          you're absolutely right, but it's on purpose/required: Hudson offers tons of methods to get the current workspace and so on, "but" (this should not be "but" but "as a consequence" in a normal case) these methods return pointer to the actual location (on the slave, etc.).
          The only one way to get the workspace on the master, for example when doing a copy-to-slave, is then to build the pointer manually (by appending /workspace to the job name). Which means: All possible cases handled by Hudson have to be manually handled in the Copy To Slave plugin.

          Show
          rseguy Romain Seguy added a comment - Thanks for the precision, I think I'll have to dive into the way Hudson handles that. In fact, when you say the other tools, like the version control tools, know that they have to check-out to that directory, but your plugin doesn't even try, it just does the 'master' check instead of checking if source and target file are the same you're absolutely right, but it's on purpose/required: Hudson offers tons of methods to get the current workspace and so on, "but" (this should not be "but" but "as a consequence" in a normal case) these methods return pointer to the actual location (on the slave, etc.). The only one way to get the workspace on the master, for example when doing a copy-to-slave, is then to build the pointer manually (by appending /workspace to the job name). Which means: All possible cases handled by Hudson have to be manually handled in the Copy To Slave plugin.
          Hide
          bleonard Brad Leonard added a comment -

          I am running into this problem when trying to copy from slave workspace to master workspace in a matrix (multi-configuration) project.

          The error as seen in console output on slave is:

          [copy-to-slave] Copying '***.msi', excluding nothing, from 'file:/C:/jenkins/workspace/project-name/label/windows/' on 'hudson.slaves.DumbSlave@e5e309d5' to 'file:/Users/build/.jenkins/jobs/project-name/configurations/axis-label/windows/workspace/' on the master.
          ERROR: Publisher com.michelin.cio.hudson.plugins.copytoslave.CopyToMasterNotifier aborted due to exception
          java.lang.InterruptedException
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:485)
          at hudson.remoting.Request$1.get(Request.java:218)
          at hudson.remoting.Request$1.get(Request.java:184)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
          at com.michelin.cio.hudson.plugins.copytoslave.MyFilePath.copyRecursiveTo(MyFilePath.java:145)
          at com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveUtils.hudson5977(CopyToSlaveUtils.java:102)
          at com.michelin.cio.hudson.plugins.copytoslave.CopyToMasterNotifier.perform(CopyToMasterNotifier.java:81)
          at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
          at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:694)
          at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:669)
          at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:647)
          at hudson.model.Build$RunnerImpl.cleanUp(Build.java:171)
          at hudson.model.Run.run(Run.java:1448)
          at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:230)

          Each unsuccessful build creates an empty directory in the slaves workspace titled 'copy-to-slave8439381225529586145.dir'. The directories listed at the top of the console output do in fact exist.

          Show
          bleonard Brad Leonard added a comment - I am running into this problem when trying to copy from slave workspace to master workspace in a matrix (multi-configuration) project. The error as seen in console output on slave is: [copy-to-slave] Copying '***.msi', excluding nothing, from 'file:/C:/jenkins/workspace/project-name/label/windows/' on 'hudson.slaves.DumbSlave@e5e309d5' to 'file:/Users/build/.jenkins/jobs/project-name/configurations/axis-label/windows/workspace/' on the master. ERROR: Publisher com.michelin.cio.hudson.plugins.copytoslave.CopyToMasterNotifier aborted due to exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at hudson.remoting.Request$1.get(Request.java:218) at hudson.remoting.Request$1.get(Request.java:184) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55) at com.michelin.cio.hudson.plugins.copytoslave.MyFilePath.copyRecursiveTo(MyFilePath.java:145) at com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveUtils.hudson5977(CopyToSlaveUtils.java:102) at com.michelin.cio.hudson.plugins.copytoslave.CopyToMasterNotifier.perform(CopyToMasterNotifier.java:81) at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:694) at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:669) at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:647) at hudson.model.Build$RunnerImpl.cleanUp(Build.java:171) at hudson.model.Run.run(Run.java:1448) at hudson.matrix.MatrixRun.run(MatrixRun.java:146) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:230) Each unsuccessful build creates an empty directory in the slaves workspace titled 'copy-to-slave8439381225529586145.dir'. The directories listed at the top of the console output do in fact exist.
          Hide
          abigos Andy Bigos added a comment -

          I have a similar problem with matrix configs. I'm copying the changelog.xml file into my workspace. So for a simple build I have:

          jobs/$JOB_NAME/builds/$BUILD_ID/changelog.xml

          with path set relative to HUDSON_HOME. This works for a simple project, however for a matrix project the $JOB_NAME gets expanded to the matrix element job name - this means the copy will fail as the source directory is not on the master. In the case of a matrix projects I think the plugin needs to use the 'base' project path, not the path for a matrix element.

          Can you confirm this is an issue? I can log it as new issue if you think it's different to the current issue.

          Thanks
          Andy

          Show
          abigos Andy Bigos added a comment - I have a similar problem with matrix configs. I'm copying the changelog.xml file into my workspace. So for a simple build I have: jobs/$JOB_NAME/builds/$BUILD_ID/changelog.xml with path set relative to HUDSON_HOME. This works for a simple project, however for a matrix project the $JOB_NAME gets expanded to the matrix element job name - this means the copy will fail as the source directory is not on the master. In the case of a matrix projects I think the plugin needs to use the 'base' project path, not the path for a matrix element. Can you confirm this is an issue? I can log it as new issue if you think it's different to the current issue. Thanks Andy

            People

            • Assignee:
              danielpetisme Daniel Petisme
              Reporter:
              psoetens psoetens
            • Votes:
              6 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: