Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27152

Store sh control files outside of workspace

    Details

    • Similar Issues:

      Description

      Currently control files are generated in a subdirectory of the workspace. This is causing various issues because some build steps are not expecting these extra files which are internal to Jenkins.

      These could be generated instead in a temporary directory or in a specific subfolder of the slave root instead.

        Attachments

          Issue Links

            Activity

            vlatombe Vincent Latombe created issue -
            vlatombe Vincent Latombe made changes -
            Field Original Value New Value
            Summary Store bourneshell control file outside of workspace Store bourneshell control files outside of workspace
            vlatombe Vincent Latombe made changes -
            Link This issue is related to JENKINS-26105 [ JENKINS-26105 ]
            vlatombe Vincent Latombe made changes -
            Remote Link This issue links to "[jenkins-users] Workflow Plugin - Performing a Release (Web Link)" [ 12127 ]
            Hide
            sumdumgai A C added a comment -

            This is a very significant flaw with this plugin. The temporary-directory solution in the workspace is still far less than ideal, this really should use a system-provided temporary directory.

            Show
            sumdumgai A C added a comment - This is a very significant flaw with this plugin. The temporary-directory solution in the workspace is still far less than ideal, this really should use a system-provided temporary directory.
            Hide
            jglick Jesse Glick added a comment -

            Agreed. A subfolder of the slave root would probably be safest.

            A little trickier than merely changing the directory location, though, since the volumes parameter here must be adjusted to include that location, meaning some kind of API from durable-task by which docker-workflow could find that temporary directory.

            Show
            jglick Jesse Glick added a comment - Agreed. A subfolder of the slave root would probably be safest. A little trickier than merely changing the directory location, though, since the volumes parameter here must be adjusted to include that location, meaning some kind of API from durable-task by which docker-workflow could find that temporary directory.
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-26133 [ JENKINS-26133 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-30240 [ JENKINS-30240 ]
            Hide
            jglick Jesse Glick added a comment -

            It was also found that CLI Git checkouts using SSH private keys create temporary files which must be put in a standardized location. Seems there needs to be some shadow directory created on demand for any workspace where temporary files may be placed which are only intended to be accessed by build steps running in that workspace.

            credentials-binding should also be updated to use such a location for its temporary files.

            The lack of this API was just never noticed until Image.inside came along, because plugins simply assumed that build steps using a workspace could also access arbitrary files created on the same computer by the slave agent.

            Show
            jglick Jesse Glick added a comment - It was also found that CLI Git checkouts using SSH private keys create temporary files which must be put in a standardized location. Seems there needs to be some shadow directory created on demand for any workspace where temporary files may be placed which are only intended to be accessed by build steps running in that workspace. credentials-binding should also be updated to use such a location for its temporary files . The lack of this API was just never noticed until Image.inside came along, because plugins simply assumed that build steps using a workspace could also access arbitrary files created on the same computer by the slave agent.
            jglick Jesse Glick made changes -
            Labels robustness workflow api docker robustness workflow
            jglick Jesse Glick made changes -
            Summary Store bourneshell control files outside of workspace Store sh control files outside of workspace
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-30884 [ JENKINS-30884 ]
            Hide
            cruftex Jens Wilke added a comment - - edited

            Just came across this issue, since I am trying to cleanup the workspace with this code sequence:

            sh 'find -mindepth 1 -maxdepth 1 \\! -name .git -exec rm {} +'
            

            My intention is to clean everything but keep the .git directory. However, this fails, because it also tries to remove the temporary directory, which e.g. contains the script file that it is currently executing, like: ./.4cba202f/script.sh

            Since the directory is arbitrary, there is no easy workaround.

            I think a gradual improvement would be to give the directory at least a consistent name or prefix, like .jenkins-tmp/4cba202f/.... This way it is possible to stay away from it.

            Show
            cruftex Jens Wilke added a comment - - edited Just came across this issue, since I am trying to cleanup the workspace with this code sequence: sh 'find -mindepth 1 -maxdepth 1 \\! -name .git -exec rm {} +' My intention is to clean everything but keep the .git directory. However, this fails, because it also tries to remove the temporary directory, which e.g. contains the script file that it is currently executing, like: ./.4cba202f/script.sh Since the directory is arbitrary, there is no easy workaround. I think a gradual improvement would be to give the directory at least a consistent name or prefix, like .jenkins-tmp/4cba202f/... . This way it is possible to stay away from it.
            Hide
            jglick Jesse Glick added a comment -

            Jens Wilke like this?

            Show
            jglick Jesse Glick added a comment - Jens Wilke like this?
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 11 (Web Link)" [ 13534 ]
            Hide
            cruftex Jens Wilke added a comment -

            The PR changes the tmp directory to something like .jenkins-4cba202f.

            I think having a separate directory like .jenkins-tmp is not quite easy since it has the problem of "when is the last user of the directory gone, so we can delete it?", correct? This leads automatically to the more complex solution of the tmp-space problem....

            The PR is good for me. However, I would suggest to make a longer more descriptive prefix, e.g. .jenkins-tmp-workflow-step-sh-.... So whenever somebody steps on a directory like this, it is obvious who is to blame

            Show
            cruftex Jens Wilke added a comment - The PR changes the tmp directory to something like .jenkins-4cba202f . I think having a separate directory like .jenkins-tmp is not quite easy since it has the problem of "when is the last user of the directory gone, so we can delete it?", correct? This leads automatically to the more complex solution of the tmp-space problem.... The PR is good for me. However, I would suggest to make a longer more descriptive prefix, e.g. .jenkins-tmp-workflow-step-sh-... . So whenever somebody steps on a directory like this, it is obvious who is to blame
            Hide
            cruftex Jens Wilke added a comment -

            BTW, just for some amusement, my current solution to cleanup the workspace, looks like this:

            sh 't=`dirname $0`; t=`basename $t`; find -mindepth 1 -maxdepth 1 \\! -name .git \\! -name $t -exec rm -rf {} +'
            
            Show
            cruftex Jens Wilke added a comment - BTW, just for some amusement, my current solution to cleanup the workspace, looks like this: sh 't=`dirname $0`; t=`basename $t`; find -mindepth 1 -maxdepth 1 \\! -name .git \\! -name $t -exec rm -rf {} +'
            Hide
            jglick Jesse Glick added a comment -

            since it has the problem of "when is the last user of the directory gone, so we can delete it?", correct?

            Right, that is why I did not try that.

            BTW as a workaround you might try

            dir('.git') {sh 'cd ..; find -mindepth 1 -maxdepth 1 \\! -name .git -exec rm {} +'}
            
            Show
            jglick Jesse Glick added a comment - since it has the problem of "when is the last user of the directory gone, so we can delete it?", correct? Right, that is why I did not try that. BTW as a workaround you might try dir( '.git' ) {sh 'cd ..; find -mindepth 1 -maxdepth 1 \\! -name .git -exec rm {} +' }
            Hide
            deepchip Martin d'Anjou added a comment -

            I just ran into this problem (use case below), and I plan to workaround the problem with this line in .gitignore:

            .[a-z0-9][a-z0-9]*/**
            

            My use case is with the Gradle axion-release plugin. The plugin has a verifyRelease task that prevents the release from occurring if there are untracked files in the git repo. Since I run gradle inside an sh step, the temporary files cause the gradle task to fail, and the release fails:

            [Workflow] sh
            [my-workflow] Running shell script
            + ./gradlew verifyRelease
            :verifyRelease
            Looking for uncommitted changes.. FAILED
            
            Staged changes:
            
            Unstaged changes:
                added: .29913d6a/script.sh
                added: .29913d6a/jenkins-log.txt
                added: .29913d6a/pid
            
            :verifyRelease FAILED
            
            ...
            [Workflow] End of Workflow
            ERROR: script returned exit code 1
            Finished: FAILURE
            
            Show
            deepchip Martin d'Anjou added a comment - I just ran into this problem (use case below), and I plan to workaround the problem with this line in .gitignore : .[a-z0-9][a-z0-9]*/** My use case is with the Gradle axion-release plugin. The plugin has a verifyRelease task that prevents the release from occurring if there are untracked files in the git repo. Since I run gradle inside an sh step, the temporary files cause the gradle task to fail, and the release fails: [Workflow] sh [my-workflow] Running shell script + ./gradlew verifyRelease :verifyRelease Looking for uncommitted changes.. FAILED Staged changes: Unstaged changes: added: .29913d6a/script.sh added: .29913d6a/jenkins-log.txt added: .29913d6a/pid :verifyRelease FAILED ... [Workflow] End of Workflow ERROR: script returned exit code 1 Finished: FAILURE
            Hide
            cobexer Ing. Christoph Obexer added a comment -

            Any news on this?
            I need to use git clean -fdx to have reliable builds and fast turnaround times (avoiding a full clone / checkout every time).
            AFAIU the proposed PR would not yet fix this?

            Show
            cobexer Ing. Christoph Obexer added a comment - Any news on this? I need to use git clean -fdx to have reliable builds and fast turnaround times (avoiding a full clone / checkout every time). AFAIU the proposed PR would not yet fix this?
            Hide
            jglick Jesse Glick added a comment -

            Ing. Christoph Obexer no, the proposed PR would not handle the case of git clean -fdx calls. On the other hand that case can be handled by passing the appropriate extension to GitSCM rather than running this Git command manually.

            Show
            jglick Jesse Glick added a comment - Ing. Christoph Obexer no, the proposed PR would not handle the case of git clean -fdx calls. On the other hand that case can be handled by passing the appropriate extension to GitSCM rather than running this Git command manually.
            Hide
            cobexer Ing. Christoph Obexer added a comment -

            Jesse Glick you mean I could pass extensions: [[$class: 'CleanBeforeCheckout']] to checkout scm somehow? I tried those:

            node { bq. scm.extensions.add([$class: 'CleanBeforeCheckout']) bq. scm.extensions.add(new CleanBeforeCheckout()) bq. def myscm = scm; bq. myscm.extensions + [new CleanBeforeCheckout()] bq. checkout scm bq. }

            none of them fails(after allowing the field access, ...), but none of them works either. I do that in a multibranch workflow btw, so I need to use checkout scm AFAIK. scm.extensions always only contains one item and never the one I add to the list.

            Any Ideas?

            Show
            cobexer Ing. Christoph Obexer added a comment - Jesse Glick you mean I could pass extensions: [ [$class: 'CleanBeforeCheckout'] ] to checkout scm somehow? I tried those: node { bq. scm.extensions.add([$class: 'CleanBeforeCheckout']) bq. scm.extensions.add(new CleanBeforeCheckout()) bq. def myscm = scm; bq. myscm.extensions + [new CleanBeforeCheckout()] bq. checkout scm bq. } none of them fails(after allowing the field access, ...), but none of them works either. I do that in a multibranch workflow btw, so I need to use checkout scm AFAIK. scm.extensions always only contains one item and never the one I add to the list. Any Ideas?
            Hide
            jglick Jesse Glick added a comment -

            Not sure about usage from workflow-multibranch. Really AbstractGitSCMSource needs to be able to configure extensions. In principle

            scm.extensions.add(new hudson.plugins.git.extensions.impl.CleanBeforeCheckout())
            checkout scm
            

            ought to work but I have not tested it.

            Show
            jglick Jesse Glick added a comment - Not sure about usage from workflow-multibranch . Really AbstractGitSCMSource needs to be able to configure extensions. In principle scm.extensions.add( new hudson.plugins.git.extensions.impl.CleanBeforeCheckout()) checkout scm ought to work but I have not tested it.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java
            http://jenkins-ci.org/commit/durable-task-plugin/79990a9f6b5e7daf731bc46f535cd09bb8c2cff3
            Log:
            JENKINS-27152 Ameliorate problem by at least using a predictable control directory name.
            This makes it easy to add to .gitignore and the like.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java http://jenkins-ci.org/commit/durable-task-plugin/79990a9f6b5e7daf731bc46f535cd09bb8c2cff3 Log: JENKINS-27152 Ameliorate problem by at least using a predictable control directory name. This makes it easy to add to .gitignore and the like.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java
            http://jenkins-ci.org/commit/durable-task-plugin/d652a99ee2820c33d84e366069a9b14e3a0e2ca3
            Log:
            Merge pull request #11 from jglick/control-dir-prefix-JENKINS-27152

            JENKINS-27152 Ameliorate problem by using a predictable control directory name

            Compare: https://github.com/jenkinsci/durable-task-plugin/compare/0a836c1cec26...d652a99ee282

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java http://jenkins-ci.org/commit/durable-task-plugin/d652a99ee2820c33d84e366069a9b14e3a0e2ca3 Log: Merge pull request #11 from jglick/control-dir-prefix- JENKINS-27152 JENKINS-27152 Ameliorate problem by using a predictable control directory name Compare: https://github.com/jenkinsci/durable-task-plugin/compare/0a836c1cec26...d652a99ee282
            Hide
            cobexer Ing. Christoph Obexer added a comment -

            Jesse Glick scm.extensions can't be modified apparently because somewhere inside the list is actually a hudson.util.CopyOnWriteList...
            trying scm.extensions.data = scm.extensions.data << new hudson.plugins.git.extensions.impl.CleanBeforeCheckout() causes this exception:
            {{Caused by: org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: unclassified method hudson.util.CopyOnWriteList leftShift hudson.plugins.git.extensions.impl.CleanBeforeCheckout
            at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:73)
            at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149)
            at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146)
            at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:15)
            at WorkflowScript.run(WorkflowScript:40)}}
            which can't be approved...

            I'll create a separate issue for the scm.extensions problem to not spam this issue any longer

            Show
            cobexer Ing. Christoph Obexer added a comment - Jesse Glick scm.extensions can't be modified apparently because somewhere inside the list is actually a hudson.util.CopyOnWriteList... trying scm.extensions.data = scm.extensions.data << new hudson.plugins.git.extensions.impl.CleanBeforeCheckout() causes this exception: {{Caused by: org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: unclassified method hudson.util.CopyOnWriteList leftShift hudson.plugins.git.extensions.impl.CleanBeforeCheckout at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:73) at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:149) at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:146) at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:15) at WorkflowScript.run(WorkflowScript:40)}} which can't be approved... I'll create a separate issue for the scm.extensions problem to not spam this issue any longer
            Hide
            jglick Jesse Glick added a comment -

            Ing. Christoph Obexer well CopyOnWriteList.add just replaces its internal array list, but that should be transparent. So something else is amiss.

            Show
            jglick Jesse Glick added a comment - Ing. Christoph Obexer well CopyOnWriteList.add just replaces its internal array list, but that should be transparent. So something else is amiss.
            Hide
            jglick Jesse Glick added a comment -

            Without introducing a new explicit API, it would probably suffice to have every plugin consent to keep temporary files inside the slave FS root, with that whole root being mounted by Image.inside.

            Show
            jglick Jesse Glick added a comment - Without introducing a new explicit API, it would probably suffice to have every plugin consent to keep temporary files inside the slave FS root, with that whole root being mounted by Image.inside .
            cleclerc Cyrille Le Clerc made changes -
            Link This issue is blocking JENKINS-32624 [ JENKINS-32624 ]
            cleclerc Cyrille Le Clerc made changes -
            Link This issue is blocking JENKINS-32943 [ JENKINS-32943 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            jglick Jesse Glick added a comment -

            Working on a series of patches to use …/workspace/folder/job@tmp as the root of temporary files for a workspace …/workspace/folder/job, and making docker-workflow mount that location. Seems to be working so far, though I had to improve the fix of JENKINS-25678. Remaining to be done: CliGitAPIImpl; FileBinding (JENKINS-32943); AgentServer (JENKINS-32624).

            Show
            jglick Jesse Glick added a comment - Working on a series of patches to use …/workspace/folder/job@tmp as the root of temporary files for a workspace …/workspace/folder/job , and making docker-workflow mount that location. Seems to be working so far, though I had to improve the fix of JENKINS-25678 . Remaining to be done: CliGitAPIImpl ; FileBinding ( JENKINS-32943 ); AgentServer ( JENKINS-32624 ).
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-32624 [ JENKINS-32624 ]
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-32624 [ JENKINS-32624 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-32943 [ JENKINS-32943 ]
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-32943 [ JENKINS-32943 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 19 (Web Link)" [ 13907 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "docker-workflow PR 33 (Web Link)" [ 13908 ]
            rtyler R. Tyler Croy made changes -
            Link This issue is related to JENKINS-32624 [ JENKINS-32624 ]
            rtyler R. Tyler Croy made changes -
            Link This issue is related to JENKINS-32624 [ JENKINS-32624 ]
            Hide
            cleclerc Cyrille Le Clerc added a comment - - edited

            Some use cases:

            (?)Linux user home

            Maybe interesting to locate the linux user.home under this workspace working folder. /!\ It may break many things if a plugin does this without 'collaborating' with other plugins but it could be a "job config" or a "jenkins global config".

            credentials-binding-plugin

            ssh-agent

            config-file-provider

            Maven Pipeline Integration

            • Would need to create a tmp folder to prepend to the PATH in which we would create an ephemeral "mvn" file that would inject the proper command line parameters that cannot be passed as environment variables (--settings, --global-settings, --debug, --errors, --batch-mode...). See Sonatype >>6.1. Maven Command Line Options
            • it would be interesting to optionally specify a local-repo under this specific folder (see -Dmaven.repo.local=...)

            Gradle Pipeline Step

            Show
            cleclerc Cyrille Le Clerc added a comment - - edited Some use cases: (?)Linux user home Maybe interesting to locate the linux user.home under this workspace working folder. /!\ It may break many things if a plugin does this without 'collaborating' with other plugins but it could be a "job config" or a "jenkins global config". .ssh/config can only be defined under $HOME . Are there some config that we would like to define "per job"? See http://www.cyberciti.biz/faq/create-ssh-config-file-on-linux-unix/ git: most git configuration can be passed through environment variables and thus it is not required to specify a per workspace setup. Moreover, the PREFIX  environment variable allow to define the "system wide" $PREFIX/etc/gitconfig https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables credentials-binding-plugin Use a "secrets" folder to store file based credentials: see FileBinding.secretsDir(workspace) ssh-agent Creates a tmp file for the local socket in java.io.tmp: AgentServer.createLocalSocketAddress() File socket = File.createTempFile("jenkins", ".jnr"); config-file-provider Creates a tmp folder to store config files ManagedFileUtil. createTempFile() Maven Pipeline Integration Would need to create a tmp folder to prepend to the PATH in which we would create an ephemeral "mvn" file that would inject the proper command line parameters that cannot be passed as environment variables ( --settings , --global-settings , --debug , --errors , --batch-mode ...). See Sonatype >>6.1. Maven Command Line Options it would be interesting to optionally specify a local-repo under this specific folder (see -Dmaven.repo.local=... ) Gradle Pipeline Step Maybe interesting to specify a custom GRADLE_USER_HOME , see Gradle >> Chapter 11. The Build Environment maybe interesting to have a per workspace local.repo
            Hide
            jglick Jesse Glick added a comment -

            Maybe interesting to locate the Linux user.home under this workspace working folder.

            $HOME that is. Seems too dangerous, and anyway users may reasonably be expecting configuration files in ~slaveacct/ to be honored.

            .ssh/config can only be defined under $HOME

            If true, this is a limitation of SSH we would need to work around, if any plugin wishes to modify that configuration. According to man ssh, this is not true:

            -F configfile
            Specifies an alternative per-user configuration file. If a configuration file is given on the command line, the system-wide configuration file (/etc/ssh/ssh_config) will be ignored. The default for the per-user configuration file is ~/.ssh/config.

            BTW Docker used to have such a mandated location for registry credentials, but as of newer versions of the client this is no longer true.

            Show
            jglick Jesse Glick added a comment - Maybe interesting to locate the Linux user.home under this workspace working folder. $HOME that is. Seems too dangerous, and anyway users may reasonably be expecting configuration files in ~slaveacct/ to be honored. .ssh/config can only be defined under $HOME If true, this is a limitation of SSH we would need to work around, if any plugin wishes to modify that configuration. According to man ssh , this is not true: -F configfile Specifies an alternative per-user configuration file. If a configuration file is given on the command line, the system-wide configuration file ( /etc/ssh/ssh_config ) will be ignored. The default for the per-user configuration file is ~/.ssh/config . BTW Docker used to have such a mandated location for registry credentials, but as of newer versions of the client this is no longer true.
            Hide
            jglick Jesse Glick added a comment -

            Plan to file a core PR to not only introduce a formal API for this temporary location (currently planned as a helper method in WorkspaceList), but also to consider it in some places, like workspace cleanup.

            Show
            jglick Jesse Glick added a comment - Plan to file a core PR to not only introduce a formal API for this temporary location (currently planned as a helper method in WorkspaceList ), but also to consider it in some places, like workspace cleanup.
            Hide
            jglick Jesse Glick added a comment -

            Should also have a Pipeline step akin to pwd but for this temporary location, to make it easy to add other temporary files with custom meanings. Maybe even File.createTempFile semantics as an option.

            Show
            jglick Jesse Glick added a comment - Should also have a Pipeline step akin to pwd but for this temporary location, to make it easy to add other temporary files with custom meanings. Maybe even File.createTempFile semantics as an option.
            jglick Jesse Glick made changes -
            Remote Link This issue links to "core PR 2066 (Web Link)" [ 14003 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "credentials-binding PR 13 (Web Link)" [ 14004 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "config-file-provider PR 13 (Web Link)" [ 14005 ]
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            As you mention java.io.File#createTempFile, java.io.tmpdir seem to be a "static" value defined at startup time (java.io.File.TempDirectory#tmpdir). Due to this "static" behavior, it does not seem possible to have a job specific java.io.tmpdir, located under job@tmp.

            Could we mount the java.io.tmpdir in docker-pipeline containers to ensure that temporary files created with java.io.File#createTempFile are visible in docker-pipeline steps? Shall we discuss this in another Jira issue?

            Show
            cleclerc Cyrille Le Clerc added a comment - As you mention java.io.File#createTempFile , java.io.tmpdir seem to be a "static" value defined at startup time ( java.io.File.TempDirectory#tmpdir ). Due to this "static" behavior, it does not seem possible to have a job specific java.io.tmpdir , located under job@tmp . Could we mount the java.io.tmpdir in docker-pipeline containers to ensure that temporary files created with java.io.File#createTempFile are visible in docker-pipeline steps? Shall we discuss this in another Jira issue?
            Hide
            jglick Jesse Glick added a comment -

            CliGitAPIImpl is a special case. It ignores the Launcher provided to SCM.checkout, and thus works unmodified even inside withContainerStep: the command is run outside the container, directly by the agent. It ought to be adjusted to use the official temporary directory, but it is not strictly necessary.

            Show
            jglick Jesse Glick added a comment - CliGitAPIImpl is a special case. It ignores the Launcher provided to SCM.checkout , and thus works unmodified even inside withContainerStep : the command is run outside the container, directly by the agent. It ought to be adjusted to use the official temporary directory, but it is not strictly necessary.
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            Jesse Glick by "official temporary directory", do you mean java.io.tmpdir and java.io.File#createTempFile or do you mean PR #33 workspace@tmp?

            Show
            cleclerc Cyrille Le Clerc added a comment - Jesse Glick by "official temporary directory", do you mean java.io.tmpdir and java.io.File#createTempFile or do you mean PR #33 workspace@tmp ?
            jglick Jesse Glick made changes -
            Remote Link This issue links to "ssh-agent PR 11 (Web Link)" [ 14011 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "acceptance test (Web Link)" [ 14012 ]
            Hide
            jglick Jesse Glick added a comment -

            Cyrille Le Clerc the latter.

            I think everything is now written with the exception of the proposed Pipeline step.

            Show
            jglick Jesse Glick added a comment - Cyrille Le Clerc the latter. I think everything is now written with the exception of the proposed Pipeline step.
            Hide
            cleclerc Cyrille Le Clerc added a comment - - edited

            > Cyrille Le Clerc the latter.

            To clarify, this mean that java.io.tmpdir will not be mounted by the docker-pipeline plugin in Docker containers launched by docker.image('my-image').inside{...}, and that files created in plugin through java.io.File#createTempFile() will not be visible in docker containers. Correct?

            > I think everything is now written

            Do you need assistance to test all the plugins with their pull request (ssh-agent, config-file-provider, credentials-binding)?

            > with the exception of the proposed Pipeline step.

            I'm not sure I understand which plugin you are referring to? CliGitAPIImpl?

            Show
            cleclerc Cyrille Le Clerc added a comment - - edited > Cyrille Le Clerc the latter. To clarify, this mean that java.io.tmpdir will not be mounted by the docker-pipeline plugin in Docker containers launched by docker.image('my-image').inside{... }, and that files created in plugin through java.io.File#createTempFile() will not be visible in docker containers. Correct? > I think everything is now written Do you need assistance to test all the plugins with their pull request (ssh-agent, config-file-provider, credentials-binding)? > with the exception of the proposed Pipeline step. I'm not sure I understand which plugin you are referring to? CliGitAPIImpl ?
            Hide
            jglick Jesse Glick added a comment -

            To clarify, […] Correct?

            Correct.

            Do you need assistance to test all the plugins

            Well everything has automated tests, but of course extra exploratory testing would be welcome.

            I'm not sure I understand which plugin you are referring to?

            PR coming shortly.

            Show
            jglick Jesse Glick added a comment - To clarify, […] Correct? Correct. Do you need assistance to test all the plugins Well everything has automated tests, but of course extra exploratory testing would be welcome. I'm not sure I understand which plugin you are referring to? PR coming shortly.
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-plugin PR 350 (Web Link)" [ 14016 ]
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            I tested the PR for the config-file-provider. I won't have the time for the other tests (credentials-binding...)

            Show
            cleclerc Cyrille Le Clerc added a comment - I tested the PR for the config-file-provider. I won't have the time for the other tests (credentials-binding...)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/model/WorkspaceCleanupThread.java
            core/src/main/java/hudson/slaves/WorkspaceList.java
            test/src/test/java/hudson/model/WorkspaceCleanupThreadTest.java
            http://jenkins-ci.org/commit/jenkins/307bfc17385c14aefcb623ccc3e7144e4f01e2a3
            Log:
            JENKINS-27152 Introduce common API WorkspaceList.tempDir.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/WorkspaceCleanupThread.java core/src/main/java/hudson/slaves/WorkspaceList.java test/src/test/java/hudson/model/WorkspaceCleanupThreadTest.java http://jenkins-ci.org/commit/jenkins/307bfc17385c14aefcb623ccc3e7144e4f01e2a3 Log: JENKINS-27152 Introduce common API WorkspaceList.tempDir.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/model/WorkspaceCleanupThread.java
            core/src/main/java/hudson/slaves/WorkspaceList.java
            test/src/test/java/hudson/model/WorkspaceCleanupThreadTest.java
            http://jenkins-ci.org/commit/jenkins/f187d706e5534b1c34fac6186b719678e43f6403
            Log:
            JENKINS-27152 Merging #2066.

            Compare: https://github.com/jenkinsci/jenkins/compare/48e42ae94a69...f187d706e553

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/WorkspaceCleanupThread.java core/src/main/java/hudson/slaves/WorkspaceList.java test/src/test/java/hudson/model/WorkspaceCleanupThreadTest.java http://jenkins-ci.org/commit/jenkins/f187d706e5534b1c34fac6186b719678e43f6403 Log: JENKINS-27152 Merging #2066. Compare: https://github.com/jenkinsci/jenkins/compare/48e42ae94a69...f187d706e553
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java
            src/test/java/org/jenkinsci/plugins/durabletask/BourneShellScriptTest.java
            http://jenkins-ci.org/commit/durable-task-plugin/85be167e1efbd0ef2e813f197c3ab7ba5031cbca
            Log:
            [FIXED JENKINS-27152] Keep control files outside of the workspace.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java src/test/java/org/jenkinsci/plugins/durabletask/BourneShellScriptTest.java http://jenkins-ci.org/commit/durable-task-plugin/85be167e1efbd0ef2e813f197c3ab7ba5031cbca Log: [FIXED JENKINS-27152] Keep control files outside of the workspace.
            scm_issue_link SCM/JIRA link daemon made changes -
            Status In Progress [ 3 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java
            src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java
            src/main/java/org/jenkinsci/plugins/durabletask/WindowsBatchScript.java
            src/test/java/org/jenkinsci/plugins/durabletask/BourneShellScriptTest.java
            http://jenkins-ci.org/commit/durable-task-plugin/d66df29a6b0a56c0f0a221f8480e130d5bd751fb
            Log:
            Merge pull request #19 from jglick/temp-dir-JENKINS-27152

            JENKINS-27152 Store control files outside of the workspace

            Compare: https://github.com/jenkinsci/durable-task-plugin/compare/92014f174b60...d66df29a6b0a

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/durabletask/BourneShellScript.java src/main/java/org/jenkinsci/plugins/durabletask/FileMonitoringTask.java src/main/java/org/jenkinsci/plugins/durabletask/WindowsBatchScript.java src/test/java/org/jenkinsci/plugins/durabletask/BourneShellScriptTest.java http://jenkins-ci.org/commit/durable-task-plugin/d66df29a6b0a56c0f0a221f8480e130d5bd751fb Log: Merge pull request #19 from jglick/temp-dir- JENKINS-27152 JENKINS-27152 Store control files outside of the workspace Compare: https://github.com/jenkinsci/durable-task-plugin/compare/92014f174b60...d66df29a6b0a
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgentFactory.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/AgentServer.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgent.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentFactory.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentStarter.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/mina/MinaRemoteAgentFactory.java
            http://jenkins-ci.org/commit/ssh-agent-plugin/d2e0f53b6eda56012c2bd5d9bc553b31667462a2
            Log:
            JENKINS-32624 JENKINS-27152 Use a standardized temporary directory when possible.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgentFactory.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/AgentServer.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgent.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentFactory.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentStarter.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/mina/MinaRemoteAgentFactory.java http://jenkins-ci.org/commit/ssh-agent-plugin/d2e0f53b6eda56012c2bd5d9bc553b31667462a2 Log: JENKINS-32624 JENKINS-27152 Use a standardized temporary directory when possible.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/findbugs/excludesFilter.xml
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgentFactory.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/AgentServer.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgent.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentFactory.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentStarter.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/mina/MinaRemoteAgentFactory.java
            src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepWorkflowTest.java
            http://jenkins-ci.org/commit/ssh-agent-plugin/f96025a616f9e5407befea51d76619ec5b266026
            Log:
            Merge pull request #11 from jglick/temp-dir-JENKINS-27152

            JENKINS-27152 Use a standardized directory for $SSH_AUTH_SOCK

            Compare: https://github.com/jenkinsci/ssh-agent-plugin/compare/12f6ff0ccaee...f96025a616f9

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/findbugs/excludesFilter.xml src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgentFactory.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/AgentServer.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgent.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentFactory.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentStarter.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/mina/MinaRemoteAgentFactory.java src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepWorkflowTest.java http://jenkins-ci.org/commit/ssh-agent-plugin/f96025a616f9e5407befea51d76619ec5b266026 Log: Merge pull request #11 from jglick/temp-dir- JENKINS-27152 JENKINS-27152 Use a standardized directory for $SSH_AUTH_SOCK Compare: https://github.com/jenkinsci/ssh-agent-plugin/compare/12f6ff0ccaee...f96025a616f9
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            CHANGES.md
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java
            basic-steps/src/main/java/org/jenkinsci/plugins/workflow/steps/PwdStep.java
            basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/config.jelly
            basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/help-tmp.html
            http://jenkins-ci.org/commit/workflow-plugin/8c74333321f007e829b2a9581af643b792ef47a6
            Log:
            JENKINS-27152 Option for pwd step to return temp directory.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: CHANGES.md aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java basic-steps/src/main/java/org/jenkinsci/plugins/workflow/steps/PwdStep.java basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/config.jelly basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/help-tmp.html http://jenkins-ci.org/commit/workflow-plugin/8c74333321f007e829b2a9581af643b792ef47a6 Log: JENKINS-27152 Option for pwd step to return temp directory.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            CHANGES.md
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java
            basic-steps/src/main/java/org/jenkinsci/plugins/workflow/steps/PwdStep.java
            basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/config.jelly
            basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/help-tmp.html
            http://jenkins-ci.org/commit/workflow-plugin/7bbcab0390b6d97dc40d4dbaf8dde34ef2c5af0c
            Log:
            Merge pull request #350 from jglick/temp-dir-JENKINS-27152

            JENKINS-27152 Option for pwd step to return temp directory

            Compare: https://github.com/jenkinsci/workflow-plugin/compare/7d2004ea4824...7bbcab0390b6

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: CHANGES.md aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java basic-steps/src/main/java/org/jenkinsci/plugins/workflow/steps/PwdStep.java basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/config.jelly basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/help-tmp.html http://jenkins-ci.org/commit/workflow-plugin/7bbcab0390b6d97dc40d4dbaf8dde34ef2c5af0c Log: Merge pull request #350 from jglick/temp-dir- JENKINS-27152 JENKINS-27152 Option for pwd step to return temp directory Compare: https://github.com/jenkinsci/workflow-plugin/compare/7d2004ea4824...7bbcab0390b6
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #4478
            JENKINS-27152 Introduce common API WorkspaceList.tempDir. (Revision 307bfc17385c14aefcb623ccc3e7144e4f01e2a3)

            Result = SUCCESS
            jesse glick : 307bfc17385c14aefcb623ccc3e7144e4f01e2a3
            Files :

            • core/src/main/java/hudson/model/WorkspaceCleanupThread.java
            • test/src/test/java/hudson/model/WorkspaceCleanupThreadTest.java
            • core/src/main/java/hudson/slaves/WorkspaceList.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #4478 JENKINS-27152 Introduce common API WorkspaceList.tempDir. (Revision 307bfc17385c14aefcb623ccc3e7144e4f01e2a3) Result = SUCCESS jesse glick : 307bfc17385c14aefcb623ccc3e7144e4f01e2a3 Files : core/src/main/java/hudson/model/WorkspaceCleanupThread.java test/src/test/java/hudson/model/WorkspaceCleanupThreadTest.java core/src/main/java/hudson/slaves/WorkspaceList.java
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java
            http://jenkins-ci.org/commit/docker-workflow-plugin/8832083218f687d9d04ba238c99bb1f9a048cc06
            Log:
            JENKINS-27152 Expect to mount workspace@tmp directory.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java http://jenkins-ci.org/commit/docker-workflow-plugin/8832083218f687d9d04ba238c99bb1f9a048cc06 Log: JENKINS-27152 Expect to mount workspace@tmp directory.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/docker/workflow/AbstractEndpointStepExecution.java
            src/main/java/org/jenkinsci/plugins/docker/workflow/DockerDSL.java
            src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java
            src/test/java/org/jenkinsci/plugins/docker/workflow/DockerDSLTest.java
            src/test/java/org/jenkinsci/plugins/docker/workflow/RegistryEndpointStepTest.java
            src/test/java/org/jenkinsci/plugins/docker/workflow/ServerEndpointStepTest.java
            src/test/java/org/jenkinsci/plugins/docker/workflow/WithContainerStepTest.java
            http://jenkins-ci.org/commit/docker-workflow-plugin/2312931df9030b814fed6522e931762f1a93e433
            Log:
            Merge pull request #33 from jglick/temp-dir-JENKINS-27152

            JENKINS-27152 Mount workspace@tmp

            Compare: https://github.com/jenkinsci/docker-workflow-plugin/compare/b60bb77ccc51...2312931df903

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/docker/workflow/AbstractEndpointStepExecution.java src/main/java/org/jenkinsci/plugins/docker/workflow/DockerDSL.java src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java src/test/java/org/jenkinsci/plugins/docker/workflow/DockerDSLTest.java src/test/java/org/jenkinsci/plugins/docker/workflow/RegistryEndpointStepTest.java src/test/java/org/jenkinsci/plugins/docker/workflow/ServerEndpointStepTest.java src/test/java/org/jenkinsci/plugins/docker/workflow/WithContainerStepTest.java http://jenkins-ci.org/commit/docker-workflow-plugin/2312931df9030b814fed6522e931762f1a93e433 Log: Merge pull request #33 from jglick/temp-dir- JENKINS-27152 JENKINS-27152 Mount workspace@tmp Compare: https://github.com/jenkinsci/docker-workflow-plugin/compare/b60bb77ccc51...2312931df903
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/test/java/plugins/WorkflowPluginTest.java
            http://jenkins-ci.org/commit/acceptance-test-harness/ef45669766be39c4b1d203ef5350a47873e5e342
            Log:
            JENKINS-27152 Integration test for Docker Pipeline, Git, and SSH Agent plugins.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/test/java/plugins/WorkflowPluginTest.java http://jenkins-ci.org/commit/acceptance-test-harness/ef45669766be39c4b1d203ef5350a47873e5e342 Log: JENKINS-27152 Integration test for Docker Pipeline, Git, and SSH Agent plugins.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/org/jenkinsci/test/acceptance/docker/fixtures/GitContainer.java
            src/main/java/org/jenkinsci/test/acceptance/junit/WithCredentials.java
            src/main/java/org/jenkinsci/test/acceptance/plugins/credentials/BaseStandardCredentials.java
            src/main/java/org/jenkinsci/test/acceptance/plugins/credentials/Credential.java
            src/main/java/org/jenkinsci/test/acceptance/plugins/credentials/UserPwdCredential.java
            src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_credentials/SshPrivateKeyCredential.java
            src/test/java/plugins/GitPluginTest.java
            src/test/java/plugins/WorkflowPluginTest.java
            http://jenkins-ci.org/commit/acceptance-test-harness/6681f400d9a84ded4c39e8a8c4c982c1a2196295
            Log:
            Merge pull request #81 from jglick/temp-dir-JENKINS-27152

            JENKINS-27152 Test of standardized temp directory with Git operations

            Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/500a376b7556...6681f400d9a8

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/test/acceptance/docker/fixtures/GitContainer.java src/main/java/org/jenkinsci/test/acceptance/junit/WithCredentials.java src/main/java/org/jenkinsci/test/acceptance/plugins/credentials/BaseStandardCredentials.java src/main/java/org/jenkinsci/test/acceptance/plugins/credentials/Credential.java src/main/java/org/jenkinsci/test/acceptance/plugins/credentials/UserPwdCredential.java src/main/java/org/jenkinsci/test/acceptance/plugins/ssh_credentials/SshPrivateKeyCredential.java src/test/java/plugins/GitPluginTest.java src/test/java/plugins/WorkflowPluginTest.java http://jenkins-ci.org/commit/acceptance-test-harness/6681f400d9a84ded4c39e8a8c4c982c1a2196295 Log: Merge pull request #81 from jglick/temp-dir- JENKINS-27152 JENKINS-27152 Test of standardized temp directory with Git operations Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/500a376b7556...6681f400d9a8
            Hide
            cobexer Ing. Christoph Obexer added a comment -

            Updating to 1.8 my pipeline script:
            ```groovy
            node('server') {
            dir ('.git')

            { bat 'cd .. && git clean -fdx' }

            }
            ```
            Caused the same problem differently:
            warning: failed to remove .git@tmp/durable-263b366c/jenkins-log.txt
            The batch file cannot be found.

            Apparently the control files are NOT stored outside the workspace but only in the parent directory next to the cwd as $CWD@tmp.
            Wouldn't it be simpler to store these files in $SLAVE_HOME/tmp/<job name>/...? aka a parallel hierarchy next to the workspace directory?

            Show
            cobexer Ing. Christoph Obexer added a comment - Updating to 1.8 my pipeline script: ```groovy node('server') { dir ('.git') { bat 'cd .. && git clean -fdx' } } ``` Caused the same problem differently: warning: failed to remove .git@tmp/durable-263b366c/jenkins-log.txt The batch file cannot be found. Apparently the control files are NOT stored outside the workspace but only in the parent directory next to the cwd as $CWD@tmp. Wouldn't it be simpler to store these files in $SLAVE_HOME/tmp/<job name>/...? aka a parallel hierarchy next to the workspace directory?
            Hide
            jglick Jesse Glick added a comment -

            Ing. Christoph Obexer I do not understand the purpose of that script. Try simply

            node('server') {
              bat 'git clean -fdx'
            }
            

            Wouldn't it be simpler to store these files in …

            There are various future options for choice of the temporary directory but until the core API gets into a reasonable LTS version, and the plugins needing to select a temporary directory can use it, they all need to have identical copies of some simple logic, for which I chose ../$(basename)@tmp as a one-liner that works well enough in most cases. Your proposal is similar to one of the options already discussed, but it is rather more complicated than you think, since you need to consider

            • temporary slave outages which result in Slave.getRootPath being null
            • alternate workspaces like jobname@2 created by concurrent builds
            • people who specify custom workspace directories
            • Unix slave FS roots exceeding roughly 108 characters (you really do not want to know)
            • obscure Windows restrictions yet to be discovered

            All fixable in one place with time, but I certainly do not want to be applying the same fix to six or seven places, especially when a mismatch between docker-workflow and one of the others will cause a Dockerized build to break.

            Show
            jglick Jesse Glick added a comment - Ing. Christoph Obexer I do not understand the purpose of that script. Try simply node( 'server' ) { bat 'git clean -fdx' } Wouldn't it be simpler to store these files in … There are various future options for choice of the temporary directory but until the core API gets into a reasonable LTS version, and the plugins needing to select a temporary directory can use it, they all need to have identical copies of some simple logic, for which I chose ../$(basename)@tmp as a one-liner that works well enough in most cases. Your proposal is similar to one of the options already discussed, but it is rather more complicated than you think, since you need to consider temporary slave outages which result in Slave.getRootPath being null alternate workspaces like jobname@2 created by concurrent builds people who specify custom workspace directories Unix slave FS roots exceeding roughly 108 characters (you really do not want to know) obscure Windows restrictions yet to be discovered All fixable in one place with time, but I certainly do not want to be applying the same fix to six or seven places, especially when a mismatch between docker-workflow and one of the others will cause a Dockerized build to break.
            Hide
            cleclerc Cyrille Le Clerc added a comment - - edited

            Jesse Glick what is your recommendation to use this "workspace@tmp" in pipeline scripts to create temporary files?

            Here is a use case with a "tmpFile" that I would like to locate in "workspace@tmp".

            def tmpFile=".aws-ec2-instances-status.json" // should be located in 'workspace@tmp'
            sh "aws ec2 describe-instances ...  2>&1 | tee ${tmpFile}"
            def awsEc2StatusAsJson = readFile(tmpFile)
            ...
            

            It may be convenient to have a pipeline step "createTempFile()".

            Show
            cleclerc Cyrille Le Clerc added a comment - - edited Jesse Glick what is your recommendation to use this "workspace@tmp" in pipeline scripts to create temporary files? Here is a use case with a " tmpFile " that I would like to locate in "workspace@tmp". def tmpFile= ".aws-ec2-instances-status.json" // should be located in 'workspace@tmp' sh "aws ec2 describe-instances ... 2>&1 | tee ${tmpFile}" def awsEc2StatusAsJson = readFile(tmpFile) ... It may be convenient to have a pipeline step " createTempFile() ".
            Hide
            cobexer Ing. Christoph Obexer added a comment -

            Jesse Glick the bat 'cd .. && git clean -fdx' was required for previous versions where the control files were in the current working directory. The 1.8 update broke compatibility with that workaround because the control files moved from the CWD to the parent directory which moved them back to where they can't be. With version 1.8 the bat 'git clean -fdx' is what is needed now.

            This incompatibility means that the change to the Jenkinsfile needs to be back-ported to all affected versions before they can build a new build successfully, however doing so immediately would trigger maintenance builds unnecessarily (Multi-Branch Pipeline Job) - which makes this incompatibility annoying.

            And thanks for the long explanation!

            Show
            cobexer Ing. Christoph Obexer added a comment - Jesse Glick the bat 'cd .. && git clean -fdx' was required for previous versions where the control files were in the current working directory. The 1.8 update broke compatibility with that workaround because the control files moved from the CWD to the parent directory which moved them back to where they can't be. With version 1.8 the bat 'git clean -fdx' is what is needed now. This incompatibility means that the change to the Jenkinsfile needs to be back-ported to all affected versions before they can build a new build successfully, however doing so immediately would trigger maintenance builds unnecessarily (Multi-Branch Pipeline Job) - which makes this incompatibility annoying. And thanks for the long explanation!
            Hide
            jglick Jesse Glick added a comment -

            until the core API gets into a reasonable LTS version

            Sadly according to #jenkins-meeting it just missed making it into the next line (Daniel Beck & Kohsuke Kawaguchi chose 1.651), so will have to wait for a 2.0 dep I guess.

            It may be convenient to have a pipeline step "createTempFile()".

            pwd tmp: true
            

            This incompatibility means that the change to the Jenkinsfile needs to be back-ported to all affected versions before they can build a new build successfully, however doing so immediately would trigger maintenance builds unnecessarily […] which makes this incompatibility annoying.

            Sorry. Do not see an easy way around that.

            Show
            jglick Jesse Glick added a comment - until the core API gets into a reasonable LTS version Sadly according to #jenkins-meeting it just missed making it into the next line ( Daniel Beck & Kohsuke Kawaguchi chose 1.651), so will have to wait for a 2.0 dep I guess. It may be convenient to have a pipeline step " createTempFile() ". pwd tmp: true This incompatibility means that the change to the Jenkinsfile needs to be back-ported to all affected versions before they can build a new build successfully, however doing so immediately would trigger maintenance builds unnecessarily […] which makes this incompatibility annoying. Sorry. Do not see an easy way around that.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/configfiles/buildwrapper/ManagedFileUtil.java
            http://jenkins-ci.org/commit/config-file-provider-plugin/eb67346e042b09a535ca7e343a5e563706a19e79
            Log:
            JENKINS-27152 Use a temporary directory for managed files near the workspace.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/configfiles/buildwrapper/ManagedFileUtil.java http://jenkins-ci.org/commit/config-file-provider-plugin/eb67346e042b09a535ca7e343a5e563706a19e79 Log: JENKINS-27152 Use a temporary directory for managed files near the workspace.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java
            http://jenkins-ci.org/commit/workflow-support-plugin/88b7a89577178ad99f718b7e08d73750378f3d98
            Log:
            JENKINS-27152 Option for pwd step to return temp directory.
            Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java http://jenkins-ci.org/commit/workflow-support-plugin/88b7a89577178ad99f718b7e08d73750378f3d98 Log: JENKINS-27152 Option for pwd step to return temp directory. Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java
            basic-steps/src/main/java/org/jenkinsci/plugins/workflow/steps/PwdStep.java
            basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/config.jelly
            basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/help-tmp.html
            http://jenkins-ci.org/commit/workflow-basic-steps-plugin/ad40c7688bac0e9f3405d17eea0013800ecbe6d8
            Log:
            JENKINS-27152 Option for pwd step to return temp directory.
            Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java basic-steps/src/main/java/org/jenkinsci/plugins/workflow/steps/PwdStep.java basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/config.jelly basic-steps/src/main/resources/org/jenkinsci/plugins/workflow/steps/PwdStep/help-tmp.html http://jenkins-ci.org/commit/workflow-basic-steps-plugin/ad40c7688bac0e9f3405d17eea0013800ecbe6d8 Log: JENKINS-27152 Option for pwd step to return temp directory. Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java
            http://jenkins-ci.org/commit/pipeline-build-step-plugin/65490d1e880eafd7bd9f07e00b41ac6edd5e4cab
            Log:
            JENKINS-27152 Option for pwd step to return temp directory.
            Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java http://jenkins-ci.org/commit/pipeline-build-step-plugin/65490d1e880eafd7bd9f07e00b41ac6edd5e4cab Log: JENKINS-27152 Option for pwd step to return temp directory. Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java
            http://jenkins-ci.org/commit/pipeline-input-step-plugin/bc8f08b30e54645269b2ab4af07cddfd34b31e7a
            Log:
            JENKINS-27152 Option for pwd step to return temp directory.
            Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java http://jenkins-ci.org/commit/pipeline-input-step-plugin/bc8f08b30e54645269b2ab4af07cddfd34b31e7a Log: JENKINS-27152 Option for pwd step to return temp directory. Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java
            http://jenkins-ci.org/commit/pipeline-stage-step-plugin/d6dbe7480cab9cb96dd584c2066de164ba36c868
            Log:
            JENKINS-27152 Option for pwd step to return temp directory.
            Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java http://jenkins-ci.org/commit/pipeline-stage-step-plugin/d6dbe7480cab9cb96dd584c2066de164ba36c868 Log: JENKINS-27152 Option for pwd step to return temp directory. Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java
            http://jenkins-ci.org/commit/workflow-multibranch-plugin/7e4f7eff243617c192afebeb9914144435d30ecb
            Log:
            JENKINS-27152 Option for pwd step to return temp directory.
            Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java http://jenkins-ci.org/commit/workflow-multibranch-plugin/7e4f7eff243617c192afebeb9914144435d30ecb Log: JENKINS-27152 Option for pwd step to return temp directory. Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/db56cffc53abb11bb7d1ea869dac3c8055cb0458
            Log:
            JENKINS-27152 Option for pwd step to return temp directory.
            Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: aggregator/src/test/java/org/jenkinsci/plugins/workflow/steps/PwdStepTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/db56cffc53abb11bb7d1ea869dac3c8055cb0458 Log: JENKINS-27152 Option for pwd step to return temp directory. Originally-Committed-As: 8c74333321f007e829b2a9581af643b792ef47a6
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 161348 ] JNJira + In-Review [ 196728 ]
            abayer Andrew Bayer made changes -
            Labels api docker robustness workflow api docker pipeline robustness workflow
            abayer Andrew Bayer made changes -
            Labels api docker pipeline robustness workflow api docker pipeline robustness
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal OSS-343 (Web Link)" [ 18909 ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal CLTS-1188 (Web Link)" [ 19153 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                vlatombe Vincent Latombe
              • Votes:
                7 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: