Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32624

sshagent{} ignored when executed in docker.image().inside{...}

    Details

    • Similar Issues:

      Description

      sshagent('my-ssh-key'){...} is ignored in docker.image('cloudbees:java-build-tools').inside{...}. The same sshagent step works in 'standard' linux node.

      See attached logs

      sshagent(){...} ignored in docker.image(...).inside{...} 
      docker.image ('cloudbees/java-build-tools:0.0.7.1').inside {
          sh 'ls -al ~/.ssh/ || true'
          sshagent(['aws-cleclerc-ssh-key-ubuntu-cleclerc']) {
             sh 'echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK'
             sh 'ls -al $SSH_AUTH_SOCK || true'
             sh "ssh -vvv -o StrictHostKeyChecking=no ubuntu@docker-agent.beesshop.org uname -a"
          }
      }
      
      sshagent(){...} WORKS in node{...} 
      node {
          sh 'ls -al ~/.ssh/ || true'
          sshagent(['aws-cleclerc-ssh-key-ubuntu-cleclerc']) {
             sh 'echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK'
             sh 'ls -al $SSH_AUTH_SOCK || true'
             sh "ssh -vvv -o StrictHostKeyChecking=no ubuntu@docker-agent.beesshop.org uname -a"
          }
      }
      

        Attachments

          Issue Links

            Activity

            cleclerc Cyrille Le Clerc created issue -
            cleclerc Cyrille Le Clerc made changes -
            Field Original Value New Value
            Environment CloudBees Docker Workflow 1.2
            Jenkins ver. 1.625.3.1 (CloudBees Jenkins Enterprise 15.11)
            cleclerc Cyrille Le Clerc made changes -
            Description {{sshagent(...)\{...\}}} is ignored in {{docker.image(...).inside\{...\}}}. The same sshagent step works in 'standard' linux node.

            See attached logs

            {code:title=sshagent()\{...\} ignored in docker.image(...).inside\{...\} }
            docker.image ('cloudbees/java-build-tools:0.0.7.1').inside {
                sh 'ls -al ~/.ssh/ || true'
                sshagent(['aws-cleclerc-ssh-key-ubuntu-cleclerc']) {
                   sh 'echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK'
                   sh 'ls -al $SSH_AUTH_SOCK || true'
                   sh "ssh -vvv -o StrictHostKeyChecking=no ubuntu@docker-agent.beesshop.org uname -a"
                }
            }
            {code}

            {code:title=sshagent()\{...\} WORKS in node\{...\} }
            node {
                sh 'ls -al ~/.ssh/ || true'
                sshagent(['aws-cleclerc-ssh-key-ubuntu-cleclerc']) {
                   sh 'echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK'
                   sh 'ls -al $SSH_AUTH_SOCK || true'
                   sh "ssh -vvv -o StrictHostKeyChecking=no ubuntu@docker-agent.beesshop.org uname -a"
                }
            }
            {code}
            {{sshagent('my-ssh-key')\{...\}}} is ignored in {{docker.image('cloudbees:java-build-tools').inside\{...\}}}. The same sshagent step works in 'standard' linux node.

            See attached logs

            {code:title=sshagent()\{...\} ignored in docker.image(...).inside\{...\} }
            docker.image ('cloudbees/java-build-tools:0.0.7.1').inside {
                sh 'ls -al ~/.ssh/ || true'
                sshagent(['aws-cleclerc-ssh-key-ubuntu-cleclerc']) {
                   sh 'echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK'
                   sh 'ls -al $SSH_AUTH_SOCK || true'
                   sh "ssh -vvv -o StrictHostKeyChecking=no ubuntu@docker-agent.beesshop.org uname -a"
                }
            }
            {code}

            {code:title=sshagent()\{...\} WORKS in node\{...\} }
            node {
                sh 'ls -al ~/.ssh/ || true'
                sshagent(['aws-cleclerc-ssh-key-ubuntu-cleclerc']) {
                   sh 'echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK'
                   sh 'ls -al $SSH_AUTH_SOCK || true'
                   sh "ssh -vvv -o StrictHostKeyChecking=no ubuntu@docker-agent.beesshop.org uname -a"
                }
            }
            {code}
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            According to Jesse Glick, this issue depends on JENKINS-27152 to handle the tmp directory properly.

            Show
            cleclerc Cyrille Le Clerc added a comment - According to Jesse Glick , this issue depends on JENKINS-27152 to handle the tmp directory properly.
            cleclerc Cyrille Le Clerc made changes -
            Link This issue depends on JENKINS-27152 [ JENKINS-27152 ]
            cleclerc Cyrille Le Clerc made changes -
            Component/s ssh-agent-plugin [ 17509 ]
            Hide
            jglick Jesse Glick added a comment -

            A facet of JENKINS-27152.

            Show
            jglick Jesse Glick added a comment - A facet of JENKINS-27152 .
            jglick Jesse Glick made changes -
            Link This issue duplicates JENKINS-27152 [ JENKINS-27152 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Duplicate [ 3 ]
            jglick Jesse Glick made changes -
            Link This issue depends on JENKINS-27152 [ JENKINS-27152 ]
            rtyler R. Tyler Croy made changes -
            Link This issue is related to JENKINS-27152 [ JENKINS-27152 ]
            rtyler R. Tyler Croy made changes -
            Link This issue is related to JENKINS-27152 [ JENKINS-27152 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgentFactory.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/AgentServer.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgent.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentFactory.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentStarter.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/mina/MinaRemoteAgentFactory.java
            http://jenkins-ci.org/commit/ssh-agent-plugin/d2e0f53b6eda56012c2bd5d9bc553b31667462a2
            Log:
            JENKINS-32624 JENKINS-27152 Use a standardized temporary directory when possible.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgentFactory.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/AgentServer.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgent.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentFactory.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/jna/JNRRemoteAgentStarter.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/mina/MinaRemoteAgentFactory.java http://jenkins-ci.org/commit/ssh-agent-plugin/d2e0f53b6eda56012c2bd5d9bc553b31667462a2 Log: JENKINS-32624 JENKINS-27152 Use a standardized temporary directory when possible.
            Hide
            cleclerc Cyrille Le Clerc added a comment - - edited

            Successfully tested.

            Pipeline Script

            node {
                docker.image ('cloudbees/java-build-tools:0.0.7.1').inside {
                    sh 'whoami'
                    sh 'ls -al ~/.ssh/ || true'
                    sshagent(['aws-cleclerc-ssh-key-ubuntu-cleclerc']) {
                       sh 'echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK'
                       sh 'ls -al $SSH_AUTH_SOCK || true'
                       sh "ssh -o StrictHostKeyChecking=no ubuntu@jenkins-agent.beesshop.org uname -a"
                    }
                }
            }
            

            build logs

            Started by user admin
            [Pipeline] Allocate node : Start
            Running on jenkins-agent.example.com in /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent
            [Pipeline] node {
            [Pipeline] sh
            [test-ssh-agent] Running shell script
            + docker inspect -f . cloudbees/java-build-tools:0.0.7.1
            .
            [Pipeline] Run build steps inside a Docker container : Start
            $ docker run -t -d -u 1000:1000 -w /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent -v /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent:/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent:rw -v /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp:/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** cloudbees/java-build-tools:0.0.7.1 cat
            [Pipeline] withDockerContainer {
            [Pipeline] sh
            [test-ssh-agent] Running shell script
            + whoami
            jenkins
            [Pipeline] sh
            [test-ssh-agent] Running shell script
            + ls -al /home/jenkins/.ssh/
            ls: cannot access /home/jenkins/.ssh/: No such file or directory
            + true
            [Pipeline] SSH Agent : Start
            [ssh-agent] Using credentials ubuntu (cleclerc-us-east.pem)
            [ssh-agent] Looking for ssh-agent implementation...
            [ssh-agent]   Java/JNR ssh-agent
            [ssh-agent] Started.
            [Pipeline] sshagent {
            [Pipeline] sh
            [test-ssh-agent] Running shell script
            + echo SSH_AUTH_SOCK=/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851
            SSH_AUTH_SOCK=/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851
            [Pipeline] sh
            [test-ssh-agent] Running shell script
            + ls -al /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851
            srw------- 1 jenkins jenkins 0 Mar  4 10:03 /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851
            [Pipeline] sh
            [test-ssh-agent] Running shell script
            + ssh -o StrictHostKeyChecking=no ubuntu@jenkins-agent.example.com uname -a
            Warning: Permanently added 'jenkins-agent.example.com,52.90.106.69' (ECDSA) to the list of known hosts.
            Linux ip-10-0-0-151 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
            [Pipeline] } //sshagent
            [Pipeline] SSH Agent : End
            [Pipeline] } //withDockerContainer
            $ docker stop e24c314f6d82ac7d15033a996a56bf33e25096b4b47a0138f1c205559c4adc32
            $ docker rm -f e24c314f6d82ac7d15033a996a56bf33e25096b4b47a0138f1c205559c4adc32
            [Pipeline] Run build steps inside a Docker container : End
            [Pipeline] } //node
            [Pipeline] Allocate node : End
            [Pipeline] End of Pipeline
            Finished: SUCCESS
            
            Show
            cleclerc Cyrille Le Clerc added a comment - - edited Successfully tested. Pipeline Script node { docker.image ( 'cloudbees/java-build-tools:0.0.7.1' ).inside { sh 'whoami' sh 'ls -al ~/.ssh/ || true ' sshagent([ 'aws-cleclerc-ssh-key-ubuntu-cleclerc' ]) { sh 'echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK' sh 'ls -al $SSH_AUTH_SOCK || true ' sh "ssh -o StrictHostKeyChecking=no ubuntu@jenkins-agent.beesshop.org uname -a" } } } build logs Started by user admin [Pipeline] Allocate node : Start Running on jenkins-agent.example.com in /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent [Pipeline] node { [Pipeline] sh [test-ssh-agent] Running shell script + docker inspect -f . cloudbees/java-build-tools:0.0.7.1 . [Pipeline] Run build steps inside a Docker container : Start $ docker run -t -d -u 1000:1000 -w /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent -v /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent:/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent:rw -v /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp:/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** cloudbees/java-build-tools:0.0.7.1 cat [Pipeline] withDockerContainer { [Pipeline] sh [test-ssh-agent] Running shell script + whoami jenkins [Pipeline] sh [test-ssh-agent] Running shell script + ls -al /home/jenkins/.ssh/ ls: cannot access /home/jenkins/.ssh/: No such file or directory + true [Pipeline] SSH Agent : Start [ssh-agent] Using credentials ubuntu (cleclerc-us-east.pem) [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Java/JNR ssh-agent [ssh-agent] Started. [Pipeline] sshagent { [Pipeline] sh [test-ssh-agent] Running shell script + echo SSH_AUTH_SOCK=/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851 SSH_AUTH_SOCK=/home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851 [Pipeline] sh [test-ssh-agent] Running shell script + ls -al /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851 srw------- 1 jenkins jenkins 0 Mar 4 10:03 /home/ubuntu/jenkins-aws-home/workspace/tests/test-ssh-agent@tmp/ssh330183201329153851 [Pipeline] sh [test-ssh-agent] Running shell script + ssh -o StrictHostKeyChecking=no ubuntu@jenkins-agent.example.com uname -a Warning: Permanently added 'jenkins-agent.example.com,52.90.106.69' (ECDSA) to the list of known hosts. Linux ip-10-0-0-151 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux [Pipeline] } //sshagent [Pipeline] SSH Agent : End [Pipeline] } //withDockerContainer $ docker stop e24c314f6d82ac7d15033a996a56bf33e25096b4b47a0138f1c205559c4adc32 $ docker rm -f e24c314f6d82ac7d15033a996a56bf33e25096b4b47a0138f1c205559c4adc32 [Pipeline] Run build steps inside a Docker container : End [Pipeline] } //node [Pipeline] Allocate node : End [Pipeline] End of Pipeline Finished: SUCCESS
            sdomula Stan Domula made changes -
            Comment [ I'm having the exact same issue if the script is executed via Jenkinsfile with the GitHub Organisation Plugin. Same script copied into a Pipeline Job works. It works while using Jenkinsfile and using node{} instead of docker.image.inside.

            Jenkins: 2.7.1 LTS
            SSH Agent Plugin: 1.13 ]
            sdomula Stan Domula made changes -
            Comment [ The same error occures if the job's workspace tmp path + socket name gets longer than 108. It works with node, but not with docker.image.inside {}

            It seems like it is not creating the link under '/tmp/../' if the sockets path gets longer than UNIX_PATH_MAX.

            {code}
            [Pipeline] {
            [Pipeline] sshagent
            [ssh-agent] Using credentials ubuntu (develop ssh key)
            [ssh-agent] Looking for ssh-agent implementation...
            [ssh-agent] Java/JNR ssh-agent
            [ssh-agent] Skipped registering BouncyCastle, not running on a remote agent
            [ssh-agent] Started.
            [Pipeline] {
            [Pipeline] sh
            [asdfasdlkfajsdlkjfklasdjlfjaklsdjfjaklsdjflkjasdkljfljasdljflkajsdjfasjdlfjaskldjfljasdlkjfl] Running shell script
            + echo SSH_AUTH_SOCK=/tmp/ssh6669368941114379204
            SSH_AUTH_SOCK=/tmp/ssh6669368941114379204
            [Pipeline] sh
            [asdfasdlkfajsdlkjfklasdjlfjaklsdjfjaklsdjflkjasdkljfljasdljflkajsdjfasjdlfjaskldjfljasdlkjfl] Running shell script
            + ls -al /tmp/ssh6669368941114379204
            ls: /tmp/ssh6669368941114379204: No such file or directory
            {code} ]
            Hide
            sdomula Stan Domula added a comment -

            Still not fully fixed.

            If the socket name gets longer than 108 chars it tries to create the socket in /tmp that doesn't work inside docker. So this is still an issue when using GitHub Organistaion Plugin. There it happens that paths are getting longer.

            Show
            sdomula Stan Domula added a comment - Still not fully fixed. If the socket name gets longer than 108 chars it tries to create the socket in /tmp that doesn't work inside docker. So this is still an issue when using GitHub Organistaion Plugin. There it happens that paths are getting longer.
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 168281 ] JNJira + In-Review [ 198387 ]
            Hide
            jspiewak Joshua Spiewak added a comment -

            If one uses the standard location of `/var/lib/jenkins` as the remote root on the Docker daemon agent, for a multi-branch pipeline job the shortest possible path that minimize will generate is too long:

            echo "/var/lib/jenkins/workspace/a-R6R3BLSUWMBHF2ZTDGNA6MR4UMMWUNRCZRZ2356BQKJBSSLTLWYA@tmp/ssh9122165901071866621" | wc -c
            109
            
            Show
            jspiewak Joshua Spiewak added a comment - If one uses the standard location of `/var/lib/jenkins` as the remote root on the Docker daemon agent, for a multi-branch pipeline job the shortest possible path that minimize will generate is too long: echo "/ var /lib/jenkins/workspace/a-R6R3BLSUWMBHF2ZTDGNA6MR4UMMWUNRCZRZ2356BQKJBSSLTLWYA@tmp/ssh9122165901071866621" | wc -c 109
            Hide
            jglick Jesse Glick added a comment -

            There is no “standard location” for Docker-based agent; depends on the image you use. So you need to use something shorter. The path name length limitation for sshagent sockets is an unfortunate artifact of a weird default in typical Linux kernels. I do not know of a satisfactory fix.

            Show
            jglick Jesse Glick added a comment - There is no “standard location” for Docker-based agent; depends on the image you use. So you need to use something shorter. The path name length limitation for sshagent sockets is an unfortunate artifact of a weird default in typical Linux kernels. I do not know of a satisfactory fix.
            Hide
            jspiewak Joshua Spiewak added a comment -

            Sure, we don't have to call it a "standard location" for agents, but given that the Jenkins package on, say, Ubuntu, installs to /var/lib/jenkins, maintaining a similar location on the agents would seem to be a sensible way to keep systems consistent, and far more obvious than setting it to /j.

            Perhaps while this is being iterated upon there could be a feature toggle that could allow folks to fallback to the previous behavior if they need to?

            Show
            jspiewak Joshua Spiewak added a comment - Sure, we don't have to call it a "standard location" for agents, but given that the Jenkins package on, say, Ubuntu, installs to /var/lib/jenkins, maintaining a similar location on the agents would seem to be a sensible way to keep systems consistent, and far more obvious than setting it to /j. Perhaps while this is being iterated upon there could be a feature toggle that could allow folks to fallback to the previous behavior if they need to?
            Hide
            jglick Jesse Glick added a comment -

            Which previous behavior are you referring to?

            Show
            jglick Jesse Glick added a comment - Which previous behavior are you referring to?
            Hide
            erliz Stanislav Vetlovskiy added a comment - - edited

            I faced with the same issue, all was good on simple pipeline, but when i switched to multi-branch pipeline job, tasks in sshagent fall with error

            SSH_AUTH_SOCK=/tmp/ssh3569037635123687207
            + ls -al /tmp/ssh3569037635123687207
            ls: cannot access /tmp/ssh3569037635123687207: No such file or directory
            

            is there any advice to fix it or some workaround?

            Show
            erliz Stanislav Vetlovskiy added a comment - - edited I faced with the same issue, all was good on simple pipeline, but when i switched to multi-branch pipeline job, tasks in sshagent fall with error SSH_AUTH_SOCK=/tmp/ssh3569037635123687207 + ls -al /tmp/ssh3569037635123687207 ls: cannot access /tmp/ssh3569037635123687207: No such file or directory is there any advice to fix it or some workaround?
            Hide
            jspiewak Joshua Spiewak added a comment -

            I am referring to the previous behavior that stored the workspace in sub-folders without the base32 of the hash that was introduced for JENKINS-34564.

            Show
            jspiewak Joshua Spiewak added a comment - I am referring to the previous behavior that stored the workspace in sub-folders without the base32 of the hash that was introduced for JENKINS-34564 .
            Hide
            jglick Jesse Glick added a comment -

            The previous naming pattern for branch projects can be restored with a system property, see release notes.

            Show
            jglick Jesse Glick added a comment - The previous naming pattern for branch projects can be restored with a system property, see release notes.
            jknurek J Knurek made changes -
            Link This issue is related to JENKINS-42582 [ JENKINS-42582 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                cleclerc Cyrille Le Clerc
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: