Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36997

sshAgent {} inside docker.image().inside {} does not work with long project name

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: ssh-agent-plugin
    • Labels:
      None
    • Environment:
      Jenkins: 2.7.1 LTS
      docker-workflow-plugin: 1.7
      ssh-agent-plugin: 1.13
    • Similar Issues:

      Description

      If the socket can not be created in the tmp directory of the job, which happens if the socket path would get longer than 108, it will create the socket directly under /tmp. If I am using docker.image().inside this does not work. If I am using node {} it does work as expected.

      When using the GitHub Organisation Plugin I can not avoid long job names.

        Attachments

          Issue Links

            Activity

            sdomula Stan Domula created issue -
            sdomula Stan Domula made changes -
            Field Original Value New Value
            Priority Minor [ 4 ] Major [ 3 ]
            sdomula Stan Domula made changes -
            Description If the socket can not be created in the tmp directory of the job, which happens if the socket path would get longer than 108, it will create the socket directly under /tmp. If I am using docker.image().inside this does not work. If I am using node {} it does work as expected.
            If the socket can not be created in the tmp directory of the job, which happens if the socket path would get longer than 108, it will create the socket directly under /tmp. If I am using docker.image().inside this does not work. If I am using node {} it does work as expected.

            When using the GitHub Organisation Plugin I can not avoid long job names.
            Hide
            jglick Jesse Glick added a comment -

            Yes the SSH Agent plugin falls back to the system temporary directory when the path is going to be longer than 108 characters, the maximum typically supported by Linux kernels. Unfortunately this breaks the assumption of Docker Pipeline that the container and (agent) host share filesystems only in the workspace and its associated temporary directory.

            Might work to use ws with a (shortish) absolute path inside node but outside inside. Or use withCredentials to access the SSH private key explicitly rather than using sshAgent.

            Show
            jglick Jesse Glick added a comment - Yes the SSH Agent plugin falls back to the system temporary directory when the path is going to be longer than 108 characters, the maximum typically supported by Linux kernels. Unfortunately this breaks the assumption of Docker Pipeline that the container and (agent) host share filesystems only in the workspace and its associated temporary directory. Might work to use ws with a (shortish) absolute path inside node but outside inside . Or use withCredentials to access the SSH private key explicitly rather than using sshAgent .
            Hide
            shanemcd Shane McDonald added a comment - - edited

            I ran into this as well. I got around the issue by checking "Use custom child workspace" in my job config, and setting "Child Directory" to ${SHORT_COMBINATION}

            Show
            shanemcd Shane McDonald added a comment - - edited I ran into this as well. I got around the issue by checking "Use custom child workspace" in my job config, and setting "Child Directory" to ${SHORT_COMBINATION}
            Hide
            jglick Jesse Glick added a comment -

            Avoid that workaround; use ws instead.

            Show
            jglick Jesse Glick added a comment - Avoid that workaround; use ws instead.
            Hide
            shanemcd Shane McDonald added a comment -

            Interesting... Couple of questions:

            • Why should I avoid this?
            • What is ws?
            Show
            shanemcd Shane McDonald added a comment - Interesting... Couple of questions: Why should I avoid this? What is ws ?
            Hide
            jglick Jesse Glick added a comment -

            Look at the documentation.

            Show
            jglick Jesse Glick added a comment - Look at the documentation.
            Hide
            shanemcd Shane McDonald added a comment -

            Thanks for being so helpful.

            Show
            shanemcd Shane McDonald added a comment - Thanks for being so helpful.
            Hide
            jglick Jesse Glick added a comment -

            Would perhaps be bypassed by PR 2, TBD.

            Show
            jglick Jesse Glick added a comment - Would perhaps be bypassed by PR 2, TBD.
            jglick Jesse Glick made changes -
            Remote Link This issue links to "ssh-agent PR 2 (Web Link)" [ 15029 ]
            Hide
            morlajb2 mor lajb added a comment -

            HI , I have the same problem - I am using jenkins version 2.32.1
            Docker Pipeline - 1.9.1
            Pipeline - 2.4
            Pipeline: Basic Steps -2.3
            Pipeline: Job - 2.9
            SSH Agent Plugin 1.13

            I couldn't workaround it , any idea ?

            here is my pipeline

            _node ('ubuntu-aws'){
            timestamps
            {
            docker.image('node').inside {
            sshagent(['XXX'])

            { stage "git checkout" checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], gitTool: 'Default', submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'YYY', url: 'git@xxx:yyy.git']]]) stage "npm install" sh "npm install" }

            } // ssh agent
            }
            }

            _

            Show
            morlajb2 mor lajb added a comment - HI , I have the same problem - I am using jenkins version 2.32.1 Docker Pipeline - 1.9.1 Pipeline - 2.4 Pipeline: Basic Steps -2.3 Pipeline: Job - 2.9 SSH Agent Plugin 1.13 I couldn't workaround it , any idea ? here is my pipeline _node ('ubuntu-aws'){ timestamps { docker.image('node').inside { sshagent( ['XXX'] ) { stage "git checkout" checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], gitTool: 'Default', submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'YYY', url: 'git@xxx:yyy.git']]]) stage "npm install" sh "npm install" } } // ssh agent } } _
            Hide
            jglick Jesse Glick added a comment -

            Again: only known workaround is to use the ws Pipeline step with a short absolute path, taking care to avoid collisions with other builds somehow (not generally easy). Proposed alternate plugin implementation would probably solve this issue; needs review and testing.

            Show
            jglick Jesse Glick added a comment - Again: only known workaround is to use the ws Pipeline step with a short absolute path, taking care to avoid collisions with other builds somehow (not generally easy). Proposed alternate plugin implementation would probably solve this issue; needs review and testing.
            jglick Jesse Glick made changes -
            Remote Link This issue links to "ssh-agent PR 17 (Web Link)" [ 15208 ]
            Hide
            jglick Jesse Glick added a comment -

            PR 17 is the new version.

            Show
            jglick Jesse Glick added a comment - PR 17 is the new version.
            Hide
            jglick Jesse Glick added a comment -

            For those using multibranch projects, read release notes. Since the default workspace directory name will be up to 80 characters, and you must stay within 108 for sshagent to work, that means you must limit your “remote FS root” to well under 28 characters.

            Show
            jglick Jesse Glick added a comment - For those using multibranch projects, read release notes . Since the default workspace directory name will be up to 80 characters, and you must stay within 108 for sshagent to work, that means you must limit your “remote FS root” to well under 28 characters.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/test/java/plugins/WorkflowPluginTest.java
            http://jenkins-ci.org/commit/acceptance-test-harness/7c815e3bcdb83c68dbfa1d8778ce6465fe7b3517
            Log:
            JENKINS-36997 Noting that weird script is a workaround for a PATH_MAX problem.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/test/java/plugins/WorkflowPluginTest.java http://jenkins-ci.org/commit/acceptance-test-harness/7c815e3bcdb83c68dbfa1d8778ce6465fe7b3517 Log: JENKINS-36997 Noting that weird script is a workaround for a PATH_MAX problem.
            jglick Jesse Glick made changes -
            Component/s docker-workflow-plugin [ 20625 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgent.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/exec/ExecRemoteAgent.java
            src/main/java/com/cloudbees/jenkins/plugins/sshagent/exec/ExecRemoteAgentFactory.java
            src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBase.java
            src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapperTest.java
            src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepWorkflowTest.java
            http://jenkins-ci.org/commit/ssh-agent-plugin/8d02c6ca20ce514e8737a15b24bbfa7557930273
            Log:
            Merge pull request #18 from jglick/ExecRemoteAgent-JENKINS-36997

            JENKINS-36997 CLI implementation of RemoteAgent

            Compare: https://github.com/jenkinsci/ssh-agent-plugin/compare/f38bb5f22fea...8d02c6ca20ce

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/com/cloudbees/jenkins/plugins/sshagent/RemoteAgent.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapper.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepExecution.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/exec/ExecRemoteAgent.java src/main/java/com/cloudbees/jenkins/plugins/sshagent/exec/ExecRemoteAgentFactory.java src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBase.java src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentBuildWrapperTest.java src/test/java/com/cloudbees/jenkins/plugins/sshagent/SSHAgentStepWorkflowTest.java http://jenkins-ci.org/commit/ssh-agent-plugin/8d02c6ca20ce514e8737a15b24bbfa7557930273 Log: Merge pull request #18 from jglick/ExecRemoteAgent- JENKINS-36997 JENKINS-36997 CLI implementation of RemoteAgent Compare: https://github.com/jenkinsci/ssh-agent-plugin/compare/f38bb5f22fea...8d02c6ca20ce
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-39540 [ JENKINS-39540 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-32704 [ JENKINS-32704 ]
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-29810 [ JENKINS-29810 ]
            jglick Jesse Glick made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            mkobit Mike Kobit added a comment -

            We tried this out with:

            sshagent(['CredId']) {
              docker.inside('image') {
                // git operations
              }
            }

            and it did not work.

             

            Using the following did work:

            docker.inside('image') {
              sshagent(['CredId']) {
                // git operations
              }
            }

            That makes sense to me based on how the execution works now with using a CLI implementation.

            Thanks for fixing this!

            Show
            mkobit Mike Kobit added a comment - We tried this out with: sshagent([ 'CredId' ]) { docker.inside( 'image' ) { // git operations } } and it did not work.   Using the following did work: docker.inside( 'image' ) { sshagent([ 'CredId' ]) { // git operations } } That makes sense to me based on how the execution works now with using a CLI implementation. Thanks for fixing this!
            Hide
            jglick Jesse Glick added a comment -

            Right, the ssh-agent needs to be run inside the container so its socket is in the same kernel namespace as the commands which try to access it.

            Show
            jglick Jesse Glick added a comment - Right, the ssh-agent needs to be run inside the container so its socket is in the same kernel namespace as the commands which try to access it.
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-42093 [ JENKINS-42093 ]
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-42346 [ JENKINS-42346 ]
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-43050 [ JENKINS-43050 ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal OSS-1802 (Web Link)" [ 18558 ]
            Hide
            hermain Hermann Schweizer added a comment - - edited

            Why is this marked as resolved? I use a multibranch pipeline and still suffer from this issue.
            I have a master slave setup. Which workspaces now need to be shorter than 108? master or slave or both?
            Simply doing ws("test") before the image.inside didn't help although the resulting ws path is really short.

            Outside of the container git cloning with inside sshagent(..){ works. Inside container it doesn't and the log says:
            docker exec 9859e94c20b84efd81e1752417c1f5144fa198ba76b14e1670f8993512af7d60 ssh-agent
            SSH_AUTH_SOCK=/tmp/ssh-GoP5qbl4iakN/agent.13
            SSH_AGENT_PID=20

            ```
            $ docker exec --env SSH_AGENT_PID=20 --env SSH_AUTH_SOCK=/tmp/ssh-GoP5qbl4iakN/agent.13 9859e94c20b84efd81e1752417c1f5144fa198ba76b14e1670f8993512af7d60 ssh-add /home/INT/jenkins/short@tmp/private_key_3688909095782238252.key
            Identity added: /home/INT/jenkins/short@tmp/private_key_3688909095782238252.key (/home/INT/jenkins/short@tmp/private_key_3688909095782238252.key)

            ```

            Show
            hermain Hermann Schweizer added a comment - - edited Why is this marked as resolved? I use a multibranch pipeline and still suffer from this issue. I have a master slave setup. Which workspaces now need to be shorter than 108? master or slave or both? Simply doing ws("test") before the image.inside didn't help although the resulting ws path is really short. Outside of the container git cloning with inside sshagent(..){ works. Inside container it doesn't and the log says: docker exec 9859e94c20b84efd81e1752417c1f5144fa198ba76b14e1670f8993512af7d60 ssh-agent SSH_AUTH_SOCK=/tmp/ssh-GoP5qbl4iakN/agent.13 SSH_AGENT_PID=20 ``` $ docker exec --env SSH_AGENT_PID=20 --env SSH_AUTH_SOCK=/tmp/ssh-GoP5qbl4iakN/agent.13 9859e94c20b84efd81e1752417c1f5144fa198ba76b14e1670f8993512af7d60 ssh-add /home/INT/jenkins/short@tmp/private_key_3688909095782238252.key Identity added: /home/INT/jenkins/short@tmp/private_key_3688909095782238252.key (/home/INT/jenkins/short@tmp/private_key_3688909095782238252.key) ```
            Hide
            duemir Denys Digtiar added a comment - - edited

            Hermann Schweizer As Mike and Jesse alluded to, the new Agent implementation was added which is based on the CLI ssh-agent. If you have a CLI available inside the docker container and use the `sshagent` inside the docker.inside() closure, your git clone should work.

            Look for the message like "Exec ssh-agent (binary ssh-agent on a remote machine)" or any errors that mention ssh-agent

            Show
            duemir Denys Digtiar added a comment - - edited Hermann Schweizer As Mike and Jesse alluded to, the new Agent implementation was added which is based on the CLI ssh-agent . If you have a CLI available inside the docker container and use the `sshagent` inside the docker.inside() closure, your git clone should work. Look for the message like "Exec ssh-agent (binary ssh-agent on a remote machine)" or any errors that mention ssh-agent
            devopsfido Bob O made changes -
            Comment [ Having exactly the same issue as [~hermain] any explanation. ]
            Hide
            hermain Hermann Schweizer added a comment -

            Bob O I managed to resolve the issue but this page was offline at the time so I forgot to mention it here:

            The problem in my case was that my git was not a known host inside the container:

            image.inside("-u root:root --network=host") {
                        sshagent(credentials: [config.gitKeyCredentialsId] ) {
                            sh "mkdir ~/.ssh"
                            sh "ssh-keyscan git.myCompany.com >> ~/.ssh/known_hosts"
                           // call scripts that do git clone...
                        }
                    }

             

            I hope that helps you and anyone else with a similar problem.

            Show
            hermain Hermann Schweizer added a comment - Bob O I managed to resolve the issue but this page was offline at the time so I forgot to mention it here: The problem in my case was that my git was not a known host inside the container: image.inside("-u root:root --network=host") {             sshagent(credentials: [config.gitKeyCredentialsId] ) {                 sh "mkdir ~/.ssh"                 sh "ssh-keyscan git.myCompany.com >> ~/.ssh/known_hosts"                // call scripts that do git clone...             }         }   I hope that helps you and anyone else with a similar problem.
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-51811 [ JENKINS-51811 ]
            Hide
            eshepelyuk Evgeny Shepelyuk added a comment - - edited

            Hello, I'm using latest 1.17 version of SSH Agent Plugin and still expirience the same issue.
            Although docker image has ssh-agent cmd line command inside - the socket is created under {{/tmp} so it's inaccesisble

            Show
            eshepelyuk Evgeny Shepelyuk added a comment - - edited Hello, I'm using latest 1.17 version of SSH Agent Plugin and still expirience the same issue. Although docker image has ssh-agent cmd line command inside - the socket is created under {{/tmp} so it's inaccesisble
            Hide
            jglick Jesse Glick added a comment -

            Evgeny Shepelyuk maybe ssh-agent is not in $PATH or something.

            Show
            jglick Jesse Glick added a comment - Evgeny Shepelyuk maybe ssh-agent is not in $PATH or something.
            Hide
            mcw Matt C. Wilson added a comment -

            Hermann Schweizer Thank you!  This was exactly our issue.

             

            Works with SSH Agent Plugin v1.17 for us, using declarative pipeline and just a regular agent block:

            pipeline {
              agent { docker {
                image 'my_build_image'
                ...
              } }
              ...
              stage("foo") {
                sshagent(credentials: [config.gitKeyCredentialsId] ) {
                  sh "mkdir ~/.ssh && ssh-keyscan git.myCompany.com >> ~/.ssh/known_hosts"
                  // call scripts that do git clone...
                }
              }
            }
            
            
            Show
            mcw Matt C. Wilson added a comment - Hermann Schweizer  Thank you!  This was exactly our issue.   Works with SSH Agent Plugin v1.17 for us, using declarative pipeline and just a regular agent block: pipeline { agent { docker { image 'my_build_image' ... } } ... stage( "foo" ) { sshagent(credentials: [config.gitKeyCredentialsId] ) {       sh "mkdir ~/.ssh && ssh-keyscan git.myCompany.com >> ~/.ssh/known_hosts"        // call scripts that do git clone...     } } }

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                sdomula Stan Domula
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: