Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40484

Unable to use withMaven() step inside docker container for old versions of Docker

    Details

    • Similar Issues:

      Description

      I'm not able to use withMaven step inside docker container.

      [Pipeline] withMaven
      $ docker exec ffff env printenv MAVEN_HOME
      $ docker exec ffff env printenv M2_HOME
      $ docker exec ffff env /bin/sh -c "which mvn"
      Using maven exec: /opt/apache-maven-3.3.9/bin/mvn
      Using global settings config with name MavenGlobal
      Replacing all maven server entries not found in credentials list is false
      [Pipeline] {
      [Pipeline] sh
      
      [test-XXX] Running shell script
      nohup: failed to run command `sh': No such file or directory
      
      [Pipeline] }
      [Pipeline] // withMaven

      My Jenkinsfile pipeline:

      node('docker') {
          stage('Checkout') {
              checkout scm
          }
          buildInDocker('linux') {
              stage('Maven') {
                  withMaven(globalMavenSettingsConfig: '11111111-2222-3333-4444-555555555555') {
                          sh 'mvn clean test'
                  }
              }
          }
      }

      It looks like PATH env gets overwritten.

        Attachments

          Issue Links

            Activity

            testuser7 Test User created issue -
            dantran dan tran made changes -
            Field Original Value New Value
            Priority Minor [ 4 ] Critical [ 2 ]
            dantran dan tran made changes -
            Priority Critical [ 2 ] Major [ 3 ]
            Hide
            dantran dan tran added a comment -

            Love to see this fixed

            Show
            dantran dan tran added a comment - Love to see this fixed
            Hide
            bkmeneguello Bruno Meneguello added a comment -

            Same here, with this simple use case:

            node('master') {
                docker.image('maven').inside {
                    withMaven {
                        sh 'mvn -version'
                    }
                }
            }
            
            Show
            bkmeneguello Bruno Meneguello added a comment - Same here, with this simple use case: node( 'master' ) { docker.image( 'maven' ).inside { withMaven { sh 'mvn -version' } } }
            Hide
            alobato Alvaro Lobato added a comment -

            Does this work for you?

            node('master') {
                docker.image('maven').inside {
                        sh 'mvn -version'
                }
            }
            
            Show
            alobato Alvaro Lobato added a comment - Does this work for you? node( 'master' ) { docker.image( 'maven' ).inside { sh 'mvn -version' } }
            Hide
            bkmeneguello Bruno Meneguello added a comment -

            Yes, it does

            Show
            bkmeneguello Bruno Meneguello added a comment - Yes, it does
            Hide
            testuser7 Test User added a comment -

            Simple Jenkinsfile to show how env are changed:

            node('docker') {
                docker.image('linux').inside {
                    echo 'env.PATH=' + env.PATH
                    sh 'echo $PATH > path'
                    path = readFile('path')
                    echo 'PATH=' + path
                    withMaven {
                        echo 'env.PATH.mvn=' + env.PATH
                        env.PATH = env.PATH + ':/bin'
                        sh 'echo $PATH > path_mvn'
                        path_mvn = readFile('path_mvn')
                        echo 'PATH.mvn=' + path_mvn
                        }
                }
            }
            env.PATH=/usr/local/bin:/usr/bin
            PATH=/opt/apache-maven-3.3.9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
            env.PATH.mvn=/var/lib/jenkins/workspace/repository-test_develop-ISA36EXDSAND7KY74323XF3CUR2ZV65I43UWYEDK2UMP3I5UQYMQ@tmp/withMavenaf31c1e4:/usr/local/bin:/usr/bin
            PATH.mvn=/var/lib/jenkins/workspace/repository-test_develop-ISA36EXDSAND7KY74323XF3CUR2ZV65I43UWYEDK2UMP3I5UQYMQ@tmp/withMavenaf31c1e4:/var/lib/jenkins/workspace/repository-test_develop-ISA36EXDSAND7KY74323XF3CUR2ZV65I43UWYEDK2UMP3I5UQYMQ@tmp/withMavenaf31c1e4:/usr/local/bin:/usr/bin:/bin
            Show
            testuser7 Test User added a comment - Simple Jenkinsfile to show how env are changed: node( 'docker' ) { docker.image( 'linux' ).inside { echo 'env.PATH=' + env.PATH sh 'echo $PATH > path' path = readFile( 'path' ) echo 'PATH=' + path withMaven { echo 'env.PATH.mvn=' + env.PATH env.PATH = env.PATH + ':/bin' sh 'echo $PATH > path_mvn' path_mvn = readFile( 'path_mvn' ) echo 'PATH.mvn=' + path_mvn } } } env.PATH=/usr/local/bin:/usr/bin PATH=/opt/apache-maven-3.3.9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin env.PATH.mvn=/ var /lib/jenkins/workspace/repository-test_develop-ISA36EXDSAND7KY74323XF3CUR2ZV65I43UWYEDK2UMP3I5UQYMQ@tmp/withMavenaf31c1e4:/usr/local/bin:/usr/bin PATH.mvn=/ var /lib/jenkins/workspace/repository-test_develop-ISA36EXDSAND7KY74323XF3CUR2ZV65I43UWYEDK2UMP3I5UQYMQ@tmp/withMavenaf31c1e4:/ var /lib/jenkins/workspace/repository-test_develop-ISA36EXDSAND7KY74323XF3CUR2ZV65I43UWYEDK2UMP3I5UQYMQ@tmp/withMavenaf31c1e4:/usr/local/bin:/usr/bin:/bin
            Hide
            dantran dan tran added a comment -

            relate to JENKINS-41339 ??

            Show
            dantran dan tran added a comment - relate to JENKINS-41339 ??
            Hide
            ncosta Nuno Costa added a comment - - edited

            The workaround to pass the variables from outside withMaven block, mentioned by Test User here works on a Jenkins 2.32.2 but not on Jenkins 2.46.

            Ubuntu 12.04
            Jenkins 2.32.2
            Pipeline Maven Plugin 0.5
            Durable Task Plugin 1.13

            CentOS 7
            Jenkins 2.46
            Pipeline Maven Plugin 0.5
            Durable Task Plugin 1.13

            Buildserver
            Ubuntu 14.04 + docker 1.13.1

            Buildserver
            CentOS7 + docker 1.13.1

            Docker image:
            Ubuntu 12.04
            Maven 3.1.1

            Show
            ncosta Nuno Costa added a comment - - edited The workaround to pass the variables from outside withMaven block, mentioned by Test User here works on a Jenkins 2.32.2 but not on Jenkins 2.46. Ubuntu 12.04 Jenkins 2.32.2 Pipeline Maven Plugin 0.5 Durable Task Plugin 1.13 CentOS 7 Jenkins 2.46 Pipeline Maven Plugin 0.5 Durable Task Plugin 1.13 Buildserver Ubuntu 14.04 + docker 1.13.1 Buildserver CentOS7 + docker 1.13.1 Docker image: Ubuntu 12.04 Maven 3.1.1
            Hide
            ncosta Nuno Costa added a comment - - edited

            This can be replicated with below code. CentOS node build will fail with

            [number1] nohup: failed to run command `sh': No such file or directory
            

            Commented is some troubleshoot steps.
            Can be resolved by getting the container path into a file, pass it into a variable and finally pass that to withEnv.

            Be aware that echo adds a new line to the file and in my case, this was also triggering the same issue. Use trim() to remove the extra line.
            The use of printf instead of echo will avoid the trim() usage.

            CentOS 7
            Jenkins 2.46

            parallel(
                number1: {
                    node('centos-buildserver') {
                        buildenv = docker.image("custom-ubuntu12.04-maven")
                        buildenv.inside("-v /var/lib/jenkins/.m2/repository/${EXECUTOR_NUMBER}:/home/jenkins/.m2/repository"){
                            stage("test") {
                                /*sh 'echo $PATH > path'
                                sh 'printf $PATH > path'
                                path = readFile('path').trim()
                                path = readFile('path')
                                pathtrim = readFile('path').trim()
                                echo '#######'
                                echo 'no trim: #'+path+'#'
                                echo '#######'
                                echo 'with trim: #'+pathtrim+'#'
                                echo '#######'*/
                                sh 'ls -l'
                                //withEnv(['PATH+workaround='+path]) {
                                    withMaven(mavenSettingsConfig: 'settings-id'){
                                        echo 'inside withmaven'
                                        sh 'ls -l'
                                    }
                                //}
                            }
                        }
                    }
                },
                number2: {
                    node('ubuntu-buildserver') {
                        buildenv = docker.image("custom-ubuntu12.04-maven")
                        buildenv.inside("-v /var/lib/jenkins/.m2/repository/${EXECUTOR_NUMBER}:/home/jenkins/.m2/repository"){
                            stage("test") {
                                /*sh 'echo $PATH > path'
                                sh 'printf $PATH > path'
                                path = readFile('path').trim()
                                path = readFile('path')
                                pathtrim = readFile('path').trim()
                                echo '#######'
                                echo '#'+path+'#'
                                echo '#######'
                                echo '#'+pathtrim+'#'
                                echo '#######'*/
                                sh 'ls -l'
                                //withEnv(['PATH+workaround='+path]) {
                                    withMaven(mavenSettingsConfig: 'settings-id'){
                                        echo 'inside withmaven'
                                        sh 'ls -l'
                                    }
                                //}
                            }
                        }
                    }
                }
            )
            
            Show
            ncosta Nuno Costa added a comment - - edited This can be replicated with below code. CentOS node build will fail with [number1] nohup: failed to run command `sh': No such file or directory Commented is some troubleshoot steps. Can be resolved by getting the container path into a file, pass it into a variable and finally pass that to withEnv. Be aware that echo adds a new line to the file and in my case, this was also triggering the same issue. Use trim() to remove the extra line. The use of printf instead of echo will avoid the trim() usage. CentOS 7 Jenkins 2.46 parallel( number1: { node('centos-buildserver') { buildenv = docker.image("custom-ubuntu12.04-maven") buildenv.inside("-v /var/lib/jenkins/.m2/repository/${EXECUTOR_NUMBER}:/home/jenkins/.m2/repository"){ stage("test") { /*sh 'echo $PATH > path' sh 'printf $PATH > path' path = readFile('path').trim() path = readFile('path') pathtrim = readFile('path').trim() echo '#######' echo 'no trim: #'+path+'#' echo '#######' echo 'with trim: #'+pathtrim+'#' echo '#######'*/ sh 'ls -l' //withEnv(['PATH+workaround='+path]) { withMaven(mavenSettingsConfig: 'settings-id'){ echo 'inside withmaven' sh 'ls -l' } //} } } } }, number2: { node('ubuntu-buildserver') { buildenv = docker.image("custom-ubuntu12.04-maven") buildenv.inside("-v /var/lib/jenkins/.m2/repository/${EXECUTOR_NUMBER}:/home/jenkins/.m2/repository"){ stage("test") { /*sh 'echo $PATH > path' sh 'printf $PATH > path' path = readFile('path').trim() path = readFile('path') pathtrim = readFile('path').trim() echo '#######' echo '#'+path+'#' echo '#######' echo '#'+pathtrim+'#' echo '#######'*/ sh 'ls -l' //withEnv(['PATH+workaround='+path]) { withMaven(mavenSettingsConfig: 'settings-id'){ echo 'inside withmaven' sh 'ls -l' } //} } } } } )
            Hide
            alobato Alvaro Lobato added a comment -

            I've done multiple tests and I can't reproduce this issue. Can you give me the exact versions of the pipeline plugins you are running? The easiest way to help me reproduce this is to install the Support Plugin and generate a support bundle which will contain all the envirornment information.

            Did you check if this is related to JENKINS-41339?

            Show
            alobato Alvaro Lobato added a comment - I've done multiple tests and I can't reproduce this issue. Can you give me the exact versions of the pipeline plugins you are running? The easiest way to help me reproduce this is to install the Support Plugin and generate a support bundle which will contain all the envirornment information. Did you check if this is related to JENKINS-41339 ?
            dantran dan tran made changes -
            Attachment image-2017-03-07-12-58-37-245.png [ 36322 ]
            dantran dan tran made changes -
            Attachment image-2017-03-07-12-58-37-245.png [ 36322 ]
            Hide
            dantran dan tran added a comment - - edited

            here is my simple pipeline script running against 2.46.1. This time it has a different error 
            [pipeline-test] Running shell script
            env: nohup: No such file or directory

             

            {code}

            pipeline {

              agent {
                docker {
                image "maven"
                }
              }

              stages {

                stage("run") {
                  steps {
                    withMaven( mavenLocalRepo: '.repository') {
                       sh "echo hello"

                     }
                }
              }

              }

             

            I now attached the support bundle

              }

             

             
            Show
            dantran dan tran added a comment - - edited here is my simple pipeline script running against 2.46.1. This time it has a different error  [pipeline-test] Running shell script env: nohup: No such file or directory   {code} pipeline {   agent {     docker {     image "maven"     }   }   stages {     stage("run") {       steps {         withMaven( mavenLocalRepo: '.repository') {            sh "echo hello"          }     }   }   }   I now attached the support bundle   }    
            dantran dan tran made changes -
            Link This issue relates to JENKINS-41204 [ JENKINS-41204 ]
            dantran dan tran made changes -
            Attachment support_2017-04-22_18.28.56.zip [ 37302 ]
            Hide
            cleclerc Cyrille Le Clerc added a comment - - edited

            Cannot reproduce. If your job fails with "nohup" not found, I suspect that the host linux instance is missing "nohup" or maybe you have used a "weird" Docker image "maven" with missing "nohup".

            Please re open a ticket if you are still facing issues.

             My test is based on

            • pipeline-maven-spy-2.1.1-beta-1.hpi (should behave the same on older versions)
            • "maven:3.5.0-jdk-8" and executed
            • Xenial 16:04 ubuntu build agent.

            https://gist.github.com/cyrille-leclerc/20af6c45ce04f6a3c727342bdf474159

            node {
                stage('build') {
                    docker.image('maven:3.5.0-jdk-8').inside {
                        git 'https://github.com/spring-projects/spring-petclinic.git'
                        withMaven() {
                            sh 'mvn clean package'
                        }
                    }
                }
            }
             
            Show
            cleclerc Cyrille Le Clerc added a comment - - edited Cannot reproduce. If your job fails with "nohup" not found, I suspect that the host linux instance is missing "nohup" or maybe you have used a "weird" Docker image "maven" with missing "nohup". Please re open a ticket if you are still facing issues.  My test is based on pipeline-maven-spy-2.1.1-beta-1.hpi (should behave the same on older versions) "maven:3.5.0-jdk-8" and executed Xenial 16:04 ubuntu build agent. https://gist.github.com/cyrille-leclerc/20af6c45ce04f6a3c727342bdf474159 node { stage( 'build' ) { docker.image( 'maven:3.5.0-jdk-8' ).inside { git 'https: //github.com/spring-projects/spring-petclinic.git' withMaven() { sh 'mvn clean package ' } } } }  
            cleclerc Cyrille Le Clerc made changes -
            Status Open [ 1 ] Closed [ 6 ]
            Assignee Alvaro Lobato [ alobato ] Cyrille Le Clerc [ cleclerc ]
            Resolution Cannot Reproduce [ 5 ]
            Hide
            dantran dan tran added a comment -

            Cyrille Le Clerc could you try the my provided declarative pipeline?

            Show
            dantran dan tran added a comment - Cyrille Le Clerc could you try the my provided declarative pipeline?
            Hide
            dantran dan tran added a comment -

            ran your example 

            node {
            stage('build') {
            docker.image('maven:3.5.0-jdk-8').inside {
            git 'https://github.com/spring-projects/spring-petclinic.git'
            withMaven( mavenLocalRepo: '.repository', globalMavenSettingsConfig : 'maven-global', mavenSettingsConfig : 'maven-user')

            { sh 'mvn clean package' }

            }
            }
            }

            need to add my maven settings to get it going. Still see 'nohup' issue
             

             

             

            Show
            dantran dan tran added a comment - ran your example  node { stage('build') { docker.image('maven:3.5.0-jdk-8').inside { git 'https://github.com/spring-projects/spring-petclinic.git' withMaven( mavenLocalRepo: '.repository', globalMavenSettingsConfig : 'maven-global', mavenSettingsConfig : 'maven-user') { sh 'mvn clean package' } } } } need to add my maven settings to get it going. Still see 'nohup' issue      
            dantran dan tran made changes -
            Resolution Cannot Reproduce [ 5 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            Hide
            dantran dan tran added a comment -

            It turns out my jenkins slave/node configuration has
            Prepare jobs environment checked and Unset System Environment Variables checked

            This is common settings at all of our jenkins slave that I have no control

            I think it is still a bug since it works with using configFileProvider directly

            configFileProvider([
            configFile(fileId: 'maven-user', variable: 'MVN_USER_SETTINGS'),
            configFile(fileId: 'maven-global', variable: 'MVN_GLOBAL_SETTINGS')
            ])

            { sh " mvn ..." }
            Show
            dantran dan tran added a comment - It turns out my jenkins slave/node configuration has Prepare jobs environment checked and Unset System Environment Variables checked This is common settings at all of our jenkins slave that I have no control I think it is still a bug since it works with using configFileProvider directly configFileProvider([ configFile(fileId: 'maven-user', variable: 'MVN_USER_SETTINGS'), configFile(fileId: 'maven-global', variable: 'MVN_GLOBAL_SETTINGS') ]) { sh " mvn ..." }
            Hide
            ncosta Nuno Costa added a comment -

            Alvaro Lobato, can you replicate the issue with this steps?

            Cyrille Le Clerc , I tried your test case on a ubuntu 14.04 build node and it ran ok.
            Running it on a pretty simple Centos7 build node, failed with the nohup error.

            No global environment variables are set on Jenkins "Configure System - Global properties"

            Connecting to the centos buildserver and switch to the jenkins user, I get this results:

            [jenkins@centos7-buildserver-02 ~]$ ll /usr/bin/bash
            -rwxr-xr-x. 1 root root 960472 Dec 7 00:19 /usr/bin/bash
            [jenkins@centos7-buildserver-02 ~]$ ll /usr/bin/sh
            lrwxrwxrwx. 1 root root 4 Dec 15 12:10 /usr/bin/sh -> bash
            [jenkins@centos7-buildserver-02 ~]$ ll /bin/sh
            lrwxrwxrwx. 1 root root 4 Dec 15 12:10 /bin/sh -> bash
            [jenkins@centos7-buildserver-02 ~]$ ll /bin/bash
            -rwxr-xr-x. 1 root root 960472 Dec 7 00:19 /bin/bash
            ...
            [jenkins@centos7-buildserver-02 ~]$ which nohup
            /bin/nohup
            [jenkins@centos7-buildserver-02 ~]$ whereis nohup
            nohup: /usr/bin/nohup /usr/share/man/man1/nohup.1.gz /usr/share/man/man1p/nohup.1p.gz
            [jenkins@centos7-buildserver-02 ~]$ echo $PATH
            /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/var/lib/jenkins/.local/bin:/var/lib/jenkins/bin
            ...
            [jenkins@centos7-buildserver-02 ~]$ which sh
            /bin/sh
            [jenkins@centos7-buildserver-02 ~]$ whereis sh
            sh: /usr/bin/sh /usr/share/man/man1/sh.1.gz /usr/share/man/man1p/sh.1p.gz
            ...
            bash_profile
            PATH=$PATH:$HOME/.local/bin:$HOME/bin

            Adding an echo $PATH inside the build stage but outside the container, I noticed that the /bin was missing from there.
            This PATH value seems to be configured by the Jenkins ssh agent connection and the PATH variable set on jenkins user bash_profile is not used.

            node('centos-buildserver-02') {
            stage('build') {
                    sh 'echo $PATH'
                    sh 'which nohup'
                    sh 'which sh'
                    docker.image('maven:3.5.0-jdk-8').inside {
                        git 'https://github.com/spring-projects/spring-petclinic.git'
                        sh 'echo $PATH'
                        sh 'which nohup'
                        sh 'which sh'
                        withMaven() {
                            sh 'mvn clean package'
                        }
                    }
                }
            }
            

            node agent config from the connection log
            PATH=/usr/local/bin:/usr/bin

            After adding PATH with /usr/local/bin:/usr/bin:/bin as a environment variable on that specific node, on "Node Properties - Environment variables", the build run sucessfully.

            What is strange is that the agent node connection log still misses the added /bin and only shows PATH=/usr/local/bin:/usr/bin after disconnect and reconnect, although the sh'echo $PATH' shows "/usr/local/bin:/usr/bin:/bin"

            [Pipeline] stage
            [Pipeline] { (build)
            [Pipeline] sh
            [withMaven-test2] Running shell script
            + echo /usr/local/bin:/usr/bin:/bin
            /usr/local/bin:/usr/bin:/bin
            [Pipeline] sh
            

            On Ubuntu 14.04, sh is only installed on /bin/sh

            node agent config
            PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

            jenkins@ub14-buildserver-02:~$ echo $PATH
             /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

            Is the behaviour shown on CentOS7 "working as designed" or something that needs to be corrected in Jenkins/Pipeline/ssh agent itself?

            Show
            ncosta Nuno Costa added a comment - Alvaro Lobato , can you replicate the issue with this steps? Cyrille Le Clerc , I tried your test case on a ubuntu 14.04 build node and it ran ok. Running it on a pretty simple Centos7 build node, failed with the nohup error. No global environment variables are set on Jenkins "Configure System - Global properties" Connecting to the centos buildserver and switch to the jenkins user, I get this results: [jenkins@centos7-buildserver-02 ~]$ ll /usr/bin/bash -rwxr-xr-x. 1 root root 960472 Dec 7 00:19 /usr/bin/bash [jenkins@centos7-buildserver-02 ~]$ ll /usr/bin/sh lrwxrwxrwx. 1 root root 4 Dec 15 12:10 /usr/bin/sh -> bash [jenkins@centos7-buildserver-02 ~]$ ll /bin/sh lrwxrwxrwx. 1 root root 4 Dec 15 12:10 /bin/sh -> bash [jenkins@centos7-buildserver-02 ~]$ ll /bin/bash -rwxr-xr-x. 1 root root 960472 Dec 7 00:19 /bin/bash ... [jenkins@centos7-buildserver-02 ~]$ which nohup /bin/nohup [jenkins@centos7-buildserver-02 ~]$ whereis nohup nohup: /usr/bin/nohup /usr/share/man/man1/nohup.1.gz /usr/share/man/man1p/nohup.1p.gz [jenkins@centos7-buildserver-02 ~]$ echo $PATH /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/ var /lib/jenkins/.local/bin:/ var /lib/jenkins/bin ... [jenkins@centos7-buildserver-02 ~]$ which sh /bin/sh [jenkins@centos7-buildserver-02 ~]$ whereis sh sh: /usr/bin/sh /usr/share/man/man1/sh.1.gz /usr/share/man/man1p/sh.1p.gz ... bash_profile PATH=$PATH:$HOME/.local/bin:$HOME/bin Adding an echo $PATH inside the build stage but outside the container, I noticed that the /bin was missing from there. This PATH value seems to be configured by the Jenkins ssh agent connection and the PATH variable set on jenkins user bash_profile is not used. node( 'centos-buildserver-02' ) { stage( 'build' ) { sh 'echo $PATH' sh 'which nohup' sh 'which sh' docker.image( 'maven:3.5.0-jdk-8' ).inside { git 'https: //github.com/spring-projects/spring-petclinic.git' sh 'echo $PATH' sh 'which nohup' sh 'which sh' withMaven() { sh 'mvn clean package ' } } } } node agent config from the connection log PATH=/usr/local/bin:/usr/bin After adding PATH with /usr/local/bin:/usr/bin:/bin as a environment variable on that specific node, on "Node Properties - Environment variables" , the build run sucessfully. What is strange is that the agent node connection log still misses the added /bin and only shows PATH=/usr/local/bin:/usr/bin after disconnect and reconnect, although the sh'echo $PATH' shows "/usr/local/bin:/usr/bin:/bin" [Pipeline] stage [Pipeline] { (build) [Pipeline] sh [withMaven-test2] Running shell script + echo /usr/local/bin:/usr/bin:/bin /usr/local/bin:/usr/bin:/bin [Pipeline] sh On Ubuntu 14.04, sh is only installed on /bin/sh node agent config PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games jenkins@ub14-buildserver-02:~$ echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games Is the behaviour shown on CentOS7 "working as designed" or something that needs to be corrected in Jenkins/Pipeline/ssh agent itself?
            Hide
            stour Stéphane Tournié added a comment - - edited

            I'm getting this error too when I configure a "provided settings.xml" and/or a "provided global settings.xml" in the Maven configuration in tools global configuration page.
            When I change it to "use default Maven settings" and call the withMaven() method without any parameter I no more get the nohup: failed to run command `sh': No such file or directory error.

            But then I'm getting another error that I cannot solve so far: script returned exit code -2.

            Show
            stour Stéphane Tournié added a comment - - edited I'm getting this error too when I configure a "provided settings.xml" and/or a "provided global settings.xml" in the Maven configuration in tools global configuration page. When I change it to "use default Maven settings" and call the withMaven() method without any parameter I no more get the nohup: failed to run command `sh': No such file or directory error. But then I'm getting another error that I cannot solve so far: script returned exit code -2 .
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            Stéphane Tournié can you please provide more details with the pipeline script, the version of the plugin and the logs? This problem seem to be specific to some hosts (centos / rhel) and does not seem to happen on Ubuntu.

            Show
            cleclerc Cyrille Le Clerc added a comment - Stéphane Tournié can you please provide more details with the pipeline script, the version of the plugin and the logs? This problem seem to be specific to some hosts (centos / rhel) and does not seem to happen on Ubuntu.
            Hide
            stour Stéphane Tournié added a comment -

            Here is my pipeline script:

            node('docker-installed') {
              docker.image('maven:3.3.9-jdk-8-alpine').inside() {
                git branch: "${branch}", url: "$url"
                withMaven() {
                  sh "mvn -f workspace/project/pom.xml clean install -s $settingsXml"
                }
              }
            }
            

            I'm running Jenkins 2.64 and pipeline-maven-plugin 2.2.0. I'm running Jenkins in a Docker container and I use a remote SSH slave that has Docker installed to execute my pipelines that require Docker.

            Here are the logs of an execution of this stage.

            Show
            stour Stéphane Tournié added a comment - Here is my pipeline script: node( 'docker-installed' ) { docker.image( 'maven:3.3.9-jdk-8-alpine' ).inside() { git branch: "${branch}" , url: "$url" withMaven() { sh "mvn -f workspace/project/pom.xml clean install -s $settingsXml" } } } I'm running Jenkins 2.64 and pipeline-maven-plugin 2.2.0. I'm running Jenkins in a Docker container and I use a remote SSH slave that has Docker installed to execute my pipelines that require Docker. Here are the logs of an execution of this stage.
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            dan tran, Nuno Costa, Stéphane Tournié, Test User can you please confirm the detailed version of the Operating System of your build agent and the version of the docker engine on this agent? The problem seem to only happen on RHEL/CentOS 6 and 7 and on some versions of the docker engine

             

            Nuno Costa I found above that you mentioned "CentOS7 + docker 1.13.1", can you please confirm?

            Show
            cleclerc Cyrille Le Clerc added a comment - dan tran , Nuno Costa , Stéphane Tournié , Test User can you please confirm the detailed version of the Operating System of your build agent and the version of the docker engine on this agent? The problem seem to only happen on RHEL/CentOS 6 and 7 and on some versions of the docker engine   Nuno Costa I found above that you mentioned "CentOS7 + docker 1.13.1", can you please confirm?
            cleclerc Cyrille Le Clerc made changes -
            Summary Unable to use withMaven step inside docker container Unable to use withMaven() step inside docker container for old versions of Docker
            dantran dan tran made changes -
            Attachment Capture.PNG [ 38674 ]
            Hide
            dantran dan tran added a comment - - edited

            Mine is sles12sp2/docker 1.12.6. Only happens if I unset system environment variable at node configuration, see

            Show
            dantran dan tran added a comment - - edited Mine is sles12sp2/docker 1.12.6. Only happens if I unset system environment variable at node configuration, see
            Hide
            stour Stéphane Tournié added a comment -

            My build agent is Centos OS 7 + Docker 17.05.0-ce.

            Show
            stour Stéphane Tournié added a comment - My build agent is Centos OS 7 + Docker 17.05.0-ce.
            Hide
            cleclerc Cyrille Le Clerc added a comment - - edited

            Another platform where the problem is reported: 

            Master: Red Hat Enterprise Linux Server release 6.7 (Santiago)

            Build agent: Red Hat Enterprise Linux Server release 7.3 (Maipo) + docker 1.12.6

             problem does not seem to happen with docker 1.11

             

            Show
            cleclerc Cyrille Le Clerc added a comment - - edited Another platform where the problem is reported:  Master: Red Hat Enterprise Linux Server release 6.7 (Santiago) Build agent: Red Hat Enterprise Linux Server release 7.3 (Maipo) + docker 1.12.6  problem does not seem to happen with docker 1.11  
            Hide
            ncosta Nuno Costa added a comment - - edited

            Cyrille Le Clerc, On Production, we are on CentOS7 with below package versions. We are using kernel 4.x because we need it to use the docker overlay driver.

            docker-engine.x86_64 >> 1.13.1-1.el7.centos >> @docker-main
            kernel-lt.x86_64 >> 4.4.50-1.el7.elrepo >> @elrepo-kernel
            

            I also have the same issue on my Test Build Server VM with CentOS7 with:

            docker-ce.x86_64 >> 17.06.0.ce-1.el7.centos >> @docker-ce-stable
            kernel.x86_64 >> 3.10.0-514.26.1.el7 >> @updates
            kernel-lt.x86_64 >> 4.4.75-1.el7.elrepo >> @elrepo-kernel

            Starting the Test Build Server with kernel 3 or 4, provides the same results, when building:

            nohup: failed to run command ‘sh’: No such file or directory

             

            Show
            ncosta Nuno Costa added a comment - - edited Cyrille Le Clerc , On Production, we are on CentOS7 with below package versions. We are using kernel 4.x because we need it to use the docker overlay driver. docker-engine.x86_64 >> 1.13.1-1.el7.centos >> @docker-main kernel-lt.x86_64 >> 4.4.50-1.el7.elrepo >> @elrepo-kernel I also have the same issue on my Test Build Server VM with CentOS7 with: docker-ce.x86_64 >> 17.06.0.ce-1.el7.centos >> @docker-ce-stable kernel.x86_64 >> 3.10.0-514.26.1.el7 >> @updates kernel-lt.x86_64 >> 4.4.75-1.el7.elrepo >> @elrepo-kernel Starting the Test Build Server with kernel 3 or 4, provides the same results, when building: nohup: failed to run command ‘sh’: No such file or directory  
            jamesdumay James Dumay made changes -
            Labels blue-pipeline-scrub
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Cyrille Le Clerc
            Path:
            jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java
            http://jenkins-ci.org/commit/pipeline-maven-plugin/0193d7c1b1be95024d35b149d37f050b415fcb20
            Log:
            JENKINS-40484 Better troubleshooting messages

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java http://jenkins-ci.org/commit/pipeline-maven-plugin/0193d7c1b1be95024d35b149d37f050b415fcb20 Log: JENKINS-40484 Better troubleshooting messages
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Cyrille Le Clerc
            Path:
            jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java
            http://jenkins-ci.org/commit/pipeline-maven-plugin/59702c5b9631b80522cbad939a9931d08b05f04a
            Log:
            Merge pull request #65 from jenkinsci/JENKINS-40484

            JENKINS-40484 Better troubleshooting messages

            Compare: https://github.com/jenkinsci/pipeline-maven-plugin/compare/b9bee3c497cc...59702c5b9631

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java http://jenkins-ci.org/commit/pipeline-maven-plugin/59702c5b9631b80522cbad939a9931d08b05f04a Log: Merge pull request #65 from jenkinsci/ JENKINS-40484 JENKINS-40484 Better troubleshooting messages Compare: https://github.com/jenkinsci/pipeline-maven-plugin/compare/b9bee3c497cc...59702c5b9631
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            I have added some log messages as part of 2.5.1 to better understand the value of the PATH environment variable.

            Could you please enable logs at level FINER on org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution and attach the logs generated by a build execution to this ticket.

            The new log messages will help us understand the value of the PATH environment variable:

            Jul 17, 2017 4:26:54 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Options: [...]
            
            Jul 17, 2017 4:26:54 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            env.PATH: /usr/local/...
            
            ...
            
            Jul 17, 2017 4:27:01 PM FINER org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            ExpanderImpl(overrides: {GLOBAL_MVN_SETTINGS=/Users/cyrilleleclerc/...
            
            Jul 17, 2017 4:27:01 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            envOverride: {GLOBAL_MVN_SETTINGS=/Users/cyrilleleclerc/...
            
            Jul 17, 2017 4:27:11 PM FINER org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            ExpanderImpl.expand - env before expand: {__CF_USER_TEXT_ENCODING=...
            
            Jul 17, 2017 4:27:11 PM FINER org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            ExpanderImpl.expand - env after expand: {__CF_USER_TEXT_ENCODING=...
            
            Show
            cleclerc Cyrille Le Clerc added a comment - I have added some log messages as part of 2.5.1 to better understand the value of the PATH environment variable. Could you please enable logs at level FINER on org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution and attach the logs generated by a build execution to this ticket. The new log messages will help us understand the value of the PATH environment variable: Jul 17, 2017 4:26:54 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Options: [...] Jul 17, 2017 4:26:54 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution env.PATH: /usr/local/... ... Jul 17, 2017 4:27:01 PM FINER org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution ExpanderImpl(overrides: {GLOBAL_MVN_SETTINGS=/Users/cyrilleleclerc/... Jul 17, 2017 4:27:01 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution envOverride: {GLOBAL_MVN_SETTINGS=/Users/cyrilleleclerc/... Jul 17, 2017 4:27:11 PM FINER org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution ExpanderImpl.expand - env before expand: {__CF_USER_TEXT_ENCODING=... Jul 17, 2017 4:27:11 PM FINER org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution ExpanderImpl.expand - env after expand: {__CF_USER_TEXT_ENCODING=...
            Hide
            testuser7 Test User added a comment -

            Logs from version 2.5.2:

            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Maven: null
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Jdk: null
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            MavenOpts: 
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Settings Config: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Settings FilePath: 
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Global settings Config: null
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Global settings FilePath: 
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Options: []
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            env.PATH: /usr/local/bin:/usr/bin
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Computer: slave1
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Env: {_=/usr/bin/java, HOME=/var/lib/jenkins, LOGNAME=jenkins, MAIL=/var/mail/jenkins, NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat, OLDPWD=/var/lib/jenkins, PATH=/usr/local/bin:/usr/bin, PWD=/var/lib/jenkins, SELINUX_LEVEL_REQUESTED=, SELINUX_ROLE_REQUESTED=, SELINUX_USE_CURRENT_RANGE=, SHELL=/bin/bash, SHLVL=1, SSH_CLIENT=*.*.*.* * *, SSH_CONNECTION=*.*.*.* * *.*.*.* *, USER=jenkins, XDG_RUNTIME_DIR=/run/user/*, XDG_SESSION_ID=*, XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt}
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Step running within docker.image()
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Setting up maven: null
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Searching for Maven through MAVEN_HOME and M2_HOME environment variables...
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            Calling printenv on docker container...
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            NO maven installation discovered on docker container through MAVEN_HOME and M2_HOME environment variables
            
            Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
            No Maven Installation or MAVEN_HOME found, looking for mvn executable by using which/where command
            
            Show
            testuser7 Test User added a comment - Logs from version 2.5.2: Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Maven: null Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Jdk: null Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution MavenOpts: Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Settings Config: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Settings FilePath: Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Global settings Config: null Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Global settings FilePath: Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Options: [] Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution env.PATH: /usr/local/bin:/usr/bin Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Computer: slave1 Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Env: {_=/usr/bin/java, HOME=/var/lib/jenkins, LOGNAME=jenkins, MAIL=/var/mail/jenkins, NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat, OLDPWD=/var/lib/jenkins, PATH=/usr/local/bin:/usr/bin, PWD=/var/lib/jenkins, SELINUX_LEVEL_REQUESTED=, SELINUX_ROLE_REQUESTED=, SELINUX_USE_CURRENT_RANGE=, SHELL=/bin/bash, SHLVL=1, SSH_CLIENT=*.*.*.* * *, SSH_CONNECTION=*.*.*.* * *.*.*.* *, USER=jenkins, XDG_RUNTIME_DIR=/run/user/*, XDG_SESSION_ID=*, XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt} Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Step running within docker.image() Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Setting up maven: null Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Searching for Maven through MAVEN_HOME and M2_HOME environment variables... Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Calling printenv on docker container... Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution NO maven installation discovered on docker container through MAVEN_HOME and M2_HOME environment variables Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution No Maven Installation or MAVEN_HOME found, looking for mvn executable by using which/where command
            Hide
            cleclerc Cyrille Le Clerc added a comment - - edited

            Detailed logs of an happy path: https://gist.github.com/cyrille-leclerc/c23d403761ac802f7a8955df40631906

            Sep 06, 2017 10:04:23 PM FINEST org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution$ExpanderImpl expand
            ...
            PATH=/home/ubuntu/jenkins-aws-home/workspace/plugins/pipeline-maven-plugin/spring-petclinic/spring-petclinic-in-docker-pipeline-plugin@tmp/withMavene56239cc:
            /usr/local/sbin:
            /usr/local/bin:
            /usr/sbin:
            /usr/bin:
            /sbin:
            /bin:
            /usr/games:
            /usr/local/games
            
            sh 'env'
            ...
            PATH=/usr/local/sbin:
            /usr/local/bin:
            /usr/sbin:
            /usr/bin:
            /sbin:
            /bin
            
            Show
            cleclerc Cyrille Le Clerc added a comment - - edited Detailed logs of an happy path: https://gist.github.com/cyrille-leclerc/c23d403761ac802f7a8955df40631906 Sep 06, 2017 10:04:23 PM FINEST org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution$ExpanderImpl expand ... PATH=/home/ubuntu/jenkins-aws-home/workspace/plugins/pipeline-maven-plugin/spring-petclinic/spring-petclinic-in-docker-pipeline-plugin@tmp/withMavene56239cc: /usr/local/sbin: /usr/local/bin: /usr/sbin: /usr/bin: /sbin: /bin: /usr/games: /usr/local/games sh 'env' ... PATH=/usr/local/sbin: /usr/local/bin: /usr/sbin: /usr/bin: /sbin: /bin
            cleclerc Cyrille Le Clerc made changes -
            Link This issue is related to JENKINS-41339 [ JENKINS-41339 ]
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            Our problem seem to be the same as JENKINS-41339 and JENKINS-41204. I didn't find a path to move forward, the troubleshooting message we have added don't help. I'll ping Jesse Glick to see if he can help us as this seem to be more related to "pipeline core" and "docker-pipeline" with old Docker engines than to be a withMaven specific issue.

            Show
            cleclerc Cyrille Le Clerc added a comment - Our problem seem to be the same as JENKINS-41339 and JENKINS-41204 . I didn't find a path to move forward, the troubleshooting message we have added don't help. I'll ping Jesse Glick to see if he can help us as this seem to be more related to "pipeline core" and "docker-pipeline" with old Docker engines than to be a withMaven specific issue.
            cleclerc Cyrille Le Clerc made changes -
            Remote Link This issue links to "Page (Jenkins Wiki)" [ 17646 ]
            Hide
            cleclerc Cyrille Le Clerc added a comment - - edited

             I have added a "known limitation" to the withMaven plugin page: "withMaven() not supported in "docker.image('xxx').inside{...}" with old versions of the Docker engine

            Show
            cleclerc Cyrille Le Clerc added a comment - - edited  I have added a "known limitation" to the withMaven plugin page: " withMaven() not supported in "docker.image('xxx').inside{...}" with old versions of the Docker engine " 
            Hide
            dantran dan tran added a comment -

            for my case(sles12sp2), same issue found with latest jenkins plugin + docker 17.4

            Show
            dantran dan tran added a comment - for my case(sles12sp2), same issue found with latest jenkins plugin + docker 17.4
            Hide
            jglick Jesse Glick added a comment -

            Cyrille Le Clerc I am not sure what the relation is to the issues you mention. Most likely a bug in docker-workflow, or perhaps JENKINS-39307 or similar. At any rate I am not working on this any more, and generally recommend stopping use of Image.inside at the first sign of trouble.

            Show
            jglick Jesse Glick added a comment - Cyrille Le Clerc I am not sure what the relation is to the issues you mention. Most likely a bug in docker-workflow , or perhaps  JENKINS-39307 or similar. At any rate I am not working on this any more, and generally recommend stopping use of Image.inside at the first sign of trouble.
            ncosta Nuno Costa made changes -
            ncosta Nuno Costa made changes -
            Hide
            ncosta Nuno Costa added a comment -

            I've seen this issue using both docker 1.x and 17.x versions.
            At the moment all buildservers are using 17.x versions.

            Noticed that the the build runs OK on a specific version (container ID 88714384d642) of maven:3.5.0-jdk-8 image.
            According to docker images command, it was created 7 weeks ago.

            Also noticed that with latest image (ID cbf114925530 created ~31 hours ago) and a older one (ID 66091267e43d created 4 months ago), the build fails with the sh error.

            Compared the logs created by Jenkins Logger (as Cyrille Le Clerc mentioned in this comment but I didn't notice anything that could indicate any issue. Both Sucess and Failure builds had env.PATH: /usr/local/bin:/usr/bin

            Attaching console and logger logs from failed and sucessfull builds:
            20170915-v.2.78-JENKINS-40484-CONSOLE-LOG-NOK.txt
            20170915-v.2.78-JENKINS-40484-JENKINS-LOG-NOK.txt

            20170915-v.2.78-JENKINS-40484-CONSOLE-LOG-OK.txt
            20170915-v.2.78-JENKINS-40484-JENKINS-LOG-OK.txt

            What I could see from Console log was the sh command being used from a different location.
            Build finishes OK when sh is called from /usr/bin/sh (image ID 88714384d642) but fails when sh is called from /bin/sh (image ID cbf114925530).
            Used the which sh and whereis sh commands to check.

            Retested today with a fully patched centos7 node and centos7 jenkins 2.78 server(fully updated plugins as well).

            Additionally, tested on a fully patched ubuntu 14.04 server and I don't have any build issues using image ID cbf114925530.

            NOTE: Not sure how we can pull a specific ID of a docker image.
            In my case, I was manually tagging the latest version, adding the ID, since I'm following this issue.
            This way I could check if a new update was made on the image, every time I pulled the image.

            Jenkinsfile used for testing:

            node('mydocker') {
            deleteDir()
            stage('build') {
                    sh 'echo $PATH'
                    sh 'which nohup'
                    sh 'which sh'
                    docker.image('maven:3.5.0-jdk-8-cbf114925530').inside {
                        git 'https://github.com/spring-projects/spring-petclinic.git'
                        sh 'echo $PATH'
                        sh 'which nohup'
                        sh 'which sh'
                        withMaven() {
                            sh 'mvn clean package'
                        }
                    }
                }
            deleteDir()
            }
            
            Show
            ncosta Nuno Costa added a comment - I've seen this issue using both docker 1.x and 17.x versions. At the moment all buildservers are using 17.x versions. Noticed that the the build runs OK on a specific version (container ID 88714384d642) of maven:3.5.0-jdk-8 image. According to docker images command, it was created 7 weeks ago. Also noticed that with latest image (ID cbf114925530 created ~31 hours ago) and a older one (ID 66091267e43d created 4 months ago), the build fails with the sh error. Compared the logs created by Jenkins Logger (as Cyrille Le Clerc mentioned in this comment but I didn't notice anything that could indicate any issue. Both Sucess and Failure builds had env.PATH: /usr/local/bin:/usr/bin Attaching console and logger logs from failed and sucessfull builds: 20170915-v.2.78-JENKINS-40484-CONSOLE-LOG-NOK.txt 20170915-v.2.78-JENKINS-40484-JENKINS-LOG-NOK.txt 20170915-v.2.78-JENKINS-40484-CONSOLE-LOG-OK.txt 20170915-v.2.78-JENKINS-40484-JENKINS-LOG-OK.txt What I could see from Console log was the sh command being used from a different location. Build finishes OK when sh is called from /usr/bin/sh (image ID 88714384d642) but fails when sh is called from /bin/sh (image ID cbf114925530). Used the which sh and whereis sh commands to check. Retested today with a fully patched centos7 node and centos7 jenkins 2.78 server(fully updated plugins as well). Additionally, tested on a fully patched ubuntu 14.04 server and I don't have any build issues using image ID cbf114925530. NOTE: Not sure how we can pull a specific ID of a docker image. In my case, I was manually tagging the latest version, adding the ID, since I'm following this issue. This way I could check if a new update was made on the image, every time I pulled the image. Jenkinsfile used for testing: node( 'mydocker' ) { deleteDir() stage( 'build' ) { sh 'echo $PATH' sh 'which nohup' sh 'which sh' docker.image( 'maven:3.5.0-jdk-8-cbf114925530' ).inside { git 'https: //github.com/spring-projects/spring-petclinic.git' sh 'echo $PATH' sh 'which nohup' sh 'which sh' withMaven() { sh 'mvn clean package ' } } } deleteDir() }
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            Jesse Glick can you please point us to documentation for an alternative to docker.image().inside{}?

            Show
            cleclerc Cyrille Le Clerc added a comment - Jesse Glick can you please point us to documentation for an alternative to docker.image().inside{}?
            Show
            jglick Jesse Glick added a comment - https://github.com/jenkinsci/docker-workflow-plugin/pull/105/files  for example.
            Show
            jglick Jesse Glick added a comment - Or: https://github.com/jglick/ephemeral-docker-agent-plugin  (CC Andrew Bayer , Nicolas De Loof )
            Hide
            abayer Andrew Bayer added a comment -
            Show
            abayer Andrew Bayer added a comment - FYI, https://github.com/jenkinsci/docker-workflow-plugin/pull/97 does seem to resolve this.
            Hide
            abayer Andrew Bayer added a comment -

            Fixed in docker-workflow 1.14, which I'm releasing as we speak.

            Show
            abayer Andrew Bayer added a comment - Fixed in docker-workflow 1.14, which I'm releasing as we speak.
            abayer Andrew Bayer made changes -
            Status Reopened [ 4 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            dantran dan tran added a comment -

            awesome!! thank you very much for this long waited fix

            Show
            dantran dan tran added a comment - awesome!! thank you very much for this long waited fix
            nv035674 Nathan Vahrenberg made changes -
            Link This issue is related to JENKINS-47809 [ JENKINS-47809 ]
            cleclerc Cyrille Le Clerc made changes -
            Link This issue is related to JENKINS-47823 [ JENKINS-47823 ]
            Hide
            timdowney Tim Downey added a comment -

            Hello, the fix for this in Docker Pipeline 1.14 seems to have broken things.  I've been using withMaven & Docker successfully for a while now and things have been working up until 1.13.

            Here's what I've been doing in the builds.  In this case, I'm using a generated settings.xml, but I have another build that just passes in an identifier for a managed settings.xml and I get the same results.

             

            // I'm generating this elsewhere
            def settingsFilePath = 'target/generated-settings.xml'
            
            // simplified for brevity
            withDockerContainer('maven:3.5.0-jdk-8-alpine') {
                withMaven(mavenSettingsFilePath: settingsFilePath) {
                   sh 'mvn -U -X clean install'
                }
            }

            Since upgrading to 1.14, attempts to set the settings.xml fail because of what appears to be changes to the user home.

            Here's what I'm seeing.  Notice that the debug logic shows in the target/generated-settings.xml as in use, but before below in the true Maven debug, you can see that user settings got set to ?./m2/settings.xml instead.

            This did not happen with 1.13.

             

            $ docker run -t -d -u 1000:482 -w /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy --volumes-from e24fc4ecfb9e762918e653126582a67aa6c907086b279d581d1b998f4915aa3d -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat maven:3.5.0-jdk-8-alpine
            [Pipeline] {
            [Pipeline] withMaven
            [withMaven] Options: [org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher[disabled=true], org.jenkinsci.plugins.pipeline.maven.publishers.TasksScannerPublisher[disabled=true]]
            [withMaven] Available options: 
            [withMaven] use JDK installation provided by the build agent
            $ docker exec 145d411e24f49e2140dd3a1a1fb9db26dfe0fa5e7d61553528bcd946e9478914 env printenv MAVEN_HOME
            [withMaven] use Maven installation provided by the build agent with the environment variable MAVEN_HOME=/usr/share/maven
            [withMaven] use Maven settings provided on the build agent 'target/generated-settings.xml' 
            [Pipeline] {
            [Pipeline] echo
            Downloading com.oneadp.taas.apigateway:taas-apigateway-payroll:$VERSION to temp folder
            [Pipeline] sh
            [deploy] Running shell script
            + mvn -U -X org.apache.maven.plugins:maven-dependency-plugin:3.0.1:copy -Dartifact=com.oneadp.taas.apigateway:taas-apigateway-payroll:1.0.2-SNAPSHOT:zip -DoutputDirectory=target -Dmdep.useBaseVersion=true
            Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e" 
            Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T19:39:06Z)
            Maven home: /usr/share/maven
            Java version: 1.8.0_131, vendor: Oracle Corporation
            Java home: /usr/lib/jvm/java-1.8-openjdk/jre
            Default locale: en_US, platform encoding: UTF-8
            OS name: "linux", version: "3.10.0-514.21.2.el7.x86_64", arch: "amd64", family: "unix"
            [DEBUG]   Included /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/pipeline-maven-spy.jar
            [DEBUG] Populating class realm maven.ext
            [DEBUG]   Included /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/pipeline-maven-spy.jar
            [jenkins-maven-event-spy] INFO generate /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/maven-spy-20171106-211420-4734323727761713157281.log.tmp ...
            [DEBUG] Created new class realm maven.api
            [DEBUG] Importing foreign packages into class realm maven.api
            [DEBUG]   Imported: javax.enterprise.inject.* < maven.ext
            [DEBUG]   Imported: javax.enterprise.util.* < maven.ext
            
            *** lots clipped ***
            
            [DEBUG]   Imported: org.slf4j.* < maven.ext
            [DEBUG]   Imported: org.slf4j.helpers.* < maven.ext
            [DEBUG]   Imported: org.slf4j.spi.* < maven.ext
            [DEBUG] Populating class realm maven.api
            [INFO] Error stacktraces are turned on.
            [DEBUG] Message scheme: color
            [DEBUG] Message styles: debug info warning error success failure strong mojo project
            [DEBUG] Reading global settings from /usr/share/maven/conf/settings.xml
            [DEBUG] Reading user settings from ?/.m2/settings.xml
            [DEBUG] Reading global toolchains from /usr/share/maven/conf/toolchains.xml
            [DEBUG] Reading user toolchains from ?/.m2/toolchains.xml
            [DEBUG] Using local repository at /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy/?/.m2/repository
            [DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10.0 for /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy/?/.m2/repository
            [INFO] Scanning for projects...
            [DEBUG] Extension realms for project org.apache.maven:standalone-pom:pom:1: (none)
            [DEBUG] Looking up lifecycle mappings for packaging pom from ClassRealm[maven.ext, parent: ClassRealm[plexus.core, parent: null]]
            [DEBUG] Using transporter WagonTransporter with priority -1.0 for https://repo.maven.apache.org/maven2
            [DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://repo.maven.apache.org/maven2
            Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/3.0.1/maven-dependency-plugin-3.0.1.pom
            [DEBUG] Writing tracking file /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy/?/.m2/repository/org/apache/maven/plugins/maven-dependency-plugin/3.0.1/maven-dependency-plugin-3.0.1.pom.lastUpdated
            [INFO] ------------------------------------------------------------------------
            [INFO] BUILD FAILURE
            [INFO] ------------------------------------------------------------------------
            [INFO] Total time: 02:07 min
            

            This issue may be related

            https://stackoverflow.com/questions/12465186/why-does-maven-use-as-my-home-directory

            Thanks for any suggestions.

             

             

             

            Show
            timdowney Tim Downey added a comment - Hello, the fix for this in Docker Pipeline 1.14 seems to have broken things.  I've been using withMaven & Docker successfully for a while now and things have been working up until 1.13. Here's what I've been doing in the builds.  In this case, I'm using a generated settings.xml, but I have another build that just passes in an identifier for a managed settings.xml and I get the same results.   // I'm generating this elsewhere def settingsFilePath = 'target/generated-settings.xml' // simplified for brevity withDockerContainer( 'maven:3.5.0-jdk-8-alpine' ) { withMaven(mavenSettingsFilePath: settingsFilePath) { sh 'mvn -U -X clean install' } } Since upgrading to 1.14, attempts to set the settings.xml fail because of what appears to be changes to the user home. Here's what I'm seeing.  Notice that the debug logic shows in the target/generated-settings.xml as in use, but before below in the true Maven debug, you can see that user settings got set to ?./m2/settings.xml instead. This did not happen with 1.13.   $ docker run -t -d -u 1000:482 -w /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy --volumes-from e24fc4ecfb9e762918e653126582a67aa6c907086b279d581d1b998f4915aa3d -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat maven:3.5.0-jdk-8-alpine [Pipeline] { [Pipeline] withMaven [withMaven] Options: [org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher[disabled=true], org.jenkinsci.plugins.pipeline.maven.publishers.TasksScannerPublisher[disabled=true]] [withMaven] Available options: [withMaven] use JDK installation provided by the build agent $ docker exec 145d411e24f49e2140dd3a1a1fb9db26dfe0fa5e7d61553528bcd946e9478914 env printenv MAVEN_HOME [withMaven] use Maven installation provided by the build agent with the environment variable MAVEN_HOME=/usr/share/maven [withMaven] use Maven settings provided on the build agent 'target/generated-settings.xml' [Pipeline] { [Pipeline] echo Downloading com.oneadp.taas.apigateway:taas-apigateway-payroll:$VERSION to temp folder [Pipeline] sh [deploy] Running shell script + mvn -U -X org.apache.maven.plugins:maven-dependency-plugin:3.0.1:copy -Dartifact=com.oneadp.taas.apigateway:taas-apigateway-payroll:1.0.2-SNAPSHOT:zip -DoutputDirectory=target -Dmdep.useBaseVersion=true Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e" Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T19:39:06Z) Maven home: /usr/share/maven Java version: 1.8.0_131, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-1.8-openjdk/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-514.21.2.el7.x86_64", arch: "amd64", family: "unix" [DEBUG] Included /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/pipeline-maven-spy.jar [DEBUG] Populating class realm maven.ext [DEBUG] Included /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/pipeline-maven-spy.jar [jenkins-maven-event-spy] INFO generate /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/maven-spy-20171106-211420-4734323727761713157281.log.tmp ... [DEBUG] Created new class realm maven.api [DEBUG] Importing foreign packages into class realm maven.api [DEBUG] Imported: javax.enterprise.inject.* < maven.ext [DEBUG] Imported: javax.enterprise.util.* < maven.ext *** lots clipped *** [DEBUG] Imported: org.slf4j.* < maven.ext [DEBUG] Imported: org.slf4j.helpers.* < maven.ext [DEBUG] Imported: org.slf4j.spi.* < maven.ext [DEBUG] Populating class realm maven.api [INFO] Error stacktraces are turned on. [DEBUG] Message scheme: color [DEBUG] Message styles: debug info warning error success failure strong mojo project [DEBUG] Reading global settings from /usr/share/maven/conf/settings.xml [DEBUG] Reading user settings from ?/.m2/settings.xml [DEBUG] Reading global toolchains from /usr/share/maven/conf/toolchains.xml [DEBUG] Reading user toolchains from ?/.m2/toolchains.xml [DEBUG] Using local repository at /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy/?/.m2/repository [DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10.0 for /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy/?/.m2/repository [INFO] Scanning for projects... [DEBUG] Extension realms for project org.apache.maven:standalone-pom:pom:1: (none) [DEBUG] Looking up lifecycle mappings for packaging pom from ClassRealm[maven.ext, parent: ClassRealm[plexus.core, parent: null]] [DEBUG] Using transporter WagonTransporter with priority -1.0 for https://repo.maven.apache.org/maven2 [DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://repo.maven.apache.org/maven2 Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/3.0.1/maven-dependency-plugin-3.0.1.pom [DEBUG] Writing tracking file /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy/?/.m2/repository/org/apache/maven/plugins/maven-dependency-plugin/3.0.1/maven-dependency-plugin-3.0.1.pom.lastUpdated [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:07 min This issue may be related https://stackoverflow.com/questions/12465186/why-does-maven-use-as-my-home-directory Thanks for any suggestions.      
            timdowney Tim Downey made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            Tim Downey, dan tran can you please test 3.0.3-beta-1 and prepend the '$MVN_CMD' variable to the '$PATH' variable in each 'sh' step that invokes 'mvn'?

            https://github.com/jenkinsci/pipeline-maven-plugin/releases/tag/pipeline-maven-3.0.3-beta-1

             

            node("linux-agent-with-docker") {
                docker.image('maven').inside {
                    withMaven(mavenSettingsConfig: 'my-maven-settings') {
            
                        git "https://github.com/cyrille-leclerc/my-jar.git"
            
                        // prepend the $MVN_CMD variable to the $PATH environment variable in each "sh" step that invokes mvn 
                        sh 'export PATH=$MVN_CMD:$PATH && mvn help:effective-settings'
                    }
                }
            }
            
            Show
            cleclerc Cyrille Le Clerc added a comment - Tim Downey , dan tran can you please test 3.0.3-beta-1 and prepend the '$MVN_CMD' variable to the '$PATH' variable in each 'sh' step that invokes 'mvn'? https://github.com/jenkinsci/pipeline-maven-plugin/releases/tag/pipeline-maven-3.0.3-beta-1   node( "linux-agent-with-docker" ) { docker.image( 'maven' ).inside { withMaven(mavenSettingsConfig: 'my-maven-settings' ) { git "https: //github.com/cyrille-leclerc/my-jar.git" // prepend the $MVN_CMD variable to the $PATH environment variable in each "sh" step that invokes mvn sh 'export PATH=$MVN_CMD:$PATH && mvn help:effective-settings' } } }
            Hide
            timdowney Tim Downey added a comment -

            Ok, running with 3.0.3-beta-1 and Docker Pipeline 1.14, I'm still showing the same error.  It is ignoring my configured settings.xml.  You can see it getting hung up at the bottom.  I'm behind a corporate proxy and without the settings.xml pointing at my Artifactory, I can't pull anything in.

            I see the warning about sh and exporting the PATH.  I'm not going to have to do that in the future, am I?  That's would be a breaking change across a lot of builds and seems fairly ugly as well.

             

            [Pipeline] node
            Running on bf3998a459f6-ba9afd01 in /home/jenkins/workspace/JENKINS-40484
            [Pipeline] {
            [Pipeline] sh
            [JENKINS-40484] Running shell script
            + docker inspect -f . maven
            
            Error: No such object: maven
            [Pipeline] sh
            [JENKINS-40484] Running shell script
            + docker pull maven
            Using default tag: latest
            latest: Pulling from library/maven
            3e17c6eae66c: Already exists
            fdfb54153de7: Pulling fs layer
            
            ** snip **
            
            9b99cd964015: Pull complete
            9d5978195527: Pull complete
            Digest: sha256:e7c0e5eb42306cbf81aebf793f9bdf15b5e5e8702c683cba626dfb161db25c4d
            Status: Downloaded newer image for maven:latest
            [Pipeline] withDockerContainer
            bf3998a459f6-ba9afd01 seems to be running inside container bf3998a459f6307d84239443d665267c974714d868d90e35a560972f7db52e2b
            $ docker run -t -d -u 1000:482 -w /home/jenkins/workspace/JENKINS-40484 --volumes-from bf3998a459f6307d84239443d665267c974714d868d90e35a560972f7db52e2b -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat maven
            [Pipeline] {
            [withMaven] Options: []
            [withMaven] Available options: 
            [withMaven] WARNING: "withMaven(){...}" step running within "docker.image('image').inside {...}". Since the Docker Pipeline Plugin version 1.14, you MUST prepend the 'MVN_CMD' environment variable to the 'PATH' environment variable in every 'sh' step that invokes 'mvn'. See Pipeline Maven Plugin FAQ.
            [withMaven] Sample:
            [withMaven]    sh "export PATH=$MVN_CMD:$PATH && mvn ..."
            [withMaven] use JDK installation provided by the build agent
            [withMaven] use Maven settings provided by the Jenkins Managed Configuration File 'my-maven-settings' 
            [withMaven] use Maven settings.xml 'my-maven-settings' with NO Maven servers credentials provided by Jenkins
            $ docker exec 06b6784e5f8ec73ccca484bc3ba35e7fbc6514a89265331fff8c416b356e0941 env printenv MAVEN_HOME
            [withMaven] use Maven installation provided by the build agent with the environment variable MAVEN_HOME=/usr/share/maven
            [Pipeline] withMaven
            [Pipeline] {
            [Pipeline] git
            Cloning the remote Git repository
            Cloning repository https://github.com/cyrille-leclerc/my-jar.git
             > git init /home/jenkins/workspace/JENKINS-40484 # timeout=10
            Fetching upstream changes from https://github.com/cyrille-leclerc/my-jar.git
             > git --version # timeout=10
             > git fetch --tags --progress https://github.com/cyrille-leclerc/my-jar.git +refs/heads/*:refs/remotes/origin/*
             > git config remote.origin.url https://github.com/cyrille-leclerc/my-jar.git # timeout=10
             > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
             > git config remote.origin.url https://github.com/cyrille-leclerc/my-jar.git # timeout=10
            Fetching upstream changes from https://github.com/cyrille-leclerc/my-jar.git
             > git fetch --tags --progress https://github.com/cyrille-leclerc/my-jar.git +refs/heads/*:refs/remotes/origin/*
             > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
             > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
            Checking out Revision 1a00e567ba6fbc47f7757c5cb160fa4483fbd33b (refs/remotes/origin/master)
             > git config core.sparsecheckout # timeout=10
             > git checkout -f 1a00e567ba6fbc47f7757c5cb160fa4483fbd33b
             > git branch -a -v --no-abbrev # timeout=10
             > git checkout -b master 1a00e567ba6fbc47f7757c5cb160fa4483fbd33b
            Commit message: "[maven-release-plugin] prepare for next development iteration"
            First time build. Skipping changelog.
            [Pipeline] sh
            [JENKINS-40484] Running shell script
            + export PATH=/home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6/mvn:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
            + mvn help:effective-settings
            Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6" 
            [jenkins-maven-event-spy] INFO generate /home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6/maven-spy-20171107-215042-2478671379137199221991.log.tmp ...
            [INFO] Scanning for projects...
            Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-release-plugin/2.5.3/maven-release-plugin-2.5.3.pom

             

            Show
            timdowney Tim Downey added a comment - Ok, running with 3.0.3-beta-1 and Docker Pipeline 1.14, I'm still showing the same error.  It is ignoring my configured settings.xml.  You can see it getting hung up at the bottom.  I'm behind a corporate proxy and without the settings.xml pointing at my Artifactory, I can't pull anything in. I see the warning about sh and exporting the PATH.  I'm not going to have to do that in the future, am I?  That's would be a breaking change across a lot of builds and seems fairly ugly as well.   [Pipeline] node Running on bf3998a459f6-ba9afd01 in /home/jenkins/workspace/JENKINS-40484 [Pipeline] { [Pipeline] sh [JENKINS-40484] Running shell script + docker inspect -f . maven Error: No such object: maven [Pipeline] sh [JENKINS-40484] Running shell script + docker pull maven Using default tag: latest latest: Pulling from library/maven 3e17c6eae66c: Already exists fdfb54153de7: Pulling fs layer ** snip ** 9b99cd964015: Pull complete 9d5978195527: Pull complete Digest: sha256:e7c0e5eb42306cbf81aebf793f9bdf15b5e5e8702c683cba626dfb161db25c4d Status: Downloaded newer image for maven:latest [Pipeline] withDockerContainer bf3998a459f6-ba9afd01 seems to be running inside container bf3998a459f6307d84239443d665267c974714d868d90e35a560972f7db52e2b $ docker run -t -d -u 1000:482 -w /home/jenkins/workspace/JENKINS-40484 --volumes-from bf3998a459f6307d84239443d665267c974714d868d90e35a560972f7db52e2b -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat maven [Pipeline] { [withMaven] Options: [] [withMaven] Available options: [withMaven] WARNING: "withMaven(){...}" step running within "docker.image('image').inside {...}". Since the Docker Pipeline Plugin version 1.14, you MUST prepend the 'MVN_CMD' environment variable to the 'PATH' environment variable in every 'sh' step that invokes 'mvn'. See Pipeline Maven Plugin FAQ. [withMaven] Sample: [withMaven] sh "export PATH=$MVN_CMD:$PATH && mvn ..." [withMaven] use JDK installation provided by the build agent [withMaven] use Maven settings provided by the Jenkins Managed Configuration File 'my-maven-settings' [withMaven] use Maven settings.xml 'my-maven-settings' with NO Maven servers credentials provided by Jenkins $ docker exec 06b6784e5f8ec73ccca484bc3ba35e7fbc6514a89265331fff8c416b356e0941 env printenv MAVEN_HOME [withMaven] use Maven installation provided by the build agent with the environment variable MAVEN_HOME=/usr/share/maven [Pipeline] withMaven [Pipeline] { [Pipeline] git Cloning the remote Git repository Cloning repository https://github.com/cyrille-leclerc/my-jar.git > git init /home/jenkins/workspace/JENKINS-40484 # timeout=10 Fetching upstream changes from https://github.com/cyrille-leclerc/my-jar.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/cyrille-leclerc/my-jar.git +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url https://github.com/cyrille-leclerc/my-jar.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/cyrille-leclerc/my-jar.git # timeout=10 Fetching upstream changes from https://github.com/cyrille-leclerc/my-jar.git > git fetch --tags --progress https://github.com/cyrille-leclerc/my-jar.git +refs/heads/*:refs/remotes/origin/* > git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 1a00e567ba6fbc47f7757c5cb160fa4483fbd33b (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 1a00e567ba6fbc47f7757c5cb160fa4483fbd33b > git branch -a -v --no-abbrev # timeout=10 > git checkout -b master 1a00e567ba6fbc47f7757c5cb160fa4483fbd33b Commit message: "[maven-release-plugin] prepare for next development iteration" First time build. Skipping changelog. [Pipeline] sh [JENKINS-40484] Running shell script + export PATH=/home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6/mvn:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + mvn help:effective-settings Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6" [jenkins-maven-event-spy] INFO generate /home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6/maven-spy-20171107-215042-2478671379137199221991.log.tmp ... [INFO] Scanning for projects... Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-release-plugin/2.5.3/maven-release-plugin-2.5.3.pom  
            Hide
            dantran dan tran added a comment -

            I cant test with beta ATM, however, the fix looks good on my sample basic declarative pipeline. I need do

            here is the result of your request

            [INFO]
            Effective user-specific configuration settings:

            <?xml version="1.0" encoding="UTF-8"?>
            <!-- ====================================================================== -->
            <!-- -->
            <!-- Generated by Maven Help Plugin on 2017-11-08T01:04:15 -->
            <!-- See: http://maven.apache.org/plugins/maven-help-plugin/ -->
            <!-- -->
            <!-- ====================================================================== -->

            <!-- ====================================================================== -->
            <!-- -->
            <!-- Effective Settings for '?' on '58a31892ecc5' -->
            <!-- -->
            <!-- ====================================================================== -->

            <settings xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
            <localRepository xmlns="http://maven.apache.org/SETTINGS/1.1.0">/space/jenkins/skysandbox/workspace/pipeline-test/?/.m2/repository</localRepository>
            <pluginGroups xmlns="http://maven.apache.org/SETTINGS/1.1.0">
            <pluginGroup>org.apache.maven.plugins</pluginGroup>
            <pluginGroup>org.codehaus.mojo</pluginGroup>
            </pluginGroups>
            </settings>

            [INFO] ------------------------------------------------------------------------
            [INFO] BUILD SUCCESS
            [INFO] ------------------------------------------------------------------------
            [INFO] Total time: 28.352 s

            Show
            dantran dan tran added a comment - I cant test with beta ATM, however, the fix looks good on my sample basic declarative pipeline. I need do here is the result of your request [INFO] Effective user-specific configuration settings: <?xml version="1.0" encoding="UTF-8"?> <!-- ====================================================================== --> <!-- --> <!-- Generated by Maven Help Plugin on 2017-11-08T01:04:15 --> <!-- See: http://maven.apache.org/plugins/maven-help-plugin/ --> <!-- --> <!-- ====================================================================== --> <!-- ====================================================================== --> <!-- --> <!-- Effective Settings for '?' on '58a31892ecc5' --> <!-- --> <!-- ====================================================================== --> <settings xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd "> <localRepository xmlns="http://maven.apache.org/SETTINGS/1.1.0">/space/jenkins/skysandbox/workspace/pipeline-test/?/.m2/repository</localRepository> <pluginGroups xmlns="http://maven.apache.org/SETTINGS/1.1.0"> <pluginGroup>org.apache.maven.plugins</pluginGroup> <pluginGroup>org.codehaus.mojo</pluginGroup> </pluginGroups> </settings> [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 28.352 s
            Hide
            timdowney Tim Downey added a comment -

            dan tran and Cyrille Le Clerc – I'm not sure I understand the original problem that led to this issue.  I've been using withDockerContainer & withMaven for months with no issue.  What is the problem that led to this change?

            Since this change any attempt to change the configured settings.xml is ignored and instead goes looking in <workspace>/?/.m2.

            Cyrille Le Clerc – In trying your example, you can see that  [withMaven] is attempting to configure things properly

            [withMaven] use Maven settings.xml 'my-maven-settings' with NO Maven servers credentials provided by Jenkins

            but later when Maven actually runs, this is ignored

            [DEBUG] Reading user settings from ?/.m2/settings.xml

             

            Show
            timdowney Tim Downey added a comment - dan tran and Cyrille Le Clerc – I'm not sure I understand the original problem that led to this issue.  I've been using withDockerContainer & withMaven for months with no issue.  What is the problem that led to this change? Since this change any attempt to change the configured settings.xml is ignored and instead goes looking in <workspace>/?/.m2. Cyrille Le Clerc – In trying your example, you can see that  [withMaven] is attempting to configure things properly [withMaven] use Maven settings.xml 'my-maven-settings' with NO Maven servers credentials provided by Jenkins but later when Maven actually runs, this is ignored [DEBUG] Reading user settings from ?/.m2/settings.xml  
            Hide
            dantran dan tran added a comment -

            i can confirm that my build does not see config provider settings

            Show
            dantran dan tran added a comment - i can confirm that my build does not see config provider settings
            Show
            cleclerc Cyrille Le Clerc added a comment - - edited Please follow-up on JENKINS-47823 . See my comment https://issues.jenkins-ci.org/browse/JENKINS-47823?focusedCommentId=319156&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-319156
            jamesdumay James Dumay made changes -
            Link This issue relates to JENKINS-48050 [ JENKINS-48050 ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal CD-206 (Web Link)" [ 18982 ]
            Hide
            stodorov Steve Todorov added a comment -

            Cyrille Le Clerc I would like to start by saying that really appreciate your work and efforts in providing a fix/workaround for this issue. 

            However, I'm quite disappointed by this change. When we upgraded back in November 2017 (JENKINS-47805) all of our builds turned red because of the `$PATH` changes in the Docker Pipeline plugin. This makes me doubt what's the purpose of using `withMaven` if it can't do the things it's supposed to do and we're constantly having to use "workarounds"? To name just a few - we need to always have a `post -> always` block that ensures `jUnit` records the tests, now we have to use `configFileProvider` for the `settings.xml`, maybe use the mvn wrapper script... We could just as "easily" add 4 or 5 more pipeline declarations and do things manually instead of using `withMaven`. So what's exactly does `withMaven` do other than decorate the syntax?

            Show
            stodorov Steve Todorov added a comment - Cyrille Le Clerc I would like to start by saying that really appreciate your work and efforts in providing a fix/workaround for this issue.  However, I'm quite disappointed by this change. When we upgraded back in November 2017 ( JENKINS-47805 ) all of our builds turned red because of the `$PATH` changes in the Docker Pipeline plugin. This makes me doubt what's the purpose of using `withMaven` if it can't do the things it's supposed to do and we're constantly having to use "workarounds"? To name just a few - we need to always have a `post -> always` block that ensures `jUnit` records the tests, now we have to use `configFileProvider` for the `settings.xml`, maybe use the mvn wrapper script... We could just as "easily" add 4 or 5 more pipeline declarations and do things manually instead of using `withMaven`. So what's exactly does `withMaven` do other than decorate the syntax?
            Hide
            acejam Joshua Noble added a comment - - edited

            I hit this issue again today for the second time. Docker Pipeline 1.14 is unusable and causes `withMaven` to break when used from within a Docker container. Using containers is one of the best features of pipelines, so this is pretty critical functionality IMO.

            Show
            acejam Joshua Noble added a comment - - edited I hit this issue again today for the second time. Docker Pipeline 1.14 is unusable and causes `withMaven` to break when used from within a Docker container. Using containers is one of the best features of pipelines, so this is pretty critical functionality IMO.
            allan_burdajewicz Allan BURDAJEWICZ made changes -
            Link This issue relates to JENKINS-43590 [ JENKINS-43590 ]
            Hide
            acejam Joshua Noble added a comment -

            Any updates on this?

            Show
            acejam Joshua Noble added a comment - Any updates on this?
            Hide
            stodorov Steve Todorov added a comment - - edited

            ++Joshua Noble try reverting to an older version of Docker Pipeline. I have been using the older version since this issue hit me in JENKINS-47805. It seems that you can fix this either by using a mvnw, downgrade to an older version of Docker Pipeline or wait for JENKINS-48050 which looks like it will fix it for good.

            Show
            stodorov Steve Todorov added a comment - - edited ++ Joshua Noble try reverting to an older version of Docker Pipeline. I have been using the older version since this issue hit me in JENKINS-47805 . It seems that you can fix this either by using a mvnw, downgrade to an older version of Docker Pipeline or wait for JENKINS-48050  which looks like it will fix it for good.
            Hide
            acejam Joshua Noble added a comment -

            I spent a good amount of time looking into this and came up with an acceptable workaround that works for my team. We have hundreds of branches across hundreds of repos, so updating all of the Java/Maven related Jenkinsfiles (on every branch) to use $MVN_CMD instead of "mvn" was not going to happen anytime soon. We also had several pending plugin updates, many of which required the latest version of the pipeline-maven plugin. In our case, we have a private internal Docker image that we use for all maven builds. It's forked off the official Docker Hub Maven image and includes a few additional tools and packages that we need.

            What I ended up doing was creating a bash script that simply wraps the "mvn" command and checks for existence of $MVN_CMD. If $MVN_CMD exists, it calls that and passes along the same parameters. If $MVN_CMD doesn't exist, it simply calls the real maven binary located at /usr/bin/mvn. This bash script lives at /usr/local/bin/mvn, which exists in our $PATH before the real maven binary, which is /usr/bin/mvn.

            Simply create a bash script like below, and ensure it has execute permissions: (chmod +x mvn.sh) 

            #!/bin/bash
            
            if [[ -v MVN_CMD ]]; then
              $MVN_CMD "$@"
            else
              /usr/bin/mvn "$@"
            fi
            

            Then inside of your Dockerfile, add the following instruction:

            COPY mvn.sh /usr/local/bin/mvn
            

            This has been tested with the maven:3.5.3-jdk-8 Docker image.

            Show
            acejam Joshua Noble added a comment - I spent a good amount of time looking into this and came up with an acceptable workaround that works for my team. We have hundreds of branches across hundreds of repos, so updating all of the Java/Maven related Jenkinsfiles (on every branch) to use $MVN_CMD instead of "mvn" was not going to happen anytime soon. We also had several pending plugin updates, many of which required the latest version of the pipeline-maven plugin. In our case, we have a private internal Docker image that we use for all maven builds. It's forked off the official Docker Hub Maven image and includes a few additional tools and packages that we need. What I ended up doing was creating a bash script that simply wraps the "mvn" command and checks for existence of $MVN_CMD. If $MVN_CMD exists, it calls that and passes along the same parameters. If $MVN_CMD doesn't exist, it simply calls the real maven binary located at /usr/bin/mvn. This bash script lives at /usr/local/bin/mvn, which exists in our $PATH before the real maven binary, which is /usr/bin/mvn. Simply create a bash script like below, and ensure it has execute permissions: (chmod +x mvn.sh)   #!/bin/bash if [[ -v MVN_CMD ]]; then $MVN_CMD "$@" else /usr/bin/mvn "$@" fi Then inside of your Dockerfile, add the following instruction: COPY mvn.sh /usr/local/bin/mvn This has been tested with the maven:3.5.3-jdk-8 Docker image.
            Hide
            timdowney Tim Downey added a comment -

            Thanks for the writeup Joshua Noble – That's pretty much exactly what Josh Trow and I have done to deal with the problem.

            Show
            timdowney Tim Downey added a comment - Thanks for the writeup Joshua Noble – That's pretty much exactly what Josh Trow and I have done to deal with the problem.

              People

              • Assignee:
                cleclerc Cyrille Le Clerc
                Reporter:
                testuser7 Test User
              • Votes:
                11 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated: