Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49063

Jenkins on Docker (LTS) running locally cannot build Pipelines from local repos

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Labels:
      None
    • Environment:
      * Jenkins 2.89.3 (a container spun up from a recent "jenkinsci/blueocean" Docker image update), running on ...
      * Docker engine 17.12.0-ce, which in turn is running on ...
      * macOS High Sierra 10.13.2
    • Similar Issues:

      Description

      The LTS release of Jenkins on Docker (installed on a user's local computer) looks like it can no longer build Pipelines from locally cloned repos.

      We have hundreds of readers following the Tutorials in the Jenkins User Documentation like this one. (I.e. Over 900 people have forked the repo that this tutorial uses since about September/October last year.)

      Up until a couple of weeks ago (afaik) and at least since mid-December 2017 when I last tested one of these tutorials, these Tutorials were working fine.

      However, either a recent change in Jenkins or Docker seems to have stopped Jenkins from successfully building Pipelines from locally cloned repos.

      Here's an example of the output I received from following the Build a Java app with Maven tutorial up to the first 'Build' stage, based on my environment (above):

      Started by user Alex
      Obtained Jenkinsfile from git /home/Documents/GitHub/simple-java-maven-app
      [Pipeline] node
      Running on Jenkins in /var/jenkins_home/workspace/simple-java-maven-app
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Declarative: Checkout SCM)
      [Pipeline] checkout
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
       > git config remote.origin.url /home/Documents/GitHub/simple-java-maven-app # timeout=10
      Fetching upstream changes from /home/Documents/GitHub/simple-java-maven-app
       > git --version # timeout=10
       > git fetch --tags --progress /home/Documents/GitHub/simple-java-maven-app +refs/heads/*:refs/remotes/origin/*
       > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
       > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
      Checking out Revision 11ab29736c7e974f76be79e885ec3e2ccd1edd3c (refs/remotes/origin/master)
       > git config core.sparsecheckout # timeout=10
       > git checkout -f 11ab29736c7e974f76be79e885ec3e2ccd1edd3c
      Commit message: "Add initial Jenkinsfile"
      First time build. Skipping changelog.
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] withEnv
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Declarative: Agent Setup)
      [Pipeline] sh
      [simple-java-maven-app] Running shell script
      + docker pull maven:3-alpine
      3-alpine: Pulling from library/maven
      Digest: sha256:400fb45cfcedfc492ff91319ea0a58f570172f8cf65f40b9b14931f44a6c1417
      Status: Image is up to date for maven:3-alpine
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] sh
      [simple-java-maven-app] Running shell script
      + docker inspect -f . maven:3-alpine
      .
      [Pipeline] withDockerContainer
      Jenkins does not seem to be running inside a container
      $ docker run -t -d -u 0:0 -v /root/.m2:/root/.m2 -w /var/jenkins_home/workspace/simple-java-maven-app -v /var/jenkins_home/workspace/simple-java-maven-app:/var/jenkins_home/workspace/simple-java-maven-app:rw,z -v /var/jenkins_home/workspace/simple-java-maven-app@tmp:/var/jenkins_home/workspace/simple-java-maven-app@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat maven:3-alpine
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Build)
      [Pipeline] sh
      [simple-java-maven-app] Running shell script
      sh: can't create /var/jenkins_home/workspace/simple-java-maven-app@tmp/durable-d0777806/jenkins-log.txt: nonexistent directory
      sh: can't create /var/jenkins_home/workspace/simple-java-maven-app@tmp/durable-d0777806/jenkins-result.txt: nonexistent directory
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      $ docker stop --time=1 62fb7efe019d5a88ed5d0c265d663184dab9668e62f4a26732dc3521ff797bd3
      $ docker rm -f 62fb7efe019d5a88ed5d0c265d663184dab9668e62f4a26732dc3521ff797bd3
      [Pipeline] // withDockerContainer
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      ERROR: script returned exit code -2
      Finished: FAILURE
      

      Other Jenkins users who've been following the tutorials in the Jenkins User Documentation have reported similar buggy behavior (see rows 44 and 45 of the documentation feedback form), one of which appears to indicate that this issue is also a problem with Docker on Windows.

      This buggy behavior even prevents the ability to successfully run a Pipeline from a local Git repo in Blue Ocean.

      I've attached links to the relevant Jenkins User Google Group post and GitHub repo issue, which a user who identified this issue (early on) had raised.

        Attachments

          Issue Links

            Activity

            Hide
            ggaskell Giles Gaskell added a comment - - edited

            There is another similar/related GitHub issue which appears to be relevant (I'm not sure entirely) but I decided to post it here anyway in case it helps resolve this issue: https://github.com/jenkinsci/docker/issues/626

            Show
            ggaskell Giles Gaskell added a comment - - edited There is another similar/related GitHub issue which appears to be relevant (I'm not sure entirely) but I decided to post it here anyway in case it helps resolve this issue: https://github.com/jenkinsci/docker/issues/626
            Hide
            ggaskell Giles Gaskell added a comment - - edited

            I've also verified that this issue also occurs in the Build a Node.js and React app with npm (I gather somewhat unsurprisingly) too.

            Show
            ggaskell Giles Gaskell added a comment - - edited I've also verified that this issue also occurs in the Build a Node.js and React app with npm (I gather somewhat unsurprisingly) too.
            Hide
            ggaskell Giles Gaskell added a comment - - edited

            Added the GitHub pull request 128 (Docker Pipeline plugin) link to this JIRA issue, which when merged and a new version of this plugin is released, should resolve this issue.

            Show
            ggaskell Giles Gaskell added a comment - - edited Added the GitHub pull request 128 (Docker Pipeline plugin) link to this JIRA issue, which when merged and a new version of this plugin is released, should resolve this issue.
            Show
            csanchez Carlos Sanchez added a comment - PR https://github.com/jenkinsci/docker-workflow-plugin/pull/128 was merged
            Hide
            gtirloni Giovanni Tirloni added a comment - - edited

            Is this really fixed? Or am I looking at a different issue here?

            Jenkins 2.121.1 (running as a systemd unit on master and a remote build node through SSH Slave plugin)
            Docker Pipeline plugin 1.17 (released in May 2018)
            Docker CE 18.03 running on CentOS 7.5
             

            pipeline {
                agent { dockerfile true }    stages {
                    stage('Test') {
                        steps {
                            sh 'cd /app && npm test'
                        }
                    }
                }
            } 

            And then the test stage doesn't run inside the Docker container that was started:

            [Pipeline] withDockerContainer
            ci-node does not seem to be running inside a container
            $ docker run -t -d -u 1020:1020 -w /home/jenkins/workspace/gpii-express_PR-7-something -v /home/jenkins/workspace/gpii-express_PR-7-something:/home/jenkins/workspace/gpii-express_PR-7-something:rw,z -v /home/jenkins/workspace/gpii-express_PR-7-something@tmp:/home/jenkins/workspace/gpii-express_PR-7-something@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** 1d0c4aca5306042f527e42c2e15dd885f78a0ac3 cat
            $ docker top 9d555195979c311991213dbcc0ef367c66b86972ba8deeb44322b122b2c02c41 -eo pid,comm
            [Pipeline] {
            [Pipeline] stage
            [Pipeline] { (Test)
            [Pipeline] sh
            [gpii-express_PR-7-something] Running shell script
            + cd /app
            + npm test
            
            > gpii-express@1.0.15 pretest /app
            > node node_modules/rimraf/rimraf coverage/* reports/*
            
            
            > gpii-express@1.0.15 test /app
            > node node_modules/istanbul/lib/cli.js cover tests/all-tests.js --print summary
            
            /app/node_modules/mkdirp/index.js:90
                                throw err0;
                                ^
            
            Error: EACCES: permission denied, mkdir '/app/reports'
                at Object.mkdirSync (fs.js:738:3)
                at Function.sync (/app/node_modules/mkdirp/index.js:71:13)
                at Object.run (/app/node_modules/istanbul/lib/command/common/run-with-cover.js:129:16)
                at CoverCommand.run (/app/node_modules/istanbul/lib/command/cover.js:27:22)
                at runCommand (/app/node_modules/istanbul/lib/cli.js:83:19)
                at runToCompletion (/app/node_modules/istanbul/lib/cli.js:87:5)
                at Object.<anonymous> (/app/node_modules/istanbul/lib/cli.js:93:5)
                at Module._compile (internal/modules/cjs/loader.js:702:30)
                at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10)
                at Module.load (internal/modules/cjs/loader.js:612:32)
            npm ERR! Test failed.  See above for more details.
            [Pipeline] }
            [Pipeline] // stage
            [Pipeline] }
            $ docker stop --time=1 9d555195979c311991213dbcc0ef367c66b86972ba8deeb44322b122b2c02c41
            $ docker rm -f 9d555195979c311991213dbcc0ef367c66b86972ba8deeb44322b122b2c02c41
            [Pipeline] // withDockerContainer
            [Pipeline] }
            [Pipeline] // withEnv
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] End of Pipeline
            

            Running the Docker image that Jenkins built works just fine:

            $ docker run --rm -ti 1d0c4aca5306042f527e42c2e15dd885f78a0ac3 /bin/sh
            /app $ npm test
            
            > gpii-express@1.0.15 pretest /app
            > node node_modules/rimraf/rimraf coverage/* reports/*
            
            
            > gpii-express@1.0.15 test /app
            > node node_modules/istanbul/lib/cli.js cover tests/all-tests.js --print summary
            ...
            16:53:58.388:  jq: ***************
            16:53:58.388:  jq: All tests concluded: 167/167 total passed in 15718ms - PASS
            16:53:58.389:  jq: ***************
            /app $ exit
            
            Show
            gtirloni Giovanni Tirloni added a comment - - edited Is this really fixed? Or am I looking at a different issue here? Jenkins 2.121.1 (running as a systemd unit on master and a remote build node through SSH Slave plugin) Docker Pipeline plugin 1.17 (released in May 2018) Docker CE 18.03 running on CentOS 7.5   pipeline { agent { dockerfile true } stages { stage('Test') { steps { sh 'cd /app && npm test' } } } } And then the test stage doesn't run inside the Docker container that was started: [Pipeline] withDockerContainer ci-node does not seem to be running inside a container $ docker run -t -d -u 1020:1020 -w /home/jenkins/workspace/gpii-express_PR-7-something -v /home/jenkins/workspace/gpii-express_PR-7-something:/home/jenkins/workspace/gpii-express_PR-7-something:rw,z -v /home/jenkins/workspace/gpii-express_PR-7-something@tmp:/home/jenkins/workspace/gpii-express_PR-7-something@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** 1d0c4aca5306042f527e42c2e15dd885f78a0ac3 cat $ docker top 9d555195979c311991213dbcc0ef367c66b86972ba8deeb44322b122b2c02c41 -eo pid,comm [Pipeline] { [Pipeline] stage [Pipeline] { (Test) [Pipeline] sh [gpii-express_PR-7-something] Running shell script + cd /app + npm test > gpii-express@1.0.15 pretest /app > node node_modules/rimraf/rimraf coverage/* reports/* > gpii-express@1.0.15 test /app > node node_modules/istanbul/lib/cli.js cover tests/all-tests.js --print summary /app/node_modules/mkdirp/index.js:90 throw err0; ^ Error: EACCES: permission denied, mkdir '/app/reports' at Object.mkdirSync (fs.js:738:3) at Function.sync (/app/node_modules/mkdirp/index.js:71:13) at Object.run (/app/node_modules/istanbul/lib/command/common/run-with-cover.js:129:16) at CoverCommand.run (/app/node_modules/istanbul/lib/command/cover.js:27:22) at runCommand (/app/node_modules/istanbul/lib/cli.js:83:19) at runToCompletion (/app/node_modules/istanbul/lib/cli.js:87:5) at Object.<anonymous> (/app/node_modules/istanbul/lib/cli.js:93:5) at Module._compile (internal/modules/cjs/loader.js:702:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:713:10) at Module.load (internal/modules/cjs/loader.js:612:32) npm ERR! Test failed. See above for more details. [Pipeline] } [Pipeline] // stage [Pipeline] } $ docker stop --time=1 9d555195979c311991213dbcc0ef367c66b86972ba8deeb44322b122b2c02c41 $ docker rm -f 9d555195979c311991213dbcc0ef367c66b86972ba8deeb44322b122b2c02c41 [Pipeline] // withDockerContainer [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Running the Docker image that Jenkins built works just fine: $ docker run --rm -ti 1d0c4aca5306042f527e42c2e15dd885f78a0ac3 /bin/sh /app $ npm test > gpii-express@1.0.15 pretest /app > node node_modules/rimraf/rimraf coverage/* reports/* > gpii-express@1.0.15 test /app > node node_modules/istanbul/lib/cli.js cover tests/all-tests.js --print summary ... 16:53:58.388: jq: *************** 16:53:58.388: jq: All tests concluded: 167/167 total passed in 15718ms - PASS 16:53:58.389: jq: *************** /app $ exit
            Hide
            eduardorn Eduardo Rodriguez added a comment -

            Is this fixed? I'm still seeing the same issue as well.

            Show
            eduardorn Eduardo Rodriguez added a comment - Is this fixed? I'm still seeing the same issue as well.
            Hide
            sadzeih Alexis Guerville added a comment -

            This issue is still happening for us too with the jenkins helm chart and docker pipeline plugin

            Show
            sadzeih Alexis Guerville added a comment - This issue is still happening for us too with the jenkins helm chart and docker pipeline plugin

              People

              • Assignee:
                ndeloof Nicolas De Loof
                Reporter:
                ggaskell Giles Gaskell
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: