Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47106

Not possible to use local docker images in pipeline

    Details

    • Similar Issues:

      Description

      Version 1.2 of pipeline-model-definition-plugin changes the behavior of docker images. `docker pull` is now performed for docker images specified in `agent`. Previously this was not performed.

      This change affects use cases where the docker image only exists locally. It will obviously fail if `docker pull` is called.

      The change was introduced in https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/182

      The simple workaround is to patch the plugin to revert this change, but it would be nice to avoid that.

      Hopefully this could be reverted to old behavior or/and an option added for configuration.

        Attachments

          Issue Links

            Activity

            oysols Øystein Olsen created issue -
            Hide
            pwilcock Peter Wilcock added a comment - - edited

            Also confirmed. Support for local images +1

             

            The original bug requested that this behaviour be toggled with a flag. That seems to be a sensible option. 

            Show
            pwilcock Peter Wilcock added a comment - - edited Also confirmed. Support for local images +1   The original bug requested that this behaviour be toggled with a flag. That seems to be a sensible option. 
            Hide
            dene14 Denis Boulas added a comment -

            Worth to add that there is no workaround currently as it seems.

            I've tried to replace local image with remote, however Jenkins fails to authenticate with ECR prior to actual check for image update... So I would change priority or provide any working workaround for this for now.

            Show
            dene14 Denis Boulas added a comment - Worth to add that there is no workaround currently as it seems. I've tried to replace local image with remote, however Jenkins fails to authenticate with ECR prior to actual check for image update... So I would change priority or provide any working workaround for this for now.
            Hide
            scottles Scott Hilleard added a comment - - edited

            Our build system has been hit by this problem too. All our pipelines now fail as they need to use locally built images!

            Show
            scottles Scott Hilleard added a comment - - edited Our build system has been hit by this problem too. All our pipelines now fail as they need to use locally built images!
            Hide
            danack Dan Ackroyd added a comment -

            I think it would be appropriate for the priority to be updated to be 'major' or 'critical', and for there to be some announcement about this being a breaking bug.

            Upgrading to this version of the plugin, when you have images that are stored locally, apparently breaks the build pipeline. 

            So anyone using the previous version and having local images in their pipeline need to be aware this bug will break their build.

            (apologies in advance for cross-posting, but this caused quite a bit of headaches for us, which other people shouldn't have to suffer.)

            Show
            danack Dan Ackroyd added a comment - I think it would be appropriate for the priority to be updated to be 'major' or 'critical', and for there to be some announcement about this being a breaking bug. Upgrading to this version of the plugin, when you have images that are stored locally, apparently breaks the build pipeline.  So anyone using the previous version and having local images in their pipeline need to be aware this bug will break their build. (apologies in advance for cross-posting, but this caused quite a bit of headaches for us, which other people shouldn't have to suffer.)
            abayer Andrew Bayer made changes -
            Field Original Value New Value
            Priority Minor [ 4 ] Blocker [ 1 ]
            Hide
            abayer Andrew Bayer added a comment -

            Any chance someone could give me a reproduction case from scratch so that I can write a unit test? I'd like to be able to verify the correct behavior in the fix, so that I can retain the JENKINS-46277 fix as well.

            Show
            abayer Andrew Bayer added a comment - Any chance someone could give me a reproduction case from scratch so that I can write a unit test? I'd like to be able to verify the correct behavior in the fix, so that I can retain the JENKINS-46277 fix as well.
            Hide
            abayer Andrew Bayer added a comment -

            Also, it's be immensely helpful if someone could provide the logs/errors they get with the current behavior.

            Show
            abayer Andrew Bayer added a comment - Also, it's be immensely helpful if someone could provide the logs/errors they get with the current behavior.
            Hide
            pwilcock Peter Wilcock added a comment - - edited

            Andrew Bayer: To reproduce I merely need to build the local image on the node so that it resides in the local docker repo. The image: argument of the docker agent just specifies that local image. 

            docker {
              image 'build_linux:latest'
            }

            The pipeline error log thusly looks like:

            12:09:13 + docker pull build_linux:latest
            12:09:14 Error response from daemon: repository build_linux not found: does not exist or no pull access
            
            Show
            pwilcock Peter Wilcock added a comment - - edited Andrew Bayer : To reproduce I merely need to build the local image on the node so that it resides in the local docker repo. The image: argument of the docker agent just specifies that local image.  docker { image 'build_linux:latest' } The pipeline error log thusly looks like: 12:09:13 + docker pull build_linux:latest 12:09:14 Error response from daemon: repository build_linux not found: does not exist or no pull access
            abayer Andrew Bayer made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            abayer Andrew Bayer made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            abayer Andrew Bayer added a comment -

            PR up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/199 for changing to alwaysDoDockerPull in options, with the default being not to pull.

            Show
            abayer Andrew Bayer added a comment - PR up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/199 for changing to alwaysDoDockerPull in options , with the default being not to pull.
            abayer Andrew Bayer made changes -
            Remote Link This issue links to "PR #199 (Web Link)" [ 17790 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            content/doc/book/pipeline/syntax.adoc
            http://jenkins-ci.org/commit/jenkins.io/2546503a7f816ed1cb9b764c2d01b19079bf1253
            Log:
            JENKINS-47106 Add docs for alwaysDoDockerPull option

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: content/doc/book/pipeline/syntax.adoc http://jenkins-ci.org/commit/jenkins.io/2546503a7f816ed1cb9b764c2d01b19079bf1253 Log: JENKINS-47106 Add docs for alwaysDoDockerPull option
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: R. Tyler Croy
            Path:
            content/doc/book/pipeline/syntax.adoc
            http://jenkins-ci.org/commit/jenkins.io/10a1828a6afd68ac8341969bb7abb21937775b2f
            Log:
            Merge pull request #1152 from abayer/jenkins-47106

            JENKINS-47106 Add docs for alwaysDoDockerPull option

            Compare: https://github.com/jenkins-infra/jenkins.io/compare/9e529c025017...10a1828a6afd

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: R. Tyler Croy Path: content/doc/book/pipeline/syntax.adoc http://jenkins-ci.org/commit/jenkins.io/10a1828a6afd68ac8341969bb7abb21937775b2f Log: Merge pull request #1152 from abayer/jenkins-47106 JENKINS-47106 Add docs for alwaysDoDockerPull option Compare: https://github.com/jenkins-infra/jenkins.io/compare/9e529c025017...10a1828a6afd
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Agent.groovy
            pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/options/impl/AlwaysDoDockerPull.java
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/options/impl/AlwaysDoDockerPull/help.jelly
            pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java
            pipeline-model-definition/src/test/resources/dockerPullLocalImage.groovy
            pipeline-model-extensions/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/DeclarativeAgent.java
            http://jenkins-ci.org/commit/pipeline-model-definition-plugin/1c275cd3958c595fb32decdc80177efd46af5b19
            Log:
            [FIXED JENKINS-47106] Only do explicit docker pull if specified

            Automatically doing docker pull was breaking usage of local images. It
            may make sense to re-enable the automatic pull for the top-level
            agent, though.

            Also added a way to store arbitrary DeclarativeOptions on the
            DeclarativeAgent, so that we don't have to hardcode any new options in
            the future.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Agent.groovy pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/options/impl/AlwaysDoDockerPull.java pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/options/impl/AlwaysDoDockerPull/help.jelly pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/AgentTest.java pipeline-model-definition/src/test/resources/dockerPullLocalImage.groovy pipeline-model-extensions/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/agent/DeclarativeAgent.java http://jenkins-ci.org/commit/pipeline-model-definition-plugin/1c275cd3958c595fb32decdc80177efd46af5b19 Log: [FIXED JENKINS-47106] Only do explicit docker pull if specified Automatically doing docker pull was breaking usage of local images. It may make sense to re-enable the automatic pull for the top-level agent, though. Also added a way to store arbitrary DeclarativeOptions on the DeclarativeAgent, so that we don't have to hardcode any new options in the future.
            abayer Andrew Bayer made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            scottles Scott Hilleard added a comment -

            Andrew Bayer Thanks for getting a fix implemented so quickly!

            Show
            scottles Scott Hilleard added a comment - Andrew Bayer Thanks for getting a fix implemented so quickly!
            Hide
            rmf Ryan Fitzsimon added a comment -

            AFAICT, a regression was introduced by a367db7c2a727e5ea87381777b72b4c84b52f1a2.

            http://jenkins-ci.org/commit/pipeline-model-definition-plugin/a367db7c2a727e5ea87381777b72b4c84b52f1a2 

            It seems this commit removes the check of alwaysDoDockerPull for agents that aren't within a stage, making it impossible to use a local image in top level scope.

            I could easily be wrong, please let me know if I've misunderstood something.

            Show
            rmf Ryan Fitzsimon added a comment - AFAICT, a regression was introduced by a367db7c2a727e5ea87381777b72b4c84b52f1a2. http://jenkins-ci.org/commit/pipeline-model-definition-plugin/a367db7c2a727e5ea87381777b72b4c84b52f1a2   It seems this commit removes the check of alwaysDoDockerPull for agents that aren't within a stage, making it impossible to use a local image in top level scope. I could easily be wrong, please let me know if I've misunderstood something.
            rmf Ryan Fitzsimon made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            rmf Ryan Fitzsimon made changes -
            Comment [ AFAICT, a regression was introduced by a367db7c2a727e5ea87381777b72b4c84b52f1a2.

            [http://jenkins-ci.org/commit/pipeline-model-definition-plugin/a367db7c2a727e5ea87381777b72b4c84b52f1a2

            It seems this commit removes the check of alwaysDoDockerPull for agents that aren't within a stage, making it impossible to use a local image in top level scope.

            I could easily be wrong, please let me know if I've misunderstood something. ]
            Hide
            johnson_xu Johnson Xu added a comment -

             I met same issue as Ryan Fitzsimon.

            When I set 'alwaysPull false' within stage block, it works fine as expect to use local image. However, if the same config is set within the top 'pipeline/agent' block, it still always do 'docker pull' and the local image does not work.

            Show
            johnson_xu Johnson Xu added a comment -  I met same issue as Ryan Fitzsimon . When I set 'alwaysPull false' within stage block, it works fine as expect to use local image. However, if the same config is set within the top 'pipeline/agent' block, it still always do 'docker pull' and the local image does not work.
            Hide
            abayer Andrew Bayer added a comment -

            Well, that was dumb on my part. Will get a fix up today or tomorrow. Sorry!

            Show
            abayer Andrew Bayer added a comment - Well, that was dumb on my part. Will get a fix up today or tomorrow. Sorry!
            Hide
            dcioiu Dan Cioiu added a comment -

            Hi Andrew Bayer,

            Any news on this regression?

            Show
            dcioiu Dan Cioiu added a comment - Hi Andrew Bayer , Any news on this regression?
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Segev Finer
            Path:
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy
            http://jenkins-ci.org/commit/pipeline-model-definition-plugin/5c4946495236afe8c778b9662ffd754923a728e5
            Log:
            [FIXED JENKINS-47106] Support alwaysPull in the top level agent section

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Segev Finer Path: pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy http://jenkins-ci.org/commit/pipeline-model-definition-plugin/5c4946495236afe8c778b9662ffd754923a728e5 Log: [FIXED JENKINS-47106] Support alwaysPull in the top level agent section
            scm_issue_link SCM/JIRA link daemon made changes -
            Status Reopened [ 4 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy
            http://jenkins-ci.org/commit/pipeline-model-definition-plugin/fa8ed8bb0861220ff36b730656dc12701ba86ffd
            Log:
            Merge pull request #254 from segevfiner/jenkins-47106-top-level-agent-local-image

            [FIXED JENKINS-47106] Support alwaysPull in the top level agent section

            Compare: https://github.com/jenkinsci/pipeline-model-definition-plugin/compare/f5deef1255d0...fa8ed8bb0861

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy http://jenkins-ci.org/commit/pipeline-model-definition-plugin/fa8ed8bb0861220ff36b730656dc12701ba86ffd Log: Merge pull request #254 from segevfiner/jenkins-47106-top-level-agent-local-image [FIXED JENKINS-47106] Support alwaysPull in the top level agent section Compare: https://github.com/jenkinsci/pipeline-model-definition-plugin/compare/f5deef1255d0...fa8ed8bb0861
            Hide
            bitwiseman Liam Newman added a comment -

            Bulk closing resolved issues.

            Show
            bitwiseman Liam Newman added a comment - Bulk closing resolved issues.
            bitwiseman Liam Newman made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                oysols Øystein Olsen
              • Votes:
                1 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: