Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55577

docker agent using a custom Dockerfile is broken with docker buildkit

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: docker-workflow-plugin
    • Labels:
    • Environment:
      ubuntu 16.04
      docker 18.09.0
      jenkins 2.150.1
      docker-workflow-plugin 1.17
    • Similar Issues:

      Description

      When enabling buildkit in docker (either via `/etc/docker/daemon.json` or a global env var in jenkins: DOCKER_BUILDKIT=1), the job fails just after building the docker image used for the agent:

       https://jenkins.io/doc/book/pipeline/docker/#dockerfile

      [Pipeline] // stage
      [Pipeline] withEnv
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Declarative: Agent Setup)
      [Pipeline] isUnix
      [Pipeline] readFile
      [Pipeline] sh
      [workspace] Running shell script
      + docker build -t xxxx -f Dockerfile .
      
      ... buildkit logs ...
      
      [Pipeline] dockerFingerprintFrom
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] End of Pipeline
      java.io.IOException: Cannot retrieve .Id from 'docker inspect ubuntu:18.04'
      	at org.jenkinsci.plugins.docker.workflow.client.DockerClient.inspectRequiredField(DockerClient.java:220)
      	at org.jenkinsci.plugins.docker.workflow.FromFingerprintStep$Execution.run(FromFingerprintStep.java:133)
      	at org.jenkinsci.plugins.docker.workflow.FromFingerprintStep$Execution.run(FromFingerprintStep.java:85)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
      	at hudson.security.ACL.impersonate(ACL.java:290)
      	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE
      

      Analysis:

      jenkins tries to get information about the base image used in the Dockerfile: `dockerFingerprintFrom`. It does so by assuming the base image is available in `docker images`, as it was just used to build the agent image.

      (This has a small timing issue if a system-wide `docker image prune` happens between the build and the call to `dockerFingerprintFrom`.)

      This worked great with the old docker builder, which effectively did a `docker pull` of the base image, but buildkit does not behave like that: it keeps things completely internally.

      $ cat Dockerfile
      FROM ubuntu:18.04
      $ DOCKER_BUILDKIT=1 docker build .
      [+] Building 1.7s (5/5) FINISHED                                                                                                                                                                                                             
       => [internal] load .dockerignore                                                                                                                                                                                                       0.0s
       => => transferring context: 2B                                                                                                                                                                                                         0.0s
       => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
       => => transferring dockerfile: 37B                                                                                                                                                                                                     0.0s
       => [internal] load metadata for docker.io/library/ubuntu:18.04                                                                                                                                                                         1.6s
       => [1/1] FROM docker.io/library/ubuntu:18.04@sha256:868fd30a0e47b8d8ac485df174795b5e2fe8a6c8f056cc707b232d65b8a1ab68                                                                                                                   0.0s
       => => resolve docker.io/library/ubuntu:18.04@sha256:868fd30a0e47b8d8ac485df174795b5e2fe8a6c8f056cc707b232d65b8a1ab68                                                                                                                   0.0s
       => exporting to image                                                                                                                                                                                                                  0.0s
       => => exporting layers                                                                                                                                                                                                                 0.0s
       => => writing image sha256:645e081eb3fdb8c828216921411fc0c08335e4fd098c151b44aba797fa334839                                                                                                                                            0.0s
      $ docker inspect ubuntu:18.04
      []
      Error: No such object: ubuntu:18.04
      

        Attachments

          Issue Links

            Activity

            Hide
            fabiang Fabian Grutschus added a comment - - edited

            That true. So the only option would be to get the sha256 from the output and pull the image by that digest?

            Show
            fabiang Fabian Grutschus added a comment - - edited That true. So the only option would be to get the sha256 from the output and pull the image by that digest?
            Hide
            thomas_deepomatic Thomas Riccardi added a comment -

            Parsing the output of buildkit is ambitious: it changes (new buildkit v0.5.0 changed it for example, but not the part that would interest us...).
            I'm not sure it's a robust way to do it, but I don't see another way to do it (ideally buildkit should give us a structured build report, but I don't think this exists...)

            Anyway, buildkit is used more and more, maybe the priority of this issue should be increased? Is there a maintainer somewhere?

            Show
            thomas_deepomatic Thomas Riccardi added a comment - Parsing the output of buildkit is ambitious: it changes (new buildkit v0.5.0 changed it for example, but not the part that would interest us...). I'm not sure it's a robust way to do it, but I don't see another way to do it (ideally buildkit should give us a structured build report, but I don't think this exists...) Anyway, buildkit is used more and more, maybe the priority of this issue should be increased? Is there a maintainer somewhere?
            Hide
            fabiang Fabian Grutschus added a comment -

            buildctl has the option --export-cache=mode=max, which seems to export all layers when running Buildkit. It seems there is no way to pass this option within docker build or by an environment variable. Unfortunately I can't test it by myself, since I can't build Builkit from the repository.

            Show
            fabiang Fabian Grutschus added a comment - buildctl has the option --export-cache=mode=max , which seems to export all layers when running Buildkit. It seems there is no way to pass this option within docker build or by an environment variable. Unfortunately I can't test it by myself, since I can't build Builkit from the repository.
            Hide
            jugglefish Peter Niederlag added a comment -

            anyone in on this? As BUILDKIT provides way better handling of secrets it should really be possible to use this on jenkins.

            Show
            jugglefish Peter Niederlag added a comment - anyone in on this? As BUILDKIT provides way better handling of secrets it should really be possible to use this on jenkins.
            Hide
            viceice Michael Kriese added a comment - - edited

            This can be closed as fixed with docker-workflow-plugin 1.19 and pipeline-model-definition-plugin 1.4.0 (not yet released)

            https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/350

            Show
            viceice Michael Kriese added a comment - - edited This can be closed as fixed with docker-workflow-plugin 1.19 and pipeline-model-definition-plugin 1.4.0 (not yet released) https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/350

              People

              • Assignee:
                Unassigned
                Reporter:
                thomas_deepomatic Thomas Riccardi
              • Votes:
                7 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated: