Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52576

Logs are flooded when sleep/touch are not available in the docker image

    Details

    • Similar Issues:

      Description

      I have tried to build images on my k8s cluster using kaniko and the kubernetes-plugin. While for some images the process works just fine for others I encountered a misbehaviour.

      In particular, the Jenkins build logs are flooded with the following lines, making the build fail : 

      *12:51:38* sh: sleep: not found
      *12:51:38* sh: touch: not found
      

      I tried to debug the issue and eventually, I realized that Jenkins is running a command using nohup that is not able to find the sleep and touch utilities (the kaniko image is extremely minimal) because they are not in the folder where Jenkins expects them, but they are under the /busybox folder.

      I managed to mitigate the problem by adding an environment variable in order to overwrite the PATH:

        stages {
          stage('Build with Kaniko') {
            environment {
              PATH = "/busybox:$PATH"
            }
            steps {
              container(name: 'kaniko', shell: '/busybox/sh') {
                  sh '''#!/busybox/sh
                  /kaniko/executor -v warn -f `pwd`/Dockerfile -c `pwd` -d 'gcr.io/camunda-public/bpmn-io-demo:tmp'
                  '''
              }
            }
          }
        }
      
      

      This approach partially fixes the problem. The logs are still flooded, but for a shorter period of time. Moreover, the following line appears at a certain point.

      12:51:38 sh: touch: not found
      12:51:39 sh: sleep: not found
      12:51:39 sh: touch: not found
      12:51:39 sh: sleep: Text file busy
      12:51:39 sh: touch: Text file busy
      12:51:39 sh: sleep: Text file busy
      

       
      I am very confident that the responsible of this problem is the following line: https://github.com/jenkinsci/kubernetes-plugin/blob/6f5db53a096b241a7cb6bb9f26e898f5c2e04f38/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L531 since I can see in the debug log that a small bash script using the sleep/touch commands is called by Jenkins using nohup.

      I believe the problem is related (but I haven't verified) only to Docker multi-stage builds.

      EDIT: Here you have the command that I believe is responsible for the issue:

      Executing command: "nohup" "sh" "-c" "{ while [ -d '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf' -a \! -f '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-result.txt' ]; do touch '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-log.txt'; sleep 3; done } & jsc=durable-49cb86e613753ffa205fa30f607a20d9; JENKINS_SERVER_COOKIE=\$jsc '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/script.sh' > '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-log.txt' 2>&1; echo \$? > '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-result.txt.tmp'; mv '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-result.txt.tmp' '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-result.txt'; wait" 
      exit
      

        Attachments

          Activity

          Hide
          hypnoce Francis JAC added a comment -

          The problem occurs because kaniko deletes the full file system between two stages, including the /bin folder with touch and sleep.

          Show
          hypnoce Francis JAC added a comment - The problem occurs because kaniko deletes the full file system between two stages, including the /bin folder with touch and sleep.
          Hide
          andrea_giardini Andrea Giardini added a comment -

          What do you think it is the best way to solve this problem?

          Show
          andrea_giardini Andrea Giardini added a comment - What do you think it is the best way to solve this problem?
          Hide
          danielbeck Daniel Beck added a comment -

          Does Durable Task 1.25 fix this? https://plugins.jenkins.io/durable-task mentions JENKINS-52881.

          Show
          danielbeck Daniel Beck added a comment - Does Durable Task 1.25 fix this? https://plugins.jenkins.io/durable-task mentions JENKINS-52881 .
          Hide
          vbehar Vincent Behar added a comment -

          Hi guys, FYI this has been fixed upstream in kaniko: https://github.com/GoogleContainerTools/kaniko/pull/369

           

          So building multi-stages images won't delete the /busybox directory anymore, and the touch/sleep commands will still be available.

           

          I believe this ticket can now be closed

          Show
          vbehar Vincent Behar added a comment - Hi guys, FYI this has been fixed upstream in kaniko: https://github.com/GoogleContainerTools/kaniko/pull/369   So building multi-stages images won't delete the /busybox directory anymore, and the touch/sleep commands will still be available.   I believe this ticket can now be closed
          Hide
          nickolashkraus Nickolas Kraus added a comment -

          We were able to mitigate this issue by using a statically-linked Busybox binary, then adding the following to our Jenkins agent Docker image:

          Dockerfile

          FROM busybox:1.30.1 as busybox
          
          COPY --from=busybox /bin /busybox
          
          ENV PATH=/busybox:$PATH
          
          # Docker volumes include an entry in /proc/self/mountinfo. This file is used
          # when kaniko builds the list of whitelisted directories. Whitelisted
          # directories are persisted between stages and are not included in the final
          # Docker image.
          VOLUME /busybox
          

           The Dockerfile for our kaniko Jenkins agent can be found here.

          Show
          nickolashkraus Nickolas Kraus added a comment - We were able to mitigate this issue by using a statically-linked Busybox binary, then adding the following to our Jenkins agent Docker image: Dockerfile FROM busybox:1.30.1 as busybox COPY --from=busybox /bin /busybox ENV PATH=/busybox:$PATH # Docker volumes include an entry in /proc/self/mountinfo. This file is used # when kaniko builds the list of whitelisted directories. Whitelisted # directories are persisted between stages and are not included in the final # Docker image. VOLUME /busybox  The Dockerfile for our kaniko Jenkins agent can be found here .

            People

            • Assignee:
              Unassigned
              Reporter:
              andrea_giardini Andrea Giardini
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: