-
Bug
-
Resolution: Fixed
-
Minor
-
None
I have tried to build images on my k8s cluster using kaniko and the kubernetes-plugin. While for some images the process works just fine for others I encountered a misbehaviour.
In particular, the Jenkins build logs are flooded with the following lines, making the build fail :
*12:51:38* sh: sleep: not found *12:51:38* sh: touch: not found
I tried to debug the issue and eventually, I realized that Jenkins is running a command using nohup that is not able to find the sleep and touch utilities (the kaniko image is extremely minimal) because they are not in the folder where Jenkins expects them, but they are under the /busybox folder.
I managed to mitigate the problem by adding an environment variable in order to overwrite the PATH:
stages {
stage('Build with Kaniko') {
environment {
PATH = "/busybox:$PATH"
}
steps {
container(name: 'kaniko', shell: '/busybox/sh') {
sh '''#!/busybox/sh
/kaniko/executor -v warn -f `pwd`/Dockerfile -c `pwd` -d 'gcr.io/camunda-public/bpmn-io-demo:tmp'
'''
}
}
}
}
This approach partially fixes the problem. The logs are still flooded, but for a shorter period of time. Moreover, the following line appears at a certain point.
12:51:38 sh: touch: not found 12:51:39 sh: sleep: not found 12:51:39 sh: touch: not found 12:51:39 sh: sleep: Text file busy 12:51:39 sh: touch: Text file busy 12:51:39 sh: sleep: Text file busy
I am very confident that the responsible of this problem is the following line: https://github.com/jenkinsci/kubernetes-plugin/blob/6f5db53a096b241a7cb6bb9f26e898f5c2e04f38/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L531 since I can see in the debug log that a small bash script using the sleep/touch commands is called by Jenkins using nohup.
I believe the problem is related (but I haven't verified) only to Docker multi-stage builds.
EDIT: Here you have the command that I believe is responsible for the issue:
Executing command: "nohup" "sh" "-c" "{ while [ -d '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf' -a \! -f '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-result.txt' ]; do touch '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-log.txt'; sleep 3; done } & jsc=durable-49cb86e613753ffa205fa30f607a20d9; JENKINS_SERVER_COOKIE=\$jsc '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/script.sh' > '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-log.txt' 2>&1; echo \$? > '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-result.txt.tmp'; mv '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-result.txt.tmp' '/home/jenkins/workspace/bpmn-io_bpmn-io-demo_newCI-F5F74DO475B5SPC6Z6AH66CH2AZ6IJS4HMDREKRSG7AXXJGZAUCA@tmp/durable-48787ddf/jenkins-result.txt'; wait" exit
