Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56396

A pipeline with a bad containerTemplate does not fail

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • kubernetes-plugin
    • None
    • Jenkins 1.50.3, kubernetes plugin 1.14.8

      If a pipeline has a bad containerTemplate definition, the build won't fail, but will infinitely try to complete. The build job effectively hangs.

      Expected behavior would be to have the job fail if a Pod can be created for the build.

      Reproduce the issue with the following pipeline (notice the bad image reference):

      def label = "mypod-${UUID.randomUUID().toString()}"podTemplate(cloud: 'k8s-dc1', label: label,
  containers: [
    containerTemplate(name: 'alpine', image: 'fakeimage', alwaysPullImage: false, ttyEnabled: true, command: 'cat')
  ]
)
{  node(label) {
    stage('alpine') {
      container('alpine') {
          sh "pwd && ls -la"
      }
    }
  }
}

      In my environment, most users don't have access to the namespace where build pods are created, and there is no feedback to the user in Jenkins that there is a bad containerTemplate. 

      When I run the build above, I see this in my build job, which only shows that it is waiting on the Pod (node) to become available.

      Started by user Admin
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
‘Jenkins’ doesn’t have label ‘mypod-7000f2a8-224b-4871-8ded-543b6b1b728d’

      Meanwhile, I see this in Kubernetes

      ~ # kubectl get pod -o wide --watch
NAME                                 READY   STATUS    RESTARTS   AGE    IP             NODE                 NOMINATED NODE   READINESS GATES
jenkins-deployment-67c96975d-khpvb   2/2     Running   0          3d4h   10.233.66.66   high-memory-node-5   <none>           <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   0/2   Pending   0     0s    <none>   <none>   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   0/2   Pending   0     0s    <none>   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   0/2   ContainerCreating   0     0s    <none>   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   1/2   ErrImagePull   0     3s    10.233.119.96   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   1/2   ImagePullBackOff   0     4s    10.233.119.96   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   1/2   ErrImagePull   0     31s   10.233.119.96   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   1/2   ImagePullBackOff   0     45s   10.233.119.96   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   1/2   ErrImagePull   0     57s   10.233.119.96   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   1/2   ImagePullBackOff   0     70s   10.233.119.96   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   1/2   Terminating   0     100s   10.233.119.96   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   1/2   Terminating   0     100s   10.233.119.96   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   0/2   Terminating   0     102s   <none>   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   0/2   Terminating   0     103s   <none>   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-f15q2   0/2   Terminating   0     103s   <none>   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk   0/2   Pending   0     0s    <none>   <none>   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk   0/2   Pending   0     0s    <none>   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk   0/2   ContainerCreating   0     0s    <none>   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk   1/2   ErrImagePull   0     3s    10.233.119.97   high-memory-node-2   <none>   <none>
mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk   1/2   ImagePullBackOff   0     5s    10.233.119.97   high-memory-node-2   <none>   <none>
...

      You can see that after failing to start the Pod several times, Kubernetes (or the plugin) terminates the first pod and starts a new pod, which exhibits the same inability to start. This continues indefinitely. Here is a "describe" of the pod

      ~ # kubectl describe pod mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk
Name:               mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk
Namespace:          jenkins-master-test
Priority:           0
PriorityClassName:  <none>
Node:               high-memory-node-2/10.0.40.14
Start Time:         Mon, 04 Mar 2019 20:21:32 +0000
Labels:             jenkins=slave
                    jenkins/mypod-7000f2a8-224b-4871-8ded-543b6b1b728d=true
Annotations:        buildUrl: http://jenkins-service:8080/job/testmaxconn/143/
                    kubernetes.io/limit-ranger:
                      LimitRanger plugin set: cpu, memory request for container alpine; cpu, memory limit for container alpine; cpu, memory request for containe...
Status:             Pending
IP:                 10.233.119.97
Containers:
  alpine:
    Container ID:
    Image:         fakeimage
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      cat
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:     25m
      memory:  256Mi
    Environment:
      JENKINS_SECRET:      7ed09a2770cad6e52f0f2d0b9a47dea3e5bdde1cde75a6309c7fbc72f91cf333
      JENKINS_TUNNEL:      jenkins-service:50000
      JENKINS_AGENT_NAME:  mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk
      JENKINS_NAME:        mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk
      JENKINS_URL:         http://jenkins-service:8080/
      HOME:                /home/jenkins
    Mounts:
      /home/jenkins from workspace-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qjfqj (ro)
  jnlp:
    Container ID:   docker://07b2f1e8d10b13b0b2980bb53fb556f38470e49b6c94a9dc6be61d6874f06b88
    Image:          jenkins/jnlp-slave:alpine
    Image ID:       docker-pullable://jenkins/jnlp-slave@sha256:8e330d8bc461440c797d543b9872e54b328da3ef3b052180bb5aed33204d4384
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 04 Mar 2019 20:21:34 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:     25m
      memory:  256Mi
    Environment:
      JENKINS_SECRET:      7ed09a2770cad6e52f0f2d0b9a47dea3e5bdde1cde75a6309c7fbc72f91cf333
      JENKINS_TUNNEL:      jenkins-service:50000
      JENKINS_AGENT_NAME:  mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk
      JENKINS_NAME:        mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk
      JENKINS_URL:         http://jenkins-service:8080/
      HOME:                /home/jenkins
    Mounts:
      /home/jenkins from workspace-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qjfqj (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  workspace-volume:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  default-token-qjfqj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qjfqj
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From                         Message
  ----     ------     ----               ----                         -------
  Normal   Scheduled  15s                default-scheduler            Successfully assigned jenkins-master-test/mypod-7000f2a8-224b-4871-8ded-543b6b1b728d-p2gh8-v8qdk to high-memory-node-2
  Normal   Pulling    13s                kubelet, high-memory-node-2  pulling image "fakeimage"
  Warning  Failed     13s                kubelet, high-memory-node-2  Failed to pull image "fakeimage": rpc error: code = Unknown desc = Error response from daemon: pull access denied for fakeimage, repository does not exist or may require 'docker login'
  Warning  Failed     13s                kubelet, high-memory-node-2  Error: ErrImagePull
  Normal   Pulled     13s                kubelet, high-memory-node-2  Container image "jenkins/jnlp-slave:alpine" already present on machine
  Normal   Created    13s                kubelet, high-memory-node-2  Created container
  Normal   Started    13s                kubelet, high-memory-node-2  Started container
  Normal   BackOff    11s (x2 over 12s)  kubelet, high-memory-node-2  Back-off pulling image "fakeimage"
  Warning  Failed     11s (x2 over 12s)  kubelet, high-memory-node-2  Error: ImagePullBackOff

            csanchez Carlos Sanchez
            dwatroustrinet Daniel Watrous
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: