Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60987

How to terminate Job container with non-0 gracePeriodSeconds

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • kubernetes-plugin
    • None
    • Kubernetes 1.12.8
      Kubernetes Plugin 1.19.3

      hi, 

      I'm running CI with the Jenkins Kubernetes plugin. In the job container, an internal k8s cluster is started by kind. So lots of containers are started in the container. 

      I'm trying to clean contains on termination (SIGTERM) to avoid cgroup leaking, but found that the pod is terminated with 0 grace period seconds. So it's not possible for us to clean stale containers when the job is aborted.

      Kubelet logs:

      Feb 06 12:28:34 172.16.5.67 kubelet[15125]: I0206 12:28:34.209179   15125 kubelet.go:1899] SyncLoop (DELETE, "api"): "build-tidb-operator-master-kind-249-9b2v8-gz33b-2krwn_jenkins-ci(d654cf7e-4893-11ea-b4a3-1866da9d20ee)"
Feb 06 12:28:34 172.16.5.67 kubelet[15125]: I0206 12:28:34.209712   15125 kuberuntime_container.go:555] Killing container "docker://280bf48b2cda6f0dcc2bb4a6ac317913962054d70a00ed974bfe280df78b1719" with 0 second grace period
Feb 06 12:28:34 172.16.5.67 kubelet[15125]: I0206 12:28:34.209802   15125 kuberuntime_container.go:555] Killing container "docker://b38a77770d867e24ecd606fc4a9391763c35553b61de35f7f8d013d49c7c6a0a" with 0 second grace period
Feb 06 12:28:34 172.16.5.67 kubelet[15125]: I0206 12:28:34.211965   15125 kubelet.go:1893] SyncLoop (REMOVE, "api"): "build-tidb-operator-master-kind-249-9b2v8-gz33b-2krwn_jenkins-ci(d654cf7e-4893-11ea-b4a3-1866da9d20ee)"
Feb 06 12:28:34 172.16.5.67 kubelet[15125]: I0206 12:28:34.234719   15125 kubelet_pods.go:1083] Killing unwanted pod "build-tidb-operator-master-kind-249-9b2v8-gz33b-2krwn"
Feb 06 12:28:34 172.16.5.67 kubelet[15125]: I0206 12:28:34.234785   15125 kuberuntime_container.go:555] Killing container "docker://280bf48b2cda6f0dcc2bb4a6ac317913962054d70a00ed974bfe280df78b1719" with 0 second grace period
Feb 06 12:28:34 172.16.5.67 kubelet[15125]: I0206 12:28:34.234794   15125 kuberuntime_container.go:555] Killing container "docker://b38a77770d867e24ecd606fc4a9391763c35553b61de35f7f8d013d49c7c6a0a" with 0 second grace period

      Note that job pod spec has `TerminationGracePeriodSeconds` set to 30, so I guess in deletion, `metadata.DeletionGracePeriodSeconds` is set to 0.

       Do you know how to clean stale resources when the job is aborted? Can Kubernetes plugin be configured to abort the pod with non-zero DeletionGracePeriodSeconds?

            Unassigned Unassigned
            cofyc Yecheng Fu
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: