We use kubernetes plugin for running freestyle projects.

We configured default pod template and labeled it, for example "k8s-runner".

We configured a freestyle project and restricted to run this project in "k8s-runner".

After start - the job wait up to 5 minutes if there is already running jobs on this label ("k8s-runner").

If there are no running jobs - job will start faster (1-2 minutes). But this is also very slow.

I made an experiment and found out where to start digging .

I started on long job on this runner (with label k8s-runner), and waited while the build is really started in pod. The pod name was - k8s-runner-job-zz1rv.

After this action i started another job with restriction on same label.

I measured the time (mm:ss) of second job and get those results:

00:00 - I press on "Run job". I instantly see new build is created. And the status is "(pending—All nodes of label ‘cd-k8s-ansible’ are offline)"

00:10 - The build status changes to "(pending—Waiting for next available executor on `k8s-runner-job-zz1rv`)"

03:01 - The build status changes to"(pending—All nodes of label ‘cd-k8s-ansible’ are offline)"

03:24 - The build status changes to "(pending—-k8s-runner-job-aq4c2 is offline)"

03:30 - The build is running.

I looked in kubernetes logs and i see that in 03:23 the pod "k8s-runner-job-aq4c2" creation started.

In this measurement we can notice that jenkins is waiting ~3 minutes for pod, that is already running the code for other build. And after 3 minutes, jenkins decides to create another pod.

By my observations - this time is not fixed. Sometimes is 1 minutes, sometimes 5 minutes.

I tried to debug the code section that is responsible for build running in jenkins and kubernetes plugin, but my java experience is not so high, and I cant find where is blocking place in the code. 

We use this plugin for half of year, and by my observation, the jobs are started very slow.

We tried to change jenkins version, upgrade kubernetes plugin, play with configuration of plugin, try other kubernetes cluster.

Nothing helped us.