Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49940

Concurrent build limits not honored on kubernetes container environment with Jenkins 2.89.4

    Details

    • Similar Issues:

      Description

      After upgrading to Jenkins to  2.89.4 we noticed the throttle plugin 2.0.1 doesn't always prevent jobs from running in parallel as expected.

      Steps to reproduce:
      1. Create a Throttle category with

      •  Maximum Total Concurrent Builds -> 0
      • Maximum Concurrent Builds Per Node -> 3
      • Maximum Concurrent Builds Per Node Labeled As Above -> 3.

      2. Created Kubernetes Pod Template with

      • Max number of instances -> 3
      •  Container Cap -> 3

      3. Create 3 jobs using the category, restricted to a node with two executors

      4. Request builds of all 3 jobs.

      What should happen:
      4. Jobs run in sequence; 1 then 2 then 3.

      What actually happens:
      5. Job 1 starts building, jobs 2 and 3 wait in queue (OK).
      6. After job 1 finishes, both job 2 and 3 start running (not OK).

      Setup Details:

      Jenkins :  2.89.4
      Throttle Concurrent Builds : 2.0.1
      Kubernetes : 1.3.1

       

        Attachments

          Activity

          Hide
          akshay1909 Akshay Chaknalwar added a comment -

           

          Tried following scenario

          • Set limit to 3 for concurrent jobs
          • Started 6 jobs at once
          • 3 jobs started and 3 jobs were in queue
          • As soon as 2 jobs from currently running jobs completed, 3 remaining jobs started and total current jobs are 4 now.
          • Expected was to start only 1 job from queue

           

          Started provisioning Kubernetes Pod Template from k8s with 1 executors. Remaining excess workload: 1.97
          Mar 05, 2018 11:27:02 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
          Started provisioning Kubernetes Pod Template from k8s with 1 executors. Remaining excess workload: 0.97
          Mar 05, 2018 11:27:02 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply
          Started provisioning Kubernetes Pod Template from k8s with 1 executors. Remaining excess workload: -0.03
          Mar 05, 2018 11:27:13 PM INFO hudson.slaves.NodeProvisioner$2 run
          Kubernetes Pod Template provisioning successfully completed. We have now 4 computer(s)
          Mar 05, 2018 11:27:13 PM INFO hudson.slaves.NodeProvisioner$2 run
          Kubernetes Pod Template provisioning successfully completed. We have now 5 computer(s)
          Mar 05, 2018 11:27:13 PM INFO hudson.slaves.SlaveComputer tryReconnect
          Attempting to reconnect qa-testslave9
          Mar 05, 2018 11:27:13 PM INFO hudson.slaves.NodeProvisioner$2 run
          Kubernetes Pod Template provisioning successfully completed. We have now 6 computer(s)
          Mar 05, 2018 11:27:13 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision
          Excess workload after pending Kubernetes agents: 0
          

           

          Show
          akshay1909 Akshay Chaknalwar added a comment -   Tried following scenario Set limit to 3 for concurrent jobs Started 6 jobs at once 3 jobs started and 3 jobs were in queue As soon as 2 jobs from currently running jobs completed, 3 remaining jobs started and total current jobs are 4 now. Expected was to start only 1 job from queue   Started provisioning Kubernetes Pod Template from k8s with 1 executors. Remaining excess workload: 1.97 Mar 05, 2018 11:27:02 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply Started provisioning Kubernetes Pod Template from k8s with 1 executors. Remaining excess workload: 0.97 Mar 05, 2018 11:27:02 PM INFO hudson.slaves.NodeProvisioner$StandardStrategyImpl apply Started provisioning Kubernetes Pod Template from k8s with 1 executors. Remaining excess workload: -0.03 Mar 05, 2018 11:27:13 PM INFO hudson.slaves.NodeProvisioner$2 run Kubernetes Pod Template provisioning successfully completed. We have now 4 computer(s) Mar 05, 2018 11:27:13 PM INFO hudson.slaves.NodeProvisioner$2 run Kubernetes Pod Template provisioning successfully completed. We have now 5 computer(s) Mar 05, 2018 11:27:13 PM INFO hudson.slaves.SlaveComputer tryReconnect Attempting to reconnect qa-testslave9 Mar 05, 2018 11:27:13 PM INFO hudson.slaves.NodeProvisioner$2 run Kubernetes Pod Template provisioning successfully completed. We have now 6 computer(s) Mar 05, 2018 11:27:13 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud provision Excess workload after pending Kubernetes agents: 0  

            People

            • Assignee:
              Unassigned
              Reporter:
              akshay1909 Akshay Chaknalwar
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: