Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50268

instanceCap reached with preexisting pods because labels are empty

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: kubernetes-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.108
      Kubernetes Plugin 1.2.1
      Kubernetes 1.7.12
    • Similar Issues:

      Description

      By default, when the Kubernetes plugin checks for the number of agents in the namespace, it get the number of ALL pods in the namespace, not just the ones created by the plugin.

      This is because when the current list of agents is being fetched, it filters all pods in the namespace by the pod labels applied to all agents created by the Jenkins cloud, which defaults to no labels. See: https://github.com/jenkinsci/kubernetes-plugin/blob/e026c003c90cec99328c1311c62c4b9f431e5f26/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesCloud.java#L451

      This has some critical consequences. In our case, we had 10 pods running in the namespace already, and a containerCap of 10. Because of this, it became impossible to create a new agent from the Kubernetes cloud integration. 

       

      I have a couple suggestions to fix this.

      1) Set the default Kubernetes cloud label list to be non-empty. Perhaps default to a label like `kubernetes-cloud: <cloud-name>`.

      2) Make the list of pod labels editable from the Jenkins management console. Right now, the only way to set this value (which is the needed workaround for the problem) is through a groovy script.

        Attachments

          Issue Links

            Activity

            Hide
            csanchez Carlos Sanchez added a comment -

            on upgrade if labels were not set they are set to jenkins=slave, see https://github.com/jenkinsci/kubernetes-plugin/blob/e026c003c90cec99328c1311c62c4b9f431e5f26/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesCloud.java#L635
            what do you have as labels in your config.xml file ?

            Show
            csanchez Carlos Sanchez added a comment - on upgrade if labels were not set they are set to jenkins=slave, see https://github.com/jenkinsci/kubernetes-plugin/blob/e026c003c90cec99328c1311c62c4b9f431e5f26/src/main/java/org/csanchez/jenkins/plugins/kubernetes/KubernetesCloud.java#L635 what do you have as labels in your config.xml file ?
            Hide
            jutley Jake Utley added a comment -

            I see what you're saying. Our KubernetesCloud config looks like this:

            <clouds>
              <org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin="kubernetes@1.2.1">
                <name>kubernetes</name>
                <templates/>
                <serverUrl>controller</serverUrl>
                <skipTlsVerify>false</skipTlsVerify>
                <namespace>jenkins-ci-stg</namespace>
                <jenkinsUrl>http://<<REDACTED>></jenkinsUrl>
                <containerCap>0</containerCap>
                <retentionTimeout>5</retentionTimeout>
                <connectTimeout>0</connectTimeout>
                <readTimeout>0</readTimeout>
                <maxRequestsPerHost>32</maxRequestsPerHost>
              </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud>
            </clouds>

            But if we do a "Reload from disk", then this gets added:

            <labels class="com.google.common.collect.ImmutableMap">
              <entry>
                <string>jenkins</string>
                <string>slave</string>
              </entry>
            </labels>

            So the behavior does follow your description. However, I don't think that the plugin should have to depend on a reload to get into the proper state.

            In our case, we try to minimize our dependence on a persistent Jenkins configuration. We handle all our configuration in Groovy scripts that run at initialization time. In the case of our Kubernetes cloud, we have a script that looks like this:

            def jenkins = Jenkins.getInstance()
            def kubernetesCloud = new KubernetesCloud('kubernetes')
            kubernetesCloud.setServerUrl('controller')
            kubernetesCloud.setSkipTlsVerify(false)
            kubernetesCloud.setNamespace('jenkins-ci-stg')
            kubernetesCloud.setJenkinsUrl('<<REDACTED>>')
            kubernetesCloud.setContainerCapStr('0')
            jenkins.clouds.replace(kubernetesCloud)
            jenkins.save()
            

            This means that every time we start up, we get a fresh cloud, and the code you linked is not running.

             

            This may be a bit of a unique usage, but the cloud should still start up with all its "defaults" in place. Perhaps the code in readResolve() that sets the defaults should be pulled into its own method, and then be called from both readResolve and the constructor?

            Show
            jutley Jake Utley added a comment - I see what you're saying. Our KubernetesCloud config looks like this: <clouds> <org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin= "kubernetes@1.2.1" > <name>kubernetes</name> <templates/> <serverUrl>controller</serverUrl> <skipTlsVerify> false </skipTlsVerify> <namespace>jenkins-ci-stg</namespace> <jenkinsUrl>http: //<<REDACTED>></jenkinsUrl> <containerCap>0</containerCap> <retentionTimeout>5</retentionTimeout> <connectTimeout>0</connectTimeout> <readTimeout>0</readTimeout> <maxRequestsPerHost>32</maxRequestsPerHost> </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud> </clouds> But if we do a "Reload from disk", then this gets added: <labels class= "com.google.common.collect.ImmutableMap" > <entry> <string>jenkins</string> <string>slave</string> </entry> </labels> So the behavior does follow your description. However, I don't think that the plugin should have to depend on a reload to get into the proper state. In our case, we try to minimize our dependence on a persistent Jenkins configuration. We handle all our configuration in Groovy scripts that run at initialization time. In the case of our Kubernetes cloud, we have a script that looks like this: def jenkins = Jenkins.getInstance() def kubernetesCloud = new KubernetesCloud( 'kubernetes' ) kubernetesCloud.setServerUrl( 'controller' ) kubernetesCloud.setSkipTlsVerify( false ) kubernetesCloud.setNamespace( 'jenkins-ci-stg' ) kubernetesCloud.setJenkinsUrl( '<<REDACTED>>' ) kubernetesCloud.setContainerCapStr( '0' ) jenkins.clouds.replace(kubernetesCloud) jenkins.save() This means that every time we start up, we get a fresh cloud, and the code you linked is not running.   This may be a bit of a unique usage, but the cloud should still start up with all its "defaults" in place. Perhaps the code in readResolve() that sets the defaults should be pulled into its own method, and then be called from both readResolve and the constructor?
            Hide
            csanchez Carlos Sanchez added a comment -

            If you are creating objects in groovy you'll need to set the labels yourself
            Readresolve is used to handle upgrades without overriding existing configuration

            Show
            csanchez Carlos Sanchez added a comment - If you are creating objects in groovy you'll need to set the labels yourself Readresolve is used to handle upgrades without overriding existing configuration
            Show
            csanchez Carlos Sanchez added a comment - https://github.com/jenkinsci/kubernetes-plugin/pull/325

              People

              • Assignee:
                csanchez Carlos Sanchez
                Reporter:
                jutley Jake Utley
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: