Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48189

Kubernetes Plugin - Reset Connection and Read Timeout to 0 and fails to change back

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Cannot Reproduce
    • Component/s: core, kubernetes-plugin
    • Environment:
      Dockerized Jenkins (2.80-2.91) Running on a Azure Kubernetes
      Kubernetes Plugin (1.1)

      Operating System CoreOS
      Java 1.8

      Web Browser - Several Chrome 55, Safari 11.0.1
    • Similar Issues:

      Description

      The configuration of Kuberenetes starts with
      Connection Timeout 5
      Read Timeout 15

      After several runs of jobs (happens after a day and sometimes a month) the configuration is reseted to
      Connection Timeout 0
      Read Timeout 0

      Any change via browser to change it with Apply/Save doesn't pass and the values are kept to 0

      This result is creating slaves that starts with error, probably due to lack of contact with master.

      Workaround
      ----------------

      kubectl exec -it jenkins_pod bash
      edit the /var/jenkins_home/config.xml

      edit the variables.

      Delete all slaves, afterward kill the master pod. The running pod starts with 5 and 15. and activity is back to normal

      I put it on Major but it should be Critical, only because I have a workaround for this.

        Attachments

          Activity

          Hide
          chenfli Chen Fliesher added a comment -

          Moving to Critical since nobody pickup this issue

          Show
          chenfli Chen Fliesher added a comment - Moving to Critical since nobody pickup this issue
          Hide
          hb648x Haim Benbenishti added a comment -

          In addition to Chen's comment,

          The configuration of Kuberenetes starts indeed with 
          Connection Timeout 5
          Read Timeout 15

          But it gets reset even without running any job, after an unkown time.

          Any change via browser to change it with Apply/Save doesn't pass and the values remains with value 0.

          This causes slaves creation failures, as you can see, and it became very critical to us:

          jenkins-jenkins-69bc5644cc-52s5w 1/1 Running 0 3d
          jenkins-slave-6kj6j-51shw 0/4 Error 0 35m
          jenkins-slave-6kj6j-5md92 0/4 Error 0 36m
          jenkins-slave-6kj6j-7817c 0/4 Error 0 36m
          jenkins-slave-6kj6j-9gr9m 0/4 Error 0 35m
          jenkins-slave-6kj6j-bp6ng 0/4 Error 0 37m
          jenkins-slave-6kj6j-dqmfj 0/4 Error 0 35m
          jenkins-slave-6kj6j-hbhqw 0/4 Error 0 36m
          jenkins-slave-6kj6j-lplhm 0/4 Error 0 37m
          jenkins-slave-6kj6j-z4651 0/4 Error 0 20h
          jenkins-slave-6kj6j-zsjhs 0/4 Error 0 38m

          Inside the Docker, the problematic configuration:
          /var/jenkins_home/config.xml:

          <serverUrl>https://kubernetes.default</serverUrl>
          <skipTlsVerify>false</skipTlsVerify>
          <namespace>default</namespace>
          <jenkinsUrl>http://jenkins-jenkins:8080</jenkinsUrl>
          <jenkinsTunnel>jenkins-jenkins-agent:50000</jenkinsTunnel>
          <containerCap>10</containerCap>
          <retentionTimeout>5</retentionTimeout>
          <connectTimeout>0</connectTimeout>
          <readTimeout>0</readTimeout>
          <maxRequestsPerHost>32</maxRequestsPerHost>
          </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud>
          </clouds>

           

          Show
          hb648x Haim Benbenishti added a comment - In addition to Chen's comment, The configuration of Kuberenetes starts indeed with  Connection Timeout 5 Read Timeout 15 But it gets reset even without running any job, after an unkown time. Any change via browser to change it with Apply/Save doesn't pass and the values remains with value 0. This causes slaves creation failures, as you can see, and it became very critical to us: jenkins-jenkins-69bc5644cc-52s5w 1/1 Running 0 3d jenkins-slave-6kj6j-51shw 0/4 Error 0 35m jenkins-slave-6kj6j-5md92 0/4 Error 0 36m jenkins-slave-6kj6j-7817c 0/4 Error 0 36m jenkins-slave-6kj6j-9gr9m 0/4 Error 0 35m jenkins-slave-6kj6j-bp6ng 0/4 Error 0 37m jenkins-slave-6kj6j-dqmfj 0/4 Error 0 35m jenkins-slave-6kj6j-hbhqw 0/4 Error 0 36m jenkins-slave-6kj6j-lplhm 0/4 Error 0 37m jenkins-slave-6kj6j-z4651 0/4 Error 0 20h jenkins-slave-6kj6j-zsjhs 0/4 Error 0 38m Inside the Docker, the problematic configuration: /var/jenkins_home/config.xml: <serverUrl> https://kubernetes.default </serverUrl> <skipTlsVerify>false</skipTlsVerify> <namespace>default</namespace> <jenkinsUrl> http://jenkins-jenkins:8080 </jenkinsUrl> <jenkinsTunnel>jenkins-jenkins-agent:50000</jenkinsTunnel> <containerCap>10</containerCap> <retentionTimeout>5</retentionTimeout> <connectTimeout>0</connectTimeout> <readTimeout>0</readTimeout> <maxRequestsPerHost>32</maxRequestsPerHost> </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud> </clouds>  
          Hide
          csanchez Carlos Sanchez added a comment -

          I believe this is fixed in the latest versions, can you try?

          Show
          csanchez Carlos Sanchez added a comment - I believe this is fixed in the latest versions, can you try?
          Hide
          chenfli Chen Fliesher added a comment -

          Closing, it is not an issue any more

          Show
          chenfli Chen Fliesher added a comment - Closing, it is not an issue any more
          Hide
          chancelarkst chance schultz added a comment -

          Is there a way to re-open this? I am still experiencing issues with this in v1.8.4 of the kubernetes plugin. 

          Show
          chancelarkst chance schultz added a comment - Is there a way to re-open this? I am still experiencing issues with this in v1.8.4 of the kubernetes plugin. 

            People

            • Assignee:
              csanchez Carlos Sanchez
              Reporter:
              chenfli Chen Fliesher
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: