Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57352

Test connection fails and also unable to get K8s pods via Jenkins using service account

    Details

    • Similar Issues:

      Description

      PS: Jenkins is deployed in K8s and not using openshift [do not intend to]

      Provisioning of K8s pods from Kubernetes Jenkins plugin using the my cloud credentials is failing, hence created service account using below command:

      kubectl  -n mynamespace create serviceaccount jenkins
      

      Then after selecting service account authentication credentials tried Test Connection in K8s-plugin cloud configuration page, but connection failed with below error:

      Error testing connection https://api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:mynamespace:default" cannot list resource "pods" in API group "" in the namespace "mynamespace": Unexpected user-id: system:serviceaccount:mynamespace:default.
      

      Also when jenkins triggered to run in labeled k8s pods, fails with below error:

      Failed to count the # of live instances on Kubernetes
      io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods?labelSelector=jenkins%3Dslave. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "mynamespace": Unexpected user-id: system:anonymous.
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:472)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:409)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
      	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:328)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:584)
      	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:49)
      	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.addProvisionedSlave(KubernetesCloud.java:493)
      	at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.provision(KubernetesCloud.java:448)
      	at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:715)
      	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:320)
      	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61)
      	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:809)
      	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:72)
      	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Activity

          Hide
          csanchez Carlos Sanchez added a comment -

          looks like it is picking the default service account, but I've never seen the "Unexpected user-id" error
          How do you configure the service account in jenkins?

          Show
          csanchez Carlos Sanchez added a comment - looks like it is picking the default service account, but I've never seen the "Unexpected user-id" error How do you configure the service account in jenkins?
          Hide
          bluegriffin Harsha GV added a comment -

          Using secret text by adding token string of service account

          Show
          bluegriffin Harsha GV added a comment - Using secret text by adding token string of service account
          Hide
          csanchez Carlos Sanchez added a comment -

          what kubernetes version? distro? cloud?

          Show
          csanchez Carlos Sanchez added a comment - what kubernetes version? distro? cloud?
          Hide
          bluegriffin Harsha GV added a comment -

          K8s:
          Client v1.11.5
          Server v1.13.5

          Jenkins deployed in K8s.

          Also could you please point me to any documentation to get the minimum roles and rolebindings needs to be assigned to a Jenkins service account - used by k8s plugin to to dynamically provision a Jenkins agent, run a single build, then tear-down that agent?

          Show
          bluegriffin Harsha GV added a comment - K8s: Client v1.11.5 Server v1.13.5 Jenkins deployed in K8s. Also could you please point me to any documentation to get the minimum roles and rolebindings needs to be assigned to a Jenkins service account - used by k8s plugin to to dynamically provision a Jenkins agent, run a single build, then tear-down that agent?
          Hide
          csanchez Carlos Sanchez added a comment -
          Show
          csanchez Carlos Sanchez added a comment - what kubernetes ? EKS? GKE?...? Roles are here https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml
          Hide
          bluegriffin Harsha GV added a comment - - edited

          Using GKE kubernetes.

          Service account is created and its roles were applied as defined in the https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml.

          But test connection still throws the error:

          Error testing connection https://api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
          
          Show
          bluegriffin Harsha GV added a comment - - edited Using GKE kubernetes. Service account is created and its roles were applied as defined in the https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/kubernetes/service-account.yml . But test connection still throws the error: Error testing connection https: //api.k8s2.apac.cloud.net:6443: Failure executing: GET at: https://api.k8s2.apac.cloud.net:6443/api/v1/namespaces/mynamespace/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
          Hide
          jglick Jesse Glick added a comment -

          Probably either a cluster or Jenkins credentials misconfiguration. Unclear how to reproduce from scratch.

          Show
          jglick Jesse Glick added a comment - Probably either a cluster or Jenkins credentials misconfiguration. Unclear how to reproduce from scratch.
          Hide
          jmukhtar junaid mukhtar added a comment -

          I was able to reproduce the issue by using an older version of Kubernetes-plugin running on dedicated EC2 instance and trying to connect to the EKS cluster.

           

          Kubernetes Plugin: 1.13.5

          EKS: 1.13

           

          ```WARNING: Failed to count the # of live instances on Kubernetes
          io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://XXXXX.sk1.eu-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/pods?labelSelector=jenkins%3Dslave. Message: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "kube-system". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "kube-system", metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).```

          Show
          jmukhtar junaid mukhtar added a comment - I was able to reproduce the issue by using an older version of Kubernetes-plugin running on dedicated EC2 instance and trying to connect to the EKS cluster.   Kubernetes Plugin: 1.13.5 EKS: 1.13   ```WARNING: Failed to count the # of live instances on Kubernetes io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://XXXXX.sk1.eu-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/pods?labelSelector=jenkins%3Dslave . Message: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "kube-system". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "kube-system", metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).```

            People

            • Assignee:
              csanchez Carlos Sanchez
              Reporter:
              bluegriffin Harsha GV
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: