Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47591

Allow each Kubernetes slave pod to use unique Persistent Volume / Storage

    Details

    • Type: New Feature
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: kubernetes-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.73.1
      Kubernetes Plugin 1.1
    • Similar Issues:
    • Released As:
      kubernetes-1.19.2

      Description

      We are experiencing extremely degraded performance on build slaves when running builds using the Kubernetes plugin when multiple build jobs run at the same time.  The degraded performance seems to be related to high IO on the disk / volume being mounted to slave pods when more than one slave pod runs on the same host.  As there does not seem to be a way to configure such that we can specify that each slave pod runs on a different host node, we typically end up with multiple slave pods running on the same host node and since each pod mounts the same device (either a volume on the host, or persistent volume) we end up with extreme performance degradation (jobs that usually take minutes taking hours) because of the high i/o on the backend storage.  It would be nice to allow for persistent volume claims to be configured such that each slave pod would dynamically provision its own persistent volume–which is what typically happens in kubernetes itself.  Otherwise a mechanism to configure such that each pod could have its own storage or to configure affinity / anti-affinity for slave pods so on a single slave would run per node, would help us past this issue.  

      In addition for verbosity I have tried to configure several unique pod templates mapped to unique storage via unique persistent volume claims (i.e. worker1, worker2, worker3 mapped to worker1-volume-claim, worker2-volume-claim, worker3-volume-claim), but this configuration does not seem to work as expected.  It seems the worker2 and worker3 pod templates are never used to generate slave pods–even though I have configured such that each pod template should only generate one slave node. When more than one job is run the job with either queue and wait for worker1 to finish a build or will generate a second slave based on the template for worker1–and as a result we have the previous performance issue because both pods are mounted to the same storage.  

      This issue is really impacting our ability to run multiple build jobs at the same time.  

        Attachments

          Issue Links

            Activity

            Hide
            csanchez Carlos Sanchez added a comment -

            I believe this should be possible to configure now with the yaml syntax

            Show
            csanchez Carlos Sanchez added a comment - I believe this should be possible to configure now with the yaml syntax
            Hide
            vicpl Vic K added a comment -

            As far as I understand, it's possible to use PVC for home directory through yaml, however, PVC has to be referred by name, which means it'll be shared between all slave pods. It doesn't really solve the problem.

            What would be great is an ability to provide PVC template that plugin would use to create a pvc for every slave pod.  

            Show
            vicpl Vic K added a comment - As far as I understand, it's possible to use PVC for home directory through yaml, however, PVC has to be referred by name, which means it'll be shared between all slave pods. It doesn't really solve the problem. What would be great is an ability to provide PVC template that plugin would use to create a pvc for every slave pod.  
            Hide
            csanchez Carlos Sanchez added a comment -

            That's a different issue, and you can only use pwc templates with statefulsets

            Show
            csanchez Carlos Sanchez added a comment - That's a different issue, and you can only use pwc templates with statefulsets
            Hide
            pgomez Pablo Gomez added a comment -

            Hi, we are facing the same issue. We have performance problems accessing from slaves to our persistent storage when we have multiple slaves running in the same node. Is there any solution or alternative? Is there any way to configure a separate claim for each pod?

            Show
            pgomez Pablo Gomez added a comment - Hi, we are facing the same issue. We have performance problems accessing from slaves to our persistent storage when we have multiple slaves running in the same node. Is there any solution or alternative? Is there any way to configure a separate claim for each pod?
            Show
            csanchez Carlos Sanchez added a comment - WIP at https://github.com/jenkinsci/kubernetes-plugin/pull/600
            Hide
            torstenreinhard Torsten Reinhard added a comment -

            Carlos Sanchez
            Can this dynamic PVC cause issues, like described at https://groups.google.com/forum/#!topic/jenkinsci-users/W83Ew2Og5Zo ? How to avoid those issues ?

            Show
            torstenreinhard Torsten Reinhard added a comment - Carlos Sanchez Can this dynamic PVC cause issues, like described at https://groups.google.com/forum/#!topic/jenkinsci-users/W83Ew2Og5Zo ? How to avoid those issues ?

              People

              • Assignee:
                runzexia runze xia
                Reporter:
                tdonfro Tim D'Onfro
              • Votes:
                1 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: