Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42422

Add support for directory caching in pod jobs

    Details

    • Similar Issues:

      Description

      It would be great to be able to "cache" directories between job executions.
      In some cases it helps to greatly speedup job execution.
      Similar to Travis: https://docs.travis-ci.com/user/caching/.

      Now I achieve it with persistent volume and pipeline function doing explicit (re)store steps.

      What can be added to kubernetes-plugin:
      -option to manage caches (specifically drop cache for specific job)
      -add DSL construct like podTemplate(... cachePaths: ["A", "B/C"])
      -default strategy for cache management (shared NFS-backed volume or provisioned PVC per job)

        Attachments

          Activity

          Hide
          the_rc R C added a comment -

          [Comment almost lost thanks to JIRA]

          Right, there's that, too, but I would be OK if jobs were scheduled starting from the lowest index available. In our case, with a hot cache (which is what we'd like to achieve), an agent's longest run wouldn't be orders of magnitude worse than the average or best case. That is still a net win, even with the occasional inefficiency here and there, like job 1 being idle every once in a while.

          I came across someone else's [experience with stateful agents|https://hiya.com/blog/2017/10/02/kubernetes-base-jenkins-stateful-agents/.] We don't spend 20 minutes doing setup work like they do, but there are pipelines where installing dependencies takes up longer than the actual tests.

          Is the current design of the plugin too reliant on creating pods manually? The least invasive way might be for the scaling to be handled by a HorizontalPodAutoscaler (it should work with StatefulSets). Then the plugin would pick available slaves in alphabetical order. Is there much left to the Kubernetes plugin, at that point?

           

          Show
          the_rc R C added a comment - [Comment almost lost thanks to JIRA] Right, there's that, too, but I would be OK if jobs were scheduled starting from the lowest index available. In our case, with a hot cache (which is what we'd like to achieve), an agent's longest run wouldn't be orders of magnitude worse than the average or best case. That is still a net win, even with the occasional inefficiency here and there, like job 1 being idle every once in a while. I came across someone else's [experience with stateful agents| https://hiya.com/blog/2017/10/02/kubernetes-base-jenkins-stateful-agents/ .] We don't spend 20 minutes doing setup work like they do, but there are pipelines where installing dependencies takes up longer than the actual tests. Is the current design of the plugin too reliant on creating pods manually? The least invasive way might be for the scaling to be handled by a HorizontalPodAutoscaler (it should work with StatefulSets). Then the plugin would pick available slaves in alphabetical order. Is there much left to the Kubernetes plugin, at that point?  
          Hide
          csanchez Carlos Sanchez added a comment -

          I wrote about using a ReplicationController for agents like 4 years ago, but Jenkins needs to be able to kill the jobs that have finished, not a random one. https://www.infoq.com/articles/scaling-docker-kubernetes-v1
          At that point you could just have a pool of agents in k8s instead of dynamic allocation and use the swarm plugin to register them automatically

          About the original issue description, I would use a shared filesystem that supports multiple mounts for writing (NFS, gluster,...) and then it would be easier for developers, while being harder to setup. Doing something to support other mount-once volumes will be harder. There could be an option to copy things from a directory before the build and back after the build, but that's already implemented with stash()/unstash().
          Looking for concrete ideas on what to do, but I would not likely implement them as I have limited time

          Show
          csanchez Carlos Sanchez added a comment - I wrote about using a ReplicationController for agents like 4 years ago, but Jenkins needs to be able to kill the jobs that have finished, not a random one. https://www.infoq.com/articles/scaling-docker-kubernetes-v1 At that point you could just have a pool of agents in k8s instead of dynamic allocation and use the swarm plugin to register them automatically About the original issue description, I would use a shared filesystem that supports multiple mounts for writing (NFS, gluster,...) and then it would be easier for developers, while being harder to setup. Doing something to support other mount-once volumes will be harder. There could be an option to copy things from a directory before the build and back after the build, but that's already implemented with stash()/unstash(). Looking for concrete ideas on what to do, but I would not likely implement them as I have limited time
          Hide
          yrsurya suryatej yaramada added a comment -

          Can we achieve this by using EFS by any chance if so I would like to know how? we already mount jenkins master /var/lib/jenkins which is running on EC2 to EFS. we using kubernetes plugin to start dynamic agents and run our maven/gradle/npm jobs but those jobs takiing some extra time as cache is not happening so for every build it downloading from artifactory. I tried with mounting /root/.m2 directory but problem is not able to start another job till this job finishes. I would like to know if any work around for this. 

          Really appreciated
          `
          Multi-Attach error for volume "pvc-4e98f6b9-ea38-11e9-aa7d-02dbf42e9b46" Volume is already used by pod(s) sandbox-surya-maven-cache-3-8hk2s-n2v0v-l4f9h

          Show
          yrsurya suryatej yaramada added a comment - Can we achieve this by using EFS by any chance if so I would like to know how? we already mount jenkins master /var/lib/jenkins which is running on EC2 to EFS. we using kubernetes plugin to start dynamic agents and run our maven/gradle/npm jobs but those jobs takiing some extra time as cache is not happening so for every build it downloading from artifactory. I tried with mounting /root/.m2 directory but problem is not able to start another job till this job finishes. I would like to know if any work around for this.  Really appreciated ` Multi-Attach error for volume "pvc-4e98f6b9-ea38-11e9-aa7d-02dbf42e9b46" Volume is already used by pod(s) sandbox-surya-maven-cache-3-8hk2s-n2v0v-l4f9h
          Hide
          fauretristan Tristan FAURE added a comment - - edited

          I also have the same issue : https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/jenkinsci-users/X61BK83LHLU/hztGcbkvBgAJ

          I don't really know travis but i assume it is equivalent to https://docs.gitlab.com/ee/ci/caching/

          about Carlos Sanchez comments :

          • we did not find either how to create pvc in the yaml syntax it was always overrident when we tried
          • stash / unstash or archiveArtifacts could help us but how can configure it to store in a pv ? how long the data is kept if there are no unstash ?
            • edit: as we want data available across jobs, stash unstash does not seem relevant
          Show
          fauretristan Tristan FAURE added a comment - - edited I also have the same issue : https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/jenkinsci-users/X61BK83LHLU/hztGcbkvBgAJ I don't really know travis but i assume it is equivalent to https://docs.gitlab.com/ee/ci/caching/ about Carlos Sanchez comments : we did not find either how to create pvc in the yaml syntax it was always overrident when we tried stash / unstash or archiveArtifacts could help us but how can configure it to store in a pv ? how long the data is kept if there are no unstash ? edit: as we want data available across jobs, stash unstash does not seem relevant
          Hide
          seakip18 Matthew Ludlum added a comment - - edited

          Carlos Sanchez  Vincent Latombe - I'm interested on working on something related to this. Most competing CI platforms offer something like this:

          https://help.github.com/en/actions/configuring-and-managing-workflows/caching-dependencies-to-speed-up-workflows

          https://docs.gitlab.com/ee/ci/caching/

          https://docs.travis-ci.com/user/caching/

           

          The question I am juggling around is whether this is something that either the pipeline side or the agent side should handle.

          Show
          seakip18 Matthew Ludlum added a comment - - edited Carlos Sanchez   Vincent Latombe - I'm interested on working on something related to this. Most competing CI platforms offer something like this: https://help.github.com/en/actions/configuring-and-managing-workflows/caching-dependencies-to-speed-up-workflows https://docs.gitlab.com/ee/ci/caching/ https://docs.travis-ci.com/user/caching/   The question I am juggling around is whether this is something that either the pipeline side or the agent side should handle.

            People

            • Assignee:
              Unassigned
              Reporter:
              electroma Roman Safronov
            • Votes:
              6 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated: