Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17236

Pluggable artifact transfer & storage

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Similar Issues:

      Description

      Currently artifact archiving follows a fixed model: artifacts are defined in ArtifactArchiver (or MavenArtifactArchiver) configuration; they are copied from slave to master using FilePath.copyRecursiveTo; they are stored in ${Run.rootDir}/archive; and they are served by browsing this directory in master.

      This system has obvious scalability limitations: channel transfer is not particularly efficient; and storing enormous files in the builds directory can cause disk overruns.

      The CloudBees Fast Archiver plugin tries to address the transfer speed issue by using a better algorithm. This helps, but the implementation (replacing the ArtifactArchiver build step) is very awkward, and cannot be made to work properly with Maven projects, fingerprints, etc.

      Some installations may anyway not need to copy files at all. The master and slave might share a common network filesystem, or might have a way of efficiently transferring file references (e.g. ZFS send).

      Finally, there should be the option to store artifacts in a better way: on a big slow drive, in a compressed archive, etc.

      To handle all these requirements we need an extension point that encompasses both transfer and storage of artifacts for each build.

        Attachments

          Issue Links

            Activity

            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/model/Run.java
            core/src/main/java/hudson/tasks/ArtifactArchiver.java
            core/src/main/java/jenkins/model/ArtifactManager.java
            core/src/main/java/jenkins/model/StandardArtifactManager.java
            maven-plugin/src/main/java/hudson/maven/MavenBuild.java
            maven-plugin/src/main/java/hudson/maven/MavenBuildProxy.java
            maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java
            maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
            maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactRecord.java
            http://jenkins-ci.org/commit/jenkins/e1eea67bb2fa1a356492e91313d84a523f441b34
            Log:
            JENKINS-17236 Initial version of ArtifactManager (no Maven support yet).


            You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
            For more options, visit https://groups.google.com/groups/opt_out.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/Run.java core/src/main/java/hudson/tasks/ArtifactArchiver.java core/src/main/java/jenkins/model/ArtifactManager.java core/src/main/java/jenkins/model/StandardArtifactManager.java maven-plugin/src/main/java/hudson/maven/MavenBuild.java maven-plugin/src/main/java/hudson/maven/MavenBuildProxy.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactRecord.java http://jenkins-ci.org/commit/jenkins/e1eea67bb2fa1a356492e91313d84a523f441b34 Log: JENKINS-17236 Initial version of ArtifactManager (no Maven support yet). – You received this message because you are subscribed to the Google Groups "Jenkins Commits" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out .
            Hide
            jglick Jesse Glick added a comment -

            API should be fleshed out; still need to add Maven support. Working on a prototype plugin archiving to ${build.rootDir}/archive.zip.

            Show
            jglick Jesse Glick added a comment - API should be fleshed out; still need to add Maven support. Working on a prototype plugin archiving to ${build.rootDir}/archive.zip .
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            core/src/main/java/hudson/FilePath.java
            core/src/main/java/hudson/model/DirectoryBrowserSupport.java
            core/src/main/java/hudson/model/Run.java
            core/src/main/java/hudson/tasks/ArtifactArchiver.java
            core/src/main/java/hudson/util/DirScanner.java
            core/src/main/java/jenkins/model/ArtifactManager.java
            core/src/main/java/jenkins/model/ArtifactManagerConfiguration.java
            core/src/main/java/jenkins/model/ArtifactManagerFactory.java
            core/src/main/java/jenkins/model/ArtifactManagerFactoryDescriptor.java
            core/src/main/java/jenkins/model/StandardArtifactManager.java
            core/src/main/java/jenkins/util/VirtualFile.java
            core/src/main/resources/jenkins/model/ArtifactManagerConfiguration/config.groovy
            core/src/test/java/hudson/model/RunTest.java
            maven-plugin/src/main/java/hudson/maven/MavenBuild.java
            maven-plugin/src/main/java/hudson/maven/MavenBuildProxy.java
            maven-plugin/src/main/java/hudson/maven/MavenModuleSetBuild.java
            maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java
            maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
            maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactRecord.java
            maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
            test/src/test/java/hudson/maven/MavenMultiModuleTest.java
            test/src/test/java/hudson/model/RunTest.java
            http://jenkins-ci.org/commit/jenkins/b698b672ec5d4033115d84b2bc2b179812080bdd
            Log:
            [FIXED JENKINS-17236] Introduce ArtifactManager

            Merge branch 'ArtifactManager-JENKINS-17236'

            Compare: https://github.com/jenkinsci/jenkins/compare/eea972ec11a1...b698b672ec5d

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/model/DirectoryBrowserSupport.java core/src/main/java/hudson/model/Run.java core/src/main/java/hudson/tasks/ArtifactArchiver.java core/src/main/java/hudson/util/DirScanner.java core/src/main/java/jenkins/model/ArtifactManager.java core/src/main/java/jenkins/model/ArtifactManagerConfiguration.java core/src/main/java/jenkins/model/ArtifactManagerFactory.java core/src/main/java/jenkins/model/ArtifactManagerFactoryDescriptor.java core/src/main/java/jenkins/model/StandardArtifactManager.java core/src/main/java/jenkins/util/VirtualFile.java core/src/main/resources/jenkins/model/ArtifactManagerConfiguration/config.groovy core/src/test/java/hudson/model/RunTest.java maven-plugin/src/main/java/hudson/maven/MavenBuild.java maven-plugin/src/main/java/hudson/maven/MavenBuildProxy.java maven-plugin/src/main/java/hudson/maven/MavenModuleSetBuild.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactRecord.java maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java test/src/test/java/hudson/maven/MavenMultiModuleTest.java test/src/test/java/hudson/model/RunTest.java http://jenkins-ci.org/commit/jenkins/b698b672ec5d4033115d84b2bc2b179812080bdd Log: [FIXED JENKINS-17236] Introduce ArtifactManager Merge branch 'ArtifactManager- JENKINS-17236 ' Compare: https://github.com/jenkinsci/jenkins/compare/eea972ec11a1...b698b672ec5d
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2874
            JENKINS-17236 Initial version of ArtifactManager (no Maven support yet). (Revision e1eea67bb2fa1a356492e91313d84a523f441b34)

            Result = SUCCESS
            Jesse Glick : e1eea67bb2fa1a356492e91313d84a523f441b34
            Files :

            • maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
            • maven-plugin/src/main/java/hudson/maven/MavenBuildProxy.java
            • core/src/main/java/hudson/model/Run.java
            • core/src/main/java/jenkins/model/ArtifactManager.java
            • core/src/main/java/hudson/tasks/ArtifactArchiver.java
            • maven-plugin/src/main/java/hudson/maven/MavenBuild.java
            • maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactRecord.java
            • maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java
            • core/src/main/java/jenkins/model/StandardArtifactManager.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2874 JENKINS-17236 Initial version of ArtifactManager (no Maven support yet). (Revision e1eea67bb2fa1a356492e91313d84a523f441b34) Result = SUCCESS Jesse Glick : e1eea67bb2fa1a356492e91313d84a523f441b34 Files : maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java maven-plugin/src/main/java/hudson/maven/MavenBuildProxy.java core/src/main/java/hudson/model/Run.java core/src/main/java/jenkins/model/ArtifactManager.java core/src/main/java/hudson/tasks/ArtifactArchiver.java maven-plugin/src/main/java/hudson/maven/MavenBuild.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactRecord.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java core/src/main/java/jenkins/model/StandardArtifactManager.java
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/hudson/maven/MavenBuild.java
            src/main/java/hudson/maven/MavenBuildProxy.java
            src/main/java/hudson/maven/reporters/MavenArtifact.java
            src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
            src/main/java/hudson/maven/reporters/MavenArtifactRecord.java
            http://jenkins-ci.org/commit/maven-plugin/fa816189a1581da05c3a9a604c6d7460232b7db7
            Log:
            JENKINS-17236 Initial version of ArtifactManager (no Maven support yet).
            Originally-Committed-As: e1eea67bb2fa1a356492e91313d84a523f441b34

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/maven/MavenBuild.java src/main/java/hudson/maven/MavenBuildProxy.java src/main/java/hudson/maven/reporters/MavenArtifact.java src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java src/main/java/hudson/maven/reporters/MavenArtifactRecord.java http://jenkins-ci.org/commit/maven-plugin/fa816189a1581da05c3a9a604c6d7460232b7db7 Log: JENKINS-17236 Initial version of ArtifactManager (no Maven support yet). Originally-Committed-As: e1eea67bb2fa1a356492e91313d84a523f441b34
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/model/Run.java
            http://jenkins-ci.org/commit/jenkins/c1fcb08e08c3ccae1451602af8ea20788dc9950e
            Log:
            JENKINS-17236 Run.deleteArtifacts needs to go through ArtifactManager.
            (Also relevant to JENKINS-22395 diagnostics.)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/Run.java http://jenkins-ci.org/commit/jenkins/c1fcb08e08c3ccae1451602af8ea20788dc9950e Log: JENKINS-17236 Run.deleteArtifacts needs to go through ArtifactManager. (Also relevant to JENKINS-22395 diagnostics.)
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3347
            JENKINS-17236 Run.deleteArtifacts needs to go through ArtifactManager. (Revision c1fcb08e08c3ccae1451602af8ea20788dc9950e)

            Result = SUCCESS
            Jesse Glick : c1fcb08e08c3ccae1451602af8ea20788dc9950e
            Files :

            • core/src/main/java/hudson/model/Run.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3347 JENKINS-17236 Run.deleteArtifacts needs to go through ArtifactManager. (Revision c1fcb08e08c3ccae1451602af8ea20788dc9950e) Result = SUCCESS Jesse Glick : c1fcb08e08c3ccae1451602af8ea20788dc9950e Files : core/src/main/java/hudson/model/Run.java

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: