Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12379

Archive the artifacts should allow specifying the target artifacts path

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: copyartifact-plugin, core
    • Labels:
    • Environment:
      Any environment...
    • Similar Issues:

      Description

      This is a problem that was mentioned on multiple occasions and strangely enough, it was never really answered or sometimes not
      understood.

      Let me provide some explanation:

      Job A
      Produces the following artifact:
      Build/release/x86/<multiple dlls>
      build/release/x86/fr/<some resource dll>
      build/release/x86/ru/<some resource dll>

      Job B
      Needs job A artifacts in the following folder: "lib/myJobA/"
      However it should not get "build/release/x86" Job A artifact folder. What we really want here is:
      lib/myJobA/<multiple dlls>
      lib/myJobA/fr/<some resource dll>
      lib/myJobA/ru/<some resource dll>

      However, unless I'm missing something obvious, neither "Archive the artifacts" nor "Copy artifacts from another project" will let you do that.
      "Archive the artifacts" will mirror the whole specified path "Build/release/x86/..." and "Copy artifacts from another project" will also copy the whole path to the specified target. (ie: lib/myJobA/build/release/x86/<multiple dlls>)

      "flatten" option will not help as it will not keep "fr" and "ru" folder.
      For that specific example, a possible workaround is to create 3 separate "Copy artifacts" build steps and to copy each folder
      individually in the specific target folder (using the flatten option). However, not only it is quite cumbersome, but also makes it quite difficult to refactor the job at a later time with so much data specific to jobA workspace.
      I suppose that a script could help (I'm new to Jenkis) but I believe that this should be a core feature of Jenkins "Archive the
      artifacts" (even though it would be possible to implement it at the "copy artifact" level, it seems much better to have jobB agnostic of jobA workspace specifics).

      Teamcity has a smart approach to handle this:
      If you specify: build/release/x86/*/.dll, it will omit "build/release/x86/" when archiving the artifacts.

      It also allows for more explicit target definition by using "=>" (ie: windows/.zip => winfiles,unix/.tgz => linuxFiles)
      Having at least the first option seems a must-have. In order to keep compatibility, it should be possible to check an option: [x] rebase (or re-root or whatever sounds good in English). By checking this option (unchecked by default), one would agree to have the artifacts path interpreted à la Teamcity.

      Hope all this makes sense.

        Attachments

          Issue Links

            Activity

            Hide
            frederic_latour Frederic Latour added a comment -

            Two things following the change from bug to Improvement and the addition of copyartifact as a component:

            1. A bug does not always mean an error is thrown. As it stands now, you have either to generate your artifacts by taking into account that you won't be able to rebase/change the path or add an additional script build step to move your artifacts within the current workspace before archiving your artifacts or create multiple "copy artifacts" build steps in your downstream project. I would consider this problem a functional bug.

            2. It would be great that copyartifact also solve this problem. However, having this solved in the core is a must have. Basically, you want to expose your artifacts with a certain folder structure that is not necessary identical to the folder structure used within your workspace by the build.

            Show
            frederic_latour Frederic Latour added a comment - Two things following the change from bug to Improvement and the addition of copyartifact as a component: 1. A bug does not always mean an error is thrown. As it stands now, you have either to generate your artifacts by taking into account that you won't be able to rebase/change the path or add an additional script build step to move your artifacts within the current workspace before archiving your artifacts or create multiple "copy artifacts" build steps in your downstream project. I would consider this problem a functional bug. 2. It would be great that copyartifact also solve this problem. However, having this solved in the core is a must have. Basically, you want to expose your artifacts with a certain folder structure that is not necessary identical to the folder structure used within your workspace by the build.
            Hide
            frederic_latour Frederic Latour added a comment -

            While trying to find out some workaround to this problem, I can say that I was a bit too optimistic.
            Basically executing a script as the last build step before archiving the artifacts won't help that much because you would need to put your artifacts in your workspace root which would basically make it impossible to specify artifacts files and folders.

            Show
            frederic_latour Frederic Latour added a comment - While trying to find out some workaround to this problem, I can say that I was a bit too optimistic. Basically executing a script as the last build step before archiving the artifacts won't help that much because you would need to put your artifacts in your workspace root which would basically make it impossible to specify artifacts files and folders.
            Hide
            tomobilplus Tobias Knapp added a comment - - edited

            I also run in trouble with same issue. I would like to have a functionality that I just specify a directory path in workspace where the artefacts are copies in, without creating recursive directory structure. The UI in my mind is like:

            Artifacts to copy: /Build/projA/*/.*
            Target directory: /lib/projA

            Remove leading directories: YES (YES/NO)

            Show
            tomobilplus Tobias Knapp added a comment - - edited I also run in trouble with same issue. I would like to have a functionality that I just specify a directory path in workspace where the artefacts are copies in, without creating recursive directory structure. The UI in my mind is like: Artifacts to copy: /Build/projA/* / .* Target directory: /lib/projA Remove leading directories: YES (YES/NO)
            Hide
            frederic_latour Frederic Latour added a comment -

            The teamcity approach is much more flexible than having a "remove leading directories" option.
            This would be however better than nothing.

            Show
            frederic_latour Frederic Latour added a comment - The teamcity approach is much more flexible than having a "remove leading directories" option. This would be however better than nothing.
            Hide
            stasl Stas Levin added a comment -

            I came across the same issue, and since I have quite a few nested directories it makes it impractical to create a copy artifacts step for each one.

            It would be really great to have this functionality built in, as right now it would seem like one has to resort to using batch scripting black magia, which is never a good sign, and rarely pleasant.

            Show
            stasl Stas Levin added a comment - I came across the same issue, and since I have quite a few nested directories it makes it impractical to create a copy artifacts step for each one. It would be really great to have this functionality built in, as right now it would seem like one has to resort to using batch scripting black magia, which is never a good sign, and rarely pleasant.
            Hide
            iristyle Iristyle added a comment -

            I would much prefer being able to structure the dir layout somehow when the artifacts are created. I get unnecessary nesting of archive/foo/*.

            Show
            iristyle Iristyle added a comment - I would much prefer being able to structure the dir layout somehow when the artifacts are created. I get unnecessary nesting of archive/foo/*.
            Hide
            qwert a bc added a comment -

            It's a common requirement for many serious projects.

            Show
            qwert a bc added a comment - It's a common requirement for many serious projects.
            Hide
            kmilesdev Kenneth Miles added a comment -

            To note, the "Publish Over SSH Plugin" handles this in a really simple way, by providing a "Remove prefix" field:

            https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH+Plugin

            Show
            kmilesdev Kenneth Miles added a comment - To note, the "Publish Over SSH Plugin" handles this in a really simple way, by providing a "Remove prefix" field: https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH+Plugin
            Hide
            danielbeck Daniel Beck added a comment - - edited

            Would the proposed solution in the attached screenshots work for everyone? It essentially splits pattern and user-defined common prefix in two. It's not as powerful as the TeamCity approach, but should cover almost all use cases without having to resort to shell scripts.

            Show
            danielbeck Daniel Beck added a comment - - edited Would the proposed solution in the attached screenshots work for everyone? It essentially splits pattern and user-defined common prefix in two. It's not as powerful as the TeamCity approach, but should cover almost all use cases without having to resort to shell scripts.
            Hide
            danielbeck Daniel Beck added a comment -

            I guess not.

            Show
            danielbeck Daniel Beck added a comment - I guess not.
            Hide
            danielbeck Daniel Beck added a comment -

            As I need this feature now, implementing as proposed above.

            Show
            danielbeck Daniel Beck added a comment - As I need this feature now, implementing as proposed above.
            Hide
            danielbeck Daniel Beck added a comment -
            Show
            danielbeck Daniel Beck added a comment - Proposed solution at https://github.com/jenkinsci/jenkins/pull/1493
            Hide
            wgroper Greg Roper added a comment -

            I think this would be a perfect solution for my needs.

            Show
            wgroper Greg Roper added a comment - I think this would be a perfect solution for my needs.
            Hide
            drewsky7821 Drew Horn added a comment -

            Hi Daniel Beck,

            Looks like this thread lost some traction. The proposed solution looks good, is there anything I can do to help get the pull request pushed through?

            Thanks!

            Show
            drewsky7821 Drew Horn added a comment - Hi Daniel Beck , Looks like this thread lost some traction. The proposed solution looks good, is there anything I can do to help get the pull request pushed through? Thanks!
            Hide
            danielbeck Daniel Beck added a comment -

            Thanks for the reminder. I'll try to revisit Jesse's objection pointing to continued user interest.

            Show
            danielbeck Daniel Beck added a comment - Thanks for the reminder. I'll try to revisit Jesse's objection pointing to continued user interest.
            Hide
            miq miq added a comment -

            I have exactly the same problem and would love to see a fix upstream.

            Show
            miq miq added a comment - I have exactly the same problem and would love to see a fix upstream.

              People

              • Assignee:
                danielbeck Daniel Beck
                Reporter:
                frederic_latour Frederic Latour
              • Votes:
                24 Vote for this issue
                Watchers:
                24 Start watching this issue

                Dates

                • Created:
                  Updated: