Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12379

Archive the artifacts should allow specifying the target artifacts path

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: copyartifact-plugin, core
    • Labels:
    • Environment:
      Any environment...

      Description

      This is a problem that was mentioned on multiple occasions and strangely enough, it was never really answered or sometimes not
      understood.

      Let me provide some explanation:

      Job A
      Produces the following artifact:
      Build/release/x86/<multiple dlls>
      build/release/x86/fr/<some resource dll>
      build/release/x86/ru/<some resource dll>

      Job B
      Needs job A artifacts in the following folder: "lib/myJobA/"
      However it should not get "build/release/x86" Job A artifact folder. What we really want here is:
      lib/myJobA/<multiple dlls>
      lib/myJobA/fr/<some resource dll>
      lib/myJobA/ru/<some resource dll>

      However, unless I'm missing something obvious, neither "Archive the artifacts" nor "Copy artifacts from another project" will let you do that.
      "Archive the artifacts" will mirror the whole specified path "Build/release/x86/..." and "Copy artifacts from another project" will also copy the whole path to the specified target. (ie: lib/myJobA/build/release/x86/<multiple dlls>)

      "flatten" option will not help as it will not keep "fr" and "ru" folder.
      For that specific example, a possible workaround is to create 3 separate "Copy artifacts" build steps and to copy each folder
      individually in the specific target folder (using the flatten option). However, not only it is quite cumbersome, but also makes it quite difficult to refactor the job at a later time with so much data specific to jobA workspace.
      I suppose that a script could help (I'm new to Jenkis) but I believe that this should be a core feature of Jenkins "Archive the
      artifacts" (even though it would be possible to implement it at the "copy artifact" level, it seems much better to have jobB agnostic of jobA workspace specifics).

      Teamcity has a smart approach to handle this:
      If you specify: build/release/x86/*/.dll, it will omit "build/release/x86/" when archiving the artifacts.

      It also allows for more explicit target definition by using "=>" (ie: windows/.zip => winfiles,unix/.tgz => linuxFiles)
      Having at least the first option seems a must-have. In order to keep compatibility, it should be possible to check an option: [x] rebase (or re-root or whatever sounds good in English). By checking this option (unchecked by default), one would agree to have the artifacts path interpreted à la Teamcity.

      Hope all this makes sense.

        Issue Links

          Activity

          Hide
          frederic_latour Frederic Latour added a comment -

          Two things following the change from bug to Improvement and the addition of copyartifact as a component:

          1. A bug does not always mean an error is thrown. As it stands now, you have either to generate your artifacts by taking into account that you won't be able to rebase/change the path or add an additional script build step to move your artifacts within the current workspace before archiving your artifacts or create multiple "copy artifacts" build steps in your downstream project. I would consider this problem a functional bug.

          2. It would be great that copyartifact also solve this problem. However, having this solved in the core is a must have. Basically, you want to expose your artifacts with a certain folder structure that is not necessary identical to the folder structure used within your workspace by the build.

          Show
          frederic_latour Frederic Latour added a comment - Two things following the change from bug to Improvement and the addition of copyartifact as a component: 1. A bug does not always mean an error is thrown. As it stands now, you have either to generate your artifacts by taking into account that you won't be able to rebase/change the path or add an additional script build step to move your artifacts within the current workspace before archiving your artifacts or create multiple "copy artifacts" build steps in your downstream project. I would consider this problem a functional bug. 2. It would be great that copyartifact also solve this problem. However, having this solved in the core is a must have. Basically, you want to expose your artifacts with a certain folder structure that is not necessary identical to the folder structure used within your workspace by the build.
          Hide
          frederic_latour Frederic Latour added a comment -

          While trying to find out some workaround to this problem, I can say that I was a bit too optimistic.
          Basically executing a script as the last build step before archiving the artifacts won't help that much because you would need to put your artifacts in your workspace root which would basically make it impossible to specify artifacts files and folders.

          Show
          frederic_latour Frederic Latour added a comment - While trying to find out some workaround to this problem, I can say that I was a bit too optimistic. Basically executing a script as the last build step before archiving the artifacts won't help that much because you would need to put your artifacts in your workspace root which would basically make it impossible to specify artifacts files and folders.
          Hide
          tomobilplus Tobias Knapp added a comment - - edited

          I also run in trouble with same issue. I would like to have a functionality that I just specify a directory path in workspace where the artefacts are copies in, without creating recursive directory structure. The UI in my mind is like:

          Artifacts to copy: /Build/projA/*/.*
          Target directory: /lib/projA

          Remove leading directories: YES (YES/NO)

          Show
          tomobilplus Tobias Knapp added a comment - - edited I also run in trouble with same issue. I would like to have a functionality that I just specify a directory path in workspace where the artefacts are copies in, without creating recursive directory structure. The UI in my mind is like: Artifacts to copy: /Build/projA/* / .* Target directory: /lib/projA Remove leading directories: YES (YES/NO)
          Hide
          frederic_latour Frederic Latour added a comment -

          The teamcity approach is much more flexible than having a "remove leading directories" option.
          This would be however better than nothing.

          Show
          frederic_latour Frederic Latour added a comment - The teamcity approach is much more flexible than having a "remove leading directories" option. This would be however better than nothing.
          Hide
          stasl Stas Levin added a comment -

          I came across the same issue, and since I have quite a few nested directories it makes it impractical to create a copy artifacts step for each one.

          It would be really great to have this functionality built in, as right now it would seem like one has to resort to using batch scripting black magia, which is never a good sign, and rarely pleasant.

          Show
          stasl Stas Levin added a comment - I came across the same issue, and since I have quite a few nested directories it makes it impractical to create a copy artifacts step for each one. It would be really great to have this functionality built in, as right now it would seem like one has to resort to using batch scripting black magia, which is never a good sign, and rarely pleasant.
          Hide
          iristyle Iristyle added a comment -

          I would much prefer being able to structure the dir layout somehow when the artifacts are created. I get unnecessary nesting of archive/foo/*.

          Show
          iristyle Iristyle added a comment - I would much prefer being able to structure the dir layout somehow when the artifacts are created. I get unnecessary nesting of archive/foo/*.
          Hide
          qwert a bc added a comment -

          It's a common requirement for many serious projects.

          Show
          qwert a bc added a comment - It's a common requirement for many serious projects.
          Hide
          kmilesdev Kenneth Miles added a comment -

          To note, the "Publish Over SSH Plugin" handles this in a really simple way, by providing a "Remove prefix" field:

          https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH+Plugin

          Show
          kmilesdev Kenneth Miles added a comment - To note, the "Publish Over SSH Plugin" handles this in a really simple way, by providing a "Remove prefix" field: https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH+Plugin
          Hide
          danielbeck Daniel Beck added a comment - - edited

          Would the proposed solution in the attached screenshots work for everyone? It essentially splits pattern and user-defined common prefix in two. It's not as powerful as the TeamCity approach, but should cover almost all use cases without having to resort to shell scripts.

          Show
          danielbeck Daniel Beck added a comment - - edited Would the proposed solution in the attached screenshots work for everyone? It essentially splits pattern and user-defined common prefix in two. It's not as powerful as the TeamCity approach, but should cover almost all use cases without having to resort to shell scripts.
          Hide
          danielbeck Daniel Beck added a comment -

          I guess not.

          Show
          danielbeck Daniel Beck added a comment - I guess not.
          Hide
          danielbeck Daniel Beck added a comment -

          As I need this feature now, implementing as proposed above.

          Show
          danielbeck Daniel Beck added a comment - As I need this feature now, implementing as proposed above.
          Hide
          danielbeck Daniel Beck added a comment -
          Show
          danielbeck Daniel Beck added a comment - Proposed solution at https://github.com/jenkinsci/jenkins/pull/1493
          Hide
          wgroper Greg Roper added a comment -

          I think this would be a perfect solution for my needs.

          Show
          wgroper Greg Roper added a comment - I think this would be a perfect solution for my needs.

            People

            • Assignee:
              danielbeck Daniel Beck
              Reporter:
              frederic_latour Frederic Latour
            • Votes:
              16 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated: