Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12134

Copying an artifact should (optionally) fingerprint all artifacts

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: copyartifact-plugin
    • Labels:
      None

      Issue Links

        Activity

        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Jørgen P. Tjernø
        Path:
        src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
        http://jenkins-ci.org/commit/copyartifact-plugin/81c6bec56889ccfee9ee626397dbcb8a882457d6
        Log:
        JENKINS-12134 Automatically fingerprint files.

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jørgen P. Tjernø Path: src/main/java/hudson/plugins/copyartifact/CopyArtifact.java http://jenkins-ci.org/commit/copyartifact-plugin/81c6bec56889ccfee9ee626397dbcb8a882457d6 Log: JENKINS-12134 Automatically fingerprint files.
        Hide
        dogfood dogfood added a comment -

        Integrated in plugins_copyartifact #73
        JENKINS-12134 Automatically fingerprint files.

        jtjerno : 81c6bec56889ccfee9ee626397dbcb8a882457d6
        Files :

        • src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
        Show
        dogfood dogfood added a comment - Integrated in plugins_copyartifact #73 JENKINS-12134 Automatically fingerprint files. jtjerno : 81c6bec56889ccfee9ee626397dbcb8a882457d6 Files : src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
        Hide
        ridesmet Ringo De Smet added a comment - - edited

        The current implementation of this ticket leads to a serious performance degradation for me. While I have all freestyle projects with fingerprinting disabled, the plugin now starts to fingerprint files.

        Setup: I have one specific project with 17000+ archived artifacts. These artifacts are used as a prerequisite for a number of other projects. Fingerprinting all these files adds about 15 minutes to the build time of each of these projects which is unacceptible.

        Please make the fingerprinting optional, or follow the fingerprinting setting of the source and/or target projects.

        Show
        ridesmet Ringo De Smet added a comment - - edited The current implementation of this ticket leads to a serious performance degradation for me. While I have all freestyle projects with fingerprinting disabled , the plugin now starts to fingerprint files. Setup: I have one specific project with 17000+ archived artifacts. These artifacts are used as a prerequisite for a number of other projects. Fingerprinting all these files adds about 15 minutes to the build time of each of these projects which is unacceptible. Please make the fingerprinting optional, or follow the fingerprinting setting of the source and/or target projects.
        Hide
        jorgenpt Jørgen Tjernø added a comment -

        Here's a suggested fix for this: https://github.com/jenkinsci/copyartifact-plugin/pull/8
        It requires you to enable "Fingerprint all artifacts" on the source for the destination to fingerprint the files.

        In addition, Kohsuke mentioned that he could look into reworking the digesting so that it's done during transfer. This would most likely add a negligible overhead to the copy process, and make the digest process basically free.

        How long does your "Copy Artifact" process take if the fingerprinting takes 15 minutes?

        Show
        jorgenpt Jørgen Tjernø added a comment - Here's a suggested fix for this: https://github.com/jenkinsci/copyartifact-plugin/pull/8 It requires you to enable "Fingerprint all artifacts" on the source for the destination to fingerprint the files. In addition, Kohsuke mentioned that he could look into reworking the digesting so that it's done during transfer. This would most likely add a negligible overhead to the copy process, and make the digest process basically free. How long does your "Copy Artifact" process take if the fingerprinting takes 15 minutes?
        Hide
        dogfood dogfood added a comment -

        Integrated in plugins_copyartifact #78
        [FIXED JENKINS-12134] hide the overhead of fingerprinting into I/O.

        Kohsuke Kawaguchi : 3cf80e567c8171428be377ddb2ec6855a7570dc3
        Files :

        • pom.xml
        • src/main/java/hudson/plugins/copyartifact/FilePathCopyMethod.java
        • src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
        • src/main/java/hudson/plugins/copyartifact/FingerprintingCopyMethod.java
        • src/main/java/hudson/plugins/copyartifact/Copier.java
        • src/main/java/hudson/plugins/copyartifact/CopyMethod.java
        Show
        dogfood dogfood added a comment - Integrated in plugins_copyartifact #78 [FIXED JENKINS-12134] hide the overhead of fingerprinting into I/O. Kohsuke Kawaguchi : 3cf80e567c8171428be377ddb2ec6855a7570dc3 Files : pom.xml src/main/java/hudson/plugins/copyartifact/FilePathCopyMethod.java src/main/java/hudson/plugins/copyartifact/CopyArtifact.java src/main/java/hudson/plugins/copyartifact/FingerprintingCopyMethod.java src/main/java/hudson/plugins/copyartifact/Copier.java src/main/java/hudson/plugins/copyartifact/CopyMethod.java
        Hide
        jorgenpt Jørgen Tjernø added a comment -

        Ringo, would you mind testing the performance of 0.21? It has something that should hopefully be a great performance improvement from 0.20, but still fingerprints. Let us know if you're still seeing unacceptable performance levels.

        Show
        jorgenpt Jørgen Tjernø added a comment - Ringo, would you mind testing the performance of 0.21? It has something that should hopefully be a great performance improvement from 0.20, but still fingerprints. Let us know if you're still seeing unacceptable performance levels.
        Hide
        mindless Alan Harder added a comment -

        Did this new change introduce JENKINS-12412 ?

        Show
        mindless Alan Harder added a comment - Did this new change introduce JENKINS-12412 ?
        Hide
        mandeepr Mandeep Rai added a comment -

        Hi there,

        I personally don't have the need for fingerprinting for some of my builds, and since they generate a large number of small files, fingerprinting is actually slower than the copy itself. With the latest release that hides the overhead of fingerprinting, it is much better, but the fingerprinting still takes longer than the actual IO of the copy. Could we please make the fingerprinting optional rather than automatic? I wouldn't mind if the option defaulted to being "on"

        Thanks,
        Mandeep

        Show
        mandeepr Mandeep Rai added a comment - Hi there, I personally don't have the need for fingerprinting for some of my builds, and since they generate a large number of small files, fingerprinting is actually slower than the copy itself. With the latest release that hides the overhead of fingerprinting, it is much better, but the fingerprinting still takes longer than the actual IO of the copy. Could we please make the fingerprinting optional rather than automatic? I wouldn't mind if the option defaulted to being "on" Thanks, Mandeep
        Hide
        peter_schuetze peter_schuetze added a comment -

        Agree with Mandeep, Currently, don't use the fingerprinting. I use the copy artifact plugin to deploy to different dev environment (as a self service for the dev teams). The same artifact will then be fingerprinted several times (at least once for every environment). So the optional fingerprinting would be a great feature.

        Show
        peter_schuetze peter_schuetze added a comment - Agree with Mandeep, Currently, don't use the fingerprinting. I use the copy artifact plugin to deploy to different dev environment (as a self service for the dev teams). The same artifact will then be fingerprinted several times (at least once for every environment). So the optional fingerprinting would be a great feature.
        Hide
        ridesmet Ringo De Smet added a comment -

        Looking at priorities, I don't understand why work is being done on a feature that results in a regression to a number of us, while JENKINS-9741 has more votes and more followers. Wouldn't that ticket be more important then?

        Show
        ridesmet Ringo De Smet added a comment - Looking at priorities, I don't understand why work is being done on a feature that results in a regression to a number of us, while JENKINS-9741 has more votes and more followers. Wouldn't that ticket be more important then?
        Hide
        jorgenpt Jørgen Tjernø added a comment -

        Work is being done on this because this is an open source project: no-one is paid to do engineering effort just to please the masses who use copyartifact-plugin. I filed the ticket, I contributed the patch (and then Kohsuke contributed a change to the copying algorithm to make it faster). It's a third party plugin with individual contributors "scratching their own itch" - which is what happened here.

        Motivations aside, you didn't respond to my previous request, Ringo:
        Would you mind testing the performance of 0.21? It has something that should hopefully be a great performance improvement from 0.20, but still fingerprints. Let us know if you're still seeing unacceptable performance levels. (Preferably with some numbers on before / after performance for 0.19 v 0.21)

        And yes, JENKINS-9741 should probably be resolved. I've encountered it myself, but the workaround was trivial enough for me to not care to delve into the root cause. Maybe it's time I did that now.

        Show
        jorgenpt Jørgen Tjernø added a comment - Work is being done on this because this is an open source project: no-one is paid to do engineering effort just to please the masses who use copyartifact-plugin. I filed the ticket, I contributed the patch (and then Kohsuke contributed a change to the copying algorithm to make it faster). It's a third party plugin with individual contributors "scratching their own itch" - which is what happened here. Motivations aside, you didn't respond to my previous request, Ringo: Would you mind testing the performance of 0.21? It has something that should hopefully be a great performance improvement from 0.20, but still fingerprints. Let us know if you're still seeing unacceptable performance levels. (Preferably with some numbers on before / after performance for 0.19 v 0.21) And yes, JENKINS-9741 should probably be resolved. I've encountered it myself, but the workaround was trivial enough for me to not care to delve into the root cause. Maybe it's time I did that now.
        Hide
        jorgenpt Jørgen Tjernø added a comment -

        Actually, 0.21 should have fixed JENKINS-9741 - please test it. The new code Kohsuke wrote to do better copies also preserves permissions.

        Show
        jorgenpt Jørgen Tjernø added a comment - Actually, 0.21 should have fixed JENKINS-9741 - please test it. The new code Kohsuke wrote to do better copies also preserves permissions.
        Hide
        ridesmet Ringo De Smet added a comment -

        I can confirm mandeepr's observations. I upgraded from Copy Artifact 1.18 to 1.21. Here are some time differences for projects that all copy the archived artifacts of a common upstream project (18k+ files):

        • project 1: 36 min to 1 hr 1 min
        • project 2: 40 min to 1 hr 13 min

        So again the request to not fingerprint if the source and/or target project has fingerprinting disabled. I am going to downgrade the plugin again as the above numbers are not acceptable for us.

        And Jørgen, you refer version numbers of the plugin starting with zero (0.19, 0.21). Shouldn't this be 1.19 and 1.21?

        Show
        ridesmet Ringo De Smet added a comment - I can confirm mandeepr's observations . I upgraded from Copy Artifact 1.18 to 1.21. Here are some time differences for projects that all copy the archived artifacts of a common upstream project (18k+ files): project 1: 36 min to 1 hr 1 min project 2: 40 min to 1 hr 13 min So again the request to not fingerprint if the source and/or target project has fingerprinting disabled. I am going to downgrade the plugin again as the above numbers are not acceptable for us. And Jørgen, you refer version numbers of the plugin starting with zero (0.19, 0.21). Shouldn't this be 1.19 and 1.21?
        Hide
        spyder Andrew Herron added a comment - - edited

        Please let us turn this behaviour off. I'm having a really weird problem where I have two "see fingerprints" links in each build of project A, and when I click on either one all I see is the artifacts this plugin copied in. I want to fingerprint those along with other things I have set to fingerprint from project B, which are copied in via a separate dependency tool rather than this plugin.

        If I look at the target file in project B's build fingerprints I can see the link was made successfully, but in project A's build entry there is no way to find that link (and thus it does not show up as a change in dependency in the build changes list).

        Show
        spyder Andrew Herron added a comment - - edited Please let us turn this behaviour off. I'm having a really weird problem where I have two "see fingerprints" links in each build of project A, and when I click on either one all I see is the artifacts this plugin copied in. I want to fingerprint those along with other things I have set to fingerprint from project B, which are copied in via a separate dependency tool rather than this plugin. If I look at the target file in project B's build fingerprints I can see the link was made successfully, but in project A's build entry there is no way to find that link (and thus it does not show up as a change in dependency in the build changes list).
        Hide
        alikulin Alexander Likulin added a comment -

        I've got two "see fingerprints" too. If I look into build.xml I can see two <hudson.tasks.Fingerprinter_-FingerprintAction> tag with different content.

        Show
        alikulin Alexander Likulin added a comment - I've got two "see fingerprints" too. If I look into build.xml I can see two <hudson.tasks.Fingerprinter_-FingerprintAction> tag with different content.
        Hide
        krw1243 Kelsey Prantis added a comment -

        I've created a pull request to add a checkbox to control the fingerprinting of artifacts: https://github.com/jenkinsci/copyartifact-plugin/pull/10

        Show
        krw1243 Kelsey Prantis added a comment - I've created a pull request to add a checkbox to control the fingerprinting of artifacts: https://github.com/jenkinsci/copyartifact-plugin/pull/10
        Hide
        spyder Andrew Herron added a comment -

        Is there going to be any movement on that pull request? Or do I have to make a custom build of the plugin?

        Show
        spyder Andrew Herron added a comment - Is there going to be any movement on that pull request? Or do I have to make a custom build of the plugin?
        Hide
        msolnit msolnit added a comment -

        +1. This issue also appears to break aggregated test results, presumably because of the duplicate "<hudson.tasks.Fingerprinter_-FingerprintAction>" XML elements.

        Show
        msolnit msolnit added a comment - +1. This issue also appears to break aggregated test results, presumably because of the duplicate "<hudson.tasks.Fingerprinter_-FingerprintAction>" XML elements.
        Hide
        scm_issue_link SCM/JIRA link daemon added a comment -

        Code changed in jenkins
        User: Kelsey Prantis
        Path:
        src/main/java/hudson/plugins/copyartifact/Copier.java
        src/main/java/hudson/plugins/copyartifact/CopyArtifact.java
        src/main/java/hudson/plugins/copyartifact/FilePathCopyMethod.java
        src/main/java/hudson/plugins/copyartifact/FingerprintingCopyMethod.java
        src/main/resources/hudson/plugins/copyartifact/CopyArtifact/config.jelly
        src/main/webapp/help-flatten-optional.html
        src/test/java/hudson/plugins/copyartifact/CopyArtifactTest.java
        http://jenkins-ci.org/commit/copyartifact-plugin/d10dad4341bc2495f469dc5cd4534bbe5e58b13c
        Log:
        JENKINS-12134 - Make fingerprinting artifacts optional.

        Show
        scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kelsey Prantis Path: src/main/java/hudson/plugins/copyartifact/Copier.java src/main/java/hudson/plugins/copyartifact/CopyArtifact.java src/main/java/hudson/plugins/copyartifact/FilePathCopyMethod.java src/main/java/hudson/plugins/copyartifact/FingerprintingCopyMethod.java src/main/resources/hudson/plugins/copyartifact/CopyArtifact/config.jelly src/main/webapp/help-flatten-optional.html src/test/java/hudson/plugins/copyartifact/CopyArtifactTest.java http://jenkins-ci.org/commit/copyartifact-plugin/d10dad4341bc2495f469dc5cd4534bbe5e58b13c Log: JENKINS-12134 - Make fingerprinting artifacts optional.
        Hide
        mwebber mwebber added a comment -

        In v1.29 of the Copy Artifact Plugin, fingerprinting was made optional.

        Show
        mwebber mwebber added a comment - In v1.29 of the Copy Artifact Plugin, fingerprinting was made optional.

          People

          • Assignee:
            mindless Alan Harder
            Reporter:
            jorgenpt Jørgen Tjernø
          • Votes:
            7 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: