Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16301

Fingerprint performance

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      A user working with files in the 100+ Mb range asked whether fingerprinting them was expensive. Probably it is not that expensive but it is likely the implementation could be optimized and at least measured.

      As far as the I/O goes, FilePath.digest uses FileInputStream rather than NIO—hmm, maybe OK. It does not use buffering, which may or may not be an issue here. Then it calls Util.getDigestOf.

      That uses the JRE’s MD5 implementation which AFAIK takes advantage of any cryptographic hardware acceleration when available. It does read into a 1024-byte buffer however which seems excessively small. (And uses a proxy stream rather than simply calling the update method, which is odd.) http://stackoverflow.com/questions/9321912/very-slow-when-generaing-md5-using-java-with-large-file suggests using a custom library though this may be overkill; others suggest DigestUtils from Commons Codec. A

        Attachments

          Issue Links

            Activity

            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            changelog.html
            core/src/main/java/hudson/Util.java
            test/src/test/java/hudson/maven/MavenBuildTest.java
            http://jenkins-ci.org/commit/jenkins/e0a3a1ddf5fa08e80a94a565dd05c39f76de6e93
            Log:
            [FIXED JENKINS-18178] Reverting an inessential part of the fix of JENKINS-16301 since it broke some Maven builds.
            Apparently the Maven 2 process factory fails to adequately insulate the Maven process from Jenkins library dependencies;
            if you specify a build extension that can override dependencies used by Jenkins core classes!
            A real fix would involve changing class loader delegation, since presumably similar bugs could still occur.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/hudson/Util.java test/src/test/java/hudson/maven/MavenBuildTest.java http://jenkins-ci.org/commit/jenkins/e0a3a1ddf5fa08e80a94a565dd05c39f76de6e93 Log: [FIXED JENKINS-18178] Reverting an inessential part of the fix of JENKINS-16301 since it broke some Maven builds. Apparently the Maven 2 process factory fails to adequately insulate the Maven process from Jenkins library dependencies; if you specify a build extension that can override dependencies used by Jenkins core classes! A real fix would involve changing class loader delegation, since presumably similar bugs could still occur.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/FilePath.java
            core/src/main/java/hudson/Util.java
            maven-plugin/src/main/java/hudson/maven/ExecutedMojo.java
            maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java
            maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
            http://jenkins-ci.org/commit/jenkins/11953750ab422bcec28d12a436867d0888313aaa
            Log:
            [JENKINS-16301 Use a buffered input stream to get somewhat better performance also from Maven archiving.
            Introducing Util.getDigestOf(File) as a convenience.

            Compare: https://github.com/jenkinsci/jenkins/compare/d18a27e5bed4...11953750ab42

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/Util.java maven-plugin/src/main/java/hudson/maven/ExecutedMojo.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java http://jenkins-ci.org/commit/jenkins/11953750ab422bcec28d12a436867d0888313aaa Log: [ JENKINS-16301 Use a buffered input stream to get somewhat better performance also from Maven archiving. Introducing Util.getDigestOf(File) as a convenience. Compare: https://github.com/jenkinsci/jenkins/compare/d18a27e5bed4...11953750ab42
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2748
            [FIXED JENKINS-18178] Reverting an inessential part of the fix of JENKINS-16301 since it broke some Maven builds. (Revision e0a3a1ddf5fa08e80a94a565dd05c39f76de6e93)
            [JENKINS-16301 Use a buffered input stream to get somewhat better performance also from Maven archiving. (Revision 11953750ab422bcec28d12a436867d0888313aaa)

            Result = SUCCESS
            Jesse Glick : e0a3a1ddf5fa08e80a94a565dd05c39f76de6e93
            Files :

            • core/src/main/java/hudson/Util.java
            • changelog.html
            • test/src/test/java/hudson/maven/MavenBuildTest.java

            Jesse Glick : 11953750ab422bcec28d12a436867d0888313aaa
            Files :

            • maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
            • maven-plugin/src/main/java/hudson/maven/ExecutedMojo.java
            • maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java
            • core/src/main/java/hudson/FilePath.java
            • core/src/main/java/hudson/Util.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2748 [FIXED JENKINS-18178] Reverting an inessential part of the fix of JENKINS-16301 since it broke some Maven builds. (Revision e0a3a1ddf5fa08e80a94a565dd05c39f76de6e93) [ JENKINS-16301 Use a buffered input stream to get somewhat better performance also from Maven archiving. (Revision 11953750ab422bcec28d12a436867d0888313aaa) Result = SUCCESS Jesse Glick : e0a3a1ddf5fa08e80a94a565dd05c39f76de6e93 Files : core/src/main/java/hudson/Util.java changelog.html test/src/test/java/hudson/maven/MavenBuildTest.java Jesse Glick : 11953750ab422bcec28d12a436867d0888313aaa Files : maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java maven-plugin/src/main/java/hudson/maven/ExecutedMojo.java maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/Util.java
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/hudson/maven/ExecutedMojo.java
            src/main/java/hudson/maven/reporters/MavenArtifact.java
            src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
            http://jenkins-ci.org/commit/maven-plugin/db2b69174c511a1cba440ce3683de4cd6676734b
            Log:
            [JENKINS-16301 Use a buffered input stream to get somewhat better performance also from Maven archiving.
            Introducing Util.getDigestOf(File) as a convenience.
            Originally-Committed-As: 11953750ab422bcec28d12a436867d0888313aaa

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/maven/ExecutedMojo.java src/main/java/hudson/maven/reporters/MavenArtifact.java src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java http://jenkins-ci.org/commit/maven-plugin/db2b69174c511a1cba440ce3683de4cd6676734b Log: [ JENKINS-16301 Use a buffered input stream to get somewhat better performance also from Maven archiving. Introducing Util.getDigestOf(File) as a convenience. Originally-Committed-As: 11953750ab422bcec28d12a436867d0888313aaa
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            test/src/test/java/hudson/maven/MavenBuildTest.java
            http://jenkins-ci.org/commit/maven-plugin/1342c5f819d6672ce53238822a93dddc087f0bd8
            Log:
            Reverting an inessential part of the fix of JENKINS-16301 since it broke some Maven builds. Apparently the Maven 2 process factory fails to adequately insulate the Maven process from Jenkins library dependencies; if you specify a build extension that can override dependencies used by Jenkins core classes! A real fix would involve changing class loader delegation, since presumably similar bugs could still occur.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: test/src/test/java/hudson/maven/MavenBuildTest.java http://jenkins-ci.org/commit/maven-plugin/1342c5f819d6672ce53238822a93dddc087f0bd8 Log: Reverting an inessential part of the fix of JENKINS-16301 since it broke some Maven builds. Apparently the Maven 2 process factory fails to adequately insulate the Maven process from Jenkins library dependencies; if you specify a build extension that can override dependencies used by Jenkins core classes! A real fix would involve changing class loader delegation, since presumably similar bugs could still occur.

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                jglick Jesse Glick
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: