Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19066

Do not spawn multiple files for fingerprints

    Details

    • Similar Issues:

      Description

      Fingerprints are stored in a large number of small XML files in a certain folder (by default, called fingerprints). This makes it really slow to move around data by file copying, as the OS will search the files one-by-one in order to copy, and they tend to be very scattered on the physical disk space. Further this is very space-inneficient, due to a lot of wasted disk space due to file allocation tables which typically allocates multiples of 4 kb blocks for files. Further, space is wasted in some directory allocation tables too.

      Please, put the fingerprints together in a single file, or at least group them in a smaller number of files.

        Attachments

          Activity

          Hide
          trejkaz trejkaz added a comment - - edited

          Here's a quick back of the envelope calculation for our system.

          Compile:

          • Number of artifacts
            • Windows - 13,911
            • Linux - 13,818
            • total = 27,729
          • Keep max 20 builds with artifacts
            • total = 554,580

          Release:

          • Number of artifacts
            • Windows - 272
            • Linux - 278
            • Mac - 274
            • total = 824
          • Keep max 1 builds with artifacts
            • total = 824

          Total files expected from builds we're keeping: 554,580 + 824 = 555,404. (Compared to 15,273,772 files actually present.)

          Fingerprint cleanup.log (took a while to find because it wasn't in /var/log like the main Jenkins log...) has 8 million lines and all lines start with either "possibly trimming" or "deleting obsolete".

          Show
          trejkaz trejkaz added a comment - - edited Here's a quick back of the envelope calculation for our system. Compile: Number of artifacts Windows - 13,911 Linux - 13,818 total = 27,729 Keep max 20 builds with artifacts total = 554,580 Release: Number of artifacts Windows - 272 Linux - 278 Mac - 274 total = 824 Keep max 1 builds with artifacts total = 824 Total files expected from builds we're keeping: 554,580 + 824 = 555,404. (Compared to 15,273,772 files actually present.) Fingerprint cleanup.log (took a while to find because it wasn't in /var/log like the main Jenkins log...) has 8 million lines and all lines start with either "possibly trimming" or "deleting obsolete".
          Hide
          danielbeck Daniel Beck added a comment -

          Uh, yeah, Jenkins cannot properly handle 28k artifacts per build. Just like it cannot really deal without some limitations with 100k projects or 5k connected slaves. I'm not surprised.

          You could just disable fingerprinting for these projects as well. Or do you need to know about every single one of these files where it originated?

          Show
          danielbeck Daniel Beck added a comment - Uh, yeah, Jenkins cannot properly handle 28k artifacts per build. Just like it cannot really deal without some limitations with 100k projects or 5k connected slaves. I'm not surprised. You could just disable fingerprinting for these projects as well. Or do you need to know about every single one of these files where it originated?
          Hide
          trejkaz trejkaz added a comment -

          "Fingerprint all copied artifacts" is unchecked for these projects already, unless you're referring to another setting we haven't been able to find yet.

          Show
          trejkaz trejkaz added a comment - "Fingerprint all copied artifacts" is unchecked for these projects already, unless you're referring to another setting we haven't been able to find yet.
          Hide
          danielbeck Daniel Beck added a comment -

          There's an option to fingerprint archived artifacts in the Advanced section of the post build step.

          Show
          danielbeck Daniel Beck added a comment - There's an option to fingerprint archived artifacts in the Advanced section of the post build step.
          Hide
          trejkaz trejkaz added a comment -

          Not seeing it. I only see the one I already mentioned. Screenshot:

          Show
          trejkaz trejkaz added a comment - Not seeing it. I only see the one I already mentioned. Screenshot:

            People

            • Assignee:
              marcsanfacon Marc Sanfacon
              Reporter:
              victorwss Victor Silva
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: