Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59311

Poor performance when using 'grep'

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: pipeline
    • Labels:
      None
    • Environment:
      jenkins ver 2.176.3
      jenkins ver 2.190.1
    • Similar Issues:

      Description

      I have a bunch of freestyle jobs, that compile and build an installer for my software on two compilers and in debug/release. Each of them takes aroung half an hour.

      Then I tried to switch the build to a pipeline build, integrated build of debug and release to one job and additionally integrated several other stuff (cppcheck,...). This build ended up in taking 3-4 hours for quite a while (which I did not expect and regard it as slow also).

      Since some weeks now my pipeline job suddenly started taking 8 hours while the freestyle jobs duration staid the same.

      Something about my environment: it's a jenkins-installation as service on a windows server 2016 machine. I am not aware of any changes on the machine installation or job configuration since the job got slower. It's not a single step in the pipeline that eats all the time. It's the overall performance and the time is distributed over all steps. E.g. the compile and build installer step (which is the entire job done by the freestyle job) alone takes 3 hours.

      Unfortunately I don't have the build history long enough to be able to identify plugin-versions from before and after the buildtime-increase.

      Please contact me in case I can help with any additional information.

        Attachments

          Activity

          Hide
          chrisfetz Christoph Fetzer added a comment - - edited

          I maybe got the point:

          My build deep inside makes use of a compiler | tee | grep,... and therefore is splitting the output into a file at one point.

          Not too far ago I switched from a formerly deprected warnings parser to the warnings next generation plugin and therefore had to insert another | tee - split. I'm not yet 100% sure if this exactly matches the point were buildtimes increased but at least it might be close to. I'll set up a test.

          At least my way quicker freestyle build does not make use of the inner  | tee | grep!

          On the other hand this | tee provides a list of other problems and I'd like to avoid it at any level - also for warnings next generation. Is there another way to split build output to a file and in the same step keep the return code?

          Show
          chrisfetz Christoph Fetzer added a comment - - edited I maybe got the point: My build deep inside makes use of a compiler | tee | grep,... and therefore is splitting the output into a file at one point. Not too far ago I switched from a formerly deprected warnings parser to the warnings next generation plugin and therefore had to insert another | tee - split. I'm not yet 100% sure if this exactly matches the point were buildtimes increased but at least it might be close to. I'll set up a test. At least my way quicker freestyle build does not make use of the inner  | tee | grep! On the other hand this | tee provides a list of other problems and I'd like to avoid it at any level - also for warnings next generation. Is there another way to split build output to a file and in the same step keep the return code?
          Hide
          chrisfetz Christoph Fetzer added a comment - - edited

          In fact, it is the grep. But that's in there for a longer time. Is there some real relation between a grep (from msys) and some components of jenkins that might explain the significant decrease in the speed?

           

          Second issue I observed with warnings-ng - for my tests I changed the use of recordIssued from:

          recordIssues sourceCodeEncoding: 'ISO-8859-1',
                   tools: [groovyScript(parserId: 'MSVC-Warning', pattern:'**/build-makedep.log', reportEncoding:'ISO-8859-1'),
                                  groovyScript(parserId: 'MSVC-Error', pattern:'**/build-makedep.log', reportEncoding:'ISO-8859-1') ]
          

          to

           recordIssues sourceCodeEncoding: 'ISO-8859-1',
                              tools: [groovyScript(parserId: 'MSVC-Warning', reportEncoding:'ISO-8859-1'),
                                  groovyScript(parserId: 'MSVC-Error', reportEncoding:'ISO-8859-1') ]
          

          (note the missing "pattern:") but that does not work as documented (scanning the console output without the need of an intermediate file).

           

          Ecuse me, that's not an issue, that's documented - but ugly!

          Show
          chrisfetz Christoph Fetzer added a comment - - edited In fact, it is the grep. But that's in there for a longer time. Is there some real relation between a grep (from msys) and some components of jenkins that might explain the significant decrease in the speed?   Second issue I observed with warnings-ng - for my tests I changed the use of recordIssued from: recordIssues sourceCodeEncoding: 'ISO-8859-1' , tools: [groovyScript(parserId: 'MSVC-Warning' , pattern: '**/build-makedep.log' , reportEncoding: 'ISO-8859-1' ), groovyScript(parserId: 'MSVC-Error' , pattern: '**/build-makedep.log' , reportEncoding: 'ISO-8859-1' ) ] to recordIssues sourceCodeEncoding: 'ISO-8859-1' , tools: [groovyScript(parserId: 'MSVC-Warning' , reportEncoding: 'ISO-8859-1' ), groovyScript(parserId: 'MSVC-Error' , reportEncoding: 'ISO-8859-1' ) ] (note the missing "pattern:") but that does not work as documented (scanning the console output without the need of an intermediate file).   Ecuse me, that's not an issue, that's documented - but ugly!
          Hide
          chrisfetz Christoph Fetzer added a comment -

          The problem seems to have disappeared after the latest LTS update to v2.190.1. At least at the first test build after that update.

          Show
          chrisfetz Christoph Fetzer added a comment - The problem seems to have disappeared after the latest LTS update to v2.190.1. At least at the first test build after that update.
          Hide
          chrisfetz Christoph Fetzer added a comment -

          Was solved somewhere between V2.176.2 and v2.190.1.

          Relation to any plugin or module not clear.

          Solution is released.

          Show
          chrisfetz Christoph Fetzer added a comment - Was solved somewhere between V2.176.2 and v2.190.1. Relation to any plugin or module not clear. Solution is released.
          Hide
          chrisfetz Christoph Fetzer added a comment - - edited

          This jenkins installation is driving me nuts - the slow behaviour came back.

          The last quick run was on September, 27th. After that the build failed for some days due to issues with the code and the libraries used. The first completing build after that on Oct. 9th was slow again (same times - ~45min vs 3:40hours).

          OK, I noticed, there are failed builds with indicatino of slow behaviour since October, 2nd. The builds before failed after only 6mins.

          I implemented every updates in that time as soon as I realised them. There were a number of them in that time frame. Is there a log that shows the plugins updated and the version change? With that I could restore the state of September 27th and apply all the updates step by step until the build gets slower again.

           

          Show
          chrisfetz Christoph Fetzer added a comment - - edited This jenkins installation is driving me nuts - the slow behaviour came back. The last quick run was on September, 27th. After that the build failed for some days due to issues with the code and the libraries used. The first completing build after that on Oct. 9th was slow again (same times - ~45min vs 3:40hours). OK, I noticed, there are failed builds with indicatino of slow behaviour since October, 2nd. The builds before failed after only 6mins. I implemented every updates in that time as soon as I realised them. There were a number of them in that time frame. Is there a log that shows the plugins updated and the version change? With that I could restore the state of September 27th and apply all the updates step by step until the build gets slower again.  
          Hide
          chrisfetz Christoph Fetzer added a comment -

          The speed decrease dissappeared again.

          Maybe it got lost a little through the history of all the comments:

          • the decrease happens only in one step, not distributed over all
          • in case it runs slowly, removing a '| grep' from a makefile build rule immediately takes down the speed decrease

          I now started tracking all the plugin updates to maybe get a correlation between jenkins updates and speed impact

          Show
          chrisfetz Christoph Fetzer added a comment - The speed decrease dissappeared again. Maybe it got lost a little through the history of all the comments: the decrease happens only in one step, not distributed over all in case it runs slowly, removing a '| grep' from a makefile build rule immediately takes down the speed decrease I now started tracking all the plugin updates to maybe get a correlation between jenkins updates and speed impact

            People

            • Assignee:
              Unassigned
              Reporter:
              chrisfetz Christoph Fetzer
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: