Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51654

Performance low with Jenkins core

    Details

    • Similar Issues:

      Description

      Hi there!

      We recently migrated from Jenkins core 2.46.3 with active choice plugin 1.5.3 to Jenkins core 2.107.2 with active choice plugin 2.1.

      Before the update, our jobs were working fine, but since the update, the UI using this plugin has gone much much slower.

      When Jenkins has just booted, it is pretty fast. However, the longer Jenkins has been running, the slower it gets (or so it seems). After a few days running, it takes from 10 to 30 minutes to display the arguments of a "build with parameters" page. We also observed a CPU peak on the Jenkins server during that time.

      We tried reverting the active choice plugin to 2.0 and even 1.5.3 - still with Jenkins core 2.107.2 but this didn't help. This is why I suspect the problem comes from a compatibility with the Jenkins core.

      In the specific job that is so slow, we use about 9 parameters.  Among them 7 of them are or the type Active Choices Reactive Parameter. They are running scripts referencing the first two parameters.

      In general, it is also not clear to me how this plugin works in terms of which parameter gets "resolved" first, when is which script called, etc.

      • Is it also something other observe?
      • Is there a work around?
      • Is there some document about how the plugin works internally?

      This issue has become a blocker for us, that's why I am happy to provide more details for troubleshooting and even to contribute to fixing the plugin if I am given some support.

      By the way, there's definitely a few issues, but this plugin is awesome! Thanks to all the contributors.

      Cheers,

       

       

       
       

        Attachments

          Activity

          Hide
          kinow Bruno P. Kinoshita added a comment -

          Thanks for reporting the bug Bernard Landon.

          Some days ago I worked on an issue that could be related to this one https://issues.jenkins-ci.org/browse/JENKINS-39593

          I do not have an instance of Jenkins with enough jobs and builds to reproduce the slowness. That's my main challenge when working on these two issues.

          What would be grand, is if someone could help by providing a simple case to reproduce the slowness. Something like "with this 2 jobs, everything works OK, but if you do this and that, then you will notice that the performance has downgraded significantly etc etc".

          Having a single job that shows the slowness, and perhaps its config.xml attached here, would be helpful. Or any other thing that you can think that could be helpful.

          First step will be to make sure whether it's a JS, Java, or JS+Java issue. Then investigate whether it was a change from 1.4 -> 2.x, or if something changed in Jenkins. I can spare some hours every month for the Active Choice, but pull requests, code review, etc, more than welcome (:

          Cheers

          Bruno

          Show
          kinow Bruno P. Kinoshita added a comment - Thanks for reporting the bug Bernard Landon . Some days ago I worked on an issue that could be related to this one https://issues.jenkins-ci.org/browse/JENKINS-39593 I do not have an instance of Jenkins with enough jobs and builds to reproduce the slowness. That's my main challenge when working on these two issues. What would be grand, is if someone could help by providing a simple case to reproduce the slowness. Something like "with this 2 jobs, everything works OK, but if you do this and that, then you will notice that the performance has downgraded significantly etc etc". Having a single job that shows the slowness, and perhaps its config.xml attached here, would be helpful. Or any other thing that you can think that could be helpful. First step will be to make sure whether it's a JS, Java, or JS+Java issue. Then investigate whether it was a change from 1.4 -> 2.x, or if something changed in Jenkins. I can spare some hours every month for the Active Choice, but pull requests, code review, etc, more than welcome (: Cheers Bruno
          Hide
          bernard87 Bernard Landon added a comment -

          Hi there!

          Thanks a lot for your quick reply.

          Reproducing this issue is a bit tricky as the slowness seems to be only perceptible after a few days of the Jenkins master running.

          I tried running tests with several workers querying the page and even implemented a Selenium test to make sure the parameters will be changed so the scripts associated would be triggered. I did not manage to reproduce it in my dev env. So somehow it must be different from my prod env.

          I believe, making sure all the param scripts had a valid fallback script and also refactoring a bit using a hidden param to do all the networking in one param only seem to have helped tremendously.

          To give you an idea of our setup, we had maybe 5-10 scripted parameters. Every param depending on all the others before. 

          Say you have params:

          • A
          • B depends on A
          • C depends on B and C
          • D depends on A, B and C
          • E depends on A, B, C and D.

          In this scenario, one would rather have the script of each param run sequentially instead of in parallel. For instance, if A changes, it is a waste of resources to evaluate the script for D before the script for B is finished. Do you get the idea?

          Also, is this situation deterministic? Is it possible to tell what script will be run first? Because if not, some funny situations might happen. Scripts might ran way more time than they need to.

          At the moment we think we have a stable situation with plugin version 1.5.3. We want to update again to 2.0+ and do more testing as well. We'll post here if this is relevant.

          Thanks for the support and the great work! 

           

           

           

           

           
           

          Show
          bernard87 Bernard Landon added a comment - Hi there! Thanks a lot for your quick reply. Reproducing this issue is a bit tricky as the slowness seems to be only perceptible after a few days of the Jenkins master running. I tried running tests with several workers querying the page and even implemented a Selenium test to make sure the parameters will be changed so the scripts associated would be triggered. I did not manage to reproduce it in my dev env. So somehow it must be different from my prod env. I believe, making sure all the param scripts had a valid fallback script and also refactoring a bit using a hidden param to do all the networking in one param only seem to have helped tremendously. To give you an idea of our setup, we had maybe 5-10 scripted parameters. Every param depending on all the others before.  Say you have params: A B depends on A C depends on B and C D depends on A, B and C E depends on A, B, C and D. In this scenario, one would rather have the script of each param run sequentially instead of in parallel. For instance, if A changes, it is a waste of resources to evaluate the script for D before the script for B is finished. Do you get the idea? Also, is this situation deterministic? Is it possible to tell what script will be run first? Because if not, some funny situations might happen. Scripts might ran way more time than they need to. At the moment we think we have a stable situation with plugin version 1.5.3. We want to update again to 2.0+ and do more testing as well. We'll post here if this is relevant. Thanks for the support and the great work!             

            People

            • Assignee:
              kinow Bruno P. Kinoshita
              Reporter:
              bernard87 Bernard Landon
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: