Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54073

"Pipeline: Job" plugin causes Jenkins Master CPU to peg to ~100% during pipelines with parallel steps

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      workflow-api 2.31

      Description

      The graph here in this description more clearly depicts whats going on. I'm the only one using this jenkins server today...

      Around 5pm, i kicked off a pipeine. The pipeline builds a container and then uses that container to run some pytests...

      It is during that last "postbuild" phase, the CPU runs real hot. While the pytest shards are running, they are just doing a pytest run and then i capture the junit.xml file.

      The reason why this is a problem is that with too many of these running at the same time, Jenkins blips out and we cannot contact the web interface because when the CPU is pegged.. it basically crashes the master.

      Here are details about my jenkins:

      My jenkins is 100% current:

       

        Attachments

          Issue Links

            Activity

            piratejohnny Jon B created issue -
            piratejohnny Jon B made changes -
            Field Original Value New Value
            Summary Jenkins Master Host CPU pegs at 50% Jenkins Master Host CPU pegs during parallel jobs
            piratejohnny Jon B made changes -
            Attachment image-2018-10-14-17-35-11-037.png [ 44776 ]
            piratejohnny Jon B made changes -
            Description AWS CPU Reports:

            !image-2018-10-14-14-48-32-774.png!

            SSH'ed into the Jenkins Master Host. Using nmon, I see:

            !image-2018-10-14-14-50-10-820.png!

            Docker stats on the host shows me:

            !image-2018-10-14-14-50-53-021.png!

             
            !image-2018-10-14-17-35-11-037.png|width=671,height=140!
            piratejohnny Jon B made changes -
            Description !image-2018-10-14-17-35-11-037.png|width=671,height=140! The graph here in this description more clearly depicts whats going on. I'm the only one using this jenkins server today... you can 

            !image-2018-10-14-17-35-11-037.png|width=657,height=137!
            piratejohnny Jon B made changes -
            Attachment image-2018-10-14-17-36-52-958.png [ 44777 ]
            piratejohnny Jon B made changes -
            Description The graph here in this description more clearly depicts whats going on. I'm the only one using this jenkins server today... you can 

            !image-2018-10-14-17-35-11-037.png|width=657,height=137!
            The graph here in this description more clearly depicts whats going on. I'm the only one using this jenkins server today... you can 

            !image-2018-10-14-17-35-11-037.png|width=657,height=137!

            Around 5pm, i kicked off a pipeine. The pipeline builds a container and then uses that container to run some pytests...

            !image-2018-10-14-17-36-52-958.png|width=329,height=367!

            It is during that last "postbuild" phase, the CPU runs real hot. While the pytest shards are running, they are just doing a pytest run and then i capture the junit.xml file.

            The reason why this is a problem is that with too many of these running at the same time, Jenkins blips out and we cannot contact the web interface because when the CPU is pegged.. it basically crashes the master.

            Here are details about my jenkins:

            My jenkins is 100% current:
             * Jenkins ver. 2.147
             * Please note I did today accept the plugin updates described on [https://github.com/jenkinsci/jep/blob/master/jep/210/README.adoc#abstract]

             
            piratejohnny Jon B made changes -
            Attachment image-2018-10-14-14-48-32-774.png [ 44775 ]
            piratejohnny Jon B made changes -
            Attachment image-2018-10-14-14-50-10-820.png [ 44774 ]
            piratejohnny Jon B made changes -
            Attachment image-2018-10-14-14-50-53-021.png [ 44773 ]
            piratejohnny Jon B made changes -
            Summary Jenkins Master Host CPU pegs during parallel jobs Jenkins Master Host CPU pegs to ~100% during parallel jobs
            oleg_nenashev Oleg Nenashev made changes -
            Component/s pipeline [ 21692 ]
            Component/s core [ 15593 ]
            piratejohnny Jon B made changes -
            Attachment image-2018-10-15-00-07-35-969.png [ 44780 ]
            piratejohnny Jon B made changes -
            Attachment image-2018-10-15-00-09-21-112.png [ 44781 ]
            piratejohnny Jon B made changes -
            Summary Jenkins Master Host CPU pegs to ~100% during parallel jobs "Pipeline: Job" plugin causes Jenkins Master CPU to peg to ~100% during pipelines with parallel steps
            jglick Jesse Glick made changes -
            Labels performance regression
            jglick Jesse Glick made changes -
            Component/s workflow-job-plugin [ 21716 ]
            Component/s pipeline [ 21692 ]
            jglick Jesse Glick made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-38381 [ JENKINS-38381 ]
            kutzi kutzi made changes -
            Attachment threaddump_jenkins-54073.txt [ 44820 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-api PR 81 (Web Link)" [ 21937 ]
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            svanoort Sam Van Oort made changes -
            Priority Major [ 3 ] Blocker [ 1 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "update-center2 PR 247 (Web Link)" [ 21939 ]
            piratejohnny Jon B made changes -
            Description The graph here in this description more clearly depicts whats going on. I'm the only one using this jenkins server today... you can 

            !image-2018-10-14-17-35-11-037.png|width=657,height=137!

            Around 5pm, i kicked off a pipeine. The pipeline builds a container and then uses that container to run some pytests...

            !image-2018-10-14-17-36-52-958.png|width=329,height=367!

            It is during that last "postbuild" phase, the CPU runs real hot. While the pytest shards are running, they are just doing a pytest run and then i capture the junit.xml file.

            The reason why this is a problem is that with too many of these running at the same time, Jenkins blips out and we cannot contact the web interface because when the CPU is pegged.. it basically crashes the master.

            Here are details about my jenkins:

            My jenkins is 100% current:
             * Jenkins ver. 2.147
             * Please note I did today accept the plugin updates described on [https://github.com/jenkinsci/jep/blob/master/jep/210/README.adoc#abstract]

             
            The graph here in this description more clearly depicts whats going on. I'm the only one using this jenkins server today...

            !image-2018-10-14-17-35-11-037.png|width=657,height=137!

            Around 5pm, i kicked off a pipeine. The pipeline builds a container and then uses that container to run some pytests...

            !image-2018-10-14-17-36-52-958.png|width=329,height=367!

            It is during that last "postbuild" phase, the CPU runs real hot. While the pytest shards are running, they are just doing a pytest run and then i capture the junit.xml file.

            The reason why this is a problem is that with too many of these running at the same time, Jenkins blips out and we cannot contact the web interface because when the CPU is pegged.. it basically crashes the master.

            Here are details about my jenkins:

            My jenkins is 100% current:
             * Jenkins ver. 2.147
             * Please note I did today accept the plugin updates described on [https://github.com/jenkinsci/jep/blob/master/jep/210/README.adoc#abstract]

             
            jglick Jesse Glick made changes -
            Component/s durable-task-plugin [ 18622 ]
            Component/s workflow-durable-task-step-plugin [ 21715 ]
            Component/s workflow-job-plugin [ 21716 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-52729 [ JENKINS-52729 ]
            jglick Jesse Glick made changes -
            Component/s workflow-api-plugin [ 21711 ]
            Component/s durable-task-plugin [ 18622 ]
            Component/s workflow-durable-task-step-plugin [ 21715 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            dnusbaum Devin Nusbaum made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Released As workflow-api 2.31

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                piratejohnny Jon B
              • Votes:
                2 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: