Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54073

"Pipeline: Job" plugin causes Jenkins Master CPU to peg to ~100% during pipelines with parallel steps

    Details

    • Similar Issues:
    • Released As:
      workflow-api 2.31

      Description

      The graph here in this description more clearly depicts whats going on. I'm the only one using this jenkins server today...

      Around 5pm, i kicked off a pipeine. The pipeline builds a container and then uses that container to run some pytests...

      It is during that last "postbuild" phase, the CPU runs real hot. While the pytest shards are running, they are just doing a pytest run and then i capture the junit.xml file.

      The reason why this is a problem is that with too many of these running at the same time, Jenkins blips out and we cannot contact the web interface because when the CPU is pegged.. it basically crashes the master.

      Here are details about my jenkins:

      My jenkins is 100% current:

       

        Attachments

          Issue Links

            Activity

            Hide
            dnusbaum Devin Nusbaum added a comment -

            A fix for this was just released in version 2.31 of the Pipeine API Plugin. (Note that this bug should only affect you if you are running workflow-job 2.26)

            Show
            dnusbaum Devin Nusbaum added a comment - A fix for this was just released in version 2.31 of the Pipeine API Plugin. (Note that this bug should only affect you if you are running workflow-job 2.26)
            Hide
            piratejohnny Jon B added a comment -

            Sorry for having disappeared on this jira.. I will upgrade this plugin tonight and rerun my pipeline to see if it still pegs the cpu to 100% and confirm by tomorrow. Thank you for your hard work on this!

            Show
            piratejohnny Jon B added a comment - Sorry for having disappeared on this jira.. I will upgrade this plugin tonight and rerun my pipeline to see if it still pegs the cpu to 100% and confirm by tomorrow. Thank you for your hard work on this!
            Hide
            piratejohnny Jon B added a comment -

            Question: I see this message: "2.27 (2018 Nov 01) - WARNING: Although major issues have been fixed since 2.26, this version carries extra risk and is not fully backwards compatible with 2.25 or older; consider waiting a few days to update in critical environments."

            If I upgrade my plugins and it pegs to 100% cpu again, will I be able to downgrade back to a healthy state with the jenkins downgrade option? Or, should I snapshot my master just in case i have to do a full restore of the master?

            Show
            piratejohnny Jon B added a comment - Question: I see this message: "2.27 (2018 Nov 01) - WARNING: Although major issues have been fixed since 2.26, this version carries extra risk and is not fully backwards compatible with 2.25 or older; consider waiting a few days to update in critical environments." If I upgrade my plugins and it pegs to 100% cpu again, will I be able to downgrade back to a healthy state with the jenkins downgrade option? Or, should I snapshot my master just in case i have to do a full restore of the master?
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Jon B I would always encourage you to back up your master before updating plugins just in case. As far as I am aware, the compatibility notice is because if you downgrade from workflow-job 2.26 or 2.27 back to 2.25 or older, the per-step logs for builds you ran in versions 2.26-2.27 will be unreadable, the full build log will be viewable but its formatting will be a little messed up, and any in-progress builds will fail, although I have not specifically tested a downgrade to be aware of all potential issues.

            Show
            dnusbaum Devin Nusbaum added a comment - Jon B I would always encourage you to back up your master before updating plugins just in case. As far as I am aware, the compatibility notice is because if you downgrade from workflow-job 2.26 or 2.27 back to 2.25 or older, the per-step logs for builds you ran in versions 2.26-2.27 will be unreadable, the full build log will be viewable but its formatting will be a little messed up, and any in-progress builds will fail, although I have not specifically tested a downgrade to be aware of all potential issues.
            Hide
            piratejohnny Jon B added a comment -

            Sounds good...

             

            A scheduled task snapshots the master each day but I'll run an extra one before doing this upgrade.

             

            Thank you.

             

            Show
            piratejohnny Jon B added a comment - Sounds good...   A scheduled task snapshots the master each day but I'll run an extra one before doing this upgrade.   Thank you.  

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                piratejohnny Jon B
              • Votes:
                2 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: