Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56581

org.jenkinsci.remoting.protocol.IOHub takes 100% CPU

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: remoting
    • Labels:
    • Environment:
      v2.150.2, Windows Server 2012 R2, x64
    • Similar Issues:

      Description

      Our 4 Jenkins installations (all on the same HW) are running at 100% CPU after some time, until they get restarted. Sometimes all of them or only some of them are affected by this.

      I had a closer look at one installation and found out IOHub is running out of control. I added a logger to the class and it prints dozens of "0 scheduled tasks to process" per secound, nothing else.

      See the attached screenshots of VisualVM for details, the jenkis was idle at the point of record. 
      I could provide more infos if someone tells what is needed.

       

      Kind regards

      Jörg

       

        Attachments

          Activity

          Hide
          joerg1985 Jörg Sautter added a comment - - edited

          I think this https://stackoverflow.com/questions/9939989/java-nio-selector-select-returns-0-although-channels-are-ready is the reason why the IOHub is taking all of our CPU power.

          The current implementation relies on the returned number of ready channels, but this is not the total number of ready channel. It is only the number of channels gone ready while the select is performed. 

          We now changed the impementation from checking the returned number to keys.isEmpty() and evaluate this on our Jenkins installations.

          I will keep you updated and will do a PR in some days, if the issue is fixed by this change.

          Show
          joerg1985 Jörg Sautter added a comment - - edited I think this  https://stackoverflow.com/questions/9939989/java-nio-selector-select-returns-0-although-channels-are-ready is the reason why the IOHub is taking all of our CPU power. The current implementation relies on the returned number of ready channels, but this is not the total number of ready channel. It is only the number of channels gone ready while the select is performed.  We now changed the impementation from checking the returned number to keys.isEmpty() and evaluate this on our Jenkins installations. I will keep you updated and will do a PR in some days, if the issue is fixed by this change.
          Hide
          jthompson Jeff Thompson added a comment -

          Interesting. It would be great if this resolves the issues you are seeing.

          Show
          jthompson Jeff Thompson added a comment - Interesting. It would be great if this resolves the issues you are seeing.
          Hide
          lifeofguenter Gunter Grodotzki added a comment -
          Show
          lifeofguenter Gunter Grodotzki added a comment - Potentially related to: https://issues.jenkins-ci.org/browse/JENKINS-58573
          Hide
          joerg1985 Jörg Sautter added a comment -

          Hopefully the last update to this issue:

          After implementing an workaround to the not blocking selector (basically adding yield and sleep statements), another thread was consuming 100% CPU.
          It was a jetty thread with exactly the same issue like the Jenkins thread before.

          There is a jetty issue about this and a comment with the hint to update the jdk to 11.0.2, since the update none of Jenkins installations had the issue again.

          Thanks for your help!

          Kind regards

          Jörg

          Show
          joerg1985 Jörg Sautter added a comment - Hopefully the last update to this issue: After implementing an workaround to the not blocking selector (basically adding yield and sleep statements), another thread was consuming 100% CPU. It was a jetty thread with exactly the same issue like the Jenkins thread before. There is a jetty issue about this and a comment with the hint to update the jdk to 11.0.2, since the update none of Jenkins installations had the issue again. Thanks for your help! Kind regards Jörg
          Hide
          jthompson Jeff Thompson added a comment -

          Jörg Sautter, so updating to Java 11 resolved this for you? That's great information. Any reason not to close this information, then?

          Show
          jthompson Jeff Thompson added a comment - Jörg Sautter , so updating to Java 11 resolved this for you? That's great information. Any reason not to close this information, then?

            People

            • Assignee:
              jthompson Jeff Thompson
              Reporter:
              joerg1985 Jörg Sautter
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: