Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22456

github plugin too many open files unclosed github-polling.log filehandles causes all builds to fail

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Github plugin creates and doesn't close thousands of github-polling.log filehandles, causing all builds to fail. Presents as "too many open files" error on attempting to open a file over the max open files limit, e.g.,

      Caused by: java.io.IOException: Too many open files

      or

      Caused by: java.io.FileNotFoundException: [...] (Too many open files)

      Github Plugin Version: 1.8
      (Jenkins Version: 1.55x)

        Attachments

          Issue Links

            Activity

            Hide
            webb webb phillips added a comment -

            We've worked around the problem by no longer using the github plugin for polling/queueing builds, so if this doesn't affect anyone else, it's no longer critical.

            Might I suggest though that a better implementation of the polling feature would be to do github polling per repo instead of per job? We have hundreds of jobs using the same repo, and at the least, generating many fewer github-polling.log files would cause the unclosed filehandles to build up much more slowly.

            Show
            webb webb phillips added a comment - We've worked around the problem by no longer using the github plugin for polling/queueing builds, so if this doesn't affect anyone else, it's no longer critical. Might I suggest though that a better implementation of the polling feature would be to do github polling per repo instead of per job? We have hundreds of jobs using the same repo, and at the least, generating many fewer github-polling.log files would cause the unclosed filehandles to build up much more slowly.
            Hide
            jglick Jesse Glick added a comment -

            If you can reproduce the problem, you can use the File Leak Detector plugin to pinpoint the cause.

            Show
            jglick Jesse Glick added a comment - If you can reproduce the problem, you can use the File Leak Detector plugin to pinpoint the cause.
            Hide
            webb webb phillips added a comment - - edited

            I confirmed (/file-handles) that job-specific github-polling.log files were not being closed. Running garbage-collection (/gc) and then revisiting open file handles (/file-handles) showed that github-polling.log files from e.g., 45 minutes earlier were still open. Not using the github plugin for polling solved the problem well enough for our needs, and I don't plan to reproduce the problem. If you would like to close as "Won't fix" or "Can't reproduce" that's totally fine by me.

            Show
            webb webb phillips added a comment - - edited I confirmed (/file-handles) that job-specific github-polling.log files were not being closed. Running garbage-collection (/gc) and then revisiting open file handles (/file-handles) showed that github-polling.log files from e.g., 45 minutes earlier were still open. Not using the github plugin for polling solved the problem well enough for our needs, and I don't plan to reproduce the problem. If you would like to close as "Won't fix" or "Can't reproduce" that's totally fine by me.
            Hide
            jglick Jesse Glick added a comment -

            I installed the File Leak Detector plugin and activated it, but could not reproduce. Various accesses to github-polling.log all reliably close the file. However I also did not manage to get a hook added, despite using a valid API token to connect to my own repo:

            java.io.FileNotFoundException: https://api.github.com/repos/jglick/simple-maven-project-with-tests/hooks
            	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624)
            	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
            	at org.kohsuke.github.Requester.parse(Requester.java:383)
            	at org.kohsuke.github.Requester._to(Requester.java:185)
            	at org.kohsuke.github.Requester.to(Requester.java:160)
            	at org.kohsuke.github.GHRepository.createHook(GHRepository.java:754)
            	at com.cloudbees.jenkins.GitHubPushTrigger.createJenkinsHook(GitHubPushTrigger.java:161)
            
            Show
            jglick Jesse Glick added a comment - I installed the File Leak Detector plugin and activated it, but could not reproduce. Various accesses to github-polling.log all reliably close the file. However I also did not manage to get a hook added, despite using a valid API token to connect to my own repo: java.io.FileNotFoundException: https://api.github.com/repos/jglick/simple-maven-project-with-tests/hooks at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254) at org.kohsuke.github.Requester.parse(Requester.java:383) at org.kohsuke.github.Requester._to(Requester.java:185) at org.kohsuke.github.Requester.to(Requester.java:160) at org.kohsuke.github.GHRepository.createHook(GHRepository.java:754) at com.cloudbees.jenkins.GitHubPushTrigger.createJenkinsHook(GitHubPushTrigger.java:161)
            Hide
            jglick Jesse Glick added a comment -

            Possibly a side effect of JENKINS-27041.

            Show
            jglick Jesse Glick added a comment - Possibly a side effect of JENKINS-27041 .

              People

              • Assignee:
                Unassigned
                Reporter:
                webb webb phillips
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: