Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49745

Too many CrumbFilter log entries per second

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The hudson.security.csrf.CrumbFilter generates so many log entries it causes parts of Jenkins to stall until the rate of log messages slows down.

      2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 418axxxx20cb74b577eaae393aa8ac0e. Will check remaining parameters for a valid one...
      2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403.

       
      The amount of these logs was causing my Jenkins to stop working: The executors were not being released by jobs (even after they were done running) until the log entry could be written.

      I checked through the logs and all the entries I have are for these URLs (there could be more, due to the logs rolling so quick):

      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getDisconnectedSlaves
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOfflineSlaves
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getRunningJobs
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getSlaves
      • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getTasksInQueue

      I would suggest that either these logs shouldn't be generated at all, or only as FINE or FINER entries.

        Attachments

          Issue Links

            Activity

            Hide
            docwhat Christian Höltje added a comment - - edited

            I'm glad my extra research helped identify the culprit (dashboard-view).

            I'm not familiar enough with Jelly to know exactly what's going on but it looks like some of the problems are:

            • It doesn't check the return value (or try/catch if done that way) in case there is an error (e.g. a 403). This means there is nothing to bounce the page to the login page (if it requires auth) or refetch a CSRF token.
            • It fetches everything, even if only some items are needed.
            • This stuff, if everything is needed, could be returned as a single struct.
            • Did I miss anything? I'm not familiar with how the CSRF stuff is implemented in Jenkins.

            It also seems wrong that if the permission for Jenkins are such that anyone (anonymous) can view a URL, then GET requests for that URL shouldn't need a CSRF token and should never raise a 403.

            Show
            docwhat Christian Höltje added a comment - - edited I'm glad my extra research helped identify the culprit (dashboard-view). I'm not familiar enough with Jelly to know exactly what's going on but it looks like some of the problems are: It doesn't check the return value (or try / catch if done that way) in case there is an error (e.g. a 403). This means there is nothing to bounce the page to the login page (if it requires auth) or refetch a CSRF token. It fetches everything, even if only some items are needed. This stuff, if everything is needed, could be returned as a single struct. Did I miss anything? I'm not familiar with how the CSRF stuff is implemented in Jenkins. It also seems wrong that if the permission for Jenkins are such that anyone (anonymous) can view a URL, then GET requests for that URL shouldn't need a CSRF token and should never raise a 403.
            Hide
            danielbeck Daniel Beck added a comment -

            It also seems wrong that if the permission for Jenkins are such that anyone (anonymous) can view a URL, then GET requests for that URL shouldn't need a CSRF token and should never raise a 403.

            GET doesn't have CSRF protection, this sends POST. Possibly because it's the default.

            Show
            danielbeck Daniel Beck added a comment - It also seems wrong that if the permission for Jenkins are such that anyone (anonymous) can view a URL, then GET requests for that URL shouldn't need a CSRF token and should never raise a 403. GET doesn't have CSRF protection, this sends POST. Possibly because it's the default.
            Hide
            tgr Tobias Gruetzmacher added a comment -

            This is now a bit saner:

            • Only one request for all fields
            • After 10 unsuccessful requests, further updates are suspended

            Test build will probably be available later: https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fdashboard-view-plugin/branches

             

            Show
            tgr Tobias Gruetzmacher added a comment - This is now a bit saner: Only one request for all fields After 10 unsuccessful requests, further updates are suspended Test build will probably be available later: https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fdashboard-view-plugin/branches  
            Hide
            danielbeck Daniel Beck added a comment -
            • After 10 unsuccessful requests, further updates are suspended

            Might be interesting for you to look at what the "Jenkins is (re)starting" screen in Jenkins core does – it distinguishes between getting no response or getting a 500 response from anything different. This way, the UI survives a Jenkins restart, but would show legitimate errors if they appear.

            https://github.com/jenkinsci/jenkins/blob/b1ff026023c71202cdfc1894a33cd0c85ddc11e2/core/src/main/resources/hudson/util/HudsonIsRestarting/index.jelly#L60

            https://github.com/jenkinsci/jenkins/blob/b1ff026023c71202cdfc1894a33cd0c85ddc11e2/war/src/main/webapp/scripts/loading.js

            Show
            danielbeck Daniel Beck added a comment - After 10 unsuccessful requests, further updates are suspended Might be interesting for you to look at what the "Jenkins is (re)starting" screen in Jenkins core does – it distinguishes between getting no response or getting a 500 response from anything different. This way, the UI survives a Jenkins restart, but would show legitimate errors if they appear. https://github.com/jenkinsci/jenkins/blob/b1ff026023c71202cdfc1894a33cd0c85ddc11e2/core/src/main/resources/hudson/util/HudsonIsRestarting/index.jelly#L60 https://github.com/jenkinsci/jenkins/blob/b1ff026023c71202cdfc1894a33cd0c85ddc11e2/war/src/main/webapp/scripts/loading.js
            Hide
            tgr Tobias Gruetzmacher added a comment -

            Fixed in dashboard-view plugin 2.10.

            Show
            tgr Tobias Gruetzmacher added a comment - Fixed in dashboard-view plugin 2.10.

              People

              • Assignee:
                tgr Tobias Gruetzmacher
                Reporter:
                docwhat Christian Höltje
              • Votes:
                2 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: