Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40344

Leaving a page open past session expiry fills the logs on the master with "Found invalid crumb" warnings

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Similar Issues:

      Description

      I noticed that I had thousands of WARNING messages in my master logs this morning because some users are leaving Jenkins home pages open past the user's session expiry.

      I understand that part of the problem here is the busy-wait looping on /ajaxBuildQueue, but finding an entire log file filled with this garbage seems like a bug

      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:46 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:05:51 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:51 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:05:51 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:51 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:05:56 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:56 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:05:56 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:05:56 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:01 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:01 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:01 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:01 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:06 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:06 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:06 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:06 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:11 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:11 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:11 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:11 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:16 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:16 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:16 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:16 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:21 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:21 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:21 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:21 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:26 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:26 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:26 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:26 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      Dec 09, 2016 4:06:31 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:31 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxBuildQueue. Returning 403.
      Dec 09, 2016 4:06:31 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: Found invalid crumb 2c7b06359e83df2535c0520c1ae79753.  Will check remaining parameters for a valid one...
      Dec 09, 2016 4:06:31 PM hudson.security.csrf.CrumbFilter doFilter
      WARNING: No valid crumb was included in request for /ajaxExecutors. Returning 403.
      

        Attachments

          Issue Links

            Activity

            Hide
            aheritier Arnaud Héritier added a comment -

            \O/ Thanks Daniel Beck

            Show
            aheritier Arnaud Héritier added a comment - \O/ Thanks Daniel Beck
            Hide
            docwhat Christian Höltje added a comment - - edited

            This isn't really fixed.  I have had to resort to changing the log levels (the URL /log/levels) to prevent it from logging.

            I'm seeing things like this (from the support logs, because it was more informative):

            2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 418axxxx20cb74b577eaae393aa8ac0e. Will check remaining parameters for a valid one...
            2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403.

             

            The amount of these logs was causing my Jenkins to stop working: The executors were not being released by jobs (even after they were done running) until the log entry could be written.

            I checked through the logs and all the entries I have are for these URLs (there could be more, due to the logs rolling so quick):

            • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getDisconnectedSlaves
            • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOfflineSlaves
            • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves
            • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getRunningJobs
            • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getSlaves
            • /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getTasksInQueue

             

            Show
            docwhat Christian Höltje added a comment - - edited This isn't really fixed.  I have had to resort to changing the log levels (the URL /log/levels) to prevent it from logging. I'm seeing things like this (from the support logs, because it was more informative): 2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 418axxxx20cb74b577eaae393aa8ac0e. Will check remaining parameters for a valid one... 2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403.   The amount of these logs was causing my Jenkins to stop working: The executors were not being released by jobs (even after they were done running) until the log entry could be written. I checked through the logs and all the entries I have are for these URLs (there could be more, due to the logs rolling so quick): /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getDisconnectedSlaves /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOfflineSlaves /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getRunningJobs /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getSlaves /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getTasksInQueue  
            Hide
            danielbeck Daniel Beck added a comment -

            Christian Höltje What version of Jenkins?

            Show
            danielbeck Daniel Beck added a comment - Christian Höltje What version of Jenkins?
            Hide
            danielbeck Daniel Beck added a comment -

            Even on current versions of Jenkins, this should still happen for Christian Höltje. The error message explains why:

             2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403.

            This seems to be about a different user (joecool) having logged in since, or a crumb issuer that takes session information into account. IOW, it's not just an expired session, there's another valid session.

            The problem and fix here was about a logged out (session expired) user spamming the log; you're asking for no log messages when a logged in user sends a crumb that's invalid for them. That is a different issue.

            Show
            danielbeck Daniel Beck added a comment - Even on current versions of Jenkins, this should still happen for Christian Höltje . The error message explains why: 2018-02-24 05:17:10.406+0000 [id=20011] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /$stapler/bound/419618ba-22aa-4afb-8528-b112a604cce9/getOnlineSlaves by joecool. Returning 403. This seems to be about a different user (joecool) having logged in since, or a crumb issuer that takes session information into account. IOW, it's not just an expired session, there's another valid session. The problem and fix here was about a logged out (session expired) user spamming the log; you're asking for no log messages when a logged in user sends a crumb that's invalid for them. That is a different issue.
            Hide
            docwhat Christian Höltje added a comment -

            The Jenkins version is 2.89.4.

            I'll open a new ticket for my case.  Thanks!

            Show
            docwhat Christian Höltje added a comment - The Jenkins version is 2.89.4. I'll open a new ticket for my case.  Thanks!

              People

              • Assignee:
                danielbeck Daniel Beck
                Reporter:
                rtyler R. Tyler Croy
              • Votes:
                4 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: