Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37206

ATH occasionally hangs under load - looks like an SSE client connect issue (timeout?)

    Details

    • Similar Issues:
    • Sprint:
      1.0-beta-1, pacific, atlantic, 1.0-b05/b-06

      Description

      Only seems to happen when the CI server is under a lot of load. Guess .... the SSE client connect fails (maybe due to a timeout) and doesn't handle it properly, resulting in the ATH test blocking waiting for an event that never arrives.

      Here's the error log I've seen ...

      2016-08-04 17:42:51,212 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - ========================================
      2016-08-04 17:42:51,216 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - SSE Gateway error to http://127.0.0.1:59010/sse-gateway/configure?batchId=4: 
      2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - { UNSENT: 0,
      2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   OPENED: 1,
      2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   HEADERS_RECEIVED: 2,
      2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   LOADING: 3,
      2016-08-04 17:42:51,231 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   DONE: 4,
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   readyState: 4,
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   onreadystatechange: [Function],
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   responseText: '{"status":"error","message":"No \'subscribe\' or \'unsubscribe\' configurations provided in configuration request."}',
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   responseXML: '',
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   status: 400,
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   statusText: null,
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   withCredentials: false,
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   open: [Function],
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   setDisableHeaderCheck: [Function],
      2016-08-04 17:42:51,232 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   setRequestHeader: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   getResponseHeader: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   getAllResponseHeaders: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   getRequestHeader: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   send: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   handleError: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   abort: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   addEventListener: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   removeEventListener: [Function],
      2016-08-04 17:42:51,233 [Thread-4] ERROR  (InputStreamHandler.java:60) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner -   dispatchEvent: [Function] }
      master59010|Aug 04, 2016 5:42:52 PM org.jenkinsci.plugins.ssegateway.sse.EventDispatcherFactory start WARNING: No CrumbIssuer on Jenkins instance. Some POSTs might not work.
      2016-08-04 17:42:52,068 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Connected to the Jenkins SSE Gateway.
      2016-08-04 17:42:52,074 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Subscribed to the "job" event channel.
      master59010|Aug 04, 2016 5:43:23 PM org.jenkinsci.plugins.ssegateway.sse.EventDispatcherFactory start WARNING: No CrumbIssuer on Jenkins instance. Some POSTs might not work.
      2016-08-04 17:43:23,071 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Connected to the Jenkins SSE Gateway.
      2016-08-04 17:43:23,074 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Subscribed to the "job" event channel.
      master59010|Aug 04, 2016 5:43:54 PM org.jenkinsci.plugins.ssegateway.sse.EventDispatcherFactory start WARNING: No CrumbIssuer on Jenkins instance. Some POSTs might not work.
      2016-08-04 17:43:54,076 [Thread-3] INFO   (InputStreamHandler.java:47) com.github.eirslett.maven.plugins.frontend.lib.DefaultGulpRunner - Connected to the Jenkins SSE Gateway.
      

      From https://ci.blueocean.io/job/Acceptance-Tests/144/consoleFull

      Although ... https://ci.blueocean.io/job/Acceptance-Tests/145/consoleText gets into what looks like the same blocked state, but without the previous SSE connect failure error. Hmmm ... I wonder is something causing Nightwatch to somehow not close out the previous test suite properly and then block not starting the next.

        Attachments

          Activity

          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Tom Fennelly
          Path:
          package.json
          src/main/js/api/sse.js
          src/main/js/globals.js
          http://jenkins-ci.org/commit/blueocean-acceptance-test/125172d45887d4d8075864a0e265e2004d0de1c4
          Log:
          [FIX JENKINS-37206] Updated to use newer SSE client for stability (#61)

          • Updated to use newer SSE client
          • Updated to use newer SSE client
          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Tom Fennelly Path: package.json src/main/js/api/sse.js src/main/js/globals.js http://jenkins-ci.org/commit/blueocean-acceptance-test/125172d45887d4d8075864a0e265e2004d0de1c4 Log: [FIX JENKINS-37206] Updated to use newer SSE client for stability (#61) Updated to use newer SSE client Updated to use newer SSE client

            People

            • Assignee:
              tfennelly Tom FENNELLY
              Reporter:
              tfennelly Tom FENNELLY
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: