Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31487

Job status not updated in web UI during build

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Environment:
    • Similar Issues:

      Description

      Job status is not updated in the web browser during build. The progress bar appears and the dot starts blinking, but they never move - even when the job is finished. Reloading the page shows proper status (if job is completed) or proper progress (if it is still building).

      Problem observed with latest Firefox and Chrome (other browsers not checked). Inspecting AJAX calls in browser's console reveals that AJAX requests are sent each few seconds and all get 200 OK, but the response body is empty. (On a properly working system the response body is not empty while the job is building).

      jenkins.log shows no error or exception - in fact even at log level 7, it shows nothing about the AJAX call. It just denotes at INFO level that a job is starting, then that it completes.

        Attachments

          Activity

          Hide
          daften Dieter Blomme added a comment -

          We also observe this behaviour on our instance since the latest jenkins update we performed.

          Show
          daften Dieter Blomme added a comment - We also observe this behaviour on our instance since the latest jenkins update we performed.
          Hide
          danielbeck Daniel Beck added a comment -

          Which progress bar is this? The global executors list, or the builds history widget of a job, or the progress bar shown to the top right when viewing an in progress console output?

          Show
          danielbeck Daniel Beck added a comment - Which progress bar is this? The global executors list, or the builds history widget of a job, or the progress bar shown to the top right when viewing an in progress console output?
          Hide
          daften Dieter Blomme added a comment -

          Build history widget of a job definitely.

          Show
          daften Dieter Blomme added a comment - Build history widget of a job definitely.
          Hide
          alex_shepherd Alex Shepherd added a comment -

          Confirming this still occurs on Jenkins ver. 1.644 as shown in the attached image - I've included the AJAX POST request in the attached screenshot to help explain.

          Show
          alex_shepherd Alex Shepherd added a comment - Confirming this still occurs on Jenkins ver. 1.644 as shown in the attached image - I've included the AJAX POST request in the attached screenshot to help explain.
          Hide
          olteanra Rares Oltean added a comment -

          Hi guys! Any update on this? It still reproduces in 1.655
          No progress bars or build pipeline cards are updated in my Jenkins instance. It's very bothersome!

          Show
          olteanra Rares Oltean added a comment - Hi guys! Any update on this? It still reproduces in 1.655 No progress bars or build pipeline cards are updated in my Jenkins instance. It's very bothersome!
          Hide
          joshuaayson Joshua Ayson added a comment -

          We are seeing the same issue as well with LTS 1.651.1.

          Show
          joshuaayson Joshua Ayson added a comment - We are seeing the same issue as well with LTS 1.651.1.
          Hide
          danielbeck Daniel Beck added a comment -

          Is anyone affected who doesn't use custom CSS/JS to style Jenkins?

          Show
          danielbeck Daniel Beck added a comment - Is anyone affected who doesn't use custom CSS/JS to style Jenkins?
          Hide
          feuchtcc_cn Calla Feucht added a comment -

          Daniel Beck - Yep – no custom CSS/JS here, and we see this issue with v1.651.2.

          Show
          feuchtcc_cn Calla Feucht added a comment - Daniel Beck - Yep – no custom CSS/JS here, and we see this issue with v1.651.2.
          Hide
          danielbeck Daniel Beck added a comment -

          Anyone remember what the first version of Jenkins was to have this problem?

          Show
          danielbeck Daniel Beck added a comment - Anyone remember what the first version of Jenkins was to have this problem?
          Hide
          feuchtcc_cn Calla Feucht added a comment -

          Hmm I don't know where/when it started, but I do know that it was not present in v1.596.

          Show
          feuchtcc_cn Calla Feucht added a comment - Hmm I don't know where/when it started, but I do know that it was not present in v1.596.
          Hide
          petere Peter Eisentraut added a comment -

          No custom CSS/JS here.

          Show
          petere Peter Eisentraut added a comment - No custom CSS/JS here.
          Hide
          feuchtcc_cn Calla Feucht added a comment -

          Daniel Beck - Any further luck with this?

          Show
          feuchtcc_cn Calla Feucht added a comment - Daniel Beck - Any further luck with this?
          Hide
          feuchtcc_cn Calla Feucht added a comment -

          As a note, I'm realizing that it's more than just the progress bar that isn't updating. I've also noticed the following elements failing to update:

          • status of upstream projects (solid ball versus blinking ball, color, etc.)
          • initial run in jobs' build history (when clicking "Build Now" on the job overview page)
          Show
          feuchtcc_cn Calla Feucht added a comment - As a note, I'm realizing that it's more than just the progress bar that isn't updating. I've also noticed the following elements failing to update: status of upstream projects (solid ball versus blinking ball, color, etc.) initial run in jobs' build history (when clicking "Build Now" on the job overview page)
          Hide
          feuchtcc_cn Calla Feucht added a comment -

          Another behavior I've noticed: when on a job's overview page while a build is in progress, if that build finishes and another starts, both builds will appear to have active progress bars even though only the latter is running.

          Show
          feuchtcc_cn Calla Feucht added a comment - Another behavior I've noticed: when on a job's overview page while a build is in progress, if that build finishes and another starts, both builds will appear to have active progress bars even though only the latter is running.
          Hide
          allan_burdajewicz Allan BURDAJEWICZ added a comment -

          I did troubleshoot this. I can easily reproduce with Jenkins 1.651 and Jenkins 2.7.1 too. Here is the simplest scenario:

          • Create a Freestyle Job
          • Add a Shell task "sleep 10" so that you can notice the behavior of the progress bar
          • Build the job

          You should see the build job progress bar refreshing and switching to complete

          • Configure the Freestyle Job
          • Check the "This build is parameterized" and add a String Parameter
          • Build the job

          The build start and the progress bar hangs like previous screenshots shows. So seems to be related to parameterized builds.

          Show
          allan_burdajewicz Allan BURDAJEWICZ added a comment - I did troubleshoot this. I can easily reproduce with Jenkins 1.651 and Jenkins 2.7.1 too. Here is the simplest scenario: Create a Freestyle Job Add a Shell task "sleep 10" so that you can notice the behavior of the progress bar Build the job You should see the build job progress bar refreshing and switching to complete Configure the Freestyle Job Check the "This build is parameterized" and add a String Parameter Build the job The build start and the progress bar hangs like previous screenshots shows. So seems to be related to parameterized builds .
          Hide
          danielbeck Daniel Beck added a comment -

          Allan BURDAJEWICZ Actually, I was able to observe this in 2.14 for those progress bars that have been present when the page was loaded. Those that appeared later were correctly progressed. That also means that clicking (e.g.) "Status" to load the job index page will effectively halt progress of the build just triggered clicking "Build Now".

          I suspect that the page redirect due to the parameters form resulted in your observation, but it has nothing to do with parameterized builds.

          Show
          danielbeck Daniel Beck added a comment - Allan BURDAJEWICZ Actually, I was able to observe this in 2.14 for those progress bars that have been present when the page was loaded. Those that appeared later were correctly progressed. That also means that clicking (e.g.) "Status" to load the job index page will effectively halt progress of the build just triggered clicking "Build Now". I suspect that the page redirect due to the parameters form resulted in your observation, but it has nothing to do with parameterized builds.
          Hide
          danielbeck Daniel Beck added a comment -

          Anyone want to test 1.632 and 1.633 to determine whether this issue was introduced in the latter release, with build history pagination?

          Show
          danielbeck Daniel Beck added a comment - Anyone want to test 1.632 and 1.633 to determine whether this issue was introduced in the latter release, with build history pagination?
          Hide
          gerardck Gerard CK added a comment -

          Hi Daniel,
          The last version that has a working build progress bar is 1.625.2
          The progress bar is broken in every versions after that.
          Cheers

          Show
          gerardck Gerard CK added a comment - Hi Daniel, The last version that has a working build progress bar is 1.625.2 The progress bar is broken in every versions after that. Cheers
          Hide
          danielbeck Daniel Beck added a comment -

          Gerard CK You're saying it's broken in 1.625.3 as well?

          Show
          danielbeck Daniel Beck added a comment - Gerard CK You're saying it's broken in 1.625.3 as well?
          Hide
          gerardck Gerard CK added a comment -

          Yes this is correct.

          Show
          gerardck Gerard CK added a comment - Yes this is correct.
          Hide
          danielbeck Daniel Beck added a comment -

          Gerard CK Weird. 1.625.3 is only a security update – while we've had some issues recently with security fixes that broke things, I don't see a related change there.

          (For reference: https://github.com/jenkinsci/jenkins/compare/jenkins-1.625.2...jenkins-1.625.3 and https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2015-12-09)

          Did you make sure that a timing/page load issue with loading the builds widget as I mention in my previous comment cannot be responsible?

          Show
          danielbeck Daniel Beck added a comment - Gerard CK Weird. 1.625.3 is only a security update – while we've had some issues recently with security fixes that broke things, I don't see a related change there. (For reference: https://github.com/jenkinsci/jenkins/compare/jenkins-1.625.2...jenkins-1.625.3 and https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2015-12-09 ) Did you make sure that a timing/page load issue with loading the builds widget as I mention in my previous comment cannot be responsible?
          Hide
          danielbeck Daniel Beck added a comment -

          I confirm it's as described by Gerard CK, this issue was introduced between 1.625.2 and 1.625.3.

          Show
          danielbeck Daniel Beck added a comment - I confirm it's as described by Gerard CK , this issue was introduced between 1.625.2 and 1.625.3.
          Hide
          danielbeck Daniel Beck added a comment -

          First culprit in old releases was https://github.com/jenkinsci/jenkins/pull/1856 – but as comments indicate, it's no longer a problem in current releases.

          The difference is, that change completely disabled AJAXing, while the problem here is that requests are sent, but responses are empty.

          Back to bisecting from the start…

          Show
          danielbeck Daniel Beck added a comment - First culprit in old releases was https://github.com/jenkinsci/jenkins/pull/1856 – but as comments indicate, it's no longer a problem in current releases. The difference is, that change completely disabled AJAXing, while the problem here is that requests are sent, but responses are empty. Back to bisecting from the start…
          Hide
          danielbeck Daniel Beck added a comment -

          In regular releases, https://github.com/jenkinsci/jenkins/commit/078d3c7c4b5ffc5eae2f8c300a681bc340dfec60 is the first broken commit via git bisect, ignoring the aforementioned issue with AJAX requests not getting sent at all. So in weeklies, 1.633 should be the first broken one.

          Not sure what's going on with 1.625.3, but 078d3c7c seems to be the real culprit.

          Show
          danielbeck Daniel Beck added a comment - In regular releases, https://github.com/jenkinsci/jenkins/commit/078d3c7c4b5ffc5eae2f8c300a681bc340dfec60 is the first broken commit via git bisect , ignoring the aforementioned issue with AJAX requests not getting sent at all. So in weeklies, 1.633 should be the first broken one. Not sure what's going on with 1.625.3, but 078d3c7c seems to be the real culprit.
          Hide
          danielbeck Daniel Beck added a comment -

          Still occurs in 2.17 (after fix for JENKINS-31791).

          Show
          danielbeck Daniel Beck added a comment - Still occurs in 2.17 (after fix for JENKINS-31791 ).
          Hide
          danielbeck Daniel Beck added a comment -

          Appears to be caused by the wrong offset sent for nextBuildNumberToFetch – the client says he's just not interested in build that already exist when the initial page was loaded.

          Noticed another problem in the process though, if no build is running when the page loads, there's never going to be AJAX updates. Not sure whether caused by my WIP or in general, but this'll take some more time.

          Show
          danielbeck Daniel Beck added a comment - Appears to be caused by the wrong offset sent for nextBuildNumberToFetch – the client says he's just not interested in build that already exist when the initial page was loaded. Noticed another problem in the process though, if no build is running when the page loads, there's never going to be AJAX updates. Not sure whether caused by my WIP or in general, but this'll take some more time.
          Hide
          matthewreiter Matthew Reiter added a comment - - edited

          I've attached a patch that works for me... I'm not sure if it's the best approach but might be a good place to start. The patch is relative to version 2.20. jobStatus.patch

          Show
          matthewreiter Matthew Reiter added a comment - - edited I've attached a patch that works for me... I'm not sure if it's the best approach but might be a good place to start. The patch is relative to version 2.20. jobStatus.patch
          Hide
          danielbeck Daniel Beck added a comment -

          Matthew Reiter Looks a lot like what I did locally. Same issue as mine though AFAICT, if there was no build running when the page loaded, the list is never updated. Can you confirm the same behavior?

          Show
          danielbeck Daniel Beck added a comment - Matthew Reiter Looks a lot like what I did locally. Same issue as mine though AFAICT, if there was no build running when the page loaded, the list is never updated. Can you confirm the same behavior?
          Hide
          matthewreiter Matthew Reiter added a comment - - edited

          Daniel Beck Yes, I am seeing the same behavior. I opened two browser windows pointing at the job page, kicked off the build on one window and saw that the other window didn't see the build until I refreshed the page.

          Show
          matthewreiter Matthew Reiter added a comment - - edited Daniel Beck Yes, I am seeing the same behavior. I opened two browser windows pointing at the job page, kicked off the build on one window and saw that the other window didn't see the build until I refreshed the page.
          Hide
          matthewreiter Matthew Reiter added a comment -

          I've updated the patch to include a fix for the list not updating if no builds were running (nextBuildNumberToFetch was null, resulting in a syntax error in the javascript code) and to make it work with concurrent builds if they finish out of order. jobStatus2.patch

          Show
          matthewreiter Matthew Reiter added a comment - I've updated the patch to include a fix for the list not updating if no builds were running (nextBuildNumberToFetch was null, resulting in a syntax error in the javascript code) and to make it work with concurrent builds if they finish out of order. jobStatus2.patch
          Hide
          danielbeck Daniel Beck added a comment -

          Matthew Reiter Could you submit this as a PR on GitHub?

          Show
          danielbeck Daniel Beck added a comment - Matthew Reiter Could you submit this as a PR on GitHub?
          Hide
          matthewreiter Matthew Reiter added a comment -

          I've created the pull request: https://github.com/jenkinsci/jenkins/pull/2542

          Show
          matthewreiter Matthew Reiter added a comment - I've created the pull request: https://github.com/jenkinsci/jenkins/pull/2542
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Matthew Reiter
          Path:
          core/src/main/java/hudson/widgets/BuildHistoryWidget.java
          core/src/main/java/hudson/widgets/HistoryWidget.java
          core/src/main/resources/hudson/widgets/HistoryWidget/index.jelly
          http://jenkins-ci.org/commit/jenkins/0268b988d5c88cd29be12ed25e95d5bc448c2840
          Log:
          [FIXED JENKINS-31487] (#2542)

          There were two issues preventing the build history from updating properly:
          1) The next build number being fetched wasn't taking into account running builds, so any builds already running when the page is refreshed would be ignored. The fix was to use nextBuildNumberToFetch if it is available (which is the case if there are running builds) and to fall back to the next build otherwise.
          2) The first transient build key (used to clear out builds from the history that are being updated) wasn't being set when the page first loads. This was fixed by making getHistoryPageFilter calculate the value so that it happens in all cases rather than just during the ajax call.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Matthew Reiter Path: core/src/main/java/hudson/widgets/BuildHistoryWidget.java core/src/main/java/hudson/widgets/HistoryWidget.java core/src/main/resources/hudson/widgets/HistoryWidget/index.jelly http://jenkins-ci.org/commit/jenkins/0268b988d5c88cd29be12ed25e95d5bc448c2840 Log: [FIXED JENKINS-31487] (#2542) There were two issues preventing the build history from updating properly: 1) The next build number being fetched wasn't taking into account running builds, so any builds already running when the page is refreshed would be ignored. The fix was to use nextBuildNumberToFetch if it is available (which is the case if there are running builds) and to fall back to the next build otherwise. 2) The first transient build key (used to clear out builds from the history that are being updated) wasn't being set when the page first loads. This was fixed by making getHistoryPageFilter calculate the value so that it happens in all cases rather than just during the ajax call.
          Hide
          kmandeville Kevin Mandeville added a comment -

          Is there any chance of getting this fix into the 1.x version?

          Show
          kmandeville Kevin Mandeville added a comment - Is there any chance of getting this fix into the 1.x version?
          Hide
          danielbeck Daniel Beck added a comment -

          Is there any chance of getting this fix into the 1.x version?

          We are no longer maintaining any releases of the 1.x line. Regular weekly releases are at 2.25, LTS releases are at 2.19.x.

          Show
          danielbeck Daniel Beck added a comment - Is there any chance of getting this fix into the 1.x version? We are no longer maintaining any releases of the 1.x line. Regular weekly releases are at 2.25, LTS releases are at 2.19.x.

            People

            • Assignee:
              danielbeck Daniel Beck
              Reporter:
              mistoffelees Assen Totin
            • Votes:
              14 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: