Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61475

Infinite loop in StandardGraphLookupView

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Minor
    • Resolution: Unresolved
    • Component/s: workflow-api-plugin
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      About once a week, multiple threads handling HTTP requests matching the patterns /blue/rest/organizations/jenkins/pipelines/<folder>/pipelines/<job>/runs/<number>/ or /blue/organizations/jenkins/<folder>%2F<job>/detail/<job>/<number>/pipeline/ will get stuck in an infinite loop in StandardGraphLookupView.bruteForceScanForEnd.

      Each thread that get stuck ties a CPU core, and they tend to happen in batches of about 20-30 threads. This results in the Jenkins web UI appearing to be very slow to users. The resolution so far has been to use the JavaMelody monitoring plugin to kill the stuck threads one by one.

      I've attached stack traces from two example threads showing that they get stuck in the same place, as well as a snapshot of the CPU usage at the time. There was nothing supicious going on at the time, and I've been unable to manually reproduce the issue.

      I believe something like https://github.com/francoiscampbell/workflow-api-plugin/compare/master...graph-lookup-concurrent should address the issue, but I'm not sure how to verify that.

        Attachments

          Activity

          Hide
          fcampbell Francois Campbell added a comment -

          All plugins and their versions:

          ace-editor 1.1
          amazon-ecs 1.27-SNAPSHOT (private-ea869ac0-francoiscampbell)
          analysis-core 1.96
          android-lint 2.6
          ansicolor 0.6.3
          ant 1.11
          antisamy-markup-formatter 1.8
          apache-httpcomponents-client-4-api 4.5.10-2.0
          authentication-tokens 1.3
          authorize-project 1.3.0
          aws-credentials 1.28
          aws-java-sdk 1.11.723
          basic-branch-build-strategies 1.3.2
          blueocean 1.22.0
          blueocean-autofavorite 1.2.4
          blueocean-bitbucket-pipeline 1.22.0
          blueocean-commons 1.22.0
          blueocean-config 1.22.0
          blueocean-core-js 1.22.0
          blueocean-dashboard 1.22.0
          blueocean-display-url 2.3.1
          blueocean-events 1.22.0
          blueocean-git-pipeline 1.22.0
          blueocean-github-pipeline 1.22.0
          blueocean-i18n 1.22.0
          blueocean-jira 1.22.0
          blueocean-jwt 1.22.0
          blueocean-personalization 1.22.0
          blueocean-pipeline-api-impl 1.22.0
          blueocean-pipeline-editor 1.22.0
          blueocean-pipeline-scm-api 1.22.0
          blueocean-rest 1.22.0
          blueocean-rest-impl 1.22.0
          blueocean-web 1.22.0
          bouncycastle-api 2.18
          branch-api 2.5.5
          build-timeout 1.19.1
          cloudbees-bitbucket-branch-source 2.7.0
          cloudbees-folder 6.11.1
          cobertura 1.15
          code-coverage-api 1.1.3
          command-launcher 1.4
          conditional-buildstep 1.3.6
          credentials 2.3.3
          credentials-binding 1.21
          datadog 1.0.2
          datatheorem-mobile-app-security 2.0.1
          display-url-api 2.3.2
          docker-commons 1.16
          docker-workflow 1.22
          durable-task 1.33
          embeddable-build-status 2.0.3
          external-monitor-job 1.7
          fail-the-build-plugin 1.0
          favorite 2.3.2
          git 4.2.0
          git-client 3.2.0
          git-server 1.9
          github 1.29.5
          github-api 1.106
          github-branch-source 2.6.0
          handlebars 1.1.1
          handy-uri-templates-2-api 2.1.8-1.0
          htmlpublisher 1.22
          http_request 1.8.24
          jackson2-api 2.10.2
          jacoco 3.0.5
          javadoc 1.5
          jdk-tool 1.4
          jenkins-design-language 1.22.0
          jira 3.0.12
          job-dsl 1.76
          jquery-detached 1.2.1
          jsch 0.1.55.2
          junit 1.28
          ldap 1.21
          lockable-resources 2.7
          mailer 1.30
          mapdb-api 1.0.9.0
          matrix-auth 2.5
          matrix-project 1.14
          maven-plugin 3.4
          mercurial 2.8
          metrics 4.0.2.6
          metrics-datadog 1.0
          momentjs 1.1.1
          monitoring 1.82.0
          naginator 1.18
          opsgenie 1.8
          pam-auth 1.6
          parallel-test-executor 1.12
          parameterized-scheduler 0.8
          parameterized-trigger 2.36
          pipeline-aws 1.39
          pipeline-build-step 2.11
          pipeline-github-lib 1.0
          pipeline-graph-analysis 1.10
          pipeline-input-step 2.11
          pipeline-milestone-step 1.3.1
          pipeline-model-api 1.5.1
          pipeline-model-declarative-agent 1.1.1
          pipeline-model-definition 1.5.1
          pipeline-model-extensions 1.5.1
          pipeline-rest-api 2.13
          pipeline-stage-step 2.3
          pipeline-stage-tags-metadata 1.5.1
          pipeline-stage-view 2.13
          pipeline-timeline 1.0.3
          pipeline-utility-steps 2.5.0
          plain-credentials 1.7
          postbuildscript 2.9.1
          pubsub-light 1.13
          rebuild 1.31
          resource-disposer 0.14
          run-condition 1.2
          saml 1.1.5
          scm-api 2.6.3
          script-security 1.70
          simple-theme-plugin 0.6
          slack 2.36
          sse-gateway 1.22
          ssh-credentials 1.18.1
          ssh-slaves 1.31.1
          strict-crumb-issuer 2.1.0
          structs 1.20
          throttle-concurrents 2.0.2
          timestamper 1.11.1
          token-macro 2.11
          trilead-api 1.0.5
          variant 1.3
          windows-slaves 1.6
          workflow-aggregator 2.6
          workflow-api 2.40
          workflow-basic-steps 2.19
          workflow-cps 2.80
          workflow-cps-global-lib 2.15
          workflow-durable-task-step 2.35
          workflow-job 2.36
          workflow-multibranch 2.21
          workflow-scm-step 2.10
          workflow-step-api 2.22
          workflow-support 3.4
          ws-cleanup 0.38

          Show
          fcampbell Francois Campbell added a comment - All plugins and their versions: ace-editor 1.1 amazon-ecs 1.27-SNAPSHOT (private-ea869ac0-francoiscampbell) analysis-core 1.96 android-lint 2.6 ansicolor 0.6.3 ant 1.11 antisamy-markup-formatter 1.8 apache-httpcomponents-client-4-api 4.5.10-2.0 authentication-tokens 1.3 authorize-project 1.3.0 aws-credentials 1.28 aws-java-sdk 1.11.723 basic-branch-build-strategies 1.3.2 blueocean 1.22.0 blueocean-autofavorite 1.2.4 blueocean-bitbucket-pipeline 1.22.0 blueocean-commons 1.22.0 blueocean-config 1.22.0 blueocean-core-js 1.22.0 blueocean-dashboard 1.22.0 blueocean-display-url 2.3.1 blueocean-events 1.22.0 blueocean-git-pipeline 1.22.0 blueocean-github-pipeline 1.22.0 blueocean-i18n 1.22.0 blueocean-jira 1.22.0 blueocean-jwt 1.22.0 blueocean-personalization 1.22.0 blueocean-pipeline-api-impl 1.22.0 blueocean-pipeline-editor 1.22.0 blueocean-pipeline-scm-api 1.22.0 blueocean-rest 1.22.0 blueocean-rest-impl 1.22.0 blueocean-web 1.22.0 bouncycastle-api 2.18 branch-api 2.5.5 build-timeout 1.19.1 cloudbees-bitbucket-branch-source 2.7.0 cloudbees-folder 6.11.1 cobertura 1.15 code-coverage-api 1.1.3 command-launcher 1.4 conditional-buildstep 1.3.6 credentials 2.3.3 credentials-binding 1.21 datadog 1.0.2 datatheorem-mobile-app-security 2.0.1 display-url-api 2.3.2 docker-commons 1.16 docker-workflow 1.22 durable-task 1.33 embeddable-build-status 2.0.3 external-monitor-job 1.7 fail-the-build-plugin 1.0 favorite 2.3.2 git 4.2.0 git-client 3.2.0 git-server 1.9 github 1.29.5 github-api 1.106 github-branch-source 2.6.0 handlebars 1.1.1 handy-uri-templates-2-api 2.1.8-1.0 htmlpublisher 1.22 http_request 1.8.24 jackson2-api 2.10.2 jacoco 3.0.5 javadoc 1.5 jdk-tool 1.4 jenkins-design-language 1.22.0 jira 3.0.12 job-dsl 1.76 jquery-detached 1.2.1 jsch 0.1.55.2 junit 1.28 ldap 1.21 lockable-resources 2.7 mailer 1.30 mapdb-api 1.0.9.0 matrix-auth 2.5 matrix-project 1.14 maven-plugin 3.4 mercurial 2.8 metrics 4.0.2.6 metrics-datadog 1.0 momentjs 1.1.1 monitoring 1.82.0 naginator 1.18 opsgenie 1.8 pam-auth 1.6 parallel-test-executor 1.12 parameterized-scheduler 0.8 parameterized-trigger 2.36 pipeline-aws 1.39 pipeline-build-step 2.11 pipeline-github-lib 1.0 pipeline-graph-analysis 1.10 pipeline-input-step 2.11 pipeline-milestone-step 1.3.1 pipeline-model-api 1.5.1 pipeline-model-declarative-agent 1.1.1 pipeline-model-definition 1.5.1 pipeline-model-extensions 1.5.1 pipeline-rest-api 2.13 pipeline-stage-step 2.3 pipeline-stage-tags-metadata 1.5.1 pipeline-stage-view 2.13 pipeline-timeline 1.0.3 pipeline-utility-steps 2.5.0 plain-credentials 1.7 postbuildscript 2.9.1 pubsub-light 1.13 rebuild 1.31 resource-disposer 0.14 run-condition 1.2 saml 1.1.5 scm-api 2.6.3 script-security 1.70 simple-theme-plugin 0.6 slack 2.36 sse-gateway 1.22 ssh-credentials 1.18.1 ssh-slaves 1.31.1 strict-crumb-issuer 2.1.0 structs 1.20 throttle-concurrents 2.0.2 timestamper 1.11.1 token-macro 2.11 trilead-api 1.0.5 variant 1.3 windows-slaves 1.6 workflow-aggregator 2.6 workflow-api 2.40 workflow-basic-steps 2.19 workflow-cps 2.80 workflow-cps-global-lib 2.15 workflow-durable-task-step 2.35 workflow-job 2.36 workflow-multibranch 2.21 workflow-scm-step 2.10 workflow-step-api 2.22 workflow-support 3.4 ws-cleanup 0.38
          Hide
          jjathman Joe Athman added a comment -

          I believe we are running in to the same issue. We just started using version 2.204.6 and this began.

          Show
          jjathman Joe Athman added a comment - I believe we are running in to the same issue. We just started using version 2.204.6 and this began.

            People

            • Assignee:
              Unassigned
              Reporter:
              fcampbell Francois Campbell
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: