Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37018

Blue Ocean dashboard page never finishes loading

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Component/s: blueocean-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.15, blue ocean 1.0-alpha-4, matrix project 1.7.1, test stability 1.0, config history 2.15
    • Similar Issues:
    • Epic Link:
    • Sprint:
      1.0-pre-beta-1

      Description

      When I try opening the Blue Ocean dashboard view, it never finishes loading, and instead says "No pipelines found."

      If I open the Chrome developer tools, I can see it making a request that never finishes:
      https://<ROOTURL>/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject

      If I use curl to retrieve that page, it takes 15 minutes to load, and produces a 53-megabyte json file that's truncated at the end.

      The most notable large sections, from what I can tell (with job and test class names changed for privacy):

      {"_class":"io.jenkins.blueocean.service.embedded.rest.ActionProxiesImpl","_links":{"self":{"_class":"io.jenkins.blueocean.rest.hal.Link","href":"/blue/rest/organizations/jenkins/pipelines/REDACTED-JOB-NAME/jobConfigHistory/"}},"_class":"hudson.plugins.jobConfigHistory.JobConfigHistoryProjectAction","jobConfigHistory":[

      {"currentName":"","date":"2016-07-27_01-38-02","hasConfig":true,"job":"REDACTED-JOB-NAME","oldName":"","operation":"Changed","user":"SYSTEM","userID":"SYSTEM"}

      ,

      (I'd say I see two-digit numbers of these "currentName" entries)

      {"_class":"io.jenkins.blueocean.service.embedded.rest.ActionProxiesImpl","_links":{"self":"_class":"io.jenkins.blueocean.rest.hal.Link","href":"/blue/rest/organizations/jenkins/pipelines/FOLDER_NAME_HERE/pipelines/JOB_NAME_HERE/runs/4/testReport/"}},"_class":"hudson.tasks.test.MatrixTestResult","failCount":11,"skipCount":21,"totalCount":3644,"childReports":[{"child":

      {"_class":"hudson.matrix.MatrixRun","number":4,"url":"https://<BASEURL_HERE>/job/FOLDER_HERE/job/MATRIXJOB_NAME_HERE/AXIS_NAME_HERE/4/"}

      ,"result":{"_class":"hudson.tasks.junit.TestResult","testActions":[],"duration":6332.9766,"empty":false,"failCount":3,"passCou{nt":903,"skipCount":5,"suites":[{"cases":[{"testActions":[

      {"_class":"de.esailors.jenkins.teststability.StabilityTestAction"}

      ],"age":0,"className":"CLASS NAME HERE","duration":2.051724,"failedSince":0,"name":"TEST NAME HERE","skipped":false,"status":"PASSED"},

      (Thousands of these StabilityTestAction entries)

        Attachments

          Issue Links

            Activity

            Hide
            danagoyette Dana Goyette added a comment - - edited

            To more clearly indicate the size of the file: I used sed to replace every instance of "},{" with "},\n{", and the file ended up with 202958 lines.

            Show
            danagoyette Dana Goyette added a comment - - edited To more clearly indicate the size of the file: I used sed to replace every instance of "},{" with "},\n{" , and the file ended up with 202958 lines.
            Hide
            jamesdumay James Dumay added a comment - - edited

            Hi Dana,

            Thanks for filing this ticket. I suspect one of the reasons you are seeing this is because we have not implemented pagination on the dashboard screen (JENKINS-35826) which would cause a massive JSON file being loaded.

            I'd love to get a better description on the size of your instance:

            1. How many developers use your Jenkins instance?
            2. Are there multiple teams?
            3. How do you organize your jobs? In Folders? If so, do you use folders to organize by project/team/etc?
            4. How many jobs are on your system?
            5. What job types do you make use of?

            Thanks,
            James

            Show
            jamesdumay James Dumay added a comment - - edited Hi Dana, Thanks for filing this ticket. I suspect one of the reasons you are seeing this is because we have not implemented pagination on the dashboard screen ( JENKINS-35826 ) which would cause a massive JSON file being loaded. I'd love to get a better description on the size of your instance: How many developers use your Jenkins instance? Are there multiple teams? How do you organize your jobs? In Folders? If so, do you use folders to organize by project/team/etc? How many jobs are on your system? What job types do you make use of? Thanks, James
            Hide
            jamesdumay James Dumay added a comment -

            Vivek Pandey I suspect looking at that data that Dana may have a lot of test cases due to the use of matrix jobs. Could that blow the size of the dashboard call out considerably?

            Show
            jamesdumay James Dumay added a comment - Vivek Pandey I suspect looking at that data that Dana may have a lot of test cases due to the use of matrix jobs. Could that blow the size of the dashboard call out considerably?
            Hide
            danagoyette Dana Goyette added a comment - - edited

            From memory and from a quick check of my Jenkins instance, I'd say I have the following:

            • 3 products under test.
            • Between 6 and 10 platforms per product (platform is the matrix axis).
            • 2 actively-tested branches per product.
            • 2 types of test suite jobs per branch (one type with 100 or so tests, one with 1000-4000 tests depending on product).
            • Each suite job runs 3 times per week.
            • Builds are kept for 28 days.
            • Job config history plugin is installed.
            • Jobs are regenerated by DSL, and job-config-history records a change even if nothing really changed.
            • I tried disabling the plugin, and it didn't change much in Blue Ocean.
            • Test reliability plugin is installed, and has been used in the past, but right now the jobs aren't set to use it.
            Show
            danagoyette Dana Goyette added a comment - - edited From memory and from a quick check of my Jenkins instance, I'd say I have the following: 3 products under test. Between 6 and 10 platforms per product (platform is the matrix axis). 2 actively-tested branches per product. 2 types of test suite jobs per branch (one type with 100 or so tests, one with 1000-4000 tests depending on product). Each suite job runs 3 times per week. Builds are kept for 28 days. Job config history plugin is installed. Jobs are regenerated by DSL, and job-config-history records a change even if nothing really changed. I tried disabling the plugin, and it didn't change much in Blue Ocean. Test reliability plugin is installed, and has been used in the past, but right now the jobs aren't set to use it.
            Hide
            jamesdumay James Dumay added a comment -

            Thanks thats great context for us

            Show
            jamesdumay James Dumay added a comment - Thanks thats great context for us
            Hide
            vivek Vivek Pandey added a comment -

            James Dumay Test data is not served as part of response in latest run of pipeline. However there is some action data with test description and high level stats is present.

            We can also consider option to not inline actions, instead only provide links to it, but then UI needs to make separate call to get action object. I think pagination should mitigate this behavior so maybe thats all is needed.

            There is a bigger issue. Backend API doesn't handle matrix project correctly. In fact matrix is not modeled in backend API. Each combination of matrix axis is instance of particular run. For example if you have two axis, OS and JDK, so a run/build will be repeated for each combination of OS and JDK combination. I have opened issue https://issues.jenkins-ci.org/browse/JENKINS-37104. It might have impact on UI depending on how we want to expose matrix projects.

            Show
            vivek Vivek Pandey added a comment - James Dumay Test data is not served as part of response in latest run of pipeline. However there is some action data with test description and high level stats is present. We can also consider option to not inline actions, instead only provide links to it, but then UI needs to make separate call to get action object. I think pagination should mitigate this behavior so maybe thats all is needed. There is a bigger issue. Backend API doesn't handle matrix project correctly. In fact matrix is not modeled in backend API. Each combination of matrix axis is instance of particular run. For example if you have two axis, OS and JDK, so a run/build will be repeated for each combination of OS and JDK combination. I have opened issue https://issues.jenkins-ci.org/browse/JENKINS-37104 . It might have impact on UI depending on how we want to expose matrix projects.
            Hide
            vivek Vivek Pandey added a comment -

            Dana Goyette Based on configuration provided, I expected about 120 (3*10*2*2) pipelines/jobs and each inlining its own run and its actions. It is worth while to investigate which is the culprit inning these 1000 of actions per build. Would it be possible to provide output response in full?

            Show
            vivek Vivek Pandey added a comment - Dana Goyette Based on configuration provided, I expected about 120 (3*10*2*2) pipelines/jobs and each inlining its own run and its actions. It is worth while to investigate which is the culprit inning these 1000 of actions per build. Would it be possible to provide output response in full?
            Hide
            danagoyette Dana Goyette added a comment -

            If you can tell me a way to get the full response, I can run it through 'sed' to replace any sensitive stuff with placeholder values, such as "product1-branch1". (When I tried using curl, the output was truncated.)

            Show
            danagoyette Dana Goyette added a comment - If you can tell me a way to get the full response, I can run it through 'sed' to replace any sensitive stuff with placeholder values, such as "product1-branch1". (When I tried using curl, the output was truncated.)
            Hide
            vivek Vivek Pandey added a comment -

            Hi Dana Goyette,

            http://localhost:8080/jenkins/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject&start=0&limit=100 should give you 100 results. If you want to include less you can, basically I want to look at actions that are getting serialized in there so send them across after scrubbing. thanks.

            Show
            vivek Vivek Pandey added a comment - Hi Dana Goyette , http://localhost:8080/jenkins/blue/rest/search/?q=type:pipeline;excludedFromFlattening:jenkins.branch.MultiBranchProject&start=0&limit=100 should give you 100 results. If you want to include less you can, basically I want to look at actions that are getting serialized in there so send them across after scrubbing. thanks.
            Hide
            danagoyette Dana Goyette added a comment -

            I've attached a tar.gz of the json (after running through 'jq' and sanitizing a bunch of stuff). Hopefully that's helpful.

            Show
            danagoyette Dana Goyette added a comment - I've attached a tar.gz of the json (after running through 'jq' and sanitizing a bunch of stuff). Hopefully that's helpful.
            Hide
            vivek Vivek Pandey added a comment -

            Thanks Dana Goyette. I will have an update after analyzing the json response.

            Show
            vivek Vivek Pandey added a comment - Thanks Dana Goyette . I will have an update after analyzing the json response.
            Hide
            vivek Vivek Pandey added a comment - - edited

            We are going to address it by

            • list matrix projects in blueocean dashboard (JENKINS-37104)
            • clicking on it redirects user to classic matrix project page (JENKINS-37427)
            Show
            vivek Vivek Pandey added a comment - - edited We are going to address it by list matrix projects in blueocean dashboard ( JENKINS-37104 ) clicking on it redirects user to classic matrix project page ( JENKINS-37427 )
            Hide
            vivek Vivek Pandey added a comment -

            Michael NealeJames Dumay To resolve this ticket we need JENKINS-37104, JENKINS-37427 and pagination ticket to be completed. Can you link pagination JIRA issue that Keith is working on to this ticket?

            Show
            vivek Vivek Pandey added a comment - Michael Neale James Dumay To resolve this ticket we need JENKINS-37104 , JENKINS-37427 and pagination ticket to be completed. Can you link pagination JIRA issue that Keith is working on to this ticket?
            Hide
            jamesdumay James Dumay added a comment -

            Vivek Pandey already linked. I think we should close this ticket against it.

            Show
            jamesdumay James Dumay added a comment - Vivek Pandey already linked. I think we should close this ticket against it.
            Hide
            trejkaz trejkaz added a comment -

            I'm seeing the progress bar taking forever to load as well, and the "original" ticket this was marked against doesn't sound like it was talking about the same problem.

            Time to reopen?

             

            Show
            trejkaz trejkaz added a comment - I'm seeing the progress bar taking forever to load as well, and the "original" ticket this was marked against doesn't sound like it was talking about the same problem. Time to reopen?  
            Hide
            jamesdumay James Dumay added a comment -

            trejkaz can you please open a new ticket with both a HAR file and a support bundle attached? Additional screenshots would help too.

            Show
            jamesdumay James Dumay added a comment - trejkaz can you please open a new ticket with both a HAR file and a support bundle attached? Additional screenshots would help too.

              People

              • Assignee:
                vivek Vivek Pandey
                Reporter:
                danagoyette Dana Goyette
              • Votes:
                1 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: