Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58085

BlueOcean UI stuck in "Waiting for run to start"

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Component/s: blueocean-plugin
    • Labels:
      None
    • Environment:
      Jenkins 2.180, BlueOcean 1.17.0
    • Similar Issues:
    • Released As:
      blue-ocean 1.19.0

      Description

      We recently upgraded BlueOcean from 1.16.0 to 1.17.0 and we started observing a weird behaviour in the BlueOcean pipeline UI.

      Frequently (not always) the pipeline UI stops updating the progress while the pipeline is running and the UI is stuck at "Waiting for run to start" (see attached screenshot). When it happens, it does not recover until the pipeline execution completes: once completed, the UI is correctly updated (all steps are green).

      We've also noticed that - when happens - the underlying requests sent by the browser to the endpoint https://jenkins.DOMAIN/blue/rest/organizations/jenkins/pipelines/PROJECT/branches/master/runs/ID/nodes/ID/steps/ always return an empty array "[]" instead of the expected array of steps. On the contrary, during the execution of the pipeline, if we look at the "Console Output" (old Jenkins UI) we can correctly see the progress of the pipeline even when the BlueOcean UI is stuck at "Waiting for run to start".

      This issue looks disappear if we rollback all BlueOcean plugins from 1.17.0 to 1.16.0.

        Attachments

        1. jenkins_build1.mov
          970 kB
          Ian Wallace-Hoyt
        2. jenkins_build1.png
          89 kB
          Ian Wallace-Hoyt
        3. jenkins_build2.mov
          1.07 MB
          Ian Wallace-Hoyt
        4. jenkins_build2.png
          96 kB
          Ian Wallace-Hoyt
        5. screenshot_2019-06-18_at_14.52.11.png
          116 kB
          Marco Pracucci
        6. Screenshot 2019-10-17 at 10.08.17.png
          16 kB
          Bas Broere

          Issue Links

            Activity

            Hide
            ian_ookla Ian Wallace-Hoyt added a comment -

            The issue repros nearly 100% of the time using this pipeline when a "buddy build" runs.

            Key details:

            • Jenkins 2.204.1
            • Blue Ocean 1.21.0
            • Amazon EC2 plugin 1.47
            • ec2-android-toolchain: this is an ec2 instance provisioned using the ec2 plugin
            • Issue repros when a PR is opened, this causing
              • "Release: build" stage is skipped
              • Stages "Buddy: build apks" and "Buddy: jvm verify" execute in parallel
            • "Buddy: build apks" stage always shows "Queued: Waiting for run to start" in BlueOcean, even though it runs and completes.
            • When "Buddy: build apks" stage completes successfully, BlueOcean updates to show all the correct stage information.
            • "Buddy: jvm verify" stage shows progress correctly in BlueOcean
            def buildAgent = 'ec2-android-toolchain'
            
            pipeline {
                agent none
            
                options{
                    timestamps()
                    timeout(time: 2, unit: 'HOURS')
                    parallelsAlwaysFailFast()
                }
            
                parameters {
                    string(
                            defaultValue: "",
                            description: 'If set, reruns the device test stages only,using artifact from specified build id, skipping all other stages',
                            name: "RERUN_DEVICETEST",
                    )
            
                    booleanParam(
                            defaultValue: false,
                            description: 'Publish release build to crashlytics beta',
                            name: "PUBLISH_BETA",
                    )
                }
            
                environment {
                    //Disable gradle daemon
                    GRADLE_OPTS = "-Dorg.gradle.daemon=false"
                }
            
                stages {
                    stage('Jenkins') {
                        parallel {
                            stage('Release: build') {
                                when {
                                    beforeAgent true
                                    equals expected: PipelineMode.RELEASE, actual: getPipelineMode()
                                    expression { !isDeviceTestOnly() }
                                }
            
                                agent {
                                    label buildAgent
                                }
            
                                environment {
                                    // Number of build numbers to allocate when building the project
                                    // (release builds only)
                                    O2_BUILD_VERSION_BLOCK_SIZE = "64"
            
                                    ORG_GRADLE_PROJECT_publishToBeta = "${params.PUBLISH_BETA ? '1' : '0'}"
                                }
            
                                steps {
                                    installJenkinsSshKey()
                                    checkoutRepo(steps)
                                    sh "./other/jenkins/job_release_build.sh"
                                }
            
                                post {
                                    always {
                                        addArtifacts()
                                        processJvmJunitResults()
                                    }
                                }
                            }
            
                            stage('Buddy: build apks') {
                                when {
                                    beforeAgent true
                                    equals expected: PipelineMode.BUDDY, actual: getPipelineMode()
                                    expression { !isDeviceTestOnly() }
                                }
            
                                agent {
                                    label buildAgent
                                }
            
                                steps {
                                    installJenkinsSshKey()
                                    checkoutRepo(steps)
                                    sh '''#!/bin/bash
            
                                        . "other/jenkins/build_lib.sh"
            
                                        "$ST_ROOT/other/jenkins/initialize_build.sh"
            
                                        execGradle --stacktrace jenkinsBuddyStageBuild
            
                                        "$ST_ROOT/other/jenkins/job_build_upload_sharedartifacts.sh"
                                    '''
                                }
            
                                post {
                                    always {
                                        addArtifacts()
                                    }
                                }
                            }
            
                            stage('Buddy: jvm verify') {
                                when {
                                    beforeAgent true
                                    equals expected: PipelineMode.BUDDY, actual: getPipelineMode()
                                    expression { !isDeviceTestOnly() }
                                }
            
                                agent {
                                    label buildAgent
                                }
            
                                steps {
                                    installJenkinsSshKey()
                                    checkoutRepo(steps)
                                    sh '''#!/bin/bash
            
                                        . "other/jenkins/build_lib.sh"
            
                                        "$ST_ROOT/other/jenkins/initialize_build.sh"
            
                                        execGradle --stacktrace jenkinsBuddyStageVerifyAndTest
                                    '''
                                }
            
                                post {
                                    always {
                                        processJvmJunitResults()
                                    }
                                }
                            }
                        }
                    }
                }
            
                post {
                    failure {
                        notifyUnhealthy()
                    }
            
                    fixed {
                        notifyFixed()
                    }
            
                    unstable {
                        notifyUnhealthy()
                    }
                }
            }
            
            def checkoutRepo(steps) {
                steps.sh '''#!/bin/bash
                git submodule init
                git submodule update
                '''
            }
            
            /**
             * This should only be used on non-ec2 agents. Roles are used to properly authorize
             * agents running in ev2
             *
             * @param cl
             * @return
             */
            @SuppressWarnings("GroovyInfiniteRecursion") // arg list is different and doesn't recurse
            def withAwsCredentials(Closure cl) {
                withAwsCredentials('**********', cl)
            }
            
            /**
             * Installs the jenkins ssh key managed by jenkins
             */
            def installJenkinsSshKey() {
                installSshKey('jenkins-ssh-key',
                        "${env.HOME}/.ssh/id_rsa")
            }
            
            /**
             * Installs an ssh managed by jenkins to the specified path
             *
             * @param credentialId
             * @param destPath
             */
            def installSshKey(String credentialId, String destPath) {
                withCredentials([sshUserPrivateKey(
                        keyFileVariable: 'THEKEY',
                        credentialsId: credentialId)]) {
            
                    sh """
                        cp "$THEKEY" "$destPath"
                        chmod 0600 "$destPath"
                    """
                }
            }
            
            enum PipelineMode {
                BUDDY, // Buddy mode for pipeline
                RELEASE, // Release mode for pipeline
                UNKNOWN // Unknown/unsupported mode. All stages skipped.
            }
            
            /**
             * Each jenkins pipeline execution runs in a single mode
             */
            def getPipelineMode() {
                if (env.CHANGE_ID != null) {
                    return PipelineMode.BUDDY
                } else if (isMainlineBranch()) {
                    return PipelineMode.RELEASE
                } else {
                    return PipelineMode.UNKNOWN
                }
            }
            
            /**
             * Returns true if any of the current branches are a mainline branch, otherwise false
             */
            def isMainlineBranch() {
                def mainlines = [ /^master$/, /^release\/.*$/, /^topic\/.*$/ ]
                return null != scm.branches.find { branch ->
                    mainlines.find { mainlineRegEx ->
                        return branch ==~ mainlineRegEx
                    }
                }
            }
            
            /**
             * Return true if only the device test stages should run
             * @return
             */
            def isDeviceTestOnly() {
                return !params.RERUN_DEVICETEST.isEmpty()
            }
            
            /**
             * Returns true if the device test stage(s) should run
             */
            def shouldRunDeviceTest() {
                return getPipelineMode() != PipelineMode.UNKNOWN
            }
            
            /**
             * Given a steps element, configures it to run the given shard indexes
             */
            def defineStepsForDeviceTest(steps, int totalShards, int...shardIndexes) {
                steps.with {
                    checkoutRepo(steps)
            
                    withAwsCredentials {
                        def url = getArtifactShareUrl("build", deviceTestInputBuildId())
                        sh "./other/jenkins/devicetest_ensure_ready.sh '$url'"
                    }
            
                    script {
                        // PR builds merge the PR HEAD into the target branch. In this case
                        // we just want to pass the PR branch to the manual job. If
                        // it isn't a PR build, the pass the commit.
                        def commitToBuild = env.CHANGE_BRANCH?.trim()
                        if (!commitToBuild) {
                            commitToBuild = env.GIT_COMMIT
                        }
            
                        build job: "/Manual Builds/speedtestnet-android/deviceTest",
                                parameters: [
                                        string(name: 'O2_INPUT_URL', value:
                                                getArtifactShareUrl("build", deviceTestInputBuildId())),
                                        string(name: 'O2_OUTPUT_URL', value:
                                                getArtifactShareUrl("deviceTest", deviceTestInputBuildId())),
                                        string(name: 'O2_COMMIT', value: commitToBuild),
                                        string(name: 'O2_BRANCH', value: env.GIT_BRANCH),
                                        string(name: 'O2_SHARD_COUNT', value: "${totalShards}"),
                                        string(name: 'O2_SHARD_INDEXES', value: shardIndexes.join(','))
                                ],
            
                                // Don't fail the stage if downstream job fails. We want to process
                                // test results from unstable builds in our pipeline
                                propagate: false
                    }
                }
            }
            
            /**
             * Build id to use as input for device test stage
             */
            def deviceTestInputBuildId() {
                return params.RERUN_DEVICETEST.isEmpty()
                        ? env.BUILD_ID
                        : params.RERUN_DEVICETEST
            }
            
            /**
             * Get the artifact share path for the given stage.
             * @param stage the stage with which the path is associated
             * @param buildId (optional) build id to use, defaults to current build id
             */
            def getArtifactShareUrl(String stage, String buildId = env.BUILD_ID) {
                return sh(returnStdout: true, script: "./other/jenkins/share_artifacts.sh "
                        + "--action getShareUrl "
                        + "--jobName ${env.JOB_NAME} --jobBuild ${buildId} --jobStage ${stage}"
                ).trim()
            }
            
            /**
             * Process junit results from the jvm tests
             */
            def processJvmJunitResults() {
                junit '**/build/test-results/**/*.xml'
            }
            /**
             * Add the archived apk's as artifacts
             */
            def addArtifacts() {
                archiveArtifacts artifacts:"Mobile4/build/outputs/apk/**/*.apk", fingerprint:true
            }
            
            def notifyUnhealthy() {
                if (getPipelineMode() != PipelineMode.RELEASE) {
                    return
                }
            
                slackSend(channel: '#team-android-dev',
                        color: 'danger',
                        message: "<${currentBuild.absoluteUrl}|${currentBuild.fullDisplayName}>"
                                + " :jenkins_angry:  is unhealthy.")
            }
            
            def notifyFixed() {
                if (getPipelineMode() != PipelineMode.RELEASE) {
                    return
                }
            
                slackSend(channel: '#team-android-dev',
                        color: 'good',
                        message: "<${currentBuild.absoluteUrl}|${currentBuild.fullDisplayName}>"
                                + ":jenkins: is healthy again.")
            }
            Show
            ian_ookla Ian Wallace-Hoyt added a comment - The issue repros nearly 100% of the time using this pipeline when a "buddy build" runs. Key details: Jenkins 2.204.1 Blue Ocean 1.21.0 Amazon EC2 plugin 1.47 ec2-android-toolchain: this is an ec2 instance provisioned using the ec2 plugin Issue repros when a PR is opened, this causing "Release: build" stage is skipped Stages "Buddy: build apks" and "Buddy: jvm verify" execute in parallel "Buddy: build apks" stage always shows "Queued: Waiting for run to start" in BlueOcean, even though it runs and completes. When "Buddy: build apks" stage completes successfully, BlueOcean updates to show all the correct stage information. "Buddy: jvm verify" stage shows progress correctly in BlueOcean def buildAgent = 'ec2-android-toolchain' pipeline { agent none options{ timestamps() timeout(time: 2, unit: 'HOURS' ) parallelsAlwaysFailFast() } parameters { string( defaultValue: "", description: 'If set, reruns the device test stages only,using artifact from specified build id, skipping all other stages' , name: "RERUN_DEVICETEST" , ) booleanParam( defaultValue: false , description: 'Publish release build to crashlytics beta' , name: "PUBLISH_BETA" , ) } environment { //Disable gradle daemon GRADLE_OPTS = "-Dorg.gradle.daemon= false " } stages { stage( 'Jenkins' ) { parallel { stage( 'Release: build' ) { when { beforeAgent true equals expected: PipelineMode.RELEASE, actual: getPipelineMode() expression { !isDeviceTestOnly() } } agent { label buildAgent } environment { // Number of build numbers to allocate when building the project // (release builds only) O2_BUILD_VERSION_BLOCK_SIZE = "64" ORG_GRADLE_PROJECT_publishToBeta = "${params.PUBLISH_BETA ? '1' : '0' }" } steps { installJenkinsSshKey() checkoutRepo(steps) sh "./other/jenkins/job_release_build.sh" } post { always { addArtifacts() processJvmJunitResults() } } } stage( 'Buddy: build apks' ) { when { beforeAgent true equals expected: PipelineMode.BUDDY, actual: getPipelineMode() expression { !isDeviceTestOnly() } } agent { label buildAgent } steps { installJenkinsSshKey() checkoutRepo(steps) sh '''#!/bin/bash . "other/jenkins/build_lib.sh" "$ST_ROOT/other/jenkins/initialize_build.sh" execGradle --stacktrace jenkinsBuddyStageBuild "$ST_ROOT/other/jenkins/job_build_upload_sharedartifacts.sh" ''' } post { always { addArtifacts() } } } stage( 'Buddy: jvm verify' ) { when { beforeAgent true equals expected: PipelineMode.BUDDY, actual: getPipelineMode() expression { !isDeviceTestOnly() } } agent { label buildAgent } steps { installJenkinsSshKey() checkoutRepo(steps) sh '''#!/bin/bash . "other/jenkins/build_lib.sh" "$ST_ROOT/other/jenkins/initialize_build.sh" execGradle --stacktrace jenkinsBuddyStageVerifyAndTest ''' } post { always { processJvmJunitResults() } } } } } } post { failure { notifyUnhealthy() } fixed { notifyFixed() } unstable { notifyUnhealthy() } } } def checkoutRepo(steps) { steps.sh '''#!/bin/bash git submodule init git submodule update ''' } /** * This should only be used on non-ec2 agents. Roles are used to properly authorize * agents running in ev2 * * @param cl * @ return */ @SuppressWarnings( "GroovyInfiniteRecursion" ) // arg list is different and doesn't recurse def withAwsCredentials(Closure cl) { withAwsCredentials( '**********' , cl) } /** * Installs the jenkins ssh key managed by jenkins */ def installJenkinsSshKey() { installSshKey( 'jenkins-ssh-key' , "${env.HOME}/.ssh/id_rsa" ) } /** * Installs an ssh managed by jenkins to the specified path * * @param credentialId * @param destPath */ def installSshKey( String credentialId, String destPath) { withCredentials([sshUserPrivateKey( keyFileVariable: 'THEKEY' , credentialsId: credentialId)]) { sh """ cp "$THEKEY" "$destPath" chmod 0600 "$destPath" """ } } enum PipelineMode { BUDDY, // Buddy mode for pipeline RELEASE, // Release mode for pipeline UNKNOWN // Unknown/unsupported mode. All stages skipped. } /** * Each jenkins pipeline execution runs in a single mode */ def getPipelineMode() { if (env.CHANGE_ID != null ) { return PipelineMode.BUDDY } else if (isMainlineBranch()) { return PipelineMode.RELEASE } else { return PipelineMode.UNKNOWN } } /** * Returns true if any of the current branches are a mainline branch, otherwise false */ def isMainlineBranch() { def mainlines = [ /^master$/, /^release\/.*$/, /^topic\/.*$/ ] return null != scm.branches.find { branch -> mainlines.find { mainlineRegEx -> return branch ==~ mainlineRegEx } } } /** * Return true if only the device test stages should run * @ return */ def isDeviceTestOnly() { return !params.RERUN_DEVICETEST.isEmpty() } /** * Returns true if the device test stage(s) should run */ def shouldRunDeviceTest() { return getPipelineMode() != PipelineMode.UNKNOWN } /** * Given a steps element, configures it to run the given shard indexes */ def defineStepsForDeviceTest(steps, int totalShards, int ...shardIndexes) { steps.with { checkoutRepo(steps) withAwsCredentials { def url = getArtifactShareUrl( "build" , deviceTestInputBuildId()) sh "./other/jenkins/devicetest_ensure_ready.sh '$url' " } script { // PR builds merge the PR HEAD into the target branch. In this case // we just want to pass the PR branch to the manual job. If // it isn't a PR build, the pass the commit. def commitToBuild = env.CHANGE_BRANCH?.trim() if (!commitToBuild) { commitToBuild = env.GIT_COMMIT } build job: "/Manual Builds/speedtestnet-android/deviceTest" , parameters: [ string(name: 'O2_INPUT_URL' , value: getArtifactShareUrl( "build" , deviceTestInputBuildId())), string(name: 'O2_OUTPUT_URL' , value: getArtifactShareUrl( "deviceTest" , deviceTestInputBuildId())), string(name: 'O2_COMMIT' , value: commitToBuild), string(name: 'O2_BRANCH' , value: env.GIT_BRANCH), string(name: 'O2_SHARD_COUNT' , value: "${totalShards}" ), string(name: 'O2_SHARD_INDEXES' , value: shardIndexes.join( ',' )) ], // Don't fail the stage if downstream job fails. We want to process // test results from unstable builds in our pipeline propagate: false } } } /** * Build id to use as input for device test stage */ def deviceTestInputBuildId() { return params.RERUN_DEVICETEST.isEmpty() ? env.BUILD_ID : params.RERUN_DEVICETEST } /** * Get the artifact share path for the given stage. * @param stage the stage with which the path is associated * @param buildId (optional) build id to use, defaults to current build id */ def getArtifactShareUrl( String stage, String buildId = env.BUILD_ID) { return sh(returnStdout: true , script: "./other/jenkins/share_artifacts.sh " + "--action getShareUrl " + "--jobName ${env.JOB_NAME} --jobBuild ${buildId} --jobStage ${stage}" ).trim() } /** * Process junit results from the jvm tests */ def processJvmJunitResults() { junit '**/build/test-results /**/ *.xml' } /** * Add the archived apk's as artifacts */ def addArtifacts() { archiveArtifacts artifacts: "Mobile4/build/outputs/apk /**/ *.apk" , fingerprint: true } def notifyUnhealthy() { if (getPipelineMode() != PipelineMode.RELEASE) { return } slackSend(channel: '#team-android-dev' , color: 'danger' , message: "<${currentBuild.absoluteUrl}|${currentBuild.fullDisplayName}>" + " :jenkins_angry: is unhealthy." ) } def notifyFixed() { if (getPipelineMode() != PipelineMode.RELEASE) { return } slackSend(channel: '#team-android-dev' , color: 'good' , message: "<${currentBuild.absoluteUrl}|${currentBuild.fullDisplayName}>" + ":jenkins: is healthy again." ) }
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Ian Wallace-Hoyt Thanks for the Jenkinsfile! My best guess is that the unfixed part of the issue has to do with having when expressions on more than one stage, or maybe beforeAgent: true, but I'm not sure. Do you have a screenshot of what the graph looks like for your Pipeline when you have the issue?

            Given the comments, I'll go ahead and reopen the issue. I am not sure whether the fixes in Blue Ocean 1.19.0 fixed a significant amount of the ways this problem could happen, and we are just left with some special cases, or if there are still a lot of outstanding issues.

            Show
            dnusbaum Devin Nusbaum added a comment - Ian Wallace-Hoyt Thanks for the Jenkinsfile! My best guess is that the unfixed part of the issue has to do with having when expressions on more than one stage, or maybe beforeAgent: true , but I'm not sure. Do you have a screenshot of what the graph looks like for your Pipeline when you have the issue? Given the comments, I'll go ahead and reopen the issue. I am not sure whether the fixes in Blue Ocean 1.19.0 fixed a significant amount of the ways this problem could happen, and we are just left with some special cases, or if there are still a lot of outstanding issues.
            Hide
            brianjmurrell Brian J Murrell added a comment -

            Devin Nusbaum A screenshot of the graph for the pipeline when in this state looks just like the existing two screenshots.

            Show
            brianjmurrell Brian J Murrell added a comment - Devin Nusbaum A screenshot of the graph for the pipeline when in this state looks just like the existing two screenshots.
            Hide
            dnusbaum Devin Nusbaum added a comment -

            Brian J Murrell Yes, I mean for Ian Wallace-Hoyt's Pipeline in particular, since we have the associated Jenkinsfile we could use the graph to see what other stages are being shown or not and in what state. Even better would be to have a screenshot of the graph from build with the issue and a screenshot of the graph for the previous build, since the problem relates to how those graphs are being combined.

            Show
            dnusbaum Devin Nusbaum added a comment - Brian J Murrell Yes, I mean for Ian Wallace-Hoyt 's Pipeline in particular, since we have the associated Jenkinsfile we could use the graph to see what other stages are being shown or not and in what state. Even better would be to have a screenshot of the graph from build with the issue and a screenshot of the graph for the previous build, since the problem relates to how those graphs are being combined.
            Hide
            ian_ookla Ian Wallace-Hoyt added a comment -

            Devin Nusbaum

             

            I opened a new PR and let it build. When it completed successfully, I rebuilt it. Same thing in both cases, the "Buddy: build apks" stage shows a queued.

             

            I took the screenshots and video after verifying that the stage was actually building but looking at the node view.

             

            jenkins_build1.png

            jenkins_build1.mov

            jenkins_buld2.png

            jenkins_buld2.mov

             

            Show
            ian_ookla Ian Wallace-Hoyt added a comment - Devin Nusbaum   I opened a new PR and let it build. When it completed successfully, I rebuilt it. Same thing in both cases, the "Buddy: build apks" stage shows a queued.   I took the screenshots and video after verifying that the stage was actually building but looking at the node view.   jenkins_build1.png jenkins_build1.mov jenkins_buld2.png jenkins_buld2.mov  

              People

              • Assignee:
                Unassigned
                Reporter:
                pracucci Marco Pracucci
              • Votes:
                24 Vote for this issue
                Watchers:
                32 Start watching this issue

                Dates

                • Created:
                  Updated: