Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37323

Ensure that Pipeline Graph can handle stage blocks

    Details

    • Epic Link:
    • Sprint:
      1.0-pre-beta-1, 1.0-beta-1
    • Similar Issues:

      Description

      Vivek to add notes after meeting with Sam

        Attachments

          Issue Links

            Activity

            Hide
            vivek Vivek Pandey added a comment -

            James Dumay Summary of discussion with Sam:

            Concerns:

            • With label block support there might be some steps out of stage blocks. Below 'echo 'I am not part of a stage' is not inside any stage, how does blueocean plans to present them for visualization? A virtual node (without label) to capture such hanging steps?
            
            stage('part-one') {
              echo 'I am part of a valid stage'
            }
            echo 'I am not part of a stage'
            stage('part-two') {
              echo 'I am part of a valid stage too!'
            }
            
            
            • Label Blocks support going to result in to support for nested stages, nested parallels. How does BlueOcean plans to handle them? There is also flattening of BO node, and they are listed to get logs for each steps. While nested stages can be treated as sequential BO nodes, however parallels nested inside branches might be tricky and a visualization scheme will make it easier for implementation. Or maybe these are edge cases and can be deferred for later BO API implementation.
            Show
            vivek Vivek Pandey added a comment - James Dumay Summary of discussion with Sam: Stage View pipeline to support Label Block Label block PR along with Stage View enhancements to release together, code completion in a week and week after release. https://github.com/jenkinsci/workflow-api-plugin/pull/6 where api implementation for walking is implemented https://github.com/jenkinsci/pipeline-stage-step-plugin/pull/4 where Label block is implemented https://github.com/jenkinsci/pipeline-graph-analysis-plugin implements timing analysis support, waiting to be released I am going to spend sometime next week to use workflow-api for blueocean use case to get any missing thing or any feedback to Sam before they are released. Concerns: With label block support there might be some steps out of stage blocks. Below 'echo 'I am not part of a stage' is not inside any stage, how does blueocean plans to present them for visualization? A virtual node (without label) to capture such hanging steps? stage( 'part-one' ) { echo 'I am part of a valid stage' } echo 'I am not part of a stage' stage( 'part-two' ) { echo 'I am part of a valid stage too!' } Label Blocks support going to result in to support for nested stages, nested parallels. How does BlueOcean plans to handle them? There is also flattening of BO node, and they are listed to get logs for each steps. While nested stages can be treated as sequential BO nodes, however parallels nested inside branches might be tricky and a visualization scheme will make it easier for implementation. Or maybe these are edge cases and can be deferred for later BO API implementation.
            Hide
            jamesdumay James Dumay added a comment -

            Vivek Pandey first concern is tracked in JENKINS-35836 - pretty low priority right now. On the second concern, I am not sure. Nested stages seems like a very bad idea - I haven't heard of a good case for it yet. Michael Neale will follow up.

            Show
            jamesdumay James Dumay added a comment - Vivek Pandey first concern is tracked in JENKINS-35836 - pretty low priority right now. On the second concern, I am not sure. Nested stages seems like a very bad idea - I haven't heard of a good case for it yet. Michael Neale will follow up.
            Hide
            jamesdumay James Dumay added a comment - - edited

            Vivek Pandey for the moment can we ensure that block scoped stages work the same way as regular stages, except for the edge cases you described above?

            Show
            jamesdumay James Dumay added a comment - - edited Vivek Pandey for the moment can we ensure that block scoped stages work the same way as regular stages, except for the edge cases you described above?
            Hide
            michaelneale Michael Neale added a comment -

            I am very keen on labelled block stages to NOT be nestable as I don't see why they should be.

            Show
            michaelneale Michael Neale added a comment - I am very keen on labelled block stages to NOT be nestable as I don't see why they should be.
            Hide
            vivek Vivek Pandey added a comment - - edited

            I have basic implementation using new ForkScanner API on branch task/JENKINS-37323.

            • Existing PipelineNodeGraphBuilder is re-written. It was implemented using FlowGrahWalker, now its going to use ForkScanner API.
            • ForkScanner and ChunkVisitor APIs look good. I am concerned about implementing nested parallel, stages inside parallels. Needs bit more work and it needs care implementing and more than that a clear visualization and DAG model. In first implementation, nested parallels/stages to be ignored
            • I found a possible bug dealing with parallel branch events being investigated by Sam. Fixed by Sam in workflow-api-plugin: https://github.com/jenkinsci/workflow-api-plugin/pull/10/commits/33e079e1e9b161e58cfdab4d224c1ee036117e89.

            TODOs

            • (Re)Implement union of partial run with last successful run using ForkScanner
            • Add tests with legacy stage as well as block scoped stages. Also add tests for nested parallels/stages inside parallel branch Need to determine expected behavior of nested paralleles/stgages.
            Show
            vivek Vivek Pandey added a comment - - edited I have basic implementation using new ForkScanner API on branch task/ JENKINS-37323 . Existing PipelineNodeGraphBuilder is re-written. It was implemented using FlowGrahWalker, now its going to use ForkScanner API. ForkScanner and ChunkVisitor APIs look good. I am concerned about implementing nested parallel, stages inside parallels. Needs bit more work and it needs care implementing and more than that a clear visualization and DAG model. In first implementation, nested parallels/stages to be ignored I found a possible bug dealing with parallel branch events being investigated by Sam. Fixed by Sam in workflow-api-plugin: https://github.com/jenkinsci/workflow-api-plugin/pull/10/commits/33e079e1e9b161e58cfdab4d224c1ee036117e89 . TODOs (Re)Implement union of partial run with last successful run using ForkScanner Add tests with legacy stage as well as block scoped stages. Also add tests for nested parallels/stages inside parallel branch Need to determine expected behavior of nested paralleles/stgages.
            Hide
            michaelneale Michael Neale added a comment -

            Note that we aren't clear on what will be visualised for nested stages (ugh) or parallel with non parallel elements at this stage.

            Show
            michaelneale Michael Neale added a comment - Note that we aren't clear on what will be visualised for nested stages (ugh) or parallel with non parallel elements at this stage.
            Hide
            vivek Vivek Pandey added a comment -

            Michael Neale Ok, I won't touch this area till we are clear on how we want to handle this case.

            Show
            vivek Vivek Pandey added a comment - Michael Neale Ok, I won't touch this area till we are clear on how we want to handle this case.
            Hide
            vivek Vivek Pandey added a comment -

            I have some more update. Instead of making one big change to support both block scoped stage as well as replace FlowGraphWalker by Fork scanner - it brings optimization to scanning process. Later is considerable amount of work as it requires complete re-write.

            Basically this ticket only adds support for block scoped stages. task/JENKINS-37323 is ready with this support as soon as pipeline-stage-step gets released.

            I am going to open another ticket to replace FlowGraphWalker by ForkedScanner.

            Show
            vivek Vivek Pandey added a comment - I have some more update. Instead of making one big change to support both block scoped stage as well as replace FlowGraphWalker by Fork scanner - it brings optimization to scanning process. Later is considerable amount of work as it requires complete re-write. Basically this ticket only adds support for block scoped stages. task/ JENKINS-37323 is ready with this support as soon as pipeline-stage-step gets released. I am going to open another ticket to replace FlowGraphWalker by ForkedScanner.
            Show
            vivek Vivek Pandey added a comment - Opened https://issues.jenkins-ci.org/browse/JENKINS-37667
            Hide
            vivek Vivek Pandey added a comment -

            Michael Neale James Dumay There is still a TODO of handling nested case. I think in first version of it we ignore them? I need to add some tests and make sure:

            • branches nested within parallel branch are ignored
            • stages nested within parallel branch are ignored
            Show
            vivek Vivek Pandey added a comment - Michael Neale James Dumay There is still a TODO of handling nested case. I think in first version of it we ignore them? I need to add some tests and make sure: branches nested within parallel branch are ignored stages nested within parallel branch are ignored
            Hide
            jamesdumay James Dumay added a comment -

            Yes please ignore them for the time being. I am hoping that we will have nesting disabled until we can sort out the visualisation.

            Show
            jamesdumay James Dumay added a comment - Yes please ignore them for the time being. I am hoping that we will have nesting disabled until we can sort out the visualisation.
            Hide
            vivek Vivek Pandey added a comment -

            Ok, I have implemented support to ignore nested stages/branches inside parallels. Please note that the /steps API are going to include nested stage/parallels steps. As soon as workflow-api/stage-step plugins are released then I will do final test and open a PR.

            Show
            vivek Vivek Pandey added a comment - Ok, I have implemented support to ignore nested stages/branches inside parallels. Please note that the /steps API are going to include nested stage/parallels steps. As soon as workflow-api/stage-step plugins are released then I will do final test and open a PR.
            Hide
            michaelneale Michael Neale added a comment - - edited

            I see that Andrew has moved pipeline config to using them - so does that mean it has been released?

            Also a question, if some steps are in stage blocks, and some outside, how does that look? (in clasic script). cc James Dumay

            Show
            michaelneale Michael Neale added a comment - - edited I see that Andrew has moved pipeline config to using them - so does that mean it has been released? Also a question, if some steps are in stage blocks, and some outside, how does that look? (in clasic script). cc James Dumay
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Thorsten Scherler
            Path:
            src/test/js/log-karaoke/stages.js
            src/test/resources/test_scripts/stages-with-wait.groovy
            http://jenkins-ci.org/commit/blueocean-acceptance-test/d1ebf33b61197b57a7db256061ef179a7b8786e1
            Log:
            JENKINS-37323 Implement test cases with new syntax

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Thorsten Scherler Path: src/test/js/log-karaoke/stages.js src/test/resources/test_scripts/stages-with-wait.groovy http://jenkins-ci.org/commit/blueocean-acceptance-test/d1ebf33b61197b57a7db256061ef179a7b8786e1 Log: JENKINS-37323 Implement test cases with new syntax
            Hide
            jamesdumay James Dumay added a comment -

            Vivek Pandey if you could attach some screenshots here with what Michael Neale was referring to that would be perfect.

            Show
            jamesdumay James Dumay added a comment - Vivek Pandey if you could attach some screenshots here with what Michael Neale was referring to that would be perfect.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Thorsten Scherler
            Path:
            PULL_REQUEST_TEMPLATE
            README.md
            package.json
            src/main/js/page_objects/blueocean/bluePipelineBranch.js
            src/main/js/page_objects/blueocean/bluePipelineRunDetail.js
            src/test/js/edgeCases/folder.js
            src/test/js/failing.js
            src/test/js/log-karaoke/stages.js
            src/test/resources/test_scripts/noStagesFailing.groovy
            http://jenkins-ci.org/commit/blueocean-acceptance-test/d3b6e5710b61371256067ffd17aeafc061b748a3
            Log:
            Merge remote-tracking branch 'origin/master' into JENKINS-37323

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Thorsten Scherler Path: PULL_REQUEST_TEMPLATE README.md package.json src/main/js/page_objects/blueocean/bluePipelineBranch.js src/main/js/page_objects/blueocean/bluePipelineRunDetail.js src/test/js/edgeCases/folder.js src/test/js/failing.js src/test/js/log-karaoke/stages.js src/test/resources/test_scripts/noStagesFailing.groovy http://jenkins-ci.org/commit/blueocean-acceptance-test/d3b6e5710b61371256067ffd17aeafc061b748a3 Log: Merge remote-tracking branch 'origin/master' into JENKINS-37323
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Thorsten Scherler
            Path:
            src/main/js/page_objects/blueocean/bluePipelineRunDetail.js
            src/test/js/queued.js
            http://jenkins-ci.org/commit/blueocean-acceptance-test/f73dda189ce92619f4999a0135932178c72c56c1
            Log:
            Merge remote-tracking branch 'origin/master' into JENKINS-37323

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Thorsten Scherler Path: src/main/js/page_objects/blueocean/bluePipelineRunDetail.js src/test/js/queued.js http://jenkins-ci.org/commit/blueocean-acceptance-test/f73dda189ce92619f4999a0135932178c72c56c1 Log: Merge remote-tracking branch 'origin/master' into JENKINS-37323
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Thorsten Scherler
            Path:
            src/main/js/page_objects/blueocean/bluePipelineActivity.js
            src/test/js/log-karaoke/stages.js
            src/test/resources/test_scripts/stages-with-wait-block-syntax.groovy
            src/test/resources/test_scripts/stages-with-wait-pipelineModel-syntax.groovy
            src/test/resources/test_scripts/stages-with-wait.groovy
            http://jenkins-ci.org/commit/blueocean-acceptance-test/5afc5f5bd8590828a1ba5fadf400233009752bc0
            Log:
            JENKINS-37323 adding all known pipeline syntax as test case

            Compare: https://github.com/jenkinsci/blueocean-acceptance-test/compare/d1ebf33b6119...5afc5f5bd859

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Thorsten Scherler Path: src/main/js/page_objects/blueocean/bluePipelineActivity.js src/test/js/log-karaoke/stages.js src/test/resources/test_scripts/stages-with-wait-block-syntax.groovy src/test/resources/test_scripts/stages-with-wait-pipelineModel-syntax.groovy src/test/resources/test_scripts/stages-with-wait.groovy http://jenkins-ci.org/commit/blueocean-acceptance-test/5afc5f5bd8590828a1ba5fadf400233009752bc0 Log: JENKINS-37323 adding all known pipeline syntax as test case Compare: https://github.com/jenkinsci/blueocean-acceptance-test/compare/d1ebf33b6119...5afc5f5bd859
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Thorsten Scherler
            Path:
            .gitignore
            pom.xml
            src/main/js/page_objects/blueocean/bluePipelineRunDetail.js
            src/test/js/edgeCases/folder.js
            src/test/js/failing.js
            src/test/js/log-karaoke/freestyle.js
            src/test/js/log-karaoke/freestylePing.js
            src/test/js/log-karaoke/noStages.js
            src/test/js/log-karaoke/stages.js
            src/test/resources/test_scripts/freestylePing.sh
            http://jenkins-ci.org/commit/blueocean-acceptance-test/d3547caa37af515fcb1174bbd22265106aebc828
            Log:
            Merge remote-tracking branch 'origin/master' into JENKINS-37323

            Compare: https://github.com/jenkinsci/blueocean-acceptance-test/compare/5afc5f5bd859...d3547caa37af

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Thorsten Scherler Path: .gitignore pom.xml src/main/js/page_objects/blueocean/bluePipelineRunDetail.js src/test/js/edgeCases/folder.js src/test/js/failing.js src/test/js/log-karaoke/freestyle.js src/test/js/log-karaoke/freestylePing.js src/test/js/log-karaoke/noStages.js src/test/js/log-karaoke/stages.js src/test/resources/test_scripts/freestylePing.sh http://jenkins-ci.org/commit/blueocean-acceptance-test/d3547caa37af515fcb1174bbd22265106aebc828 Log: Merge remote-tracking branch 'origin/master' into JENKINS-37323 Compare: https://github.com/jenkinsci/blueocean-acceptance-test/compare/5afc5f5bd859...d3547caa37af
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Thorsten Scherler
            Path:
            src/main/js/page_objects/blueocean/bluePipelineRunDetail.js
            src/test/js/log-karaoke/stages.js
            http://jenkins-ci.org/commit/blueocean-acceptance-test/b1ec3e1f137cd328d0ee83d5b0e56287b14a86b7
            Log:
            JENKINS-37323 fix merge problem and make sure we returning always self and not coditional

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Thorsten Scherler Path: src/main/js/page_objects/blueocean/bluePipelineRunDetail.js src/test/js/log-karaoke/stages.js http://jenkins-ci.org/commit/blueocean-acceptance-test/b1ec3e1f137cd328d0ee83d5b0e56287b14a86b7 Log: JENKINS-37323 fix merge problem and make sure we returning always self and not coditional
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Thorsten Scherler
            Path:
            src/main/js/page_objects/blueocean/bluePipelineActivity.js
            src/main/js/page_objects/blueocean/bluePipelineRunDetail.js
            src/test/js/log-karaoke/stages.js
            src/test/resources/test_scripts/stages-with-wait-block-syntax.groovy
            src/test/resources/test_scripts/stages-with-wait-pipelineModel-syntax.groovy
            http://jenkins-ci.org/commit/blueocean-acceptance-test/9f546a028e30a1b1c46282c9c55be028414cf57a
            Log:
            [FIX JENKINS-38017 JENKINS-37323] Implement test cases with new syntax (#30)

            • JENKINS-37323 fix merge problem and make sure we returning always self and not coditional
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Thorsten Scherler Path: src/main/js/page_objects/blueocean/bluePipelineActivity.js src/main/js/page_objects/blueocean/bluePipelineRunDetail.js src/test/js/log-karaoke/stages.js src/test/resources/test_scripts/stages-with-wait-block-syntax.groovy src/test/resources/test_scripts/stages-with-wait-pipelineModel-syntax.groovy http://jenkins-ci.org/commit/blueocean-acceptance-test/9f546a028e30a1b1c46282c9c55be028414cf57a Log: [FIX JENKINS-38017 JENKINS-37323] Implement test cases with new syntax (#30) JENKINS-37323 Implement test cases with new syntax JENKINS-37323 adding all known pipeline syntax as test case JENKINS-37323 fix merge problem and make sure we returning always self and not coditional

              People

              • Assignee:
                vivek Vivek Pandey
                Reporter:
                jamesdumay James Dumay
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: