Details

    • Sprint:
      Pipeline - July/August
    • Similar Issues:

      Description

      This ticket is closed – please read carefully

      Hi everyone,

      I am aware that this ticket is valued by many people. However, on review there are too many disparate use cases and problems attached to this issue that are already described elsewhere.

      This issue means too many things for too many people and makes it unsatisfiable with a single change. The appropriate action here is to close this ticket and ask watchers to watch and vote for a specific ticket that will solve their problem.

      If you are interested in these problems, please watch and vote for the following:

      • JENKINS-39203 - The entire pipeline gets marked as unstable rather than the stage or parallel where the unstable status was set.
      • JENKINS-45579 - Capability to mark a stage as "failed" with message and have the pipeline continue executing.
      • JENKINS-43995 - Infrastructure to allow any step to have its own distinct status. This enables the tickets listed above.
      • JENKINS-46170 - Add notes to stages and parallels

      I may have missed a few use cases. If that is the case:

      • Search JIRA for an issue matching your use case. If you find one, link it in the comments and I will update the ticket. Be sure to leave a comment on the linked ticket describing the problem you are trying to solve and then your desired solution to the problem.
      • If you cannot find an existing issue, raise a new one and comment here. Leave a comment on your new ticket describing problem you are trying to solve and then your desired solution to the problem.

      Thanks,
      James Dumay

      Original request
      It is often useful to provide more information up-front than a simple red or green "this step passed" or "this step failed". Our current in-house CI software has a "summary" field in its "Running Steps" equivalent, which can be filled with other info when a step finishes. For example, we have a step "upload-to-storage" which can optionally replace a "/latest/" symlink to the output it just uploaded. Either replacing or not replacing the symlink is correct, so we populate our "summary" field with the decision taken.

      I've attached an example from our current CI system - note that by allowing a step to populate its own summary, it means all pertinent information is available immediately to developers, on one page, rather than needing to always click through to log output for high-level information.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            JENKINS-26107 suggests a label for a block, but a (generally static) label set at the start, whereas this is mainly about adding summary information at the end. So that would probably have to be a different step.

            JENKINS-26523 notes that the ability to set Run.result may be useful.

            Show
            jglick Jesse Glick added a comment - JENKINS-26107 suggests a label for a block, but a (generally static) label set at the start , whereas this is mainly about adding summary information at the end . So that would probably have to be a different step. JENKINS-26523 notes that the ability to set Run.result may be useful.
            Hide
            jglick Jesse Glick added a comment -

            A related request was for a stage to be marked with a custom icon, to indicate what kind of stage this way, or to add some key/value pairs such as name of environment we deployed to.

            Show
            jglick Jesse Glick added a comment - A related request was for a stage to be marked with a custom icon, to indicate what kind of stage this way, or to add some key/value pairs such as name of environment we deployed to.
            Hide
            directhex Jo Shields added a comment - - edited

            Okay, to try & better explain what we're trying to achieve, here's our current implementation, which is split into three parts:

            Custom logic in the job shell script to annotate results

            roughly speaking, this looks like:

            STEP="System.Security-4.5"
            STARTTIME=`date +%s`
            echo "*** start: ${STEP}"
            ${TIMEOUTCMD} --signal=ABRT --kill-after=60s 5m make -w -C mcs/class/System.Security run-test TEST_HARNESS_FLAGS=-labels PROFILE=net_4_x && echo -e "*** end($(echo $(date +%s) - ${STARTTIME} | bc)): ${STEP}: \\e[42mPassed\e[0m" || echo -e "*** end($(echo $(date +%s) - ${STARTTIME} | bc)): ${STEP}: \\e[43mUnstable\e[0m"

            This assigns blocks of tests (roughly equivalent to Pipeline stages) to run together, with a common timeout, and either a "pass" status or an "unstable" status (unstable indicating this block had a non-zero exit status, but that execution of the script should continue). For steps where failure should be deemed critical, the colour code is red instead of yellow, status text says "failed", and there's some extra exit 1's thrown in.

            Groovy script to parse the build log and generate a job status table

            Pretty much just builds an HTML <table>, pairing "*** start" and "*** end" lines to build the table structure and write the execution time, and adding the most recent test runner output as an at-a-glance summary

            Aggregate the data from multiple jobs in a giant table, with a custom plugin

            this gives the most dense data view we can manage - 76 steps, 5 architectures, with an at-a-glance look for developers of "did I break tests grouped into X on architecture Y", or "is builder Z misbehaving and in need of attention"

            I would love to move as much of this crap into a Jenkinsfile as possible. Certainly parts 1 and 2 of the above seem theoretically feasible to me, but there are gaps that means I'd just end up emulating bash really badly in Groovy, rather than taking advantage of Pipeline features (because the Pipeline versions are missing things)

            Show
            directhex Jo Shields added a comment - - edited Okay, to try & better explain what we're trying to achieve, here's our current implementation, which is split into three parts: Custom logic in the job shell script to annotate results roughly speaking, this looks like: STEP="System.Security-4.5" STARTTIME=`date +%s` echo "*** start: ${STEP}" ${TIMEOUTCMD} --signal=ABRT --kill-after=60s 5m make -w -C mcs/class/System.Security run-test TEST_HARNESS_FLAGS=-labels PROFILE=net_4_x && echo -e "*** end($(echo $(date +%s) - ${STARTTIME} | bc)): ${STEP}: \\e[42mPassed\e[0m" || echo -e "*** end($(echo $(date +%s) - ${STARTTIME} | bc)): ${STEP}: \\e[43mUnstable\e[0m" This assigns blocks of tests (roughly equivalent to Pipeline stages) to run together, with a common timeout, and either a "pass" status or an "unstable" status (unstable indicating this block had a non-zero exit status, but that execution of the script should continue). For steps where failure should be deemed critical, the colour code is red instead of yellow, status text says "failed", and there's some extra exit 1 's thrown in. Groovy script to parse the build log and generate a job status table Pretty much just builds an HTML <table>, pairing "*** start" and "*** end" lines to build the table structure and write the execution time, and adding the most recent test runner output as an at-a-glance summary Aggregate the data from multiple jobs in a giant table, with a custom plugin this gives the most dense data view we can manage - 76 steps, 5 architectures, with an at-a-glance look for developers of "did I break tests grouped into X on architecture Y", or "is builder Z misbehaving and in need of attention" I would love to move as much of this crap into a Jenkinsfile as possible. Certainly parts 1 and 2 of the above seem theoretically feasible to me, but there are gaps that means I'd just end up emulating bash really badly in Groovy, rather than taking advantage of Pipeline features (because the Pipeline versions are missing things)
            Hide
            jglick Jesse Glick added a comment -

            I am already working on JENKINS-26107, which allows you to define a display label for a block (which could contain one step, a series of steps, some function calls, etc.). What I would propose is another step status that could take various parameters such as:

            • result: 'ABORTED' (or any other hudson.model.Result)
            • message: 'skipped 5 tests'
            • tags: [env: 'production', version: 123]
            • badge: 'url'
            • error: x (after catch (x))

            When run, the step would look up the enclosing BlockStartNode and attach some actions to it. (The BlockEndNode would be more logical, but that is not created until the block is actually closed. Alternately, the step could be placed outside and just after the block, but that makes it harder to access local variables and the like.) So your Jenkinsfile could look like

            label('Security tests') {
              sh 'make test'
              def failureCount = Integer.parseInt(readFile 'failureCount')
              if (failureCount == 0) {
                status badge: 'http://mycdn.com/checkmark.png'
              } else {
                status badge: 'http://mycdn.com/redx.png', result: 'UNSTABLE', message: "${failureCount} test(s) failed"
              }
            }
            

            which I think is reasonably readable, and allows the job to record all relevant information in the flow graph in a way that is recoverable from the Java APIs (and, albeit awkwardly for now, from the REST API). The basic Pipeline Steps UI can certainly show all this information, and some kind of unobtrusive formatted text can be added to the textual build log too.

            Then the question becomes how sophisticated you can expect the “stock” stage view visualization to be when it comes to representing this information. JENKINS-33185 proposes a significant overhaul to that view, in the context of which it would make sense to display this status information directly.

            The UI you are showing requires more than that, however, since you are displaying multiple results per build according to matrix configurations. JENKINS-27395 requests a way for the Jenkinsfile to scope things like archived test results to a particular configuration of matrix axes, where this information would then become accessible to other plugins. In particular the stage view (or whatever the visualization of Pipeline builds is called by then) would need to render matrices of results when defined.

            No matter what gets implemented in Pipeline, you are not going to have a UI as well tuned to your use case as what you currently have in your custom plugin, but perhaps the generic UI would be good enough, or at least your custom plugin could become a thinner veneer.

            My immediate question is what features are safe to release incrementally. JENKINS-26107 itself is complete, but I want to spend some time studying or even prototyping this issue, JENKINS-28119, JENKINS-27395, JENKINS-26132, and JENKINS-33185 to make sure the design is not foreclosing or impeding future expansions.

            Show
            jglick Jesse Glick added a comment - I am already working on JENKINS-26107 , which allows you to define a display label for a block (which could contain one step, a series of steps, some function calls, etc.). What I would propose is another step status that could take various parameters such as: result: 'ABORTED' (or any other hudson.model.Result ) message: 'skipped 5 tests' tags: [env: 'production', version: 123] badge: 'url' error: x (after catch (x) ) When run, the step would look up the enclosing BlockStartNode and attach some actions to it. (The BlockEndNode would be more logical, but that is not created until the block is actually closed. Alternately, the step could be placed outside and just after the block, but that makes it harder to access local variables and the like.) So your Jenkinsfile could look like label( 'Security tests' ) { sh 'make test' def failureCount = Integer .parseInt(readFile 'failureCount' ) if (failureCount == 0) { status badge: 'http: //mycdn.com/checkmark.png' } else { status badge: 'http: //mycdn.com/redx.png' , result: 'UNSTABLE' , message: "${failureCount} test(s) failed" } } which I think is reasonably readable, and allows the job to record all relevant information in the flow graph in a way that is recoverable from the Java APIs (and, albeit awkwardly for now, from the REST API). The basic Pipeline Steps UI can certainly show all this information, and some kind of unobtrusive formatted text can be added to the textual build log too. Then the question becomes how sophisticated you can expect the “stock” stage view visualization to be when it comes to representing this information. JENKINS-33185 proposes a significant overhaul to that view, in the context of which it would make sense to display this status information directly. The UI you are showing requires more than that, however, since you are displaying multiple results per build according to matrix configurations. JENKINS-27395 requests a way for the Jenkinsfile to scope things like archived test results to a particular configuration of matrix axes, where this information would then become accessible to other plugins. In particular the stage view (or whatever the visualization of Pipeline builds is called by then) would need to render matrices of results when defined. No matter what gets implemented in Pipeline, you are not going to have a UI as well tuned to your use case as what you currently have in your custom plugin, but perhaps the generic UI would be good enough, or at least your custom plugin could become a thinner veneer. My immediate question is what features are safe to release incrementally. JENKINS-26107 itself is complete, but I want to spend some time studying or even prototyping this issue, JENKINS-28119 , JENKINS-27395 , JENKINS-26132 , and JENKINS-33185 to make sure the design is not foreclosing or impeding future expansions.
            Hide
            jglick Jesse Glick added a comment -

            Another example use case is that github-branch-source could look for the resulting status notes and publish them to the GH Status API, so you would be able to see progress of your build in different sections right in a pull request.

            Show
            jglick Jesse Glick added a comment - Another example use case is that github-branch-source could look for the resulting status notes and publish them to the GH Status API, so you would be able to see progress of your build in different sections right in a pull request.
            Hide
            svanoort Sam Van Oort added a comment -

            Per JENKINS-36087 this should probably get some sort of stage view UI representation.

            Show
            svanoort Sam Van Oort added a comment - Per JENKINS-36087 this should probably get some sort of stage view UI representation.
            Hide
            svanoort Sam Van Oort added a comment -

            To flesh out general implementation thoughts (in case I knock this out as a rainy day or weekend project).

            Work breaks down roughly like this:

            Pipeline core plugins work

            • Action type to attach to the Blocks (probably on the BlockEndNode, so it can be accumulated during execution and then written out). Probably will take a

              Map<String, Object>

              for field, value pairs. Alternately, single Name,Value pairs for each action (with extension point for defined annotations that includes type restrictions).
            • Built-in annotation: custom status (implementation of the action, or a default entry in map, if we go that way).
            • Marker step object 'annotate' that attaches the action in question (so the flow graph shows when it is entered) – this gets propagated up to containing blocks, I think?
            • Pipeline core UI (tree view): show annotations on steps

            Stage View Work

            • Pipeline rest API (stage view): add annotation info to the objects.
            • Pipeline Stage view UI: add a new JS extension point impl (example exists) for the annotation displays, to display annotation notes and create a popup with them.
            • Stage view REST/UI: override stage StatusExt status using status annotations on block OR add additional information to StageExt objects that the UI consumes if present (extending the existing status coloring) --> this is probably what is needed (for parallel use too).
            Show
            svanoort Sam Van Oort added a comment - To flesh out general implementation thoughts (in case I knock this out as a rainy day or weekend project). Work breaks down roughly like this: Pipeline core plugins work Action type to attach to the Blocks (probably on the BlockEndNode, so it can be accumulated during execution and then written out). Probably will take a Map<String, Object> for field, value pairs. Alternately, single Name,Value pairs for each action (with extension point for defined annotations that includes type restrictions). Built-in annotation: custom status (implementation of the action, or a default entry in map, if we go that way). Marker step object 'annotate' that attaches the action in question (so the flow graph shows when it is entered) – this gets propagated up to containing blocks, I think? Pipeline core UI (tree view): show annotations on steps Stage View Work Pipeline rest API (stage view): add annotation info to the objects. Pipeline Stage view UI: add a new JS extension point impl (example exists) for the annotation displays, to display annotation notes and create a popup with them. Stage view REST/UI: override stage StatusExt status using status annotations on block OR add additional information to StageExt objects that the UI consumes if present (extending the existing status coloring) --> this is probably what is needed (for parallel use too).
            Hide
            svanoort Sam Van Oort added a comment -

            Untagging stage view, since support in UI would probably be through Blue Ocean instead.

            Show
            svanoort Sam Van Oort added a comment - Untagging stage view, since support in UI would probably be through Blue Ocean instead.
            Hide
            svanoort Sam Van Oort added a comment -
            Show
            svanoort Sam Van Oort added a comment - Proof-of-concept approach here: https://github.com/jenkinsci/workflow-api-plugin/pull/12
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Sam Van Oort
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/actions/TagsAction.java
            http://jenkins-ci.org/commit/workflow-api-plugin/f6de209ae70887dd47a28075f1972bf1642b68d7
            Log:
            Merge pull request #12 from jenkinsci/tagsaction-JENKINS-26522

            Create a new TagsAction JENKINS-26522

            Compare: https://github.com/jenkinsci/workflow-api-plugin/compare/8ca82ae8180d...f6de209ae708

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: src/main/java/org/jenkinsci/plugins/workflow/actions/TagsAction.java http://jenkins-ci.org/commit/workflow-api-plugin/f6de209ae70887dd47a28075f1972bf1642b68d7 Log: Merge pull request #12 from jenkinsci/tagsaction- JENKINS-26522 Create a new TagsAction JENKINS-26522 Compare: https://github.com/jenkinsci/workflow-api-plugin/compare/8ca82ae8180d...f6de209ae708
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Sam Van Oort
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/actions/TagsAction.java
            http://jenkins-ci.org/commit/workflow-api-plugin/428319414b7fa7288bec1652202188a1ea52daa5
            Log:
            Revert "Create a new TagsAction JENKINS-26522"

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: src/main/java/org/jenkinsci/plugins/workflow/actions/TagsAction.java http://jenkins-ci.org/commit/workflow-api-plugin/428319414b7fa7288bec1652202188a1ea52daa5 Log: Revert "Create a new TagsAction JENKINS-26522 "
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Sam Van Oort
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/actions/TagsAction.java
            http://jenkins-ci.org/commit/workflow-api-plugin/43f7e86b0323d7031429b5e5552393eb9d027bfe
            Log:
            Merge pull request #14 from jenkinsci/revert-12-tagsaction-JENKINS-26522

            Revert "Create a new TagsAction JENKINS-26522"

            Compare: https://github.com/jenkinsci/workflow-api-plugin/compare/f6de209ae708...43f7e86b0323

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: src/main/java/org/jenkinsci/plugins/workflow/actions/TagsAction.java http://jenkins-ci.org/commit/workflow-api-plugin/43f7e86b0323d7031429b5e5552393eb9d027bfe Log: Merge pull request #14 from jenkinsci/revert-12-tagsaction- JENKINS-26522 Revert "Create a new TagsAction JENKINS-26522 " Compare: https://github.com/jenkinsci/workflow-api-plugin/compare/f6de209ae708...43f7e86b0323
            Hide
            alexdean Alex Dean added a comment - - edited

            Sam Van Oort could you provide a brief example of how this will be used in a Jenkins file? From your description I'm imagining something like this, but I'm not sure if I've understood correctly or not.

            node {
              stage('example') {
                if (condition) {
                  note('condition was true')
                } else {
                  note('condition was false')
                }
              }
            }
            

            and then the "condition was (true|false)" notation would be reported in the project's stage view?

            Show
            alexdean Alex Dean added a comment - - edited Sam Van Oort could you provide a brief example of how this will be used in a Jenkins file? From your description I'm imagining something like this, but I'm not sure if I've understood correctly or not. node { stage( 'example' ) { if (condition) { note( 'condition was true ' ) } else { note( 'condition was false ' ) } } } and then the "condition was (true|false)" notation would be reported in the project's stage view?
            Hide
            svanoort Sam Van Oort added a comment -

            Alex Dean There's a variety of different ways to use it, I think.

            The simplest ones:

            // Annotations are string-string key:value pairs now
            stage('example') {
               annotate('WebLink','http://example.com');
               annotate('MySpecialCondition','true');
            }
            

            Possibly we might want to allow it to take blocks?

            // Annotations are string-string key:value pairs now
            stage('example') {
               annotate('WebLink','http://example.com') {
                 sh './deploy.sh www.example.com'
               }
            }
            

            Also, we could use it for status-overrides, if simple support is added to the pipeline-graph-analysis plugin:

            // Annotations are string-string key:value pairs now
            stage('example') {
               try {
                  sh 'mvn flakyTest'
               } catch (Exception ex) {
                   stageStatus 'UNSTABLE'; // Sets *just this* stage status to unstable (once status computation supports it) without changing the overall build result. 
                   annotate('currentStage.status', 'UNSTABLE'); // actually creates the same result?
                }
            }
            

            Or at least that's what would make sense to me, not sure what others think.

            Show
            svanoort Sam Van Oort added a comment - Alex Dean There's a variety of different ways to use it, I think. The simplest ones: // Annotations are string-string key:value pairs now stage( 'example' ) { annotate( 'WebLink' , 'http: //example.com' ); annotate( 'MySpecialCondition' , ' true ' ); } Possibly we might want to allow it to take blocks? // Annotations are string-string key:value pairs now stage( 'example' ) { annotate( 'WebLink' , 'http: //example.com' ) { sh './deploy.sh www.example.com' } } Also, we could use it for status-overrides, if simple support is added to the pipeline-graph-analysis plugin: // Annotations are string-string key:value pairs now stage( 'example' ) { try { sh 'mvn flakyTest' } catch (Exception ex) { stageStatus 'UNSTABLE' ; // Sets *just this * stage status to unstable (once status computation supports it) without changing the overall build result. annotate( 'currentStage.status' , 'UNSTABLE' ); // actually creates the same result? } } Or at least that's what would make sense to me, not sure what others think.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Sam Van Oort
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/actions/TagsAction.java
            http://jenkins-ci.org/commit/workflow-api-plugin/a5a7e8cce052f430149575b02fb41d145b89147b
            Log:
            Revert "Revert "Create a new TagsAction JENKINS-26522""

            This reverts commit 428319414b7fa7288bec1652202188a1ea52daa5.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Sam Van Oort Path: src/main/java/org/jenkinsci/plugins/workflow/actions/TagsAction.java http://jenkins-ci.org/commit/workflow-api-plugin/a5a7e8cce052f430149575b02fb41d145b89147b Log: Revert "Revert "Create a new TagsAction JENKINS-26522 "" This reverts commit 428319414b7fa7288bec1652202188a1ea52daa5.
            Hide
            svanoort Sam Van Oort added a comment -

            James Dumay Not quite in review – part of this is, but there's a bit more to be done before we can consider it complete.

            Show
            svanoort Sam Van Oort added a comment - James Dumay Not quite in review – part of this is, but there's a bit more to be done before we can consider it complete.
            Hide
            asanami Alex Sanami added a comment -

            Is it also possible to expose in the API, so subsequent stages in the pipeline can find the status of previous stages.

            Show
            asanami Alex Sanami added a comment - Is it also possible to expose in the API, so subsequent stages in the pipeline can find the status of previous stages.
            Hide
            anomalizer Arvind Jayaprakash added a comment -

            The stageStatus directive as shown above looks good for manual overrides.

            We will also have to get existing plugins to start setting this as opposed to the build status. For example, junit plugin on test failure, findbugs on exceeding violation thresholds etc. etc.

            Show
            anomalizer Arvind Jayaprakash added a comment - The stageStatus directive as shown above looks good for manual overrides. We will also have to get existing plugins to start setting this as opposed to the build status. For example, junit plugin on test failure, findbugs on exceeding violation thresholds etc. etc.
            Hide
            vanta Krzysztof Wolny added a comment -

            How about making a step/stage programatically disabled/skipped (like in Gradle) so I can decide in runtime, based on previous steps/stages results if I want to execute next one(s) or not?

            Show
            vanta Krzysztof Wolny added a comment - How about making a step/stage programatically disabled/skipped (like in Gradle) so I can decide in runtime, based on previous steps/stages results if I want to execute next one(s) or not?
            Hide
            jglick Jesse Glick added a comment -

            Krzysztof Wolny that is something you can do already. Please reserve this issue for discussion of visualizations.

            Show
            jglick Jesse Glick added a comment - Krzysztof Wolny that is something you can do already. Please reserve this issue for discussion of visualizations.
            Hide
            vanta Krzysztof Wolny added a comment -

            Jesse Glick I didn't know, can you redirect me to place when I can check it?

            Show
            vanta Krzysztof Wolny added a comment - Jesse Glick I didn't know, can you redirect me to place when I can check it?
            Hide
            jglick Jesse Glick added a comment -

            Just write whatever logic you need in your script. Please do not use JIRA for advice/support.

            Show
            jglick Jesse Glick added a comment - Just write whatever logic you need in your script. Please do not use JIRA for advice/support.
            Hide
            m4dc4p Justin Bailey added a comment -

            Sam Van Oort Did this PR land elsewhere? Would really like to have this in order to replace a 'promoted build' job with a pipeline.

            Show
            m4dc4p Justin Bailey added a comment - Sam Van Oort Did this PR land elsewhere? Would really like to have this in order to replace a 'promoted build' job with a pipeline.
            Hide
            jglick Jesse Glick added a comment -

            Justin Bailey I suppose you are rather looking for JENKINS-36089.

            Show
            jglick Jesse Glick added a comment - Justin Bailey I suppose you are rather looking for JENKINS-36089 .
            Hide
            abayer Andrew Bayer added a comment -

            So we've got TagsAction for behind-the-scenes string-to-string metadata/annotations currently. There's no way to add tags from within a Pipeline directly, but I'm still not 100% convinced of that use case.

            What I'm thinking about now is twofold - first, a more status-specific Action - perhaps called something as dumb as FlowNodeStatusAction, which would allow steps to set their status to a Result, either at the same time they set the entire build result or just setting the step status on its own. I'd like this to not solely be for Block - there's real value in being able to mark specific steps with their own statuses, and then aggregating the status of steps within a block to determine the block's status, etc. This would enable us to resolve issues like JENKINS-42863 and/or JENKINS-39203. Worth mentioning that this alone won't solve those issues, but it would provide the tools to solve those.

            Second would be non-string metadata - most immediately and specifically, test results (hi there, JENKINS-26395). But I know there are some potential problems with adding too much data to FlowNode actions directly, so we'd probably go with the same approach the junit plugin (and its many cousins) are using with runs - saving the results themselves in a separate file and pointing to that file from the action on the FlowNode

            Sam Van Oort - thoughts?

            Show
            abayer Andrew Bayer added a comment - So we've got TagsAction for behind-the-scenes string-to-string metadata/annotations currently. There's no way to add tags from within a Pipeline directly, but I'm still not 100% convinced of that use case. What I'm thinking about now is twofold - first, a more status-specific Action - perhaps called something as dumb as FlowNodeStatusAction , which would allow steps to set their status to a Result , either at the same time they set the entire build result or just setting the step status on its own. I'd like this to not solely be for Block - there's real value in being able to mark specific steps with their own statuses, and then aggregating the status of steps within a block to determine the block's status, etc. This would enable us to resolve issues like  JENKINS-42863 and/or  JENKINS-39203 . Worth mentioning that this alone won't solve those issues, but it would provide the tools to solve those. Second would be non-string metadata - most immediately and specifically, test results (hi there, JENKINS-26395 ). But I know there are some potential problems with adding too much data to FlowNode actions directly, so we'd probably go with the same approach the junit plugin (and its many cousins) are using with runs - saving the results themselves in a separate file and pointing to that file from the action on the FlowNode .  Sam Van Oort - thoughts?
            Hide
            svanoort Sam Van Oort added a comment -

            Andrew Bayer I'd rather not add another Action  - we added the TagsAction specifically with an eye towards this, and Result values have straightforward canonical string representations.  

            I think aggregating statuses within a block is not actually a trivial process because of the number of potential combinations of different statuses in different orders and across parallels.  I prefer to allow setting a single override for the overall block that determines its status (but can be modified later) which keeps the handling simple.  However either way we'd want a concrete and detailed proposal to work from – this is not something to do ad-hoc because we've been bitten by ignoring these layered complexities.

            For test results with the TagsAction we would not be encapsulating a complete result, but an identifier for the test result (perhaps indicating the flownode it runs with) – similarly to how you describe. 

            Show
            svanoort Sam Van Oort added a comment - Andrew Bayer I'd rather not add another Action  - we added the TagsAction specifically with an eye towards this, and Result values have straightforward canonical string representations.   I think aggregating statuses within a block is not actually a trivial process because of the number of potential combinations of different statuses in different orders and across parallels.  I prefer to allow setting a single override for the overall block that determines its status (but can be modified later) which keeps the handling simple.  However either way we'd want a concrete and detailed proposal to work from – this is not something to do ad-hoc because we've been bitten by ignoring these layered complexities. For test results with the TagsAction we would not be encapsulating a complete result, but an identifier for the test result (perhaps indicating the flownode it runs with) – similarly to how you describe. 
            Hide
            abayer Andrew Bayer added a comment - - edited

            Aaaaah. TagsAction does make sense there. re: block status - would you put the status/test results pointer/whatever on the BlockStartStep or BlockEndStep? My instincts tend towards BlockStartStep, since we don't have the BlockEndStep yet at the time that something like junit could be adding test results/setting status, but I could be wrong. =)

            Show
            abayer Andrew Bayer added a comment - - edited Aaaaah. TagsAction does make sense there. re: block status - would you put the status/test results pointer/whatever on the BlockStartStep or BlockEndStep ? My instincts tend towards BlockStartStep , since we don't have the BlockEndStep yet at the time that something like junit could be adding test results/setting status, but I could be wrong. =)
            Hide
            svanoort Sam Van Oort added a comment -

            Andrew Bayer Convention (at least with ErrorActions, which we should follow the semantics of) is to put the status/results on the current node and the BlockEndNode.   This follows the convention that only the current head (tip) nodes are mutable, which permits caching and allows some simplifying assumptions to be made.  It also avoids the insanity of backtracking to modify every single BlockStartNode up to the beginning of the run, because we only need to track the current state. 

            I'm sure we could find a straightforward way to keep information about currently executing blocks to simplify dealing with incomplete ones – I have several ideas for the future roadmap along those lines already.

            Show
            svanoort Sam Van Oort added a comment - Andrew Bayer Convention (at least with ErrorActions, which we should follow the semantics of) is to put the status/results on the  current  node and the  BlockEndNode.   This follows the convention that only the current head (tip) nodes are mutable, which permits caching and allows some simplifying assumptions to be made.  It also avoids the insanity of backtracking to modify every single BlockStartNode up to the beginning of the run, because we only need to track the current state.  I'm sure we could find a straightforward way to keep information about currently executing blocks to simplify dealing with incomplete ones – I have several ideas for the future roadmap along those lines already.
            Hide
            abayer Andrew Bayer added a comment -

            Sam Van Oort Ok, so how would we propagate from the current node to the BlockEndNode? I may well be wrong, but I assumed that wasn't created until the end of the block.

            Show
            abayer Andrew Bayer added a comment - Sam Van Oort Ok, so how would we propagate from the current node to the BlockEndNode ? I may well be wrong, but I assumed that wasn't created until the end of the block.
            Hide
            abayer Andrew Bayer added a comment -

            Sam Van Oort Ok, dug into workflow-cps to figure that out. We aren't actually going from current node -> StepEndNode - the ErrorAction is ending up on StepEndNode via handling of an abnormal Outcome (with a Throwable) in CpsBodyExecution. So I my gut instinct at this point is that we'll need CpsBodyExecution.SuccessAdapter to traverse its nodes to find relevant TagsAction instances and add them on the StepEndNode. Which...I don't like. Maybe it makes more sense to do something like:

            • When we start a body, add a TagsAction to its StepContext, somehow scoped to this specific body (so that nested bodies can each have their own TagsAction).
            • When we hit a relevant status or test result, etc, add (if not existing) a TagsAction to the current node and add the relevant tags to it. This may not be needed - we may opt to bypass the current node completely, but I feel like we want to have that information on the specific node as well as on the whole block. Your call.
            • At the same time, add the relevant tags to the body's TagsAction from the StepContext. Some logic will be needed for knowing, e.g., that status can't get better, just worse, tags that can have multiple values (i.e., while ideally we'd have only one junit invocation per block, we can't guarantee that, so we need to be able to record multiple test results per block - maybe TagsAction should support lists of Strings for values?).
            • At completion of the body, in both FailureAdapter and SuccessAdapter, pull the body's TagsAction from the StepContext and add any values in it to the StepEndNode.
            • And finally, propagate up the body's TagsAction contents to any possible parent body's TagsAction.

            How's that sound?

            Show
            abayer Andrew Bayer added a comment - Sam Van Oort Ok, dug into workflow-cps to figure that out. We aren't actually going from current node -> StepEndNode - the ErrorAction is ending up on StepEndNode via handling of an abnormal Outcome (with a Throwable ) in CpsBodyExecution . So I my gut instinct at this point is that we'll need CpsBodyExecution.SuccessAdapter to traverse its nodes to find relevant TagsAction instances and add them on the StepEndNode . Which...I don't like. Maybe it makes more sense to do something like: When we start a body, add a TagsAction to its StepContext , somehow scoped to this specific body (so that nested bodies can each have their own TagsAction ). When we hit a relevant status or test result, etc, add (if not existing) a TagsAction to the current node and add the relevant tags to it. This may not be needed - we may opt to bypass the current node completely, but I feel like we want to have that information on the specific node as well as on the whole block. Your call. At the same time, add the relevant tags to the body's TagsAction from the StepContext . Some logic will be needed for knowing, e.g., that status can't get better, just worse, tags that can have multiple values (i.e., while ideally we'd have only one junit invocation per block, we can't guarantee that, so we need to be able to record multiple test results per block - maybe TagsAction should support lists of Strings for values?). At completion of the body, in both FailureAdapter and SuccessAdapter , pull the body's TagsAction from the StepContext  and add any values in it to the StepEndNode . And finally, propagate up the body's TagsAction contents to any possible parent body's TagsAction . How's that sound?
            Hide
            abayer Andrew Bayer added a comment -

            Thinking further - some tag names will imply/require additional behavior, like status. So maybe we also need an extension point for "special tag names", that would provide a tag name and a method for actually setting the tag. Something like this:

            public abstract class SpecialTag implements ExtensionPoint {
                @Nonnull
                public abstract String getTagName();
            
                public void setSpecialTag(@Nonnull TagsAction action, @Nonnull String value);
            
                public static ExtensionList<SpecialTag> all() {
                    return ExtensionList.lookup(SpecialTag.class);
                }
            }

            Then, the theoretical StatusTag would be something like:

            @Extension
            public class StatusTag extends SpecialTag {
                public String getTagName() {
                    return "BLOCK_STATUS";
                }
            
                public void setSpecialTag(@Nonnull TagsAction action, @Nonnull String value) {
                    if (value != null) {
                        String currentValue = action.getTagValue(getTagName());
                        if (currentValue == null || Result.fromString(value).isWorseThan(Result.fromString(currentValue))) {
                            action.addTag(this, value);
                        }
                    }
                }
            }

            And over in TagsAction itself, we'd change addTag to two versions...

            public void addTag(@CheckForNull String tag, @CheckForNull String value) {
                if (tag != null && value != null && !tag.isEmpty() && !value.isEmpty()) {
                    boolean foundSpecialTag = false;
                    for (SpecialTag t : all()) {
                        if (t.getTagName().equals(tagName)) {
                            t.setSpecialTag(this, value);
                            foundSpecialTag = true;
                        }
                    }
                    if (!foundSpecialTag) {
                        tags.put(tag, value);
                    }
                }
            }
            
            public void addTag(@Nonnull SpecialTag specialTag, @Nonnull String value) {
                tags.put(specialTag.getTagName(), value);
            }
            

            Just a thought, but hey.

            Show
            abayer Andrew Bayer added a comment - Thinking further - some tag names will imply/require additional behavior, like status. So maybe we also need an extension point for "special tag names", that would provide a tag name and a method for actually setting the tag. Something like this: public abstract class SpecialTag implements ExtensionPoint { @Nonnull public abstract String getTagName(); public void setSpecialTag(@Nonnull TagsAction action, @Nonnull String value); public static ExtensionList<SpecialTag> all() { return ExtensionList.lookup(SpecialTag.class); } } Then, the theoretical StatusTag would be something like: @Extension public class StatusTag extends SpecialTag { public String getTagName() { return "BLOCK_STATUS" ; } public void setSpecialTag(@Nonnull TagsAction action, @Nonnull String value) { if (value != null ) { String currentValue = action.getTagValue(getTagName()); if (currentValue == null || Result.fromString(value).isWorseThan(Result.fromString(currentValue))) { action.addTag( this , value); } } } } And over in TagsAction itself, we'd change addTag to two versions... public void addTag(@CheckForNull String tag, @CheckForNull String value) { if (tag != null && value != null && !tag.isEmpty() && !value.isEmpty()) { boolean foundSpecialTag = false ; for (SpecialTag t : all()) { if (t.getTagName().equals(tagName)) { t.setSpecialTag( this , value); foundSpecialTag = true ; } } if (!foundSpecialTag) { tags.put(tag, value); } } } public void addTag(@Nonnull SpecialTag specialTag, @Nonnull String value) { tags.put(specialTag.getTagName(), value); } Just a thought, but hey.
            Hide
            jglick Jesse Glick added a comment -

            We aren't actually going from current node -> StepEndNode - the ErrorAction is ending up on StepEndNode via handling of an abnormal Outcome (with a Throwable) in CpsBodyExecution.

            Right, this is not baked into the system at all, it is just an effect of the fact that an exception will be thrown up unless caught. workflow-cps attaches an ErrorAction to any step which ends in an exception, no more no less.

            Would it not be simpler and more flexible to just attach any tag metadata to the current node, and have higher-level visualization APIs determine what scope(s) to associate that metadata with—later, on demand?

            I probably missed a bunch of discussion so feel free to ignore if what I am saying does not make sense or has already been considered and rejected.

            Show
            jglick Jesse Glick added a comment - We aren't actually going from current node -> StepEndNode - the ErrorAction is ending up on StepEndNode via handling of an abnormal Outcome (with a Throwable ) in CpsBodyExecution . Right, this is not baked into the system at all, it is just an effect of the fact that an exception will be thrown up unless caught. workflow-cps attaches an ErrorAction to any step which ends in an exception, no more no less. Would it not be simpler and more flexible to just attach any tag metadata to the current node, and have higher-level visualization APIs determine what scope(s) to associate that metadata with—later, on demand? I probably missed a bunch of discussion so feel free to ignore if what I am saying does not make sense or has already been considered and rejected.
            Hide
            svanoort Sam Van Oort added a comment -

            Right, this is not baked into the system at all, it is just an effect of the fact that an exception will be thrown up unless caught. workflow-cps attaches an ErrorAction to any step which ends in an exception, no more no less

            As Andrew Bayer observed correctly, it's just throwing it on there when it's closing out the blocks. 

            I think we could bind this into the execution state (or context) associated with tracking open blocks during execution, to reduce complexity of attaching the statuses. 

            Would it not be simpler and more flexible to just attach any tag metadata to the current node, and have higher-level visualization APIs determine what scope(s) to associate that metadata with—later, on demand?

            NO. Jesse Glick this sort of approach has created a nightmare for Stage View, Blue Ocean before pipeline graph analysis libs ("Bismuth") and then in implementing the complex and nasty StatusAndTiming rules + UI representations.  We cannot make these rules even more complicated – it is at the tipping point where it is unsustainable to make them more complex, and I have to walk through long explanations for the results given because they are already hard to follow.  A new status coding must replace the previous in an either-or fashion and provide a simple, explicit result following from simple rules (probably the same rules of precedence we use for BuildResults being determined by the worse result).  It also needs to trickle up to the block level.

            Feels like I could almost produce a PoC implementation in the time we're spending talking about this. 

            Show
            svanoort Sam Van Oort added a comment - Right, this is not baked into the system at all, it is just an effect of the fact that an exception will be thrown up unless caught. workflow-cps attaches an ErrorAction to any step which ends in an exception, no more no less As Andrew Bayer  observed correctly, it's just throwing it on there when it's closing out the blocks.  I think we could bind this into the execution state (or context) associated with tracking open blocks during execution, to reduce complexity of attaching the statuses.  Would it not be simpler and more flexible to just attach any tag metadata to the current node, and have higher-level visualization APIs determine what scope(s) to associate that metadata with—later, on demand? NO. Jesse Glick  this sort of approach has created a nightmare for Stage View, Blue Ocean before pipeline graph analysis libs ("Bismuth") and then in implementing the complex and nasty StatusAndTiming rules + UI representations.  We cannot make these rules even more complicated – it is at the tipping point where it is unsustainable to make them more complex, and I have to walk through long explanations for the results given because they are already hard to follow.  A new status coding must replace the previous in an either-or fashion and provide a simple, explicit result following from simple rules (probably the same rules of precedence we use for BuildResults being determined by the worse result).  It also needs to trickle up to the block level. Feels like I could almost produce a PoC implementation in the time we're spending talking about this. 
            Hide
            abayer Andrew Bayer added a comment -

            I've got an initial design doc up at https://github.com/jenkinsci/workflow-api-plugin/pull/40 - I'm stuck on figuring out how to actually propagate up to the enclosing blocks, and have been for a few days, so I'm giving up on figuring it out solo and begging for help. =)

            Show
            abayer Andrew Bayer added a comment - I've got an initial design doc up at https://github.com/jenkinsci/workflow-api-plugin/pull/40 - I'm stuck on figuring out how to actually propagate up to the enclosing blocks, and have been for a few days, so I'm giving up on figuring it out solo and begging for help. =)
            Hide
            jamesdumay James Dumay added a comment - - edited

            Hi everyone,

            I am aware that this ticket is valued by many people. However, on review there are too many disparate use cases and problems attached to this issue that are already described elsewhere.

            This issue means too many things for too many people and makes it unsatisfiable with a single change. The appropriate action here is to close this ticket and ask watchers to watch and vote for a specific ticket that will solve their problem.

            If you are interested in these problems, please watch and vote for the following:

            • JENKINS-39203 - The entire pipeline gets marked as unstable rather than the stage or parallel where the unstable status was set.
            • JENKINS-45579 - Capability to mark a stage as "failed" with message and have the pipeline continue executing.
            • JENKINS-43995 - Infrastructure to allow any step to have its own distinct status. This enables the tickets listed above.

            I may have missed a few use cases. If that is the case:

            • Search JIRA for an issue matching your use case. If you find one, link it in the comments and I will update the ticket. Be sure to leave a comment on the linked ticket describing the problem you are trying to solve and then your desired solution to the problem.
            • If you cannot find an existing issue, raise a new one and comment here. Leave a comment on your new ticket describing problem you are trying to solve and then your desired solution to the problem.

            Thanks,

            Show
            jamesdumay James Dumay added a comment - - edited Hi everyone, I am aware that this ticket is valued by many people. However, on review there are too many disparate use cases and problems attached to this issue that are already described elsewhere. This issue means too many things for too many people and makes it unsatisfiable with a single change. The appropriate action here is to close this ticket and ask watchers to watch and vote for a specific ticket that will solve their problem. If you are interested in these problems, please watch and vote for the following: JENKINS-39203 - The entire pipeline gets marked as unstable rather than the stage or parallel where the unstable status was set. JENKINS-45579 - Capability to mark a stage as "failed" with message and have the pipeline continue executing. JENKINS-43995 - Infrastructure to allow any step to have its own distinct status. This enables the tickets listed above. I may have missed a few use cases. If that is the case: Search JIRA for an issue matching your use case. If you find one, link it in the comments and I will update the ticket. Be sure to leave a comment on the linked ticket describing the problem you are trying to solve and then your desired solution to the problem. If you cannot find an existing issue, raise a new one and comment here. Leave a comment on your new ticket describing problem you are trying to solve and then your desired solution to the problem. Thanks,
            Hide
            dageissl Daniel Geißler added a comment -

            James Dumay the tickets you linked seem to focus on the state of stages, but as I read the original ticket it was also about additional information by each stage (not only if a stage fails or is unstable, but also if everything went well).

            This aspect seems to only vaguely covered by JENKINS-45579 with the message.
            Should one open a new Request for this or will it be covered in one of the menitoned.

            Show
            dageissl Daniel Geißler added a comment - James Dumay the tickets you linked seem to focus on the state of stages, but as I read the original ticket it was also about additional information by each stage (not only if a stage fails or is unstable, but also if everything went well). This aspect seems to only vaguely covered by JENKINS-45579 with the message. Should one open a new Request for this or will it be covered in one of the menitoned.
            Hide
            jamesdumay James Dumay added a comment -

            Daniel Geißler do you mean that you want to be able to attach a "note" to any stage or parallel? If so, please raise another ticket for that use cases and I can link it here.

            Show
            jamesdumay James Dumay added a comment - Daniel Geißler do you mean that you want to be able to attach a "note" to any stage or parallel? If so, please raise another ticket for that use cases and I can link it here.
            Hide
            dageissl Daniel Geißler added a comment -

            James Dumay I've opened JENKINS-46170 hope this coveres all the intentions.

            Show
            dageissl Daniel Geißler added a comment - James Dumay I've opened  JENKINS-46170 hope this coveres all the intentions.
            Hide
            macdrega Joerg Schwaerzler added a comment -

            Looks like the right initiative to close that one.

            However I now wonder whether there's any ticket out there requesting to visualize stages inside parallel blocks using Blue Ocean. Can anyone tell? Didn't find anything so far but I doubt that I'm the only one who would make use of it.

            Thanks.

            Show
            macdrega Joerg Schwaerzler added a comment - Looks like the right initiative to close that one. However I now wonder whether there's any ticket out there requesting to visualize stages inside parallel blocks using Blue Ocean. Can anyone tell? Didn't find anything so far but I doubt that I'm the only one who would make use of it. Thanks.
            Hide
            jamesdumay James Dumay added a comment -

            Joerg Schwaerzler you are looking for JENKINS-38442 but its unlikely to be implemented at this time (we can't find a sane way to visualise it). Would you mind drawing a picture of your ideal visualisation and a mock Jenkinsfile, to help us figure out what you want? (attach and comment to JENKINS-38442)

            Show
            jamesdumay James Dumay added a comment - Joerg Schwaerzler you are looking for JENKINS-38442 but its unlikely to be implemented at this time (we can't find a sane way to visualise it). Would you mind drawing a picture of your ideal visualisation and a mock Jenkinsfile, to help us figure out what you want? (attach and comment to JENKINS-38442 )
            Hide
            macdrega Joerg Schwaerzler added a comment -

            Thanks for the quick answer - found that one already.

            I totally understand that nested stages are hard to visualize. I only need a small part of it: The only thing I need is just stages inside parallel blocks - without nesting one stage inside another stage, something like:

            That should be easy to visualize.

            Probably I'll create a new ticket hoping that it won't be closed in favor of  JENKINS-38442.

            Show
            macdrega Joerg Schwaerzler added a comment - Thanks for the quick answer - found that one already. I totally understand that nested stages are hard to visualize. I only need a small part of it: The only thing I need is just stages inside parallel blocks - without nesting one stage inside another stage, something like: That should be easy to visualize. Probably I'll create a new ticket hoping that it won't be closed in favor of   JENKINS-38442 .
            Hide
            jamesdumay James Dumay added a comment -

            Mind posting this on the other ticket so that it's captured in the right place?

            Show
            jamesdumay James Dumay added a comment - Mind posting this on the other ticket so that it's captured in the right place?
            Hide
            macdrega Joerg Schwaerzler added a comment -

            Sorry my fault . Post is already done now.

            Show
            macdrega Joerg Schwaerzler added a comment - Sorry my fault . Post is already done now.
            Hide
            jamesdumay James Dumay added a comment -

            Thanks

            Show
            jamesdumay James Dumay added a comment - Thanks

              People

              • Assignee:
                Unassigned
                Reporter:
                directhex Jo Shields
              • Votes:
                115 Vote for this issue
                Watchers:
                165 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: