Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27395

distinguish test reports filed from different parts of workflow

    Details

    • Sprint:
      Pipeline - July/August, Pipeline - December
    • Similar Issues:

      Description

      JUnit test result archiving can happen multiple times inside workflow, and currently there's no way to meaningfully distinguish multiple invocations.

        Attachments

          Issue Links

            Activity

            kohsuke Kohsuke Kawaguchi created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-26522 [ JENKINS-26522 ]
            Hide
            rsandell rsandell added a comment -

            I didn't notice this ticket until after I wrote https://github.com/jenkinsci/junit-plugin/pull/27 and https://github.com/jenkinsci/parallel-test-executor-plugin/pull/16

            My suggested approach doesn't provide any visual difference in the test report, since suite names isn't rendered anywhere iiuc. But it serves a way for other parts of the flow to use that info, like the parallel test executor.

            Show
            rsandell rsandell added a comment - I didn't notice this ticket until after I wrote https://github.com/jenkinsci/junit-plugin/pull/27 and https://github.com/jenkinsci/parallel-test-executor-plugin/pull/16 My suggested approach doesn't provide any visual difference in the test report, since suite names isn't rendered anywhere iiuc. But it serves a way for other parts of the flow to use that info, like the parallel test executor.
            Hide
            jglick Jesse Glick added a comment -

            One possibility is to look up the nearest enclosing LabelAction and automatically include that in the test result metadata. Thus running JUnitResultArchiver inside label (JENKINS-26107) or a parallel branch would suffice to differentiate ambiguous test names.

            Rendering results in a matrix requires more information. Perhaps the convention could be that a label in a format acceptable to hudson.matrix.Combination.fromString (for example, os=win,jdk=7) would be eligible for display by any supporting visualizations in a matrix cell.

            Show
            jglick Jesse Glick added a comment - One possibility is to look up the nearest enclosing LabelAction and automatically include that in the test result metadata. Thus running JUnitResultArchiver inside label ( JENKINS-26107 ) or a parallel branch would suffice to differentiate ambiguous test names. Rendering results in a matrix requires more information. Perhaps the convention could be that a label in a format acceptable to hudson.matrix.Combination.fromString (for example, os=win,jdk=7 ) would be eligible for display by any supporting visualizations in a matrix cell.
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-26107 [ JENKINS-26107 ]
            Hide
            svanoort Sam Van Oort added a comment - - edited

            The other possibility I see is to attach a new TestAction to the FlowNode executing/archiving the tests (preferably something that is general enough to apply beyond JUnit), which includes some metadata about tests & parameters. When scanning the flowgraph to generate UI rendering, nodes with these actions can be discovered and aggregated to the appropriate stage/labelled block/whatever.

            This has a couple advantages:

            1. It's simple to implement and fast at runtime, since only the node in question is touched, and you don't need to scan the FlowGraph every time to get the enclosing label/stage (a process than can be unpredictable slow and can trigger significant I/O)
            2. It promotes JUnit/Test result archiving to a proper first-class pipeline step
            3. Makes it easy to tie JUnit results to a specific point in time for a pipeline, with all the traceability that implies (i.e. ability to map out the steps preceding it, see the node label used, etc)
            Show
            svanoort Sam Van Oort added a comment - - edited The other possibility I see is to attach a new TestAction to the FlowNode executing/archiving the tests (preferably something that is general enough to apply beyond JUnit), which includes some metadata about tests & parameters. When scanning the flowgraph to generate UI rendering, nodes with these actions can be discovered and aggregated to the appropriate stage/labelled block/whatever. This has a couple advantages: It's simple to implement and fast at runtime, since only the node in question is touched, and you don't need to scan the FlowGraph every time to get the enclosing label/stage (a process than can be unpredictable slow and can trigger significant I/O) It promotes JUnit/Test result archiving to a proper first-class pipeline step Makes it easy to tie JUnit results to a specific point in time for a pipeline, with all the traceability that implies (i.e. ability to map out the steps preceding it, see the node label used, etc)
            Hide
            abayer Andrew Bayer added a comment -

            Yeah, I'd like to see something significantly more generic than just junit - or just test reports, for that matter. Artifacts, static analysis, etc all would be applicable - basically, any Recorder.

            Show
            abayer Andrew Bayer added a comment - Yeah, I'd like to see something significantly more generic than just junit - or just test reports, for that matter. Artifacts, static analysis, etc all would be applicable - basically, any Recorder .
            Hide
            jglick Jesse Glick added a comment -

            It is an interesting proposal but it has one major disadvantage: that it requires Pipeline-specific code in the junit plugin, or some new bridge plugin. The more straightforward proposal, which is just to archive test results but include an associated label (as in rsandell’s original patch), requires only minor modifications to junit. Ditto for other recorders.

            I do not see any reason to think performance is a consideration here.

            Making test archiving be a first-class step is already tracked in JENKINS-34148, which would be useful for example for scripts which want to inspect results without necessarily changing the overall build result, but I not think that is really what we need here. There is basic UI in the junit plugin which is responsible for displaying test results, and this has a corresponding API which a lot of other plugins need to refer to. Making a Pipeline-specific storage system and API would allow the stage view to show test results, with a total fork of the (rather complex!) JUnit UI, but would throw out integration with any of these others. This does not feel like a maintainable direction.

            Show
            jglick Jesse Glick added a comment - It is an interesting proposal but it has one major disadvantage: that it requires Pipeline-specific code in the junit plugin, or some new bridge plugin. The more straightforward proposal, which is just to archive test results but include an associated label (as in rsandell ’s original patch), requires only minor modifications to junit . Ditto for other recorders. I do not see any reason to think performance is a consideration here. Making test archiving be a first-class step is already tracked in JENKINS-34148 , which would be useful for example for scripts which want to inspect results without necessarily changing the overall build result, but I not think that is really what we need here. There is basic UI in the junit plugin which is responsible for displaying test results, and this has a corresponding API which a lot of other plugins need to refer to. Making a Pipeline-specific storage system and API would allow the stage view to show test results, with a total fork of the (rather complex!) JUnit UI, but would throw out integration with any of these others. This does not feel like a maintainable direction.
            Hide
            abayer Andrew Bayer added a comment -

            Note that one way or the other, we'll need to make changes to Junit and other similar plugins to make sure they don't just store all their data in a single per-run hardcoded file.

            Show
            abayer Andrew Bayer added a comment - Note that one way or the other, we'll need to make changes to Junit and other similar plugins to make sure they don't just store all their data in a single per-run hardcoded file.
            Hide
            svanoort Sam Van Oort added a comment -

            To capture discussion with Andrew Bayer my proposal has been revised some: currently I think the best solution is to create a new Step for JUnit or other test results, but not attach an action to the FlowNode generated by this step exection. Instead, when run, the step will attach a reference to the FlowNodeId in the TestResult. Thus, each test result will be assignable to the exact point within the run where it was generated.

            Implementation-side, this is what I'm thinking:

            1. New step (not just a wrapper) for JUnit result archiving.
            2. The step.run method in its execution will invoke a new version of the JUnitResultArchiver.perform method, which takes a (nullable) FlowNodeId parameter or similar.
            3. The existing API will be modified to invoke the new API using a null FlowNodeId.
            4. The test results will get a (nullable) field for the flowNodeId, probably on the TestAction or TestResult action. This allows us to associate the result with the time of generation.
            5. The new JUnitResultArchiver perform method will set this flowNodeId field to store the information.
            6. For non-pipeline builds, or archived results before this, we need to handle null values (and allow serializing/deserializing without the field set).
            7. If we can find a good way, JUnitResultArchiver may will to look to see if it is inside a pipeline run and attach this field when the old perform API is called, by looking at the current heads for a FlowNode that might be an achiving step.

            UI representation here TBD.

            It is possibly worth calling the flowNodeId field something else, to allow for other uses and not make it 100% pipeline specific. I am not sure.

            Show
            svanoort Sam Van Oort added a comment - To capture discussion with Andrew Bayer my proposal has been revised some: currently I think the best solution is to create a new Step for JUnit or other test results, but not attach an action to the FlowNode generated by this step exection. Instead, when run, the step will attach a reference to the FlowNodeId in the TestResult. Thus, each test result will be assignable to the exact point within the run where it was generated. Implementation-side, this is what I'm thinking: 1. New step (not just a wrapper) for JUnit result archiving. 2. The step.run method in its execution will invoke a new version of the JUnitResultArchiver.perform method, which takes a (nullable) FlowNodeId parameter or similar. 3. The existing API will be modified to invoke the new API using a null FlowNodeId. 4. The test results will get a (nullable) field for the flowNodeId, probably on the TestAction or TestResult action. This allows us to associate the result with the time of generation. 5. The new JUnitResultArchiver perform method will set this flowNodeId field to store the information. 6. For non-pipeline builds, or archived results before this, we need to handle null values (and allow serializing/deserializing without the field set). 7. If we can find a good way, JUnitResultArchiver may will to look to see if it is inside a pipeline run and attach this field when the old perform API is called, by looking at the current heads for a FlowNode that might be an achiving step. UI representation here TBD. It is possibly worth calling the flowNodeId field something else, to allow for other uses and not make it 100% pipeline specific. I am not sure.
            Hide
            svanoort Sam Van Oort added a comment -

            Jesse Glick You may want to see the updated comment (since your comment applies to the old plan); I think with the revised proposal, the only parts that require pipeline dependencies are the step itself, and if we choose to automatically store the flowNodeId when the JUnitResultArchiver is invoked with the original perform API (such as in some current scripts).

            I think the PR that rsandell put forth to include an archiveId has a lot in common with this (although the internals seem to differ a bit).

            Show
            svanoort Sam Van Oort added a comment - Jesse Glick You may want to see the updated comment (since your comment applies to the old plan); I think with the revised proposal, the only parts that require pipeline dependencies are the step itself, and if we choose to automatically store the flowNodeId when the JUnitResultArchiver is invoked with the original perform API (such as in some current scripts). I think the PR that rsandell put forth to include an archiveId has a lot in common with this (although the internals seem to differ a bit).
            Hide
            lostgoat Andres Rodriguez added a comment - - edited

            For anyone interested in this functionality, here is a small hack I use to achieve something similar:

            def setJUnitPackageName(packageName, inFile, outFile) {
                // Prepend the testName as the package name so that we get sorted output in the
                // Jenkins test results page. The packageName is extracted from the classname
                // attribute of the testcase tag.
                // WARNING: the package attribute of the testcase tag is igonred by Jenkins
                sh "sed \"s/\\(classname=['\\\"]\\)/\\1${packageName}./g\" ${inFile} > ${outFile}"
            }
            

            With this you can set packageName of a junit file to a string of your choice, we usually use "test-name-platform" for example.

            Show
            lostgoat Andres Rodriguez added a comment - - edited For anyone interested in this functionality, here is a small hack I use to achieve something similar: def setJUnitPackageName(packageName, inFile, outFile) { // Prepend the testName as the package name so that we get sorted output in the // Jenkins test results page. The packageName is extracted from the classname // attribute of the testcase tag. // WARNING: the package attribute of the testcase tag is igonred by Jenkins sh "sed \" s/\\(classname=['\\\ "]\\)/\\1${packageName}./g\" ${inFile} > ${outFile}" } With this you can set packageName of a junit file to a string of your choice, we usually use "test-name-platform" for example.
            jglick Jesse Glick made changes -
            Epic Link JENKINS-35400 [ 171193 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 161600 ] JNJira + In-Review [ 180759 ]
            jamesdumay James Dumay made changes -
            Labels workflow blueocean workflow
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "Mailing list (Web Link)" [ 14747 ]
            abayer Andrew Bayer made changes -
            Labels blueocean workflow blueocean pipeline workflow
            abayer Andrew Bayer made changes -
            Labels blueocean pipeline workflow blueocean pipeline
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-36932 [ JENKINS-36932 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-39203 [ JENKINS-39203 ]
            jamesdumay James Dumay made changes -
            Link This issue is duplicated by JENKINS-39203 [ JENKINS-39203 ]
            abayer Andrew Bayer made changes -
            Link This issue is blocking JENKINS-41031 [ JENKINS-41031 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-41710 [ JENKINS-41710 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-41855 [ JENKINS-41855 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-39353 [ JENKINS-39353 ]
            Hide
            reinholdfuereder Reinhold Füreder added a comment -

            Some thoughts:

            • Should all test results from potentially (or now with Jenkins2/Pipeline very likely) different test types (unit vs. integration vs. acceptance tests etc) really be merged into one single test result like it seems to be done now?
            • Other CI Servers allow (i.e. this is optional) giving each test result publication a dedicated title name (e.g. "Unit Tests" vs. "Firefox99 Acceptance Tests"): would it make sense to merge only based on test result publication title and show also stats and graphs per that (test result publication title)?
            • (I think a test report filename based test result publication title naming approach is not nice)
            Show
            reinholdfuereder Reinhold Füreder added a comment - Some thoughts: Should all test results from potentially (or now with Jenkins2/Pipeline very likely) different test types (unit vs. integration vs. acceptance tests etc) really be merged into one single test result like it seems to be done now? Other CI Servers allow (i.e. this is optional) giving each test result publication a dedicated title name (e.g. "Unit Tests" vs. "Firefox99 Acceptance Tests"): would it make sense to merge only based on test result publication title and show also stats and graphs per that (test result publication title)? (I think a test report filename based test result publication title naming approach is not nice)
            Hide
            jglick Jesse Glick added a comment -

            PR 64 is the most recent proposal, but I think it is more a starting point than a complete fix.

            Show
            jglick Jesse Glick added a comment - PR 64 is the most recent proposal, but I think it is more a starting point than a complete fix.
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 64 (Web Link)" [ 17207 ]
            abayer Andrew Bayer made changes -
            Link This issue relates to JENKINS-40986 [ JENKINS-40986 ]
            jamesdumay James Dumay made changes -
            Labels blueocean pipeline blueocean cloudbees-internal-pipeline pipeline
            abayer Andrew Bayer made changes -
            Assignee Andrew Bayer [ abayer ]
            abayer Andrew Bayer made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            abayer Andrew Bayer added a comment -

            Veeeerrrry preliminary work in progress PR up at https://github.com/jenkinsci/junit-plugin/pull/75 - it subsumes https://github.com/jenkinsci/junit-plugin/pull/64, and isn't anywhere near ready yet, but I wanted to get the initial work up for discussion as I move forward.

            Show
            abayer Andrew Bayer added a comment - Veeeerrrry preliminary work in progress PR up at https://github.com/jenkinsci/junit-plugin/pull/75 - it subsumes https://github.com/jenkinsci/junit-plugin/pull/64 , and isn't anywhere near ready yet, but I wanted to get the initial work up for discussion as I move forward.
            abayer Andrew Bayer made changes -
            Remote Link This issue links to "PR #75 (Web Link)" [ 17333 ]
            abayer Andrew Bayer made changes -
            Remote Link This issue links to "PR #75 (Web Link)" [ 17333 ]
            Hide
            abayer Andrew Bayer added a comment -

            Ditched the first PR that built on top of #64, which I wasn't happy with, in favor of https://github.com/jenkinsci/junit-plugin/pull/76.

            Show
            abayer Andrew Bayer added a comment - Ditched the first PR that built on top of #64, which I wasn't happy with, in favor of https://github.com/jenkinsci/junit-plugin/pull/76 .
            abayer Andrew Bayer made changes -
            Remote Link This issue links to "PR #76 (Web Link)" [ 17334 ]
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            Andrew Bayer will your enhancement support the case where one single step generates multiple test reports that we want to associate with different names?
            I have in mind a multi-module Maven project for which I would like to have per module test reports: a single "sh 'mvn test' " step would generate multiple test reports that I want to attach with their maven module name/identifier.

            See comment https://github.com/jenkinsci/junit-plugin/pull/64#issuecomment-277625409

            Show
            cleclerc Cyrille Le Clerc added a comment - Andrew Bayer will your enhancement support the case where one single step generates multiple test reports that we want to associate with different names? I have in mind a multi-module Maven project for which I would like to have per module test reports: a single "sh 'mvn test' " step would generate multiple test reports that I want to attach with their maven module name/identifier. See comment https://github.com/jenkinsci/junit-plugin/pull/64#issuecomment-277625409
            Hide
            abayer Andrew Bayer added a comment -

            Cyrille Le Clerc So sh 'mvn test' wouldn't actually generate any test reports in the job, either now or in the future. It's the junit (or my proposed junitResults step) invocation that does that. So if, with my proposal, you ran sh 'mvn test' and then junitResults "first-module/target/surefire-reports/*/.xml" and junitResults "second-module/target/surefire-reports/*/.xml", that'd be two junitResults invocations, so whatever's displaying the tests could say "Ok, what are the test results for this step or this stage?" and get the individual execution's results. But yeah, overall, I've decided to separate this from JENKINS-41710 - they're different matters.

            Show
            abayer Andrew Bayer added a comment - Cyrille Le Clerc So sh 'mvn test' wouldn't actually generate any test reports in the job, either now or in the future. It's the junit (or my proposed junitResults step) invocation that does that. So if, with my proposal, you ran sh 'mvn test' and then junitResults "first-module/target/surefire-reports/* / .xml" and junitResults "second-module/target/surefire-reports/* / .xml" , that'd be two junitResults invocations, so whatever's displaying the tests could say "Ok, what are the test results for this step or this stage?" and get the individual execution's results. But yeah, overall, I've decided to separate this from JENKINS-41710 - they're different matters.
            abayer Andrew Bayer made changes -
            Sprint Pipeline - July/August [ 371 ]
            abayer Andrew Bayer made changes -
            Epic Link JENKINS-35400 [ 171193 ] JENKINS-45818 [ 184028 ]
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            Andrew Bayer I forgot to mention that I publish the test reports with the withMaven plugin, in org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback.TailCall#finished (here). See sample below.

            This means that a single step publishes multiple test reports.

            node {
                stage ('Build') {
                    git "https://github.com/cyrille-leclerc/multi-module-maven-project.git"
                    withMaven(maven: 'maven-3.3.9') {
                       sh "mvn clean package"
                    }
                }
            }
            

            > But yeah, overall, I've decided to separate this from JENKINS-41710 - they're different matters.

            Does it mean that https://github.com/jenkinsci/junit-plugin/pull/64 is no longer blocked and can be merged?

            Show
            cleclerc Cyrille Le Clerc added a comment - Andrew Bayer I forgot to mention that I publish the test reports with the withMaven plugin, in org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback.TailCall#finished ( here ). See sample below. This means that a single step publishes multiple test reports. node { stage ( 'Build' ) { git "https: //github.com/cyrille-leclerc/multi-module-maven-project.git" withMaven(maven: 'maven-3.3.9' ) { sh "mvn clean package " } } } > But yeah, overall, I've decided to separate this from JENKINS-41710 - they're different matters. Does it mean that https://github.com/jenkinsci/junit-plugin/pull/64 is no longer blocked and can be merged?
            Hide
            abayer Andrew Bayer added a comment -

            This is orthogonal to https://github.com/jenkinsci/junit-plugin/pull/64. And it would have no impact on the withMaven step at all, unless/until you change it to use the static method introduced for parsing and attaching test results rather than https://github.com/jenkinsci/pipeline-maven-plugin/blob/2eba9c8ac50f933567d6b3775ca4b3372e9f1d50/jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java#L268.

            Show
            abayer Andrew Bayer added a comment - This is orthogonal to https://github.com/jenkinsci/junit-plugin/pull/64 . And it would have no impact on the withMaven step at all, unless/until you change it to use the static method introduced for parsing and attaching test results rather than https://github.com/jenkinsci/pipeline-maven-plugin/blob/2eba9c8ac50f933567d6b3775ca4b3372e9f1d50/jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java#L268 .
            abayer Andrew Bayer made changes -
            Link This issue is blocking JENKINS-45826 [ JENKINS-45826 ]
            Hide
            abayer Andrew Bayer added a comment -

            XUnit plugin PR up at https://github.com/jenkinsci/xunit-plugin/pull/49 that builds on top of the JUnit one...

            Show
            abayer Andrew Bayer added a comment - XUnit plugin PR up at https://github.com/jenkinsci/xunit-plugin/pull/49 that builds on top of the JUnit one...
            abayer Andrew Bayer made changes -
            Remote Link This issue links to "xunit PR #49 (Web Link)" [ 17344 ]
            abayer Andrew Bayer made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            cleclerc Cyrille Le Clerc added a comment -

            Andrew Bayer thanks for the explanation, I'll try my best to contribute to PR-64 and make it happen.

            Show
            cleclerc Cyrille Le Clerc added a comment - Andrew Bayer thanks for the explanation, I'll try my best to contribute to PR-64 and make it happen.
            michaelneale Michael Neale made changes -
            Link This issue is blocked by JENKINS-45923 [ JENKINS-45923 ]
            jamesdumay James Dumay made changes -
            Link This issue relates to JENKINS-46166 [ JENKINS-46166 ]
            Hide
            jamesdumay James Dumay added a comment - - edited

            Hi all,

            We have a design that will differentiate tests across different stages and parallels for Blue Ocean as part of JENKINS-46166.

            Please take a look and let us know what you think.

            Thanks
            James

            Show
            jamesdumay James Dumay added a comment - - edited Hi all, We have a design that will differentiate tests across different stages and parallels for Blue Ocean as part of JENKINS-46166 . Please take a look and let us know what you think. Thanks James
            Hide
            abayer Andrew Bayer added a comment -

            Added a PR to pipeline-maven (https://github.com/jenkinsci/pipeline-maven-plugin/pull/82) that uses the junit changes as well.

            Show
            abayer Andrew Bayer added a comment - Added a PR to pipeline-maven ( https://github.com/jenkinsci/pipeline-maven-plugin/pull/82 ) that uses the junit changes as well.
            abayer Andrew Bayer made changes -
            Remote Link This issue links to "pipeline-maven PR #82 (Web Link)" [ 17501 ]
            teilo James Nord made changes -
            Component/s parallel-test-executor-plugin [ 17650 ]
            Hide
            jglick Jesse Glick added a comment -

            Andrew Bayer I have not been following details of what you are doing, but you need to verify that parallel-test-executor is able to take advantage of this metadata—it is a critical use case.

            Show
            jglick Jesse Glick added a comment - Andrew Bayer I have not been following details of what you are doing, but you need to verify that parallel-test-executor is able to take advantage of this metadata—it is a critical use case.
            Hide
            abayer Andrew Bayer added a comment -

            Looks like parallel-test-executor will need some work to be able to make any use of the additional data - theoretically, it could be reworked to, say, find just the tests for a specific stage in the previous build, which would work pretty easily.

            Show
            abayer Andrew Bayer added a comment - Looks like parallel-test-executor will need some work to be able to make any use of the additional data - theoretically, it could be reworked to, say, find just the tests for a specific stage in the previous build, which would work pretty easily.
            Hide
            jglick Jesse Glick added a comment -

            theoretically, it could be reworked to, say, find just the tests for a specific stage in the previous build

            Theory becomes practice when you file the PR.

            Show
            jglick Jesse Glick added a comment - theoretically, it could be reworked to, say, find just the tests for a specific stage in the previous build Theory becomes practice when you file the PR.
            Hide
            abayer Andrew Bayer added a comment -

            Indeed. I've started playing around with it. Trying to figure out the best way to specify a parallel branch within a stage (or possibly an arbitrarily deep nest of stages/parallel branches) at the moment. I might wait until Sam Van Oort has nailed down more performant querying for enclosing flow nodes (as discussed on the junit PR) and instead have an option that lets you just say "use the tests from this stage/branch in the previous build" without having to specify the full "path" directly.

            Show
            abayer Andrew Bayer added a comment - Indeed. I've started playing around with it. Trying to figure out the best way to specify a parallel branch within a stage (or possibly an arbitrarily deep nest of stages/parallel branches) at the moment. I might wait until Sam Van Oort has nailed down more performant querying for enclosing flow nodes (as discussed on the junit PR) and instead have an option that lets you just say "use the tests from this stage/branch in the previous build" without having to specify the full "path" directly.
            jglick Jesse Glick made changes -
            Link This issue is blocking JENKINS-35420 [ JENKINS-35420 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/graph/AtomNode.java
            src/main/java/org/jenkinsci/plugins/workflow/graph/BlockEndNode.java
            src/main/java/org/jenkinsci/plugins/workflow/graph/BlockStartNode.java
            src/main/java/org/jenkinsci/plugins/workflow/graph/FlowNode.java
            src/test/java/org/jenkinsci/plugins/workflow/graph/FlowNodeTest.java
            http://jenkins-ci.org/commit/workflow-api-plugin/3e8a1b79b07936131098267664f441335318d4e1
            Log:
            JENKINS-27395 Start recording enclosing blocks for flow nodes

            Specifically for AtomNodes, BlockStartNodes, and BlockEndNodes.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: src/main/java/org/jenkinsci/plugins/workflow/graph/AtomNode.java src/main/java/org/jenkinsci/plugins/workflow/graph/BlockEndNode.java src/main/java/org/jenkinsci/plugins/workflow/graph/BlockStartNode.java src/main/java/org/jenkinsci/plugins/workflow/graph/FlowNode.java src/test/java/org/jenkinsci/plugins/workflow/graph/FlowNodeTest.java http://jenkins-ci.org/commit/workflow-api-plugin/3e8a1b79b07936131098267664f441335318d4e1 Log: JENKINS-27395 Start recording enclosing blocks for flow nodes Specifically for AtomNodes, BlockStartNodes, and BlockEndNodes.
            Hide
            svanoort Sam Van Oort added a comment -

            Andrew Bayer  Not sure exactly what you're trying to do?  

            I had a semi-performant impl for fetching nodes within a block (using something I put into Bismuth ages ago but hadn't fully taken advantage of).  Removed it because no downstream consumer yet but I can easily put it back. 

            Only thing is that in the face of parallels the order of nodes wrt branches is pretty ill defined.

            Show
            svanoort Sam Van Oort added a comment - Andrew Bayer   Not sure exactly what you're trying to do?   I had a semi-performant impl for fetching nodes within a block (using something I put into Bismuth ages ago but hadn't fully taken advantage of).  Removed it because no downstream consumer yet but I can easily put it back.  Only thing is that in the face of parallels the order of nodes wrt branches is pretty ill defined.
            Hide
            abayer Andrew Bayer added a comment -

            I...totally didn't do anything?

            Show
            abayer Andrew Bayer added a comment - I...totally didn't do anything?
            Hide
            svanoort Sam Van Oort added a comment -

            Andrew Bayer I think the upstreams are now unblocked for you when you get back online. 

            Show
            svanoort Sam Van Oort added a comment - Andrew Bayer I think the upstreams are now unblocked for you when you get back online. 
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            pom.xml
            src/main/java/hudson/tasks/junit/JUnitParser.java
            src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
            src/main/java/hudson/tasks/junit/JUnitTask.java
            src/main/java/hudson/tasks/junit/SuiteResult.java
            src/main/java/hudson/tasks/junit/TestResult.java
            src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStep.java
            src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java
            src/main/java/hudson/tasks/test/TestResultParser.java
            src/test/java/hudson/tasks/junit/JUnitParserTest.java
            src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java
            src/test/java/hudson/tasks/junit/SuiteResult2Test.java
            src/test/java/hudson/tasks/junit/SuiteResultTest.java
            src/test/java/hudson/tasks/junit/TestResultTest.java
            src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java
            src/test/resources/hudson/tasks/junit/JUnitResultArchiverTest.zip
            src/test/resources/hudson/tasks/junit/JUnitResultArchiverTest/testData.zip
            http://jenkins-ci.org/commit/junit-plugin/375f91d78da541fec0f095d996619d5e7cdfc18f
            Log:
            JENKINS-27395 Add a junitResults step

            This uses the run's externalizable ID and the step's FlowNode.getId()
            together as a unique key to track which SuiteResults are associated
            with a particular execution, and allows getting a TestResult object
            from a run ID and 1 or more node IDs passed to an existing TestResult
            object that contains suites for that run/those nodes.

            Note that JUnitResultArchiverTest.zip has been moved around and
            renamed due to
            https://github.com/jenkinsci/jenkins-test-harness/commit/79d9603370d736fa1ede35e78b5f4c421cf2d5be
            breaking tests otherwise, since the JUnitResultArchiveTest/ directory
            will end up getting picked up as JENKINS_HOME otherwise.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pom.xml src/main/java/hudson/tasks/junit/JUnitParser.java src/main/java/hudson/tasks/junit/JUnitResultArchiver.java src/main/java/hudson/tasks/junit/JUnitTask.java src/main/java/hudson/tasks/junit/SuiteResult.java src/main/java/hudson/tasks/junit/TestResult.java src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStep.java src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java src/main/java/hudson/tasks/test/TestResultParser.java src/test/java/hudson/tasks/junit/JUnitParserTest.java src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java src/test/java/hudson/tasks/junit/SuiteResult2Test.java src/test/java/hudson/tasks/junit/SuiteResultTest.java src/test/java/hudson/tasks/junit/TestResultTest.java src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java src/test/resources/hudson/tasks/junit/JUnitResultArchiverTest.zip src/test/resources/hudson/tasks/junit/JUnitResultArchiverTest/testData.zip http://jenkins-ci.org/commit/junit-plugin/375f91d78da541fec0f095d996619d5e7cdfc18f Log: JENKINS-27395 Add a junitResults step This uses the run's externalizable ID and the step's FlowNode.getId() together as a unique key to track which SuiteResults are associated with a particular execution, and allows getting a TestResult object from a run ID and 1 or more node IDs passed to an existing TestResult object that contains suites for that run/those nodes. Note that JUnitResultArchiverTest.zip has been moved around and renamed due to https://github.com/jenkinsci/jenkins-test-harness/commit/79d9603370d736fa1ede35e78b5f4c421cf2d5be breaking tests otherwise, since the JUnitResultArchiveTest/ directory will end up getting picked up as JENKINS_HOME otherwise.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            Jenkinsfile
            pom.xml
            src/main/java/hudson/tasks/junit/CaseResult.java
            src/main/java/hudson/tasks/junit/JUnitParser.java
            src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
            src/main/java/hudson/tasks/junit/JUnitTask.java
            src/main/java/hudson/tasks/junit/SuiteResult.java
            src/main/java/hudson/tasks/junit/TestResult.java
            src/main/java/hudson/tasks/junit/TestResultAction.java
            src/main/java/hudson/tasks/junit/TestResultSummary.java
            src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStep.java
            src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java
            src/main/java/hudson/tasks/test/PipelineBlockWithTests.java
            src/main/java/hudson/tasks/test/TabulatedResult.java
            src/main/java/hudson/tasks/test/TestResultParser.java
            src/main/resources/hudson/tasks/junit/pipeline/JUnitResultsStep/config.jelly
            src/test/java/hudson/tasks/junit/JUnitParserTest.java
            src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java
            src/test/java/hudson/tasks/junit/SuiteResult2Test.java
            src/test/java/hudson/tasks/junit/SuiteResultTest.java
            src/test/java/hudson/tasks/junit/TestResultTest.java
            src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java
            src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-first-1.xml
            src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-first-2.xml
            src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-first-3.xml
            src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-second-1.xml
            src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-second-2.xml
            src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-second-3.xml
            http://jenkins-ci.org/commit/junit-plugin/5ae9ec10480d92075f03944462d8a90c8755ed86
            Log:
            Merge pull request #76 from abayer/jenkins-27395-mk2

            JENKINS-27395 Add a junitResults step

            Compare: https://github.com/jenkinsci/junit-plugin/compare/187cb92178f0...5ae9ec10480d

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: Jenkinsfile pom.xml src/main/java/hudson/tasks/junit/CaseResult.java src/main/java/hudson/tasks/junit/JUnitParser.java src/main/java/hudson/tasks/junit/JUnitResultArchiver.java src/main/java/hudson/tasks/junit/JUnitTask.java src/main/java/hudson/tasks/junit/SuiteResult.java src/main/java/hudson/tasks/junit/TestResult.java src/main/java/hudson/tasks/junit/TestResultAction.java src/main/java/hudson/tasks/junit/TestResultSummary.java src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStep.java src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java src/main/java/hudson/tasks/test/PipelineBlockWithTests.java src/main/java/hudson/tasks/test/TabulatedResult.java src/main/java/hudson/tasks/test/TestResultParser.java src/main/resources/hudson/tasks/junit/pipeline/JUnitResultsStep/config.jelly src/test/java/hudson/tasks/junit/JUnitParserTest.java src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java src/test/java/hudson/tasks/junit/SuiteResult2Test.java src/test/java/hudson/tasks/junit/SuiteResultTest.java src/test/java/hudson/tasks/junit/TestResultTest.java src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-first-1.xml src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-first-2.xml src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-first-3.xml src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-second-1.xml src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-second-2.xml src/test/resources/hudson/tasks/junit/pipeline/junit-report-testTrends-second-3.xml http://jenkins-ci.org/commit/junit-plugin/5ae9ec10480d92075f03944462d8a90c8755ed86 Log: Merge pull request #76 from abayer/jenkins-27395-mk2 JENKINS-27395 Add a junitResults step Compare: https://github.com/jenkinsci/junit-plugin/compare/187cb92178f0...5ae9ec10480d
            Hide
            abayer Andrew Bayer added a comment -

            Releasing as 1.22-beta-1 - plan is to follow that up with an xunit beta and a parallel-test-executor PR...

            Show
            abayer Andrew Bayer added a comment - Releasing as 1.22-beta-1 - plan is to follow that up with an xunit beta and a parallel-test-executor PR...
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            src/main/java/hudson/tasks/junit/CaseResult.java
            src/main/java/hudson/tasks/junit/JUnitParser.java
            src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
            src/main/java/hudson/tasks/junit/SuiteResult.java
            src/main/java/hudson/tasks/junit/TestResult.java
            src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java
            src/main/java/hudson/tasks/test/TabulatedResult.java
            src/main/java/hudson/tasks/test/TestResultParser.java
            src/test/java/hudson/tasks/junit/JUnitParserTest.java
            src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java
            src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java
            http://jenkins-ci.org/commit/junit-plugin/147692d8717493b17cc8ebf4a3a73256e1029879
            Log:
            JENKINS-27395 Removed per-run node tracking

            On further inspection, I came to the conclusion that since
            multiple-runs-per-TestResult was only possible for
            `AggregatedTestResultAction`, which doesn't support Pipeline jobs in
            the first place, it made more sense to simplify things and get rid of
            the run-and-nodes tracking of suites in favor of just nodes.

            This also necessitated gathreing the enclosing block names earlier,
            but that's probably a good change anyway.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: src/main/java/hudson/tasks/junit/CaseResult.java src/main/java/hudson/tasks/junit/JUnitParser.java src/main/java/hudson/tasks/junit/JUnitResultArchiver.java src/main/java/hudson/tasks/junit/SuiteResult.java src/main/java/hudson/tasks/junit/TestResult.java src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java src/main/java/hudson/tasks/test/TabulatedResult.java src/main/java/hudson/tasks/test/TestResultParser.java src/test/java/hudson/tasks/junit/JUnitParserTest.java src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java http://jenkins-ci.org/commit/junit-plugin/147692d8717493b17cc8ebf4a3a73256e1029879 Log: JENKINS-27395 Removed per-run node tracking On further inspection, I came to the conclusion that since multiple-runs-per-TestResult was only possible for `AggregatedTestResultAction`, which doesn't support Pipeline jobs in the first place, it made more sense to simplify things and get rid of the run-and-nodes tracking of suites in favor of just nodes. This also necessitated gathreing the enclosing block names earlier, but that's probably a good change anyway.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            src/main/java/hudson/tasks/junit/CaseResult.java
            src/main/java/hudson/tasks/junit/JUnitParser.java
            src/main/java/hudson/tasks/junit/JUnitResultArchiver.java
            src/main/java/hudson/tasks/junit/SuiteResult.java
            src/main/java/hudson/tasks/junit/TestResult.java
            src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java
            src/main/java/hudson/tasks/test/PipelineTestDetails.java
            src/main/java/hudson/tasks/test/TabulatedResult.java
            src/main/java/hudson/tasks/test/TestResultParser.java
            src/test/java/hudson/tasks/junit/JUnitParserTest.java
            src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java
            src/test/java/hudson/tasks/junit/SuiteResult2Test.java
            src/test/java/hudson/tasks/junit/SuiteResultTest.java
            src/test/java/hudson/tasks/junit/TestResultTest.java
            src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java
            http://jenkins-ci.org/commit/junit-plugin/b98cb6eba3b1207a3eba70a2e0766f3e26953b4e
            Log:
            Merge pull request #83 from abayer/jenkins-27395-remove-run-id

            JENKINS-27395 Removed per-run node tracking

            Compare: https://github.com/jenkinsci/junit-plugin/compare/1c00c38bea6c...b98cb6eba3b1

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: src/main/java/hudson/tasks/junit/CaseResult.java src/main/java/hudson/tasks/junit/JUnitParser.java src/main/java/hudson/tasks/junit/JUnitResultArchiver.java src/main/java/hudson/tasks/junit/SuiteResult.java src/main/java/hudson/tasks/junit/TestResult.java src/main/java/hudson/tasks/junit/pipeline/JUnitResultsStepExecution.java src/main/java/hudson/tasks/test/PipelineTestDetails.java src/main/java/hudson/tasks/test/TabulatedResult.java src/main/java/hudson/tasks/test/TestResultParser.java src/test/java/hudson/tasks/junit/JUnitParserTest.java src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java src/test/java/hudson/tasks/junit/SuiteResult2Test.java src/test/java/hudson/tasks/junit/SuiteResultTest.java src/test/java/hudson/tasks/junit/TestResultTest.java src/test/java/hudson/tasks/junit/pipeline/JUnitResultsStepTest.java http://jenkins-ci.org/commit/junit-plugin/b98cb6eba3b1207a3eba70a2e0766f3e26953b4e Log: Merge pull request #83 from abayer/jenkins-27395-remove-run-id JENKINS-27395 Removed per-run node tracking Compare: https://github.com/jenkinsci/junit-plugin/compare/1c00c38bea6c...b98cb6eba3b1
            Hide
            abayer Andrew Bayer added a comment -

            I've just released 1.22, which includes this. Downstream plugins, such as xunit, parallel-test-executor and Blue Ocean should come soon.

            Show
            abayer Andrew Bayer added a comment - I've just released 1.22, which includes this. Downstream plugins, such as xunit , parallel-test-executor and Blue Ocean should come soon.
            reinholdfuereder Reinhold Füreder made changes -
            Link This issue is related to JENKINS-48020 [ JENKINS-48020 ]
            Hide
            reinholdfuereder Reinhold Füreder added a comment -

            Andrew Bayer Please see JENKINS-48020 which IMHO popped up after upgrading to 1.22.1; besides that I am also wondering if any GUI differences should be already visible now...

            Show
            reinholdfuereder Reinhold Füreder added a comment - Andrew Bayer Please see JENKINS-48020 which IMHO popped up after upgrading to 1.22.1; besides that I am also wondering if any GUI differences should be already visible now...
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutor.java
            src/main/java/org/jenkinsci/plugins/parallel_test_executor/PreviousTestResultLookup.java
            src/main/java/org/jenkinsci/plugins/parallel_test_executor/SplitStep.java
            src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorTest.java
            src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorUnitTest.java
            http://jenkins-ci.org/commit/parallel-test-executor-plugin/7d09e31992b5122200ee4d48c01042eda5d7acb1
            Log:
            JENKINS-27395 Add per-stage test splitting option

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pom.xml src/main/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutor.java src/main/java/org/jenkinsci/plugins/parallel_test_executor/PreviousTestResultLookup.java src/main/java/org/jenkinsci/plugins/parallel_test_executor/SplitStep.java src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorTest.java src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorUnitTest.java http://jenkins-ci.org/commit/parallel-test-executor-plugin/7d09e31992b5122200ee4d48c01042eda5d7acb1 Log: JENKINS-27395 Add per-stage test splitting option
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutor.java
            src/main/java/org/jenkinsci/plugins/parallel_test_executor/SplitStep.java
            src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorTest.java
            src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorUnitTest.java
            http://jenkins-ci.org/commit/parallel-test-executor-plugin/89aad65707343dc98339b006fa4fe07ba1fe58b0
            Log:
            Merge pull request #31 from abayer/jenkins-27395

            JENKINS-27395 Add per-stage test splitting option

            Compare: https://github.com/jenkinsci/parallel-test-executor-plugin/compare/fde66c24dda6...89aad6570734

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pom.xml src/main/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutor.java src/main/java/org/jenkinsci/plugins/parallel_test_executor/SplitStep.java src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorTest.java src/test/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutorUnitTest.java http://jenkins-ci.org/commit/parallel-test-executor-plugin/89aad65707343dc98339b006fa4fe07ba1fe58b0 Log: Merge pull request #31 from abayer/jenkins-27395 JENKINS-27395 Add per-stage test splitting option Compare: https://github.com/jenkinsci/parallel-test-executor-plugin/compare/fde66c24dda6...89aad6570734
            jamesdumay James Dumay made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            jamesdumay James Dumay made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Hide
            jamesdumay James Dumay added a comment -

            The UI component of this work in Blue Ocean was just merged to master.

            Andrew Bayer what is the status of this ticket?

            Show
            jamesdumay James Dumay added a comment - The UI component of this work in Blue Ocean was just merged to master. Andrew Bayer what is the status of this ticket?
            jamesdumay James Dumay made changes -
            Sprint Pipeline - July/August [ 371 ] Pipeline - July/August, Pipeline - December [ 371, 446 ]
            Hide
            jrogers Jonathan Rogers added a comment -

            How does one disable this "feature"? I've been running different test suites in parallel branches and getting aggregated results. Now, it's harder to follow the history of the results for a particular test because the URL varies across runs.

            Show
            jrogers Jonathan Rogers added a comment - How does one disable this "feature"? I've been running different test suites in parallel branches and getting aggregated results. Now, it's harder to follow the history of the results for a particular test because the URL varies across runs.
            Hide
            abayer Andrew Bayer added a comment -

            Jonathan Rogers I was thinking about that and have opened JENKINS-48372 for this.

            Show
            abayer Andrew Bayer added a comment - Jonathan Rogers I was thinking about that and have opened JENKINS-48372 for this.
            Hide
            abayer Andrew Bayer added a comment -

            James Dumay I think it's basically done. Marking resolved.

            Show
            abayer Andrew Bayer added a comment - James Dumay I think it's basically done. Marking resolved.
            abayer Andrew Bayer made changes -
            Status Reopened [ 4 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            jrogers Jonathan Rogers added a comment -

            OK, I'll follow JENKINS-48372

            Show
            jrogers Jonathan Rogers added a comment - OK, I'll follow JENKINS-48372
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal CD-185 (Web Link)" [ 19117 ]
            xliiauo Xiao Liu made changes -
            Rank Ranked higher
            michaelneale Michael Neale made changes -
            Link This issue is blocked by JENKINS-45923 [ JENKINS-45923 ]
            michaelneale Michael Neale made changes -
            Link This issue is duplicated by JENKINS-41031 [ JENKINS-41031 ]
            michaelneale Michael Neale made changes -
            Issue Type Improvement [ 4 ] New Feature [ 2 ]
            michaelneale Michael Neale made changes -
            Remote Link This issue links to "Page (Jenkins Wiki)" [ 19859 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Andrew Bayer
            Path:
            jenkins-plugin/pom.xml
            jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java
            jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java
            http://jenkins-ci.org/commit/pipeline-maven-plugin/aeb7d81775305282afcaa0464c4bf72c1faf3ca9
            Log:
            JENKINS-27395 Record flow node and enclosing stage/branch for tests

            Downstream of https://github.com/jenkinsci/junit-plugin/pull/76. This
            records the `FlowNode#id` for the step that's actually recording the
            test results and the enclosing stage(s)/branch(es) of that `FlowNode`
            with each suite, along with the names of the enclosing
            stage(s)/branch(es) on each `CaseResult` for display purposes.

            This is accomplished by getting the `FlowNode` via
            `context.get(FlowNode.class)`, and then passing the configured
            `JUnitResultArchiver`, the node's ID, the enclosing stage/branch IDs,
            and the run (which is used to get the `Run#getExternalizableId()` for
            internal purposes) to the new `JUnitResultArchiver#parseAndAttach`
            static method. That method does the rest of the work.

            Once this is all released, the classic test results pages will display
            the test name with the enclosing stage/branch names prepended (if
            there are test results from multiple stages/branches in the full test
            result - i.e., only if relevant). Blue Ocean has a PR to do the same
            that's pending.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: jenkins-plugin/pom.xml jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java http://jenkins-ci.org/commit/pipeline-maven-plugin/aeb7d81775305282afcaa0464c4bf72c1faf3ca9 Log: JENKINS-27395 Record flow node and enclosing stage/branch for tests Downstream of https://github.com/jenkinsci/junit-plugin/pull/76 . This records the `FlowNode#id` for the step that's actually recording the test results and the enclosing stage(s)/branch(es) of that `FlowNode` with each suite, along with the names of the enclosing stage(s)/branch(es) on each `CaseResult` for display purposes. This is accomplished by getting the `FlowNode` via `context.get(FlowNode.class)`, and then passing the configured `JUnitResultArchiver`, the node's ID, the enclosing stage/branch IDs, and the run (which is used to get the `Run#getExternalizableId()` for internal purposes) to the new `JUnitResultArchiver#parseAndAttach` static method. That method does the rest of the work. Once this is all released, the classic test results pages will display the test name with the enclosing stage/branch names prepended (if there are test results from multiple stages/branches in the full test result - i.e., only if relevant). Blue Ocean has a PR to do the same that's pending.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Cyrille Le Clerc
            Path:
            jenkins-plugin/pom.xml
            jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java
            jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java
            http://jenkins-ci.org/commit/pipeline-maven-plugin/7224db1606420368db3f54b02a79f784c9202665
            Log:
            JENKINS-27395 Record flow node and enclosing stage/branch for tests #82

            Merge pull request #82 from abayer/jenkins-27395

            Compare: https://github.com/jenkinsci/pipeline-maven-plugin/compare/15990d246db2...7224db160642

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: jenkins-plugin/pom.xml jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java http://jenkins-ci.org/commit/pipeline-maven-plugin/7224db1606420368db3f54b02a79f784c9202665 Log: JENKINS-27395 Record flow node and enclosing stage/branch for tests #82 Merge pull request #82 from abayer/jenkins-27395 Compare: https://github.com/jenkinsci/pipeline-maven-plugin/compare/15990d246db2...7224db160642
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Cyrille Le Clerc
            Path:
            jenkins-plugin/pom.xml
            jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java
            jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java
            http://jenkins-ci.org/commit/pipeline-maven-plugin/31c1156f199515f907154938ed8e3bf5a14431c8
            Log:
            JENKINS-27395 temporarily disable bump to the jenkins-junit-pipeline 1.23 due to additional dependencies.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: jenkins-plugin/pom.xml jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/publishers/JunitTestsPublisher.java jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java http://jenkins-ci.org/commit/pipeline-maven-plugin/31c1156f199515f907154938ed8e3bf5a14431c8 Log: JENKINS-27395 temporarily disable bump to the jenkins-junit-pipeline 1.23 due to additional dependencies.
            nfalco Nikolas Falco made changes -
            Link This issue is blocking JENKINS-49344 [ JENKINS-49344 ]
            nfalco Nikolas Falco made changes -
            Link This issue is blocking JENKINS-49344 [ JENKINS-49344 ]
            Hide
            sid_s Sid S added a comment - - edited

            How do you actually consume this in a Jenkinsfile? Our tests have the same name/namespaces across different variants and each variant gets it's own build stage (it's serial for unrelated reasons below). Right now tests across multiple runs/stages are indistinguishable as identical duplicates. Currently we're using the NUnitPublisher which converts nunit3 test XMLs to junit format. We could switch from NUnitPublisher to mstest but it's unclear if we need the test transforming plugins to also be updated OR if the junit changes are sufficient (since it can pull stage names directly).

            Jenkins 2.124
            Blue Ocean 1.5.0
            JUnit 1.24
            Nunit 0.23

            This is the pipeline being used and it doesn't work i.e.

            pipeline {
              agent {
                node {
                  label 'master'
                }
              }
              stages {
                stage('Build variant1') {
                  // build steps here
                }
                stage('Publish variant 1 tests') {
                  steps {
                    step([$class: 'NUnitPublisher', testResultsPattern: 'results/variant1/*.xml', debug: false, keepJUnitReports: true, skipJUnitArchiver:false, failIfNoResults: true])
                  }
                }
                stage('Build variant2') {
                  // build steps here
                }
                stage('Publish variant 2 tests') {
                  steps {
                    step([$class: 'NUnitPublisher', testResultsPattern: 'results/variant2/*.xml', debug: false, keepJUnitReports: true, skipJUnitArchiver:false, failIfNoResults: true])
                  }
                }
              }
            }
            
            Show
            sid_s Sid S added a comment - - edited How do you actually consume this in a Jenkinsfile? Our tests have the same name/namespaces across different variants and each variant gets it's own build stage (it's serial for unrelated reasons below). Right now tests across multiple runs/stages are indistinguishable as identical duplicates. Currently we're using the NUnitPublisher which converts nunit3 test XMLs to junit format. We could switch from NUnitPublisher to mstest but it's unclear if we need the test transforming plugins to also be updated OR if the junit changes are sufficient (since it can pull stage names directly). Jenkins 2.124 Blue Ocean 1.5.0 JUnit 1.24 Nunit 0.23 This is the pipeline being used and it doesn't work i.e. pipeline { agent { node { label 'master' } } stages { stage( 'Build variant1' ) { // build steps here } stage( 'Publish variant 1 tests' ) { steps { step([$class: 'NUnitPublisher' , testResultsPattern: 'results/variant1/*.xml' , debug: false , keepJUnitReports: true , skipJUnitArchiver: false , failIfNoResults: true ]) } } stage( 'Build variant2' ) { // build steps here } stage( 'Publish variant 2 tests' ) { steps { step([$class: 'NUnitPublisher' , testResultsPattern: 'results/variant2/*.xml' , debug: false , keepJUnitReports: true , skipJUnitArchiver: false , failIfNoResults: true ]) } } } }
            Hide
            sid_s Sid S added a comment - - edited

            Documenting for future users

            1. Switched from nunit3.x to mstest's .trx (xml) test output. In dotnet this simply meant swapping the logger to the mstest one even though we're still using nunit as the actual test framework i.e.

            dotnet test --logger "trx;LogFileName=resultfile.trx"

            2. Directly used the xsl file published by the mstest folks (here) to transform .trx to junit compatible .xml files. In powershell is this a simple

            $Xslt = New-Object System.Xml.Xsl.XslCompiledTransform
            $Xslt.Load($xsltFilePath)    
            $Xslt.Transform($xmlInputFilePath, $xmlOutputFilePath)
            

            3. Use a Jenkinsfile like the one below. In the traditional Jenkins UI, the stage names get prefixed, so tests with same namespace are now seen differently. BlueOcean UI still shows them as indistinguishable / identical duplicates

            pipeline {
              agent {
                node {
                  label 'master'
                }
              }
              stages {    
                stage('variant-1') {
                  steps {
                    // build steps
                  }
                  post {
                    always {
                        junit 'results/*.xml'
                    }
                  }
                }
                stage('variant-2') {
                  steps {
                    // build steps
                  }
                  post {
                    always {
                        junit 'results/*.xml'
                    }
                  }
                }
              }
            }
            
            Show
            sid_s Sid S added a comment - - edited Documenting for future users 1. Switched from nunit3.x to mstest's .trx (xml) test output. In dotnet this simply meant swapping the logger to the mstest one even though we're still using nunit as the actual test framework i.e. dotnet test --logger "trx;LogFileName=resultfile.trx" 2. Directly used the xsl file published by the mstest folks ( here ) to transform .trx to junit compatible .xml files. In powershell is this a simple $Xslt = New- Object System .Xml.Xsl.XslCompiledTransform $Xslt.Load($xsltFilePath) $Xslt.Transform($xmlInputFilePath, $xmlOutputFilePath) 3. Use a Jenkinsfile like the one below. In the traditional Jenkins UI, the stage names get prefixed, so tests with same namespace are now seen differently. BlueOcean UI still shows them as indistinguishable / identical duplicates pipeline { agent { node { label 'master' } } stages { stage( 'variant-1' ) { steps { // build steps } post { always { junit 'results/*.xml' } } } stage( 'variant-2' ) { steps { // build steps } post { always { junit 'results/*.xml' } } } } }
            Hide
            trejkaz trejkaz added a comment -

            Something doesn't quite add up here. Our build already has stage names, but we still don't get the tests distinguished between the different stages. The only workaround idea I had so far was to use XSLT to add the prefix into the actual XML files.

             

            Show
            trejkaz trejkaz added a comment - Something doesn't quite add up here. Our build already has stage names, but we still don't get the tests distinguished between the different stages. The only workaround idea I had so far was to use XSLT to add the prefix into the actual XML files.  
            Hide
            sid_s Sid S added a comment -

            trejkaz, could you share your XSLT file or link to it?

            Show
            sid_s Sid S added a comment - trejkaz , could you share your XSLT file or link to it?
            Hide
            trejkaz trejkaz added a comment -

            We don't use an XSLT yet because we're using Gradle's reports, which are already in the right format.

             

            Show
            trejkaz trejkaz added a comment - We don't use an XSLT yet because we're using Gradle's reports, which are already in the right format.  
            Hide
            ottago Nicholas Klopfer-Webber added a comment -

            We had the same problem, but managed to work it out.

            When we started using a Jenkinsfile we copied this example, however this bypasses the code that keeps each run of tests independent.

            step([$class : 'JUnitResultArchiver', testResults: '**/build/test-results/**/*.xml'])
            

            What we now use and works is:

            junit testResults: '**/build/test-results/**/*.xml'
            
            Show
            ottago Nicholas Klopfer-Webber added a comment - We had the same problem, but managed to work it out. When we started using a Jenkinsfile we copied this example, however this bypasses the code that keeps each run of tests independent. step([$class : 'JUnitResultArchiver' , testResults: '**/build/test-results /**/ *.xml' ]) What we now use and works is: junit testResults: '**/build/test-results /**/ *.xml'
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-53362 [ JENKINS-53362 ]
            Hide
            mark_han Mark Han added a comment -

            I think there's still a problem where running a simple `./gradlew test` merges all tests into one section, even though there can be multiple flavors of tests run. Not sure if this is the case if we explicitly break up the flavors e.g. `./gradlew testDebug` Do we need to break up `./gradlew` commands between separate stages? 

            Show
            mark_han Mark Han added a comment - I think there's still a problem where running a simple `./gradlew test` merges all tests into one section, even though there can be multiple flavors of tests run. Not sure if this is the case if we explicitly break up the flavors e.g. `./gradlew testDebug` Do we need to break up `./gradlew` commands between separate stages? 
            Hide
            jglick Jesse Glick added a comment -

            Do we need to break up ./gradlew commands between separate stages?

            Currently I am afraid you do.

            Show
            jglick Jesse Glick added a comment - Do we need to break up ./gradlew commands between separate stages? Currently I am afraid you do.

              People

              • Assignee:
                abayer Andrew Bayer
                Reporter:
                kohsuke Kohsuke Kawaguchi
              • Votes:
                35 Vote for this issue
                Watchers:
                65 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: