Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41031

in Junit reports wrong output is shown for tests with same name

    Details

    • Similar Issues:

      Description

      we have a setup in which we run the same tests on windows and linux nodes. The test results are copied to the master into separate folders and a junit report is done for the base folder.

      If, for example, linux fails and windows succeeds, I see 2 failed tests with the same name and in both tests I see the same log output, in my case the log output of the successful test.

      This is quite annoying, since I see a test failure that absolutely does not match the log output, which looks completely successful.

      I will setup a stripped down test scenario that demonstrates the problem.

        Attachments

          Issue Links

            Activity

            Hide
            ronald_brindl Ronald Brindl added a comment -

            I attached 2 files, which contain a folder junit with 2 subfolders. Those two subfolders contain a junit xml each, which has the same suitname, testclassname and testname, but they differ in the result, one is successful (lin), and one is failed (win).
            And there is a Jenkinsfile in each of them which does a checkout and a junit report on those junit xml files.
            I configured this to run in a multibranch pipeline build.

            The 2 zip files are only different in the order of the junit folders, either 1-lin, 2-win or the other way round: 1-win, 2-lin.

            The order is relevant for this problem, in one case you see the successful log output in both junit reports, in the other case you see the unsuccessful log in both reports.
            I also added the junit json exported from the respective build, which clearly shows that both test results show the same log output.

            Show
            ronald_brindl Ronald Brindl added a comment - I attached 2 files, which contain a folder junit with 2 subfolders. Those two subfolders contain a junit xml each, which has the same suitname, testclassname and testname, but they differ in the result, one is successful (lin), and one is failed (win). And there is a Jenkinsfile in each of them which does a checkout and a junit report on those junit xml files. I configured this to run in a multibranch pipeline build. The 2 zip files are only different in the order of the junit folders, either 1-lin, 2-win or the other way round: 1-win, 2-lin. The order is relevant for this problem, in one case you see the successful log output in both junit reports, in the other case you see the unsuccessful log in both reports. I also added the junit json exported from the respective build, which clearly shows that both test results show the same log output.
            Hide
            ronald_brindl Ronald Brindl added a comment -

            PS: you can also find this testsetup in my github repo: https://github.com/rbrindl/jenkins-41031-junit-reports. there is a branch for each scenario.

            Show
            ronald_brindl Ronald Brindl added a comment - PS: you can also find this testsetup in my github repo: https://github.com/rbrindl/jenkins-41031-junit-reports . there is a branch for each scenario.
            Hide
            ronald_brindl Ronald Brindl added a comment -
            Successful folder is first (linux); and failing is second (windows)
            {
              "_class" : "hudson.tasks.junit.TestResult",
              "testActions" : [
                
              ],
              "duration" : 30.0,
              "empty" : false,
              "failCount" : 1,
              "passCount" : 1,
              "skipCount" : 0,
              "suites" : [
                {
                  "cases" : [
                    {
                      "testActions" : [
                        
                      ],
                      "age" : 0,
                      "className" : "this.is.TheTest",
                      "duration" : 10.0,
                      "errorDetails" : null,
                      "errorStackTrace" : null,
                      "failedSince" : 0,
                      "name" : "myTest",
                      "skipped" : false,
                      "skippedMessage" : null,
                      "status" : "PASSED",
                      "stderr" : "\n\tNo error on Linux\n  ",
                      "stdout" : "\n\tTest run on Linux\n  "
                    },
                    {
                      "testActions" : [
                        
                      ],
                      "age" : 1,
                      "className" : "this.is.TheTest",
                      "duration" : 20.0,
                      "errorDetails" : "A Failure",
                      "errorStackTrace" : "The Stacktrace",
                      "failedSince" : 1,
                      "name" : "myTest",
                      "skipped" : false,
                      "skippedMessage" : null,
                      "status" : "FAILED",
                      "stderr" : "\n\tNo error on Linux\n  ",
                      "stdout" : "\n\tTest run on Linux\n  "
                    }
                  ],
                  "duration" : 10.0,
                  "id" : null,
                  "name" : "this.is.TheTest",
                  "stderr" : "\n\tNo error on Linux\n  ",
                  "stdout" : "\n\tTest run on Linux\n  ",
                  "timestamp" : "2017-01-12T14:47:16"
                }
              ]
            }
            
            Failing folder is first (windows); and successful is second (linux)
            {
              "_class" : "hudson.tasks.junit.TestResult",
              "testActions" : [
                
              ],
              "duration" : 30.0,
              "empty" : false,
              "failCount" : 1,
              "passCount" : 1,
              "skipCount" : 0,
              "suites" : [
                {
                  "cases" : [
                    {
                      "testActions" : [
                        
                      ],
                      "age" : 1,
                      "className" : "this.is.TheTest",
                      "duration" : 20.0,
                      "errorDetails" : "A Failure",
                      "errorStackTrace" : "The Stacktrace",
                      "failedSince" : 1,
                      "name" : "myTest",
                      "skipped" : false,
                      "skippedMessage" : null,
                      "status" : "FAILED",
                      "stderr" : "\n\tError happened on windows\n  ",
                      "stdout" : "\n\tTest run on windows\n  "
                    },
                    {
                      "testActions" : [
                        
                      ],
                      "age" : 0,
                      "className" : "this.is.TheTest",
                      "duration" : 10.0,
                      "errorDetails" : null,
                      "errorStackTrace" : null,
                      "failedSince" : 0,
                      "name" : "myTest",
                      "skipped" : false,
                      "skippedMessage" : null,
                      "status" : "PASSED",
                      "stderr" : "\n\tError happened on windows\n  ",
                      "stdout" : "\n\tTest run on windows\n  "
                    }
                  ],
                  "duration" : 20.0,
                  "id" : null,
                  "name" : "this.is.TheTest",
                  "stderr" : "\n\tError happened on windows\n  ",
                  "stdout" : "\n\tTest run on windows\n  ",
                  "timestamp" : "2017-01-12T14:48:16"
                }
              ]
            }
            
            Show
            ronald_brindl Ronald Brindl added a comment - Successful folder is first (linux); and failing is second (windows) { "_class" : "hudson.tasks.junit.TestResult" , "testActions" : [ ], "duration" : 30.0, "empty" : false , "failCount" : 1, "passCount" : 1, "skipCount" : 0, "suites" : [ { "cases" : [ { "testActions" : [ ], "age" : 0, "className" : " this .is.TheTest" , "duration" : 10.0, "errorDetails" : null , "errorStackTrace" : null , "failedSince" : 0, "name" : "myTest" , "skipped" : false , "skippedMessage" : null , "status" : "PASSED" , "stderr" : "\n\tNo error on Linux\n " , "stdout" : "\n\tTest run on Linux\n " }, { "testActions" : [ ], "age" : 1, "className" : " this .is.TheTest" , "duration" : 20.0, "errorDetails" : "A Failure" , "errorStackTrace" : "The Stacktrace" , "failedSince" : 1, "name" : "myTest" , "skipped" : false , "skippedMessage" : null , "status" : "FAILED" , "stderr" : "\n\tNo error on Linux\n " , "stdout" : "\n\tTest run on Linux\n " } ], "duration" : 10.0, "id" : null , "name" : " this .is.TheTest" , "stderr" : "\n\tNo error on Linux\n " , "stdout" : "\n\tTest run on Linux\n " , "timestamp" : "2017-01-12T14:47:16" } ] } Failing folder is first (windows); and successful is second (linux) { "_class" : "hudson.tasks.junit.TestResult" , "testActions" : [ ], "duration" : 30.0, "empty" : false , "failCount" : 1, "passCount" : 1, "skipCount" : 0, "suites" : [ { "cases" : [ { "testActions" : [ ], "age" : 1, "className" : " this .is.TheTest" , "duration" : 20.0, "errorDetails" : "A Failure" , "errorStackTrace" : "The Stacktrace" , "failedSince" : 1, "name" : "myTest" , "skipped" : false , "skippedMessage" : null , "status" : "FAILED" , "stderr" : "\n\tError happened on windows\n " , "stdout" : "\n\tTest run on windows\n " }, { "testActions" : [ ], "age" : 0, "className" : " this .is.TheTest" , "duration" : 10.0, "errorDetails" : null , "errorStackTrace" : null , "failedSince" : 0, "name" : "myTest" , "skipped" : false , "skippedMessage" : null , "status" : "PASSED" , "stderr" : "\n\tError happened on windows\n " , "stdout" : "\n\tTest run on windows\n " } ], "duration" : 20.0, "id" : null , "name" : " this .is.TheTest" , "stderr" : "\n\tError happened on windows\n " , "stdout" : "\n\tTest run on windows\n " , "timestamp" : "2017-01-12T14:48:16" } ] }
            Hide
            michaelneale Michael Neale added a comment -

            Closing as I think this was fixed in https://issues.jenkins-ci.org/browse/JENKINS-27395 (out in beta 1.4) - correct me if wrong. 

            Show
            michaelneale Michael Neale added a comment - Closing as I think this was fixed in https://issues.jenkins-ci.org/browse/JENKINS-27395  (out in beta 1.4) - correct me if wrong. 

              People

              • Assignee:
                Unassigned
                Reporter:
                ronald_brindl Ronald Brindl
              • Votes:
                5 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: