Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58430

Publishing two fitnesse xml files with same test name confuses the results

    Details

    • Similar Issues:

      Description

      • In one pipeline job, run the same FitNesse tests and generate 2 xml files. So for instance:
        • FitNesse tests on Linux generates fitnesse_linux.xml
        • FitNesse tests on Windows generates fitnesse_windows.xml
          The way the tests are generated is as follows (same for windows)
          java -jar ./fitnesse-standalone.jar -c "MySuite?suite&format=xml&includehtml" -b fitnesse_windows.xml
          
      • Now in the pipeline job, the tests are published in this way:
        step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml'])
      • Then let's assume that MySuite.TestOne fails on Linux, when I go to see the Details of the failure from the FitNesseReport link in Jenkins, while TestOne shows as failed, the html details I can see from there actually shows the details of the windows run.

      To reproduce this issue, use the following pipeline. Run it once and them manually copy the two xml files attached into the workspace. Run it again and it should publish the FitNesse results. My agent is a linux agent if that makes a difference.

      pipeline {
          agent {
              label 'your-agent'
          }
          stages {
              stage ('Repro') {
                  steps {
                      echo 'Make sure the files are in the Workspace'
                  }
              }
          }
          post {
              always {
                  step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml'])
              }
          }
      }
      

      Once the results are published, you can see that the fitnesse_windows.xml has been successful while fitnesse_linux.xml has failed. You can see this in the FitNesse report that this is represented correctly:


      But if I click on the fitnesse_linux.xml and then on the Details link for test BuysideEnterprise.Ims.MySuite.TestOne then it actually shows me the details of the run in fitnesse_windows.xml instead of the linux one. It looks like the details have been overridden.

      Therefore, while I see that the test has failed, I cannot see why as it is showing me the incorrect run details:

        Attachments

          Activity

          johannmoro Johann Moro created issue -
          johannmoro Johann Moro made changes -
          Field Original Value New Value
          Attachment fitnesse_linux.xml [ 47968 ]
          Attachment fitnesse_windows.xml [ 47969 ]
          johannmoro Johann Moro made changes -
          Description * In one pipeline job, run the same FitNesse tests and generate 2 xml files. So for instance:
          ** FitNesse tests on Linux generates fitnesse_linux.xml
          ** FitNesse tests on Windows generates fitnesse_windows.xml
          The way the tests are generated is as follows (same for windows)
          {code:java}
          java -jar ./fitnesse-standalone.jar -c "MySuite?suite&format=xml&includehtml" -b fitnesse_windows.xml
          {code}
          * Now in the pipeline job, the tests are published in this way:
          {code:java}step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml']){code}
          * Then let's assume that MySuite.TestOne fails on Linux, when I go to see the Details of the failure from the FitNesseReport link in Jenkins, while TestOne shows as failed, the html details I can see from there actually shows the details of the windows run.
          * In one pipeline job, run the same FitNesse tests and generate 2 xml files. So for instance:
           ** FitNesse tests on Linux generates fitnesse_linux.xml
           ** FitNesse tests on Windows generates fitnesse_windows.xml
           The way the tests are generated is as follows (same for windows)
          {code:java}
          java -jar ./fitnesse-standalone.jar -c "MySuite?suite&format=xml&includehtml" -b fitnesse_windows.xml
          {code}

           * Now in the pipeline job, the tests are published in this way:
          {code:java}
          step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml']){code}

           * Then let's assume that MySuite.TestOne fails on Linux, when I go to see the Details of the failure from the FitNesseReport link in Jenkins, while TestOne shows as failed, the html details I can see from there actually shows the details of the windows run.

          ----
          To reproduce this issue, use the following pipeline. Run it once and them manually copy the two xml files attached into the workspace. Run it again and it should publish the FitNesse results.
          {code}
          pipeline {
              agent {
                  label 'your-agent'
              }
              stages {
                  stage ('Repro') {
                      steps {
                          echo 'Make sure the files are in the Workspace'
                      }
                  }
              }
              post {
                  always {
                      step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml'])
                  }
              }
          }
          {code}

          Once the results are published, you can see that the fitnesse_windows.xml has been successful while fitnesse_linux.xml has failed. You can see this in the FitNesse report that this is represented correctly:
          johannmoro Johann Moro made changes -
          Attachment FitNesseReport.png [ 47970 ]
          johannmoro Johann Moro made changes -
          johannmoro Johann Moro made changes -
          Description * In one pipeline job, run the same FitNesse tests and generate 2 xml files. So for instance:
           ** FitNesse tests on Linux generates fitnesse_linux.xml
           ** FitNesse tests on Windows generates fitnesse_windows.xml
           The way the tests are generated is as follows (same for windows)
          {code:java}
          java -jar ./fitnesse-standalone.jar -c "MySuite?suite&format=xml&includehtml" -b fitnesse_windows.xml
          {code}

           * Now in the pipeline job, the tests are published in this way:
          {code:java}
          step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml']){code}

           * Then let's assume that MySuite.TestOne fails on Linux, when I go to see the Details of the failure from the FitNesseReport link in Jenkins, while TestOne shows as failed, the html details I can see from there actually shows the details of the windows run.

          ----
          To reproduce this issue, use the following pipeline. Run it once and them manually copy the two xml files attached into the workspace. Run it again and it should publish the FitNesse results.
          {code}
          pipeline {
              agent {
                  label 'your-agent'
              }
              stages {
                  stage ('Repro') {
                      steps {
                          echo 'Make sure the files are in the Workspace'
                      }
                  }
              }
              post {
                  always {
                      step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml'])
                  }
              }
          }
          {code}

          Once the results are published, you can see that the fitnesse_windows.xml has been successful while fitnesse_linux.xml has failed. You can see this in the FitNesse report that this is represented correctly:
          * In one pipeline job, run the same FitNesse tests and generate 2 xml files. So for instance:
           ** FitNesse tests on Linux generates fitnesse_linux.xml
           ** FitNesse tests on Windows generates fitnesse_windows.xml
           The way the tests are generated is as follows (same for windows)
          {code:java}
          java -jar ./fitnesse-standalone.jar -c "MySuite?suite&format=xml&includehtml" -b fitnesse_windows.xml
          {code}

           * Now in the pipeline job, the tests are published in this way:
          {code:java}
          step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml']){code}

           * Then let's assume that MySuite.TestOne fails on Linux, when I go to see the Details of the failure from the FitNesseReport link in Jenkins, while TestOne shows as failed, the html details I can see from there actually shows the details of the windows run.

          ----
          To reproduce this issue, use the following pipeline. Run it once and them manually copy the two xml files attached into the workspace. Run it again and it should publish the FitNesse results.
          {code}
          pipeline {
              agent {
                  label 'your-agent'
              }
              stages {
                  stage ('Repro') {
                      steps {
                          echo 'Make sure the files are in the Workspace'
                      }
                  }
              }
              post {
                  always {
                      step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml'])
                  }
              }
          }
          {code}

          Once the results are published, you can see that the fitnesse_windows.xml has been successful while fitnesse_linux.xml has failed. You can see this in the FitNesse report that this is represented correctly:
           !FitNesseReport.png!
           !Details-of-test-not-representing-failure.png.png|thumbnail!
          But if I click on the fitnesse_linux.xml and then on the Details link for test _BuysideEnterprise.Ims.MySuite.TestOne_ then it actually shows me the details of the run in fitnesse_windows.xml instead of the linux one. It looks like the details have been overridden.

          Therefore, while I see that the test has failed, I cannot see why as it is showing me the incorrect run details:
           !Details-of-test-not-representing-failure.png!
          johannmoro Johann Moro made changes -
          Description * In one pipeline job, run the same FitNesse tests and generate 2 xml files. So for instance:
           ** FitNesse tests on Linux generates fitnesse_linux.xml
           ** FitNesse tests on Windows generates fitnesse_windows.xml
           The way the tests are generated is as follows (same for windows)
          {code:java}
          java -jar ./fitnesse-standalone.jar -c "MySuite?suite&format=xml&includehtml" -b fitnesse_windows.xml
          {code}

           * Now in the pipeline job, the tests are published in this way:
          {code:java}
          step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml']){code}

           * Then let's assume that MySuite.TestOne fails on Linux, when I go to see the Details of the failure from the FitNesseReport link in Jenkins, while TestOne shows as failed, the html details I can see from there actually shows the details of the windows run.

          ----
          To reproduce this issue, use the following pipeline. Run it once and them manually copy the two xml files attached into the workspace. Run it again and it should publish the FitNesse results.
          {code}
          pipeline {
              agent {
                  label 'your-agent'
              }
              stages {
                  stage ('Repro') {
                      steps {
                          echo 'Make sure the files are in the Workspace'
                      }
                  }
              }
              post {
                  always {
                      step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml'])
                  }
              }
          }
          {code}

          Once the results are published, you can see that the fitnesse_windows.xml has been successful while fitnesse_linux.xml has failed. You can see this in the FitNesse report that this is represented correctly:
           !FitNesseReport.png!
           !Details-of-test-not-representing-failure.png.png|thumbnail!
          But if I click on the fitnesse_linux.xml and then on the Details link for test _BuysideEnterprise.Ims.MySuite.TestOne_ then it actually shows me the details of the run in fitnesse_windows.xml instead of the linux one. It looks like the details have been overridden.

          Therefore, while I see that the test has failed, I cannot see why as it is showing me the incorrect run details:
           !Details-of-test-not-representing-failure.png!
          * In one pipeline job, run the same FitNesse tests and generate 2 xml files. So for instance:
           ** FitNesse tests on Linux generates fitnesse_linux.xml
           ** FitNesse tests on Windows generates fitnesse_windows.xml
           The way the tests are generated is as follows (same for windows)
          {code:java}
          java -jar ./fitnesse-standalone.jar -c "MySuite?suite&format=xml&includehtml" -b fitnesse_windows.xml
          {code}

           * Now in the pipeline job, the tests are published in this way:
          {code:java}
          step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml']){code}

           * Then let's assume that MySuite.TestOne fails on Linux, when I go to see the Details of the failure from the FitNesseReport link in Jenkins, while TestOne shows as failed, the html details I can see from there actually shows the details of the windows run.

          ----
          To reproduce this issue, use the following pipeline. Run it once and them manually copy the two xml files attached into the workspace. Run it again and it should publish the FitNesse results. My agent is a linux agent if that makes a difference.
          {code}
          pipeline {
              agent {
                  label 'your-agent'
              }
              stages {
                  stage ('Repro') {
                      steps {
                          echo 'Make sure the files are in the Workspace'
                      }
                  }
              }
              post {
                  always {
                      step([$class: 'FitnesseResultsRecorder', fitnessePathToXmlResultsIn: 'fitnesse_*.xml'])
                  }
              }
          }
          {code}

          Once the results are published, you can see that the fitnesse_windows.xml has been successful while fitnesse_linux.xml has failed. You can see this in the FitNesse report that this is represented correctly:
           !FitNesseReport.png!
           !Details-of-test-not-representing-failure.png.png|thumbnail!
          But if I click on the fitnesse_linux.xml and then on the Details link for test _BuysideEnterprise.Ims.MySuite.TestOne_ then it actually shows me the details of the run in fitnesse_windows.xml instead of the linux one. It looks like the details have been overridden.

          Therefore, while I see that the test has failed, I cannot see why as it is showing me the incorrect run details:
           !Details-of-test-not-representing-failure.png!
          jaruzafa jaruzafa made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          johannmoro Johann Moro added a comment -

          Note that this issue probably needs to be reproduced on Linux because JENKINS-58923 will prevent you from reproducing this on Windows.

          Show
          johannmoro Johann Moro added a comment - Note that this issue probably needs to be reproduced on Linux because JENKINS-58923 will prevent you from reproducing this on Windows.
          jaruzafa jaruzafa made changes -
          Status In Progress [ 3 ] Fixed but Unreleased [ 10203 ]
          Resolution Fixed [ 1 ]
          Hide
          jaruzafa jaruzafa added a comment -

          Fixed on fitnesse-plugin 1.30

          Show
          jaruzafa jaruzafa added a comment - Fixed on fitnesse-plugin 1.30
          jaruzafa jaruzafa made changes -
          Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              johannmoro Johann Moro
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: