Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-8976

testng plugin is not able to differentiate between multiple test-methods run as part of the same <class> in data provider mode

    Details

    • Similar Issues:

      Description

      When running parameterized tests in TestNG, Jenkins fails to keep the message for each parameterized run.

      For example, if we had 4 parameter combinations, the first 3 failed and the last one succeeded, Jenkins will correctly show the status (PASS or FAIL), but when we click to see the details of the failed test-case, it will display the message from the last parameter combination (in this case 'PASS').

        Attachments

          Activity

          Hide
          farshidce farshidce added a comment -

          Dimitar,

          If possible can you please attach the result.xml file your test execution creates. It would be easier if I can repro the bug by just parsing the xml and see the result immediately ?

          Show
          farshidce farshidce added a comment - Dimitar, If possible can you please attach the result.xml file your test execution creates. It would be easier if I can repro the bug by just parsing the xml and see the result immediately ?
          Hide
          ddimitrov Dimitar Dimitrov added a comment -

          Unfortunately, our build does not retain the result.xml files, but here is a test you can use to reproduce:

          import org.testng.annotations.DataProvider;
          import org.testng.annotations.Test;
          
          public class MixedFailuresTest {
              @DataProvider
              Object[][] data() {
                  return new Object[][] { {1}, {2}, {3}, {4}, {5}, {6}, {7}, {8}};        
              }
              
              @Test(dataProvider = "data")
              void test(int i) { assert i%2==0; }
          }
          

          Even though 4 of the executions will fail, when you click of them in Jenkins, you will see 'PASS' instead of the assertion stack trace.

          Show
          ddimitrov Dimitar Dimitrov added a comment - Unfortunately, our build does not retain the result.xml files, but here is a test you can use to reproduce: import org.testng.annotations.DataProvider; import org.testng.annotations.Test; public class MixedFailuresTest { @DataProvider Object [][] data() { return new Object [][] { {1}, {2}, {3}, {4}, {5}, {6}, {7}, {8}}; } @Test(dataProvider = "data" ) void test( int i) { assert i%2==0; } } Even though 4 of the executions will fail, when you click of them in Jenkins, you will see 'PASS' instead of the assertion stack trace.
          Hide
          farshidce farshidce added a comment -

          thanks . i was able to repro the issue.

          its a general bug with data provider test cases which we are not handling it properly yet.

          Show
          farshidce farshidce added a comment - thanks . i was able to repro the issue. its a general bug with data provider test cases which we are not handling it properly yet.
          Hide
          farshidce farshidce added a comment -

          pushed in the fix
          https://github.com/jenkinsci/testng-plugin-plugin/commit/9ed61505dbe8e23f85f81ccf326971bfb6ee9d97

          will prepare/release the plugin in couple of days after more testing

          Show
          farshidce farshidce added a comment - pushed in the fix https://github.com/jenkinsci/testng-plugin-plugin/commit/9ed61505dbe8e23f85f81ccf326971bfb6ee9d97 will prepare/release the plugin in couple of days after more testing

            People

            • Assignee:
              farshidce farshidce
              Reporter:
              ddimitrov Dimitar Dimitrov
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: