Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29716

Incorrect Execution Results when using TestNG class name results seeking strategy

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: testlink-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 1.585, Testlink 1.9.13, Google Chrome 44.0.2403.107(64bit)
    • Similar Issues:

      Description

      I have a configured the Testlink plugin on Jenkins. After I run my TestNG test cases, I proceed to update the results in Testlink.
      The test cases are displayed correctly, but all of them are updated as "Passed"

      TestLink build ID: 282

      TestLink build name: build_2015-07-27_09-40-21

      Total of 4 tests. Where 4 (+4) passed, 0 failed, 0 were blocked and 0 were not executed.

      List of test cases and execution result status

      Test case ID Test case external ID Version Name Test project ID Execution status
      87555 SOL-10503 1 Add Provider Credential Policy 67109 Passed
      87557 SOL-10504 1 Assign Prov Credential Policy to Provider Admin 67109 Passed
      87559 SOL-10505 1 Add Customer Credential Policy 67109 Passed
      87561 SOL-10506 1 Assign Cust Credential Policy to Customer Admin 67109 Passed

      In my test cases TestNG result, 2 of the test cases have failed.

      TestNG Results
      Total Tests: 4 (+4)
      Failed Tests: 2 (+2)
      com.company.abc.def.auto.applicationtest.test.GenericTest.testAdd
      com.company.abc.def.auto.applicationtest.test.GenericTest.testAdd
      Skipped Tests: 0 (±0)
      Failed Configurations: 0 (±0)
      Skipped Configurations: 0 (±0)

      I have a suspicion it might be related to my Test result seeking strategy definition :
      I use :
      TestNG class name
      Include Pattern -> output/testng-results.xml
      Key Custom Field -> TL-Jenkins

      I am using : TestLink 1.9.13 (Stormbringer)
      Jenkins ver. 1.585
      Testlink Plugin 3.10

      http://mantis.testlink.org/view.php?id=7204

        Attachments

          Activity

          Hide
          vossqa voss qa added a comment -

          Anybody able to assist with this?

          Show
          vossqa voss qa added a comment - Anybody able to assist with this?
          Hide
          arne1985 Arne Wohletz added a comment -

          I have a similar problem.
          TestNG plugin states that 1 from 1 TC passed, but the TestLink plugin results show a 'Not run'.
          The testng-results.xml file is OK, shows that the TC passed. Somehow Testlink does not confirm this result.
          I also use a
          'TestNG class name' result seeking strategy
          'Include pattern' = target/**/testng-results.xml

          and a 'Publish TestNG results' post-build-action:
          TestNG XML report pattern = target/**/testng-results.xml

          I am currently learning from this tutorial: https://wiki.jenkins-ci.org/download/attachments/753702/jenkins.pdf

          TestLink v1.9.14 (Padawan)
          TestLink Plugin v3.12
          TestNG Results Plugin v1.13
          Jenkins v1.651.3

          Can somebody help?
          Could it be that the TestLink plugin wants to retrieve the test results before those are written out in the xml file, setting it to 'not run' as a default?

          Show
          arne1985 Arne Wohletz added a comment - I have a similar problem. TestNG plugin states that 1 from 1 TC passed, but the TestLink plugin results show a 'Not run'. The testng-results.xml file is OK, shows that the TC passed. Somehow Testlink does not confirm this result. I also use a 'TestNG class name' result seeking strategy 'Include pattern' = target/**/testng-results.xml and a 'Publish TestNG results' post-build-action: TestNG XML report pattern = target/**/testng-results.xml I am currently learning from this tutorial: https://wiki.jenkins-ci.org/download/attachments/753702/jenkins.pdf TestLink v1.9.14 (Padawan) TestLink Plugin v3.12 TestNG Results Plugin v1.13 Jenkins v1.651.3 Can somebody help? Could it be that the TestLink plugin wants to retrieve the test results before those are written out in the xml file, setting it to 'not run' as a default?
          Hide
          kinow Bruno P. Kinoshita added a comment -

          > Could it be that the TestLink plugin wants to retrieve the test results before those are written out in the xml file, setting it to 'not run' as a default?

          By default, for each test case retrieved for the given test plan, the status is set to not run.

          Then it will use the strategy you chose to search for test results.

          For JUnit, it uses Jenkins' JUNit parser.

          For TAP, it uses tap4j.

          And for TestNG it uses a custom testng-parser - see https://github.com/tupilabs/testng-parser

          So it would require investigation to define whether there's something wrong with the parser, if the configuration is correct, of if it's a bug or incompatibility in the Java and/or PHP layer.

          As for the tutorial, I want to move it under www.gitbook.com, so that we can use Markdown for source, and get pull requests and issues. Then I would update it to the latest version of Jenkins, libraries and TestLink. But unfortunately I don't have the resources for doing it right now (mainly time, but also $ )

          Show
          kinow Bruno P. Kinoshita added a comment - > Could it be that the TestLink plugin wants to retrieve the test results before those are written out in the xml file, setting it to 'not run' as a default? By default, for each test case retrieved for the given test plan, the status is set to not run. Then it will use the strategy you chose to search for test results. For JUnit, it uses Jenkins' JUNit parser. For TAP, it uses tap4j. And for TestNG it uses a custom testng-parser - see https://github.com/tupilabs/testng-parser So it would require investigation to define whether there's something wrong with the parser, if the configuration is correct, of if it's a bug or incompatibility in the Java and/or PHP layer. As for the tutorial, I want to move it under www.gitbook.com, so that we can use Markdown for source, and get pull requests and issues. Then I would update it to the latest version of Jenkins, libraries and TestLink. But unfortunately I don't have the resources for doing it right now (mainly time, but also $ )
          Hide
          arne1985 Arne Wohletz added a comment -

          Hello Bruno,
          thanks for for quick response.
          I found the solution which works for me - the class name that is used in the example is actually 'jenkins.plugins.testlink.examples.TestCurrentTime' whereas in TestLink I used 'hudson.plugins.testlink.examples.TestCurrentTime' as it said in the tutorial PDF (page 15).
          Now it works perfectly!

          Maybe that also helps voss qa.

          Show
          arne1985 Arne Wohletz added a comment - Hello Bruno, thanks for for quick response. I found the solution which works for me - the class name that is used in the example is actually 'jenkins.plugins.testlink.examples.TestCurrentTime' whereas in TestLink I used 'hudson.plugins.testlink.examples.TestCurrentTime' as it said in the tutorial PDF (page 15). Now it works perfectly! Maybe that also helps voss qa .

            People

            • Assignee:
              kinow Bruno P. Kinoshita
              Reporter:
              vossqa voss qa
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: