Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52202

xUnit reports "All test reports are empty" in case of parallel steps but reports are not empty

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I have created a simple pipeline with two parallel steps.

      In each step I create a valid JUnit xml file and use xUnit to analyse it.

      During the run xunit can analyse one of the JUnit xml file, but for the other one it only logs out the message: WARNING: All test reports are empty.

      Attached pipeline.log log.txt

       

      Example pipeline:

      timestamps {
          node {
              deleteDir()
              sleep 1
              parallel(
                          dateiEins: {
              stage('generate xml file 1'){
                  dir('file1') {
                  dir('test-output') {
                      dir('junit') {
                                  writeFile file: 'TEST-de.edict.services.portaltests.SlotSessionTest_ES.xml', text: '''<?xml version="1.0" encoding="UTF-8"?>
                  <!-- Generated by de.edict.listener.JUnitReportReporterBase -->
                  <testsuite hostname="hhbld03t" name="de.edict.services.portaltests.SlotSessionTest--ES" tests="11" failures="0" timestamp="26 Jun 2018 13:51:32 GMT" time="265.564" errors="1">
                    <testcase name="testExpireSlotSessionWhileGameStartLossCondition--ES" time="17.388" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testConsistencyOfSlotsessionAndAccountDataAfterHeartbeat--ES" time="59.843" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testSlotSessionWinning--ES" time="21.539" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testExpireSlotSessionWhileGameStartTimeCondition--ES" time="17.597" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testSlotSessionLimitAmountExceeded--ES" time="13.583" classname="de.edict.services.portaltests.SlotSessionTest">
                      <error type="java.lang.AssertionError" message="Expecting message type SLOT_LOSS_LIMIT_EXCEEDED in Response.
                  Expected: is &quot;SLOT_LOSS_LIMIT_EXCEEDED&quot;
                       but: was &quot;ACCOUNT_EMPTY&quot;">
                        <![CDATA[java.lang.AssertionError: Expecting message type SLOT_LOSS_LIMIT_EXCEEDED in Response.
                  Expected: is "SLOT_LOSS_LIMIT_EXCEEDED"
                       but: was "ACCOUNT_EMPTY"
                  	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
                  	at de.edict.services.portaltests.SlotSessionTest.testSlotSessionLimitAmountExceeded(SlotSessionTest.java:288)
                  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  	at java.lang.reflect.Method.invoke(Method.java:498)
                  	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
                  	at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
                  	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
                  	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
                  	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
                  	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
                  	at org.testng.TestRunner.privateRun(TestRunner.java:767)
                  	at org.testng.TestRunner.run(TestRunner.java:617)
                  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
                  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
                  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
                  	at org.testng.SuiteRunner.run(SuiteRunner.java:254)
                  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
                  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
                  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
                  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
                  	at org.testng.TestNG.run(TestNG.java:1057)
                  	at org.testng.TestNG.privateMain(TestNG.java:1364)
                  	at org.testng.TestNG.main(TestNG.java:1333)
                  ]]>
                      </error>
                    </testcase> <!-- testSlotSessionLimitAmountExceeded-ES -->
                    <testcase name="testConsistencyOfSlotsessionAndAccountData--ES" time="18.456" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testCreateSlotSessionWithGameStart--ES" time="2.897" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testExpireSlotSessionWhileHeartbeat--ES" time="35.558" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testCheckPortalRestService--ES" time="28.069" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testCheckBORestService--ES" time="17.425" classname="de.edict.services.portaltests.SlotSessionTest"/>
                    <testcase name="testLossLimitWarning--ES" time="33.209" classname="de.edict.services.portaltests.SlotSessionTest"/>
                  </testsuite> <!-- de.edict.services.portaltests.SlotSessionTest-ES -->
                  '''
                  sh 'ls -l'
                  xunit testTimeMargin: '3000', thresholdMode: 1, thresholds: [failed(failureNewThreshold: '0', failureThreshold: '0'), skipped()], tools: [JUnit(deleteOutputFiles: false, failIfNotNew: false, pattern: 'TEST-*.xml', skipNoTestFiles: false, stopProcessingIfError: true)]
                      }
                  }
              }
          }
      },
      dateiZwei: {
          stage('generate xml file 2'){
              dir('file2') {
                  dir('test-output') {
                      dir('junit') {                    writeFile file: 'TEST-de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests--IoM_default.xml', text: '''<?xml version="1.0" encoding="UTF-8"?>
                          <!-- Generated by de.edict.listener.JUnitReportReporterBase -->
                          <testsuite hostname="hhbld01t" name="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests--IoM_default" tests="15" failures="1" timestamp="26 Jun 2018 15:41:26 GMT" time="55.391" errors="0">
                            <testcase name="testCreateBonusOfferWithCountryRestrictions--IoM_default" time="12.167" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testFirstPayInWith50EURWithVoucherCode--IoM_default" time="1.454" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testCreateBonusOfferTrigger999--IoM_default" time="2.114" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testEveryPayInDuringThePeriodWithVisa--IoM_default" time="26.713" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests">
                              <failure type="com.googlecode.sardine.impl.SardineException" message="Unexpected response">
                                <![CDATA[com.googlecode.sardine.impl.SardineException: Unexpected response
                          	at com.googlecode.sardine.impl.handler.ValidatingResponseHandler.validateResponse(ValidatingResponseHandler.java:49)
                          	at com.googlecode.sardine.impl.handler.VoidResponseHandler.handleResponse(VoidResponseHandler.java:35)
                          	at com.googlecode.sardine.impl.SardineImpl.get(SardineImpl.java:478)
                          	at com.googlecode.sardine.impl.SardineImpl.get(SardineImpl.java:458)
                          	at de.edict.eoc.test.utils.TextHelper.connectWikiWebdav(TextHelper.java:75)
                          	at de.edict.eoc.test.bonus.BonusOfferTestHelper.checkDefaultStakerestrictionsAgainstWiki(BonusOfferTestHelper.java:339)
                          	at de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests.testEveryPayInDuringThePeriodWithVisa(BonusOfferDepositTests.java:178)
                          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                          	at java.lang.reflect.Method.invoke(Method.java:498)
                          	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
                          	at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
                          	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
                          	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
                          	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
                          	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
                          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                          	at java.lang.Thread.run(Thread.java:748)
                          ]]>
                              </failure>
                            </testcase> <!-- testEveryPayInDuringThePeriodWithVisa-IoM_default -->
                            <testcase name="TryToCreateBonusWithoutCorrectPermission--IoM_default" time="0.610" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testBonusOfferWithSpecialCharacter--IoM_default" time="3.118" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testFirstPayinWithOneHourDuration--IoM_default" time="0.693" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testBonusOfferWithSpecialCharacterMinus--IoM_default" time="0.501" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testFirstPayinWithoutStartAndEnddate--IoM_default" time="0.542" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testCreateBonusOfferWithCountryAndPaymentRestrictions--IoM_default" time="2.136" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testEveryfifthPayInWithVisa--IoM_default" time="0.675" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testEveryPayInDuringThePeriod--IoM_default" time="1.956" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testFirstPayInWith50EUR--IoM_default" time="0.706" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testCreateBonusOfferSartTodayTop5Restrictions--IoM_default" time="1.055" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                            <testcase name="testCreateBonusOfferAllMax--IoM_default" time="0.951" classname="de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests"/>
                          </testsuite> <!-- de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests-IoM_default -->
                          '''
                          
                                              sh 'ls -l'
                                              xunit testTimeMargin: '3000', thresholdMode: 1, thresholds: [failed(failureNewThreshold: '0', failureThreshold: '0'), skipped()], tools: [JUnit(deleteOutputFiles: false, failIfNotNew: false, pattern: "TEST-*.xml", skipNoTestFiles: false, stopProcessingIfError: true)]
                                          }
                                      }
                                  }
                              }
                          }
                  )
                          
          
              }
          } 

        Attachments

          Activity

          Hide
          larsskj larsskj added a comment -

          I would like to raise this problem to major - at least: This bug has been introduced with the upgrade from 2.0.2 to 2.0.3 and it's not only cosmetical.

          It also means that xUnit don't fail build jobs when it finds failures in the jUnit resultfiles - meaning that continuous deployments will deploy because they're not stopped by xUnit.

          I can add that even though xUnit warns that all test reports are empty, they're parsed correctly and can be seen in the usual Jenkins testresult view for the job.

          Show
          larsskj larsskj added a comment - I would like to raise this problem to major - at least: This bug has been introduced with the upgrade from 2.0.2 to 2.0.3 and it's not only cosmetical. It also means that xUnit don't fail build jobs when it finds failures in the jUnit resultfiles - meaning that continuous deployments will deploy because they're not stopped by xUnit. I can add that even though xUnit warns that all test reports are empty, they're parsed correctly and can be seen in the usual Jenkins testresult view for the job.
          Hide
          nfalco Nikolas Falco added a comment - - edited

          I agree, I will look this weekend.

          Show
          nfalco Nikolas Falco added a comment - - edited I agree, I will look this weekend.
          Hide
          nfalco Nikolas Falco added a comment -

          With Jenkins 1.129, all plugins updated expect junit 1.20
          I'm not able to reproduce this issue. With your script I got this log:

          Running in Durability level: MAX_SURVIVABILITY
          [Pipeline] node
          Running on Jenkins in /Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline
          [Pipeline] {
          [Pipeline] deleteDir
          [Pipeline] sleep
          Sleeping for 1 s
          [Pipeline] parallel
          [Pipeline] [dateiEins] { (Branch: dateiEins)
          [Pipeline] [dateiZwei] { (Branch: dateiZwei)
          [Pipeline] [dateiEins] stage
          [Pipeline] [dateiEins] { (generate xml file 1)
          [Pipeline] [dateiZwei] stage
          [Pipeline] [dateiZwei] { (generate xml file 2)
          [Pipeline] [dateiEins] dir
          [dateiEins] Running in /Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file1
          [Pipeline] [dateiEins] {
          [Pipeline] [dateiZwei] dir
          [dateiZwei] Running in /Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file2
          [Pipeline] [dateiZwei] {
          [Pipeline] [dateiEins] writeFile
          [Pipeline] [dateiZwei] writeFile
          [Pipeline] [dateiEins] sh
          [dateiEins] [file1] Running shell script
          [Pipeline] [dateiZwei] sh
          [dateiEins] + ls -l
          [dateiEins] total 16
          [dateiEins] -rw-r--r--  1 nikolasfalco  staff  4511 Jun 30 12:24 TEST-de.edict.services.portaltests.SlotSessionTest_ES.xml
          [dateiZwei] [file2] Running shell script
          [dateiZwei] + ls -l
          [dateiZwei] total 16
          [dateiZwei] -rw-r--r--  1 nikolasfalco  staff  5665 Jun 30 12:24 TEST-de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests--IoM_default.xml
          [Pipeline] [dateiEins] xunit
          [dateiEins] INFO: Starting to record.
          [dateiEins] INFO: Processing JUnit
          [dateiEins] INFO: [JUnit] - 1 test report file(s) were found with the pattern 'TEST-*.xml' relative to '/Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file1' for the testing framework 'JUnit'.
          [dateiEins] INFO: Check 'Failed Tests' threshold.
          [dateiEins] INFO: The total number of tests for this category exceeds the specified "failure" threshold value.
          [dateiEins] INFO: Setting the build status to FAILURE
          [dateiEins] INFO: Stopping recording.
          [Pipeline] [dateiZwei] xunit
          [dateiZwei] INFO: Starting to record.
          [dateiZwei] INFO: Processing JUnit
          [dateiZwei] INFO: [JUnit] - 1 test report file(s) were found with the pattern 'TEST-*.xml' relative to '/Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file2' for the testing framework 'JUnit'.
          [Pipeline] [dateiEins] }
          [Pipeline] [dateiEins] // dir
          [Pipeline] [dateiEins] }
          [Pipeline] [dateiZwei] }
          [Pipeline] [dateiEins] // stage
          [Pipeline] [dateiZwei] // dir
          [Pipeline] [dateiEins] }
          [Pipeline] [dateiZwei] }
          [Pipeline] [dateiZwei] // stage
          [Pipeline] [dateiZwei] }
          [dateiZwei] Failed in branch dateiZwei
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          

          In case could you please attach a zip that contains the content of your workspace folder produced by the example you post in the issue?

          Show
          nfalco Nikolas Falco added a comment - With Jenkins 1.129, all plugins updated expect junit 1.20 I'm not able to reproduce this issue. With your script I got this log: Running in Durability level: MAX_SURVIVABILITY [Pipeline] node Running on Jenkins in /Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline [Pipeline] { [Pipeline] deleteDir [Pipeline] sleep Sleeping for 1 s [Pipeline] parallel [Pipeline] [dateiEins] { (Branch: dateiEins) [Pipeline] [dateiZwei] { (Branch: dateiZwei) [Pipeline] [dateiEins] stage [Pipeline] [dateiEins] { (generate xml file 1) [Pipeline] [dateiZwei] stage [Pipeline] [dateiZwei] { (generate xml file 2) [Pipeline] [dateiEins] dir [dateiEins] Running in /Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file1 [Pipeline] [dateiEins] { [Pipeline] [dateiZwei] dir [dateiZwei] Running in /Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file2 [Pipeline] [dateiZwei] { [Pipeline] [dateiEins] writeFile [Pipeline] [dateiZwei] writeFile [Pipeline] [dateiEins] sh [dateiEins] [file1] Running shell script [Pipeline] [dateiZwei] sh [dateiEins] + ls -l [dateiEins] total 16 [dateiEins] -rw-r--r-- 1 nikolasfalco staff 4511 Jun 30 12:24 TEST-de.edict.services.portaltests.SlotSessionTest_ES.xml [dateiZwei] [file2] Running shell script [dateiZwei] + ls -l [dateiZwei] total 16 [dateiZwei] -rw-r--r-- 1 nikolasfalco staff 5665 Jun 30 12:24 TEST-de.edict.services.backofficetests.bonusoffer.BonusOfferDepositTests--IoM_default.xml [Pipeline] [dateiEins] xunit [dateiEins] INFO: Starting to record. [dateiEins] INFO: Processing JUnit [dateiEins] INFO: [JUnit] - 1 test report file(s) were found with the pattern 'TEST-*.xml' relative to '/Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file1' for the testing framework 'JUnit'. [dateiEins] INFO: Check 'Failed Tests' threshold. [dateiEins] INFO: The total number of tests for this category exceeds the specified "failure" threshold value. [dateiEins] INFO: Setting the build status to FAILURE [dateiEins] INFO: Stopping recording. [Pipeline] [dateiZwei] xunit [dateiZwei] INFO: Starting to record. [dateiZwei] INFO: Processing JUnit [dateiZwei] INFO: [JUnit] - 1 test report file(s) were found with the pattern 'TEST-*.xml' relative to '/Users/nikolasfalco/git/xunit-plugin/work/workspace/pipeline/file2' for the testing framework 'JUnit'. [Pipeline] [dateiEins] } [Pipeline] [dateiEins] // dir [Pipeline] [dateiEins] } [Pipeline] [dateiZwei] } [Pipeline] [dateiEins] // stage [Pipeline] [dateiZwei] // dir [Pipeline] [dateiEins] } [Pipeline] [dateiZwei] } [Pipeline] [dateiZwei] // stage [Pipeline] [dateiZwei] } [dateiZwei] Failed in branch dateiZwei [Pipeline] // parallel [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline In case could you please attach a zip that contains the content of your workspace folder produced by the example you post in the issue?
          Hide
          nfalco Nikolas Falco added a comment -

          I got it. It happens on next builds

          Show
          nfalco Nikolas Falco added a comment - I got it. It happens on next builds

            People

            • Assignee:
              nfalco Nikolas Falco
              Reporter:
              sfin Sven Finsterwalder
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: