Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-5674

Some NUnit test results show blank "Test name" attribute in web view

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: nunit-plugin
    • Labels:
      None
    • Environment:
      Linux
    • Similar Issues:

      Description

      We have various NUnit tests that run in our project, that create .xml files for the results. Hudson has been able to use these quite well in the past, showcasing the results in web format - i.e. the list where you have the various test names, how long they took to run, and whether they passed or not.

      However, we recently uploaded a new project, which Hudson seems to have issues with - in looking at the results page, there is a list of all the tests - but the "Test name" fields are blank for all of them. They're ordered in the same order that they appear in the .xml report file, and the web page accurately shows whether they passed or not, as well as how long they took to run - but the test names simply aren't there.

      In comparing the new project with the old, there seems to be only one difference - the new ones get their test names using the NUnit "TestCaseSource" attribute. We're guessing that perhaps Hudson is unable to read the test name results for these, due to them having 4 levels of hierarchy, rather than 3 as our other ones do.

      Is there any other information you would need to have to help diagnose this? Let me know in some fashion, and I'll do what I can to get the information for you. I can also attach a screen shot, if a visual aid would help.

        Attachments

        1. APITests.cs
          2 kB
        2. BrowserNamesTest1.cs
          0.8 kB
        3. hudson-screenshot.jpg
          hudson-screenshot.jpg
          31 kB
        4. nunit_remote.xml
          87 kB
        5. nunit_remote.xml
          20 kB
        6. nunit-to-junit.xsl
          2 kB
        7. nunit-to-junit-20110330.xsl
          3 kB
        8. nunit-to-junit-20110330b.xsl
          3 kB
        9. TestResult_2.xml
          7 kB
        10. TestResult.xml
          4 kB
        11. VisualTests.cs
          15 kB

          Activity

          Hide
          timotei Timotei Dolean added a comment - - edited

          Hello,

          This issue doesn't seem to be fixed, here's the simplest usecase that still fails:

          namespace Sample.Tests
          {
             public class SampleTests
             {
                  [TestCaseSource(nameof(NamedTestCaseDataInput))]
                  public void NamedTestCaseData(string input)
                  {
                        Assert.IsNotEmpty(input);
                  }
                  public static IEnumerable NamedTestCaseDataInput()
                  {
                       return new[]
                       {
                           new TestCaseData("input1").SetName("first input"),
                           new TestCaseData("input2").SetName("second input"),
                           new TestCaseData("input3").SetName("third input")
                       };
                  }
              }
          

           

           Now, this generates the following NUnit result XML:

          <?xml version="1.0" encoding="utf-8" standalone="no"?>
          <!--This file represents the results of running a test suite-->
          <test-results name="" total="3" errors="0" failures="0" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2017-11-06" time="09:40:49">
            <environment nunit-version="1.0.0.0" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 10.0.15063.0" platform="Win32NT" cwd="F:\" machine-name="10" user="t" user-domain="U" />
            <culture-info current-culture="en-US" current-uiculture="en-US" />
            <test-suite type="Assembly" name="F:\test.dll" executed="True" result="Success" success="True" time="0.027" asserts="3">
              <properties>
                <property name="_PID" value="14568" />
                <property name="_APPDOMAIN" value="test-domain-" />
              </properties>
              <results>
                <test-suite type="TestSuite" name="Sample" executed="True" result="Success" success="True" time="0.021" asserts="3">
                  <results>
                    <test-suite type="TestSuite" name="Tests" executed="True" result="Success" success="True" time="0.021" asserts="3">
                      <results>
                        <test-suite type="TestFixture" name="SampleTests" executed="True" result="Success" success="True" time="0.019" asserts="3">
                          <results>
                            <test-suite type="ParameterizedMethod" name="NamedTestCaseData" executed="True" result="Success" success="True" time="0.017" asserts="3">
                              <results>
                                <test-case name="Sample.Tests.SampleTests.first input" executed="True" result="Success" success="True" time="0.010" asserts="1" />
                                <test-case name="Sample.Tests.SampleTests.second input" executed="True" result="Success" success="True" time="0.000" asserts="1" />
                                <test-case name="Sample.Tests.SampleTests.third input" executed="True" result="Success" success="True" time="0.000" asserts="1" />
                              </results>
                            </test-suite>
                          </results>
                        </test-suite>
                      </results>
                    </test-suite>
                  </results>
                </test-suite>
              </results>
            </test-suite>
          </test-results>
          

          Which in turn gets transformed to the following JUnit Result:

          <?xml version="1.0"?>
          <testsuites>
            <testsuite name="NamedTestCaseData" tests="3" time="0.017" failures="0" errors="0" skipped="0">
              <testcase classname="NamedTestCaseData" name="Sample.Tests.SampleTests.first input" time="0.010"/>
              <testcase classname="NamedTestCaseData" name="Sample.Tests.SampleTests.second input" time="0.000"/>
              <testcase classname="NamedTestCaseData" name="Sample.Tests.SampleTests.third input" time="0.000"/>
            </testsuite>
          </testsuites>
          

          However, the same test, without using SetName, would bring the following JUnit Result:

          <?xml version="1.0"?>
          <testsuites>
            <testsuite name="Sample.Tests.SampleTests" tests="3" time="0.017" failures="0" errors="0" skipped="0">
              <testcase classname="Sample.Tests.SampleTests" name="NamedTestCaseData(&quot;input1&quot;)" time="0.011"/>
              <testcase classname="Sample.Tests.SampleTests" name="NamedTestCaseData(&quot;input2&quot;)" time="0.000"/>
              <testcase classname="Sample.Tests.SampleTests" name="NamedTestCaseData(&quot;input3&quot;)" time="0.000"/>
            </testsuite>
          </testsuites>
          

          What it boils down is that the following bits of NUnit results are not parsed correctly:

          <test-suite type="ParameterizedMethod" name="NamedTestCaseData" executed="True" result="Success" success="True" time="0.017" asserts="3">
              <results>
                  <test-case name="Sample.Tests.SampleTests.first input" executed="True" result="Success" success="True" time="0.010" asserts="1" />
                  <test-case name="Sample.Tests.SampleTests.second input" executed="True" result="Success" success="True" time="0.000" asserts="1" />
                  <test-case name="Sample.Tests.SampleTests.third input" executed="True" result="Success" success="True" time="0.000" asserts="1" />
              </results>
          </test-suite>
          

          but the other ones, are:

          <test-suite type="ParameterizedMethod" name="NamedTestCaseData" executed="True" result="Success" success="True" time="0.017" asserts="3">
              <results>
                  <test-case name="Sample.Tests.SampleTests.NamedTestCaseData(&quot;input1&quot;)" executed="True" result="Success" success="True" time="0.011" asserts="1" />
                  <test-case name="Sample.Tests.SampleTests.NamedTestCaseData(&quot;input2&quot;)" executed="True" result="Success" success="True" time="0.000" asserts="1" />
                  <test-case name="Sample.Tests.SampleTests.NamedTestCaseData(&quot;input3&quot;)" executed="True" result="Success" success="True" time="0.000" asserts="1" />
              </results>
          </test-suite>
          

          So, instead of putting the proper classname and test name, it puts the method name.

          I'll do a Pull Request with the proper test case + a fix

          Show
          timotei Timotei Dolean added a comment - - edited Hello, This issue doesn't seem to be fixed, here's the simplest usecase that still fails: namespace Sample.Tests {     public class SampleTests    {         [TestCaseSource(nameof(NamedTestCaseDataInput))]         public void NamedTestCaseData(string input)         {               Assert.IsNotEmpty(input);         }         public static IEnumerable NamedTestCaseDataInput()         {               return new []              { new TestCaseData( "input1" ).SetName( "first input" ), new TestCaseData( "input2" ).SetName( "second input" ), new TestCaseData( "input3" ).SetName( "third input" )              };         }     }    Now, this generates the following NUnit result XML: <?xml version= "1.0" encoding= "utf-8" standalone= "no" ?> <!--This file represents the results of running a test suite--> <test-results name= "" total=" 3 " errors=" 0 " failures=" 0 " not-run=" 0 " inconclusive=" 0 " ignored=" 0 " skipped=" 0 " invalid=" 0 " date=" 2017-11-06 " time=" 09:40:49"> <environment nunit-version= "1.0.0.0" clr-version= "4.0.30319.42000" os-version= "Microsoft Windows NT 10.0.15063.0" platform= "Win32NT" cwd= "F:\" machine-name= "10" user= "t" user-domain= "U" /> <culture-info current-culture= "en-US" current-uiculture= "en-US" /> <test-suite type= "Assembly" name= "F:\test.dll" executed= "True" result= "Success" success= "True" time= "0.027" asserts= "3" > <properties> <property name= "_PID" value= "14568" /> <property name= "_APPDOMAIN" value= "test-domain-" /> </properties> <results> <test-suite type= "TestSuite" name= "Sample" executed= "True" result= "Success" success= "True" time= "0.021" asserts= "3" > <results> <test-suite type= "TestSuite" name= "Tests" executed= "True" result= "Success" success= "True" time= "0.021" asserts= "3" > <results> <test-suite type= "TestFixture" name= "SampleTests" executed= "True" result= "Success" success= "True" time= "0.019" asserts= "3" > <results> <test-suite type= "ParameterizedMethod" name= "NamedTestCaseData" executed= "True" result= "Success" success= "True" time= "0.017" asserts= "3" > <results> <test-case name= "Sample.Tests.SampleTests.first input" executed= "True" result= "Success" success= "True" time= "0.010" asserts= "1" /> <test-case name= "Sample.Tests.SampleTests.second input" executed= "True" result= "Success" success= "True" time= "0.000" asserts= "1" /> <test-case name= "Sample.Tests.SampleTests.third input" executed= "True" result= "Success" success= "True" time= "0.000" asserts= "1" /> </results> </test-suite> </results> </test-suite> </results> </test-suite> </results> </test-suite> </results> </test-suite> </test-results> Which in turn gets transformed to the following JUnit Result: <?xml version= "1.0" ?> <testsuites> <testsuite name= "NamedTestCaseData" tests= "3" time= "0.017" failures= "0" errors= "0" skipped= "0" > <testcase classname= "NamedTestCaseData" name= "Sample.Tests.SampleTests.first input" time= "0.010" /> <testcase classname= "NamedTestCaseData" name= "Sample.Tests.SampleTests.second input" time= "0.000" /> <testcase classname= "NamedTestCaseData" name= "Sample.Tests.SampleTests.third input" time= "0.000" /> </testsuite> </testsuites> However, the same test, without using SetName, would bring the following JUnit Result: <?xml version= "1.0" ?> <testsuites> <testsuite name= "Sample.Tests.SampleTests" tests= "3" time= "0.017" failures= "0" errors= "0" skipped= "0" > <testcase classname= "Sample.Tests.SampleTests" name= "NamedTestCaseData(&quot;input1&quot;)" time= "0.011" /> <testcase classname= "Sample.Tests.SampleTests" name= "NamedTestCaseData(&quot;input2&quot;)" time= "0.000" /> <testcase classname= "Sample.Tests.SampleTests" name= "NamedTestCaseData(&quot;input3&quot;)" time= "0.000" /> </testsuite> </testsuites> What it boils down is that the following bits of NUnit results are not parsed correctly: <test-suite type= "ParameterizedMethod" name= "NamedTestCaseData" executed= "True" result= "Success" success= "True" time= "0.017" asserts= "3" > <results> <test-case name= "Sample.Tests.SampleTests.first input" executed= "True" result= "Success" success= "True" time= "0.010" asserts= "1" /> <test-case name= "Sample.Tests.SampleTests.second input" executed= "True" result= "Success" success= "True" time= "0.000" asserts= "1" /> <test-case name= "Sample.Tests.SampleTests.third input" executed= "True" result= "Success" success= "True" time= "0.000" asserts= "1" /> </results> </test-suite> but the other ones, are: <test-suite type= "ParameterizedMethod" name= "NamedTestCaseData" executed= "True" result= "Success" success= "True" time= "0.017" asserts= "3" > <results> <test-case name= "Sample.Tests.SampleTests.NamedTestCaseData(&quot;input1&quot;)" executed= "True" result= "Success" success= "True" time= "0.011" asserts= "1" /> <test-case name= "Sample.Tests.SampleTests.NamedTestCaseData(&quot;input2&quot;)" executed= "True" result= "Success" success= "True" time= "0.000" asserts= "1" /> <test-case name= "Sample.Tests.SampleTests.NamedTestCaseData(&quot;input3&quot;)" executed= "True" result= "Success" success= "True" time= "0.000" asserts= "1" /> </results> </test-suite> So, instead of putting the proper classname and test name, it puts the method name. I'll do a Pull Request with the proper test case + a fix
          Hide
          timotei Timotei Dolean added a comment -
          Show
          timotei Timotei Dolean added a comment - Here's the proper fix: https://github.com/jenkinsci/nunit-plugin/pull/16
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Timotei Dolean
          Path:
          src/main/resources/hudson/plugins/nunit/nunit-to-junit.xsl
          src/test/java/hudson/plugins/nunit/NUnitToJUnitXslTest.java
          src/test/resources/hudson/plugins/nunit/JUnit-issue5674-setname.xml
          src/test/resources/hudson/plugins/nunit/NUnit-issue5674-setname.xml
          http://jenkins-ci.org/commit/nunit-plugin/0e82f81888c82ead07d1e4e55f9f1eef6c71c7b6
          Log:
          Handle named parametrized tests

          When using .SetName() on a TestCaseData
          the end result in the NUnit results file
          won't be using the format of a method call

          Issue: JENKINS-5674

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Timotei Dolean Path: src/main/resources/hudson/plugins/nunit/nunit-to-junit.xsl src/test/java/hudson/plugins/nunit/NUnitToJUnitXslTest.java src/test/resources/hudson/plugins/nunit/JUnit-issue5674-setname.xml src/test/resources/hudson/plugins/nunit/NUnit-issue5674-setname.xml http://jenkins-ci.org/commit/nunit-plugin/0e82f81888c82ead07d1e4e55f9f1eef6c71c7b6 Log: Handle named parametrized tests When using .SetName() on a TestCaseData the end result in the NUnit results file won't be using the format of a method call Issue: JENKINS-5674
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Alex Earl
          Path:
          src/main/resources/hudson/plugins/nunit/nunit-to-junit.xsl
          src/test/java/hudson/plugins/nunit/NUnitToJUnitXslTest.java
          src/test/resources/hudson/plugins/nunit/JUnit-issue5674-setname.xml
          src/test/resources/hudson/plugins/nunit/NUnit-issue5674-setname.xml
          http://jenkins-ci.org/commit/nunit-plugin/14322b5f5ee9d7caa065303b2f79d1e72deb733f
          Log:
          Merge pull request #16 from timotei/master

          Fix JENKINS-5674

          Compare: https://github.com/jenkinsci/nunit-plugin/compare/129accb1fd73...14322b5f5ee9

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Alex Earl Path: src/main/resources/hudson/plugins/nunit/nunit-to-junit.xsl src/test/java/hudson/plugins/nunit/NUnitToJUnitXslTest.java src/test/resources/hudson/plugins/nunit/JUnit-issue5674-setname.xml src/test/resources/hudson/plugins/nunit/NUnit-issue5674-setname.xml http://jenkins-ci.org/commit/nunit-plugin/14322b5f5ee9d7caa065303b2f79d1e72deb733f Log: Merge pull request #16 from timotei/master Fix JENKINS-5674 Compare: https://github.com/jenkinsci/nunit-plugin/compare/129accb1fd73...14322b5f5ee9
          Hide
          slide_o_mix Alex Earl added a comment -

          Fixed in 0.22

          Show
          slide_o_mix Alex Earl added a comment - Fixed in 0.22

            People

            • Assignee:
              slide_o_mix Alex Earl
              Reporter:
              telgabrowny telgabrowny
            • Votes:
              7 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: