Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-5674

Some NUnit test results show blank "Test name" attribute in web view

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: nunit-plugin
    • Labels:
      None
    • Environment:
      Linux
    • Similar Issues:

      Description

      We have various NUnit tests that run in our project, that create .xml files for the results. Hudson has been able to use these quite well in the past, showcasing the results in web format - i.e. the list where you have the various test names, how long they took to run, and whether they passed or not.

      However, we recently uploaded a new project, which Hudson seems to have issues with - in looking at the results page, there is a list of all the tests - but the "Test name" fields are blank for all of them. They're ordered in the same order that they appear in the .xml report file, and the web page accurately shows whether they passed or not, as well as how long they took to run - but the test names simply aren't there.

      In comparing the new project with the old, there seems to be only one difference - the new ones get their test names using the NUnit "TestCaseSource" attribute. We're guessing that perhaps Hudson is unable to read the test name results for these, due to them having 4 levels of hierarchy, rather than 3 as our other ones do.

      Is there any other information you would need to have to help diagnose this? Let me know in some fashion, and I'll do what I can to get the information for you. I can also attach a screen shot, if a visual aid would help.

        Attachments

        1. APITests.cs
          2 kB
        2. BrowserNamesTest1.cs
          0.8 kB
        3. hudson-screenshot.jpg
          hudson-screenshot.jpg
          31 kB
        4. nunit_remote.xml
          87 kB
        5. nunit_remote.xml
          20 kB
        6. nunit-to-junit.xsl
          2 kB
        7. nunit-to-junit-20110330.xsl
          3 kB
        8. nunit-to-junit-20110330b.xsl
          3 kB
        9. TestResult_2.xml
          7 kB
        10. TestResult.xml
          4 kB
        11. VisualTests.cs
          15 kB

          Activity

          Hide
          redsolo redsolo added a comment -

          It would be great if you could attach a file containing sample nunit xml that isnt parsed properly so i quickly can fix up a test case for this issue.

          Show
          redsolo redsolo added a comment - It would be great if you could attach a file containing sample nunit xml that isnt parsed properly so i quickly can fix up a test case for this issue.
          Hide
          telgabrowny telgabrowny added a comment -

          Here you go! This .xml file is the one that showcases this - the group of test results that aren't displaying correctly fall under the "APITest" suite name. The other test names show up fine in, in their results pages.

          Show
          telgabrowny telgabrowny added a comment - Here you go! This .xml file is the one that showcases this - the group of test results that aren't displaying correctly fall under the "APITest" suite name. The other test names show up fine in, in their results pages.
          Hide
          redsolo redsolo added a comment -

          If i look at the XML, you can see that you have a test suite hierarchy of:
          ModelCenterTest.APITests.APITest.<methods>

          But each test is reported as to have the following hierarchy:
          ModelCenterTest.APITests.<methods>

          Note that there are no "APITest" in the test case name whereas it exists in the test-suite hierarchy. Do you know why it looks like that?

          Any insight on how the code is structured would help me understanding this issue.

          <test-suite name="ModelCenterTest" executed="True" success="True" time="2" asserts="0">
          <results>
          <test-suite name="APITests" executed="True" success="True" time="2" asserts="0">
          <results>
          <test-suite name="APITest" executed="True" success="True" time="2" asserts="0">
          <results>
          <test-case name="ModelCenterTest.APITests.autoLink" executed="True" success="True" time="1.0" asserts="1" />
          <test-case name="ModelCenterTest.APITests.VBScriptAPI" executed="True" success="True" time="1.0" asserts="1" />

          Show
          redsolo redsolo added a comment - If i look at the XML, you can see that you have a test suite hierarchy of: ModelCenterTest.APITests.APITest.<methods> But each test is reported as to have the following hierarchy: ModelCenterTest.APITests.<methods> Note that there are no "APITest" in the test case name whereas it exists in the test-suite hierarchy. Do you know why it looks like that? Any insight on how the code is structured would help me understanding this issue. <test-suite name="ModelCenterTest" executed="True" success="True" time="2" asserts="0"> <results> <test-suite name="APITests" executed="True" success="True" time="2" asserts="0"> <results> <test-suite name="APITest" executed="True" success="True" time="2" asserts="0"> <results> <test-case name="ModelCenterTest.APITests.autoLink" executed="True" success="True" time="1.0" asserts="1" /> <test-case name="ModelCenterTest.APITests.VBScriptAPI" executed="True" success="True" time="1.0" asserts="1" />
          Hide
          redsolo redsolo added a comment -

          What version of NUnit are you using?

          Show
          redsolo redsolo added a comment - What version of NUnit are you using?
          Hide
          telgabrowny telgabrowny added a comment -

          Redsolo,

          We're not really sure why "APITest" wasn't in the test case name - it's not like these tests are generated any differently than some of our other ones, which do show their full names properly. We decided to experiment with how the test names are being generated, and now have this in our code:

          testName = testName.Substring(0, testName.Length - 4);
          yield return new TestCaseData(f).SetName("APITests." + testName);

          We're waiting on the next build to pass, so that the tests can run. If this happens to be a workaround, then we'll use it for now - but any efforts on your end are very appreciated, regardless of how it turns out.

          As far as the version of NUnit that we're using, we're utilizing 2.5.3.9345.

          Show
          telgabrowny telgabrowny added a comment - Redsolo, We're not really sure why "APITest" wasn't in the test case name - it's not like these tests are generated any differently than some of our other ones, which do show their full names properly. We decided to experiment with how the test names are being generated, and now have this in our code: testName = testName.Substring(0, testName.Length - 4); yield return new TestCaseData(f).SetName("APITests." + testName); We're waiting on the next build to pass, so that the tests can run. If this happens to be a workaround, then we'll use it for now - but any efforts on your end are very appreciated, regardless of how it turns out. As far as the version of NUnit that we're using, we're utilizing 2.5.3.9345.
          Hide
          redsolo redsolo added a comment -

          You should not have to do any workaround, I do have a solution in place but Im still not sure which test name to use (ie with APITest or not).

          Is it possible you could share the test fixture class (without any logic) so we can have a look at the class (namespace, class and method names? Is there any other logic for collecting the tests in your assembly?

          I have been discussing this on the NUnit discussion list, and the developers seems to be stumped also. If you want you can share your findings there also. http://groups.google.com/group/nunit-discuss/browse_thread/thread/cca1fcf29c878f38

          Show
          redsolo redsolo added a comment - You should not have to do any workaround, I do have a solution in place but Im still not sure which test name to use (ie with APITest or not). Is it possible you could share the test fixture class (without any logic) so we can have a look at the class (namespace, class and method names? Is there any other logic for collecting the tests in your assembly? I have been discussing this on the NUnit discussion list, and the developers seems to be stumped also. If you want you can share your findings there also. http://groups.google.com/group/nunit-discuss/browse_thread/thread/cca1fcf29c878f38
          Hide
          telgabrowny telgabrowny added a comment -

          Redsolo,

          I've attached two .cs files that go into this particular NUnit test suite - in this case, APITests (the one that's having issues), and VisualTests, whose names appear just fine in Hudson. I actually used the APITests one to create the VisualTests file, since they're pretty similar.

          Note, that the APITests have our current attempt at a workaround (like in the line of code I mentioned earlier) - the tests haven't run since then, to know if that worked. For clarification, just remove "APITests" from:

          yield return new TestCaseData(f).SetName("APITests." + testName);

          and that's how the code was, when this issue came up.

          Let me know if you need anything else,

          ~T

          Show
          telgabrowny telgabrowny added a comment - Redsolo, I've attached two .cs files that go into this particular NUnit test suite - in this case, APITests (the one that's having issues), and VisualTests, whose names appear just fine in Hudson. I actually used the APITests one to create the VisualTests file, since they're pretty similar. Note, that the APITests have our current attempt at a workaround (like in the line of code I mentioned earlier) - the tests haven't run since then, to know if that worked. For clarification, just remove "APITests" from: yield return new TestCaseData(f).SetName("APITests." + testName); and that's how the code was, when this issue came up. Let me know if you need anything else, ~T
          Hide
          telgabrowny telgabrowny added a comment -

          Redsolo,

          This is the new .xml file that was generated after we explicitly set the test name in our suite to match what Hudson seemed to need, but it still has the same issues as before. Let us know if there's anything else you need,

          ~T

          Show
          telgabrowny telgabrowny added a comment - Redsolo, This is the new .xml file that was generated after we explicitly set the test name in our suite to match what Hudson seemed to need, but it still has the same issues as before. Let us know if there's anything else you need, ~T
          Hide
          samweiss samweiss added a comment -

          Hi!

          I beleive I'm having a similar(or same issue), caused by using parameterized tests.

          I'm uploading the xml file, and a sample test that will reproduce the problem (so that you can see how the output is created).

          Please let me know if you need help with this. I might be able to have some time to help.

          Thanks!

          Sam

          Show
          samweiss samweiss added a comment - Hi! I beleive I'm having a similar(or same issue), caused by using parameterized tests. I'm uploading the xml file, and a sample test that will reproduce the problem (so that you can see how the output is created). Please let me know if you need help with this. I might be able to have some time to help. Thanks! Sam
          Hide
          samweiss samweiss added a comment -

          This is a sample test that will create the output the nunit parser is having trouble processing, as well as the test output.

          Thanks!

          Sam

          Show
          samweiss samweiss added a comment - This is a sample test that will create the output the nunit parser is having trouble processing, as well as the test output. Thanks! Sam
          Hide
          samweiss samweiss added a comment -

          Hi

          I found the problem. You will see in the test result file lines with this:
          <test-case name="GetBrowsersTest.BrowserNamesTest1.AnotherTest("IE")" executed="True" success="True" time="0.026" asserts="0" />

          The problem is that the parser cannot process the "("IE")". I've tried removing the "e;, and that didn't quite do it. It still had a problem with parenthesis. So when I changed it to

          <test-case name="GetBrowsersTest.BrowserNamesTest1.AnotherTest.IE" executed="True" success="True" time="0.026" asserts="0" />

          The file parsed just fine. Btw, these files are created by Nunit 2.5.

          This looks like it would be a very simple change to make for someone who knew the code. Unfortunately I don't know code, or how the hudson plugins work.

          Thanks!

          Sam

          Show
          samweiss samweiss added a comment - Hi I found the problem. You will see in the test result file lines with this: <test-case name="GetBrowsersTest.BrowserNamesTest1.AnotherTest("IE")" executed="True" success="True" time="0.026" asserts="0" /> The problem is that the parser cannot process the "("IE")". I've tried removing the "e;, and that didn't quite do it. It still had a problem with parenthesis. So when I changed it to <test-case name="GetBrowsersTest.BrowserNamesTest1.AnotherTest.IE" executed="True" success="True" time="0.026" asserts="0" /> The file parsed just fine. Btw, these files are created by Nunit 2.5. This looks like it would be a very simple change to make for someone who knew the code. Unfortunately I don't know code, or how the hudson plugins work. Thanks! Sam
          Hide
          m95lah m95lah added a comment - - edited

          I'm having the same issue, but from a different kind of setup.
          Basically we are just generating NUnit xml output that we want hudson to parse. We're not using NUnit itself at all.

          Anyway, when I have a result file that looks like this:

          <?xml version="1.0" standalone="no"?>
          <!--This file represents the results of running a test suite-->
          <test-results name="foo" total="2" failures="0" not-run="0" date="14/05/2010" time="10.44">
            <test-suite name="foo" success="True" time="10.44">
              <results>
                <test-case name="HelloWorld" executed="True" success="True" time="0" asserts="0"/>
                <test-case name="foo_GoodbyeWorld" executed="True" success="True" time="0" asserts="0"/>
              </results>
            </test-suite>
          </test-results>
          

          I get results that look like this:

          Basically it appears that any test-case that starts with the name of its test-suite will get its name dropped. In fact if I put "foo" anywhere in the test-case name it will be dropped!

          Show
          m95lah m95lah added a comment - - edited I'm having the same issue, but from a different kind of setup. Basically we are just generating NUnit xml output that we want hudson to parse. We're not using NUnit itself at all. Anyway, when I have a result file that looks like this: <?xml version="1.0" standalone="no"?> <!--This file represents the results of running a test suite--> <test-results name="foo" total="2" failures="0" not-run="0" date="14/05/2010" time="10.44"> <test-suite name="foo" success="True" time="10.44"> <results> <test-case name="HelloWorld" executed="True" success="True" time="0" asserts="0"/> <test-case name="foo_GoodbyeWorld" executed="True" success="True" time="0" asserts="0"/> </results> </test-suite> </test-results> I get results that look like this: Basically it appears that any test-case that starts with the name of its test-suite will get its name dropped. In fact if I put "foo" anywhere in the test-case name it will be dropped!
          Hide
          m95lah m95lah added a comment -

          A screenshot showing dropped test-case names.

          Show
          m95lah m95lah added a comment - A screenshot showing dropped test-case names.
          Hide
          telgabrowny telgabrowny added a comment -

          Redsolo,

          We wanted to let you know that m95lah's take on this seems to be correct. We renamed our APITest function to not be the same as the class name, and the subsequent nightly test showed the display bug as fixed. Yay for workarounds indeed!

          If there's anything you need from us to help you in pinning down the bug at fault so it can be fixed, just let me know. Good luck on fixing it,

          ~Taurik

          Show
          telgabrowny telgabrowny added a comment - Redsolo, We wanted to let you know that m95lah's take on this seems to be correct. We renamed our APITest function to not be the same as the class name, and the subsequent nightly test showed the display bug as fixed. Yay for workarounds indeed! If there's anything you need from us to help you in pinning down the bug at fault so it can be fixed, just let me know. Good luck on fixing it, ~Taurik
          Hide
          m95lah m95lah added a comment -

          I found a way to fix the problem. The nunit-to-junit.xsl file that does the transformation does pretty much exactly what I describe above (look at line 4):

            ...
          1 <xsl:for-each select="*/test-case[@time!='']">
          2   <xsl:variable name="testcaseName">
          3     <xsl:choose>
          4       <xsl:when test="contains(./@name, $assembly)">
          5         <xsl:value-of select="substring-after(./@name, concat($assembly,'.'))"/><!-- We either instantiate a "15" -->
          6       </xsl:when>
             ...
          

          Change it to (only line 4 changed):

            ...
          1 <xsl:for-each select="*/test-case[@time!='']">
          2   <xsl:variable name="testcaseName">
          3     <xsl:choose>
          4       <xsl:when test="starts-with(./@name, concat($assembly,'.'))">
          5         <xsl:value-of select="substring-after(./@name, concat($assembly,'.'))"/><!-- We either instantiate a "15" -->
          6       </xsl:when>
             ...
          

          And you will get at least better behaviour. Now it checks if the testcase name starts with "suite.", rather than just checking if it contains "suite", before choosing which way to show the test case name. (The substring-after function will return an empty string if the name does not start with "suite.", which is why we're getting the empty names with the old code.)

          Obviously I am not sure that this fix does not cause any other unpleasant side-effects.

          For those who want to use this fix, just change the nunit-to-junit.xsl file that is buried deep within your HUDSON_HOME/plugins/nunit directory.

          Show
          m95lah m95lah added a comment - I found a way to fix the problem. The nunit-to-junit.xsl file that does the transformation does pretty much exactly what I describe above (look at line 4): ... 1 <xsl:for-each select="*/test-case[@time!='']"> 2 <xsl:variable name="testcaseName"> 3 <xsl:choose> 4 <xsl:when test="contains(./@name, $assembly)"> 5 <xsl:value-of select="substring-after(./@name, concat($assembly,'.'))"/><!-- We either instantiate a "15" --> 6 </xsl:when> ... Change it to (only line 4 changed): ... 1 <xsl:for-each select="*/test-case[@time!='']"> 2 <xsl:variable name="testcaseName"> 3 <xsl:choose> 4 <xsl:when test="starts-with(./@name, concat($assembly,'.'))"> 5 <xsl:value-of select="substring-after(./@name, concat($assembly,'.'))"/><!-- We either instantiate a "15" --> 6 </xsl:when> ... And you will get at least better behaviour. Now it checks if the testcase name starts with "suite.", rather than just checking if it contains "suite", before choosing which way to show the test case name. (The substring-after function will return an empty string if the name does not start with "suite.", which is why we're getting the empty names with the old code.) Obviously I am not sure that this fix does not cause any other unpleasant side-effects. For those who want to use this fix, just change the nunit-to-junit.xsl file that is buried deep within your HUDSON_HOME/plugins/nunit directory.
          Hide
          yuvalr yuvalr added a comment -

          same issue here

          Show
          yuvalr yuvalr added a comment - same issue here
          Hide
          m95lah m95lah added a comment -

          Any chance of getting my fix (see my comment above) included at some stage?

          Show
          m95lah m95lah added a comment - Any chance of getting my fix (see my comment above) included at some stage?
          Hide
          emn13 emn13 added a comment -

          I'm seeing the same problem.

          The xsl transformation seems to be making various unwarranted assumptions. In addition to the above, when two namespaces are nested which have the same name, things get messy.

          The basic problem is that the idea that the substring before the test-suite name is the namespace isn't valid. The test-suite name may well occur earlier in the string at least, so string replace isn't a good strategy.

          I rewrote the xslt to determine the qualified namespace name by walking the xml tree. It also deals with parametrized tests. Finally, I cleaned up the xsl:for-each (removed unnecessary nesting) and the odd <failure> handling code (complex xsl:if logic was equivalent to xsl:apply-templates); that means that despite the new tree-walking template and the parametrized test stuff this version is actually slightly shorter.

          The order in which test cases are reported does slightly change, however; the new xsl transform lists the test-suites by document order, whereas the old code listed test results by document order. In practice this means that now A.B.C test results will be listed before A.B.C.D test results.

          Show
          emn13 emn13 added a comment - I'm seeing the same problem. The xsl transformation seems to be making various unwarranted assumptions. In addition to the above, when two namespaces are nested which have the same name, things get messy. The basic problem is that the idea that the substring before the test-suite name is the namespace isn't valid. The test-suite name may well occur earlier in the string at least, so string replace isn't a good strategy. I rewrote the xslt to determine the qualified namespace name by walking the xml tree. It also deals with parametrized tests. Finally, I cleaned up the xsl:for-each (removed unnecessary nesting) and the odd <failure> handling code (complex xsl:if logic was equivalent to xsl:apply-templates); that means that despite the new tree-walking template and the parametrized test stuff this version is actually slightly shorter. The order in which test cases are reported does slightly change, however; the new xsl transform lists the test-suites by document order, whereas the old code listed test results by document order. In practice this means that now A.B.C test results will be listed before A.B.C.D test results.
          Hide
          emn13 emn13 added a comment -

          Oh, m95lah's solution is only a partial fix; in the code he quotes the variable $assembly (which is actually the test-suite-name, not the assembly) can be incorrectly set do to the substring thing mentioned before.

          The contributor agreement looks scary, I'll check with the appropriate person at my company but I'll probably just release the new version to public domain instead of going through the bother of signing an agreement I don't understand.

          Show
          emn13 emn13 added a comment - Oh, m95lah's solution is only a partial fix; in the code he quotes the variable $assembly (which is actually the test-suite-name, not the assembly) can be incorrectly set do to the substring thing mentioned before. The contributor agreement looks scary, I'll check with the appropriate person at my company but I'll probably just release the new version to public domain instead of going through the bother of signing an agreement I don't understand.
          Hide
          emn13 emn13 added a comment -

          Yeah, it's fine - the attached version of nunit-to-junit.xsl that fixes this issue is released into public domain. Do with it what you will (i.e., include it eigen if you want it).

          Show
          emn13 emn13 added a comment - Yeah, it's fine - the attached version of nunit-to-junit.xsl that fixes this issue is released into public domain. Do with it what you will (i.e., include it eigen if you want it).
          Hide
          redsolo redsolo added a comment -

          Im going to look into this issue in the next days.

          Show
          redsolo redsolo added a comment - Im going to look into this issue in the next days.
          Hide
          redsolo redsolo added a comment -

          Thanks for the patch as Ive now used it and resolved this issue. Sorry that it took so long time to fix it. Release 0.13 containing the fix is being pushed out right now.

          Thanks again for all comments.

          Show
          redsolo redsolo added a comment - Thanks for the patch as Ive now used it and resolved this issue. Sorry that it took so long time to fix it. Release 0.13 containing the fix is being pushed out right now. Thanks again for all comments.
          Hide
          emn13 emn13 added a comment -

          This is still not really fixed. The new xslt file still doesn't deal correctly with classes nested in namespaces that have the same name (this is due to the fact that substring-after is used), and I'm seeing some tests that for some reason are incorrectly placed in a package "(root)", despite having a namespace. In addition, the java.exe process takes several minutes of CPU time to process the test results now. If you use the attached nunit-to-junit.xsl (dated 24/Aug/10 6:10 AM 2 kB), both these problems are avoided, although the xslt transform is still unusually slow (which XSLT engine is doing the transform?)

          Show
          emn13 emn13 added a comment - This is still not really fixed. The new xslt file still doesn't deal correctly with classes nested in namespaces that have the same name (this is due to the fact that substring-after is used), and I'm seeing some tests that for some reason are incorrectly placed in a package "(root)", despite having a namespace. In addition, the java.exe process takes several minutes of CPU time to process the test results now. If you use the attached nunit-to-junit.xsl (dated 24/Aug/10 6:10 AM 2 kB), both these problems are avoided, although the xslt transform is still unusually slow (which XSLT engine is doing the transform?)
          Hide
          emn13 emn13 added a comment -

          See previous comment; the version included in 0.13 is way too slow and still produces incorrect results.

          Show
          emn13 emn13 added a comment - See previous comment; the version included in 0.13 is way too slow and still produces incorrect results.
          Hide
          redsolo redsolo added a comment -

          I just saw that i have an xml file that produces a "(root)" package report with the current XSL file. I will look into it.

          re: "classes nested in namespaces that have the same name" - could you share any example on to reproduce this?

          The reason why I adjusted the attached XSL file is that it returned strange package names. I did a test with it right now, and it generated the below package names. I would rather have the namespace names instad of the assembly name.

          (root)
          ./MyTests.dll
          C:\Users\Samw\Documents\Visual Studio 2008\Projects\GetBrowsersTest\GetBrowsersTest\bin\Debug\GetBrowsersTest.dll.GetBrowsersTest.BrowserNamesTest1
          UnitTests.dll.UnitTests 
          /home/erik/coding/test/nunittests/Tests.dll.UnitTests.UnitTests 
          

          re: XSLT engine, im not sure. I havent specified any, but i will do a dig and see what i come up with. Know of any fast xslt engines?

          Show
          redsolo redsolo added a comment - I just saw that i have an xml file that produces a "(root)" package report with the current XSL file. I will look into it. re: "classes nested in namespaces that have the same name" - could you share any example on to reproduce this? The reason why I adjusted the attached XSL file is that it returned strange package names. I did a test with it right now, and it generated the below package names. I would rather have the namespace names instad of the assembly name. (root) ./MyTests.dll C:\Users\Samw\Documents\Visual Studio 2008\Projects\GetBrowsersTest\GetBrowsersTest\bin\Debug\GetBrowsersTest.dll.GetBrowsersTest.BrowserNamesTest1 UnitTests.dll.UnitTests /home/erik/coding/test/nunittests/Tests.dll.UnitTests.UnitTests re: XSLT engine, im not sure. I havent specified any, but i will do a dig and see what i come up with. Know of any fast xslt engines?
          Hide
          emn13 emn13 added a comment -

          I think the CPU slowdown was something else - it occured simultaneously because I had just updated to get this fix; but it still occurs and only occurs looking at 2 specific builds' test results (and I don't think the XSLT is saved and rerun anytime someone views the test results, right?). If I can reproduce that, I'll file a separate bug.

          Show
          emn13 emn13 added a comment - I think the CPU slowdown was something else - it occured simultaneously because I had just updated to get this fix; but it still occurs and only occurs looking at 2 specific builds' test results (and I don't think the XSLT is saved and rerun anytime someone views the test results, right?). If I can reproduce that, I'll file a separate bug.
          Hide
          emn13 emn13 added a comment -

          Ok, I had a look at the Assembly-names thingo you mention, and identified the cause. The three attached xml files (nunit_remote.xml, nunit_remote.xml and TestResult.xml) aren't valid nunit xml files according to the nunit schema:

          http://www.nunit.org/wiki/doku.php?id=dev:specs:xml_formats
          direct link:
          http://nunit.org/files/testresult_schema_25.txt

          I also verified that my copy of nunit includes the attribute "type" on all <test-suite> elements; so either these xml files weren't generated by nunit or they were generated by an old version that deviates from the documented 2.5 schema.

          I'm pretty sure it's easily fixed, the only reason I used those attributes is that it's not necessarily the case that the outermost test-suite is the only test-suite to contain such a long path. I can change the xslt to simply assume that the outermost test-suite is an assembly if it has no type attribute.

          Show
          emn13 emn13 added a comment - Ok, I had a look at the Assembly-names thingo you mention, and identified the cause. The three attached xml files (nunit_remote.xml, nunit_remote.xml and TestResult.xml) aren't valid nunit xml files according to the nunit schema: http://www.nunit.org/wiki/doku.php?id=dev:specs:xml_formats direct link: http://nunit.org/files/testresult_schema_25.txt I also verified that my copy of nunit includes the attribute "type" on all <test-suite> elements; so either these xml files weren't generated by nunit or they were generated by an old version that deviates from the documented 2.5 schema. I'm pretty sure it's easily fixed, the only reason I used those attributes is that it's not necessarily the case that the outermost test-suite is the only test-suite to contain such a long path. I can change the xslt to simply assume that the outermost test-suite is an assembly if it has no type attribute.
          Hide
          emn13 emn13 added a comment -

          Attached TestResult_2.xml; sample nunit testresults file as generated by nunit 2.5.9.10348.

          Show
          emn13 emn13 added a comment - Attached TestResult_2.xml; sample nunit testresults file as generated by nunit 2.5.9.10348.
          Hide
          emn13 emn13 added a comment -

          The TestResult_2.xml also happens to include a class called identically to it's namespace which is incorrectly parsed by the old XSLT. The new xslt doesn't include <skipped> elements - are these used by Jenkins? (Still feels a little weird calling the project jenkins :-D)

          Show
          emn13 emn13 added a comment - The TestResult_2.xml also happens to include a class called identically to it's namespace which is incorrectly parsed by the old XSLT. The new xslt doesn't include <skipped> elements - are these used by Jenkins? (Still feels a little weird calling the project jenkins :-D)
          Hide
          emn13 emn13 added a comment -

          Attached nunit-to-junit-20110330.xsl.

          This variant deals with skipped tests and omits the outermost test-suite name if it has no @type.

          Show
          emn13 emn13 added a comment - Attached nunit-to-junit-20110330.xsl. This variant deals with skipped tests and omits the outermost test-suite name if it has no @type.
          Hide
          emn13 emn13 added a comment -

          Attached nunit-to-junit-20110330b.xsl

          This version treats parametrized test fixtures sanely; i.e. each instance appears as a separate class in the namespace and has it's own methods.

          Show
          emn13 emn13 added a comment - Attached nunit-to-junit-20110330b.xsl This version treats parametrized test fixtures sanely; i.e. each instance appears as a separate class in the namespace and has it's own methods.
          Hide
          emn13 emn13 added a comment -

          Note, btw, that there are a few concepts which simple don't map very cleanly onto junit:

          Right now the last version for instance treats parametrized test cases as classes (intentionally) - if you don't do this, then tests with lots of test-cases swamp other tests on the same class. Really, with the fixed package>class>test three-level hierarchy, the most reasonable distribution over those levels depends on the way nunit is used. This could be reverted, of course to strictly follow namespace+innerclass>textfixture>testcases.

          For example of more messyness, if you introduce generic type parameters (possible both on text fixtures and test methods), or declare tests indirectly via inheritance (by applying a test-attribute on a base class method), things get confusing, fast.

          Show
          emn13 emn13 added a comment - Note, btw, that there are a few concepts which simple don't map very cleanly onto junit: Right now the last version for instance treats parametrized test cases as classes (intentionally) - if you don't do this, then tests with lots of test-cases swamp other tests on the same class. Really, with the fixed package>class>test three-level hierarchy, the most reasonable distribution over those levels depends on the way nunit is used. This could be reverted, of course to strictly follow namespace+innerclass>textfixture>testcases. For example of more messyness, if you introduce generic type parameters (possible both on text fixtures and test methods), or declare tests indirectly via inheritance (by applying a test-attribute on a base class method), things get confusing, fast.
          Hide
          slide_o_mix Alex Earl added a comment -

          Is this still an issue?

          Show
          slide_o_mix Alex Earl added a comment - Is this still an issue?
          Hide
          timotei Timotei Dolean added a comment - - edited

          Hello,

          This issue doesn't seem to be fixed, here's the simplest usecase that still fails:

          namespace Sample.Tests
          {
             public class SampleTests
             {
                  [TestCaseSource(nameof(NamedTestCaseDataInput))]
                  public void NamedTestCaseData(string input)
                  {
                        Assert.IsNotEmpty(input);
                  }
                  public static IEnumerable NamedTestCaseDataInput()
                  {
                       return new[]
                       {
                           new TestCaseData("input1").SetName("first input"),
                           new TestCaseData("input2").SetName("second input"),
                           new TestCaseData("input3").SetName("third input")
                       };
                  }
              }
          

           

           Now, this generates the following NUnit result XML:

          <?xml version="1.0" encoding="utf-8" standalone="no"?>
          <!--This file represents the results of running a test suite-->
          <test-results name="" total="3" errors="0" failures="0" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2017-11-06" time="09:40:49">
            <environment nunit-version="1.0.0.0" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 10.0.15063.0" platform="Win32NT" cwd="F:\" machine-name="10" user="t" user-domain="U" />
            <culture-info current-culture="en-US" current-uiculture="en-US" />
            <test-suite type="Assembly" name="F:\test.dll" executed="True" result="Success" success="True" time="0.027" asserts="3">
              <properties>
                <property name="_PID" value="14568" />
                <property name="_APPDOMAIN" value="test-domain-" />
              </properties>
              <results>
                <test-suite type="TestSuite" name="Sample" executed="True" result="Success" success="True" time="0.021" asserts="3">
                  <results>
                    <test-suite type="TestSuite" name="Tests" executed="True" result="Success" success="True" time="0.021" asserts="3">
                      <results>
                        <test-suite type="TestFixture" name="SampleTests" executed="True" result="Success" success="True" time="0.019" asserts="3">
                          <results>
                            <test-suite type="ParameterizedMethod" name="NamedTestCaseData" executed="True" result="Success" success="True" time="0.017" asserts="3">
                              <results>
                                <test-case name="Sample.Tests.SampleTests.first input" executed="True" result="Success" success="True" time="0.010" asserts="1" />
                                <test-case name="Sample.Tests.SampleTests.second input" executed="True" result="Success" success="True" time="0.000" asserts="1" />
                                <test-case name="Sample.Tests.SampleTests.third input" executed="True" result="Success" success="True" time="0.000" asserts="1" />
                              </results>
                            </test-suite>
                          </results>
                        </test-suite>
                      </results>
                    </test-suite>
                  </results>
                </test-suite>
              </results>
            </test-suite>
          </test-results>
          

          Which in turn gets transformed to the following JUnit Result:

          <?xml version="1.0"?>
          <testsuites>
            <testsuite name="NamedTestCaseData" tests="3" time="0.017" failures="0" errors="0" skipped="0">
              <testcase classname="NamedTestCaseData" name="Sample.Tests.SampleTests.first input" time="0.010"/>
              <testcase classname="NamedTestCaseData" name="Sample.Tests.SampleTests.second input" time="0.000"/>
              <testcase classname="NamedTestCaseData" name="Sample.Tests.SampleTests.third input" time="0.000"/>
            </testsuite>
          </testsuites>
          

          However, the same test, without using SetName, would bring the following JUnit Result:

          <?xml version="1.0"?>
          <testsuites>
            <testsuite name="Sample.Tests.SampleTests" tests="3" time="0.017" failures="0" errors="0" skipped="0">
              <testcase classname="Sample.Tests.SampleTests" name="NamedTestCaseData(&quot;input1&quot;)" time="0.011"/>
              <testcase classname="Sample.Tests.SampleTests" name="NamedTestCaseData(&quot;input2&quot;)" time="0.000"/>
              <testcase classname="Sample.Tests.SampleTests" name="NamedTestCaseData(&quot;input3&quot;)" time="0.000"/>
            </testsuite>
          </testsuites>
          

          What it boils down is that the following bits of NUnit results are not parsed correctly:

          <test-suite type="ParameterizedMethod" name="NamedTestCaseData" executed="True" result="Success" success="True" time="0.017" asserts="3">
              <results>
                  <test-case name="Sample.Tests.SampleTests.first input" executed="True" result="Success" success="True" time="0.010" asserts="1" />
                  <test-case name="Sample.Tests.SampleTests.second input" executed="True" result="Success" success="True" time="0.000" asserts="1" />
                  <test-case name="Sample.Tests.SampleTests.third input" executed="True" result="Success" success="True" time="0.000" asserts="1" />
              </results>
          </test-suite>
          

          but the other ones, are:

          <test-suite type="ParameterizedMethod" name="NamedTestCaseData" executed="True" result="Success" success="True" time="0.017" asserts="3">
              <results>
                  <test-case name="Sample.Tests.SampleTests.NamedTestCaseData(&quot;input1&quot;)" executed="True" result="Success" success="True" time="0.011" asserts="1" />
                  <test-case name="Sample.Tests.SampleTests.NamedTestCaseData(&quot;input2&quot;)" executed="True" result="Success" success="True" time="0.000" asserts="1" />
                  <test-case name="Sample.Tests.SampleTests.NamedTestCaseData(&quot;input3&quot;)" executed="True" result="Success" success="True" time="0.000" asserts="1" />
              </results>
          </test-suite>
          

          So, instead of putting the proper classname and test name, it puts the method name.

          I'll do a Pull Request with the proper test case + a fix

          Show
          timotei Timotei Dolean added a comment - - edited Hello, This issue doesn't seem to be fixed, here's the simplest usecase that still fails: namespace Sample.Tests {     public class SampleTests    {         [TestCaseSource(nameof(NamedTestCaseDataInput))]         public void NamedTestCaseData(string input)         {               Assert.IsNotEmpty(input);         }         public static IEnumerable NamedTestCaseDataInput()         {               return new []              { new TestCaseData( "input1" ).SetName( "first input" ), new TestCaseData( "input2" ).SetName( "second input" ), new TestCaseData( "input3" ).SetName( "third input" )              };         }     }    Now, this generates the following NUnit result XML: <?xml version= "1.0" encoding= "utf-8" standalone= "no" ?> <!--This file represents the results of running a test suite--> <test-results name= "" total=" 3 " errors=" 0 " failures=" 0 " not-run=" 0 " inconclusive=" 0 " ignored=" 0 " skipped=" 0 " invalid=" 0 " date=" 2017-11-06 " time=" 09:40:49"> <environment nunit-version= "1.0.0.0" clr-version= "4.0.30319.42000" os-version= "Microsoft Windows NT 10.0.15063.0" platform= "Win32NT" cwd= "F:\" machine-name= "10" user= "t" user-domain= "U" /> <culture-info current-culture= "en-US" current-uiculture= "en-US" /> <test-suite type= "Assembly" name= "F:\test.dll" executed= "True" result= "Success" success= "True" time= "0.027" asserts= "3" > <properties> <property name= "_PID" value= "14568" /> <property name= "_APPDOMAIN" value= "test-domain-" /> </properties> <results> <test-suite type= "TestSuite" name= "Sample" executed= "True" result= "Success" success= "True" time= "0.021" asserts= "3" > <results> <test-suite type= "TestSuite" name= "Tests" executed= "True" result= "Success" success= "True" time= "0.021" asserts= "3" > <results> <test-suite type= "TestFixture" name= "SampleTests" executed= "True" result= "Success" success= "True" time= "0.019" asserts= "3" > <results> <test-suite type= "ParameterizedMethod" name= "NamedTestCaseData" executed= "True" result= "Success" success= "True" time= "0.017" asserts= "3" > <results> <test-case name= "Sample.Tests.SampleTests.first input" executed= "True" result= "Success" success= "True" time= "0.010" asserts= "1" /> <test-case name= "Sample.Tests.SampleTests.second input" executed= "True" result= "Success" success= "True" time= "0.000" asserts= "1" /> <test-case name= "Sample.Tests.SampleTests.third input" executed= "True" result= "Success" success= "True" time= "0.000" asserts= "1" /> </results> </test-suite> </results> </test-suite> </results> </test-suite> </results> </test-suite> </results> </test-suite> </test-results> Which in turn gets transformed to the following JUnit Result: <?xml version= "1.0" ?> <testsuites> <testsuite name= "NamedTestCaseData" tests= "3" time= "0.017" failures= "0" errors= "0" skipped= "0" > <testcase classname= "NamedTestCaseData" name= "Sample.Tests.SampleTests.first input" time= "0.010" /> <testcase classname= "NamedTestCaseData" name= "Sample.Tests.SampleTests.second input" time= "0.000" /> <testcase classname= "NamedTestCaseData" name= "Sample.Tests.SampleTests.third input" time= "0.000" /> </testsuite> </testsuites> However, the same test, without using SetName, would bring the following JUnit Result: <?xml version= "1.0" ?> <testsuites> <testsuite name= "Sample.Tests.SampleTests" tests= "3" time= "0.017" failures= "0" errors= "0" skipped= "0" > <testcase classname= "Sample.Tests.SampleTests" name= "NamedTestCaseData(&quot;input1&quot;)" time= "0.011" /> <testcase classname= "Sample.Tests.SampleTests" name= "NamedTestCaseData(&quot;input2&quot;)" time= "0.000" /> <testcase classname= "Sample.Tests.SampleTests" name= "NamedTestCaseData(&quot;input3&quot;)" time= "0.000" /> </testsuite> </testsuites> What it boils down is that the following bits of NUnit results are not parsed correctly: <test-suite type= "ParameterizedMethod" name= "NamedTestCaseData" executed= "True" result= "Success" success= "True" time= "0.017" asserts= "3" > <results> <test-case name= "Sample.Tests.SampleTests.first input" executed= "True" result= "Success" success= "True" time= "0.010" asserts= "1" /> <test-case name= "Sample.Tests.SampleTests.second input" executed= "True" result= "Success" success= "True" time= "0.000" asserts= "1" /> <test-case name= "Sample.Tests.SampleTests.third input" executed= "True" result= "Success" success= "True" time= "0.000" asserts= "1" /> </results> </test-suite> but the other ones, are: <test-suite type= "ParameterizedMethod" name= "NamedTestCaseData" executed= "True" result= "Success" success= "True" time= "0.017" asserts= "3" > <results> <test-case name= "Sample.Tests.SampleTests.NamedTestCaseData(&quot;input1&quot;)" executed= "True" result= "Success" success= "True" time= "0.011" asserts= "1" /> <test-case name= "Sample.Tests.SampleTests.NamedTestCaseData(&quot;input2&quot;)" executed= "True" result= "Success" success= "True" time= "0.000" asserts= "1" /> <test-case name= "Sample.Tests.SampleTests.NamedTestCaseData(&quot;input3&quot;)" executed= "True" result= "Success" success= "True" time= "0.000" asserts= "1" /> </results> </test-suite> So, instead of putting the proper classname and test name, it puts the method name. I'll do a Pull Request with the proper test case + a fix
          Hide
          timotei Timotei Dolean added a comment -
          Show
          timotei Timotei Dolean added a comment - Here's the proper fix: https://github.com/jenkinsci/nunit-plugin/pull/16
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Timotei Dolean
          Path:
          src/main/resources/hudson/plugins/nunit/nunit-to-junit.xsl
          src/test/java/hudson/plugins/nunit/NUnitToJUnitXslTest.java
          src/test/resources/hudson/plugins/nunit/JUnit-issue5674-setname.xml
          src/test/resources/hudson/plugins/nunit/NUnit-issue5674-setname.xml
          http://jenkins-ci.org/commit/nunit-plugin/0e82f81888c82ead07d1e4e55f9f1eef6c71c7b6
          Log:
          Handle named parametrized tests

          When using .SetName() on a TestCaseData
          the end result in the NUnit results file
          won't be using the format of a method call

          Issue: JENKINS-5674

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Timotei Dolean Path: src/main/resources/hudson/plugins/nunit/nunit-to-junit.xsl src/test/java/hudson/plugins/nunit/NUnitToJUnitXslTest.java src/test/resources/hudson/plugins/nunit/JUnit-issue5674-setname.xml src/test/resources/hudson/plugins/nunit/NUnit-issue5674-setname.xml http://jenkins-ci.org/commit/nunit-plugin/0e82f81888c82ead07d1e4e55f9f1eef6c71c7b6 Log: Handle named parametrized tests When using .SetName() on a TestCaseData the end result in the NUnit results file won't be using the format of a method call Issue: JENKINS-5674
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Alex Earl
          Path:
          src/main/resources/hudson/plugins/nunit/nunit-to-junit.xsl
          src/test/java/hudson/plugins/nunit/NUnitToJUnitXslTest.java
          src/test/resources/hudson/plugins/nunit/JUnit-issue5674-setname.xml
          src/test/resources/hudson/plugins/nunit/NUnit-issue5674-setname.xml
          http://jenkins-ci.org/commit/nunit-plugin/14322b5f5ee9d7caa065303b2f79d1e72deb733f
          Log:
          Merge pull request #16 from timotei/master

          Fix JENKINS-5674

          Compare: https://github.com/jenkinsci/nunit-plugin/compare/129accb1fd73...14322b5f5ee9

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Alex Earl Path: src/main/resources/hudson/plugins/nunit/nunit-to-junit.xsl src/test/java/hudson/plugins/nunit/NUnitToJUnitXslTest.java src/test/resources/hudson/plugins/nunit/JUnit-issue5674-setname.xml src/test/resources/hudson/plugins/nunit/NUnit-issue5674-setname.xml http://jenkins-ci.org/commit/nunit-plugin/14322b5f5ee9d7caa065303b2f79d1e72deb733f Log: Merge pull request #16 from timotei/master Fix JENKINS-5674 Compare: https://github.com/jenkinsci/nunit-plugin/compare/129accb1fd73...14322b5f5ee9
          Hide
          slide_o_mix Alex Earl added a comment -

          Fixed in 0.22

          Show
          slide_o_mix Alex Earl added a comment - Fixed in 0.22

            People

            • Assignee:
              slide_o_mix Alex Earl
              Reporter:
              telgabrowny telgabrowny
            • Votes:
              7 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: