Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52152

xunit cannot read junit time attributes with more than three decimals or using exponential notation

    Details

    • Similar Issues:

      Description

      xunit fails to parse junit xml files where the time attribute of the testcase element contains e.g 0.123456 or 1.6e-05, because it very narrowly defines the valid format to decimal notation with no more than three decimals.

       This breaks parsing the junit output from e.g. https://github.com/catchorg/Catch2 as well as an internal unit testing framework with use at my workplace.

      INFO: [JUnit] - 34 test report file(s) were found with the pattern '**/*result.xml' relative to 'foo' for the testing framework 'JUnit'.
      WARNING: The file 'foo/junit-result.xml' is an invalid file.
      WARNING: At line 3 of file:foo/junit-result.xml:cvc-pattern-valid: Value '1.6e-05' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
      WARNING: At line 3 of file:foo/junit-result.xml:cvc-attribute.3: The value '1.6e-05' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
      ERROR: Step ‘Publish xUnit test result report’ aborted due to exception: 

       

        Attachments

          Issue Links

            Activity

            tsondergaard tsondergaard created issue -
            tsondergaard tsondergaard made changes -
            Field Original Value New Value
            Description xunit fails to parse junit xml files where the time attribute of the testcase element contains e.g 0.123456 or 1.6e-05, because it very narrowly defines the valid format to decimal notation with no more than three decimals.

             
            {noformat}
            INFO: [JUnit] - 34 test report file(s) were found with the pattern '**/*result.xml' relative to 'foo' for the testing framework 'JUnit'.
            WARNING: The file 'foo/junit-result.xml' is an invalid file.
            WARNING: At line 3 of file:foo/junit-result.xml:cvc-pattern-valid: Value '1.6e-05' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
            WARNING: At line 3 of file:foo/junit-result.xml:cvc-attribute.3: The value '1.6e-05' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
            ERROR: Step ‘Publish xUnit test result report’ aborted due to exception: {noformat}
            xunit fails to parse junit xml files where the time attribute of the testcase element contains e.g 0.123456 or 1.6e-05, because it very narrowly defines the valid format to decimal notation with no more than three decimals.

             This breaks parsing the junit output from e.g. [https://github.com/catchorg/Catch2] as well as an internal unit testing framework with use at my workplace.
            {noformat}
            INFO: [JUnit] - 34 test report file(s) were found with the pattern '**/*result.xml' relative to 'foo' for the testing framework 'JUnit'.
            WARNING: The file 'foo/junit-result.xml' is an invalid file.
            WARNING: At line 3 of file:foo/junit-result.xml:cvc-pattern-valid: Value '1.6e-05' is not facet-valid with respect to pattern '(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?' for type '#AnonType_timetestsuite'.
            WARNING: At line 3 of file:foo/junit-result.xml:cvc-attribute.3: The value '1.6e-05' of attribute 'time' on element 'testsuite' is not valid with respect to its type, 'null'.
            ERROR: Step ‘Publish xUnit test result report’ aborted due to exception: {noformat}
             
            nfalco Nikolas Falco made changes -
            Link This issue is related to JENKINS-52392 [ JENKINS-52392 ]
            Hide
            nfalco Nikolas Falco added a comment -

            Explanation was in Wiki page and support in the linked issue

            Show
            nfalco Nikolas Falco added a comment - Explanation was in Wiki page and support in the linked issue
            nfalco Nikolas Falco made changes -
            Link This issue is related to JENKINS-52392 [ JENKINS-52392 ]
            nfalco Nikolas Falco made changes -
            Status Open [ 1 ] Closed [ 6 ]
            Resolution Not A Defect [ 7 ]
            nfalco Nikolas Falco made changes -
            Link This issue is related to JENKINS-52392 [ JENKINS-52392 ]
            Hide
            tsondergaard tsondergaard added a comment -

            Nikolas Falco, could you clarify whether this is the explanation in the wiki page you refer to:

            https://wiki.jenkins.io/display/JENKINS/xUnit+Plugin#xUnitPlugin-BreakingChanges

            In case you are using a non java tools that declares to produce a JUnit compatible report, ensure the report respects one of the widely used supported schemas. If not than select as test type the build step "Custom Tool" and provide a XSL as described here.

             

            I can understand the wish to pick and follow "standards", so I have reported an issue against http://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd to have it support more than three decimals.

            https://issues.apache.org/jira/browse/SUREFIRE-1533

             

            Show
            tsondergaard tsondergaard added a comment - Nikolas Falco , could you clarify whether this is the explanation in the wiki page you refer to: https://wiki.jenkins.io/display/JENKINS/xUnit+Plugin#xUnitPlugin-BreakingChanges Official supported JUnit schema are Ant junit and Maven Surefire ). In case you are using a non java tools that declares to produce a JUnit compatible report, ensure the report respects one of the widely used supported schemas. If not than select as test type the build step "Custom Tool" and provide a XSL as described  here .   I can understand the wish to pick and follow "standards", so I have reported an issue against http://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd to have it support more than three decimals. https://issues.apache.org/jira/browse/SUREFIRE-1533  
            Hide
            nfalco Nikolas Falco added a comment -

            Read the JENKINS-52392 for a possible solution.

            About the issue opened to sufefire you wrote "Many test tools, also outside the Java world, use the junit format,"
            You have to know that JUnit is a java library and does not define a standard for report. Maven surefire/Ant has defined an own schema to support JUnit library feature and the library that rely on java time (Date.getTime()) return millisecond. For this reason does not have sense precision more than milliseconds or as someone has open a defect to parse time in format 1-E10.

            Show
            nfalco Nikolas Falco added a comment - Read the JENKINS-52392 for a possible solution. About the issue opened to sufefire you wrote "Many test tools, also outside the Java world, use the junit format," You have to know that JUnit is a java library and does not define a standard for report. Maven surefire/Ant has defined an own schema to support JUnit library feature and the library that rely on java time (Date.getTime()) return millisecond. For this reason does not have sense precision more than milliseconds or as someone has open a defect to parse time in format 1-E10.
            Hide
            tsondergaard tsondergaard added a comment -

            I believe there is nothing lost by loosening the format for the time attribute. If they feel different then other tools will have to adapt to that reality.

            Show
            tsondergaard tsondergaard added a comment - I believe there is nothing lost by loosening the format for the time attribute. If they feel different then other tools will have to adapt to that reality.
            Hide
            cmuller13 Christophe Muller added a comment - - edited

            I understand that it is cleaner to validate the XML files but it brings a lot of issues when using several test frameworks.

            We had to upgrade nose so that it now puts skipped instead of skip (which was fixed in nose2) and this was OK because it looked like an issue on nose part (when they say it "follows" junit standard). But here for catch2 it is not an issue on their part, they have better precision and it's good. What we did as a workaround is to patch the xsd file to allow for better precision of time attributes and it worked:

            --- a/junit-10.xsd 2018-11-15 18:25:44.849045904 +0100
            +++ b/junit-10.xsd 2018-11-15 18:27:26.705774044 +0100
            @@ -25,7 +25,7 @@
             <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
             <xs:simpleType name="SUREFIRE_TIME">
             <xs:restriction base="xs:string">
            - <xs:pattern value="(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?"/>
            + <xs:pattern value="(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,9})?"/>
             </xs:restriction>
             </xs:simpleType>

            It would be nice that such feature be considered in the xunit plugin IMHO.

             

            Show
            cmuller13 Christophe Muller added a comment - - edited I understand that it is cleaner to validate the XML files but it brings a lot of issues when using several test frameworks. We had to upgrade nose so that it now puts skipped instead of skip (which was fixed in nose2) and this was OK because it looked like an issue on nose part (when they say it "follows" junit standard). But here for catch2 it is not an issue on their part, they have better precision and it's good. What we did as a workaround is to patch the xsd file to allow for better precision of time attributes and it worked: --- a/junit-10.xsd 2018-11-15 18:25:44.849045904 +0100 +++ b/junit-10.xsd 2018-11-15 18:27:26.705774044 +0100 @@ -25,7 +25,7 @@ <xs:schema xmlns:xs= "http: //www.w3.org/2001/XMLSchema" > <xs:simpleType name= "SUREFIRE_TIME" > <xs:restriction base= "xs:string" > - <xs:pattern value= "(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,3})?" /> + <xs:pattern value= "(([0-9]{0,3},)*[0-9]{3}|[0-9]{0,3})*(\.[0-9]{0,9})?" /> </xs:restriction> </xs:simpleType> It would be nice that such feature be considered in the xunit plugin IMHO.  
            Hide
            nfalco Nikolas Falco added a comment -

            I had already explain the reason of why and which junit format this plugin supports.

            For any custom junit format refer to this issue and the wiki documentation here

            Show
            nfalco Nikolas Falco added a comment - I had already explain the reason of why and which junit format this plugin supports. For any custom junit format refer to this issue and the wiki documentation here
            Hide
            marek_ruszczak Marek Ruszczak added a comment -

            Ok supported formats are https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd and http://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd

            Now take a look into first one:

            <?xml version="1.0" encoding="UTF-8"?>
            
            <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
            	 elementFormDefault="qualified"
            	 attributeFormDefault="unqualified">
            
                ......
                <xs:attribute name="time" type="xs:decimal" use="required">
            
                ......
            </xs:schema>
            

            Lets see definition of decimal in http://www.w3.org/2001/XMLSchema

            XML Schema 1.1

            An XML Schema schema document for XML Schema schema documents.

            XMLSchema.xsd contains:

              <xs:simpleType name="decimal" id="decimal">
                <xs:annotation>
                  <xs:appinfo>
                    <hfp:hasFacet name="totalDigits"/>
                    <hfp:hasFacet name="fractionDigits"/>
                    <hfp:hasFacet name="pattern"/>
                    <hfp:hasFacet name="whiteSpace"/>
                    <hfp:hasFacet name="enumeration"/>
                    <hfp:hasFacet name="maxInclusive"/>
                    <hfp:hasFacet name="maxExclusive"/>
                    <hfp:hasFacet name="minInclusive"/>
                    <hfp:hasFacet name="minExclusive"/>
                    <hfp:hasProperty name="ordered" value="total"/>
                    <hfp:hasProperty name="bounded" value="false"/>
                    <hfp:hasProperty name="cardinality" value="countably infinite"/>
                    <hfp:hasProperty name="numeric" value="true"/>
                  </xs:appinfo>
                  <xs:documentation source="http://www.w3.org/TR/xmlschema-2/#decimal"/>
                </xs:annotation>
                <xs:restriction base="xs:anySimpleType">
                  <xs:whiteSpace value="collapse" fixed="true" id="decimal.whiteSpace"/>
                </xs:restriction>
               </xs:simpleType>
            

            And finally above leads to documentation which states

            3.2.3.1 Lexical representation

            *decimal* has a lexical representation consisting of a finite-length sequence of decimal digits (#x30-#x39) separated by a period as a decimal indicator. An optional leading sign is allowed. If the sign is omitted, "+" is assumed. Leading and trailing zeroes are optional. If the fractional part is zero, the period and following zero(es) can be omitted. For example: `-1.23, 12678967.543233, +100000.00, 210`.

            3.2.3.2 Canonical representation

            The canonical representation for *decimal* is defined by prohibiting certain options from the Lexical representation (§3.2.3.1). Specifically, the preceding optional "+" sign is prohibited. The decimal point is required. Leading and trailing zeroes are prohibited subject to the following: there must be at least one digit to the right and to the left of the decimal point which may be a zero.

            For me it is clear value 12678967.543233 is legal for "time" and should not lead to an error as it does.

            Nikolas Falco So IMO this should be reopened and fixed.

            Show
            marek_ruszczak Marek Ruszczak added a comment - Ok supported formats are https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd and http://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd Now take a look into first one: <?xml version= "1.0" encoding= "UTF-8" ?> <xs:schema xmlns:xs = "http://www.w3.org/2001/XMLSchema" elementFormDefault= "qualified" attributeFormDefault= "unqualified" > ...... <xs:attribute name= "time" type= "xs:decimal" use= "required" > ...... </xs:schema> Lets see definition of decimal in http://www.w3.org/2001/XMLSchema XML Schema 1.1 An XML Schema schema document for XML Schema schema documents. XMLSchema.xsd contains: <xs:simpleType name= "decimal" id= "decimal" > <xs:annotation> <xs:appinfo> <hfp:hasFacet name= "totalDigits" /> <hfp:hasFacet name= "fractionDigits" /> <hfp:hasFacet name= "pattern" /> <hfp:hasFacet name= "whiteSpace" /> <hfp:hasFacet name= "enumeration" /> <hfp:hasFacet name= "maxInclusive" /> <hfp:hasFacet name= "maxExclusive" /> <hfp:hasFacet name= "minInclusive" /> <hfp:hasFacet name= "minExclusive" /> <hfp:hasProperty name= "ordered" value= "total" /> <hfp:hasProperty name= "bounded" value= "false" /> <hfp:hasProperty name= "cardinality" value= "countably infinite" /> <hfp:hasProperty name= "numeric" value= "true" /> </xs:appinfo> <xs:documentation source= "http://www.w3.org/TR/xmlschema-2/#decimal" /> </xs:annotation> <xs:restriction base= "xs:anySimpleType" > <xs:whiteSpace value= "collapse" fixed= "true" id= "decimal.whiteSpace" /> </xs:restriction> </xs:simpleType> And finally above leads to documentation which states 3.2.3.1 Lexical representation * decimal * has a lexical representation consisting of a finite-length sequence of decimal digits (#x30-#x39) separated by a period as a decimal indicator. An optional leading sign is allowed. If the sign is omitted, "+" is assumed. Leading and trailing zeroes are optional. If the fractional part is zero, the period and following zero(es) can be omitted. For example: `-1.23, 12678967.543233, +100000.00, 210`. 3.2.3.2 Canonical representation The canonical representation for * decimal * is defined by prohibiting certain options from the Lexical representation (§3.2.3.1) . Specifically, the preceding optional "+" sign is prohibited. The decimal point is required. Leading and trailing zeroes are prohibited subject to the following: there must be at least one digit to the right and to the left of the decimal point which may be a zero. For me it is clear value 12678967.543233 is legal for "time" and should not lead to an error as it does. Nikolas Falco So IMO this should be reopened and fixed.

              People

              • Assignee:
                nfalco Nikolas Falco
                Reporter:
                tsondergaard tsondergaard
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: