Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53237

Support Maven Surefire report version 3

    Details

    • Similar Issues:

      Description

      I want to inform you that XML Reports of Surefire are not valid against XSD
      https://github.com/apache/maven-surefire/blob/master/maven-surefire-plugin/src/site/resources/xsd/surefire-test-report.xsd

      This should be fixed in Maven Surefire.
      If you use this XML and visualize the reports, pls adapt to the fix.
      https://issues.apache.org/jira/browse/SUREFIRE-1559

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment - - edited

            Tibor Digana Does SUREFIRE-1559 imply a breaking change in Maven Surefire?

            > This should be fixed in Maven Surefire. If you use this XML and visualize the reports, pls adapt to the fix.

            What is ETA of the fix? IIUC there is only an issue report without confirmed ETA or action plan (e.g. major version number bump, etc.). 

            Whatever the fix is needed on the Jenkins side, we will need to support both previous and new formats

            Show
            oleg_nenashev Oleg Nenashev added a comment - - edited Tibor Digana Does SUREFIRE-1559 imply a breaking change in Maven Surefire? > This should be fixed in Maven Surefire. If you use this XML and visualize the reports, pls adapt to the fix. What is ETA of the fix? IIUC there is only an issue report without confirmed ETA or action plan (e.g. major version number bump, etc.).  Whatever the fix is needed on the Jenkins side, we will need to support both previous and new formats
            Hide
            tibor17 Tibor Digana added a comment -

            That's the question because for me it breaks the form of report in something (only 4 elements I mentioned above) but the rest remains intact. If you do not mind in the stack trace system-out/err, then you would not see any difference.
            One way or another it was badly designed XSD/XML which could not be parsed and could not be valid. The fact that Jenkins plugins are parsing this XML without any problem means that they do not mind in the elements of (rerunError, rerunFailure, flakyFailure, flakyError). If they were used in Jenkins plugins and if the XML simple value was observed then it means that your users sometimes observed weird text in stack trace:
            <system-out>my console log...</system-out>

            Show
            tibor17 Tibor Digana added a comment - That's the question because for me it breaks the form of report in something (only 4 elements I mentioned above) but the rest remains intact. If you do not mind in the stack trace system-out/err , then you would not see any difference. One way or another it was badly designed XSD/XML which could not be parsed and could not be valid. The fact that Jenkins plugins are parsing this XML without any problem means that they do not mind in the elements of ( rerunError, rerunFailure, flakyFailure, flakyError ). If they were used in Jenkins plugins and if the XML simple value was observed then it means that your users sometimes observed weird text in stack trace: <system-out>my console log...</system-out>
            Hide
            tibor17 Tibor Digana added a comment -

            These 4 elements are relatively new. Two were missing in XSD but present in XML.
            As I said bard work on our side.

            Show
            tibor17 Tibor Digana added a comment - These 4 elements are relatively new. Two were missing in XSD but present in XML. As I said bard work on our side.
            Hide
            tibor17 Tibor Digana added a comment -

            Oleg Nenashev
            It seems you agree since you have not reply. I am going to push it to master in ASF Maven. Any objections?

            Show
            tibor17 Tibor Digana added a comment - Oleg Nenashev It seems you agree since you have not reply. I am going to push it to master in ASF Maven. Any objections?
            Hide
            nfalco Nikolas Falco added a comment - - edited

            > Tibor DiganaThe fact that Jenkins plugins are parsing this XML without any problem means that they do not mind in the elements of (rerunError, rerunFailure, flakyFailure, flakyError)

            Not true at all, plugins like xunit use the XSD to validate the XML (and with 2.22.1 is broken, files are skipped). It's not clear for me what is the gain of the stacktrace element and make rerun elements different from failure and error.

            EDIT: you missed to list the component that supports flaky elements... xunit and flaky-test-handler and for this reason you did get a response in time because no one of major interested people was informed. I'm here now because the new version had broken our organisation pipelines and I got this link from SUREFIRE-1559.

            Show
            nfalco Nikolas Falco added a comment - - edited > Tibor Digana The fact that Jenkins plugins are parsing this XML without any problem means that they do not mind in the elements of (rerunError, rerunFailure, flakyFailure, flakyError) Not true at all, plugins like xunit use the XSD to validate the XML (and with 2.22.1 is broken, files are skipped). It's not clear for me what is the gain of the stacktrace element and make rerun elements different from failure and error. EDIT: you missed to list the component that supports flaky elements... xunit and flaky-test-handler and for this reason you did get a response in time because no one of major interested people was informed. I'm here now because the new version had broken our organisation pipelines and I got this link from SUREFIRE-1559.
            Hide
            tibor17 Tibor Digana added a comment -

            I have tried to involve as many plugin as possible in Jenkins and Travis. I saw XUnit related to .Net and not Java.
            Sorry for that. Mistakes happen the same as developers make mistakes with the XML design regarding elements /flaky and rerun.
            This is the chance to improve the s/w. Btw we plan to release several release candidates for version 3.0.0 and we will break backwards compatibility in API (not in XML) this time and it is because we have no choice after 15 years. There is time and milestone where we have to say that we have to break something but hopefully not everything. We are not going to create a new plugin but after 15 years it is reasonable.
            Nikolas, the XML and old XSD was broken.
            Similar changes happened in Surefire but it was not this schema. It was also with standard output stream which was broken by user's libraries and the error was uncovered in logs however the users do not like it but they had to accept that we also do not want to be responsible for "their" mistakes in their project or used external libraries.

            Show
            tibor17 Tibor Digana added a comment - I have tried to involve as many plugin as possible in Jenkins and Travis. I saw XUnit related to .Net and not Java. Sorry for that. Mistakes happen the same as developers make mistakes with the XML design regarding elements /flaky and rerun. This is the chance to improve the s/w. Btw we plan to release several release candidates for version 3.0.0 and we will break backwards compatibility in API (not in XML) this time and it is because we have no choice after 15 years. There is time and milestone where we have to say that we have to break something but hopefully not everything. We are not going to create a new plugin but after 15 years it is reasonable. Nikolas, the XML and old XSD was broken. Similar changes happened in Surefire but it was not this schema. It was also with standard output stream which was broken by user's libraries and the error was uncovered in logs however the users do not like it but they had to accept that we also do not want to be responsible for "their" mistakes in their project or used external libraries.
            Hide
            nfalco Nikolas Falco added a comment - - edited

            The main issues is than you had introduced these changes (not back compatible with version 2.22.0) in version 2.22.1 not in a major release. The people will not expect that.

            version 2.22.0

                <rerunFailure message="expected:&lt;2&gt; but was:&lt;3&gt;" type="junit.framework.AssertionFailedError">junit.framework.AssertionFailedError: expected:&lt;2&gt; but was:&lt;3&gt;
            	at junit.framework.Assert.fail(Assert.java:57)
            	at junit.framework.Assert.failNotEquals(Assert.java:329)
            	at junit.framework.Assert.assertEquals(Assert.java:78)
            	at junit.framework.Assert.assertEquals(Assert.java:234)
            	at junit.framework.Assert.assertEquals(Assert.java:241)
            	at junit.framework.TestCase.assertEquals(TestCase.java:409)
                    ...
            </rerunFailure>
            

            version 2.22.1

                <rerunFailure type="java.lang.AssertionError">
                  <stackTrace>java.lang.AssertionError
            	at com.acme.faulttolerance.internal.test.FooTest.foo(FooTest.java:10)
            </stackTrace>
                </rerunFailure>
            

            the difference are:

            • 1 new required (not optional) element
            • 1 row stacktrace against full stracktrace
            • no XML text allowed

            Question for you Tibor Digana At this point why do not add an attribute to mark the differences between flaky failures and failure also in the XML (not only in console) ? With the XML you have to parse whole file and make logic on top to calculate real failures.

            Show
            nfalco Nikolas Falco added a comment - - edited The main issues is than you had introduced these changes (not back compatible with version 2.22.0) in version 2.22.1 not in a major release. The people will not expect that. version 2.22.0 <rerunFailure message="expected:&lt;2&gt; but was:&lt;3&gt;" type="junit.framework.AssertionFailedError">junit.framework.AssertionFailedError: expected:&lt;2&gt; but was:&lt;3&gt; at junit.framework.Assert.fail(Assert.java:57) at junit.framework.Assert.failNotEquals(Assert.java:329) at junit.framework.Assert.assertEquals(Assert.java:78) at junit.framework.Assert.assertEquals(Assert.java:234) at junit.framework.Assert.assertEquals(Assert.java:241) at junit.framework.TestCase.assertEquals(TestCase.java:409) ... </rerunFailure> version 2.22.1 <rerunFailure type="java.lang.AssertionError"> <stackTrace>java.lang.AssertionError at com.acme.faulttolerance.internal.test.FooTest.foo(FooTest.java:10) </stackTrace> </rerunFailure> the difference are: 1 new required (not optional) element 1 row stacktrace against full stracktrace no XML text allowed Question for you Tibor Digana At this point why do not add an attribute to mark the differences between flaky failures and failure also in the XML (not only in console) ? With the XML you have to parse whole file and make logic on top to calculate real failures.
            Hide
            tibor17 Tibor Digana added a comment - - edited

            Nikolas Falco
            Hi Nicolas,

            Not sure what you mean by
            At this point why do not add an attribute to mark the differences between flaky failures and failure also in the XML

            With the XML you have to parse whole file and make logic on top to calculate real failures.
            As always because this is XML and the stacktrace is not big like kilobytes.

            Not sure why you validate XML but anyway. Parsing the XML and displaying the statistics is what has to be done by the plugins. In this case we have to find a way how you detect version of the XSD. We have not had any version in the XSD which would be easier to have it in past. What alternatives are you facing in order to detect which XSD is actual?
            We can fix this as following.
            I will deploy the old XSD to the old schema location URL=https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd and I will deploy a new schema location for the current status 2.22.1 with another name of XSD file and new schema location which would have version 2.22.1.
            Do you agree or you see another solution?

            If the change present in 3.0 version, we would have to wait quite long time and there would be a risk that the rerun feature as well as flaky failures are heavily used. One way or another these Jenkins plugins do not care about Maven plugin versions because they only read the XML report as the outcome of the Maven plugins and Jenkins plugins do not use to check the version of surefire in POM. Not that easy for them. The fact that nobody claimed that the stacktrace contained std-out in one string of XML Value means that these features are not much used or the users do not print logs in tested logic on the console, but the frequency of use of these features is more probable.

            Show
            tibor17 Tibor Digana added a comment - - edited Nikolas Falco Hi Nicolas, Not sure what you mean by At this point why do not add an attribute to mark the differences between flaky failures and failure also in the XML With the XML you have to parse whole file and make logic on top to calculate real failures. As always because this is XML and the stacktrace is not big like kilobytes. Not sure why you validate XML but anyway. Parsing the XML and displaying the statistics is what has to be done by the plugins. In this case we have to find a way how you detect version of the XSD. We have not had any version in the XSD which would be easier to have it in past. What alternatives are you facing in order to detect which XSD is actual? We can fix this as following. I will deploy the old XSD to the old schema location URL= https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd and I will deploy a new schema location for the current status 2.22.1 with another name of XSD file and new schema location which would have version 2.22.1. Do you agree or you see another solution? If the change present in 3.0 version, we would have to wait quite long time and there would be a risk that the rerun feature as well as flaky failures are heavily used. One way or another these Jenkins plugins do not care about Maven plugin versions because they only read the XML report as the outcome of the Maven plugins and Jenkins plugins do not use to check the version of surefire in POM. Not that easy for them. The fact that nobody claimed that the stacktrace contained std-out in one string of XML Value means that these features are not much used or the users do not print logs in tested logic on the console, but the frequency of use of these features is more probable.
            Hide
            nfalco Nikolas Falco added a comment - - edited

            Not sure why you validate XML but anyway.

            Since JUnit it's just a library and does not produce a report since 2.0 of xunit-plugin, supported JUnit report are those produced by the ant task and maven surefire. My XSD is compatible for both. I validate report in input to reduce possible unexpected parse/transformer (XSLT for not junit report) errors. This make easier resolve bugs because XSD filter out JUnit reports produced by other tools that declare to produce a correct report but they produces custom attributes and some times a different XML structure that could failure in the parser.

            Parsing the XML and displaying the statistics is what has to be done by the plugins.

            Yes but testsuite attributes meaningful depends from version.

            What I wanted to say with At this point why do not add an attribute to mark the differences between flaky failures and failure also in the XML is that when rerun feature was introduced if had been added additional attribute for flaky failures (maybe also an attribute for runs) the XML would be full backward compatible at all because the attributes had the same meaningful of before. Now we have the root testsuite element that declare 10 tests and 2 failures when in real you have only 7 test method (but 10 runs...) and no failures.

            <testsuite name="...FooTest" time="0" tests="7" errors="0" skipped="0" failures="0" runs="10" flakyfailures="2">
            

            About the XSD version for what I can see for other software I consider a best practice publish XSD for each version and the XML declare the path to the specific XSD version

            <testsuite ... xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report" ... /> (to be compatible with actual)
            <testsuite ... xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-2.2.xsd" ... />
            <testsuite ... xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" ... />
            

            If the change present in 3.0 version, we would have to wait quite long time and there would be a risk that the rerun feature as well as flaky failures are heavily used.

            From the point of view of end user if I get an error updating from 2.x to 2.22.1 I have good reason to open a bug. In 3.x thanks to semantic versioning you can answer "we have changed the format". These kind of changes from 2.22.0 to 2.22.1 is much less reasonable, it looks more like a bug.

            Show
            nfalco Nikolas Falco added a comment - - edited Not sure why you validate XML but anyway. Since JUnit it's just a library and does not produce a report since 2.0 of xunit-plugin, supported JUnit report are those produced by the ant task and maven surefire. My XSD is compatible for both . I validate report in input to reduce possible unexpected parse/transformer (XSLT for not junit report) errors. This make easier resolve bugs because XSD filter out JUnit reports produced by other tools that declare to produce a correct report but they produces custom attributes and some times a different XML structure that could failure in the parser. Parsing the XML and displaying the statistics is what has to be done by the plugins. Yes but testsuite attributes meaningful depends from version. What I wanted to say with At this point why do not add an attribute to mark the differences between flaky failures and failure also in the XML is that when rerun feature was introduced if had been added additional attribute for flaky failures (maybe also an attribute for runs) the XML would be full backward compatible at all because the attributes had the same meaningful of before. Now we have the root testsuite element that declare 10 tests and 2 failures when in real you have only 7 test method (but 10 runs...) and no failures. <testsuite name= "...FooTest" time= "0" tests= "7" errors= "0" skipped= "0" failures= "0" runs= "10" flakyfailures= "2" > About the XSD version for what I can see for other software I consider a best practice publish XSD for each version and the XML declare the path to the specific XSD version <testsuite ... xsi:noNamespaceSchemaLocation= "https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report" ... /> (to be compatible with actual) <testsuite ... xsi:noNamespaceSchemaLocation= "https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-2.2.xsd" ... /> <testsuite ... xsi:noNamespaceSchemaLocation= "https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd" ... /> If the change present in 3.0 version, we would have to wait quite long time and there would be a risk that the rerun feature as well as flaky failures are heavily used. From the point of view of end user if I get an error updating from 2.x to 2.22.1 I have good reason to open a bug. In 3.x thanks to semantic versioning you can answer "we have changed the format". These kind of changes from 2.22.0 to 2.22.1 is much less reasonable, it looks more like a bug.
            Hide
            tibor17 Tibor Digana added a comment -

            Nikolas Falco
            If there are no objections I will introduce XML version in the XSD schema and we will proceed with plugin version 3.0.0-M1.
            We will have multiple versions of this XML in web deployment and the old versions of plugin would be validated against the old schema.
            I will open a pull request where you can have a look if you like and this way you will have the result under your control.
            Is this fair?

            Show
            tibor17 Tibor Digana added a comment - Nikolas Falco If there are no objections I will introduce XML version in the XSD schema and we will proceed with plugin version 3.0.0-M1. We will have multiple versions of this XML in web deployment and the old versions of plugin would be validated against the old schema. I will open a pull request where you can have a look if you like and this way you will have the result under your control. Is this fair?
            Hide
            nfalco Nikolas Falco added a comment -

            For me it's ok

            Show
            nfalco Nikolas Falco added a comment - For me it's ok
            Hide
            tibor17 Tibor Digana added a comment -

            Nikolas Falco
            We are near to Surefire 3.0.0-M2 release. I have made a fix for this issue. There is a new XSD version 3.0 and documentation explaining the decisions, see index.apt.vm. I have pushed a branch SUREFIRE-1590 on GitHub. Please review it. Thx.

            Show
            tibor17 Tibor Digana added a comment - Nikolas Falco We are near to Surefire 3.0.0-M2 release. I have made a fix for this issue. There is a new XSD version 3.0 and documentation explaining the decisions, see index.apt.vm . I have pushed a branch SUREFIRE-1590 on GitHub. Please review it. Thx.
            Hide
            nfalco Nikolas Falco added a comment -

            It seems ok

            Show
            nfalco Nikolas Falco added a comment - It seems ok
            Hide
            tibor17 Tibor Digana added a comment -
            Show
            tibor17 Tibor Digana added a comment - Nikolas Falco We are in release Vote for the version 3.0.0-M2 Please join us in the Vote: https://lists.apache.org/thread.html/d44c7a8198e4f4d214211e40909249bcca67bdabd7d476731f2e1665@%3Cdev.maven.apache.org%3E
            Hide
            tibor17 Tibor Digana added a comment -

            Nikolas Falco
            The version 3.0.0-M2 is available in Maven Central.
            Can you please adapt your plugin xUnit with the path of XSD?
            It has a new location
            https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd
            and old location for legacy versions
            https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd
            I hope this would work for you. Sorry.

            Show
            tibor17 Tibor Digana added a comment - Nikolas Falco The version 3.0.0-M2 is available in Maven Central. Can you please adapt your plugin xUnit with the path of XSD? It has a new location https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd and old location for legacy versions https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd I hope this would work for you. Sorry.
            Hide
            nfalco Nikolas Falco added a comment -

            The XSD embedded in this plugin is not a copy of surefure XSD, it have to support old and new version and ant junit format version. So it is not an issue if the URl is changed but YES I have to check if latest version 3.0.0-M2 cause some issue to the actual plugin. Within this weekend I will perform the needed verification. Thank you.

            Show
            nfalco Nikolas Falco added a comment - The XSD embedded in this plugin is not a copy of surefure XSD, it have to support old and new version and ant junit format version. So it is not an issue if the URl is changed but YES I have to check if latest version 3.0.0-M2 cause some issue to the actual plugin. Within this weekend I will perform the needed verification. Thank you.
            Hide
            nfalco Nikolas Falco added a comment - - edited

            Tibor Digana maybe it's the case also to open an issue to the flaky-test-handler-plugin that is the plugin that decorate JUnit reports with flaky tests in the XML.

            If you want I could open for you.

            Show
            nfalco Nikolas Falco added a comment - - edited Tibor Digana maybe it's the case also to open an issue to the flaky-test-handler-plugin that is the plugin that decorate JUnit reports with flaky tests in the XML. If you want I could open for you.
            Hide
            tibor17 Tibor Digana added a comment -

            Nikolas Falco
            It's up to you. If you find an issue, I am all the ear.
            But I do not know if your plugin can recognize the XSD version. I hope it can.

            Show
            tibor17 Tibor Digana added a comment - Nikolas Falco It's up to you. If you find an issue, I am all the ear. But I do not know if your plugin can recognize the XSD version. I hope it can.

              People

              • Assignee:
                dszczepanik Damian Szczepanik
                Reporter:
                tibor17 Tibor Digana
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: