Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15322

NOTESTS in TAP response gives parse error and stack trace from plugin

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: tap-plugin
    • Labels:
    • Environment:
      centos 6
    • Similar Issues:

      Description

      A .tap file containing a NOTESTS result (from eg a skip_all declaration in a .t file) such as

      /opt/performance/scripts/t/dbcoststest1.t .. skipped: No Relevant Tests as No DB Costs for GetSmoidTrackV11New
      Files=1, Tests=0,  1 wallclock secs ( 0.01 usr  0.02 sys +  0.54 cusr  0.04 csys =  0.61 CPU)
      Result: NOTESTS
      

      results in the following errors when running a plan from Jenkins:

      Parsing TAP test result [/var/lib/jenkins/jobs/Performance-Server-Trunk/builds/2012-09-26_16-06-21/tap/dbcostsGetSmoidTrackV11New.tap].
      org.tap4j.parser.ParserException: Error parsing TAP Stream: Missing TAP Plan.
      	at org.tap4j.parser.Tap13YamlParser.parseFile(Tap13YamlParser.java:587)
      	at org.tap4j.plugin.TapParser.parse(TapParser.java:94)
      	at org.tap4j.plugin.TapPublisher.loadResults(TapPublisher.java:240)
      	at org.tap4j.plugin.TapPublisher.perform(TapPublisher.java:182)
      	at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27)
      	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:705)
      	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:680)
      	at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:996)
      	at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:627)
      	at hudson.model.Run.run(Run.java:1438)
      	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:479)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:238)
      Caused by: org.tap4j.parser.ParserException: Missing TAP Plan.
      	at org.tap4j.parser.Tap13YamlParser.checkTAPPlanIsSet(Tap13YamlParser.java:353)
      	at org.tap4j.parser.Tap13YamlParser.postProcess(Tap13YamlParser.java:736)
      	at org.tap4j.parser.Tap13YamlParser.parseFile(Tap13YamlParser.java:585)
      	... 12 more
      

        Attachments

          Issue Links

            Activity

            alex_n Alex Newnham created issue -
            Hide
            alex_n Alex Newnham added a comment -

            One course of action would be to mark the file as skipped with no associated FAIL, given that No Tests are being explicitly requested.

            Show
            alex_n Alex Newnham added a comment - One course of action would be to mark the file as skipped with no associated FAIL, given that No Tests are being explicitly requested.
            kinow Bruno P. Kinoshita made changes -
            Field Original Value New Value
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            kinow Bruno P. Kinoshita added a comment -

            Hi Alex, I'm not familiar with NOTESTS. Could you provide an example TAP Stream, please?

            Would it be something like:

            somefile.tap
            ------------

            1..0

            ?

            Show
            kinow Bruno P. Kinoshita added a comment - Hi Alex, I'm not familiar with NOTESTS. Could you provide an example TAP Stream, please? Would it be something like: somefile.tap ------------ 1..0 ?
            Hide
            alex_n Alex Newnham added a comment -

            Hi Bruno, here's another NOTESTS .tap file, generated from skip_all as detailed in http://search.cpan.org/~mschwern/Test-Simple-0.98/lib/Test/More.pm

            We also supplied a $skip_reason (specifically " No Relevant Tests as No DB Costs for RequestConfig2V11EncoreNew").

            /opt/performance/scripts/t/dbcoststest1.t .. skipped: No Relevant Tests as No DB Costs for RequestConfig2V11EncoreNew Files=1, Tests=0,  1 wallclock secs ( 0.00 usr  0.01 sys +  0.55 cusr
            0.06 csys =  0.62 CPU)
            Result: NOTESTS
            
            Show
            alex_n Alex Newnham added a comment - Hi Bruno, here's another NOTESTS .tap file, generated from skip_all as detailed in http://search.cpan.org/~mschwern/Test-Simple-0.98/lib/Test/More.pm We also supplied a $skip_reason (specifically " No Relevant Tests as No DB Costs for RequestConfig2V11EncoreNew"). /opt/performance/scripts/t/dbcoststest1.t .. skipped: No Relevant Tests as No DB Costs for RequestConfig2V11EncoreNew Files=1, Tests=0, 1 wallclock secs ( 0.00 usr 0.01 sys + 0.55 cusr 0.06 csys = 0.62 CPU) Result: NOTESTS
            Hide
            kinow Bruno P. Kinoshita added a comment -

            Hmm, the output doesn't look like a valid TAP stream., it looks to be something specific to a Perl module or prove tool.

            In order to have a TAP output, I think you can do something similar to perl -Ilib /opt/performance/scripts/t/dbcoststest1.t

            This should give you something like

            1..0

            1. No tests run!

            I found this perl command in this blog post [1] some time ago. Maybe it will help to clarify what is happening with the plug-in, there's some interesting comments there.

            [1] http://blogs.perl.org/users/confuseacat/2011/09/perl-testing-with-jenkinshudson-avoiding-some-pitfalls.html

            Show
            kinow Bruno P. Kinoshita added a comment - Hmm, the output doesn't look like a valid TAP stream., it looks to be something specific to a Perl module or prove tool. In order to have a TAP output, I think you can do something similar to perl -Ilib /opt/performance/scripts/t/dbcoststest1.t This should give you something like 1..0 No tests run! I found this perl command in this blog post [1] some time ago. Maybe it will help to clarify what is happening with the plug-in, there's some interesting comments there. [1] http://blogs.perl.org/users/confuseacat/2011/09/perl-testing-with-jenkinshudson-avoiding-some-pitfalls.html
            kinow Bruno P. Kinoshita made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Bruno P. Kinoshita
            Path:
            pom.xml
            src/main/java/org/tap4j/plugin/TapParser.java
            src/main/java/org/tap4j/plugin/TapPublisher.java
            src/main/java/org/tap4j/plugin/model/TapTestResultResult.java
            src/main/resources/org/tap4j/plugin/TapPublisher/config.jelly
            http://jenkins-ci.org/commit/tap-plugin/ae773491476f022cf11afbe7c820b76b36bbb646
            Log:
            [FIXED JENKINS-15322] Fixed bug after a new release in tap4j. The parser was treating unknown content as subtests

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Bruno P. Kinoshita Path: pom.xml src/main/java/org/tap4j/plugin/TapParser.java src/main/java/org/tap4j/plugin/TapPublisher.java src/main/java/org/tap4j/plugin/model/TapTestResultResult.java src/main/resources/org/tap4j/plugin/TapPublisher/config.jelly http://jenkins-ci.org/commit/tap-plugin/ae773491476f022cf11afbe7c820b76b36bbb646 Log: [FIXED JENKINS-15322] Fixed bug after a new release in tap4j. The parser was treating unknown content as subtests
            scm_issue_link SCM/JIRA link daemon made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            kinow Bruno P. Kinoshita added a comment -

            Sorry, I sent the code to the wrong issue. Ignore the comment above. This issue is being marked as not a defect (not as fixed) while we discuss about prove/tap.

            Show
            kinow Bruno P. Kinoshita added a comment - Sorry, I sent the code to the wrong issue. Ignore the comment above. This issue is being marked as not a defect (not as fixed) while we discuss about prove/tap.
            kinow Bruno P. Kinoshita made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            kinow Bruno P. Kinoshita made changes -
            Status Reopened [ 4 ] Resolved [ 5 ]
            Resolution Not A Defect [ 7 ]
            Hide
            alex_n Alex Newnham added a comment -

            Thanks for the update

            We are parsing our .t files via TAP::Harness using Test::Harness v3.25 which states it's output is TAP version 12. Perhaps there's a mismatch here somewhere between the TAP produced by our Test::More module and Tap::Harness and the TAP specifications. Or maybe the plugin is expecting a different TAP version?

            Currently I can't investigate further as http://testanything.org is down at the moment.

            In our projects we can't easily switch to using prove from the command line or a bash script as our .t files aren't static, they rely on a whole bunch of variable being passed into them via the TAP::Harness, there may be other workarounds possible though.

            Show
            alex_n Alex Newnham added a comment - Thanks for the update We are parsing our .t files via TAP::Harness using Test::Harness v3.25 which states it's output is TAP version 12. Perhaps there's a mismatch here somewhere between the TAP produced by our Test::More module and Tap::Harness and the TAP specifications. Or maybe the plugin is expecting a different TAP version? Currently I can't investigate further as http://testanything.org is down at the moment. In our projects we can't easily switch to using prove from the command line or a bash script as our .t files aren't static, they rely on a whole bunch of variable being passed into them via the TAP::Harness, there may be other workarounds possible though.
            Hide
            alex_n Alex Newnham added a comment -

            Reopening.

            Show
            alex_n Alex Newnham added a comment - Reopening.
            alex_n Alex Newnham made changes -
            Resolution Not A Defect [ 7 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            Hide
            kinow Bruno P. Kinoshita added a comment -

            > Perhaps there's a mismatch here somewhere between the TAP produced by our Test::More module and Tap::Harness and the TAP specifications. Or maybe the plugin is expecting a different TAP version?

            Probably. The plug-in is expecting TAP 13, thought TAP 12 syntax was more similar to TAP 13, but never worked with this version.

            > Currently I can't investigate further as http://testanything.org is down at the moment.

            The site has been down for over a week. I'll drop an e-mail to the mailing list and hopefully someone will fix it.

            >In our projects we can't easily switch to using prove from the command line or a bash script as our .t files aren't static, they rely on a whole bunch of variable being passed into them via the TAP::Harness, there may be other workarounds possible though.

            Sounds like an interesting case. I don't have Perl in this computer (got a new one) but I will try something from this blog post [1]. In case you have some spare time to take a look on this and see if anything makes sense... my Perl skills are quite limited

            [1] http://www.effectiveperlprogramming.com/blog/1271

            Show
            kinow Bruno P. Kinoshita added a comment - > Perhaps there's a mismatch here somewhere between the TAP produced by our Test::More module and Tap::Harness and the TAP specifications. Or maybe the plugin is expecting a different TAP version? Probably. The plug-in is expecting TAP 13, thought TAP 12 syntax was more similar to TAP 13, but never worked with this version. > Currently I can't investigate further as http://testanything.org is down at the moment. The site has been down for over a week. I'll drop an e-mail to the mailing list and hopefully someone will fix it. >In our projects we can't easily switch to using prove from the command line or a bash script as our .t files aren't static, they rely on a whole bunch of variable being passed into them via the TAP::Harness, there may be other workarounds possible though. Sounds like an interesting case. I don't have Perl in this computer (got a new one) but I will try something from this blog post [1] . In case you have some spare time to take a look on this and see if anything makes sense... my Perl skills are quite limited [1] http://www.effectiveperlprogramming.com/blog/1271
            Hide
            alex_n Alex Newnham added a comment -

            http://testanything.org has been down for A Very Long Time, it seems. There's a crawled version of the site here: http://web.archive.org/web/20100124210854/http://testanything.org/wiki/index.php/Main_Page

            Browsing from there, I noticed here that

            Every test set as defined in this document should contain a version definition. A test set without a version definition is assumed to be written in TAP version 12, which is not covered by this document.

            Unfortunately I haven't yet found a definition for TAP version 12, but at least http://web.archive.org has a definition of TAP 13. It might be safe to assume that the Perl Tap::Harness module is writing valid TAP 12 - also that without a version specifier > 12 the TAP plugin for Jenkins should parse TAP as TAP 12. Maybe I can coerce my perl harness to output TAP 13 which would probably be a fix to my problem, but in the meantime some of the above may be useful to you?

            Show
            alex_n Alex Newnham added a comment - http://testanything.org has been down for A Very Long Time, it seems. There's a crawled version of the site here: http://web.archive.org/web/20100124210854/http://testanything.org/wiki/index.php/Main_Page Browsing from there, I noticed here that Every test set as defined in this document should contain a version definition. A test set without a version definition is assumed to be written in TAP version 12, which is not covered by this document. Unfortunately I haven't yet found a definition for TAP version 12, but at least http://web.archive.org has a definition of TAP 13. It might be safe to assume that the Perl Tap::Harness module is writing valid TAP 12 - also that without a version specifier > 12 the TAP plugin for Jenkins should parse TAP as TAP 12. Maybe I can coerce my perl harness to output TAP 13 which would probably be a fix to my problem, but in the meantime some of the above may be useful to you?
            Hide
            kinow Bruno P. Kinoshita added a comment -

            Hi Alex, nice finding, I'll use web.archive.org for now. I've got in touch with some guys from the tap-l mailing list, and one of them forwarded the message to other mailing lists and to the responsible of testanything.org. Hopefully it will be back online in few days.

            In the meantime, I will try to find out more about TAP 12, and why Tap::Harness does not output TAP 13 :o)

            Thanks a lot.

            Show
            kinow Bruno P. Kinoshita added a comment - Hi Alex, nice finding, I'll use web.archive.org for now. I've got in touch with some guys from the tap-l mailing list, and one of them forwarded the message to other mailing lists and to the responsible of testanything.org. Hopefully it will be back online in few days. In the meantime, I will try to find out more about TAP 12, and why Tap::Harness does not output TAP 13 :o) Thanks a lot.
            Hide
            kinow Bruno P. Kinoshita added a comment -

            Hi Alex,

            Do you know if this issue is related to JENKINS-16325, please?

            If so, maybe I can link both issues as related here so you would get update notifications from both.

            Thanks, Bruno

            Show
            kinow Bruno P. Kinoshita added a comment - Hi Alex, Do you know if this issue is related to JENKINS-16325 , please? If so, maybe I can link both issues as related here so you would get update notifications from both. Thanks, Bruno
            Hide
            alex_n Alex Newnham added a comment -

            JENKINS-16325 seems to be concerned with both

            1. skipping certain tests
            2. appending TAP output from iterating thru multiple .t test files and output to one .tap file.

            Using TAP plugin 1.9, I have not managed to get the stack trace when I only skip certain tests but I can duplicate the stack trace from the plugin when I append tests to one .tap file. I am not sure if multiple tests in one .tap file breaks the TAP spec (for a particular TAP version or perhaps all TAP versions).

            So although the stack trace from this issue looks the same as the trace in JENKINS-16325, the reasons for the traces differ. However, the author of JENKINS-16325 mentions in his comments on that issue that he is still affected by JENKINS-16326 which does pretty much duplicate this issue.

            My tests have been with the simplest TAP files possible, I may have missed some complexity, and I didn't use prove.

            Hope this is useful!
            Alex

            Show
            alex_n Alex Newnham added a comment - JENKINS-16325 seems to be concerned with both skipping certain tests appending TAP output from iterating thru multiple .t test files and output to one .tap file. Using TAP plugin 1.9, I have not managed to get the stack trace when I only skip certain tests but I can duplicate the stack trace from the plugin when I append tests to one .tap file. I am not sure if multiple tests in one .tap file breaks the TAP spec (for a particular TAP version or perhaps all TAP versions). So although the stack trace from this issue looks the same as the trace in JENKINS-16325 , the reasons for the traces differ. However, the author of JENKINS-16325 mentions in his comments on that issue that he is still affected by JENKINS-16326 which does pretty much duplicate this issue. My tests have been with the simplest TAP files possible, I may have missed some complexity, and I didn't use prove . Hope this is useful! Alex
            Hide
            kinow Bruno P. Kinoshita added a comment -

            Hi @Alex,

            IN JENKINS-16325 a user suggested:

            prove *.t --archive tap/

            That worked fine for me. One TAP Strem (in this case, a TAP file) cannot, or at least should not, contain multiple TAP files. In TAP specification a blank line in a file is ignored, as well as any non-TAP related token. So we would have to write a special parser to handle multiple TAP files within a single file.

            I would prefer avoid that if possible.

            Let me know if that works for you too.

            Thanks! /B

            Show
            kinow Bruno P. Kinoshita added a comment - Hi @Alex, IN JENKINS-16325 a user suggested: prove *.t --archive tap/ That worked fine for me. One TAP Strem (in this case, a TAP file) cannot, or at least should not, contain multiple TAP files. In TAP specification a blank line in a file is ignored, as well as any non-TAP related token. So we would have to write a special parser to handle multiple TAP files within a single file. I would prefer avoid that if possible. Let me know if that works for you too. Thanks! /B
            kinow Bruno P. Kinoshita made changes -
            Status Reopened [ 4 ] Open [ 1 ]
            Hide
            alex_n Alex Newnham added a comment -

            I can't see a use case for multiple TAP streams within a single file. None of my TAP is in this format so it's not a feature I am looking for.

            thanks
            Alex

            Show
            alex_n Alex Newnham added a comment - I can't see a use case for multiple TAP streams within a single file. None of my TAP is in this format so it's not a feature I am looking for. thanks Alex
            Hide
            alex_n Alex Newnham added a comment -

            Update: Excellent news! http://testanything.org/ is back up and running.

            Show
            alex_n Alex Newnham added a comment - Update: Excellent news! http://testanything.org/ is back up and running.
            kinow Bruno P. Kinoshita made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            kinow Bruno P. Kinoshita made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            kinow Bruno P. Kinoshita made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            Hide
            kinow Bruno P. Kinoshita added a comment -

            Yay! There are some broken links, and seems like they had to restore an old backup. I'll add tap4j information and examples there again.

            As a side note, I believe I've fixed this issue, but as I had to fix tap4j parser, I'll use a few more hours to test it.

            I'll update the Wiki with information on how to use the plug-in with prove --archive and an example with skip_all.

            Show
            kinow Bruno P. Kinoshita added a comment - Yay! There are some broken links, and seems like they had to restore an old backup. I'll add tap4j information and examples there again. As a side note, I believe I've fixed this issue, but as I had to fix tap4j parser, I'll use a few more hours to test it. I'll update the Wiki with information on how to use the plug-in with prove --archive and an example with skip_all.
            Hide
            kinow Bruno P. Kinoshita added a comment -

            The link to the bug fixed in tap4j: https://github.com/tupilabs/tap4j/issues/13

            Show
            kinow Bruno P. Kinoshita added a comment - The link to the bug fixed in tap4j: https://github.com/tupilabs/tap4j/issues/13
            Hide
            kinow Bruno P. Kinoshita added a comment -

            Wiki updated. Created the section "Running Perl tests with prove" with an example of how one can run prove and archive its output. The procedure is the same even if you use NOTESTS in TAP.

            You can follow the instructions there, but there's another bug which causes the plug-in to not list tests which plan is marked as skip all. The remaining work will continue there, please feel free to comment there.

            Both issues will be released together in 2.0.

            Show
            kinow Bruno P. Kinoshita added a comment - Wiki updated. Created the section "Running Perl tests with prove" with an example of how one can run prove and archive its output. The procedure is the same even if you use NOTESTS in TAP. You can follow the instructions there, but there's another bug which causes the plug-in to not list tests which plan is marked as skip all. The remaining work will continue there, please feel free to comment there. Both issues will be released together in 2.0.
            kinow Bruno P. Kinoshita made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            kinow Bruno P. Kinoshita made changes -
            Link This issue is blocking JENKINS-17245 [ JENKINS-17245 ]
            Hide
            kinow Bruno P. Kinoshita added a comment -

            Fixed. Will release in 2.0 with JENKINS-17245.

            Show
            kinow Bruno P. Kinoshita added a comment - Fixed. Will release in 2.0 with JENKINS-17245 .
            kinow Bruno P. Kinoshita made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            kinow Bruno P. Kinoshita added a comment -

            Fixed in 1.10

            Show
            kinow Bruno P. Kinoshita added a comment - Fixed in 1.10
            kinow Bruno P. Kinoshita made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 146045 ] JNJira + In-Review [ 206179 ]

              People

              • Assignee:
                kinow Bruno P. Kinoshita
                Reporter:
                alex_n Alex Newnham
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: