Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16326

TAP Parser can't handle SKIP_ALL

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Not A Defect
    • Component/s: tap-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      In perl test suites, it's common to do a SKIPALL on certain test files when you for instance determine that you're on the wrong OS, or if long tests are not enabled.

      <testsuites>
        <testsuite failures="0"
                   errors="0"
                   tests="0"
                   name="00_devel_smoke_binaries_t">
          <system-out><![CDATA[1..0 # SKIP Long-running tests disabled unless SLOW_TESTS environment variable set.
      ]]></system-out>
          <system-err></system-err>
        </testsuite>
      </testsuites>
      

      When the TAP parser hits these, it declares the build a failure. If the well formed XML is declaring 0 tests, how is this a failure condition?

        Attachments

        1. screenshot004.png
          screenshot004.png
          93 kB
        2. screenshot005.png
          screenshot005.png
          10 kB
        3. screenshot006.png
          screenshot006.png
          63 kB

          Activity

          Hide
          kinow Bruno P. Kinoshita added a comment -

          Hello Todd,

          Let me see if I understand it right. The following text:

          1..0 # SKIP Long-running tests disabled unless SLOW_TESTS environment variable set.

          is not interpreted as a skip all plan, right? We can fix that in tap4j (tap plug-in uses tap4j for parsing TAP files).

          /B

          Show
          kinow Bruno P. Kinoshita added a comment - Hello Todd, Let me see if I understand it right. The following text: 1..0 # SKIP Long-running tests disabled unless SLOW_TESTS environment variable set. is not interpreted as a skip all plan, right? We can fix that in tap4j (tap plug-in uses tap4j for parsing TAP files). /B
          Hide
          toddr Todd Rinaldo added a comment -

          Isn't the XML what's actually parsed? In which case, wouldn't this be what's parsed?

            <testsuite failures="0"
                       errors="0"
                       tests="0"
                       name="00_devel_smoke_binaries_t">
          
          Show
          toddr Todd Rinaldo added a comment - Isn't the XML what's actually parsed? In which case, wouldn't this be what's parsed? <testsuite failures= "0" errors= "0" tests= "0" name= "00_devel_smoke_binaries_t" >
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Not really, the plug-in looks for plan/text files containing TAP streams like the following:

          1..2
          ok 1 - All is fine...
          not ok 2 - Some message # comments

          I'll update the Wiki this week. Will try to make more clear what type of file you need to have to use the plug-in

          Show
          kinow Bruno P. Kinoshita added a comment - Not really, the plug-in looks for plan/text files containing TAP streams like the following: 1..2 ok 1 - All is fine... not ok 2 - Some message # comments I'll update the Wiki this week. Will try to make more clear what type of file you need to have to use the plug-in
          Hide
          toddr Todd Rinaldo added a comment - - edited

          Good! in that case that's easier. Here's what the output looks like in that case:

          $>prove -wlvm t/00_devel_smoke_pl.t
          t/00_devel_smoke_pl.t .. 
          1..0 # SKIP Long-running tests disabled unless SLOW_TESTS environment variable set.
          skipped: Long-running tests disabled unless SLOW_TESTS environment variable set.
          Files=1, Tests=0,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.04 cusr  0.00 csys =  0.07 CPU)
          Result: NOTESTS
          
          Show
          toddr Todd Rinaldo added a comment - - edited Good! in that case that's easier. Here's what the output looks like in that case: $>prove -wlvm t/00_devel_smoke_pl.t t/00_devel_smoke_pl.t .. 1..0 # SKIP Long -running tests disabled unless SLOW_TESTS environment variable set. skipped: Long -running tests disabled unless SLOW_TESTS environment variable set. Files=1, Tests=0, 0 wallclock secs ( 0.03 usr 0.00 sys + 0.04 cusr 0.00 csys = 0.07 CPU) Result: NOTESTS
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Hi there,

          First I created a simple freestyle job, with a shell build step (touch archive).

          Then I downloaded XML-Simple-2.20, copied the t/ folder to my job workspace and executed in another shell build step: prove t/*.t --archive archive/.

          Finally, added TAP to my project, with archive/*.t as pattern.

          The result archive/7_SaxStuff.t contains a SKIP_ALL: 1..0 # SKIP no XML::SAX

          It is interpreted neither as failure or as passed, it's simply treated as a skipped test

          Can you run your tests with prove and archive the results to an existing directory and check what happens, please?

          Thank you in advance, -B

          Show
          kinow Bruno P. Kinoshita added a comment - Hi there, First I created a simple freestyle job, with a shell build step (touch archive). Then I downloaded XML-Simple-2.20, copied the t/ folder to my job workspace and executed in another shell build step: prove t/*.t --archive archive/. Finally, added TAP to my project, with archive/*.t as pattern. The result archive/7_SaxStuff.t contains a SKIP_ALL: 1..0 # SKIP no XML::SAX It is interpreted neither as failure or as passed, it's simply treated as a skipped test Can you run your tests with prove and archive the results to an existing directory and check what happens, please? Thank you in advance, -B
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Job configuration

          Show
          kinow Bruno P. Kinoshita added a comment - Job configuration
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Marking as not a defect. If we find that it's not possible to interpret SKIP_ALL with the plug-in I'll change the bug status TY

          Show
          kinow Bruno P. Kinoshita added a comment - Marking as not a defect. If we find that it's not possible to interpret SKIP_ALL with the plug-in I'll change the bug status TY
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Fixed in 1.10

          Show
          kinow Bruno P. Kinoshita added a comment - Fixed in 1.10

            People

            • Assignee:
              kinow Bruno P. Kinoshita
              Reporter:
              toddr Todd Rinaldo
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: