Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17781

TAP Plugin: fails to report errors for TAP streams where # tests run doesn't match plan

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: tap-plugin
    • Labels:
    • Environment:
      Jenkins: 1.500
      TAP Plugin: 1.10

      Optional: perl: 5.8.9, TAP::Harness: 3.26, TAP::Harness::Archive: 0.14
    • Similar Issues:

      Description

      Thanks for your work on the Jenkins TAP Plugin!

      In our tests, we've found it ignores parse errors where # tests run does not match the plan. Given this perl test:

      $ cat fail.t
      use Test::More tests => 2;
      ok('before die');
      die "eek!";
      ok('after die');

      And a Jenkins Job with this shell build step:

      cd $JENKINS_HOME
      mkdir -p $WORKSPACE/tap-output
      set +e
      prove -m --archive $WORKSPACE/tap-output/ fail.t
      echo "prove exited with: $?"

      Plus a Publish TAP step...

      Running the job you get this TAP output:

      $ cat ~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t
      1..2
      ok 1
      eek! at fail.t line 4.

      1. Looks like you planned 2 tests but ran 1.
      2. Looks like your test exited with 255 just after 1.

      This should result in a Parse Error (because we planned 2 tests, but ran only 1). TAP::Harness confirms:

      $ cp ~/jobs/TAP-Plugin-Bug/workspace/tap-output/fail.t ~/fail.tap
      $ prove -v ~/fail.tap
      ... snip - TAP ...
      Failed 1/2 subtests

      Test Summary Report
      -------------------
      fail.tap (Wstat: 0 Tests: 1 Failed: 0)
      Parse errors: Bad plan. You planned 2 tests but ran 1.
      Files=1, Tests=1, 0 wallclock secs ( 0.02 usr + 0.03 sys = 0.05 CPU)
      Result: FAIL

      But the Jenkins TAP plugin reports a successful build:

      1 tests, 1 ok, 0 not ok, 0 skipped, 0 Bail Out!.
      ...
      No parse errors found.

      See screenshot for more details.

      I've marked this as critical: as it stands Jenkins TAP will report false positives in these cases. As a worst-case scenario this may result in some users deploying bugs to production.

        Attachments

          Activity

          Hide
          kinow Bruno P. Kinoshita added a comment -

          Hi Steve!

          Thanks for reporting the issue, and for including details. At moment I'm working on other open source projects and private projects.

          But I'll have a new cycle to work on Jenkins plug-ins within the next days. I'll take a look on this issue and will try to cut a release as soon as I've found a fix for it.

          In the meantime, feel free to comment on this issue and/or, in case you have time, submit a pull request via GitHub

          All the best,
          Bruno

          Show
          kinow Bruno P. Kinoshita added a comment - Hi Steve! Thanks for reporting the issue, and for including details. At moment I'm working on other open source projects and private projects. But I'll have a new cycle to work on Jenkins plug-ins within the next days. I'll take a look on this issue and will try to cut a release as soon as I've found a fix for it. In the meantime, feel free to comment on this issue and/or, in case you have time, submit a pull request via GitHub All the best, Bruno
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Successfully reproduced the issue here. I'll write a quick fix for it later. Thanks Steve!

          Show
          kinow Bruno P. Kinoshita added a comment - Successfully reproduced the issue here. I'll write a quick fix for it later. Thanks Steve!
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Bruno P. Kinoshita
          Path:
          src/main/java/org/tap4j/plugin/TapParser.java
          src/main/java/org/tap4j/plugin/TapPublisher.java
          src/main/resources/org/tap4j/plugin/TapPublisher/config.jelly
          http://jenkins-ci.org/commit/tap-plugin/2127053964d7d4b0b8a3038658ef231a0a214e31
          Log:
          [FIXED JENKINS-17960] [FIXED JENKINS-17781] Adding option to validate number of planned tests and number of tests executed

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Bruno P. Kinoshita Path: src/main/java/org/tap4j/plugin/TapParser.java src/main/java/org/tap4j/plugin/TapPublisher.java src/main/resources/org/tap4j/plugin/TapPublisher/config.jelly http://jenkins-ci.org/commit/tap-plugin/2127053964d7d4b0b8a3038658ef231a0a214e31 Log: [FIXED JENKINS-17960] [FIXED JENKINS-17781] Adding option to validate number of planned tests and number of tests executed
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Bruno P. Kinoshita
          Path:
          src/test/java/org/tap4j/plugin/issue16647/TestIssue16647.java
          src/test/java/org/tap4j/plugin/issue16964/TestIssue16964.java
          http://jenkins-ci.org/commit/tap-plugin/c52f18350d32a93e5242e872eb8c24f756544b7d
          Log:
          [JENKINS-17960 JENKINS-17781] Fixing tests

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Bruno P. Kinoshita Path: src/test/java/org/tap4j/plugin/issue16647/TestIssue16647.java src/test/java/org/tap4j/plugin/issue16964/TestIssue16964.java http://jenkins-ci.org/commit/tap-plugin/c52f18350d32a93e5242e872eb8c24f756544b7d Log: [JENKINS-17960 JENKINS-17781] Fixing tests
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Fixed in 1.13

          Show
          kinow Bruno P. Kinoshita added a comment - Fixed in 1.13

            People

            • Assignee:
              kinow Bruno P. Kinoshita
              Reporter:
              spurkis Steve Purkis
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: