Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-17960

Indicate if tests don't go to plan

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      One of the major features of TAP is that you have a test plan.
      If my TAP starts 1..100, but only 5 results are recorded before something broke, then I want to know about it. Likewise, if 105 results are recorded, I want some kind of error indication.

      I suggest counting this as 95 skipped tests by default, possibly with an config option to treat them as failed instead.

        Attachments

          Activity

          jameshowe James Howe created issue -
          kinow Bruno P. Kinoshita made changes -
          Field Original Value New Value
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Hi James,

          Good idea. We can include a new checkbox in the job configuration, so we'll keep the previous behaviour while we include this new feature.

          Show
          kinow Bruno P. Kinoshita added a comment - Hi James, Good idea. We can include a new checkbox in the job configuration, so we'll keep the previous behaviour while we include this new feature.
          kinow Bruno P. Kinoshita made changes -
          Labels 1.13
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Bruno P. Kinoshita
          Path:
          src/main/java/org/tap4j/plugin/TapParser.java
          src/main/java/org/tap4j/plugin/TapPublisher.java
          src/main/resources/org/tap4j/plugin/TapPublisher/config.jelly
          http://jenkins-ci.org/commit/tap-plugin/2127053964d7d4b0b8a3038658ef231a0a214e31
          Log:
          [FIXED JENKINS-17960] [FIXED JENKINS-17781] Adding option to validate number of planned tests and number of tests executed

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Bruno P. Kinoshita Path: src/main/java/org/tap4j/plugin/TapParser.java src/main/java/org/tap4j/plugin/TapPublisher.java src/main/resources/org/tap4j/plugin/TapPublisher/config.jelly http://jenkins-ci.org/commit/tap-plugin/2127053964d7d4b0b8a3038658ef231a0a214e31 Log: [FIXED JENKINS-17960] [FIXED JENKINS-17781] Adding option to validate number of planned tests and number of tests executed
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Bruno P. Kinoshita
          Path:
          src/test/java/org/tap4j/plugin/issue16647/TestIssue16647.java
          src/test/java/org/tap4j/plugin/issue16964/TestIssue16964.java
          http://jenkins-ci.org/commit/tap-plugin/c52f18350d32a93e5242e872eb8c24f756544b7d
          Log:
          [JENKINS-17960 JENKINS-17781] Fixing tests

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Bruno P. Kinoshita Path: src/test/java/org/tap4j/plugin/issue16647/TestIssue16647.java src/test/java/org/tap4j/plugin/issue16964/TestIssue16964.java http://jenkins-ci.org/commit/tap-plugin/c52f18350d32a93e5242e872eb8c24f756544b7d Log: [JENKINS-17960 JENKINS-17781] Fixing tests
          kinow Bruno P. Kinoshita made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Fixed in 1.13

          Show
          kinow Bruno P. Kinoshita added a comment - Fixed in 1.13
          kinow Bruno P. Kinoshita made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          jameshowe James Howe added a comment -

          Ah, you've made it less useful. Now I don't get any results.
          The plan check should come last.

          If I plan 1000 tests but only run 999, I should a) get all 999 results reported and b) be unstable even if they all passed.

          Show
          jameshowe James Howe added a comment - Ah, you've made it less useful. Now I don't get any results. The plan check should come last. If I plan 1000 tests but only run 999, I should a) get all 999 results reported and b) be unstable even if they all passed.
          Hide
          jameshowe James Howe added a comment -

          Like how I originally suggested.

          Show
          jameshowe James Howe added a comment - Like how I originally suggested.
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Sorry @James, let's reopen it and work on a fix again

          My bad, Bruno

          Show
          kinow Bruno P. Kinoshita added a comment - Sorry @James, let's reopen it and work on a fix again My bad, Bruno
          kinow Bruno P. Kinoshita made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Hide
          jameshowe James Howe added a comment -

          In fact, it now fails regardless of settings if there is no plan, where before it was successful.

          Show
          jameshowe James Howe added a comment - In fact, it now fails regardless of settings if there is no plan, where before it was successful.
          Hide
          jameshowe James Howe added a comment -

          Surely it's simple just to have it do this?

          Tests planned: 100, Tests run: 90, Tests passed: 90, Status: Unstable
          Tests planned: 100, Tests run: 110, Tests passed: 110, Status: Unstable
          Tests planned: 100, Tests run: 100, Tests passed: 100, Status: Stable
          etc.

          Show
          jameshowe James Howe added a comment - Surely it's simple just to have it do this? Tests planned: 100, Tests run: 90, Tests passed: 90, Status: Unstable Tests planned: 100, Tests run: 110, Tests passed: 110, Status: Unstable Tests planned: 100, Tests run: 100, Tests passed: 100, Status: Stable etc.
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Hi James, I believe I'll start a new cycle on tap-plugin today or very soon. Would you have time to test a beta version of tap-plugin before we release a new version and close this issue again?

          Show
          kinow Bruno P. Kinoshita added a comment - Hi James, I believe I'll start a new cycle on tap-plugin today or very soon. Would you have time to test a beta version of tap-plugin before we release a new version and close this issue again?
          Hide
          jameshowe James Howe added a comment -

          Yes, I think so.

          Show
          jameshowe James Howe added a comment - Yes, I think so.
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Starting a dev cycle on tap plugin looking forward to working on this issue again James. Hope you have some spare time and patience to go through this issue again.

          Show
          kinow Bruno P. Kinoshita added a comment - Starting a dev cycle on tap plugin looking forward to working on this issue again James. Hope you have some spare time and patience to go through this issue again.
          Hide
          kinow Bruno P. Kinoshita added a comment -

          > Tests planned: 100, Tests run: 90, Tests passed: 90, Status: Unstable
          > Tests planned: 100, Tests run: 110, Tests passed: 110, Status: Unstable
          > Tests planned: 100, Tests run: 100, Tests passed: 100, Status: Stable

          Tis indeed not so hard to do. However, should you have any failure, and the
          job is configured to fail the build in case of failures, your build will
          be marked as failure, instead of unstable.

          On the other hand, if you planned 100 test cases, but only 90 were execute,
          and the others were skipped, then with this new change your build will
          be marked as unstable because the number of test cases doesn't match
          the plan.

          WDYT? Will release an alpha version to the alpha update site.

          Show
          kinow Bruno P. Kinoshita added a comment - > Tests planned: 100, Tests run: 90, Tests passed: 90, Status: Unstable > Tests planned: 100, Tests run: 110, Tests passed: 110, Status: Unstable > Tests planned: 100, Tests run: 100, Tests passed: 100, Status: Stable Tis indeed not so hard to do. However, should you have any failure, and the job is configured to fail the build in case of failures, your build will be marked as failure, instead of unstable. On the other hand, if you planned 100 test cases, but only 90 were execute, and the others were skipped, then with this new change your build will be marked as unstable because the number of test cases doesn't match the plan. WDYT? Will release an alpha version to the alpha update site.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Bruno P. Kinoshita
          Path:
          src/main/java/org/tap4j/plugin/TapParser.java
          src/main/java/org/tap4j/plugin/TapPublisher.java
          src/main/java/org/tap4j/plugin/TapResult.java
          src/main/java/org/tap4j/plugin/model/TapStreamResult.java
          src/main/java/org/tap4j/plugin/model/TapTestResultResult.java
          src/test/java/org/tap4j/plugin/issue16964/TestIssue16964.java
          http://jenkins-ci.org/commit/tap-plugin/9e60ea571d359a0c445a33f791e62ca6c5f87479
          Log:
          JENKINS-17960 Indicate if tests don't go to plan

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Bruno P. Kinoshita Path: src/main/java/org/tap4j/plugin/TapParser.java src/main/java/org/tap4j/plugin/TapPublisher.java src/main/java/org/tap4j/plugin/TapResult.java src/main/java/org/tap4j/plugin/model/TapStreamResult.java src/main/java/org/tap4j/plugin/model/TapTestResultResult.java src/test/java/org/tap4j/plugin/issue16964/TestIssue16964.java http://jenkins-ci.org/commit/tap-plugin/9e60ea571d359a0c445a33f791e62ca6c5f87479 Log: JENKINS-17960 Indicate if tests don't go to plan
          Hide
          kinow Bruno P. Kinoshita added a comment -

          1.18-alpha released to http://updates.jenkins-ci.org/update-center.json

          Give it a shot and let me know if that is in the right direction. You'll see a message in the console output, but nothing in the UI yet (will work on this if you confirm we are in the right direction).

          Thanks!
          Bruno

          Show
          kinow Bruno P. Kinoshita added a comment - 1.18-alpha released to http://updates.jenkins-ci.org/update-center.json Give it a shot and let me know if that is in the right direction. You'll see a message in the console output, but nothing in the UI yet (will work on this if you confirm we are in the right direction). Thanks! Bruno
          kinow Bruno P. Kinoshita made changes -
          Status Reopened [ 4 ] Open [ 1 ]
          Hide
          jameshowe James Howe added a comment -

          > On the other hand, if you planned 100 test cases, but only 90 were execute,
          > and the others were skipped, then with this new change your build will
          > be marked as unstable because the number of test cases doesn't match
          > the plan.

          That's not following the TAP standard. Skipped tests should count.
          If Passed + Failed + Skipped = Planned then do nothing, otherwise make a Stable job Unstable.

          Show
          jameshowe James Howe added a comment - > On the other hand, if you planned 100 test cases, but only 90 were execute, > and the others were skipped, then with this new change your build will > be marked as unstable because the number of test cases doesn't match > the plan. That's not following the TAP standard. Skipped tests should count. If Passed + Failed + Skipped = Planned then do nothing, otherwise make a Stable job Unstable.
          Hide
          jameshowe James Howe added a comment -

          I'm also seeing in version 1.17 that if the plan doesn't match then all the results are discarded.
          It should report them just as for any other build.

          Show
          jameshowe James Howe added a comment - I'm also seeing in version 1.17 that if the plan doesn't match then all the results are discarded. It should report them just as for any other build.
          Hide
          jameshowe James Howe added a comment - - edited

          The logic for this feature should really be very simple.

          if (planExists && numberAtEndOfPlan != numberOfLastTest)

          { log message that tests didn't go to plan; if (status == SUCCESS) then status = UNSTABLE; }

          The checkbox should not cause anything else to happen.

          Show
          jameshowe James Howe added a comment - - edited The logic for this feature should really be very simple. if (planExists && numberAtEndOfPlan != numberOfLastTest) { log message that tests didn't go to plan; if (status == SUCCESS) then status = UNSTABLE; } The checkbox should not cause anything else to happen.
          Hide
          kinow Bruno P. Kinoshita added a comment -

          I liked this simpler logic. Going to commit now but will be able to release another alpha version only tonight. Hope you can give it another shot this week. I'd like to cut a new release by weekend.

          Thanks James!

          Show
          kinow Bruno P. Kinoshita added a comment - I liked this simpler logic. Going to commit now but will be able to release another alpha version only tonight. Hope you can give it another shot this week. I'd like to cut a new release by weekend. Thanks James!
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Bruno P. Kinoshita
          Path:
          src/main/java/org/tap4j/plugin/TapPublisher.java
          http://jenkins-ci.org/commit/tap-plugin/275c599945025e3467bb42e9d82d3089ee25dc2b
          Log:
          JENKINS-17960 Simplify logic

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Bruno P. Kinoshita Path: src/main/java/org/tap4j/plugin/TapPublisher.java http://jenkins-ci.org/commit/tap-plugin/275c599945025e3467bb42e9d82d3089ee25dc2b Log: JENKINS-17960 Simplify logic
          Hide
          jameshowe James Howe added a comment -

          Does that commit also address the problem of all the results being lost if the tests don't go to plan?

          Show
          jameshowe James Howe added a comment - Does that commit also address the problem of all the results being lost if the tests don't go to plan?
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Not that one, I believe it was fixed in https://github.com/jenkinsci/tap-plugin/commit/9e60ea571d359a0c445a33f791e62ca6c5f87479#diff-7ebebde3074c0bb507b4983802c5803cL149, already released in the experimental update center.

          This new commit only simplifies the logic as you suggested. Let me know if I missed anything. Probably I can push more commits today before releasing 1.18-2-alpha tonight.

          Show
          kinow Bruno P. Kinoshita added a comment - Not that one, I believe it was fixed in https://github.com/jenkinsci/tap-plugin/commit/9e60ea571d359a0c445a33f791e62ca6c5f87479#diff-7ebebde3074c0bb507b4983802c5803cL149 , already released in the experimental update center. This new commit only simplifies the logic as you suggested. Let me know if I missed anything. Probably I can push more commits today before releasing 1.18-2-alpha tonight.
          Hide
          kinow Bruno P. Kinoshita added a comment -

          By the way, I'm not exactly comparing the number of the last test, but comparing if the total of tests is equal to the expected number in the test plan.

          Do you see any problem in doing like that?

          Show
          kinow Bruno P. Kinoshita added a comment - By the way, I'm not exactly comparing the number of the last test, but comparing if the total of tests is equal to the expected number in the test plan. Do you see any problem in doing like that?
          Hide
          jameshowe James Howe added a comment -

          That looks like it might work now
          I'd need something in the next couple of hours in order to test it before Monday.

          Show
          jameshowe James Howe added a comment - That looks like it might work now I'd need something in the next couple of hours in order to test it before Monday.
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Due to firewall policies in the office where I am atm I can't release an alpha version to the experimental update center

          I've attached a snapshot HPI with the latest code though, don't know if you can install it in some environment and try it. If so let me know if that works and feel free to update this issue.

          JENKINS-17960 is the only pending issue I'm waiting to cut a new release. Next release will focus in JVM crashes and memory management, and the following will be code review/smaller improvements.

          Thanks!
          Bruno

          Show
          kinow Bruno P. Kinoshita added a comment - Due to firewall policies in the office where I am atm I can't release an alpha version to the experimental update center I've attached a snapshot HPI with the latest code though, don't know if you can install it in some environment and try it. If so let me know if that works and feel free to update this issue. JENKINS-17960 is the only pending issue I'm waiting to cut a new release. Next release will focus in JVM crashes and memory management, and the following will be code review/smaller improvements. Thanks! Bruno
          Hide
          kinow Bruno P. Kinoshita added a comment -

          TAP Plug-in snapshot 2014-03-12 HPI file

          Show
          kinow Bruno P. Kinoshita added a comment - TAP Plug-in snapshot 2014-03-12 HPI file
          kinow Bruno P. Kinoshita made changes -
          Attachment tap-snapshot-20140312.hpi [ 25543 ]
          Hide
          jameshowe James Howe added a comment -

          This looks like it now works as I would expect.

          Show
          jameshowe James Howe added a comment - This looks like it now works as I would expect.
          jameshowe James Howe made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Yay, thanks for testing it so quickly James. Cutting a new release this night, 1.18 should be available in the update center within hours.

          Thanks again
          Bruno

          Show
          kinow Bruno P. Kinoshita added a comment - Yay, thanks for testing it so quickly James. Cutting a new release this night, 1.18 should be available in the update center within hours. Thanks again Bruno
          Hide
          kinow Bruno P. Kinoshita added a comment -

          Fixed in 1.18

          Show
          kinow Bruno P. Kinoshita added a comment - Fixed in 1.18
          kinow Bruno P. Kinoshita made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          rtyler R. Tyler Croy made changes -
          Workflow JNJira [ 149257 ] JNJira + In-Review [ 206649 ]

            People

            • Assignee:
              kinow Bruno P. Kinoshita
              Reporter:
              jameshowe James Howe
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: