Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21835

Cucumber Test Result Plugin crashes on malformed output in Scenario Outline results produced by some tools

    Details

    • Similar Issues:

      Description

      What we did:
      Ran minimal cucumber feature test with Jenkins:

      Feature
      Feature: Sample Test
        Scenario Outline: Parsing scenarios with multiple examples
          Given I navigate to the root page
          Then I see the text 'home page'
          Examples:
          | a | b |
          | 1 | 2 |
      

      By a shell build step:

      shell build
      #!/bin/bash
      # Set up RVM (Ruby 2.0)
      . ~/.bashrc
      set -e
      CUCUMBER_OPTS='-f json -o cucumber.json features/sample.feature' rake cucumber
      

      What we got:
      [::1::]

      What we expected:
      No error, parsing of cucumber.json containing results from scenario outlines.

      [::1::]

      Output
      /var/lib/jenkins/.rvm/rubies/ruby-2.0.0-p353/bin/ruby -S bundle exec cucumber -f json -o cucumber.json features/sample.feature
      Using the default profile...
      Feature: Sample Test
      
        Scenario Outline: Parsing scenarios with multiple examples # features/sample.feature:2
          Given I navigate to the root page                        # features/step_definitions/navigation_steps.rb:15
          Then I see the text 'home page'                          # features/step_definitions/study_admin.rb:23
      
          Examples: 
            | a | b |
            | 1 | 2 |
      
      1 scenario (1 passed)
      2 steps (2 passed)
      0m1.080s
      Xvfb stopping
      Archiving artifacts
      [Cucumber Tests] Parsing results.
      [Cucumber Tests] parsing cucumber.json
      ERROR: Failed to archive cucumber reports
      java.io.IOException: Failed to parse Cucumber JSON
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:99)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:44)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl$ParseResultCallable.invoke(DefaultTestResultParserImpl.java:161)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl$ParseResultCallable.invoke(DefaultTestResultParserImpl.java:107)
      	at hudson.FilePath.act(FilePath.java:914)
      	at hudson.FilePath.act(FilePath.java:887)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl.parse(DefaultTestResultParserImpl.java:101)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:116)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberTestResultArchiver.perform(CucumberTestResultArchiver.java:93)
      	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:785)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:757)
      	at hudson.model.Build$BuildExecution.post2(Build.java:183)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:706)
      	at hudson.model.Run.execute(Run.java:1703)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:231)
      Caused by: org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberModelException: Step received before previous step handled!
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.GherkinCallback.step(GherkinCallback.java:160)
      	at gherkin.formatter.model.Step.replay(Step.java:68)
      	at gherkin.JSONParser.step(JSONParser.java:106)
      	at gherkin.JSONParser.parse(JSONParser.java:51)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:94)
      	... 17 more
      Build step 'Publish Cucumber test result report' changed build result to FAILURE
      Finished: FAILURE
      

        Attachments

          Activity

          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Kanstantsin Shautsou
          Path:
          src/main/java/org/jenkinsci/plugins/cucumber/jsontestsupport/CucumberJSONParser.java
          src/main/java/org/jenkinsci/plugins/cucumber/jsontestsupport/GherkinCallback.java
          http://jenkins-ci.org/commit/cucumber-testresult-plugin/14c313da7f51527df7468f39140a27ff44ffc33c
          Log:
          JENKINS-21835 Don't print stacktrace. Print friendly log.

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kanstantsin Shautsou Path: src/main/java/org/jenkinsci/plugins/cucumber/jsontestsupport/CucumberJSONParser.java src/main/java/org/jenkinsci/plugins/cucumber/jsontestsupport/GherkinCallback.java http://jenkins-ci.org/commit/cucumber-testresult-plugin/14c313da7f51527df7468f39140a27ff44ffc33c Log: JENKINS-21835 Don't print stacktrace. Print friendly log.
          Hide
          teilo James Nord added a comment -

          workaround enabled.

          Enabling this may cause tests that have failed to not cause the build to be marked as unstable - but there is no information about the result so there is nothing else that can be done.

          The cause of the issue has fixes upstream - so it is a matter of getting those fixes into all the relevant tools to get a real fix.

          Show
          teilo James Nord added a comment - workaround enabled. Enabling this may cause tests that have failed to not cause the build to be marked as unstable - but there is no information about the result so there is nothing else that can be done. The cause of the issue has fixes upstream - so it is a matter of getting those fixes into all the relevant tools to get a real fix.
          Hide
          dserodio Daniel Serodio added a comment - - edited

          Unfortunately https://github.com/cucumber/cucumber/issues/700 is a 404. Does anyone know which issue that was, and/or which Cucumber version (if any) contains the fix?

          Show
          dserodio Daniel Serodio added a comment - - edited Unfortunately https://github.com/cucumber/cucumber/issues/700 is a 404. Does anyone know which issue that was, and/or which Cucumber version (if any) contains the fix?
          Hide
          dserodio Daniel Serodio added a comment - - edited

          Ok, the cucumber repo was renamed to cucumber-ruby and gherkin was renamed to gherkin2. The links above are now:

          https://github.com/cucumber/cucumber-ruby/issues/700

          https://github.com/cucumber/gherkin2/issues/165

           

          Show
          dserodio Daniel Serodio added a comment - - edited Ok, the cucumber repo was renamed to cucumber-ruby and gherkin was renamed to gherkin2. The links above are now: https://github.com/cucumber/cucumber-ruby/issues/700 https://github.com/cucumber/gherkin2/issues/165  
          Hide
          teilo James Nord added a comment -

          Daniel Serodio thanks for the detective work. If you find the versions with fixes can you comment here?

          Show
          teilo James Nord added a comment - Daniel Serodio thanks for the detective work. If you find the versions with fixes can you comment here?

            People

            • Assignee:
              teilo James Nord
              Reporter:
              jwmach1 James Wilson
            • Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: