Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-1267

cc.xml reports failure for last build, although the last build was unstable/successful

    Details

    • Type: Bug
    • Status: In Review (View Workflow)
    • Priority: Major
    • Resolution: Won't Fix
    • Component/s: junit-plugin
    • Labels:
    • Environment:
      Platform: All, OS: All
    • Similar Issues:

      Description

      cc.xml reports failure for the last build of a project , although the last build
      was unstable/successful. It shows the correct lastBuildLabel, but the wrong
      lastBuildStatus.

        Attachments

          Activity

          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          Can you give us more details?

          The last few jobs and their timestamps, their results, and whether a new build
          is being scheduled or not. Also, please paste the cc.xml contents

          Show
          kohsuke Kohsuke Kawaguchi added a comment - Can you give us more details? The last few jobs and their timestamps, their results, and whether a new build is being scheduled or not. Also, please paste the cc.xml contents
          Hide
          canhuth canhuth added a comment -

          > The last few jobs and their timestamps,
          > their results, and whether a new build
          > is being scheduled or not.

          I do not know which jobs were run or wether the jobs that were running after the
          job in question succeeded or not. Hudson has been restarted since and its output
          was not logged to a file. For us the problem persists however, it is not a
          temporary phenomenon.

          > Also, please paste the cc.xml contents

          From http://hudson.ramfelt.se/cc.xml, note the second last project "XBMC TV",
          which has been listed with lastBuildStatus="Failure":

          <Projects>
          <Project activity="Sleeping" lastBuildTime="2007-08-21T20:30:53Z"
          webUrl="http://hudson.ramfelt.se/job/Airduel/" lastBuildLabel="10"
          lastBuildStatus="Success" name="Airduel"/>

          <Project activity="Sleeping" lastBuildTime="2008-02-10T23:03:14Z"
          webUrl="http://hudson.ramfelt.se/job/Hudson/" lastBuildLabel="357"
          lastBuildStatus="Success" name="Hudson"/>

          <Project activity="Sleeping" lastBuildTime="2008-01-17T13:24:59Z"
          webUrl="http://hudson.ramfelt.se/job/Hudson free style/" lastBuildLabel="11"
          lastBuildStatus="Success" name="Hudson free style"/>

          <Project activity="Sleeping" lastBuildTime="2008-02-11T07:01:58Z"
          webUrl="http://hudson.ramfelt.se/job/NMaven/" lastBuildLabel="45"
          lastBuildStatus="Failure" name="NMaven"/>

          <Project activity="Sleeping" lastBuildTime="2008-01-10T07:35:21Z"
          webUrl="http://hudson.ramfelt.se/job/NMaven maven job/" lastBuildLabel="25"
          lastBuildStatus="Failure" name="NMaven maven job"/>

          <Project activity="Sleeping" lastBuildTime="2008-01-29T18:41:52Z"
          webUrl="http://hudson.ramfelt.se/job/NUnit 2.0/" lastBuildLabel="5"
          lastBuildStatus="Failure" name="NUnit 2.0"/>

          <Project activity="Sleeping" lastBuildTime="2007-11-20T20:35:07Z"
          webUrl="http://hudson.ramfelt.se/job/XBMC Test/" lastBuildLabel="4"
          lastBuildStatus="Success" name="XBMC Test"/>

          <Project activity="Sleeping" lastBuildTime="2007-11-20T20:36:21Z"
          webUrl="http://hudson.ramfelt.se/job/XBMC TV/" lastBuildLabel="22"
          lastBuildStatus="Failure" name="XBMC TV"/>

          <Project activity="Sleeping" lastBuildTime="2007-12-30T21:38:24Z"
          webUrl="http://hudson.ramfelt.se/job/XBMC TV - Verification/"
          lastBuildLabel="11" lastBuildStatus="Success" name="XBMC TV - Verification"/>
          </Projects>

          Here is the project information from the front page (http://hudson.ramfelt.se/),
          note the "XBMC TV" project (second last, again) to be listed as "Unstable":

          Success
          100%
          W Description %
          Test Result: 0 tests failing out of a total of 40 tests. 100
          Build stability: No recent builds failed. 100
          Airduel 5 months 23 days (#10) 6 months 25 days (#1) 26 seconds

          Success
          100%
          W Description %
          Test Result: 0 tests failing out of a total of 189 tests. 100
          Build stability: No recent builds failed. 100
          Hudson 9 hours 48 minutes (#357) 4 days 9 hours (#345) 1 hour 21 minutes

          Success
          79%
          W Description %
          Build stability: 1 out of the last 5 builds failed. 79
          Test Result: 0 tests failing out of a total of 185 tests. 100
          Hudson free style 24 days (#11) 24 days (#10) 13 minutes

          Unstable
          91%
          W Description %
          Test Result: 3 tests failing out of a total of 35 tests. 91
          Build stability: No recent builds failed. 100
          NMaven 1 hour 49 minutes (#45) 6 days 13 hours (#35) 18 minutes

          Failed
          79%
          W Description %
          Build stability: 1 out of the last 5 builds failed. 79
          Test Result: 0 tests failing out of a total of 17 tests. 100
          NMaven maven job 1 month 7 days (#23) 1 month 2 days (#25) 2 minutes 59
          seconds

          Failed
          74%
          W Description %
          Build stability: 1 out of the last 4 builds failed. 74
          Test Result: 12 tests failing out of a total of 1173 tests. 98
          NUnit 2.0 15 days (#4) 12 days (#5) 9 minutes 45 seconds

          Success
          24%
          W Description %
          Build stability: 3 out of the last 4 builds failed. 24
          XBMC Test 2 months 22 days (#4) 2 months 22 days (#3) 1 minute 53 seconds

          Unstable
          33%
          W Description %
          Number of pylint violations is 804 33
          Test Result: 56 tests failing out of a total of 130 tests. 56
          Build stability: 1 out of the last 5 builds failed. 79
          XBMC TV 2 months 22 days (#22) 5 months 15 days (#16) 6 minutes 39 seconds

          Success
          79%
          W Description %
          Build stability: 1 out of the last 5 builds failed. 79
          Test Result: 0 tests failing out of a total of 2 tests. 100
          XBMC TV - Verification 1 month 12 days (#11) 6 months 18 days (#7) 59 seconds

          Show
          canhuth canhuth added a comment - > The last few jobs and their timestamps, > their results, and whether a new build > is being scheduled or not. I do not know which jobs were run or wether the jobs that were running after the job in question succeeded or not. Hudson has been restarted since and its output was not logged to a file. For us the problem persists however, it is not a temporary phenomenon. > Also, please paste the cc.xml contents From http://hudson.ramfelt.se/cc.xml , note the second last project "XBMC TV", which has been listed with lastBuildStatus="Failure": <Projects> <Project activity="Sleeping" lastBuildTime="2007-08-21T20:30:53Z" webUrl="http://hudson.ramfelt.se/job/Airduel/" lastBuildLabel="10" lastBuildStatus="Success" name="Airduel"/> <Project activity="Sleeping" lastBuildTime="2008-02-10T23:03:14Z" webUrl="http://hudson.ramfelt.se/job/Hudson/" lastBuildLabel="357" lastBuildStatus="Success" name="Hudson"/> <Project activity="Sleeping" lastBuildTime="2008-01-17T13:24:59Z" webUrl="http://hudson.ramfelt.se/job/Hudson free style/" lastBuildLabel="11" lastBuildStatus="Success" name="Hudson free style"/> <Project activity="Sleeping" lastBuildTime="2008-02-11T07:01:58Z" webUrl="http://hudson.ramfelt.se/job/NMaven/" lastBuildLabel="45" lastBuildStatus="Failure" name="NMaven"/> <Project activity="Sleeping" lastBuildTime="2008-01-10T07:35:21Z" webUrl="http://hudson.ramfelt.se/job/NMaven maven job/" lastBuildLabel="25" lastBuildStatus="Failure" name="NMaven maven job"/> <Project activity="Sleeping" lastBuildTime="2008-01-29T18:41:52Z" webUrl="http://hudson.ramfelt.se/job/NUnit 2.0/" lastBuildLabel="5" lastBuildStatus="Failure" name="NUnit 2.0"/> <Project activity="Sleeping" lastBuildTime="2007-11-20T20:35:07Z" webUrl="http://hudson.ramfelt.se/job/XBMC Test/" lastBuildLabel="4" lastBuildStatus="Success" name="XBMC Test"/> <Project activity="Sleeping" lastBuildTime="2007-11-20T20:36:21Z" webUrl="http://hudson.ramfelt.se/job/XBMC TV/" lastBuildLabel="22" lastBuildStatus="Failure" name="XBMC TV"/> <Project activity="Sleeping" lastBuildTime="2007-12-30T21:38:24Z" webUrl="http://hudson.ramfelt.se/job/XBMC TV - Verification/" lastBuildLabel="11" lastBuildStatus="Success" name="XBMC TV - Verification"/> </Projects> Here is the project information from the front page ( http://hudson.ramfelt.se/ ), note the "XBMC TV" project (second last, again) to be listed as "Unstable": Success 100% W Description % Test Result: 0 tests failing out of a total of 40 tests. 100 Build stability: No recent builds failed. 100 Airduel 5 months 23 days (#10) 6 months 25 days (#1) 26 seconds Success 100% W Description % Test Result: 0 tests failing out of a total of 189 tests. 100 Build stability: No recent builds failed. 100 Hudson 9 hours 48 minutes (#357) 4 days 9 hours (#345) 1 hour 21 minutes Success 79% W Description % Build stability: 1 out of the last 5 builds failed. 79 Test Result: 0 tests failing out of a total of 185 tests. 100 Hudson free style 24 days (#11) 24 days (#10) 13 minutes Unstable 91% W Description % Test Result: 3 tests failing out of a total of 35 tests. 91 Build stability: No recent builds failed. 100 NMaven 1 hour 49 minutes (#45) 6 days 13 hours (#35) 18 minutes Failed 79% W Description % Build stability: 1 out of the last 5 builds failed. 79 Test Result: 0 tests failing out of a total of 17 tests. 100 NMaven maven job 1 month 7 days (#23) 1 month 2 days (#25) 2 minutes 59 seconds Failed 74% W Description % Build stability: 1 out of the last 4 builds failed. 74 Test Result: 12 tests failing out of a total of 1173 tests. 98 NUnit 2.0 15 days (#4) 12 days (#5) 9 minutes 45 seconds Success 24% W Description % Build stability: 3 out of the last 4 builds failed. 24 XBMC Test 2 months 22 days (#4) 2 months 22 days (#3) 1 minute 53 seconds Unstable 33% W Description % Number of pylint violations is 804 33 Test Result: 56 tests failing out of a total of 130 tests. 56 Build stability: 1 out of the last 5 builds failed. 79 XBMC TV 2 months 22 days (#22) 5 months 15 days (#16) 6 minutes 39 seconds Success 79% W Description % Build stability: 1 out of the last 5 builds failed. 79 Test Result: 0 tests failing out of a total of 2 tests. 100 XBMC TV - Verification 1 month 12 days (#11) 6 months 18 days (#7) 59 seconds
          Hide
          canhuth canhuth added a comment -

          Looking at Functions.java:

          /**

          • Converts the Hudson build status to CruiseControl build status,
          • which is either Success, Failure, Exception, or Unknown.
            */
            public static String toCCStatus(Item i) {
            if (i instanceof Job)
            Unknown macro: { Job j = (Job) i; switch (j.getIconColor().noAnime()) { case ABORTED: case RED: case YELLOW: return "Failure"; case BLUE: return "Success"; case DISABLED: case GREY: return "Unknown"; } }

            return "Unknown";
            }

          Given this it seems that an unstable build is considered a failure. When Hudson
          considers a build successful but unstable when it has only some tests failing
          maybe it is "more correct" to interpret it as a "Success" in cc.xml, too.

          Show
          canhuth canhuth added a comment - Looking at Functions.java: /** Converts the Hudson build status to CruiseControl build status, which is either Success, Failure, Exception, or Unknown. */ public static String toCCStatus(Item i) { if (i instanceof Job) Unknown macro: { Job j = (Job) i; switch (j.getIconColor().noAnime()) { case ABORTED: case RED: case YELLOW: return "Failure"; case BLUE: return "Success"; case DISABLED: case GREY: return "Unknown"; } } return "Unknown"; } Given this it seems that an unstable build is considered a failure. When Hudson considers a build successful but unstable when it has only some tests failing maybe it is "more correct" to interpret it as a "Success" in cc.xml, too.
          Hide
          canhuth canhuth added a comment -

          Created an attachment (id=210)
          Patch to consider unstable builds to be successful builds in case of "cc.xml"

          Show
          canhuth canhuth added a comment - Created an attachment (id=210) Patch to consider unstable builds to be successful builds in case of "cc.xml"
          Hide
          kohsuke Kohsuke Kawaguchi added a comment -

          Given that cc.xml doesn't have a state that directly corresponds to Hudson's
          "unstable" state, we have to make a choice how this unstable state in Hudson is
          mapped to a cc.xml state.

          The current code makes a choice to map this to a failure. Given how other tools
          operate, such as Maven, I think it is correct to classify test failures as a
          failure.

          If we report this as a success, people will not notice test regressions. That
          defeats the purpose of Hudson.

          Show
          kohsuke Kohsuke Kawaguchi added a comment - Given that cc.xml doesn't have a state that directly corresponds to Hudson's "unstable" state, we have to make a choice how this unstable state in Hudson is mapped to a cc.xml state. The current code makes a choice to map this to a failure. Given how other tools operate, such as Maven, I think it is correct to classify test failures as a failure. If we report this as a success, people will not notice test regressions. That defeats the purpose of Hudson.
          Hide
          ssbarnea Sorin Sbarnea added a comment -

          How about code coverage? Our Jenkins instance is configured to mark a build as unstable if cove coverage goes down, which happens very often, especially because is measured using two decimals, so just removing one line of code somewhere can make the build unstable, and even worse to display it as a failure in the all cc.xml consumers.

          How can we fix this?

          Show
          ssbarnea Sorin Sbarnea added a comment - How about code coverage? Our Jenkins instance is configured to mark a build as unstable if cove coverage goes down, which happens very often, especially because is measured using two decimals, so just removing one line of code somewhere can make the build unstable, and even worse to display it as a failure in the all cc.xml consumers. How can we fix this?
          Hide
          ssbarnea Sorin Sbarnea added a comment -

          Kohsuke Kawaguchi, now after 9 years, could we revise that decision and start to expose this information inside cc.xml files? There is nothing preventing us from exposing the extended status as a new property.

          The reality is that for big projects it is normal to have unstable builds because you always have bugs that are not sorted. Also this practice is important for test driven development, where we are supposed to add tests for bugs before fixing them. Fixing a bug could take, well, years or forever.

          Show
          ssbarnea Sorin Sbarnea added a comment - Kohsuke Kawaguchi , now after 9 years, could we revise that decision and start to expose this information inside cc.xml files? There is nothing preventing us from exposing the extended status as a new property. The reality is that for big projects it is normal to have unstable builds because you always have bugs that are not sorted. Also this practice is important for test driven development, where we are supposed to add tests for bugs before fixing them. Fixing a bug could take, well, years or forever.

            People

            • Assignee:
              Unassigned
              Reporter:
              canhuth canhuth
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: