Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20706

Make Cobertura plug-in support multi-configuration (matrix) jobs

    Details

    • Similar Issues:

      Description

      Love the Cobertura plugin!

      I'm using it with Python build jobs that are set up as multi-configuration jobs; there is a user-defined axis defined for TOXENV which lets me use a Python tool called tox to run for each build, a "sub-build" (forgive me if this is the wrong terminology) for each of several Python versions – e.g.: py26, py27, py33, and pypy. Each of these Python versions is a "sub-build" of the build. Each of those sub-builds generates its own coverage.xml file. Here is how I set up these multi-configuration jobs:

      http://tox.readthedocs.org/en/latest/example/jenkins.html

      The Cobertura plugin kind of works with this setup in a somewhat incomplete way.

      If I go to the project page, I don't see the coverage graph (but I do see JUnit test results so I guess the JUnit plugin has support for multi-configuration builds). If I click on "Coverage Report", then I get back the "nodata" page. I can only view coverage by drilling into each of the individual environments. For example, I can click on the most recent build and then click the "py26" sub-build and then for that I can see a coverage report.

      What I would really like is to also be able to see some coverage information for the overall build – either by:

      1. (simpler) having the plugin pick one of the sub-builds as representative and showing that sub-build's coverage as the coverage for the overall build. At the very simplest, it could simply pick any one of the sub-builds as the coverage is sometimes slightly different for the sub-builds, but approximately the same, so showing any one is better than nothing. Fancier would be to allow the user to specify in the plugin configuration that one of the sub-builds is the one to use. For example, since I deploy to py26, that one is most important to me, so I might pick the py26 coverage as being the one to use for the overall build.

      2. (fancier) do some merging of the coverage across the sub-builds – e.g.: if a line is covered in any of the sub-builds then it is considered covered. This seems tricky. Another way to handle this is to leave it up to the user. I believe that the Python "coverage" tool has a merge sub-command that lets you merge multiple runs into one (I haven't used it yet though). It might be possible for me to manually run this in my build.

      If you don't have time to actively work on this issue, I am open to working on it myself, especially if you can give me a little guidance of where to look and what the overall approach might be. I've already cloned the git repo and set up Netbeans and I have messed around a little with setting breakpoints and debugging but I haven't been able to figure out yet how to get it to work. I am more of a Python guy than a Jenkins guy, but if you give me some pointers, I might be able to find my way and possibly send you a pull request.

      I think the main problem that I discovered while debugging is that in CoberturaProjectAction.getLastResultBuild when the plugin calls getAction on my build, it gets null (instead of a CoberturaBuildAction), so it doesn't consider the build to have coverage. So even if I attempt to manually copy a coverage.xml into the build directory, the plugin still doesn't think that the build has coverage because it calls getAction and gets null instead of a CoberturaBuildAction. I am guessing that at some point, the plugin registers this action (couldn't find where) and in the multi-config job case, it probably registers the action on the sub-jobs but not the parent job. Just a guess.

        Attachments

        1. build 25.png
          build 25.png
          137 kB
        2. Execute Python Script.png
          Execute Python Script.png
          13 kB
        3. Post build actions.png
          Post build actions.png
          25 kB
        4. project page.png
          project page.png
          45 kB
        5. TOXENV.png
          TOXENV.png
          11 kB
        6. TOXENV=py26 coverage results.png
          TOXENV=py26 coverage results.png
          144 kB

          Issue Links

            Activity

            Hide
            msabramo Marc Abramowitz added a comment -

            TOXENV User-defined Axix

            Show
            msabramo Marc Abramowitz added a comment - TOXENV User-defined Axix
            Hide
            msabramo Marc Abramowitz added a comment -

            "Execute Python Script" build step for tox

            Show
            msabramo Marc Abramowitz added a comment - "Execute Python Script" build step for tox
            Hide
            msabramo Marc Abramowitz added a comment -

            Post build actions for Cobertura and JUnit

            Show
            msabramo Marc Abramowitz added a comment - Post build actions for Cobertura and JUnit
            Hide
            msabramo Marc Abramowitz added a comment -

            Project page - note that JUnit test results are shown but not Cobertura

            Show
            msabramo Marc Abramowitz added a comment - Project page - note that JUnit test results are shown but not Cobertura
            Hide
            msabramo Marc Abramowitz added a comment -

            Note that coverage is shown for TOXENV=py26 "sub-build" of build #25. Coverage is not shown for build #25 overall.

            Show
            msabramo Marc Abramowitz added a comment - Note that coverage is shown for TOXENV=py26 "sub-build" of build #25. Coverage is not shown for build #25 overall.
            Hide
            msabramo Marc Abramowitz added a comment -

            Coverage is not shown for build #25 overall. Note that coverage is shown for TOXENV=py26 "sub-build" of build #25.

            Show
            msabramo Marc Abramowitz added a comment - Coverage is not shown for build #25 overall. Note that coverage is shown for TOXENV=py26 "sub-build" of build #25.
            Hide
            msabramo Marc Abramowitz added a comment -

            ssogabe: What do you think? If you don't want to do this, but can give me some pointers in the right direction, I might be able to send a PR.

            Show
            msabramo Marc Abramowitz added a comment - ssogabe : What do you think? If you don't want to do this, but can give me some pointers in the right direction, I might be able to send a PR.
            Hide
            msabramo Marc Abramowitz added a comment -

            I submitted a PR that supports matrix projects in a very simplistic way:

            https://github.com/jenkinsci/cobertura-plugin/pull/22

            This doesn't attempt to merge the coverage results from different matrix runs. It simply uses the coverage from the last matrix run as the coverage for the matrix build.

            Hopefully, this can be extended to some kind of intelligent aggregation of the coverage results.

            Show
            msabramo Marc Abramowitz added a comment - I submitted a PR that supports matrix projects in a very simplistic way: https://github.com/jenkinsci/cobertura-plugin/pull/22 This doesn't attempt to merge the coverage results from different matrix runs. It simply uses the coverage from the last matrix run as the coverage for the matrix build. Hopefully, this can be extended to some kind of intelligent aggregation of the coverage results.
            Hide
            msabramo Marc Abramowitz added a comment -

            Bruno P. Kinoshita: Since you were super helpful on IRC, I thought that I'd put a note of thanks here and obviously if you have other comments, let me know.

            Show
            msabramo Marc Abramowitz added a comment - Bruno P. Kinoshita : Since you were super helpful on IRC, I thought that I'd put a note of thanks here and obviously if you have other comments, let me know.
            Hide
            msabramo Marc Abramowitz added a comment -

            Changing assignee from Stephen Connolly to ssogabe, as the latter seems more active on this project recently.

            Show
            msabramo Marc Abramowitz added a comment - Changing assignee from Stephen Connolly to ssogabe , as the latter seems more active on this project recently.
            Hide
            msabramo Marc Abramowitz added a comment - - edited

            I have a new pull request that I think solves this in a better way:

            https://github.com/jenkinsci/cobertura-plugin/pull/26

            This works by creating a MatrixAggregator that on endRun, copies the coverage.xml from each MatrixRun into the root build's directory. Then in endBuild, those files are parsed and aggregated.

            The advantage of this approach is that it seems to aggregate the coverage in a nice way – i.e.: if my MatrixBuild does a MatrixRun for Python versions 2.6, 2.7, and 3.3 and each of those has lines that are not covered but all of the lines are covered by at least one MatrixRun, then the coverage for the overall build will be reported as 100%.

            Show
            msabramo Marc Abramowitz added a comment - - edited I have a new pull request that I think solves this in a better way: https://github.com/jenkinsci/cobertura-plugin/pull/26 This works by creating a MatrixAggregator that on endRun , copies the coverage.xml from each MatrixRun into the root build's directory. Then in endBuild , those files are parsed and aggregated. The advantage of this approach is that it seems to aggregate the coverage in a nice way – i.e.: if my MatrixBuild does a MatrixRun for Python versions 2.6, 2.7, and 3.3 and each of those has lines that are not covered but all of the lines are covered by at least one MatrixRun, then the coverage for the overall build will be reported as 100%.
            Hide
            msabramo Marc Abramowitz added a comment -

            I want to Cc a few people who have been helpful on this issue in the past:

            Show
            msabramo Marc Abramowitz added a comment - I want to Cc a few people who have been helpful on this issue in the past: emanuelez Bruno P. Kinoshita
            Hide
            jtalton James Talton added a comment -

            I pulled the latest code and merged in the changes from https://github.com/jenkinsci/cobertura-plugin/pull/26.

            The code coverage is indeed aggregated up and looks good on the matrix job.

            What is not working for us is that the code coverage thresholds are still being applied at each matrix run and not on the aggregated results.
            This causes a matrix run to fail even though the aggregated coverage is 100%.

            Show
            jtalton James Talton added a comment - I pulled the latest code and merged in the changes from https://github.com/jenkinsci/cobertura-plugin/pull/26 . The code coverage is indeed aggregated up and looks good on the matrix job. What is not working for us is that the code coverage thresholds are still being applied at each matrix run and not on the aggregated results. This causes a matrix run to fail even though the aggregated coverage is 100%.
            Hide
            msabramo Marc Abramowitz added a comment -

            https://github.com/jenkinsci/cobertura-plugin/pull/26

            > I would really like to use this feature. @msabramo seems to have done all the heavy lifting already. Can anyone from jenkins
            > comment what remains to be done to get it merged?

            I would love for someone more steeped in Java and Jenkins plug-ins to take this over and take it across the finish line...

            Show
            msabramo Marc Abramowitz added a comment - https://github.com/jenkinsci/cobertura-plugin/pull/26 > I would really like to use this feature. @msabramo seems to have done all the heavy lifting already. Can anyone from jenkins > comment what remains to be done to get it merged? I would love for someone more steeped in Java and Jenkins plug-ins to take this over and take it across the finish line...
            Hide
            macetw macetw added a comment -

            Marc Abramowitz  James Talton This issue is "in progress," but without activity in 3.5 years. Shall we call this "won't fix" or is there a different approach to refer to?

            Show
            macetw macetw added a comment - Marc Abramowitz   James Talton This issue is "in progress," but without activity in 3.5 years. Shall we call this "won't fix" or is there a different approach to refer to?

              People

              • Assignee:
                ssogabe ssogabe
                Reporter:
                msabramo Marc Abramowitz
              • Votes:
                5 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: