Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20706

Make Cobertura plug-in support multi-configuration (matrix) jobs

    Details

    • Similar Issues:

      Description

      Love the Cobertura plugin!

      I'm using it with Python build jobs that are set up as multi-configuration jobs; there is a user-defined axis defined for TOXENV which lets me use a Python tool called tox to run for each build, a "sub-build" (forgive me if this is the wrong terminology) for each of several Python versions – e.g.: py26, py27, py33, and pypy. Each of these Python versions is a "sub-build" of the build. Each of those sub-builds generates its own coverage.xml file. Here is how I set up these multi-configuration jobs:

      http://tox.readthedocs.org/en/latest/example/jenkins.html

      The Cobertura plugin kind of works with this setup in a somewhat incomplete way.

      If I go to the project page, I don't see the coverage graph (but I do see JUnit test results so I guess the JUnit plugin has support for multi-configuration builds). If I click on "Coverage Report", then I get back the "nodata" page. I can only view coverage by drilling into each of the individual environments. For example, I can click on the most recent build and then click the "py26" sub-build and then for that I can see a coverage report.

      What I would really like is to also be able to see some coverage information for the overall build – either by:

      1. (simpler) having the plugin pick one of the sub-builds as representative and showing that sub-build's coverage as the coverage for the overall build. At the very simplest, it could simply pick any one of the sub-builds as the coverage is sometimes slightly different for the sub-builds, but approximately the same, so showing any one is better than nothing. Fancier would be to allow the user to specify in the plugin configuration that one of the sub-builds is the one to use. For example, since I deploy to py26, that one is most important to me, so I might pick the py26 coverage as being the one to use for the overall build.

      2. (fancier) do some merging of the coverage across the sub-builds – e.g.: if a line is covered in any of the sub-builds then it is considered covered. This seems tricky. Another way to handle this is to leave it up to the user. I believe that the Python "coverage" tool has a merge sub-command that lets you merge multiple runs into one (I haven't used it yet though). It might be possible for me to manually run this in my build.

      If you don't have time to actively work on this issue, I am open to working on it myself, especially if you can give me a little guidance of where to look and what the overall approach might be. I've already cloned the git repo and set up Netbeans and I have messed around a little with setting breakpoints and debugging but I haven't been able to figure out yet how to get it to work. I am more of a Python guy than a Jenkins guy, but if you give me some pointers, I might be able to find my way and possibly send you a pull request.

      I think the main problem that I discovered while debugging is that in CoberturaProjectAction.getLastResultBuild when the plugin calls getAction on my build, it gets null (instead of a CoberturaBuildAction), so it doesn't consider the build to have coverage. So even if I attempt to manually copy a coverage.xml into the build directory, the plugin still doesn't think that the build has coverage because it calls getAction and gets null instead of a CoberturaBuildAction. I am guessing that at some point, the plugin registers this action (couldn't find where) and in the multi-config job case, it probably registers the action on the sub-jobs but not the parent job. Just a guess.

        Attachments

        1. build 25.png
          build 25.png
          137 kB
        2. Execute Python Script.png
          Execute Python Script.png
          13 kB
        3. Post build actions.png
          Post build actions.png
          25 kB
        4. project page.png
          project page.png
          45 kB
        5. TOXENV.png
          TOXENV.png
          11 kB
        6. TOXENV=py26 coverage results.png
          TOXENV=py26 coverage results.png
          144 kB

          Issue Links

            Activity

            msabramo Marc Abramowitz created issue -
            msabramo Marc Abramowitz made changes -
            Field Original Value New Value
            Description Love the Cobertura plugin!

            I'm using it with Python build jobs that are set up as multi-configuration jobs; there is a user-defined axis defined for TOXENV which lets me use a Python tool called tox to run for each build, a "sub-build" (forgive me if this is the wrong terminology) for each of several Python versions -- e.g.: py26, py27, py33, and pypy. Each of these Python versions is a "sub-build" of the build. Each of those sub-builds generates its own coverage.xml file.

            The Cobertura plugin kind of works with this setup in a somewhat incomplete way.

            If I go to the project page, I don't see the coverage graph. If I click on "Coverage Report", then I get back the "nodata" page. I can only view coverage by drilling into each of the individual environments. For example, I can click on the most recent build and then click the "py26" sub-build and then for that I can see a coverage report.

            What I would really like is to also be able to see some coverage information for the overall build -- either by:

            1. (simpler) having the plugin pick one of the sub-builds as representative and showing that sub-build's coverage as the coverage for the overall build. At the very simplest, it could simply pick any one of the sub-builds as the coverage is sometimes slightly different for the sub-builds, but approximately the same, so showing any one is better than nothing. Fancier would be to allow the user to specify in the plugin configuration that one of the sub-builds is the one to use. For example, since I deploy to py26, that one is most important to me, so I might pick the py26 coverage as being the one to use for the overall build.

            2. (fancier) do some merging of the coverage across the sub-builds -- e.g.: if a line is covered in any of the sub-builds then it is considered covered. This seems tricky. Another way to handle this is to leave it up to the user. I believe that the Python "coverage" tool has a merge sub-command that lets you merge multiple runs into one (I haven't used it yet though). It might be possible for me to manually run this in my build.

            If you don't have time to actively work on this issue, I am open to working on it myself, especially if you can give me a little guidance of where to look and what the overall approach might be. I've already cloned the git repo and set up Netbeans and I have messed around a little with setting breakpoints and debugging but I haven't been able to figure out yet how to get it to work. I am more of a Python guy than a Jenkins guy, but if you give me some pointers, I might be able to find my way and possibly send you a pull request.

            I think the main problem that I discovered while debugging is that in {{CoberturaProjectAction.getLastResultBuild}} when the plugin calls {{getAction}} on my build, it gets {{null}} (instead of a {{CoberturaBuildAction}}), so it doesn't consider the build to have coverage. So even if I attempt to manually copy a coverage.xml into the build directory, the plugin still doesn't think that the build has coverage because it calls {{getAction}} and gets {{null}} instead of a {{CoberturaBuildAction}}. I am guessing that at some point, the plugin registers this action (couldn't find where) and in the multi-config job case, it probably registers the action on the sub-jobs but not the parent job. Just a guess.
            Love the Cobertura plugin!

            I'm using it with Python build jobs that are set up as multi-configuration jobs; there is a user-defined axis defined for TOXENV which lets me use a Python tool called tox to run for each build, a "sub-build" (forgive me if this is the wrong terminology) for each of several Python versions -- e.g.: py26, py27, py33, and pypy. Each of these Python versions is a "sub-build" of the build. Each of those sub-builds generates its own coverage.xml file. Here is how I set up these multi-configuration jobs:

            http://tox.readthedocs.org/en/latest/example/jenkins.html

            The Cobertura plugin kind of works with this setup in a somewhat incomplete way.

            If I go to the project page, I don't see the coverage graph. If I click on "Coverage Report", then I get back the "nodata" page. I can only view coverage by drilling into each of the individual environments. For example, I can click on the most recent build and then click the "py26" sub-build and then for that I can see a coverage report.

            What I would really like is to also be able to see some coverage information for the overall build -- either by:

            1. (simpler) having the plugin pick one of the sub-builds as representative and showing that sub-build's coverage as the coverage for the overall build. At the very simplest, it could simply pick any one of the sub-builds as the coverage is sometimes slightly different for the sub-builds, but approximately the same, so showing any one is better than nothing. Fancier would be to allow the user to specify in the plugin configuration that one of the sub-builds is the one to use. For example, since I deploy to py26, that one is most important to me, so I might pick the py26 coverage as being the one to use for the overall build.

            2. (fancier) do some merging of the coverage across the sub-builds -- e.g.: if a line is covered in any of the sub-builds then it is considered covered. This seems tricky. Another way to handle this is to leave it up to the user. I believe that the Python "coverage" tool has a merge sub-command that lets you merge multiple runs into one (I haven't used it yet though). It might be possible for me to manually run this in my build.

            If you don't have time to actively work on this issue, I am open to working on it myself, especially if you can give me a little guidance of where to look and what the overall approach might be. I've already cloned the git repo and set up Netbeans and I have messed around a little with setting breakpoints and debugging but I haven't been able to figure out yet how to get it to work. I am more of a Python guy than a Jenkins guy, but if you give me some pointers, I might be able to find my way and possibly send you a pull request.

            I think the main problem that I discovered while debugging is that in {{CoberturaProjectAction.getLastResultBuild}} when the plugin calls {{getAction}} on my build, it gets {{null}} (instead of a {{CoberturaBuildAction}}), so it doesn't consider the build to have coverage. So even if I attempt to manually copy a coverage.xml into the build directory, the plugin still doesn't think that the build has coverage because it calls {{getAction}} and gets {{null}} instead of a {{CoberturaBuildAction}}. I am guessing that at some point, the plugin registers this action (couldn't find where) and in the multi-config job case, it probably registers the action on the sub-jobs but not the parent job. Just a guess.
            msabramo Marc Abramowitz made changes -
            Attachment TOXENV.png [ 24726 ]
            msabramo Marc Abramowitz made changes -
            Labels multi-configuration
            msabramo Marc Abramowitz made changes -
            Attachment Execute Python Script.png [ 24727 ]
            msabramo Marc Abramowitz made changes -
            Attachment Post build actions.png [ 24728 ]
            msabramo Marc Abramowitz made changes -
            Attachment project page.png [ 24729 ]
            msabramo Marc Abramowitz made changes -
            Description Love the Cobertura plugin!

            I'm using it with Python build jobs that are set up as multi-configuration jobs; there is a user-defined axis defined for TOXENV which lets me use a Python tool called tox to run for each build, a "sub-build" (forgive me if this is the wrong terminology) for each of several Python versions -- e.g.: py26, py27, py33, and pypy. Each of these Python versions is a "sub-build" of the build. Each of those sub-builds generates its own coverage.xml file. Here is how I set up these multi-configuration jobs:

            http://tox.readthedocs.org/en/latest/example/jenkins.html

            The Cobertura plugin kind of works with this setup in a somewhat incomplete way.

            If I go to the project page, I don't see the coverage graph. If I click on "Coverage Report", then I get back the "nodata" page. I can only view coverage by drilling into each of the individual environments. For example, I can click on the most recent build and then click the "py26" sub-build and then for that I can see a coverage report.

            What I would really like is to also be able to see some coverage information for the overall build -- either by:

            1. (simpler) having the plugin pick one of the sub-builds as representative and showing that sub-build's coverage as the coverage for the overall build. At the very simplest, it could simply pick any one of the sub-builds as the coverage is sometimes slightly different for the sub-builds, but approximately the same, so showing any one is better than nothing. Fancier would be to allow the user to specify in the plugin configuration that one of the sub-builds is the one to use. For example, since I deploy to py26, that one is most important to me, so I might pick the py26 coverage as being the one to use for the overall build.

            2. (fancier) do some merging of the coverage across the sub-builds -- e.g.: if a line is covered in any of the sub-builds then it is considered covered. This seems tricky. Another way to handle this is to leave it up to the user. I believe that the Python "coverage" tool has a merge sub-command that lets you merge multiple runs into one (I haven't used it yet though). It might be possible for me to manually run this in my build.

            If you don't have time to actively work on this issue, I am open to working on it myself, especially if you can give me a little guidance of where to look and what the overall approach might be. I've already cloned the git repo and set up Netbeans and I have messed around a little with setting breakpoints and debugging but I haven't been able to figure out yet how to get it to work. I am more of a Python guy than a Jenkins guy, but if you give me some pointers, I might be able to find my way and possibly send you a pull request.

            I think the main problem that I discovered while debugging is that in {{CoberturaProjectAction.getLastResultBuild}} when the plugin calls {{getAction}} on my build, it gets {{null}} (instead of a {{CoberturaBuildAction}}), so it doesn't consider the build to have coverage. So even if I attempt to manually copy a coverage.xml into the build directory, the plugin still doesn't think that the build has coverage because it calls {{getAction}} and gets {{null}} instead of a {{CoberturaBuildAction}}. I am guessing that at some point, the plugin registers this action (couldn't find where) and in the multi-config job case, it probably registers the action on the sub-jobs but not the parent job. Just a guess.
            Love the Cobertura plugin!

            I'm using it with Python build jobs that are set up as multi-configuration jobs; there is a user-defined axis defined for TOXENV which lets me use a Python tool called tox to run for each build, a "sub-build" (forgive me if this is the wrong terminology) for each of several Python versions -- e.g.: py26, py27, py33, and pypy. Each of these Python versions is a "sub-build" of the build. Each of those sub-builds generates its own coverage.xml file. Here is how I set up these multi-configuration jobs:

            http://tox.readthedocs.org/en/latest/example/jenkins.html

            The Cobertura plugin kind of works with this setup in a somewhat incomplete way.

            If I go to the project page, I don't see the coverage graph (but I *do* see JUnit test results so I guess the JUnit plugin has support for multi-configuration builds). If I click on "Coverage Report", then I get back the "nodata" page. I can only view coverage by drilling into each of the individual environments. For example, I can click on the most recent build and then click the "py26" sub-build and then for that I can see a coverage report.

            What I would really like is to also be able to see some coverage information for the overall build -- either by:

            1. (simpler) having the plugin pick one of the sub-builds as representative and showing that sub-build's coverage as the coverage for the overall build. At the very simplest, it could simply pick any one of the sub-builds as the coverage is sometimes slightly different for the sub-builds, but approximately the same, so showing any one is better than nothing. Fancier would be to allow the user to specify in the plugin configuration that one of the sub-builds is the one to use. For example, since I deploy to py26, that one is most important to me, so I might pick the py26 coverage as being the one to use for the overall build.

            2. (fancier) do some merging of the coverage across the sub-builds -- e.g.: if a line is covered in any of the sub-builds then it is considered covered. This seems tricky. Another way to handle this is to leave it up to the user. I believe that the Python "coverage" tool has a merge sub-command that lets you merge multiple runs into one (I haven't used it yet though). It might be possible for me to manually run this in my build.

            If you don't have time to actively work on this issue, I am open to working on it myself, especially if you can give me a little guidance of where to look and what the overall approach might be. I've already cloned the git repo and set up Netbeans and I have messed around a little with setting breakpoints and debugging but I haven't been able to figure out yet how to get it to work. I am more of a Python guy than a Jenkins guy, but if you give me some pointers, I might be able to find my way and possibly send you a pull request.

            I think the main problem that I discovered while debugging is that in {{CoberturaProjectAction.getLastResultBuild}} when the plugin calls {{getAction}} on my build, it gets {{null}} (instead of a {{CoberturaBuildAction}}), so it doesn't consider the build to have coverage. So even if I attempt to manually copy a coverage.xml into the build directory, the plugin still doesn't think that the build has coverage because it calls {{getAction}} and gets {{null}} instead of a {{CoberturaBuildAction}}. I am guessing that at some point, the plugin registers this action (couldn't find where) and in the multi-config job case, it probably registers the action on the sub-jobs but not the parent job. Just a guess.
            msabramo Marc Abramowitz made changes -
            Attachment TOXENV=py26 coverage results.png [ 24730 ]
            msabramo Marc Abramowitz made changes -
            Attachment build 25.png [ 24731 ]
            msabramo Marc Abramowitz made changes -
            Summary Support multi-configuration jobs with user-defined axes Support multi-configuration (matrix) jobs
            Labels multi-configuration matrix multi-configuration
            msabramo Marc Abramowitz made changes -
            Summary Support multi-configuration (matrix) jobs Make Cobertura plug-in support multi-configuration (matrix) jobs
            msabramo Marc Abramowitz made changes -
            Labels matrix multi-configuration cobertura cobertura, coverage matrix multi-configuration
            msabramo Marc Abramowitz made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            msabramo Marc Abramowitz made changes -
            Status In Progress [ 3 ] Open [ 1 ]
            msabramo Marc Abramowitz made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            msabramo Marc Abramowitz made changes -
            Assignee stephenconnolly [ stephenconnolly ] ssogabe [ ssogabe ]
            jprosevear JP Rosevear made changes -
            Link This issue is duplicated by JENKINS-6314 [ JENKINS-6314 ]
            jglick Jesse Glick made changes -
            Labels cobertura cobertura, coverage matrix multi-configuration matrix multi-configuration
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 152515 ] JNJira + In-Review [ 185481 ]

              People

              • Assignee:
                ssogabe ssogabe
                Reporter:
                msabramo Marc Abramowitz
              • Votes:
                5 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated: