Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15570

Coverage report includes classes that have been excluded from Jacoco analysis

    Details

    • Similar Issues:

      Description

      In my Maven config there are a number of classes that are excluded from Jacoco analysis. The report generated by Jacoco does not include these classes. The report generated by the Jenkins Jacoco plugin does includes these classes which gives a misleading picture and makes it difficult to set any coverage thresholds. I think this may be related to JENKINS-14975.

        Attachments

          Activity

          Hide
          ognjenb Ognjen Bubalo added a comment -

          Hi,

          Please give us more information.

          Which plugin version do you use?
          What is your current configuration on the configuration page of JaCoCo plugin?

          Cheers,
          Ogi

          Show
          ognjenb Ognjen Bubalo added a comment - Hi, Please give us more information. Which plugin version do you use? What is your current configuration on the configuration page of JaCoCo plugin? Cheers, Ogi
          Hide
          rflob Rune Flobakk added a comment -

          I am guessing this is related to classes which are generated, e.g. from WSDLs. A common use case is to generate Java source files which end up in target/generated-sources/**. This folder then becomes part of the source folder of the project, and compiled as usual into target/classes and packaged as usual. But these classes should not be part of the coverage report.

          In my case, I have a Maven module which only purpose is to generate such classes and package them into a jar. The module does not even have a src-folder because everything is generated from WSDLs/XSDs. The jacoco-maven-plugin correctly does not produce any jacoco.exec file for this module, but it seems like the Jenkins plugin sees the output classes, and no jacoco.exec, and interprets that as zero coverage.

          Show
          rflob Rune Flobakk added a comment - I am guessing this is related to classes which are generated, e.g. from WSDLs. A common use case is to generate Java source files which end up in target/generated-sources/**. This folder then becomes part of the source folder of the project, and compiled as usual into target/classes and packaged as usual. But these classes should not be part of the coverage report. In my case, I have a Maven module which only purpose is to generate such classes and package them into a jar. The module does not even have a src-folder because everything is generated from WSDLs/XSDs. The jacoco-maven-plugin correctly does not produce any jacoco.exec file for this module, but it seems like the Jenkins plugin sees the output classes, and no jacoco.exec, and interprets that as zero coverage.
          Hide
          japgolly David Barri added a comment -

          I'm having this issue too. The relevant part of my POM looks like this:

          <plugin>
          	<groupId>org.jacoco</groupId>
          	<artifactId>jacoco-maven-plugin</artifactId>
          	<version>0.6.0.201210061924</version>
          	<executions>
          		<execution>
          			<id>coverage-agent</id>
          			<phase>process-test-classes</phase>
          			<goals>
          				<goal>prepare-agent</goal>
          			</goals>
          		</execution>
          		<execution>
          			<id>coverage-report</id>
          			<phase>test</phase>
          			<goals>
          				<goal>report</goal>
          			</goals>
          		</execution>
          	</executions>
          	<configuration>
          		<excludes>
          			<!-- Exclude classes generated by AndroidAnnotations -->
          			<exclude>**/*_.*</exclude>
          			<exclude>**/*_$*</exclude>
          		</excludes>
          	</configuration>
          </plugin>
          

          When I generate a Jacoco HTML report, classes that end in an underscore like ExampleActivity_ do not appear because of the exclude settings in my POM. However from Jenkins all of those classes that the Maven build excludes are visible again.

          My Jenkins settings for this plugin are the defaults.

          Thanks!
          David

          Show
          japgolly David Barri added a comment - I'm having this issue too. The relevant part of my POM looks like this: <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.6.0.201210061924</version> <executions> <execution> <id>coverage-agent</id> <phase>process-test-classes</phase> <goals> <goal>prepare-agent</goal> </goals> </execution> <execution> <id>coverage-report</id> <phase>test</phase> <goals> <goal>report</goal> </goals> </execution> </executions> <configuration> <excludes> <!-- Exclude classes generated by AndroidAnnotations --> <exclude>**/*_.*</exclude> <exclude>**/*_$*</exclude> </excludes> </configuration> </plugin> When I generate a Jacoco HTML report, classes that end in an underscore like ExampleActivity_ do not appear because of the exclude settings in my POM. However from Jenkins all of those classes that the Maven build excludes are visible again. My Jenkins settings for this plugin are the defaults. Thanks! David
          Hide
          ognjenb Ognjen Bubalo added a comment -

          Is this issue still actual?

          If yes, please show your configuration (given paths) of JaCoCo.

          Thanks,
          ogi

          Show
          ognjenb Ognjen Bubalo added a comment - Is this issue still actual? If yes, please show your configuration (given paths) of JaCoCo. Thanks, ogi
          Hide
          zlika Thomas Zlika added a comment -

          Yes, the problem is still present with the latest 1.0.9 version.
          I have some packages excluded from code coverage using the "excludes" parameter of the jacoco-maven-plugin. The HTML page generated by this plugin are correct (the excludes packages are not taken into account). However, the Jenkins plugin take them into account and display a code coverage of 0%.
          The configuration for the Jenkins plugin is the standard one:

          • Path to exec files: **/target/jacoco.exec
          • Path to class dir: **/target/classes
            Path to source dir: **/src/main/java
          Show
          zlika Thomas Zlika added a comment - Yes, the problem is still present with the latest 1.0.9 version. I have some packages excluded from code coverage using the "excludes" parameter of the jacoco-maven-plugin. The HTML page generated by this plugin are correct (the excludes packages are not taken into account). However, the Jenkins plugin take them into account and display a code coverage of 0%. The configuration for the Jenkins plugin is the standard one: Path to exec files: **/target/jacoco.exec Path to class dir: **/target/classes Path to source dir: **/src/main/java
          Hide
          mfriedenhagen Mirko Friedenhagen added a comment - - edited

          Same here, I exclude the generated HelpMojo:

          Maven only needs the exclusion for the report mojo, though. jacoco's version is: 0.6.1.201212231917.
          Maybe the Jenkins-Plugin should just drop all .class files where no .java files are found?

          Show
          mfriedenhagen Mirko Friedenhagen added a comment - - edited Same here, I exclude the generated HelpMojo : in build https://github.com/1and1/bill-of-materials-maven-plugin/blob/a7c20afe67620df3c3be0458ddd23a596398af33/pom.xml#L134 and reporting https://github.com/1and1/bill-of-materials-maven-plugin/blob/a7c20afe67620df3c3be0458ddd23a596398af33/pom.xml#L166 Maven only needs the exclusion for the report mojo, though. jacoco 's version is: 0.6.1.201212231917. Maybe the Jenkins-Plugin should just drop all .class files where no .java files are found?
          Hide
          ognjenb Ognjen Bubalo added a comment -

          Hi,

          Hopefully I'll implement this feature and release a new version in the next 2 days.

          Cheers,
          Ogi

          Show
          ognjenb Ognjen Bubalo added a comment - Hi, Hopefully I'll implement this feature and release a new version in the next 2 days. Cheers, Ogi
          Hide
          ognjenb Ognjen Bubalo added a comment -

          It will be available in the next release.

          Show
          ognjenb Ognjen Bubalo added a comment - It will be available in the next release.
          Show
          ognjenb Ognjen Bubalo added a comment - Check this: https://groups.google.com/d/topic/jenkins-jacoco-plugin-mailing-list/LCWg0ukJ_wg/discussion
          Hide
          librucha Libor Ondrusek added a comment -

          In version of plugin 10.0.16 and Jenkins 1.583 it is not working again.
          I configured pom for exclusion one class from report, but Jenkins still compute with this class.

          Show
          librucha Libor Ondrusek added a comment - In version of plugin 10.0.16 and Jenkins 1.583 it is not working again. I configured pom for exclusion one class from report, but Jenkins still compute with this class.
          Hide
          kozaka Akos Kozak added a comment - - edited

          Still not working 100%-ly with plugin version 1.0.16 and Jenkins 1.563.

          I made a smaller maven project where I got api and be modules.

          I set:

          pom.xml
                                      <execution>
                                          <id>check-coverage</id>
                                          <goals>
                                              <goal>check</goal>
                                          </goals>
                                          <configuration>
                                              <rules>
                                                  <rule>
                                                      <element>BUNDLE</element>
                                                      <excludes>
                                                          <exclude>*api*</exclude>
                                                      </excludes>
                                                      <limits>
                                                          <limit>
                                                              <counter>LINE</counter>
                                                              <value>COVEREDRATIO</value>
                                                              <minimum>0.25</minimum>
                                                          </limit>
                                                      </limits>
                                                  </rule>
                                              </rules>
                                          </configuration>
                                      </execution>
          

          And the plugin works well. No api is visible in the coverage.

          But if I do the same on our large project, if from one module Test a call will be done on an api, and therefor in the be classes this will be visible. However I would like to exclude api.

          Module A
          A-api: Interface to A service
          A-be : Service A implementation

          Module B
          B-api: Interface to B service
          B-be: Service B implementation

          From Service B implementation we call a method over A-api Interface to A service implementation. Therefore, the B-api will be excluded, but the A-api not! This is a bit disturbing, because the A-api should be also excluded.

          I hope I have right, and it is understandable what I wrote!

          Show
          kozaka Akos Kozak added a comment - - edited Still not working 100%-ly with plugin version 1.0.16 and Jenkins 1.563. I made a smaller maven project where I got api and be modules. I set: pom.xml <execution> <id>check-coverage</id> <goals> <goal>check</goal> </goals> <configuration> <rules> <rule> <element>BUNDLE</element> <excludes> <exclude>*api*</exclude> </excludes> <limits> <limit> <counter>LINE</counter> <value>COVEREDRATIO</value> <minimum>0.25</minimum> </limit> </limits> </rule> </rules> </configuration> </execution> And the plugin works well. No api is visible in the coverage. But if I do the same on our large project, if from one module Test a call will be done on an api, and therefor in the be classes this will be visible. However I would like to exclude api. Module A A-api: Interface to A service A-be : Service A implementation Module B B-api: Interface to B service B-be: Service B implementation From Service B implementation we call a method over A-api Interface to A service implementation. Therefore, the B-api will be excluded, but the A-api not! This is a bit disturbing, because the A-api should be also excluded. I hope I have right, and it is understandable what I wrote!
          Hide
          cschulz C. S. added a comment -

          We ran into the same issue. It is important for us to get it working, because we have a lot of generated classes, which should be excluded and it would be a mess to set up each Jenkins job according the new classes.

          So we decided to exclude the classes from the report via pom. But it seems that the jacoco plugin does its own match from classes to exec file.

          Show
          cschulz C. S. added a comment - We ran into the same issue. It is important for us to get it working, because we have a lot of generated classes, which should be excluded and it would be a mess to set up each Jenkins job according the new classes. So we decided to exclude the classes from the report via pom. But it seems that the jacoco plugin does its own match from classes to exec file.
          Hide
          lordofthemoon Raj Bhaskar added a comment -

          I think I'm having the same issue. I package the Oracle JDBC driver as a local jar in my application (as it's not on maven) but it looks like although I'm excluding it in my pom, the plugin is still including it when doing its analysis.

          Show
          lordofthemoon Raj Bhaskar added a comment - I think I'm having the same issue. I package the Oracle JDBC driver as a local jar in my application (as it's not on maven) but it looks like although I'm excluding it in my pom, the plugin is still including it when doing its analysis.
          Hide
          zteve Steve Powell added a comment -

          This is an issue for us also. In a maven build with the jacoco plugin we exclude all the classes in a package; and jacoco reports in the console log that these have been excluded.

          In the jacoco check phase, no further inclusions/exclusions are supplied but thresholds are provided.

          The coverage report given by Jenkins seems to include those classes, the coverage percentages (package, line and so on) remaining the same as without exclusion. The threshold checks (in particular the line thresholds) still apply and make the build fail.

          The classes we exclude are generated into generated-sources by a third party (immutables.org) which we do not wish to test.

          Show
          zteve Steve Powell added a comment - This is an issue for us also. In a maven build with the jacoco plugin we exclude all the classes in a package; and jacoco reports in the console log that these have been excluded. In the jacoco check phase, no further inclusions/exclusions are supplied but thresholds are provided. The coverage report given by Jenkins seems to include those classes, the coverage percentages (package, line and so on) remaining the same as without exclusion. The threshold checks (in particular the line thresholds) still apply and make the build fail. The classes we exclude are generated into generated-sources by a third party (immutables.org) which we do not wish to test.
          Hide
          antoinetran Antoine Tran added a comment -

          We have this issue too. Right now, we need to duplicate exclusion information from pom.xml into this Jenkins Jacoco plugin. This issue is not solved yet.

          Show
          antoinetran Antoine Tran added a comment - We have this issue too. Right now, we need to duplicate exclusion information from pom.xml into this Jenkins Jacoco plugin. This issue is not solved yet.
          Hide
          twolfart twolfart added a comment -

          Confirmed on Jenkins 2.64, hudson.plugins.jacoco.JacocoPublisher 2.2.0.

          The code is generated to the target directory and further used in the build.
          The pom.xml contains this configuration:

                <plugin>
                  <groupId>org.jacoco</groupId>
                  <artifactId>jacoco-maven-plugin</artifactId>
                  <configuration>
                    <excludes>
                      <!-- exclude generated code from coverage report -->
                      <exclude>**/com/foo/bar/*</exclude>
                    </excludes>
                  </configuration>
                </plugin>
          

          The maven-generated JaCoCo HTML report doesn't include the classes, Jenkins includes them. Adding the ignores to the jenkins-plugin removes them as expected, but is not very convenient (two places required for configuration).

          Show
          twolfart twolfart added a comment - Confirmed on Jenkins 2.64, hudson.plugins.jacoco.JacocoPublisher 2.2.0. The code is generated to the target directory and further used in the build. The pom.xml contains this configuration: <plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <configuration> <excludes> <!-- exclude generated code from coverage report --> <exclude>**/com/foo/bar/*</exclude> </excludes> </configuration> </plugin> The maven-generated JaCoCo HTML report doesn't include the classes, Jenkins includes them. Adding the ignores to the jenkins-plugin removes them as expected, but is not very convenient (two places required for configuration).
          Hide
          jgangemi Jae Gangemi added a comment -

          is there any progress on resolving this? i still see this behavior w/ jenkins 2.70 and version 2.2.1 of the plugin.

          Show
          jgangemi Jae Gangemi added a comment - is there any progress on resolving this? i still see this behavior w/ jenkins 2.70 and version 2.2.1 of the plugin.
          Hide
          renescheibe René Scheibe added a comment -

          A workaround is to configure the exclusions in the Jenkins Jacoco-Plugin too.

          Show
          renescheibe René Scheibe added a comment - A workaround is to configure the exclusions in the Jenkins Jacoco-Plugin too.
          Hide
          jgangemi Jae Gangemi added a comment -

          is there any futher movement on this issue? configuring exclusions in maven and jenkins is not a viable work around.

          Show
          jgangemi Jae Gangemi added a comment - is there any futher movement on this issue? configuring exclusions in maven and jenkins is not a viable work around.
          Hide
          ataylor Alex Taylor added a comment -

          I was looking into this issue and it appears that the Jacoco plugin fixed this issue for ENV variables with https://github.com/jenkinsci/jacoco-plugin/pull/74 but it does not actually parse the pom.xml for the `excludes`(or `includes` for that matter) configuration. Does that sound like the actual issue here rather than the Excludes not working at all?

           

          Did I miss anything here? I was going to work on getting this fixed with a PR hopefully within the next week

          Show
          ataylor Alex Taylor added a comment - I was looking into this issue and it appears that the Jacoco plugin fixed this issue for ENV variables with https://github.com/jenkinsci/jacoco-plugin/pull/74  but it does not actually parse the pom.xml for the `excludes`(or `includes` for that matter) configuration. Does that sound like the actual issue here rather than the Excludes not working at all?   Did I miss anything here? I was going to work on getting this fixed with a PR hopefully within the next week
          Hide
          jgangemi Jae Gangemi added a comment -

          i'm confused as to why this information just isn't part of the generated results file in the first place as a block that indicates what to exclude. perhaps even better would just be excluding the classes from the result file outright. disclaimer: i have not looked into how the maven plugin works when i choose to generate a report, but it does do the proper exclusions.

          if the plugin needs to parse the pom to figure this out, then that is the fix but that doesn't really seem like the correct solution here b/c what if i'm a nodejs project that is able to generate coverage reports in jacoco format? there's not going to be any pom to parse for exclusions in that case.

          Show
          jgangemi Jae Gangemi added a comment - i'm confused as to why this information just isn't part of the generated results file in the first place as a block that indicates what to exclude. perhaps even better would just be excluding the classes from the result file outright. disclaimer: i have not looked into how the maven plugin works when i choose to generate a report, but it does do the proper exclusions. if the plugin needs to parse the pom to figure this out, then that is the fix but that doesn't really seem like the correct solution here b/c what if i'm a nodejs project that is able to generate coverage reports in jacoco format? there's not going to be any pom to parse for exclusions in that case.
          Hide
          gstanek Gabriel Stanek added a comment -

          Alex Taylor , what you've described, regarding the exclude block in the maven plugin not taking affect in Jenkins, is the issue I've seen and am interested in a fix for, and seems to align with the initial description (which is specific to maven based projects).

          Show
          gstanek Gabriel Stanek added a comment - Alex Taylor , what you've described, regarding the exclude block in the maven plugin not taking affect in Jenkins, is the issue I've seen and am interested in a fix for, and seems to align with the initial description (which is specific to maven based projects).
          Hide
          ataylor Alex Taylor added a comment -

          Thanks for the update Gabriel Stanek just wanted make sure I was understanding everyone's issue and how best I can fix it

          Show
          ataylor Alex Taylor added a comment - Thanks for the update Gabriel Stanek just wanted make sure I was understanding everyone's issue and how best I can fix it
          Hide
          alan_czajkowski Alan Czajkowski added a comment -

          really looking forward to this fix! lots of anxiety around this bug

          Show
          alan_czajkowski Alan Czajkowski added a comment - really looking forward to this fix! lots of anxiety around this bug
          Hide
          rajiv_jandial Rajiv Jandial added a comment -

          I am still seeing this as an open issue. Even I exclude the package from the pom.xml as well as from the Jenkins "exclusions" fields, those packages are still being included in the JaCoCo report.

          Show
          rajiv_jandial Rajiv Jandial added a comment - I am still seeing this as an open issue. Even I exclude the package from the pom.xml as well as from the Jenkins "exclusions" fields, those packages are still being included in the JaCoCo report.
          Hide
          pablogrisafi1975 Pablo Grisafi added a comment -

          my workaround: in the pom.xml I save all patterns  as properties:

          <coverage-exclusion-01>*/com/company/product//config//.</coverage-exclusion-01><coverage-exclusion-02>/com/company/cfa//CallDTO.</coverage-exclusion-02><coverage-exclusion-03>*/com/company/cfa/*/DAO.</coverage-exclusion-03>

          And I use them to configure the maven jacoco plugin

          <configuration> <excludes> <exclude>${coverage-exclusion-01}</exclude> <exclude>${coverage-exclusion-02}</exclude>

          etc...

           

          And then, in the pipeline file y read again the properties

           

          script{      props = readMavenPom().getProperties()     exclusionPattern = props.entrySet().findAll\{entry -> entry.key.startsWith('coverage-exclusion-')}.collect{it.value}.join(',')
              echo "exclusionPattern = ${exclusionPattern}"
          }
          jacoco( execPattern: '**/target/jacoco.exec', exclusionPattern: exclusionPattern)

           

          It works, but is ugly

           

           

          Show
          pablogrisafi1975 Pablo Grisafi added a comment - my workaround: in the pom.xml I save all patterns  as properties: < coverage-exclusion-01 > * /com/company/product/ /config/ / . </ coverage-exclusion-01 > < coverage-exclusion-02 > /com/company/cfa/ /CallDTO. </ coverage-exclusion-02 > < coverage-exclusion-03 > * /com/company/cfa/ */ DAO. </ coverage-exclusion-03 > And I use them to configure the maven jacoco plugin < configuration > < excludes > < exclude > ${coverage-exclusion-01} </ exclude > < exclude > ${coverage-exclusion-02} </ exclude > etc...   And then, in the pipeline file y read again the properties   script{      props = readMavenPom().getProperties()     exclusionPattern = props.entrySet().findAll\{entry -> entry.key.startsWith('coverage-exclusion-')}.collect{it.value}.join(',')     echo "exclusionPattern = ${exclusionPattern}" } jacoco( execPattern: '**/target/jacoco.exec', exclusionPattern: exclusionPattern)   It works, but is ugly    
          Hide
          arielma Ariel M added a comment -

          We are also facing the same issue. it's weird coz' it's working for one MS, but not for the other. is there any update for it?

          Show
          arielma Ariel M added a comment - We are also facing the same issue. it's weird coz' it's working for one MS, but not for the other. is there any update for it?

            People

            • Assignee:
              ognjenb Ognjen Bubalo
              Reporter:
              daknin Darrell King
            • Votes:
              20 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

              • Created:
                Updated: