Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42189

Groovy 2.4.8 interoperability issues

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      The issue depicted in blocked_threads.png appeared in our production server.
      (it results on a broken state that we fixed by rebooting the server)

      The result of the first investigation is the following:
      cleanUpGlobalClassSet
      https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1037

      seems to have compatibility issues with the memory leak fix implemented in groovy 2.4.8
      https://github.com/apache/groovy/pull/219/commits/f7c688e101af77170448cf0be9103c4e6dac3f74

      It seems that gradle had a similar issue, that was fixed by the following commit in groovy 2.4.8
      https://github.com/apache/groovy/commit/82c9d20a160c7e4f59630f163a68df82f5e0eba4

        Attachments

          Issue Links

            Activity

            yetanotherion Jon Alberdi created issue -
            yetanotherion Jon Alberdi made changes -
            Field Original Value New Value
            Description The issue depicted in blocked_trheads.png appeared in our production server.
            (it results on a broken state that we fixed by rebooting the server only)

            The result of the first investigation is the following:
            cleanUpGlobalClassSet
            https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1037

            seems to have compatibility issues with the memory-leak fix implemented in groovy 2.4.8
            https://github.com/apache/groovy/pull/219/commits/f7c688e101af77170448cf0be9103c4e6dac3f74

            It seems that gradle had a similar issue, that was fixed by the following commit in groovy 2.4.8
            https://github.com/apache/groovy/commit/82c9d20a160c7e4f59630f163a68df82f5e0eba4

            The issue depicted in blocked_trheads.png appeared in our production server.
            (it results on a broken state that we fixed by rebooting the server)

            The result of the first investigation is the following:
            cleanUpGlobalClassSet
            https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1037

            seems to have compatibility issues with the memory-leak fix implemented in groovy 2.4.8
            https://github.com/apache/groovy/pull/219/commits/f7c688e101af77170448cf0be9103c4e6dac3f74

            It seems that gradle had a similar issue, that was fixed by the following commit in groovy 2.4.8
            https://github.com/apache/groovy/commit/82c9d20a160c7e4f59630f163a68df82f5e0eba4

            yetanotherion Jon Alberdi made changes -
            Description The issue depicted in blocked_trheads.png appeared in our production server.
            (it results on a broken state that we fixed by rebooting the server)

            The result of the first investigation is the following:
            cleanUpGlobalClassSet
            https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1037

            seems to have compatibility issues with the memory-leak fix implemented in groovy 2.4.8
            https://github.com/apache/groovy/pull/219/commits/f7c688e101af77170448cf0be9103c4e6dac3f74

            It seems that gradle had a similar issue, that was fixed by the following commit in groovy 2.4.8
            https://github.com/apache/groovy/commit/82c9d20a160c7e4f59630f163a68df82f5e0eba4

            The issue depicted in blocked_threads.png appeared in our production server.
            (it results on a broken state that we fixed by rebooting the server)

            The result of the first investigation is the following:
            cleanUpGlobalClassSet
            https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1037

            seems to have compatibility issues with the memory-leak fix implemented in groovy 2.4.8
            https://github.com/apache/groovy/pull/219/commits/f7c688e101af77170448cf0be9103c4e6dac3f74

            It seems that gradle had a similar issue, that was fixed by the following commit in groovy 2.4.8
            https://github.com/apache/groovy/commit/82c9d20a160c7e4f59630f163a68df82f5e0eba4

            yetanotherion Jon Alberdi made changes -
            Description The issue depicted in blocked_threads.png appeared in our production server.
            (it results on a broken state that we fixed by rebooting the server)

            The result of the first investigation is the following:
            cleanUpGlobalClassSet
            https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1037

            seems to have compatibility issues with the memory-leak fix implemented in groovy 2.4.8
            https://github.com/apache/groovy/pull/219/commits/f7c688e101af77170448cf0be9103c4e6dac3f74

            It seems that gradle had a similar issue, that was fixed by the following commit in groovy 2.4.8
            https://github.com/apache/groovy/commit/82c9d20a160c7e4f59630f163a68df82f5e0eba4

            The issue depicted in blocked_threads.png appeared in our production server.
            (it results on a broken state that we fixed by rebooting the server)

            The result of the first investigation is the following:
            cleanUpGlobalClassSet
            https://github.com/jenkinsci/workflow-cps-plugin/blob/master/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java#L1037

            seems to have compatibility issues with the memory leak fix implemented in groovy 2.4.8
            https://github.com/apache/groovy/pull/219/commits/f7c688e101af77170448cf0be9103c4e6dac3f74

            It seems that gradle had a similar issue, that was fixed by the following commit in groovy 2.4.8
            https://github.com/apache/groovy/commit/82c9d20a160c7e4f59630f163a68df82f5e0eba4

            oleg_nenashev Oleg Nenashev made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Added the issue reference to the community rating on https://jenkins.io/changelog/

            Show
            oleg_nenashev Oleg Nenashev added a comment - Added the issue reference to the community rating on https://jenkins.io/changelog/
            oleg_nenashev Oleg Nenashev made changes -
            Labels pipeline
            Hide
            oleg_nenashev Oleg Nenashev added a comment - - edited

            Jon Alberdi Is there a GROOVY-x issue reported for it? We can probably fix it on the Pipeline side (and retain compatibility with prev. version ?), but I suppose it requires a fix on the Groovy side

            Show
            oleg_nenashev Oleg Nenashev added a comment - - edited Jon Alberdi Is there a GROOVY-x issue reported for it? We can probably fix it on the Pipeline side (and retain compatibility with prev. version ?), but I suppose it requires a fix on the Groovy side
            Show
            yetanotherion Jon Alberdi added a comment - Oleg Nenashev https://issues.apache.org/jira/browse/GROOVY-8092
            Hide
            yetanotherion Jon Alberdi added a comment -

            Oleg Nenashev FYI an answer was posted in https://issues.apache.org/jira/browse/GROOVY-8092,

            Two solutions are proposed:

            • A possible workaround for 2.4.8 may be to use -Dgroovy.use.classvalue.
            • upgrade to groovy 2.4.9
            Show
            yetanotherion Jon Alberdi added a comment - Oleg Nenashev FYI an answer was posted in https://issues.apache.org/jira/browse/GROOVY-8092 , Two solutions are proposed: A possible workaround for 2.4.8 may be to use -Dgroovy.use.classvalue. upgrade to groovy 2.4.9
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Both options do not seem to be feasible.

            • 2.4.9 has no ETA
            • Workaround cannot be enforced in Jenkins without really big chunk of testing

            Maybe the safest approach is to just revert the fix and wait for 4.2.9. CC Daniel Spilker

            Show
            oleg_nenashev Oleg Nenashev added a comment - Both options do not seem to be feasible. 2.4.9 has no ETA Workaround cannot be enforced in Jenkins without really big chunk of testing Maybe the safest approach is to just revert the fix and wait for 4.2.9. CC Daniel Spilker
            Hide
            daspilker Daniel Spilker added a comment -

            Oleg Nenashev I think reverting the change is the best option.

            Show
            daspilker Daniel Spilker added a comment - Oleg Nenashev I think reverting the change is the best option.
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Link This issue blocks JENKINS-33358 [ JENKINS-33358 ]
            Hide
            jglick Jesse Glick added a comment -

            The workaround implemented for Gradle takes effect in Jenkins as well, and CpsFlowExecutionTest.loaderReleased passes (albeit slowly—there is a soft leak here somewhere which did not exist in Jenkins 2.32.2); klazzF.get(classInfo) produces ClassInfo.Sentinel.class.

            The reported problem seems to be something else, a deadlock (perhaps?), but I cannot diagnose it without a complete textual thread dump as a file attachment.

            Show
            jglick Jesse Glick added a comment - The workaround implemented for Gradle takes effect in Jenkins as well, and CpsFlowExecutionTest.loaderReleased passes (albeit slowly—there is a soft leak here somewhere which did not exist in Jenkins 2.32.2); klazzF.get(classInfo) produces ClassInfo.Sentinel.class . The reported problem seems to be something else, a deadlock (perhaps?), but I cannot diagnose it without a complete textual thread dump as a file attachment.
            Hide
            jglick Jesse Glick added a comment - - edited

            The soft leak:

            groovy.lang.GroovyClassLoader$InnerLoader@26f6bf29=private static final org.codehaus.groovy.reflection.GroovyClassValue org.codehaus.groovy.reflection.ClassInfo.globalClassValue->
            org.codehaus.groovy.reflection.GroovyClassValuePreJava7@38b6b4fb-map->
            org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map@1045633-segments->
            [Lorg.codehaus.groovy.util.AbstractConcurrentMapBase$Segment;@6d1e60b7-[3]->
            org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@48faf82d-table->
            [Ljava.lang.Object;@26facae0-[14]->
            org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue@54add792-value->
            org.codehaus.groovy.reflection.ClassInfo@3cc5a7fc-weakMetaClass->
            org.codehaus.groovy.util.ManagedReference@78e07ad1-ref->
            org.codehaus.groovy.util.ReferenceType$SoftRef@69fcac1c-referent->
            groovy.lang.MetaClassImpl@12989a26-theClass->
            java.lang.Class@eb2c5d9-<changed>->
            groovy.lang.GroovyClassLoader$InnerLoader@26f6bf29
            
            Show
            jglick Jesse Glick added a comment - - edited The soft leak: groovy.lang.GroovyClassLoader$InnerLoader@26f6bf29=private static final org.codehaus.groovy.reflection.GroovyClassValue org.codehaus.groovy.reflection.ClassInfo.globalClassValue-> org.codehaus.groovy.reflection.GroovyClassValuePreJava7@38b6b4fb-map-> org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Map@1045633-segments-> [Lorg.codehaus.groovy.util.AbstractConcurrentMapBase$Segment;@6d1e60b7-[3]-> org.codehaus.groovy.reflection.GroovyClassValuePreJava7$GroovyClassValuePreJava7Segment@48faf82d-table-> [Ljava.lang.Object;@26facae0-[14]-> org.codehaus.groovy.reflection.GroovyClassValuePreJava7$EntryWithValue@54add792-value-> org.codehaus.groovy.reflection.ClassInfo@3cc5a7fc-weakMetaClass-> org.codehaus.groovy.util.ManagedReference@78e07ad1-ref-> org.codehaus.groovy.util.ReferenceType$SoftRef@69fcac1c-referent-> groovy.lang.MetaClassImpl@12989a26-theClass-> java.lang.Class@eb2c5d9-<changed>-> groovy.lang.GroovyClassLoader$InnerLoader@26f6bf29
            Hide
            jglick Jesse Glick added a comment -

            Definitely 2.4.9 would be preferable to 2.4.8. For Pipeline, the way things stand, 2.4.7 would be preferable to 2.4.9; I am trying to dig into why the memory leak regressed, but I cannot promise progress.

            Show
            jglick Jesse Glick added a comment - Definitely 2.4.9 would be preferable to 2.4.8. For Pipeline, the way things stand, 2.4.7 would be preferable to 2.4.9; I am trying to dig into why the memory leak regressed, but I cannot promise progress.
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Labels pipeline groovy pipeline regression
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 109 (Web Link)" [ 15484 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            Jenkinsfile
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/1a73b74a907c47e06d103875c89f5c7efc8890ab
            Log:
            [FIXED JENKINS-42189] In Groovy 2.4.8+, ClassInfo.klazz is replaced by .classRef.
            cleanUpGlobalClassValue needs to walk through those (not just Sentinel.class over and over);
            and cleanUpGlobalClassSet is no longer relevant since classRef is weak.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: Jenkinsfile pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java http://jenkins-ci.org/commit/workflow-cps-plugin/1a73b74a907c47e06d103875c89f5c7efc8890ab Log: [FIXED JENKINS-42189] In Groovy 2.4.8+, ClassInfo.klazz is replaced by .classRef. cleanUpGlobalClassValue needs to walk through those (not just Sentinel.class over and over); and cleanUpGlobalClassSet is no longer relevant since classRef is weak.
            jglick Jesse Glick made changes -
            Status In Review [ 10005 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            Hide
            daspilker Daniel Spilker added a comment -

            FYI: the vote to release Groovy 2.4.9 has been started: http://mail-archives.apache.org/mod_mbox/groovy-dev/201702.mbox/browser

            Show
            daspilker Daniel Spilker added a comment - FYI: the vote to release Groovy 2.4.9 has been started: http://mail-archives.apache.org/mod_mbox/groovy-dev/201702.mbox/browser
            Hide
            jglick Jesse Glick added a comment -

            Thanks for tip. As noted in GROOVY-8092 I see no problem with that.

            Show
            jglick Jesse Glick added a comment - Thanks for tip. As noted in GROOVY-8092 I see no problem with that.
            jglick Jesse Glick made changes -
            Remote Link This issue links to "PR 111 (Web Link)" [ 15490 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            content/_data/changelogs/weekly.yml
            content/_partials/changes.html.haml
            http://jenkins-ci.org/commit/jenkins.io/258844d287e599fb9239faaa25499657f987978b
            Log:
            JENKINS-42189 Warn people against using 2.47.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: content/_data/changelogs/weekly.yml content/_partials/changes.html.haml http://jenkins-ci.org/commit/jenkins.io/258844d287e599fb9239faaa25499657f987978b Log: JENKINS-42189 Warn people against using 2.47.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            content/_data/changelogs/weekly.yml
            content/_partials/changes.html.haml
            http://jenkins-ci.org/commit/jenkins.io/d31c43dafad414ecf6b72efed087c7cb48f0317e
            Log:
            Merge pull request #694 from jglick/warning-JENKINS-42189

            JENKINS-42189 Warn people against using 2.47

            Compare: https://github.com/jenkins-infra/jenkins.io/compare/f9848c1d08e2...d31c43dafad4

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: content/_data/changelogs/weekly.yml content/_partials/changes.html.haml http://jenkins-ci.org/commit/jenkins.io/d31c43dafad414ecf6b72efed087c7cb48f0317e Log: Merge pull request #694 from jglick/warning- JENKINS-42189 JENKINS-42189 Warn people against using 2.47 Compare: https://github.com/jenkins-infra/jenkins.io/compare/f9848c1d08e2...d31c43dafad4
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/test/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecutionTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/e6038f2494d55f13e2294f04c720f6fdce28508a
            Log:
            JENKINS-42189 Using https://github.com/jenkinsci/jenkins-test-harness/pull/50 to amend #109 to fail if we get another soft leak.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/test/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecutionTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/e6038f2494d55f13e2294f04c720f6fdce28508a Log: JENKINS-42189 Using https://github.com/jenkinsci/jenkins-test-harness/pull/50 to amend #109 to fail if we get another soft leak.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/test/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecutionTest.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/7e0b27433db5a4bb72fe47b826fb54ca29d5bf7b
            Log:
            Merge pull request #111 from jglick/MemoryAssert

            JENKINS-42189 Forcing a test failure in case of another soft leak

            Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/d59b417e08b8...7e0b27433db5

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/test/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecutionTest.java http://jenkins-ci.org/commit/workflow-cps-plugin/7e0b27433db5a4bb72fe47b826fb54ca29d5bf7b Log: Merge pull request #111 from jglick/MemoryAssert JENKINS-42189 Forcing a test failure in case of another soft leak Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/d59b417e08b8...7e0b27433db5
            Hide
            edilect Emmanuel Debanne added a comment - - edited

            When testing groovy 2.4.8, we needed to restart twice Jenkins. We already provided a screenshot corresponding to the first restart on on the 18th. I add an attachment with the thread dump of the second restart on the 21th, hoping it will help to investigate the deadlock. The job names have been replaced by 'AJOB'.

            Show
            edilect Emmanuel Debanne added a comment - - edited When testing groovy 2.4.8, we needed to restart twice Jenkins. We already provided a screenshot corresponding to the first restart on on the 18th. I add an attachment with the thread dump of the second restart on the 21th, hoping it will help to investigate the deadlock. The job names have been replaced by 'AJOB'.
            edilect Emmanuel Debanne made changes -
            Attachment 20170221_thread_dump.html.htm [ 36208 ]
            Hide
            jglick Jesse Glick added a comment -

            Emmanuel Debanne just please pick up the workflow-cps update and verify that this addresses the issue for you.

            Show
            jglick Jesse Glick added a comment - Emmanuel Debanne just please pick up the workflow-cps update and verify that this addresses the issue for you.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/dc00a0e56c08ee30e74ea8b848bc3ab8e69b7eff
            Log:
            Merge pull request #109 from jglick/groovy-2.4.8-JENKINS-42189

            JENKINS-42189 Fix ClassInfo cleanup in Groovy 2.4.8+

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/CpsFlowExecution.java http://jenkins-ci.org/commit/workflow-cps-plugin/dc00a0e56c08ee30e74ea8b848bc3ab8e69b7eff Log: Merge pull request #109 from jglick/groovy-2.4.8- JENKINS-42189 JENKINS-42189 Fix ClassInfo cleanup in Groovy 2.4.8+
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2033 (Web Link)" [ 18455 ]

              People

              • Assignee:
                jglick Jesse Glick
                Reporter:
                yetanotherion Jon Alberdi
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: