Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46403

Infinite loop on pipeline start

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      After creation of a new job with jobDSL the created pipeline freezes just before starts to run.
      Debugging I noticed the thread is locked at Utils.groovy#L869
      The attached stack trace shows a possible conflict with groovy MetaClass getProperty, causing the mentioned loop.
      As seen in attached file "jobDsl", the "logRotator" command causes the bug, but I think it's a pipeline-model-definition bug.

        Attachments

        1. another-config.xml
          2 kB
        2. another-thread-stack.txt
          7 kB
        3. config.xml
          2 kB
        4. jobDsl
          0.4 kB
        5. stack
          13 kB

          Activity

          Hide
          abayer Andrew Bayer added a comment -

          Can you attach the resulting Pipeline script?

          Show
          abayer Andrew Bayer added a comment - Can you attach the resulting Pipeline script?
          Hide
          abayer Andrew Bayer added a comment -

          Yeah, I really need the Jenkinsfile - somehow it's thinking the WorkflowJob instance is itself a POJO, rather than WorkflowJob, or else it wouldn't end up in that metaclass mess...

          Show
          abayer Andrew Bayer added a comment - Yeah, I really need the Jenkinsfile - somehow it's thinking the WorkflowJob instance is itself a POJO, rather than WorkflowJob , or else it wouldn't end up in that metaclass mess...
          Hide
          bkmeneguello Bruno Meneguello added a comment -

          Andrew Bayer sorry the delay. This is the generate failed build config

          Show
          bkmeneguello Bruno Meneguello added a comment - Andrew Bayer sorry the delay. This is the generate failed build config
          Hide
          tom_gl Thomas de Grenier de Latour added a comment -

          I've been hit by this bug too, with Jenkins 2.60.3 and pipeline-model-definition 1.2.2, in a similar situation: a JobDSL-generated WorkflowJob, with a logRotator defined outside of the properties list.  I will attach a config.xml too, and a stacktrace during the live lock (it's an infinite loop, with 100% cpu usage on master).  The stack changed a bit because of different core and plugin versions, but I'm pretty sure the bug is the same.

           

          Looking at the code, I think this could be the explanation:

           - here, we have a loop to remove all properties of a given type

          https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/pipeline-model-definition-1.2.2/pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy#L550

          => "as long as there is some property of this type, remove a property of this type"

           - here the implementation of Job.getProperty(Class)

          https://github.com/jenkinsci/jenkins/blob/jenkins-2.60.3/core/src/main/java/hudson/model/Job.java#L567

          => Note that it can return something which is not actually from the job properties list.

           - here the implementation of Job.removeProperty(Class)

          https://github.com/jenkinsci/jenkins/blob/jenkins-2.60.3/core/src/main/java/hudson/model/Job.java#L532

          => Note that it won't remove anything which was not in the properties list.

          The fundamental issue here is that get and remove methods are not coherent, meaning that the assumption that "when you can get something you can also remove it" is not true.  Maybe that's something that should be fixed in core, by adding a special case in removeProperty, similar to the one in getProperty.

          Now, I think it's also something you could workaround in pipeline-model-definition, by not looping when removeProperty returns null. Actually, I'm not sure you need to call getProperty at all:

          while (j.removeProperty(p.class) != null) {}

           

          Show
          tom_gl Thomas de Grenier de Latour added a comment - I've been hit by this bug too, with Jenkins 2.60.3 and pipeline-model-definition 1.2.2, in a similar situation: a JobDSL-generated WorkflowJob, with a logRotator defined outside of the properties list.  I will attach a config.xml too, and a stacktrace during the live lock (it's an infinite loop, with 100% cpu usage on master).  The stack changed a bit because of different core and plugin versions, but I'm pretty sure the bug is the same.   Looking at the code, I think this could be the explanation:  - here, we have a loop to remove all properties of a given type https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/pipeline-model-definition-1.2.2/pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy#L550 => "as long as there is some property of this type, remove a property of this type"  - here the implementation of Job.getProperty(Class) https://github.com/jenkinsci/jenkins/blob/jenkins-2.60.3/core/src/main/java/hudson/model/Job.java#L567 => Note that it can return something which is not actually from the job properties list.  - here the implementation of Job.removeProperty(Class) https://github.com/jenkinsci/jenkins/blob/jenkins-2.60.3/core/src/main/java/hudson/model/Job.java#L532 => Note that it won't remove anything which was not in the properties list. The fundamental issue here is that get and remove methods are not coherent, meaning that the assumption that "when you can get something you can also remove it" is not true.  Maybe that's something that should be fixed in core, by adding a special case in removeProperty , similar to the one in getProperty . Now, I think it's also something you could workaround in pipeline-model-definition , by not looping when removeProperty returns null . Actually, I'm not sure you need to call getProperty at all: while (j.removeProperty(p.class) != null) { }  
          Hide
          tom_gl Thomas de Grenier de Latour added a comment -

          Added link to a PR, with a fix and a test case.

          Show
          tom_gl Thomas de Grenier de Latour added a comment - Added link to a PR, with a fix and a test case.
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Thomas de Grenier de Latour
          Path:
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy
          pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTest.java
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/b475449098a9f7636c951882040b5e8209eda617
          Log:
          JENKINS-46403 - fix infinite loop running jobs with a logRotator

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Thomas de Grenier de Latour Path: pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTest.java http://jenkins-ci.org/commit/pipeline-model-definition-plugin/b475449098a9f7636c951882040b5e8209eda617 Log: JENKINS-46403 - fix infinite loop running jobs with a logRotator
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy
          pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTest.java
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/6c07792ba4b62f654adf0204e83b6b23711e2342
          Log:
          Merge pull request #206 from thomasgl-orange/JENKINS-46403

          JENKINS-46403 - fix infinite loop running jobs with a logRotator

          Compare: https://github.com/jenkinsci/pipeline-model-definition-plugin/compare/47f97fb32373...6c07792ba4b6

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTest.java http://jenkins-ci.org/commit/pipeline-model-definition-plugin/6c07792ba4b62f654adf0204e83b6b23711e2342 Log: Merge pull request #206 from thomasgl-orange/ JENKINS-46403 JENKINS-46403 - fix infinite loop running jobs with a logRotator Compare: https://github.com/jenkinsci/pipeline-model-definition-plugin/compare/47f97fb32373...6c07792ba4b6
          Hide
          abayer Andrew Bayer added a comment -

          Thanks for the fix, Thomas de Grenier de Latour!

          Show
          abayer Andrew Bayer added a comment - Thanks for the fix, Thomas de Grenier de Latour !

            People

            • Assignee:
              tom_gl Thomas de Grenier de Latour
              Reporter:
              bkmeneguello Bruno Meneguello
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: