Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43199

Credentials Binding plugin causes File Descriptor leak

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Our deployment of Jenkins with this plugin is currently on an NFS share. In testing a job with this plugin, a file descriptor was apparently left open. We found this out when attempting to delete the test job after it ran. (It was a simple echo job to make sure masking worked as expected, nothing else)

      The issue appears to be related to the anonymous sub-class of the LineTransformationOutputStream class returned by the Wrapper class and similar to the bug resolved in JENKINS-28409 for EnvInject. The sub-class needs to clean up it's parent outputstream and the super class on close.

       

      @Override
       public void close() throws IOException {
       super.close();
       out.close();
       }

        Attachments

          Issue Links

            Activity

            Hide
            ebutler Elliott Butler added a comment -

            This may potentially cause the same issue for the pipeline variant, but is not used by us at this time.

            Show
            ebutler Elliott Butler added a comment - This may potentially cause the same issue for the pipeline variant, but is not used by us at this time.
            Hide
            bagee Bill Agee added a comment - - edited

            I'm also seeing this - the versions I'm running are:

            • Jenkins 2.54
            • Credentials Plugin version 2.1.13
            • Credentials Binding Plugin version 1.11

            This issue is causing a pretty serious problem on my Jenkins master, since I have several frequently-running jobs which use the credentials plugin. After a few days those jobs cause Jenkins to have several thousand open FDs to build logs:

            $ sudo sysdig -c lsof "'proc.pid=3531 and thread.tid=3531'" | tail -n 1
            java 3531 3531 jenkins 3204 file /var/lib/jenkins/jobs/myjob/builds/886/log

            My pipeline jobs that use the credentials plugin also appear to be affected by this issue.

            For the moment I've downgraded to Jenkins 2.46.1 and the Credentials Binding Plugin version 1.8, which made the problem go away. It would be nice to get back onto newer releases, though.

            Show
            bagee Bill Agee added a comment - - edited I'm also seeing this - the versions I'm running are: Jenkins 2.54 Credentials Plugin version 2.1.13 Credentials Binding Plugin version 1.11 This issue is causing a pretty serious problem on my Jenkins master, since I have several frequently-running jobs which use the credentials plugin. After a few days those jobs cause Jenkins to have several thousand open FDs to build logs: $ sudo sysdig -c lsof " 'proc.pid=3531 and thread.tid=3531' " | tail -n 1 java 3531 3531 jenkins 3204 file / var /lib/jenkins/jobs/myjob/builds/886/log My pipeline jobs that use the credentials plugin also appear to be affected by this issue. For the moment I've downgraded to Jenkins 2.46.1 and the Credentials Binding Plugin version 1.8, which made the problem go away. It would be nice to get back onto newer releases, though.
            Hide
            shoconinja Ben Waters added a comment -

            Hi!

             

            Any update? We're using Jenkins 2.53 in our production environments and after update noticed the change. Would really appreciate it.

            Show
            shoconinja Ben Waters added a comment - Hi!   Any update? We're using Jenkins 2.53 in our production environments and after update noticed the change. Would really appreciate it.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Somebody was playing with statuses, so nobody picked the "In Review" ticket. Reverted the status to Open

            Show
            oleg_nenashev Oleg Nenashev added a comment - Somebody was playing with statuses, so nobody picked the "In Review" ticket. Reverted the status to Open
            Hide
            jglick Jesse Glick added a comment -

            From inspecting code, I see no way this would affect Pipeline builds. Freestyle builds yes, due in part to what I think is a core bug in Run.execute: listener.closeQuietly() should instead (or additionally) be logger.close().

            Show
            jglick Jesse Glick added a comment - From inspecting code, I see no way this would affect Pipeline builds. Freestyle builds yes, due in part to what I think is a core bug in Run.execute : listener.closeQuietly() should instead (or additionally) be logger.close() .
            Hide
            jglick Jesse Glick added a comment -

            Cannot find JIRA ID for mssbg to assign to.

            Show
            jglick Jesse Glick added a comment - Cannot find JIRA ID for mssbg to assign to.
            Hide
            abhishekmukherg Abhishek Mukherjee added a comment -

            For what it's worth, this bug seems to have made it into the LTS release of jenkins (2.60.1) and is incredibly disruptive. We're having to currently restart our jenkins nightly-ish as Ben is doing. I just noticed that we'd upgraded our credentials binding plugin to 1.12, so I'm going to downgrade that to 1.11, which was packaged with jenkins, with tonight's bounce and see if that helps any.

            Show
            abhishekmukherg Abhishek Mukherjee added a comment - For what it's worth, this bug seems to have made it into the LTS release of jenkins (2.60.1) and is incredibly disruptive. We're having to currently restart our jenkins nightly-ish as Ben is doing. I just noticed that we'd upgraded our credentials binding plugin to 1.12, so I'm going to downgrade that to 1.11, which was packaged with jenkins, with tonight's bounce and see if that helps any.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Jenkins LTS does not bundle plugins. Once the plugin fix is released, it will become available in previous LTS releases

            Show
            oleg_nenashev Oleg Nenashev added a comment - Jenkins LTS does not bundle plugins. Once the plugin fix is released, it will become available in previous LTS releases
            Hide
            abhishekmukherg Abhishek Mukherjee added a comment -

            Ah apologies – uninstall is greyed out and I can only downgrade in the UI so i figured that meant that it came with the jenkins LTS. Fully understand that a plugin can be updated independently (as we've already done to 1.12)

            Show
            abhishekmukherg Abhishek Mukherjee added a comment - Ah apologies – uninstall is greyed out and I can only downgrade in the UI so i figured that meant that it came with the jenkins LTS. Fully understand that a plugin can be updated independently (as we've already done to 1.12)
            Hide
            jglick Jesse Glick added a comment -

            Depending on the location of the fix; I believe this is actually a core bug, in which case the fix will come in a core update.

            Show
            jglick Jesse Glick added a comment - Depending on the location of the fix; I believe this is actually a core bug, in which case the fix will come in a core update.
            Hide
            abhishekmukherg Abhishek Mukherjee added a comment - - edited

            Would that be the linked -JENKINS-28409-, by any chance? Thanks!

            EDIT: nvm. just noticed the "Component" of that ticket being a plugin as well. please ignore me

            Show
            abhishekmukherg Abhishek Mukherjee added a comment - - edited Would that be the linked - JENKINS-28409 -, by any chance? Thanks! EDIT: nvm. just noticed the "Component" of that ticket being a plugin as well. please ignore me
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/hudson/model/Run.java
            http://jenkins-ci.org/commit/jenkins/a0a55d1da6882e8a56d90885e7cd55a22accbfe1
            Log:
            JENKINS-43199 Reliably close build log file (#2954)

            • Try harder to ensure that the logger is closed at the end of the build.
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/hudson/model/Run.java http://jenkins-ci.org/commit/jenkins/a0a55d1da6882e8a56d90885e7cd55a22accbfe1 Log: JENKINS-43199 Reliably close build log file (#2954) JENKINS-43199 Reliably close build log file. Try harder to ensure that the logger is closed at the end of the build.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            It should be solved by https://github.com/jenkinsci/jenkins/pull/2954 in 2.73

            Show
            oleg_nenashev Oleg Nenashev added a comment - It should be solved by https://github.com/jenkinsci/jenkins/pull/2954 in 2.73
            Hide
            jglick Jesse Glick added a comment -

            Is there an easy way to reproduce the bug? I did not try to write an automated test for it, so I only believe the fix is effective.

            Show
            jglick Jesse Glick added a comment - Is there an easy way to reproduce the bug? I did not try to write an automated test for it, so I only believe the fix is effective.
            Hide
            jglick Jesse Glick added a comment -

            Technically the lack of a close override in this plugin is not addressed—only the symptom, when using the core patch. Reopening for tracking purposes.

            Show
            jglick Jesse Glick added a comment - Technically the lack of a close override in this plugin is not addressed—only the symptom, when using the core patch. Reopening for tracking purposes.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Yes, it would be great to get the plugin fixed for older cores

            Show
            oleg_nenashev Oleg Nenashev added a comment - Yes, it would be great to get the plugin fixed for older cores
            Hide
            billagee Bill Agee added a comment -

            In my environment, the repro steps on Ubuntu 14.04 were:

            • Create a 'Secret text' credential with Global scope, in /credentials/store/system/domain/_/
            • Create a freestyle job
            • Enable and select the secret text value in the job UI, with:
              • Use secret text(s) or file(s) -> Bindings -> Secret text -> Variable set to MY_SECRET
            • Add a step in Build -> Execute shell and put a simple 'exit 0' statement or something similar (this step may not be necessary ... not sure)
            • Run the Jenkins job several times, while using sysdig to examine the pid and thread ID for the process handling the job, and see if you see open file handles related to the job begin stacking up as you repeatedly trigger builds:
            • sudo sysdig -c lsof "'proc.pid=3531 and thread.tid=3531'" | tail -n 1
              java 3531 3531 jenkins 3204 file /var/lib/jenkins/jobs/myjob/builds/886/log

            Show
            billagee Bill Agee added a comment - In my environment, the repro steps on Ubuntu 14.04 were: Create a 'Secret text' credential with Global scope, in /credentials/store/system/domain/_/ Create a freestyle job Enable and select the secret text value in the job UI, with: Use secret text(s) or file(s) -> Bindings -> Secret text -> Variable set to MY_SECRET Add a step in Build -> Execute shell and put a simple 'exit 0' statement or something similar (this step may not be necessary ... not sure) Run the Jenkins job several times, while using sysdig to examine the pid and thread ID for the process handling the job, and see if you see open file handles related to the job begin stacking up as you repeatedly trigger builds: sudo sysdig -c lsof "'proc.pid=3531 and thread.tid=3531'" | tail -n 1 java 3531 3531 jenkins 3204 file /var/lib/jenkins/jobs/myjob/builds/886/log

              People

              • Assignee:
                Unassigned
                Reporter:
                ebutler Elliott Butler
              • Votes:
                6 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: