Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37332

java.io.IOException: Failed to rename during Plugin Update

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
      Windows
      Jenkins 2.17
    • Similar Issues:

      Description

      We are currently unable to update plugins in our jenkins installation, because this exception occurs:

      java.io.IOException: Failed to rename C:\Program Files (x86)\Jenkins\plugins\windows-slaves.jpi.tmp to C:\Program Files (x86)\Jenkins\plugins\windows-slaves.jpi
      at hudson.model.UpdateCenter$InstallationJob.replace(UpdateCenter.java:1952)
      at hudson.model.UpdateCenter$UpdateCenterConfiguration.install(UpdateCenter.java:1178)
      at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1653)
      at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1848)
      at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1624)
      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
      at java.lang.Thread.run(Unknown Source)

      It doesn't seem to affect all plugins, but it also happends for the JUnit plugin.

        Attachments

          Issue Links

            Activity

            Hide
            nitram Martin Karing added a comment -

            I can confirm this issue happening for the windows slaves and the junit plug in.

            Show
            nitram Martin Karing added a comment - I can confirm this issue happening for the windows slaves and the junit plug in.
            Hide
            brighton29 Rob Cruz added a comment - - edited

            This is happening to me as well.

            Windows Slaves Plugin
            Failure -
            java.io.IOException: Failed to rename C:\Jenkins\plugins\junit.jpi.tmp to C:\Jenkins\plugins\junit.jpi
            at hudson.model.UpdateCenter$InstallationJob.replace(UpdateCenter.java:1952)
            at hudson.model.UpdateCenter$UpdateCenterConfiguration.install(UpdateCenter.java:1178)
            at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1653)
            at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1848)
            at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1624)
            at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
            at java.lang.Thread.run(Unknown Source)

            JUnit Plugin
            Failure -
            java.io.IOException: Failed to rename C:\Jenkins\plugins\windows-slaves.jpi.tmp to C:\Jenkins\plugins\windows-slaves.jpi
            at hudson.model.UpdateCenter$InstallationJob.replace(UpdateCenter.java:1952)
            at hudson.model.UpdateCenter$UpdateCenterConfiguration.install(UpdateCenter.java:1178)
            at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1653)
            at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1848)
            at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1624)
            at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
            at java.util.concurrent.FutureTask.run(Unknown Source)
            at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
            at java.lang.Thread.run(Unknown Source)

            Show
            brighton29 Rob Cruz added a comment - - edited This is happening to me as well. Windows Slaves Plugin Failure - java.io.IOException: Failed to rename C:\Jenkins\plugins\junit.jpi.tmp to C:\Jenkins\plugins\junit.jpi at hudson.model.UpdateCenter$InstallationJob.replace(UpdateCenter.java:1952) at hudson.model.UpdateCenter$UpdateCenterConfiguration.install(UpdateCenter.java:1178) at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1653) at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1848) at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1624) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110) at java.lang.Thread.run(Unknown Source) JUnit Plugin Failure - java.io.IOException: Failed to rename C:\Jenkins\plugins\windows-slaves.jpi.tmp to C:\Jenkins\plugins\windows-slaves.jpi at hudson.model.UpdateCenter$InstallationJob.replace(UpdateCenter.java:1952) at hudson.model.UpdateCenter$UpdateCenterConfiguration.install(UpdateCenter.java:1178) at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1653) at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1848) at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1624) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110) at java.lang.Thread.run(Unknown Source)
            Hide
            jeffdafoe Jeff Dafoe added a comment -

            On our install, it's happening for the plugins mentioned by others, as well as:
            LDAP Plugin
            Mailer Plugin
            Subversion Plug-in
            External Monitor Job Type Plugin
            Maven Integration plugin
            Javadoc Plugin
            Matrix Authorization Strategy Plugin
            OWASP Markup Formatter Plugin
            Matrix Project Plugin
            Ant Plugin
            PAM Authentication plugin

            Show
            jeffdafoe Jeff Dafoe added a comment - On our install, it's happening for the plugins mentioned by others, as well as: LDAP Plugin Mailer Plugin Subversion Plug-in External Monitor Job Type Plugin Maven Integration plugin Javadoc Plugin Matrix Authorization Strategy Plugin OWASP Markup Formatter Plugin Matrix Project Plugin Ant Plugin PAM Authentication plugin
            Hide
            danielbeck Daniel Beck added a comment -

            All listed plugins are formerly bundled plugins.

            What circumstances do these issues appear in? Are you trying to update these plugins?

            Show
            danielbeck Daniel Beck added a comment - All listed plugins are formerly bundled plugins. What circumstances do these issues appear in? Are you trying to update these plugins?
            Hide
            danielbeck Daniel Beck added a comment -

            Would be interesting to know if Jenkins has a lock on these plugin JPIs specifically, and not others, at the time the upgrade is attempted.

            Show
            danielbeck Daniel Beck added a comment - Would be interesting to know if Jenkins has a lock on these plugin JPIs specifically, and not others, at the time the upgrade is attempted.
            Hide
            danielbeck Daniel Beck added a comment -

            Is this happening only since version 2.17, or did this issue occur earlier than that? What Jenkins versions were you running when you last tried to upgrade any plugin in Jeff's list?

            Show
            danielbeck Daniel Beck added a comment - Is this happening only since version 2.17, or did this issue occur earlier than that? What Jenkins versions were you running when you last tried to upgrade any plugin in Jeff's list?
            Hide
            laurin1 Keith Davis added a comment - - edited

            I update Jenkins (first) and then any plugins every week. On 8/6, I updated to 2.17 and the all plugin updates available - no issues. On 8/11, there was no Jenkins update available and so I tried to update all plugins and these 2 failed. These are the plugins I updated on 8/11 and on 8/6.

            8/11:

            	Email Extension Plugin
            This plugin allows you to configure every aspect of email notifications. You can customize when an email is sent, who should receive it, and what the email says.	2.47	2.46
            	Git Parameter Plug-In
            This plugin allows you to choose between Git tags or sha1 of your SCM repository so Git Plugin installed is required.	0.6.2	0.6.1
            	GitHub API Plugin
            This plugin is a library plugin used by other GitHub related plugins to share the same libraries. This plugin does not have any user visible feature by itself. There's no need to install this plugin manually, although you want to keep it up to date.	1.77	1.76
            	JUnit Plugin
            Allows JUnit-format test results to be published.	1.18	1.15
            	Pipeline: Groovy
            Pipeline execution engine based on continuation passing style transformation of Groovy scripts.	2.11	2.10
            	Pipeline: Input Step
            Adds the Pipeline step ‘input’ to wait for human input or approval.	2.1	2.0
            	Pipeline: Job
            Defines a new job type for pipelines and provides their generic user interface.	2.5	2.4
            	Pipeline: Shared Groovy Libraries
            Global shared library for Pipeline scripts.	2.2	2.1
            	Windows Slaves Plugin
            Allows you to connect to Windows machines and start slave agents on them.	1.2	1.1
            
            

            8/6:

            	Durable Task Plugin
            Library offering an extension point for processes which can run outside of Jenkins yet be monitored.	1.12	1.11
            	Email Extension Plugin
            This plugin allows you to configure every aspect of email notifications. You can customize when an email is sent, who should receive it, and what the email says.	2.46	2.44
            	Git plugin
            This plugin allows use of Git as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official git client. Use exotic installations at your own risk.
            2.5.3	2.5.2
            	GitHub Organization Folder Plugin
            Pipeline-as-Code support for a whole GitHub organization. Scans all the branches & repositories in GitHub organization and build them via Jenkins pipelines automatically
            1.4	1.3
            	Pipeline: Basic Steps
            Commonly used steps for Pipelines.	2.1	2.0
            	Pipeline: Groovy
            Pipeline execution engine based on continuation passing style transformation of Groovy scripts.	2.10	2.9
            	Pipeline: Job
            Defines a new job type for pipelines and provides their generic user interface.	2.4	2.3
            	Pipeline: Nodes and Processes
            Pipeline steps locking agents and workspaces, and running external processes that may survive a Jenkins restart or slave reconnection.	2.4	2.3
            	Pipeline: REST API Plugin
            Provides a REST API to access pipeline and pipeline run data	1.7	1.6
            	Pipeline: Stage View Plugin
            Provides a swimlane view of the different stages in a pipeline.	1.7	1.6
            	Pipeline: Step API
            API for asynchronous build step primitive.	2.3	2.2
            	Structs Plugin
            Library plugin for DSL plugins that need concise names for Jenkins extensions	1.3	1.2
            
            
            
            Show
            laurin1 Keith Davis added a comment - - edited I update Jenkins (first) and then any plugins every week. On 8/6, I updated to 2.17 and the all plugin updates available - no issues. On 8/11, there was no Jenkins update available and so I tried to update all plugins and these 2 failed. These are the plugins I updated on 8/11 and on 8/6. 8/11: Email Extension Plugin This plugin allows you to configure every aspect of email notifications. You can customize when an email is sent, who should receive it, and what the email says. 2.47 2.46 Git Parameter Plug-In This plugin allows you to choose between Git tags or sha1 of your SCM repository so Git Plugin installed is required. 0.6.2 0.6.1 GitHub API Plugin This plugin is a library plugin used by other GitHub related plugins to share the same libraries. This plugin does not have any user visible feature by itself. There's no need to install this plugin manually, although you want to keep it up to date. 1.77 1.76 JUnit Plugin Allows JUnit-format test results to be published. 1.18 1.15 Pipeline: Groovy Pipeline execution engine based on continuation passing style transformation of Groovy scripts. 2.11 2.10 Pipeline: Input Step Adds the Pipeline step ‘input’ to wait for human input or approval. 2.1 2.0 Pipeline: Job Defines a new job type for pipelines and provides their generic user interface. 2.5 2.4 Pipeline: Shared Groovy Libraries Global shared library for Pipeline scripts. 2.2 2.1 Windows Slaves Plugin Allows you to connect to Windows machines and start slave agents on them. 1.2 1.1 8/6: Durable Task Plugin Library offering an extension point for processes which can run outside of Jenkins yet be monitored. 1.12 1.11 Email Extension Plugin This plugin allows you to configure every aspect of email notifications. You can customize when an email is sent, who should receive it, and what the email says. 2.46 2.44 Git plugin This plugin allows use of Git as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official git client. Use exotic installations at your own risk. 2.5.3 2.5.2 GitHub Organization Folder Plugin Pipeline-as-Code support for a whole GitHub organization. Scans all the branches & repositories in GitHub organization and build them via Jenkins pipelines automatically 1.4 1.3 Pipeline: Basic Steps Commonly used steps for Pipelines. 2.1 2.0 Pipeline: Groovy Pipeline execution engine based on continuation passing style transformation of Groovy scripts. 2.10 2.9 Pipeline: Job Defines a new job type for pipelines and provides their generic user interface. 2.4 2.3 Pipeline: Nodes and Processes Pipeline steps locking agents and workspaces, and running external processes that may survive a Jenkins restart or slave reconnection. 2.4 2.3 Pipeline: REST API Plugin Provides a REST API to access pipeline and pipeline run data 1.7 1.6 Pipeline: Stage View Plugin Provides a swimlane view of the different stages in a pipeline. 1.7 1.6 Pipeline: Step API API for asynchronous build step primitive. 2.3 2.2 Structs Plugin Library plugin for DSL plugins that need concise names for Jenkins extensions 1.3 1.2
            Hide
            jeffdafoe Jeff Dafoe added a comment -

            I updated from 1.655 to 2.16 when the issue first surfaced . The issue happens when updating plugins from the web UI.

            In Process Explorer, there are in-use JPI files matching exactly the list of plugins that are having issues while updating. The plugins that do not have a problem updating do not show their JPI files as in-use (if this information is of any use, I don't know).

            Show
            jeffdafoe Jeff Dafoe added a comment - I updated from 1.655 to 2.16 when the issue first surfaced . The issue happens when updating plugins from the web UI. In Process Explorer, there are in-use JPI files matching exactly the list of plugins that are having issues while updating. The plugins that do not have a problem updating do not show their JPI files as in-use (if this information is of any use, I don't know).
            Hide
            nitek Claus Näveke added a comment -

            The files seem to be locked by Jenkins. I tried moving windows-slaves.jpi away, but it did not work (because it was in use) until I shut down Jenkins.
            Looks like Jenkins is locking the file it is trying to update/overwrite

            Show
            nitek Claus Näveke added a comment - The files seem to be locked by Jenkins. I tried moving windows-slaves.jpi away, but it did not work (because it was in use) until I shut down Jenkins. Looks like Jenkins is locking the file it is trying to update/overwrite
            Hide
            danielbeck Daniel Beck added a comment -

            Jeff Dafoe Claus Näveke To clarify, the plugins locked by Jenkins, if they exist on your instance, are exactly the following list:
            https://github.com/jenkinsci/jenkins/blob/b81041c292cc1d2849a95d22381c84e0e1ff26af/core/src/main/java/hudson/ClassicPluginStrategy.java#L413...L427

            If (and only if) a plugin is on that list, and exists on your instance, its JPI file is locked?

            Show
            danielbeck Daniel Beck added a comment - Jeff Dafoe Claus Näveke To clarify, the plugins locked by Jenkins, if they exist on your instance, are exactly the following list: https://github.com/jenkinsci/jenkins/blob/b81041c292cc1d2849a95d22381c84e0e1ff26af/core/src/main/java/hudson/ClassicPluginStrategy.java#L413...L427 If (and only if) a plugin is on that list, and exists on your instance, its JPI file is locked?
            Hide
            nitek Claus Näveke added a comment -

            Happend to me with windows-slave and junit, so yes, only plugins from that list.

            Show
            nitek Claus Näveke added a comment - Happend to me with windows-slave and junit, so yes, only plugins from that list.
            Hide
            jeffdafoe Jeff Dafoe added a comment -

            Daniel Beck The list you referenced is exactly the same as the list that I see in sysinternals Process Explorer when I look for locked JPI files.under the jenkins process.

            Show
            jeffdafoe Jeff Dafoe added a comment - Daniel Beck The list you referenced is exactly the same as the list that I see in sysinternals Process Explorer when I look for locked JPI files.under the jenkins process.
            Hide
            danielbeck Daniel Beck added a comment -

            Responsible commit through git bisect (on OS X, with lsof, and running using java -jar jenkins.war):
            https://github.com/jenkinsci/jenkins/pull/2489/commits/2a450e5ac2ecd83a54f05c83729852073601816e

            $ git bisect good
            2a450e5ac2ecd83a54f05c83729852073601816e is the first bad commit
            commit 2a450e5ac2ecd83a54f05c83729852073601816e
            Author: Stephen Connolly <stephen.alan.connolly@gmail.com>
            Date:   Thu Jul 28 16:09:21 2016 +0100
            
                [FIXED JENKINS-37041] Ensure that detached plugins are always at least their minimum version
            
            :040000 040000 fdfb36c7fafa895b74c030135405fe2920cfc96f d0ead1a29141a585923651e2593c826ce0fdd893 M      core
            
            $ git bisect log
            git bisect start
            # bad: [e02419c61a562d991d8a068a34f357b2342747ca] [maven-release-plugin] prepare release jenkins-2.16
            git bisect bad e02419c61a562d991d8a068a34f357b2342747ca
            # good: [ae59dde643367c90ec4427490b841db9570e32ad] [maven-release-plugin] prepare release jenkins-2.15
            git bisect good ae59dde643367c90ec4427490b841db9570e32ad
            # good: [d7f62a29a6ead4ad35a64a3fcf764d9e25cb6185] Merge pull request #2480 from stephenc/jenkins-36871-prep
            git bisect good d7f62a29a6ead4ad35a64a3fcf764d9e25cb6185
            # bad: [65adca296d42c8038bad27eb1a10b40d71a163f0] Noting merge of JENKINS-37041
            git bisect bad 65adca296d42c8038bad27eb1a10b40d71a163f0
            # bad: [d92fd4cc63e19c60e374e45e045dcdce2d91e951] [JENKINS-37041] Address Code review comments
            git bisect bad d92fd4cc63e19c60e374e45e045dcdce2d91e951
            # good: [e7b39efff80ebdd50ddd8ab20ab2adee1d5b1dcc] Merge pull request #2485 from stephenc/jenkins-36996
            git bisect good e7b39efff80ebdd50ddd8ab20ab2adee1d5b1dcc
            # bad: [2a450e5ac2ecd83a54f05c83729852073601816e] [FIXED JENKINS-37041] Ensure that detached plugins are always at least their minimum version
            git bisect bad 2a450e5ac2ecd83a54f05c83729852073601816e
            # good: [8e536513e8d7b08cda63201ba14ee6e30289d713] [JENKINS-36996] Noting merge
            git bisect good 8e536513e8d7b08cda63201ba14ee6e30289d713
            # first bad commit: [2a450e5ac2ecd83a54f05c83729852073601816e] [FIXED JENKINS-37041] Ensure that detached plugins are always at least their minimum version
            
            Show
            danielbeck Daniel Beck added a comment - Responsible commit through git bisect (on OS X, with lsof , and running using java -jar jenkins.war ): https://github.com/jenkinsci/jenkins/pull/2489/commits/2a450e5ac2ecd83a54f05c83729852073601816e $ git bisect good 2a450e5ac2ecd83a54f05c83729852073601816e is the first bad commit commit 2a450e5ac2ecd83a54f05c83729852073601816e Author: Stephen Connolly <stephen.alan.connolly@gmail.com> Date: Thu Jul 28 16:09:21 2016 +0100 [FIXED JENKINS-37041] Ensure that detached plugins are always at least their minimum version :040000 040000 fdfb36c7fafa895b74c030135405fe2920cfc96f d0ead1a29141a585923651e2593c826ce0fdd893 M core $ git bisect log git bisect start # bad: [e02419c61a562d991d8a068a34f357b2342747ca] [maven-release-plugin] prepare release jenkins-2.16 git bisect bad e02419c61a562d991d8a068a34f357b2342747ca # good: [ae59dde643367c90ec4427490b841db9570e32ad] [maven-release-plugin] prepare release jenkins-2.15 git bisect good ae59dde643367c90ec4427490b841db9570e32ad # good: [d7f62a29a6ead4ad35a64a3fcf764d9e25cb6185] Merge pull request #2480 from stephenc/jenkins-36871-prep git bisect good d7f62a29a6ead4ad35a64a3fcf764d9e25cb6185 # bad: [65adca296d42c8038bad27eb1a10b40d71a163f0] Noting merge of JENKINS-37041 git bisect bad 65adca296d42c8038bad27eb1a10b40d71a163f0 # bad: [d92fd4cc63e19c60e374e45e045dcdce2d91e951] [JENKINS-37041] Address Code review comments git bisect bad d92fd4cc63e19c60e374e45e045dcdce2d91e951 # good: [e7b39efff80ebdd50ddd8ab20ab2adee1d5b1dcc] Merge pull request #2485 from stephenc/jenkins-36996 git bisect good e7b39efff80ebdd50ddd8ab20ab2adee1d5b1dcc # bad: [2a450e5ac2ecd83a54f05c83729852073601816e] [FIXED JENKINS-37041] Ensure that detached plugins are always at least their minimum version git bisect bad 2a450e5ac2ecd83a54f05c83729852073601816e # good: [8e536513e8d7b08cda63201ba14ee6e30289d713] [JENKINS-36996] Noting merge git bisect good 8e536513e8d7b08cda63201ba14ee6e30289d713 # first bad commit: [2a450e5ac2ecd83a54f05c83729852073601816e] [FIXED JENKINS-37041] Ensure that detached plugins are always at least their minimum version
            Hide
            danielbeck Daniel Beck added a comment -

            Or maybe not. Trying to locate/fix the issue wasn't all that successful.

            Oleg Nenashev told me he'll look into this tomorrow, so for record keeping, assigning to him for now.

            Show
            danielbeck Daniel Beck added a comment - Or maybe not. Trying to locate/fix the issue wasn't all that successful. Oleg Nenashev told me he'll look into this tomorrow, so for record keeping, assigning to him for now.
            Hide
            oleg_nenashev Oleg Nenashev added a comment - - edited

            Here is a cause stacktrace for the runaway file descriptor (on 2.16 war). Fixing the issue.

            #46 /Users/nenashev/Documents/jenkins/test/jenkins-2.16/jenkins/plugins/mailer.jpi by thread:Loading bundled plugins on Fri Aug 19 13:50:21 CEST 2016
            	at java.util.zip.ZipFile.<init>(ZipFile.java:146)
            	at java.util.jar.JarFile.<init>(JarFile.java:154)
            	at java.util.jar.JarFile.<init>(JarFile.java:91)
            	at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93)
            	at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69)
            	at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84)
            	at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122)
            	at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
            	at java.net.URL.openStream(URL.java:1037)
            	at hudson.PluginManager.parsePluginManifest(PluginManager.java:975)
            	at hudson.PluginManager.getPluginVersion(PluginManager.java:775)
            	at hudson.PluginManager.getPluginVersion(PluginManager.java:768)
            	at hudson.PluginManager.getPluginVersion(PluginManager.java:756)
            	at hudson.PluginManager.loadDetachedPlugins(PluginManager.java:722)
            	at hudson.LocalPluginManager.loadBundledPlugins(LocalPluginManager.java:85)
            	at hudson.PluginManager$1$1.run(PluginManager.java:374)
            	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
            	at jenkins.model.Jenkins$7.runTask(Jenkins.java:1026)
            	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
            	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            	at java.lang.Thread.run(Thread.java:745)
            
            Show
            oleg_nenashev Oleg Nenashev added a comment - - edited Here is a cause stacktrace for the runaway file descriptor (on 2.16 war). Fixing the issue. #46 /Users/nenashev/Documents/jenkins/test/jenkins-2.16/jenkins/plugins/mailer.jpi by thread:Loading bundled plugins on Fri Aug 19 13:50:21 CEST 2016 at java.util.zip.ZipFile.<init>(ZipFile.java:146) at java.util.jar.JarFile.<init>(JarFile.java:154) at java.util.jar.JarFile.<init>(JarFile.java:91) at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:93) at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:84) at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:122) at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150) at java.net.URL.openStream(URL.java:1037) at hudson.PluginManager.parsePluginManifest(PluginManager.java:975) at hudson.PluginManager.getPluginVersion(PluginManager.java:775) at hudson.PluginManager.getPluginVersion(PluginManager.java:768) at hudson.PluginManager.getPluginVersion(PluginManager.java:756) at hudson.PluginManager.loadDetachedPlugins(PluginManager.java:722) at hudson.LocalPluginManager.loadBundledPlugins(LocalPluginManager.java:85) at hudson.PluginManager$1$1.run(PluginManager.java:374) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:1026) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            So the Jenkins code behaves correctly, but there is an issue with JARURLConnection, which leaks handles in the common use-case, because it does not cleanup underlying resources.

            Nice reading: http://www.genevski.com/2010/04/javaneturl-and-jarurlconnection-may.html

            Show
            oleg_nenashev Oleg Nenashev added a comment - So the Jenkins code behaves correctly, but there is an issue with JARURLConnection, which leaks handles in the common use-case, because it does not cleanup underlying resources. Nice reading: http://www.genevski.com/2010/04/javaneturl-and-jarurlconnection-may.html
            Hide
            oleg_nenashev Oleg Nenashev added a comment -
            Show
            oleg_nenashev Oleg Nenashev added a comment - Pull request: https://github.com/jenkinsci/jenkins/pull/2516
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/main/java/hudson/PluginManager.java
            core/src/main/java/hudson/Util.java
            core/src/test/java/hudson/PluginManagerTest.java
            http://jenkins-ci.org/commit/jenkins/96c97860b0f018094c20283b15f3ddb3bdd9effe
            Log:
            [FIXED JENKINS-37332] - Prevent File descriptor leaks when reading manifests from JARs (#2516)

            • JENKINS-37332 - Improve diagnostics of non-closed streams during reading of the manifests in PluginManager
            • JENKINS-37332 - Leakless processing of JarUrlConnection during Manifest parsing
            • JENKINS-37332 - Also implement leak-safe method for retrieving file modification date
            • JENKINS-37332 - Add spotcheck methods for manifest file access + Javadoc
            • JENKINS-37332 - Also test multi-line and empty attributes in the test
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/hudson/PluginManager.java core/src/main/java/hudson/Util.java core/src/test/java/hudson/PluginManagerTest.java http://jenkins-ci.org/commit/jenkins/96c97860b0f018094c20283b15f3ddb3bdd9effe Log: [FIXED JENKINS-37332] - Prevent File descriptor leaks when reading manifests from JARs (#2516) JENKINS-37332 - Improve diagnostics of non-closed streams during reading of the manifests in PluginManager JENKINS-37332 - Leakless processing of JarUrlConnection during Manifest parsing JENKINS-37332 - Also implement leak-safe method for retrieving file modification date JENKINS-37332 - Add spotcheck methods for manifest file access + Javadoc JENKINS-37332 - Also test multi-line and empty attributes in the test
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The fix should be released in 2.19

            Show
            oleg_nenashev Oleg Nenashev added a comment - The fix should be released in 2.19
            Hide
            brianeray Brian Ray added a comment -

            Just hit this after upgrading to LTS 2.7.3 on Windows. If my records are correct I was able to update bundled plugins without issue on LTS 2.7.2. Is the workaround simply to downgrade to 2.7.2 until there's a new LTS release with the current fix?

            Or is there a sneaky hack to stay on 2.7.3 and update the bundled plugins?

            Thanks

            Show
            brianeray Brian Ray added a comment - Just hit this after upgrading to LTS 2.7.3 on Windows. If my records are correct I was able to update bundled plugins without issue on LTS 2.7.2. Is the workaround simply to downgrade to 2.7.2 until there's a new LTS release with the current fix? Or is there a sneaky hack to stay on 2.7.3 and update the bundled plugins? Thanks
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The fix should have been integraed into 2.7.3 Checking

            Show
            oleg_nenashev Oleg Nenashev added a comment - The fix should have been integraed into 2.7.3 Checking
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Well, it was not
            CC Daniel Beck Oliver Gondža

            Show
            oleg_nenashev Oleg Nenashev added a comment - Well, it was not CC Daniel Beck Oliver Gondža
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Brian Ray
            Are you sure it is caused by this problem? https://github.com/jenkinsci/jenkins/commit/2a450e5ac2ecd83a54f05c83729852073601816e has not been backported to 2.7.3, hence there should not be such problem.

            Could you please run your instance with http://file-leak-detector.kohsuke.org/ and report the list of open file handles?

            Show
            oleg_nenashev Oleg Nenashev added a comment - Brian Ray Are you sure it is caused by this problem? https://github.com/jenkinsci/jenkins/commit/2a450e5ac2ecd83a54f05c83729852073601816e has not been backported to 2.7.3, hence there should not be such problem. Could you please run your instance with http://file-leak-detector.kohsuke.org/ and report the list of open file handles?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -
            Show
            oleg_nenashev Oleg Nenashev added a comment - Oops. Logs say it has been actually backported: https://github.com/jenkinsci/jenkins/commit/6ea7883fed497b68eec9f89ec89082edf1b39ce5
            Hide
            brianeray Brian Ray added a comment -

            Thank you Oleg Nenashev for the quick response and making this an LTS candidate. My process explorer revealed the same leaked file descriptors on the bundled plugin JPIs.

            My workaround was to downgrade to 2.7.2, upgrade these plugins, then upgrade once again to 2.7.3.

            Show
            brianeray Brian Ray added a comment - Thank you Oleg Nenashev for the quick response and making this an LTS candidate. My process explorer revealed the same leaked file descriptors on the bundled plugin JPIs. My workaround was to downgrade to 2.7.2, upgrade these plugins, then upgrade once again to 2.7.3.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/main/java/hudson/PluginManager.java
            core/src/main/java/hudson/Util.java
            core/src/test/java/hudson/PluginManagerTest.java
            http://jenkins-ci.org/commit/jenkins/37edc1a3b0c5670c24bd06d157f4de91b99f8391
            Log:
            [FIXED JENKINS-37332] - Prevent File descriptor leaks when reading manifests from JARs (#2516)

            • JENKINS-37332 - Improve diagnostics of non-closed streams during reading of the manifests in PluginManager
            • JENKINS-37332 - Leakless processing of JarUrlConnection during Manifest parsing
            • JENKINS-37332 - Also implement leak-safe method for retrieving file modification date
            • JENKINS-37332 - Add spotcheck methods for manifest file access + Javadoc
            • JENKINS-37332 - Also test multi-line and empty attributes in the test

            (cherry picked from commit 96c97860b0f018094c20283b15f3ddb3bdd9effe)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/hudson/PluginManager.java core/src/main/java/hudson/Util.java core/src/test/java/hudson/PluginManagerTest.java http://jenkins-ci.org/commit/jenkins/37edc1a3b0c5670c24bd06d157f4de91b99f8391 Log: [FIXED JENKINS-37332] - Prevent File descriptor leaks when reading manifests from JARs (#2516) JENKINS-37332 - Improve diagnostics of non-closed streams during reading of the manifests in PluginManager JENKINS-37332 - Leakless processing of JarUrlConnection during Manifest parsing JENKINS-37332 - Also implement leak-safe method for retrieving file modification date JENKINS-37332 - Add spotcheck methods for manifest file access + Javadoc JENKINS-37332 - Also test multi-line and empty attributes in the test (cherry picked from commit 96c97860b0f018094c20283b15f3ddb3bdd9effe)
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Daniel Beck It has been reopened intentionally since the bug is in LTS now

            Show
            oleg_nenashev Oleg Nenashev added a comment - Daniel Beck It has been reopened intentionally since the bug is in LTS now
            Hide
            cobexer Ing. Christoph Obexer added a comment -

            Oliver Gondža I also have this issue on our Jenkins instance (IIRC with the Mailer plugin).

            Show
            cobexer Ing. Christoph Obexer added a comment - Oliver Gondža I also have this issue on our Jenkins instance (IIRC with the Mailer plugin).
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            2.7.4 release discussion: https://groups.google.com/forum/#!topic/jenkinsci-dev/zIq6EgUPQBI
            Please vote there if you're affected

            Show
            oleg_nenashev Oleg Nenashev added a comment - 2.7.4 release discussion: https://groups.google.com/forum/#!topic/jenkinsci-dev/zIq6EgUPQBI Please vote there if you're affected
            Hide
            olivergondza Oliver Gondža added a comment -

            Fixed on 2.7 LTS line in 2.7.4.

            Show
            olivergondza Oliver Gondža added a comment - Fixed on 2.7 LTS line in 2.7.4.

              People

              • Assignee:
                oleg_nenashev Oleg Nenashev
                Reporter:
                nitek Claus Näveke
              • Votes:
                13 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: