Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19741

Massive jobs execution causes issues in lastSuccessfulBuild permalink update

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: core
    • Environment:
      1) 1.509.3, Linux server (RHEL6)
      2) 1.509.4 LTS, SL Linux 6.4
    • Similar Issues:

      Description

      We run massive parallel builds of the free-style job (~120 simultaneous runs). Most of runs take about 15 seconds, to job frequently updates the build statistics.

      Several jobs print warnings to the log. I suppose that issue is caused by parallel updates of the file.

      Sep 25, 2013 10:42:48 AM jenkins.model.PeepholePermalink updateCache
      WARNING: Failed to update hudson.model.FreeStyleProject@3771fd63[SBC_Run_Any_Batch] lastSuccessfulBuild permalink for MY_JOB #3830
      java.nio.file.NoSuchFileException: /var/lib/jenkins/jobs/MY_JOB/builds/lastSuccessfulBuild.tmp
      at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
      at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
      at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
      at sun.nio.fs.UnixFileSystemProvider.readSymbolicLink(UnixFileSystemProvider.java:495)
      at java.nio.file.Files.readSymbolicLink(Files.java:1382)
      at sun.reflect.GeneratedMethodAccessor249.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at hudson.Util.resolveSymlink(Util.java:1213)
      at jenkins.model.PeepholePermalink.writeSymlink(PeepholePermalink.java:169)
      at jenkins.model.PeepholePermalink.updateCache(PeepholePermalink.java:137)
      at jenkins.model.PeepholePermalink.resolve(PeepholePermalink.java:114)
      at jenkins.model.PeepholePermalink$RunListenerImpl.onCompleted(PeepholePermalink.java:212)
      at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:199)
      at hudson.model.Run.execute(Run.java:1642)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:237)

        Attachments

          Issue Links

            Activity

            Hide
            hjhafner Hans-Juergen Hafner added a comment -

            We have also a job which is executed concurrent. In average approximately 12 are running at the same time. We also see problems with update of premalinks.
            For us not working links for this job are no big issue but the "NoSuchFileException" is flooding Jenkins log.
            This job is running ~1300 times a day and therefore the exception is adding ~1.5 MBytes to log.
            Is there a way to reduce logs for such a case?

            Show
            hjhafner Hans-Juergen Hafner added a comment - We have also a job which is executed concurrent. In average approximately 12 are running at the same time. We also see problems with update of premalinks. For us not working links for this job are no big issue but the "NoSuchFileException" is flooding Jenkins log. This job is running ~1300 times a day and therefore the exception is adding ~1.5 MBytes to log. Is there a way to reduce logs for such a case?
            Hide
            smr88 Steve Robbins added a comment - - edited

            I don't think this has anything to do with parallel jobs execution ... except perhaps that it is easier to expose the race condition?

            We see the same failures to update permalinks – both lastSuccessfulBuild and lastFailedBuild – on a small installation without parallel jobs (one node, with #executors set to 1) and with low execution rates. Also the server is Windows Server 2012 R2 Standard, so not a linux-only issue. Jenkins ver. 1.642.2.

            Another difference is that instead of NoSuchFile, we see DirectoryNotEmptyException:

            Mar 05, 2016 4:06:09 AM jenkins.model.PeepholePermalink updateCache
            WARNING: Failed to update hudson.model.FreeStyleProject@320509[40018_mmiDotNetMath_main] lastSuccessfulBuild permalink for 40018_mmiDotNetMath_main #767
            java.nio.file.DirectoryNotEmptyException: C:\Jenkins\jobs\40018_mmiDotNetMath_main\builds\lastSuccessfulBuild
            at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
            at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
            at java.nio.file.Files.delete(Unknown Source)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
            at java.lang.reflect.Method.invoke(Unknown Source)
            at hudson.Util.deleteFile(Util.java:255)
            at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:113)
            at jenkins.model.PeepholePermalink.writeSymlink(PeepholePermalink.java:200)
            at jenkins.model.PeepholePermalink.updateCache(PeepholePermalink.java:150)
            at jenkins.model.PeepholePermalink$RunListenerImpl.onCompleted(PeepholePermalink.java:237)
            at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:201)
            at hudson.model.Run.execute(Run.java:1783)
            at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
            at hudson.model.ResourceController.execute(ResourceController.java:98)
            at hudson.model.Executor.run(Executor.java:410)

            Show
            smr88 Steve Robbins added a comment - - edited I don't think this has anything to do with parallel jobs execution ... except perhaps that it is easier to expose the race condition? We see the same failures to update permalinks – both lastSuccessfulBuild and lastFailedBuild – on a small installation without parallel jobs (one node, with #executors set to 1) and with low execution rates. Also the server is Windows Server 2012 R2 Standard, so not a linux-only issue. Jenkins ver. 1.642.2. Another difference is that instead of NoSuchFile, we see DirectoryNotEmptyException: Mar 05, 2016 4:06:09 AM jenkins.model.PeepholePermalink updateCache WARNING: Failed to update hudson.model.FreeStyleProject@320509 [40018_mmiDotNetMath_main] lastSuccessfulBuild permalink for 40018_mmiDotNetMath_main #767 java.nio.file.DirectoryNotEmptyException: C:\Jenkins\jobs\40018_mmiDotNetMath_main\builds\lastSuccessfulBuild at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source) at java.nio.file.Files.delete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at hudson.Util.deleteFile(Util.java:255) at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:113) at jenkins.model.PeepholePermalink.writeSymlink(PeepholePermalink.java:200) at jenkins.model.PeepholePermalink.updateCache(PeepholePermalink.java:150) at jenkins.model.PeepholePermalink$RunListenerImpl.onCompleted(PeepholePermalink.java:237) at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:201) at hudson.model.Run.execute(Run.java:1783) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410)
            Hide
            israodin Dimitri Goldshtein added a comment -

            Hi all
            We Have some problem with lasBuild on Jenkins ver. 1.651.3

            WARNING: Failed to update hudson.model.FreeStyleProject@1c6fcad[CD-CFW_Selenium_Part1] lastSuccessfulBuild permalink for CD-CFW_Selenium_Part1 #5631
            java.nio.file.DirectoryNotEmptyException: c:\Jenkins\jobs\CD-CFW_Selenium_Part1\builds\lastSuccessfulBuild
            at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
            at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
            at java.nio.file.Files.delete(Unknown Source)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
            at java.lang.reflect.Method.invoke(Unknown Source)
            at hudson.Util.deleteFile(Util.java:257)
            at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:113)
            at jenkins.model.PeepholePermalink.writeSymlink(PeepholePermalink.java:200)
            at jenkins.model.PeepholePermalink.updateCache(PeepholePermalink.java:150)
            at jenkins.model.PeepholePermalink.resolve(PeepholePermalink.java:122)
            at hudson.model.Job.getLastSuccessfulBuild(Job.java:913)
            at jenkins.tasks.SimpleBuildStep$LastBuildActionFactory.createFor(SimpleBuildStep.java:113)
            at jenkins.tasks.SimpleBuildStep$LastBuildActionFactory.createFor(SimpleBuildStep.java:99)
            at hudson.model.Actionable.createFor(Actionable.java:107)
            at hudson.model.Actionable.getAllActions(Actionable.java:98)
            at hudson.model.Actionable.getAction(Actionable.java:165)
            at hudson.plugins.nunit.NUnitPublisher.getProjectAction(NUnitPublisher.java:94)
            at hudson.tasks.BuildStepCompatibilityLayer.getProjectActions(BuildStepCompatibilityLayer.java:98)
            at hudson.model.Project.createTransientActions(Project.java:252)
            at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:755)
            at hudson.model.AbstractProject.onLoad(AbstractProject.java:339)
            at hudson.model.Project.onLoad(Project.java:98)
            at hudson.model.Items.load(Items.java:327)
            at jenkins.model.Jenkins$18.run(Jenkins.java:2720)
            at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
            at jenkins.model.Jenkins$8.runTask(Jenkins.java:926)
            at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
            at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

            Show
            israodin Dimitri Goldshtein added a comment - Hi all We Have some problem with lasBuild on Jenkins ver. 1.651.3 WARNING: Failed to update hudson.model.FreeStyleProject@1c6fcad [CD-CFW_Selenium_Part1] lastSuccessfulBuild permalink for CD-CFW_Selenium_Part1 #5631 java.nio.file.DirectoryNotEmptyException: c:\Jenkins\jobs\CD-CFW_Selenium_Part1\builds\lastSuccessfulBuild at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source) at java.nio.file.Files.delete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at hudson.Util.deleteFile(Util.java:257) at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:113) at jenkins.model.PeepholePermalink.writeSymlink(PeepholePermalink.java:200) at jenkins.model.PeepholePermalink.updateCache(PeepholePermalink.java:150) at jenkins.model.PeepholePermalink.resolve(PeepholePermalink.java:122) at hudson.model.Job.getLastSuccessfulBuild(Job.java:913) at jenkins.tasks.SimpleBuildStep$LastBuildActionFactory.createFor(SimpleBuildStep.java:113) at jenkins.tasks.SimpleBuildStep$LastBuildActionFactory.createFor(SimpleBuildStep.java:99) at hudson.model.Actionable.createFor(Actionable.java:107) at hudson.model.Actionable.getAllActions(Actionable.java:98) at hudson.model.Actionable.getAction(Actionable.java:165) at hudson.plugins.nunit.NUnitPublisher.getProjectAction(NUnitPublisher.java:94) at hudson.tasks.BuildStepCompatibilityLayer.getProjectActions(BuildStepCompatibilityLayer.java:98) at hudson.model.Project.createTransientActions(Project.java:252) at hudson.model.AbstractProject.updateTransientActions(AbstractProject.java:755) at hudson.model.AbstractProject.onLoad(AbstractProject.java:339) at hudson.model.Project.onLoad(Project.java:98) at hudson.model.Items.load(Items.java:327) at jenkins.model.Jenkins$18.run(Jenkins.java:2720) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$8.runTask(Jenkins.java:926) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)
            Hide
            inetov Yuriy Inetov added a comment - - edited

            My case: JENKINS-54863

            I do not understand how much they are the same..

            Show
            inetov Yuriy Inetov added a comment - - edited My case: JENKINS-54863 I do not understand how much they are the same..
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Yuriy Inetov The error messages are not 100% same, because the AtomicFile logic has changed significantly since 2013. But yes, the rootcause is the same

            Show
            oleg_nenashev Oleg Nenashev added a comment - Yuriy Inetov The error messages are not 100% same, because the AtomicFile logic has changed significantly since 2013. But yes, the rootcause is the same

              People

              • Assignee:
                Unassigned
                Reporter:
                oleg_nenashev Oleg Nenashev
              • Votes:
                7 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated: