Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-2487

"Please wait, getting ready to work" takes very long

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Incomplete
    • Component/s: other
    • Labels:
      None
    • Environment:
      Platform: All, OS: All
    • Similar Issues:

      Description

      We're running quite an extensive setup with dozens of projects and thousands of
      builds for each project. At the moment we're purging build artefacts but keep
      the actual build history+metadata indefinitely. Maybe because of that, startup
      up hudson now takes in excess of 40 minutes. vmstat shows the disk being busy.

      It would be preferable if startup time of hudson were unaffected by the number
      of builds or projects. I'm putting this as "enhancement", but it's really a
      rather important one for us.

        Attachments

          Issue Links

            Activity

            blackmuzzle blackmuzzle created issue -
            Hide
            ejono ejono added a comment -

            I have about 100 projects, many of which have thousands of archived builds, and
            it takes about 45 minutes to start up as well.

            Show
            ejono ejono added a comment - I have about 100 projects, many of which have thousands of archived builds, and it takes about 45 minutes to start up as well.
            Hide
            ejono ejono added a comment -

            adding myself to the CC list

            Show
            ejono ejono added a comment - adding myself to the CC list
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -
                • Issue 3523 has been marked as a duplicate of this issue. ***
            Show
            kohsuke Kohsuke Kawaguchi added a comment - Issue 3523 has been marked as a duplicate of this issue. ***
            kohsuke Kohsuke Kawaguchi made changes -
            Field Original Value New Value
            Link This issue is duplicated by JENKINS-3523 [ JENKINS-3523 ]
            Hide
            bradfritz bradfritz added a comment -

            adding myself to CC list

            Show
            bradfritz bradfritz added a comment - adding myself to CC list
            Hide
            rtyler R. Tyler Croy added a comment -

            Adding myself to CC

            Show
            rtyler R. Tyler Croy added a comment - Adding myself to CC
            Hide
            mdonohue mdonohue added a comment -
                • Issue 2890 has been marked as a duplicate of this issue. ***
            Show
            mdonohue mdonohue added a comment - Issue 2890 has been marked as a duplicate of this issue. ***
            mdonohue mdonohue made changes -
            Link This issue is duplicated by JENKINS-2890 [ JENKINS-2890 ]
            Show
            huybrechts huybrechts added a comment - See http://n4.nabble.com/Restart-hudson-takes-too-much-time-td386719.html for some tips.
            Hide
            mdillon mdillon added a comment -

            I notice in the referenced thread that Kohsuke mentions refactoring the Hudson core to allow build data to be lazily loaded. I would absolutely love to see that happen.

            Is this JIRA issue the one that that work would be done under?

            Show
            mdillon mdillon added a comment - I notice in the referenced thread that Kohsuke mentions refactoring the Hudson core to allow build data to be lazily loaded. I would absolutely love to see that happen. Is this JIRA issue the one that that work would be done under?
            Hide
            abarbieri Andrea Barbieri added a comment -

            I'd love to see this happening too.
            Downstream buildview plugin forexample takes 'ages' with several thousands of builds to go through even when changing the source code to reduce the amount of debug/info generated.

            Show
            abarbieri Andrea Barbieri added a comment - I'd love to see this happening too. Downstream buildview plugin forexample takes 'ages' with several thousands of builds to go through even when changing the source code to reduce the amount of debug/info generated.
            Hide
            salimfadhley salimfadhley added a comment -

            Bump!

            This is one of the worst aspects of Hudson. Can anybody suggest a work-around? Would simply discarding some historic builds reduce the startup-time? What is Hudson actually doing that could take so long?

            Show
            salimfadhley salimfadhley added a comment - Bump! This is one of the worst aspects of Hudson. Can anybody suggest a work-around? Would simply discarding some historic builds reduce the startup-time? What is Hudson actually doing that could take so long?
            Hide
            croesus Croesus Kall added a comment - - edited

            My setup takes nearly two hours to start. It has more than 100 jobs. It runs on unix, and the iostat and vmstat commands seem to denote minimal activity during startup.
            When fine logging is enabled, the log first gets a long stream of:

            FINE: Queue maintenance started hudson.model.Queue
            
            Show
            croesus Croesus Kall added a comment - - edited My setup takes nearly two hours to start. It has more than 100 jobs. It runs on unix, and the iostat and vmstat commands seem to denote minimal activity during startup. When fine logging is enabled, the log first gets a long stream of: FINE: Queue maintenance started hudson.model.Queue
            Hide
            taksan taksan added a comment -

            We also have an environment with 400+ jobs, 48 slaves and literally hundreds of thousands of builds.
            With this setup, our jenkins takes about 80min to start.

            Show
            taksan taksan added a comment - We also have an environment with 400+ jobs, 48 slaves and literally hundreds of thousands of builds. With this setup, our jenkins takes about 80min to start.
            Hide
            hmeerlo Herman Meerlo added a comment -

            Ok, this is really ridiculous. What on earth is it doing for 45 minutes?

            Show
            hmeerlo Herman Meerlo added a comment - Ok, this is really ridiculous. What on earth is it doing for 45 minutes?
            Hide
            martin_naughton martin naughton added a comment -

            is there any logs to see what it is doing on startup? It seems from the comments that it is recursively finding each build and loading it in to memory. Makes jenkins unscalable from a start up point of view.

            Any action on this?

            Show
            martin_naughton martin naughton added a comment - is there any logs to see what it is doing on startup? It seems from the comments that it is recursively finding each build and loading it in to memory. Makes jenkins unscalable from a start up point of view. Any action on this?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            There's no info to analyze the issue.
            This issue could be caused by plugins, memory overflows, disk throughput, etc.
            As example, my installations with ~100 slaves, 1000 jobs and 50Gb of the build history start within 10 minutes, hence it is not a "system-wide" problem

            Please submit Jenkins startup logs or run your Jenkins installations inside the profiler.

            Show
            oleg_nenashev Oleg Nenashev added a comment - There's no info to analyze the issue. This issue could be caused by plugins, memory overflows, disk throughput, etc. As example, my installations with ~100 slaves, 1000 jobs and 50Gb of the build history start within 10 minutes, hence it is not a "system-wide" problem Please submit Jenkins startup logs or run your Jenkins installations inside the profiler.
            Hide
            francisdb francisdb added a comment -

            And a service that needs 10 minutes to start is ok?

            Show
            francisdb francisdb added a comment - And a service that needs 10 minutes to start is ok?
            Hide
            taksan taksan added a comment -

            Just some insight for this issue:

            We had profiled jenkins during our slow startup and found that what was really taking a long time was reading all the builds.

            To work around the problem, we did an exhaustive cleanup and setup a very aggressive policy to clean up old builds (only a few selected jobs keep a longer history).

            We were able to reduce the startup time from 80min to 5 minutes.

            Other things that help: because the problem is due to intensive I/O reading every build file, improving the hardware also helps to alleviate the problem.

            Show
            taksan taksan added a comment - Just some insight for this issue: We had profiled jenkins during our slow startup and found that what was really taking a long time was reading all the builds. To work around the problem, we did an exhaustive cleanup and setup a very aggressive policy to clean up old builds (only a few selected jobs keep a longer history). We were able to reduce the startup time from 80min to 5 minutes. Other things that help: because the problem is due to intensive I/O reading every build file, improving the hardware also helps to alleviate the problem.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            @francisdb
            > And a service that needs 10 minutes to start is ok?
            Many services like DBMS may require several hours to startup, so several minutes are not the showstopper in general.

            Jenkins developers have implemented much features to decrease the data traffic on the startup (lazy loading of builds, delayed tasks, etc.). BTW, Jenkins definitely need to initialize much data objects and to establish connections with nodes, hence it is not possible to nullify the initialization time.

            Any plugin can "contribute" to the startup time, hence it is mandatory to know about the Jenkins configuration to localize your issues.

            Show
            oleg_nenashev Oleg Nenashev added a comment - @francisdb > And a service that needs 10 minutes to start is ok? Many services like DBMS may require several hours to startup, so several minutes are not the showstopper in general. Jenkins developers have implemented much features to decrease the data traffic on the startup (lazy loading of builds, delayed tasks, etc.). BTW, Jenkins definitely need to initialize much data objects and to establish connections with nodes, hence it is not possible to nullify the initialization time. Any plugin can "contribute" to the startup time, hence it is mandatory to know about the Jenkins configuration to localize your issues.
            Hide
            svalkanesko Sergey Valkanesko added a comment -

            >Many services like DBMS may require several hours to startup, so several minutes are not the showstopper in general.

            This statement received in email notification made me recover my password to reply. Seriously? Do you really believe that this should be considered an excuse for this issue?

            Show
            svalkanesko Sergey Valkanesko added a comment - >Many services like DBMS may require several hours to startup, so several minutes are not the showstopper in general. This statement received in email notification made me recover my password to reply. Seriously? Do you really believe that this should be considered an excuse for this issue?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Nobody excuses for the "issue".
            Many systems (including Jenkins) are slow by design due to the runtime specifics.
            In the case of Jenkins there're known bottlenecks: queue handling, security checks, listeners, etc.
            The performance could be improved step-by-step, but it is not a single issue in the code.

            If you have such strong complains, just bother to provide the info about your installation and to profile your Jenkins installation.
            Any other contributions to Jenkins core or plugins would be appreciated as well.

            Show
            oleg_nenashev Oleg Nenashev added a comment - Nobody excuses for the "issue". Many systems (including Jenkins) are slow by design due to the runtime specifics. In the case of Jenkins there're known bottlenecks: queue handling, security checks, listeners, etc. The performance could be improved step-by-step, but it is not a single issue in the code. If you have such strong complains, just bother to provide the info about your installation and to profile your Jenkins installation. Any other contributions to Jenkins core or plugins would be appreciated as well.
            Hide
            francisdb francisdb added a comment -

            If all builds ever executed need to be read in before jenkins kan show it's main ui this indeed sounds like a design issue. Anyway, I no longer have a big instance running so I can live with the ~1 min startup time we have here.

            Show
            francisdb francisdb added a comment - If all builds ever executed need to be read in before jenkins kan show it's main ui this indeed sounds like a design issue. Anyway, I no longer have a big instance running so I can live with the ~1 min startup time we have here.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            @francisdb
            > If all builds ever executed need to be read in before jenkins kan show it's main ui this indeed sounds like a design issue
            This issue has been already solved in 1.485 (JENKINS-8754). However, there were many regression issues caused by the lazy loading, hence it is recommended to use 1.532+
            Now Jenkins loads only the required set if builds. Some plugins may require all builds to be loaded, but it is a topic for additional issues on Jenkins JIRA.

            Show
            oleg_nenashev Oleg Nenashev added a comment - @francisdb > If all builds ever executed need to be read in before jenkins kan show it's main ui this indeed sounds like a design issue This issue has been already solved in 1.485 ( JENKINS-8754 ). However, there were many regression issues caused by the lazy loading, hence it is recommended to use 1.532+ Now Jenkins loads only the required set if builds. Some plugins may require all builds to be loaded, but it is a topic for additional issues on Jenkins JIRA.
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue depends on JENKINS-8754 [ JENKINS-8754 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-20046 [ JENKINS-20046 ]
            Hide
            jglick Jesse Glick added a comment -

            This issue is too vague and general to be worth keeping open. If there are specific startup performance problems reproducible in current versions of Jenkins they should be filed with details.

            Show
            jglick Jesse Glick added a comment - This issue is too vague and general to be worth keeping open. If there are specific startup performance problems reproducible in current versions of Jenkins they should be filed with details.
            jglick Jesse Glick made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Incomplete [ 4 ]
            Hide
            danielbeck Daniel Beck added a comment -
            Show
            danielbeck Daniel Beck added a comment - Relevant information for filing performance-related issues can be found here: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-25473 [ JENKINS-25473 ]
            Hide
            draperp Paul Draper added a comment - - edited

            @huybrechts, http://n4.nabble.com/Restart-hudson-takes-too-much-time-td386719.html is dead.

            @Jesse this is pretty specific. Create "dozens of projects and thousands of builds" (as per the issue description) and then start Jenkins.
            Large numbers of projects and builds slowing down Jenkins startup has been corroborated by many people on this issue.

            Are you not able to reproduce this issue?
            What details do you lack?

            Show
            draperp Paul Draper added a comment - - edited @huybrechts, http://n4.nabble.com/Restart-hudson-takes-too-much-time-td386719.html is dead. @Jesse this is pretty specific. Create "dozens of projects and thousands of builds" (as per the issue description) and then start Jenkins. Large numbers of projects and builds slowing down Jenkins startup has been corroborated by many people on this issue. Are you not able to reproduce this issue? What details do you lack?
            Hide
            danielbeck Daniel Beck added a comment -

            has been corroborated by many people on this issue.

            It's less clear if you consider that Jenkins implemented lazy loading of build records from Oct 2012 to some time in early/mid 2013 (until the major issues were worked out).

            Then there's the case that some plugins have a tendency to require loading all build records, which defeats this optimization.

            What details do you lack?

            Stack traces would be helpful. Lists of installed plugins. Maybe even Support Core plugin support bundles. And of course everything with recent Jenkins versions only.

            And even then, at least my expectation is that many different causes contribute to a long startup time, which will need to be fixed one by one rather than gathered in a catch-all issue.

            Show
            danielbeck Daniel Beck added a comment - has been corroborated by many people on this issue. It's less clear if you consider that Jenkins implemented lazy loading of build records from Oct 2012 to some time in early/mid 2013 (until the major issues were worked out). Then there's the case that some plugins have a tendency to require loading all build records, which defeats this optimization. What details do you lack? Stack traces would be helpful. Lists of installed plugins. Maybe even Support Core plugin support bundles. And of course everything with recent Jenkins versions only. And even then, at least my expectation is that many different causes contribute to a long startup time, which will need to be fixed one by one rather than gathered in a catch-all issue.
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 132560 ] JNJira + In-Review [ 186551 ]

              People

              • Assignee:
                Unassigned
                Reporter:
                blackmuzzle blackmuzzle
              • Votes:
                13 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: