Details

    • Similar Issues:

      Description

      This just started when I updated yesterday. I went from 1.529 to 1.536 and also updated all my plugins. After that the end of my builds started to have a huge pause. The e-mail gets sent which was the end of the run before. Now after I get the e-mail, run is still going on

      BUILD SUCCESSFUL
      
      Total time: 11 mins 32.749 secs
      Build step 'Invoke Gradle script' changed build result to SUCCESS
      [CHECKSTYLE] Collecting checkstyle analysis files...
      [CHECKSTYLE] Finding all files that match the pattern app*/build/reports/checkstyle/*.xml
      [CHECKSTYLE] Parsing 26 files in /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-ak/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-ak/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-cn/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-cn/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-ct/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-ct/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-dt/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-dt/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-ga/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-ga/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-hi/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-hi/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-ia/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-ia/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-la/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-la/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-nj/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-nj/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-nm/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-nm/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-se/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-se/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-ut/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app-ut/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app/build/reports/checkstyle/main.xml of module  with 0 warnings.
      [CHECKSTYLE] Successfully parsed file /nobackup_st/csb/jenkins_slave_romaine/workspace/SEERDMS/app/build/reports/checkstyle/test.xml of module  with 0 warnings.
      [CHECKSTYLE] Computing warning deltas based on reference build #1158
      [WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
      Archiving artifacts
      Recording test results
      Email was triggered for: Success
      Sending email for trigger: Success
      Sending email to: seerdmsdev@imsweb.com criderp@imsweb.com
      

      There is a 4-5 minute pause here. Other jobs are still waiting for it to finish. Then only the success line is printed and the job completes.

      Finished: SUCCESS
      

      I can't figure out what is going on during that time but it definitely never happened before this update.

      There is a bug in 1.536 which does not allow configuration changes to save, so I rolled back to 1.529 and I still see the pause. It must be one of the plugins, but I can't figure out which one.

        Attachments

          Activity

          Hide
          danielbeck Daniel Beck added a comment -

          Ivan: While the build is in the delay phase, go to /threadDump to see what it does. It should be called "Executor #3 for nodename" or similar.

          Candidates off the top of my head:

          • Deleting old builds (log rotation)
          • Notifying build listeners that do whatever
          • If running on a slave, synchronization

          Please post the thread dump for further investigation.

          Show
          danielbeck Daniel Beck added a comment - Ivan: While the build is in the delay phase, go to /threadDump to see what it does. It should be called "Executor #3 for nodename" or similar. Candidates off the top of my head: Deleting old builds (log rotation) Notifying build listeners that do whatever If running on a slave, synchronization Please post the thread dump for further investigation.
          Hide
          jhansche Joe Hansche added a comment -

          To add a little more info to this: I've noticed the slowdown getting worse as the master has been running for a long time. Seems like a restart of Jenkins tends to clear something out, and the delays are not as bad (not really noticeable as "problematic"). Our last restart was when the master had been running for about 5-6 weeks, and the post-build delay was around 2 minutes. After the restart, the jobs finished without a noticeable lag.

          Show
          jhansche Joe Hansche added a comment - To add a little more info to this: I've noticed the slowdown getting worse as the master has been running for a long time. Seems like a restart of Jenkins tends to clear something out, and the delays are not as bad (not really noticeable as "problematic"). Our last restart was when the master had been running for about 5-6 weeks, and the post-build delay was around 2 minutes. After the restart, the jobs finished without a noticeable lag.
          Hide
          tomahawk1187 Anargyros Tomaras added a comment - - edited

          We are experiencing exactly the same problem.

          Here is an example with a 12 minute !! pause

          Show
          tomahawk1187 Anargyros Tomaras added a comment - - edited We are experiencing exactly the same problem. Here is an example with a 12 minute !! pause
          Hide
          danielbeck Daniel Beck added a comment - - edited

          Anargyros Tomaras Are you starting a bunch of processes during the build? E.g. to execute tests? If so, you could experience JENKINS-55106.

          Also https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump

          Show
          danielbeck Daniel Beck added a comment - - edited Anargyros Tomaras Are you starting a bunch of processes during the build? E.g. to execute tests? If so, you could experience JENKINS-55106 . Also https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump
          Hide
          tomahawk1187 Anargyros Tomaras added a comment -

          Daniel Beck msbuild and vstest.console are run with the parallel flag so they should be creating a few processes yes

          Show
          tomahawk1187 Anargyros Tomaras added a comment - Daniel Beck  msbuild and vstest.console are run with the parallel flag so they should be creating a few processes yes

            People

            • Assignee:
              Unassigned
              Reporter:
              ctmay4 ctmay4
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: