Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21145

Several concurrent builds waits for each other

    Details

    • Similar Issues:

      Description

      Hi,

      We have an upstream project(with configuration matrix) which trigers downstream one. Downstream project has "Throttle Concurrent Builds" option enabled with "Throttle this project alone", "Maximum Total Concurrent Builds=2", "Maximum Concurrent Builds Per Node=2"

      Both upstream and downstream projects have "xUnit" related steps in "Post-build Actions"

      Issue there is when we have triggered 10 concurrent builds(#100 - #109) of the downstream project and the third one (#102) requires more time to finish - all further builds (#103 - #109) will be locked until this third build finish.

      Other example, when two builds there were executing simultaneously, one of them failed and took long time to finish.
      It looks like successful one was waiting for unsuccessful one.

      Just compare end of logs for these pairs of tests.
      Pay attention on time stamps: gap between "Stopping recording" and "Sending e-mails":

      jenkins/view/QA/job/_SIP/job/pjsip_c/2900/console
      jenkins/view/QA/job/_SIP/job/pjsip_c/2901/console

      11:07:30 [xUnit] [INFO] - Starting to record.
      11:07:30 [xUnit] [INFO] - Processing JUnit
      11:07:30 [xUnit] [INFO] - [JUnit] - 3 test report file(s) were found with the pattern 'results-pjsip/*.xml' relative to '/tmp/workspace/_SIP/pjsip_c' for the testing framework 'JUnit'.
      11:07:35 [xUnit] [INFO] - Check 'Failed Tests' threshold.
      11:07:35 [xUnit] [INFO] - The total number of tests for this category exceeds the specified 'failure' threshold value.
      11:07:35 [xUnit] [INFO] - Setting the build status to FAILURE
      11:07:35 [xUnit] [INFO] - Stopping recording.
      11:07:35 Build step 'Publish xUnit test result report' changed build result to FAILURE
      11:07:35 Sending e-mails to:
      11:08:05 Notifying upstream projects of job completion
      11:08:05 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys,Version=390
      11:08:05 Finished: FAILURE

      VS

      10:56:40 [xUnit] [INFO] - Setting the build status to SUCCESS
      10:56:40 [xUnit] [INFO] - Stopping recording.
      11:07:34 Sending e-mails to:
      11:08:04 Notifying upstream projects of job completion
      11:08:04 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys_ora,Version=380
      11:08:04 Finished: SUCCESS

      And:
      jenkins/view/QA/job/_SIP/job/pjsip_c/2904/console
      jenkins/view/QA/job/_SIP/job/pjsip_c/2905/console

      10:34:52 [xUnit] [INFO] - Starting to record.
      10:34:52 [xUnit] [INFO] - Processing JUnit
      10:34:53 [xUnit] [INFO] - [JUnit] - 3 test report file(s) were found with the pattern 'results-pjsip/*.xml' relative to '/tmp/workspace/_SIP/pjsip_c' for the testing framework 'JUnit'.
      10:34:57 [xUnit] [INFO] - Check 'Failed Tests' threshold.
      10:34:57 [xUnit] [INFO] - The total number of tests for this category exceeds the specified 'failure' threshold value.
      10:34:57 [xUnit] [INFO] - Setting the build status to FAILURE
      10:34:57 [xUnit] [INFO] - Stopping recording.
      10:34:57 Build step 'Publish xUnit test result report' changed build result to FAILURE
      10:34:57 Sending e-mails to:
      10:35:36 Notifying upstream projects of job completion
      10:35:36 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys_ora,Version=390
      10:35:36 Finished: FAILURE

      VS

      10:21:28 [xUnit] [INFO] - Setting the build status to SUCCESS
      10:21:28 [xUnit] [INFO] - Stopping recording.
      10:34:57 Sending e-mails to:
      10:35:31 Notifying upstream projects of job completion
      10:35:31 Join notifier cannot find upstream project: _SIP/pjsip_cron/VM=etsys,Version=371
      10:35:31 Finished: SUCCESS

      Could you please advise how we can workaround this issue. Any help is appreciated.

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Post-Build actions (publishers) define the locking behavior in concurrent builds on their own. I've re-assigned the issue to the xunit plugin

            Show
            oleg_nenashev Oleg Nenashev added a comment - Post-Build actions (publishers) define the locking behavior in concurrent builds on their own. I've re-assigned the issue to the xunit plugin
            Hide
            danielbeck Daniel Beck added a comment -

            This looks more like email-ext before 2.37 when it synchronized on BuildStepMonitor to send accurate subjects ('Still Unstable', 'Fixed', etc.). (JENKINS-16376)

            What version of email-ext plugin do you use?

            Show
            danielbeck Daniel Beck added a comment - This looks more like email-ext before 2.37 when it synchronized on BuildStepMonitor to send accurate subjects ('Still Unstable', 'Fixed', etc.). ( JENKINS-16376 ) What version of email-ext plugin do you use?
            Hide
            danielbeck Daniel Beck added a comment -

            Given that this was filed before Email-ext 2.37 with JENKINS-16376 fix was out, I'm resolving this one optimistically

            Show
            danielbeck Daniel Beck added a comment - Given that this was filed before Email-ext 2.37 with JENKINS-16376 fix was out, I'm resolving this one optimistically

              People

              • Assignee:
                Unassigned
                Reporter:
                vbaskoff Vitalii Baskov
              • Votes:
                1 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: