Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-25638

Cannot prevent concurrent build pipelines

    Details

    • Similar Issues:

      Description

      There is no way to inhibit a subsequent instance of a build pipeline from starting while another is in progress.

      I've created a Test job that takes the workspace of the upstream job as a parameter so that it can be equally applied to Gerrit check ins, master updates, and release builds. It's fine to have different pipelines running because they operate in separate workspaces. But when a second instance of a particular pipeline begins before the first has finished, it may remove files that the test relies upon. As a result, I see frequent failures of downstream jobs.

      As a workaround, I've reduced the number of executors to one, and set the priority of downstream jobs to be higher than upstream jobs, but this is still not a guarantee. And it's definitely not ideal because some upstream jobs take a long time and having more executors allows faster jobs to complete.

        Attachments

          Activity

          Hide
          patbos Patrik Boström added a comment -

          Try under "Advanced Project Options" -> "Block build when downstream project is building"

          Show
          patbos Patrik Boström added a comment - Try under "Advanced Project Options" -> "Block build when downstream project is building"
          Hide
          andrewatfornax Andrew Hill added a comment -

          "Advanced Project Options" does not appear to be valid for Multibranch Pipeline projects.

          Show
          andrewatfornax Andrew Hill added a comment - "Advanced Project Options" does not appear to be valid for Multibranch Pipeline projects.
          Hide
          kithril Bruce Fraser added a comment - - edited

          We have been using the "Block build when downstream project is building" on all of our jobs for the last year. We have confirmed that even with this set it fails at least some of the time, in what I suspect is a race condition. What we see is that on some rare occasions, a second pipeline job will start just a second after one of the downstream jobs starts. The log below is crude, but shows the Process ID, date and time, Job Start and End, the workspace inherited from the upstream job and a database inherited from an upstream job.

          At 00:23:49 there was a downstream job started, then at 00:23:50 the concurrent job starts doing a git clean and obliterates files being used by the downstream job. Gerit2 and Master are both at the head of pipelines. All other jobs are downstream.

          104 19/04/16 23:38:34 Start Gerit2 /home/jenkins/workspace/Gerrit2
          5238 20/04/16 00:01:30 End   Master /home/jenkins/workspace/Master/debug/no/simtime/yes debug=no simtime=yes
          5238 20/04/16 00:01:44 Start Master /home/jenkins/workspace/Master/debug/yes/simtime/yes debug=yes simtime=yes
          104 20/04/16 00:04:08 End   Gerit2 /home/jenkins/workspace/Gerrit2
          3028 20/04/16 00:04:19 Start Export /home/jenkins/workspace/Gerrit2 NZDEV2
          5238 20/04/16 00:22:54 End   Master /home/jenkins/workspace/Master/debug/yes/simtime/yes debug=yes simtime=yes
          3028 20/04/16 00:23:37 End   Export /home/jenkins/workspace/Gerrit2 NZDEV2
          2661 20/04/16 00:23:49 Start PBuild /home/jenkins/workspace/Gerrit2 NZDEV2
          105 20/04/16 00:23:50 Start Gerit2 /home/jenkins/workspace/Gerrit2
          3029 20/04/16 00:23:52 Start Export /home/jenkins/workspace/Master/debug/no/simtime/no NZDEV2
          5239 20/04/16 00:24:08 Start Master /home/jenkins/workspace/Master/debug/no/simtime/no debug=no simtime=no
          3029 20/04/16 00:25:18 End   Export /home/jenkins/workspace/Master/debug/no/simtime/no NZOK
          105 20/04/16 00:50:23 End   Gerit2 /home/jenkins/workspace/Gerrit2
          3030 20/04/16 00:50:34 Start Export /home/jenkins/workspace/Gerrit2 NZDEV2
          5239 20/04/16 00:50:46 End   Master /home/jenkins/workspace/Master/debug/no/simtime/no debug=no simtime=no
          5239 20/04/16 00:51:08 Start Master /home/jenkins/workspace/Master/debug/yes/simtime/no debug=yes simtime=no
          

          Jenkins and all plugins were upgraded to the latest just last week.

          Show
          kithril Bruce Fraser added a comment - - edited We have been using the "Block build when downstream project is building" on all of our jobs for the last year. We have confirmed that even with this set it fails at least some of the time, in what I suspect is a race condition. What we see is that on some rare occasions, a second pipeline job will start just a second after one of the downstream jobs starts. The log below is crude, but shows the Process ID, date and time, Job Start and End, the workspace inherited from the upstream job and a database inherited from an upstream job. At 00:23:49 there was a downstream job started, then at 00:23:50 the concurrent job starts doing a git clean and obliterates files being used by the downstream job. Gerit2 and Master are both at the head of pipelines. All other jobs are downstream. 104 19/04/16 23:38:34 Start Gerit2 /home/jenkins/workspace/Gerrit2 5238 20/04/16 00:01:30 End Master /home/jenkins/workspace/Master/debug/no/simtime/yes debug=no simtime=yes 5238 20/04/16 00:01:44 Start Master /home/jenkins/workspace/Master/debug/yes/simtime/yes debug=yes simtime=yes 104 20/04/16 00:04:08 End Gerit2 /home/jenkins/workspace/Gerrit2 3028 20/04/16 00:04:19 Start Export /home/jenkins/workspace/Gerrit2 NZDEV2 5238 20/04/16 00:22:54 End Master /home/jenkins/workspace/Master/debug/yes/simtime/yes debug=yes simtime=yes 3028 20/04/16 00:23:37 End Export /home/jenkins/workspace/Gerrit2 NZDEV2 2661 20/04/16 00:23:49 Start PBuild /home/jenkins/workspace/Gerrit2 NZDEV2 105 20/04/16 00:23:50 Start Gerit2 /home/jenkins/workspace/Gerrit2 3029 20/04/16 00:23:52 Start Export /home/jenkins/workspace/Master/debug/no/simtime/no NZDEV2 5239 20/04/16 00:24:08 Start Master /home/jenkins/workspace/Master/debug/no/simtime/no debug=no simtime=no 3029 20/04/16 00:25:18 End Export /home/jenkins/workspace/Master/debug/no/simtime/no NZOK 105 20/04/16 00:50:23 End Gerit2 /home/jenkins/workspace/Gerrit2 3030 20/04/16 00:50:34 Start Export /home/jenkins/workspace/Gerrit2 NZDEV2 5239 20/04/16 00:50:46 End Master /home/jenkins/workspace/Master/debug/no/simtime/no debug=no simtime=no 5239 20/04/16 00:51:08 Start Master /home/jenkins/workspace/Master/debug/yes/simtime/no debug=yes simtime=no Jenkins and all plugins were upgraded to the latest just last week.
          Hide
          zionyx KY Lee added a comment -

          My current setup:

          Jenkins 2.19.1, 
          Pipeline plugins:
          Pipeline: API 2.4
          Pipeline: Basic Steps 2.2
          Pipeline: Groovy 2.19
          Pipeline: Input Step 2.1
          Pipeline: Job 2.7
          Pipeline: REST API Plugin 2.1
          Pipeline: Stage View Plugin 2.1
          Pipeline: Step API 2.4
          Pipeline: Supporting APIs 2.8

          My pipeline job ignores Block builds when certain jobs are running. It clashes with other ongoing builds on the same node which uses the same folder.

          Before migrating to the pipeline job, blocking mechanism worked perfectly well for us.

          Show
          zionyx KY Lee added a comment - My current setup: Jenkins 2.19.1, Pipeline plugins: Pipeline: API 2.4 Pipeline: Basic Steps 2.2 Pipeline: Groovy 2.19 Pipeline: Input Step 2.1 Pipeline: Job 2.7 Pipeline: REST API Plugin 2.1 Pipeline: Stage View Plugin 2.1 Pipeline: Step API 2.4 Pipeline: Supporting APIs 2.8 My pipeline job ignores Block builds when certain jobs are running . It clashes with other ongoing builds on the same node which uses the same folder. Before migrating to the pipeline job, blocking mechanism worked perfectly well for us.
          Hide
          grahamrhay Graham Hay added a comment -

          This was opt-in in the old world, to quote from the help:

          "When this option is checked, multiple builds of this project may be executed in parallel.
          By default, only a single build of a project is executed at a time — any other requests to start building that project will remain in the build queue until the first build is complete.
          This is a safe default, as projects can often require exclusive access to certain resources, such as a database, or a piece of hardware."

          Removing a "safe default" doesn't seem a good idea; particularly when there is no way to opt out (I don't have an upstream build, but I still only want one running at a time).

          Show
          grahamrhay Graham Hay added a comment - This was opt-in in the old world, to quote from the help: "When this option is checked, multiple builds of this project may be executed in parallel. By default, only a single build of a project is executed at a time — any other requests to start building that project will remain in the build queue until the first build is complete. This is a safe default, as projects can often require exclusive access to certain resources, such as a database, or a piece of hardware." Removing a "safe default" doesn't seem a good idea; particularly when there is no way to opt out (I don't have an upstream build, but I still only want one running at a time).

            People

            • Assignee:
              Unassigned
              Reporter:
              kithril Bruce Fraser
            • Votes:
              12 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated: