Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-6819

Jobs are triggered twice occasionally

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Not A Defect
    • Component/s: core
    • Labels:
      None
    • Environment:
      Master: Hudson ver. 1.363, RHEL Linux 2.6.18-164.el5
      Slaves: RHEL Linux 2.6.9-55.0.2.ELhugemem & Windows XP SP3
    • Similar Issues:

      Description

      After Hudson update from version 1.361 to 1.363 we occasionally see jobs being triggered twice. This seems to happen in both cases, when upstream job triggers downstream job, and remotely triggering a job via curl in a script.

      All of our jobs are free-style jobs.

        Attachments

          Issue Links

            Activity

            Hide
            jcarsique Julien Carsique added a comment -

            We encounter same issue. And when it happens the duplicated jobs' views became unreachable (neither the job view, neither any of both build views). Other views on other jobs work fine.

            Show
            jcarsique Julien Carsique added a comment - We encounter same issue. And when it happens the duplicated jobs' views became unreachable (neither the job view, neither any of both build views). Other views on other jobs work fine.
            Hide
            mindless Alan Harder added a comment -

            can someone post steps to see the problem in a new install of 1.363?

            Show
            mindless Alan Harder added a comment - can someone post steps to see the problem in a new install of 1.363?
            Hide
            abayer Andrew Bayer added a comment -

            For what it's worth, I'm having trouble reproducing it in a fresh install of 1.363 - I've only seen it in upgrades. I think it's not the dependency graph changes - it seems to be a race condition, possibly related to Kohsuke's refactoring of pgweiss's logic for preventing more or less this exact thing from happening.

            Show
            abayer Andrew Bayer added a comment - For what it's worth, I'm having trouble reproducing it in a fresh install of 1.363 - I've only seen it in upgrades. I think it's not the dependency graph changes - it seems to be a race condition, possibly related to Kohsuke's refactoring of pgweiss's logic for preventing more or less this exact thing from happening.
            Hide
            jcarsique Julien Carsique added a comment -

            After downgrading Hudson to 1.362, I can see the two successive builds of same job were run from the same trigger (the builds nuxeo-features-5.3-notests/973/ and 972/ said they were both run from the build nuxeo-jsf-5.3-notests/504).
            These all are Maven jobs, configured to trigger a build when a snapshot dependency is built.

            Show
            jcarsique Julien Carsique added a comment - After downgrading Hudson to 1.362, I can see the two successive builds of same job were run from the same trigger (the builds nuxeo-features-5.3-notests/973/ and 972/ said they were both run from the build nuxeo-jsf-5.3-notests/504). These all are Maven jobs, configured to trigger a build when a snapshot dependency is built.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            On the upstream project (nuxeo-jsf-5.3-notests/504) does it report that it's triggering nuxeo-features-5.3-notests twice?

            Show
            kohsuke Kohsuke Kawaguchi added a comment - On the upstream project (nuxeo-jsf-5.3-notests/504) does it report that it's triggering nuxeo-features-5.3-notests twice?
            Hide
            kohsuke Kohsuke Kawaguchi added a comment - - edited

            abayer: I doubt if the concurrent fix change that you refer to can cause this problem. The problem there is that if you actually have two builds of the same job queued (which requires a parameterization), then it's possible that those two builds end up executed concurrently, even if it's supposed to be prohibited.

            In case of this problem, there should be just one build in the queue.

            Still digging deeper...

            Show
            kohsuke Kohsuke Kawaguchi added a comment - - edited abayer: I doubt if the concurrent fix change that you refer to can cause this problem. The problem there is that if you actually have two builds of the same job queued (which requires a parameterization), then it's possible that those two builds end up executed concurrently, even if it's supposed to be prohibited. In case of this problem, there should be just one build in the queue. Still digging deeper...
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            I withdraw my comment above. I just came up with a hypothesis that causes this problem.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - I withdraw my comment above. I just came up with a hypothesis that causes this problem.
            Hide
            jcarsique Julien Carsique added a comment -

            @kohsuke: nuxeo-jsf-notests/504 says "downstream builds: nuxeo-features-5.3-notests #974-#976", nothing about builds 972 and 973; whereas nuxeo-features-5.3-notests/974 was started by an URL trigger (we've set a hook on Mercurial to trigger a build from URL when there's a commit on a given branch); nuxeo-jsf-notests/503 says "downstream builds: nuxeo-features-5.3-notests #969-#971".

            Show
            jcarsique Julien Carsique added a comment - @kohsuke: nuxeo-jsf-notests/504 says "downstream builds: nuxeo-features-5.3-notests #974-#976", nothing about builds 972 and 973; whereas nuxeo-features-5.3-notests/974 was started by an URL trigger (we've set a hook on Mercurial to trigger a build from URL when there's a commit on a given branch); nuxeo-jsf-notests/503 says "downstream builds: nuxeo-features-5.3-notests #969-#971".
            Hide
            mindless Alan Harder added a comment -

            Changes included in 1.364:

            r32256 | kohsuke | 2010-06-23 17:17:13 -0700 (Wed, 23 Jun 2010)
            Changed paths:
            M /trunk/hudson/main/core/src/main/java/hudson/model/DependencyGraph.java

            [JENKINS-6819] tweaking the implementation a bit so that the code never ever have multiple edges between the same src and dst.

            Not that I was able to spot where the previous code did that, but this implementation is bit more straightforward.

            ------------------------------------------------------------------------

            r32258 | kohsuke | 2010-06-23 18:40:12 -0700 (Wed, 23 Jun 2010)
            Changed paths:
            M /trunk/hudson/main/core/src/main/java/hudson/model/Queue.java
            M /trunk/www/changelog.html

            [FIXED JENKINS-6819] Here is the race condition that induces this problem.

            • Task X becomes buildable
            • Thread E1 enters the pop() method, assigns X to an executor E2.
              Note that at this point X is not removed from the buildables list yet
            • Thread E1 goes to offer.event.block and blocks, losing the lock
            • Thread E3 enters the pop() method, assigns X to anogher executor E4.
              This is possible because X is still in the buildables list.
            • Thread E3 goes to offer.event.block and blocks, losing the lock.
            • Thread E2 wakes up, obtains the lock and execute the rest of the
              pop() method. It decides that it's assigned X, it removes X from
              the buildables list.
            • Thread E2 leaves the pop() method, loses the lock, and starts building X
            • Thread E4 wakes up, obtains the lock, and executes the rest of the
              pop() method. It decides that it's assigned X. It tries to remove X
              from the buildables list. This remove operation becomes no-op,
              but we don't check that.
            • Thread E4 leaves the pop(), and start building X.

            Thus we end up building X twice.

            To avoid this problem, I've introduced the 4th state "pending", and moves BuildableItem from buildables to pendings as soon as we assigned it to the executor. New BuildableItem cannot be created
            while its another copy is either in the buildables or pendings state.

            Show
            mindless Alan Harder added a comment - Changes included in 1.364: r32256 | kohsuke | 2010-06-23 17:17:13 -0700 (Wed, 23 Jun 2010) Changed paths: M /trunk/hudson/main/core/src/main/java/hudson/model/DependencyGraph.java [ JENKINS-6819 ] tweaking the implementation a bit so that the code never ever have multiple edges between the same src and dst. Not that I was able to spot where the previous code did that, but this implementation is bit more straightforward. ------------------------------------------------------------------------ r32258 | kohsuke | 2010-06-23 18:40:12 -0700 (Wed, 23 Jun 2010) Changed paths: M /trunk/hudson/main/core/src/main/java/hudson/model/Queue.java M /trunk/www/changelog.html [FIXED JENKINS-6819] Here is the race condition that induces this problem. Task X becomes buildable Thread E1 enters the pop() method, assigns X to an executor E2. Note that at this point X is not removed from the buildables list yet Thread E1 goes to offer.event.block and blocks, losing the lock Thread E3 enters the pop() method, assigns X to anogher executor E4. This is possible because X is still in the buildables list. Thread E3 goes to offer.event.block and blocks, losing the lock. Thread E2 wakes up, obtains the lock and execute the rest of the pop() method. It decides that it's assigned X, it removes X from the buildables list. Thread E2 leaves the pop() method, loses the lock, and starts building X Thread E4 wakes up, obtains the lock, and executes the rest of the pop() method. It decides that it's assigned X. It tries to remove X from the buildables list. This remove operation becomes no-op, but we don't check that. Thread E4 leaves the pop(), and start building X. Thus we end up building X twice. To avoid this problem, I've introduced the 4th state "pending", and moves BuildableItem from buildables to pendings as soon as we assigned it to the executor. New BuildableItem cannot be created while its another copy is either in the buildables or pendings state.
            Hide
            hhuynh hhuynh added a comment -

            I've upgraded to Hudson 1.364 and all of my projects (Maven) are themselves the upstream project. It leads to a infinite loop a project triggering itself as soon as it's done.

            Show
            hhuynh hhuynh added a comment - I've upgraded to Hudson 1.364 and all of my projects (Maven) are themselves the upstream project. It leads to a infinite loop a project triggering itself as soon as it's done.
            Hide
            hhuynh hhuynh added a comment -

            I've attached a graphic to show a project is calling itself repeately

            Show
            hhuynh hhuynh added a comment - I've attached a graphic to show a project is calling itself repeately
            Hide
            tbingaman Timothy Bingaman added a comment -

            I've had to hack in a workaround in the Ivy plugin for this same issue. The fix should probably be made in core to not allow self-referencing triggers (which was the previous behaviour), but I just wanted to get my builds working again locally.

            svn di -c 32210 ivy
            Index: ivy/src/main/java/hudson/ivy/IvyModule.java
            ===================================================================
            --- ivy/src/main/java/hudson/ivy/IvyModule.java	(revision 32209)
            +++ ivy/src/main/java/hudson/ivy/IvyModule.java	(revision 32210)
            @@ -418,7 +418,8 @@
                             upstream = src;
                         }
             
            -            if (!hasDependency(graph, upstream, downstream))
            +            // Create the build dependency, ignoring self-referencing or already existing deps
            +            if (upstream != downstream && !hasDependency(graph, upstream, downstream))
                             graph.addDependency(new IvyThresholdDependency(upstream, downstream, Result.SUCCESS));
                     }
                 }
            

            The code here is very similar to the Hudson Maven plugin, so the same workaround could probably be applied there in MavenModule.java as a temporary measure.

            I've still had to go back to Hudson version 1.362 anyway though since the locks and latches plugin has been totally broken by these queue scheduling changes and we rely on that heavily as well.

            Show
            tbingaman Timothy Bingaman added a comment - I've had to hack in a workaround in the Ivy plugin for this same issue. The fix should probably be made in core to not allow self-referencing triggers (which was the previous behaviour), but I just wanted to get my builds working again locally. svn di -c 32210 ivy Index: ivy/src/main/java/hudson/ivy/IvyModule.java =================================================================== --- ivy/src/main/java/hudson/ivy/IvyModule.java (revision 32209) +++ ivy/src/main/java/hudson/ivy/IvyModule.java (revision 32210) @@ -418,7 +418,8 @@ upstream = src; } - if (!hasDependency(graph, upstream, downstream)) + // Create the build dependency, ignoring self-referencing or already existing deps + if (upstream != downstream && !hasDependency(graph, upstream, downstream)) graph.addDependency(new IvyThresholdDependency(upstream, downstream, Result.SUCCESS)); } } The code here is very similar to the Hudson Maven plugin, so the same workaround could probably be applied there in MavenModule.java as a temporary measure. I've still had to go back to Hudson version 1.362 anyway though since the locks and latches plugin has been totally broken by these queue scheduling changes and we rely on that heavily as well.
            Hide
            hhuynh hhuynh added a comment -

            Could we please resolve this bug? It's quite a serious blocker

            Show
            hhuynh hhuynh added a comment - Could we please resolve this bug? It's quite a serious blocker
            Hide
            mwebber Matthew Webber added a comment -

            I have seen the problem in 1.372.

            Show
            mwebber Matthew Webber added a comment - I have seen the problem in 1.372.
            Hide
            ssbarnea Sorin Sbarnea added a comment -

            I observed the same behavior and I think I know why this is happening. Double clicking in browser can generate this, it can happen more often than you may think.

            A simple solution for this would be prevent this from happening, how about http://greatwebguy.com/programming/dom/prevent-double-submit-with-jquery/ ?

            This happened once to me while presenting hudson to a first time user. When I told her to start a new job she asked me why there are two running jobs. Please do not count this as an usage issue

            Show
            ssbarnea Sorin Sbarnea added a comment - I observed the same behavior and I think I know why this is happening. Double clicking in browser can generate this, it can happen more often than you may think. A simple solution for this would be prevent this from happening, how about http://greatwebguy.com/programming/dom/prevent-double-submit-with-jquery/ ? This happened once to me while presenting hudson to a first time user. When I told her to start a new job she asked me why there are two running jobs. Please do not count this as an usage issue
            Hide
            tiainpa tiainpa added a comment -

            I'm quite sure that double clicking is not the issue for us, because we are not clicking anything manually.

            We are starting the chain of jobs periodically from one job, and from the first job onwards everything is done either using the "build other jobs" option of Hudson or remotely triggering the jobs using wget / curl.

            Show
            tiainpa tiainpa added a comment - I'm quite sure that double clicking is not the issue for us, because we are not clicking anything manually. We are starting the chain of jobs periodically from one job, and from the first job onwards everything is done either using the "build other jobs" option of Hudson or remotely triggering the jobs using wget / curl.
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            With modern versions of Jenkins, the build console output reports the list of causes that triggered a build. For those who are experiencing double invocations, please report what you see there in two (or more) builds that are fired.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - With modern versions of Jenkins, the build console output reports the list of causes that triggered a build. For those who are experiencing double invocations, please report what you see there in two (or more) builds that are fired.
            Hide
            tiainpa tiainpa added a comment -

            We just recently updated our Hudson from version 1.362 to Jenkins 1.400 through the manage section, and we haven't seen this issue anymore for some reason.

            We are just glad we were finally able to update!

            Show
            tiainpa tiainpa added a comment - We just recently updated our Hudson from version 1.362 to Jenkins 1.400 through the manage section, and we haven't seen this issue anymore for some reason. We are just glad we were finally able to update!
            Hide
            fredg Fred G added a comment -

            Issue seems to be fixed with newer versions of Jenkins. Please re-open if there are still problems.

            Show
            fredg Fred G added a comment - Issue seems to be fixed with newer versions of Jenkins. Please re-open if there are still problems.
            Hide
            accmunsayac Christian Munsayac added a comment -

            This is also hapenning in 2.15 version. Jobs builds twice using poll scm cron or manually clicking the build. The job executes twice

            Show
            accmunsayac Christian Munsayac added a comment - This is also hapenning in 2.15 version. Jobs builds twice using poll scm cron or manually clicking the build. The job executes twice
            Hide
            zardoz Luis Panadero Guardeño added a comment - - edited

            Happening on 2.7.4 . I'm trying to update git plugins to see if fix this . We aren't using SCM polling. It's a task scheduled to be executed only one time per day, that was working fine before we change the git branch that is checked. It's very annoying as it's a heavy task that enters on a infinite loop. The duplicated jobs shows that Jenkins detected changes on the repository, but not show any change.

            UPDATE : Updating the git plug-in don't fix this.

            Show
            zardoz Luis Panadero Guardeño added a comment - - edited Happening on 2.7.4 . I'm trying to update git plugins to see if fix this . We aren't using SCM polling. It's a task scheduled to be executed only one time per day, that was working fine before we change the git branch that is checked. It's very annoying as it's a heavy task that enters on a infinite loop. The duplicated jobs shows that Jenkins detected changes on the repository, but not show any change. UPDATE : Updating the git plug-in don't fix this.
            Hide
            zardoz Luis Panadero Guardeño added a comment - - edited

            Looks that gets fixed if I clean the workspace. I would say tomorrow.

            Show
            zardoz Luis Panadero Guardeño added a comment - - edited Looks that gets fixed if I clean the workspace. I would say tomorrow.
            Hide
            t_westling Tomas Westling added a comment -

            I have seen this happen in Jenkins 1.651.3, however, only when the Jenkins master is under heavy CPU load.
            Normally, everything works as it should, but on certain times when the Jenkins master CPU is running on max, we see two downstream
            builds of one project started, both builds pointing to the same parent build.
            This is done using the conditional build step + parameterized trigger and a schedule for the parent project.

            Show
            t_westling Tomas Westling added a comment - I have seen this happen in Jenkins 1.651.3, however, only when the Jenkins master is under heavy CPU load. Normally, everything works as it should, but on certain times when the Jenkins master CPU is running on max, we see two downstream builds of one project started, both builds pointing to the same parent build. This is done using the conditional build step + parameterized trigger and a schedule for the parent project.
            Hide
            ssbarnea Sorin Sbarnea added a comment -

            This is still happening with most recent versions of Jenkins, like Jenkins ver. 2.46.1

            I can easily double tick the build button, especial on a MBP with touchpad, doing a double click is so easy.

            Show
            ssbarnea Sorin Sbarnea added a comment - This is still happening with most recent versions of Jenkins, like  Jenkins ver. 2.46.1 I can easily double tick the build button, especial on a MBP with touchpad, doing a double click is so easy.
            Hide
            sangeetawakhale sangeeta wakhale added a comment -

            I am also facing this issue with Jenkins ver. 2.46.1. I have a upstream job which is triggering a downstream job.

            post build actions>Build other projects>The downstream job is getting triggered twice.

            Also post build actions>Trigger parameterized build on other projects>The downstream job is getting triggered twice.

            Show
            sangeetawakhale sangeeta wakhale added a comment - I am also facing this issue with  Jenkins ver. 2.46.1 . I have a upstream job which is triggering a downstream job. post build actions>Build other projects>The downstream job is getting triggered twice. Also post build actions>Trigger parameterized build on other projects>The downstream job is getting triggered twice.
            Hide
            sangeetawakhale sangeeta wakhale added a comment -

            This issue is fixed for me by doing below:

            Define the downstream job name in upstream job using either Build other projects OR Trigger parameterized build on other projects.

            Removed the upstream job name in downstream job defined by Build Triggers>Build after other projects are built

             

            Show
            sangeetawakhale sangeeta wakhale added a comment - This issue is fixed for me by doing below: Define the downstream job name in upstream job using either Build other projects OR Trigger parameterized build on other projects. Removed the upstream job name in downstream job defined by Build Triggers>Build after other projects are built  
            Hide
            suni sunitha prabhu added a comment -

            I still see this issue. Do we have any workaround for this?  @Sangeeta - I don't understand your workaround.  can you please explain in detail

            Show
            suni sunitha prabhu added a comment - I still see this issue. Do we have any workaround for this?  @Sangeeta - I don't understand your workaround.  can you please explain in detail
            Hide
            shiluveru Bharath Shiluveru added a comment -

            I'm also seeing the same issue. 

            updating issue description for others:

            Downstream job is getting invoked twice when 'Trigger parameterized build on other projects' option is selected in upstream project.

            If anybody found a solution or workaround to mitigate this, please post.

            Show
            shiluveru Bharath Shiluveru added a comment - I'm also seeing the same issue.  updating issue description for others: Downstream job is getting invoked twice when ' Trigger parameterized build on other projects ' option is selected in upstream project. If anybody found a solution or workaround to mitigate this, please post.
            Hide
            shiluveru Bharath Shiluveru added a comment -

            I have tried steps mentioned by  sangeeta wakhale , this got resolved.

            Thanks !

            Show
            shiluveru Bharath Shiluveru added a comment - I have tried steps mentioned by   sangeeta wakhale , this got resolved. Thanks !
            Hide
            samt Sam Talebbeik added a comment -

            Thanks sangeeta wakhale for the tip. Your workaround fixed the issue.

            My "upstream job" was triggering a "downstream job" twice all the time using "Trigger parametrized build on other projects" plugin. My Jenkins version is 2.73.2

            I had to go to the "downstream job" -> Build Trigger -> remove "Build after other projects are built" 

            Show
            samt Sam Talebbeik added a comment - Thanks sangeeta wakhale for the tip. Your workaround fixed the issue. My "upstream job" was triggering a "downstream job" twice all the time using "Trigger parametrized build on other projects" plugin. My Jenkins version is 2.73.2 I had to go to the "downstream job" -> Build Trigger -> remove "Build after other projects are built" 
            Hide
            slide_o_mix Alex Earl added a comment -

            It looks like this was mainly improper upstream/downstream job relationships that were defined.

            Show
            slide_o_mix Alex Earl added a comment - It looks like this was mainly improper upstream/downstream job relationships that were defined.
            Hide
            jayanmn Jayan MN added a comment -

            I was searching for another issue but this masked it. Adding a link if that helps someone : https://issues.jenkins-ci.org/browse/JENKINS-58296  Projects are triggered twice when using File Parameter and Firefox 67

            Show
            jayanmn Jayan MN added a comment - I was searching for another issue but this masked it. Adding a link if that helps someone :  https://issues.jenkins-ci.org/browse/JENKINS-58296   Projects are triggered twice when using File Parameter and Firefox 67

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                tiainpa tiainpa
              • Votes:
                22 Vote for this issue
                Watchers:
                27 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: