Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-7033

Job in build queue is not executed

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: batch-task-plugin
    • Labels:
      None
    • Environment:
      Hudson ver. 1.393, matrix project, matrixtieparent plugin, post-build-batch-task plugin
    • Similar Issues:

      Description

      I have a matrix project that completely builds on a slave node ("Build on multiple nodes", "Tie parent build to a node" vie matrixtieparent plugin). As a "Post-build Action", "Invoke batch tasks" is configured to run a batch task of the same project. Usually, this works fine and the batch task is execured after the matrix configurations are build. However, when the salve node becomes idle, the bach task is still in the build queue and stays there forever with "pending - Waiting for next available executor on MySlaveNode").
      When a new job is added to the build queue, the pending job is finally executed.
      This usually happen when may builds are in the queue. However, we also see it when hudson is idle and we start one project build manually.

        Attachments

          Issue Links

            Activity

            Hide
            mindless Alan Harder added a comment -

            Can you find/post steps to reproduce the issue? How are the slave nodes configured?

            Show
            mindless Alan Harder added a comment - Can you find/post steps to reproduce the issue? How are the slave nodes configured?
            Hide
            axelheider Axel Heider added a comment -

            Unfortunately, we have not found a way to repoduce this every time. However, there is a high change that this happen when I start a jon manually with master and slave idle.

            Configuration:
            Master: Windows, 4 executors
            Slave: Linux, 2 executors, launched via SSH, "use exclusive for tied jobs, keep slave online as much as possible.

            Show
            axelheider Axel Heider added a comment - Unfortunately, we have not found a way to repoduce this every time. However, there is a high change that this happen when I start a jon manually with master and slave idle. Configuration: Master: Windows, 4 executors Slave: Linux, 2 executors, launched via SSH, "use exclusive for tied jobs, keep slave online as much as possible.
            Hide
            axelheider Axel Heider added a comment -

            I wonder that I'm the only one seeing this happen - and it appears do be reproducible every time by now.

            Show
            axelheider Axel Heider added a comment - I wonder that I'm the only one seeing this happen - and it appears do be reproducible every time by now.
            Hide
            axelheider Axel Heider added a comment - - edited

            Now I have found a way to reproduce this always:

            • Configuratuon: Windows Master, Linux Salve
            • Matrix Job triggered by SVN commit runs fully on Linux Slave (also the parent vie matrix-tie-parent plugin)
            • Job has one Batch task configured
            • this batch Task is triggered as post build action (via "Invoke batch tasks")

            When I start the build job manually, all matrix builds run and the batch task is put in the queue. But then the batch task does not run, it stays in thew queue. And it stays in the queue until another job arrives in the queue.

            So, is it a problem with the post-build-Invoke-batch-task-plugin and the way it adds a job? It appears the problem always occurs for job on slaves, it occurs rarely for jobs on the master node.

            Show
            axelheider Axel Heider added a comment - - edited Now I have found a way to reproduce this always: Configuratuon: Windows Master, Linux Salve Matrix Job triggered by SVN commit runs fully on Linux Slave (also the parent vie matrix-tie-parent plugin) Job has one Batch task configured this batch Task is triggered as post build action (via "Invoke batch tasks") When I start the build job manually, all matrix builds run and the batch task is put in the queue. But then the batch task does not run, it stays in thew queue. And it stays in the queue until another job arrives in the queue. So, is it a problem with the post-build-Invoke-batch-task-plugin and the way it adds a job? It appears the problem always occurs for job on slaves, it occurs rarely for jobs on the master node.
            Hide
            axelheider Axel Heider added a comment -

            Problem seen with Hudson 1.393 and 1.394 in Windows Master today. A matrix project building on the windows mater is configured to run 3 batch jobs after building. All three job remain in the build queue without getting executed.

            Work around is now that an empty dummy project is build every 10 minutes so the build queue gets flushed.

            Show
            axelheider Axel Heider added a comment - Problem seen with Hudson 1.393 and 1.394 in Windows Master today. A matrix project building on the windows mater is configured to run 3 batch jobs after building. All three job remain in the build queue without getting executed. Work around is now that an empty dummy project is build every 10 minutes so the build queue gets flushed.
            Hide
            axelheider Axel Heider added a comment -

            Problem still exists in 1.396

            Show
            axelheider Axel Heider added a comment - Problem still exists in 1.396
            Hide
            axelheider Axel Heider added a comment -

            Todays I've seen another strange issue, however I canno reproduce it. I've set up a Dummy job that is triggered by many matrix jobs when they are done. This is supposed to add something to the build queue, so the neglected batch tasks from the matrix jobs waiting there finally get triggered. Now the Dummy job is also waiting in the build queue with the timeout/clock symbol. All executors are available but nothing happens. If I start the Dummy job manually, nothing happens, too. I have to add a new job to the queue to get it finally flushed to the executors.

            Please have a review on the build queue handling.

            Show
            axelheider Axel Heider added a comment - Todays I've seen another strange issue, however I canno reproduce it. I've set up a Dummy job that is triggered by many matrix jobs when they are done. This is supposed to add something to the build queue, so the neglected batch tasks from the matrix jobs waiting there finally get triggered. Now the Dummy job is also waiting in the build queue with the timeout/clock symbol. All executors are available but nothing happens. If I start the Dummy job manually, nothing happens, too. I have to add a new job to the queue to get it finally flushed to the executors. Please have a review on the build queue handling.
            Hide
            axelheider Axel Heider added a comment -

            Have not seen this for a long time now, assume it was some internal race condition that has ben resolved.

            Show
            axelheider Axel Heider added a comment - Have not seen this for a long time now, assume it was some internal race condition that has ben resolved.
            Hide
            ssbarnea Sorin Sbarnea added a comment -

            I do have the problem on Jenkins 1.518 running on Windows and one specific job started to behave like this.

            SCM pollling (git) finds changes and adds the job to the queue. The queue is configured to wait for 300 seconds before building and the polling is configured at every 2 minutes.

            The job ends-up always in the queue, never building but if I manually try to build it it will work.

            C:\JenkinsWar>java -DJENKINS_HOME=C:\Jenkins -jar jenkins.war
            Running from: C:\JenkinsWar\jenkins.war
            webroot: System.getProperty("JENKINS_HOME")
            Jun 20, 2013 11:20:44 AM winstone.Logger logInternal
            INFO: Beginning extraction from war file
            Jenkins home directory: C:\Jenkins found at: System.getProperty("JENKINS_HOME")
            Jun 20, 2013 11:20:53 AM winstone.Logger logInternal
            INFO: HTTP Listener started: port=8080
            Jun 20, 2013 11:20:53 AM winstone.Logger logInternal
            INFO: Winstone Servlet Engine v0.9.10 running: controlPort=disabled
            Jun 20, 2013 11:20:55 AM jenkins.InitReactorRunner$1 onAttained
            INFO: Started initialization
            Jun 20, 2013 11:20:56 AM hudson.ClassicPluginStrategy createPluginWrapper
            INFO: Plugin translation.jpi is disabled
            Jun 20, 2013 11:20:56 AM jenkins.InitReactorRunner$1 onAttained
            INFO: Listed all plugins
            Jun 20, 2013 11:20:56 AM hudson.plugins.greenballs.PluginImpl start
            INFO: Green Balls!
            Jun 20, 2013 11:20:56 AM jenkins.InitReactorRunner$1 onAttained
            INFO: Prepared all plugins
            Jun 20, 2013 11:21:01 AM jenkins.InitReactorRunner$1 onAttained
            INFO: Started all plugins
            Jun 20, 2013 11:21:15 AM jenkins.InitReactorRunner$1 onAttained
            INFO: Augmented all extensions
            Jun 20, 2013 11:21:17 AM jenkins.InitReactorRunner$1 onAttained
            INFO: Loaded all jobs
            Jun 20, 2013 11:21:19 AM org.jenkinsci.main.modules.sshd.SSHD start
            INFO: Started SSHD at port 49165
            Jun 20, 2013 11:21:19 AM jenkins.InitReactorRunner$1 onAttained
            INFO: Completed initialization
            Jun 20, 2013 11:21:19 AM hudson.TcpSlaveAgentListener <init>
            INFO: JNLP slave agent listener started on TCP port 49166
            Jun 20, 2013 11:21:19 AM hudson.WebAppMain$2 run
            INFO: Jenkins is fully up and running
            Jun 20, 2013 11:21:53 AM hudson.triggers.SCMTrigger$Runner run
            INFO: SCM changes detected in carbon_trunk_dotnet-packages. Triggering  #58
            Jun 20, 2013 11:23:51 AM hudson.triggers.SCMTrigger$Runner run
            INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th
            e queue
            Jun 20, 2013 11:25:51 AM hudson.triggers.SCMTrigger$Runner run
            INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th
            e queue
            Jun 20, 2013 11:27:51 AM hudson.triggers.SCMTrigger$Runner run
            INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th
            e queue
            Jun 20, 2013 11:29:51 AM hudson.triggers.SCMTrigger$Runner run
            INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th
            e queue
            Jun 20, 2013 11:31:51 AM hudson.triggers.SCMTrigger$Runner run
            INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th
            e queue
            Jun 20, 2013 11:33:51 AM hudson.triggers.SCMTrigger$Runner run
            INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th
            e queue
            Jun 20, 2013 11:35:51 AM hudson.triggers.SCMTrigger$Runner run
            INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th
            e queue
            
            Show
            ssbarnea Sorin Sbarnea added a comment - I do have the problem on Jenkins 1.518 running on Windows and one specific job started to behave like this. SCM pollling (git) finds changes and adds the job to the queue. The queue is configured to wait for 300 seconds before building and the polling is configured at every 2 minutes. The job ends-up always in the queue, never building but if I manually try to build it it will work. C:\JenkinsWar>java -DJENKINS_HOME=C:\Jenkins -jar jenkins.war Running from: C:\JenkinsWar\jenkins.war webroot: System .getProperty( "JENKINS_HOME" ) Jun 20, 2013 11:20:44 AM winstone.Logger logInternal INFO: Beginning extraction from war file Jenkins home directory: C:\Jenkins found at: System .getProperty( "JENKINS_HOME" ) Jun 20, 2013 11:20:53 AM winstone.Logger logInternal INFO: HTTP Listener started: port=8080 Jun 20, 2013 11:20:53 AM winstone.Logger logInternal INFO: Winstone Servlet Engine v0.9.10 running: controlPort=disabled Jun 20, 2013 11:20:55 AM jenkins.InitReactorRunner$1 onAttained INFO: Started initialization Jun 20, 2013 11:20:56 AM hudson.ClassicPluginStrategy createPluginWrapper INFO: Plugin translation.jpi is disabled Jun 20, 2013 11:20:56 AM jenkins.InitReactorRunner$1 onAttained INFO: Listed all plugins Jun 20, 2013 11:20:56 AM hudson.plugins.greenballs.PluginImpl start INFO: Green Balls! Jun 20, 2013 11:20:56 AM jenkins.InitReactorRunner$1 onAttained INFO: Prepared all plugins Jun 20, 2013 11:21:01 AM jenkins.InitReactorRunner$1 onAttained INFO: Started all plugins Jun 20, 2013 11:21:15 AM jenkins.InitReactorRunner$1 onAttained INFO: Augmented all extensions Jun 20, 2013 11:21:17 AM jenkins.InitReactorRunner$1 onAttained INFO: Loaded all jobs Jun 20, 2013 11:21:19 AM org.jenkinsci.main.modules.sshd.SSHD start INFO: Started SSHD at port 49165 Jun 20, 2013 11:21:19 AM jenkins.InitReactorRunner$1 onAttained INFO: Completed initialization Jun 20, 2013 11:21:19 AM hudson.TcpSlaveAgentListener <init> INFO: JNLP slave agent listener started on TCP port 49166 Jun 20, 2013 11:21:19 AM hudson.WebAppMain$2 run INFO: Jenkins is fully up and running Jun 20, 2013 11:21:53 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Triggering #58 Jun 20, 2013 11:23:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue Jun 20, 2013 11:25:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue Jun 20, 2013 11:27:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue Jun 20, 2013 11:29:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue Jun 20, 2013 11:31:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue Jun 20, 2013 11:33:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue Jun 20, 2013 11:35:51 AM hudson.triggers.SCMTrigger$Runner run INFO: SCM changes detected in carbon_trunk_dotnet-packages. Job is already in th e queue

              People

              • Assignee:
                kohsuke Kohsuke Kawaguchi
                Reporter:
                axelheider Axel Heider
              • Votes:
                3 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: