Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46503

Pipeline hanging after updating Jenkinsfile but before starting to build

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Duplicate
    • Component/s: jira-plugin
    • Environment:
      Debian 8.8. Jenkins 2.60. Tomcat 8.0.14.
      Pipeline 2.5, Pipeline: Groovy 2.38, Git plugin 3.5.1, Lockable Resources 2.0.
    • Similar Issues:

      Description

      We have a linux master (4 executors, never full) running a number of pipeline jobs that all pull Jenkinsfile from git before triggering the actual build on one of a pool of a half dozen workers (osx, linux, and windows). Most builds (dozens a day) work fine, but occasionally (1-3 times a week) things will hang before actually starting to run any Groovy.

      This has been happening for months now, and various packages have been upgraded in response to no avail.

      Initially, I thought that the issue was network connectivity to the git server - but I have since pulled the pipeline script onto a local file:// repo and experience the same issue.

      When affected, builds look like this:

      Started by upstream project "Project/Project-Sync" build number 1682
      originally caused by:
       Started by an SCM change
      Checking out git file:///var/lib/jenkins/git-cache into /var/lib/jenkins/workspace/Project/Project-WIN@script to read Jenkinsfile
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
       > git config remote.origin.url file:///var/lib/jenkins/git-cache # timeout=10
      Fetching upstream changes from file:///var/lib/jenkins/git-cache
       > git --version # timeout=10
       > git fetch --tags --progress file:///var/lib/jenkins/git-cache +refs/heads/*:refs/remotes/origin/*
       > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
       > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
      Checking out Revision fd354d30a6b141d6fc81267d4708de10de5b5966 (refs/remotes/origin/master)
      Commit message: "Typo."
       > git config core.sparsecheckout # timeout=10
       > git checkout -f fd354d30a6b141d6fc81267d4708de10de5b5966
       > git rev-list fd354d30a6b141d6fc81267d4708de10de5b5966 # timeout=10

      And then they just sit there like that forever.

      A good goes more like this:

      Started by upstream project "Project/Project-Sync" build number 1684
      originally caused by:
      Started by user Ammon Lauritzen
      ...
      Checking out Revision fd354d30a6b141d6fc81267d4708de10de5b5966 (refs/remotes/origin/master)
      Commit message: "Typo."
      > git config core.sparsecheckout # timeout=10
      > git checkout -f fd354d30a6b141d6fc81267d4708de10de5b5966
      > git rev-list fd354d30a6b141d6fc81267d4708de10de5b5966 # timeout=10
      [Pipeline] node
      Running on master in /var/lib/jenkins/workspace/Project/Project-WIN
      [Pipeline] {
      [Pipeline] echo
      prepping master environment
      [Pipeline] sh
      [Project-WIN] Running shell script
      ...

      The only difference in the log is that it stops before the [Pipeline] lines start logging.

      The build cannot be cancelled normally via the UI, by restarting Tomcat, or by rebooting the server - I have to instead abort it via the script console.

      It happens more often with nightly builds than SCM-triggered CI builds, and more often on the weekends... but it happens plenty of times in the middle of the day as well. It is not always the same projects that hang, and not always projects destined for Windows slaves that hang (that was just the most recent instance of the problem).

        Attachments

          Issue Links

            Activity

            Hide
            markewaite Mark Waite added a comment - - edited

            Sverre Moe refer to JENKINS-48357 for details of the impact of Jira plugin 2.5.1. This bug was reported before the release of Jira plugin 2.5.1. I suspect that Jira plugin 2.5.1 related bugs are a "side path" from this bug. The real bug is likely still in the system as reported originally.

            Show
            markewaite Mark Waite added a comment - - edited Sverre Moe refer to JENKINS-48357 for details of the impact of Jira plugin 2.5.1. This bug was reported before the release of Jira plugin 2.5.1. I suspect that Jira plugin 2.5.1 related bugs are a "side path" from this bug. The real bug is likely still in the system as reported originally.
            Hide
            michaelneale Michael Neale added a comment -

            Mark Waite I hit this too - I am trying downgrade to 2.5.0 and will report back if it is all ok after that. This may be a dupe of https://issues.jenkins-ci.org/browse/JENKINS-43106 but I put a support bundle there. in what log did people see the JIRA error? 

             

            I swear that JIRA plugin (and the Atlassian dependencies) are a disaster. 

            Show
            michaelneale Michael Neale added a comment - Mark Waite I hit this too - I am trying downgrade to 2.5.0 and will report back if it is all ok after that. This may be a dupe of https://issues.jenkins-ci.org/browse/JENKINS-43106  but I put a support bundle there. in what log did people see the JIRA error?    I swear that JIRA plugin (and the Atlassian dependencies) are a disaster. 
            Hide
            djviking Sverre Moe added a comment -

            We have also downgraded JIRA plugin to 2.5.0 and it seems to have resolved the problem.

            Show
            djviking Sverre Moe added a comment - We have also downgraded JIRA plugin to 2.5.0 and it seems to have resolved the problem.
            Hide
            svanoort Sam Van Oort added a comment -

            Mark Waite I'm downgrading the priority to reflect the fix via JIRA... not clear if this should stay as a pipeline bug though or if we can close it out.

            Show
            svanoort Sam Van Oort added a comment - Mark Waite I'm downgrading the priority to reflect the fix via JIRA... not clear if this should stay as a pipeline bug though or if we can close it out.
            Hide
            carltongbrown Carlton Brown added a comment -

            Seeing the same issue here... no luck with the workarounds around downgrading the JRIA plugin

            Show
            carltongbrown Carlton Brown added a comment - Seeing the same issue here... no luck with the workarounds around downgrading the JRIA plugin

              People

              • Assignee:
                Unassigned
                Reporter:
                allaryin Ammon Lauritzen
              • Votes:
                7 Vote for this issue
                Watchers:
                21 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: