Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55220

Multibranch Pipeline jobs get randomly lost

    Details

    • Similar Issues:

      Description

      We are running different Multibranch Pipelines with git (Bitbucket) since almost 2 years now.

      Unfortunately jobs (aka branches) do disappear randomly. It feels like the the number of losses is increasing and our workaround (see first comment) is too complicated to keep up.

      Even worse it looks like develop and master are the most likely to disappear.

      We are not sure what causes the losses. It feels pretty randomly.

      The Scan Multibranch Pipeline Log says Detected unsupported subitem:

      [Mon Dec 17 10:28:18 CET 2018] Starting branch indexing...

      Connecting to https://bitbucket.org using *****

      Repository type: Git

      Looking up sj/sj-mobile-ios for pull requests

       

        0 pull requests were processed

      Connecting to https://bitbucket.org using ******

      Repository type: Git

      Looking up sj/sj-mobile-ios for branches

      Checking branch develop from sj/sj-mobile-ios

            'Jenkinsfile' found

          Met criteria

      Detected unsupported subitem sj-mobile-ios » develop, skipping

      Checking branch master from sj/sj-mobile-ios

            'Jenkinsfile' found

          Met criteria

      Detected unsupported subitem sj-mobile-ios » master, skipping

      Checking branch stash/iapPromotion from sj/sj-mobile-ios

            'Jenkinsfile' not found

          Skipped

       

        3 branches were processed

      Looking up sj/sj-mobile-ios for pull requests

       

        0 pull requests were processed

      [Mon Dec 17 10:28:21 CET 2018] Finished branch indexing. Indexing took 3.4 sec

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@65a9d08a[sj-ios/ffffeeeea.j2i0s0od1eao.innnntttt], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@4f2f2fc0[sj-ios/rrrreeeel.vled0mqc8i7t.2----3333], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@72d388bb[sj-ios/rrrreeeel.r1m7pv7ujr6o.5----1111], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@29a3274d[sj-ios/rrrreeeel.tgai4liiqai5.3----1111], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@63eb56a6[sj-ios/ffffeeeea.fqa397lnq84t.aggggyyyy], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@2548a24e[sj-ios/rrrreeeel.0d0a3oaeksil.5----2222], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@52e05031[sj-ios/PR-130], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@69b325d6[sj-ios/rrrreeeel.2s6jniopvdin.1----7777], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@1ffba7d3[sj-ios/rrrreeeel.c8v76qkpj7f5.1----4444], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@1c6e3ff6[sj-ios/rrrreeeel.tndhtl2k8ai1.1----6666], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@734db299[sj-ios/rrrreeeel.9dgfoipjl376.2----2222], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@f3b27ec[sj-ios/PR-129], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@22f40536[sj-ios/rrrreeeel.i2j4g40joneo.1----5555], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@2928d02b[sj-ios/ffffeeeea.i5k5i2srbpsp.nkkkkssss], skipping

      Detected unsupported subitem org.jenkinsci.plugins.workflow.job.WorkflowJob@5795381d[sj-ios/rrrreeeel.akouhq90ivkl.1----3333], skipping

      Finished: SUCCESS

       

      You can still see the job if you navigate to /project/job/develop/ but its not working anymore. Somehow the config.xml gets reseted to

      <?xml version='1.1' encoding='UTF-8'?>
      <flow-definition plugin="workflow-job@2.30">
      <actions/>
      <keepDependencies>false</keepDependencies>
      <properties/>
      <triggers/>
      <disabled>false</disabled>
      </flow-definition>

       

      We are thankful for your help.

       

       

      We are running Jenkins on AWS Bitnami Installation on Apache Tomcat Version 8.0.47

      See also screenshots of Project Configuration and Bitbuckets Webhooks.

      OS: 

      os.name Linux
      os.version 4.9.0-5-amd64

      VM:

      java.vendor Oracle Corporation
      java.vendor.url http://java.oracle.com/
      java.vendor.url.bug http://bugreport.sun.com/bugreport/
      java.version 1.8.0_151
      java.vm.info mixed mode
      java.vm.name Java HotSpot(TM) 64-Bit Server VM
      java.vm.specification.name Java Virtual Machine Specification
      java.vm.specification.vendor Oracle Corporation
      java.vm.specification.version 1.8
      java.vm.vendor Oracle Corporation
      java.vm.version 25.151-b12

       

      Installed Plugins:

      ace-editor 1.1 true
      amazon-ecr 1.6 true
      amazon-ecs 1.18 true
      analysis-core 1.95 true
      ansicolor 0.5.3 true
      ant 1.9 true
      antisamy-markup-formatter 1.5 true
      apache-httpcomponents-client-4-api 4.5.5-3.0 true
      authentication-tokens 1.3 true
      aws-credentials 1.24 true
      aws-java-sdk 1.11.457 true
      bitbucket 1.1.8 true
      bitbucket-build-status-notifier 1.4.1 true
      bitbucket-scm-trait-commit-skip 0.1.1 true
      bouncycastle-api 2.17 true
      branch-api 2.1.2 true
      build-monitor-plugin 1.12+build.201809061734 true
      build-timeout 1.19 true
      checkstyle 3.50 true
      cloudbees-bitbucket-branch-source 2.2.15 true
      cloudbees-folder 6.7 true
      cobertura 1.13 true
      code-coverage-api 1.0.5 true
      command-launcher 1.2 true
      copyartifact 1.41 true
      credentials 2.1.18 true
      credentials-binding 1.17 true
      display-url-api 2.3.0 true
      docker-build-publish 1.3.2 true
      docker-commons 1.13 true
      docker-workflow 1.17 true
      dry 2.50 true
      durable-task 1.28 true
      email-ext 2.63 true
      external-monitor-job 1.7 true
      git 3.9.1 true
      git-client 2.7.4 true
      git-parameter 0.9.6 true
      git-server 1.7 true
      github 1.29.3 true
      github-api 1.95 true
      github-branch-source 2.4.1 true
      github-organization-folder 1.6 true
      gradle 1.29 true
      handlebars 1.1.1 true
      handy-uri-templates-2-api 2.1.6-1.0 true
      icon-shim 2.0.3 true
      jackson2-api 2.9.7.1 true
      javadoc 1.4 true
      jdk-tool 1.2 true
      jquery 1.12.4-0 true
      jquery-detached 1.2.1 true
      jsch 0.1.54.2 true
      junit 1.26.1 true
      ldap 1.20 true
      locale 1.4 true
      lockable-resources 2.3 true
      log-parser 2.0 true
      mailer 1.22 true
      mapdb-api 1.0.9.0 true
      matrix-auth 2.3 true
      matrix-project 1.13 true
      maven-plugin 3.2 true
      mercurial 2.4 true
      momentjs 1.1.1 true
      multiple-scms 0.6 true
      notification 1.13 true
      pam-auth 1.4 true
      pipeline-aws 1.35 true
      pipeline-build-step 2.7 true
      pipeline-github-lib 1.0 true
      pipeline-graph-analysis 1.9 true
      pipeline-input-step 2.8 true
      pipeline-milestone-step 1.3.1 true
      pipeline-model-api 1.3.3 true
      pipeline-model-declarative-agent 1.1.1 true
      pipeline-model-definition 1.3.3 true
      pipeline-model-extensions 1.3.3 true
      pipeline-rest-api 2.10 true
      pipeline-stage-step 2.3 true
      pipeline-stage-tags-metadata 1.3.3 true
      pipeline-stage-view 2.10 true
      pipeline-utility-steps 2.2.0 true
      plain-credentials 1.4 true
      pmd 3.50 true
      pyenv-pipeline 2.0.1 true
      resource-disposer 0.12 true
      s3 0.11.2 true
      scm-api 2.3.0 true
      script-security 1.49 true
      sloccount 1.24 true
      ssh-agent 1.17 true
      ssh-credentials 1.14 true
      ssh-slaves 1.29.1 true
      structs 1.17 true
      subversion 2.12.1 true
      timestamper 1.8.10 true
      token-macro 2.5 true
      trilead-api 1.0.1 true
      warnings 4.68 true
      windows-slaves 1.3.1 true
      workflow-aggregator 2.6 true
      workflow-api 2.33 true
      workflow-basic-steps 2.13 true
      workflow-cps 2.61 true
      workflow-cps-global-lib 2.12 true
      workflow-durable-task-step 2.26 true
      workflow-job 2.30 true
      workflow-multibranch 2.20 true
      workflow-scm-step 2.7 true
      workflow-step-api 2.17 true
      workflow-support 2.23 true
      ws-cleanup 0.36 true

        Attachments

          Activity

          Hide
          stikus Konstantin Grammatikati added a comment -

          Dave Nichols

          We are having random branches disappearance in our Jenkins too (Jenkins ver. 2.190.3 standalone on CentOS machine).

          This is screenshot from Scan GitLab Project Log:

          But we have master branch and Jenkinsfile there. I don't know why Jenkins counting this branch as orphaned and don't see anything in logs and in Multibranch pipeline events section.

           

          Show
          stikus Konstantin Grammatikati added a comment - Dave Nichols We are having random branches disappearance in our Jenkins too (Jenkins ver. 2.190.3 standalone on CentOS machine). This is screenshot from  Scan GitLab Project Log : But we have master branch and Jenkinsfile there. I don't know why Jenkins counting this branch as orphaned and don't see anything in logs and in Multibranch pipeline events section.  
          Hide
          josh_watson Josh Watson added a comment - - edited

          Also getting this with Jenkins running on Centos 7, using nginx as a proxy. The jobs are triggered from other multi-branch pipelines (from subversion (still using it in 2019!) ) and I'm seeing mangled branches being created, for example branches/4.0 mutates from branches-4-0.v5ktnf to bbrraanncchh.lphb61._252-F44–00

          A workaround for this case (reusable job) is to remove the mangled branch after it fails and link it back to the correct branch. The job then works. I haven't restarted Jenkins since doing this

          Show
          josh_watson Josh Watson added a comment - - edited Also getting this with Jenkins running on Centos 7, using nginx as a proxy. The jobs are triggered from other multi-branch pipelines (from subversion (still using it in 2019!) ) and I'm seeing mangled branches being created, for example branches/4.0 mutates from branches-4-0.v5ktnf to bbrraanncchh.lphb61._252-F44–00 A workaround for this case (reusable job) is to remove the mangled branch after it fails and link it back to the correct branch. The job then works. I haven't restarted Jenkins since doing this
          Show
          morganr87 Morgan Robertson added a comment - - edited The following are issues that I believe are duplicates or are related to this: https://issues.jenkins-ci.org/browse/JENKINS-55388 https://issues.jenkins-ci.org/browse/JENKINS-57848 https://issues.jenkins-ci.org/browse/JENKINS-57398 https://issues.jenkins-ci.org/browse/JENKINS-57381
          Hide
          morganr87 Morgan Robertson added a comment -

          Do we all have the 'Discard old items' item enabled in Jenkins?  I have a suspicion that what's happening is that GitHub or Bitbucket or whatever git service we use occasionally fails to properly respond to the Jenkins poll request and then Jenkins removes the branch due to the 'Discard old items' option.  I'm going to try disabling this.

          If so, two potential fixes:

          • Allow certain branches to be excepted from the 'Discard old items' job.
          • Make every Bitbucket / GitHub / etc. reliably respond to requests (hah).
          Show
          morganr87 Morgan Robertson added a comment - Do we all have the 'Discard old items' item enabled in Jenkins?  I have a suspicion that what's happening is that GitHub or Bitbucket or whatever git service we use occasionally fails to properly respond to the Jenkins poll request and then Jenkins removes the branch due to the 'Discard old items' option.  I'm going to try disabling this. If so, two potential fixes: Allow certain branches to be excepted from the 'Discard old items' job. Make every Bitbucket / GitHub / etc. reliably respond to requests (hah).
          Hide
          m_o Malte Onnen added a comment -

          We're having this issue with Jenkins 2.222.3, Branch API 2.5.6 working with RhodeCode (hosted internally). It seems to happen more often on Jobs that are not really active in development, but resetting a master-branch seems to always be a bad idea.

          Show
          m_o Malte Onnen added a comment - We're having this issue with Jenkins 2.222.3, Branch API 2.5.6 working with RhodeCode (hosted internally). It seems to happen more often on Jobs that are not really active in development, but resetting a master-branch seems to always be a bad idea.

            People

            • Assignee:
              Unassigned
              Reporter:
              cornr Jonas Büth
            • Votes:
              17 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

              • Created:
                Updated: