Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41580

Branches gone missing after SCM-API upgrades applied

    Details

    • Similar Issues:

      Description

      Summary:
      After installing the plugins required for testing scm-api and related plugins, branches in a GitHub Multibranch project which were previously present, have disappeared from view. Pull requests, if there are any present in the project, are available in their own tab.

      Environment:
      Jenkins 2.42
      Ubuntu 15.04
      Relevant plugin versions: the complete will be in the support bundle. Here's a

      • SCM-API 2.0.2-beta-2
      • branch-api 2.0.2-beta-5
      • github-branch-source 2.0.1-beta-6

      Steps to recreate:

      1. Using a Jenkins with the pre-upgrade versions of the scm-api family of plugins, set up a known good set of GitHub multibranch pipeline jobs with lots of active branches inside.
      2. Upgrade the plugins, using the (admittedly mistake-prone and somewhat fragile process of doing so mostly by hand)
      3. Relaunch Jenkins.

      The list of branches will be empty:

      Note:
      I short-sightedly removed the unusable data that Jenkins complained about when I brought the system back up. That data should hopefully still be in the support bundle. I will also go back and recreate this issue from a backup.

        Attachments

          Activity

          Hide
          recampbell Ryan Campbell added a comment -

          We really need the Jenkins logs. Can you just attach these? I guess they were excluded from the support bundle because you didn't include the node logs.

          Show
          recampbell Ryan Campbell added a comment - We really need the Jenkins logs. Can you just attach these? I guess they were excluded from the support bundle because you didn't include the node logs.
          Hide
          recampbell Ryan Campbell added a comment - - edited

          It looks to me like we can't rebuild the state because we can't authenticate to Bitbucket. Are the BitBucket credentials still present and valid?

          Ah nevermind, this bug is about GitHub, not BitBucket, sorry.

          Show
          recampbell Ryan Campbell added a comment - - edited It looks to me like we can't rebuild the state because we can't authenticate to Bitbucket. Are the BitBucket credentials still present and valid? Ah nevermind, this bug is about GitHub, not BitBucket, sorry.
          Hide
          kshultz Karl Shultz added a comment -

          Log attached now as well.

          Show
          kshultz Karl Shultz added a comment - Log attached now as well.
          Hide
          stephenconnolly Stephen Connolly added a comment - - edited

          So here is an example of where the job is failing to load:

          Jan 30, 2017 10:27:16 PM com.cloudbees.hudson.plugins.folder.AbstractFolder loadChildren
          WARNING: could not load /var/lib/jenkins/jobs/GitHub_Projects/jobs/Multibranch_1_quality-assurance_kshultzCB/branches/Laurinburg
          java.lang.NullPointerException
          	at jenkins.branch.Branch.getName(Branch.java:147)
          	at jenkins.branch.MultiBranchProjectDescriptor$ChildNameGeneratorImpl.dirNameFromItem(MultiBranchProjectDescriptor.java:249)
          	at jenkins.branch.MultiBranchProjectDescriptor$ChildNameGeneratorImpl.dirNameFromItem(MultiBranchProjectDescriptor.java:225)
          	at com.cloudbees.hudson.plugins.folder.AbstractFolder.loadChildren(AbstractFolder.java:546)
          	at com.cloudbees.hudson.plugins.folder.AbstractFolder.onLoad(AbstractFolder.java:702)
          	at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.onLoad(ComputedFolder.java:171)
          	at jenkins.branch.MultiBranchProject.onLoad(MultiBranchProject.java:166)
          	at com.cloudbees.hudson.plugins.folder.AbstractFolder.loadChildren(AbstractFolder.java:597)
          	at com.cloudbees.hudson.plugins.folder.AbstractFolder.onLoad(AbstractFolder.java:702)
          	at com.cloudbees.hudson.plugins.folder.Folder.onLoad(Folder.java:107)
          	at hudson.model.Items.load(Items.java:372)
          	at jenkins.model.Jenkins$17.run(Jenkins.java:3067)
          	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
          	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
          	at jenkins.model.Jenkins$7.runTask(Jenkins.java:1066)
          	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
          	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:745)
          

          Which is an NPE in this function:

              public String getName() {
                  // TODO this could include a uniquifying prefix defined in BranchSource
                  return head.getName();
              }
          

          In other words, the head of a branch is null which should never happen

          So let's look at the source data:

                <branch plugin="branch-api@1.11.1">
                  <sourceId>94242f50-b6ac-4f8e-8940-65b65c5fc8fd</sourceId>
                  <scm class="hudson.plugins.git.GitSCM" plugin="git@3.0.1">
                    <configVersion>2</configVersion>
                    <userRemoteConfigs>
                      <hudson.plugins.git.UserRemoteConfig>
                        <name>origin</name>
                        <refspec>+refs/heads/*:refs/remotes/origin/*</refspec>
                        <url>https://github.com/kshultzCB/quality-assurance.git</url>
                        <credentialsId>0ec1d5e6-012b-4273-a01b-a1efb4812384</credentialsId>
                      </hudson.plugins.git.UserRemoteConfig>
                      <hudson.plugins.git.UserRemoteConfig>
                        <name>origin</name>
                        <refspec>+refs/pull/*/head:refs/remotes/origin/pr/*</refspec>
                        <url>https://github.com/kshultzCB/quality-assurance.git</url>
                        <credentialsId>0ec1d5e6-012b-4273-a01b-a1efb4812384</credentialsId>
                      </hudson.plugins.git.UserRemoteConfig>
                    </userRemoteConfigs>
                    <branches class="singleton-list">
                      <hudson.plugins.git.BranchSpec>
                        <name>Laurinburg</name>
                      </hudson.plugins.git.BranchSpec>
                    </branches>
                    <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
                    <submoduleCfg class="empty-list"/>
                    <extensions/>
                  </scm>
                  <properties/>
                </branch>
          

          So this was saved with branch-api 1.11.1 and git 3.0.1 (also workflow multibranch 2.9.2 which is outside of the snippet above) but critically it is missing the <head> entry

          If we compare this seed data with my test data set, we see:

                <branch plugin="branch-api@1.11.1">
                  <sourceId>45f20856-e3d0-437c-93ad-1532be81fdf1</sourceId>
                  <head plugin="scm-api@1.3">
                    <name>feature-1</name>
                  </head>
                  <scm class="hudson.plugins.git.GitSCM" plugin="git@3.0.1">
                    <configVersion>2</configVersion>
                    <userRemoteConfigs>
                      <hudson.plugins.git.UserRemoteConfig>
                        <name>origin</name>
                        <refspec>+refs/heads/*:refs/remotes/origin/*</refspec>
                        <url>https://github.com/cloudbeers/fu-manchu.git</url>
                        <credentialsId>github</credentialsId>
                      </hudson.plugins.git.UserRemoteConfig>
                      <hudson.plugins.git.UserRemoteConfig>
                        <name>origin</name>
                        <refspec>+refs/pull/*/head:refs/remotes/origin/pr/*</refspec>
                        <url>https://github.com/cloudbeers/fu-manchu.git</url>
                        <credentialsId>github</credentialsId>
                      </hudson.plugins.git.UserRemoteConfig>
                    </userRemoteConfigs>
                    <branches class="singleton-list">
                      <hudson.plugins.git.BranchSpec>
                        <name>feature-1</name>
                      </hudson.plugins.git.BranchSpec>
                    </branches>
                    <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
                    <submoduleCfg class="empty-list"/>
                    <extensions/>
                  </scm>
                  <properties class="java.util.concurrent.CopyOnWriteArrayList"/>
                </branch>
          

          We see the <head> entry which is what we expect.

          Next let's take a look at the archive contents:

          -rw-r--r--  0 jenkins jenkins  1887 30 Jan 19:11 var/lib/jenkins/jobs/GitHub_Projects/jobs/Multibranch_1_quality-assurance_kshultzCB/branches/Laurinburg/config.xml
          ...
          -rw-r--r--  0 jenkins jenkins    10 30 Jan 18:08 var/lib/jenkins/jobs/GitHub_Projects/jobs/Multibranch_1_quality-assurance_kshultzCB/branches/Laurinburg/name-utf8.txt
          -rw-r--r--  0 jenkins jenkins   396 30 Jan 18:08 var/lib/jenkins/jobs/GitHub_Projects/jobs/Multibranch_1_quality-assurance_kshultzCB/branches/Laurinburg/scm-revision-hash.xml
          

          So the interesting thing here is the timestamps on name-utf8.txt and scm-revision-hash.xml compared with config.xml

          Unless you have been changing your system clock, this looks very much like:

          1. System upgraded to latest and ran an index at 18:08
          2. System downgraded to pre-upgrade and ran an index or otherwise triggered a save of the branch config.xml at 19:11
          3. Upgraded again.

          If we look at the branch api 2.0.x tracking file: scm-revision-hash.xml we can see some more details:

          <?xml version='1.0' encoding='UTF-8'?>
          <jenkins.plugins.git.AbstractGitSCMSource_-SCMRevisionImpl plugin="git@3.0.2-beta-2">
            <head class="org.jenkinsci.plugins.github_branch_source.BranchSCMHead" plugin="github-branch-source@2.0.1-beta-6">
              <name>Laurinburg</name>
            </head>
            <hash>d1ecb00aeb4d9dd482d7574c7e2c02116022caf6</hash>
          </jenkins.plugins.git.AbstractGitSCMSource_-SCMRevisionImpl>
          

          So at 18:08 we know that the Git 3.0.2-beta-2 plugin and github branch source 2.0.1-beta-6 plugins were installed and that the head was of type org.jenkinsci.plugins.github_branch_source.BranchSCMHead

          That type does not exist in the 1.x versions of GitHub Branch Source, so when downgrading to the pre-upgrade version the head will be resolved as null (because it is an unknown type) and then even when we upgrade again we will now have the null

          It is for the above exact reason that we set compatibleSinceVersion == 2.0.0 which means that if you upgrade to this version you cannot downgrade without restoring from a backup as there are changes made to the configuration files on disk that will result in data loss during a downgrade below the indicated version

          Show
          stephenconnolly Stephen Connolly added a comment - - edited So here is an example of where the job is failing to load: Jan 30, 2017 10:27:16 PM com.cloudbees.hudson.plugins.folder.AbstractFolder loadChildren WARNING: could not load / var /lib/jenkins/jobs/GitHub_Projects/jobs/Multibranch_1_quality-assurance_kshultzCB/branches/Laurinburg java.lang.NullPointerException at jenkins.branch.Branch.getName(Branch.java:147) at jenkins.branch.MultiBranchProjectDescriptor$ChildNameGeneratorImpl.dirNameFromItem(MultiBranchProjectDescriptor.java:249) at jenkins.branch.MultiBranchProjectDescriptor$ChildNameGeneratorImpl.dirNameFromItem(MultiBranchProjectDescriptor.java:225) at com.cloudbees.hudson.plugins.folder.AbstractFolder.loadChildren(AbstractFolder.java:546) at com.cloudbees.hudson.plugins.folder.AbstractFolder.onLoad(AbstractFolder.java:702) at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.onLoad(ComputedFolder.java:171) at jenkins.branch.MultiBranchProject.onLoad(MultiBranchProject.java:166) at com.cloudbees.hudson.plugins.folder.AbstractFolder.loadChildren(AbstractFolder.java:597) at com.cloudbees.hudson.plugins.folder.AbstractFolder.onLoad(AbstractFolder.java:702) at com.cloudbees.hudson.plugins.folder.Folder.onLoad(Folder.java:107) at hudson.model.Items.load(Items.java:372) at jenkins.model.Jenkins$17.run(Jenkins.java:3067) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:1066) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Which is an NPE in this function : public String getName() { // TODO this could include a uniquifying prefix defined in BranchSource return head.getName(); } In other words, the head of a branch is null which should never happen So let's look at the source data: <branch plugin= "branch-api@1.11.1" > <sourceId> 94242f50-b6ac-4f8e-8940-65b65c5fc8fd </sourceId> <scm class= "hudson.plugins.git.GitSCM" plugin= "git@3.0.1" > <configVersion> 2 </configVersion> <userRemoteConfigs> <hudson.plugins.git.UserRemoteConfig> <name> origin </name> <refspec> +refs/heads/*:refs/remotes/origin/* </refspec> <url> https://github.com/kshultzCB/quality-assurance.git </url> <credentialsId> 0ec1d5e6-012b-4273-a01b-a1efb4812384 </credentialsId> </hudson.plugins.git.UserRemoteConfig> <hudson.plugins.git.UserRemoteConfig> <name> origin </name> <refspec> +refs/pull/*/head:refs/remotes/origin/pr/* </refspec> <url> https://github.com/kshultzCB/quality-assurance.git </url> <credentialsId> 0ec1d5e6-012b-4273-a01b-a1efb4812384 </credentialsId> </hudson.plugins.git.UserRemoteConfig> </userRemoteConfigs> <branches class= "singleton-list" > <hudson.plugins.git.BranchSpec> <name> Laurinburg </name> </hudson.plugins.git.BranchSpec> </branches> <doGenerateSubmoduleConfigurations> false </doGenerateSubmoduleConfigurations> <submoduleCfg class= "empty-list" /> <extensions/> </scm> <properties/> </branch> So this was saved with branch-api 1.11.1 and git 3.0.1 (also workflow multibranch 2.9.2 which is outside of the snippet above) but critically it is missing the <head> entry If we compare this seed data with my test data set, we see: <branch plugin= "branch-api@1.11.1" > <sourceId> 45f20856-e3d0-437c-93ad-1532be81fdf1 </sourceId> <head plugin= "scm-api@1.3" > <name> feature-1 </name> </head> <scm class= "hudson.plugins.git.GitSCM" plugin= "git@3.0.1" > <configVersion> 2 </configVersion> <userRemoteConfigs> <hudson.plugins.git.UserRemoteConfig> <name> origin </name> <refspec> +refs/heads/*:refs/remotes/origin/* </refspec> <url> https://github.com/cloudbeers/fu-manchu.git </url> <credentialsId> github </credentialsId> </hudson.plugins.git.UserRemoteConfig> <hudson.plugins.git.UserRemoteConfig> <name> origin </name> <refspec> +refs/pull/*/head:refs/remotes/origin/pr/* </refspec> <url> https://github.com/cloudbeers/fu-manchu.git </url> <credentialsId> github </credentialsId> </hudson.plugins.git.UserRemoteConfig> </userRemoteConfigs> <branches class= "singleton-list" > <hudson.plugins.git.BranchSpec> <name> feature-1 </name> </hudson.plugins.git.BranchSpec> </branches> <doGenerateSubmoduleConfigurations> false </doGenerateSubmoduleConfigurations> <submoduleCfg class= "empty-list" /> <extensions/> </scm> <properties class= "java.util.concurrent.CopyOnWriteArrayList" /> </branch> We see the <head> entry which is what we expect. Next let's take a look at the archive contents: -rw-r--r-- 0 jenkins jenkins 1887 30 Jan 19:11 var /lib/jenkins/jobs/GitHub_Projects/jobs/Multibranch_1_quality-assurance_kshultzCB/branches/Laurinburg/config.xml ... -rw-r--r-- 0 jenkins jenkins 10 30 Jan 18:08 var /lib/jenkins/jobs/GitHub_Projects/jobs/Multibranch_1_quality-assurance_kshultzCB/branches/Laurinburg/name-utf8.txt -rw-r--r-- 0 jenkins jenkins 396 30 Jan 18:08 var /lib/jenkins/jobs/GitHub_Projects/jobs/Multibranch_1_quality-assurance_kshultzCB/branches/Laurinburg/scm-revision-hash.xml So the interesting thing here is the timestamps on name-utf8.txt and scm-revision-hash.xml compared with config.xml Unless you have been changing your system clock, this looks very much like: 1. System upgraded to latest and ran an index at 18:08 2. System downgraded to pre-upgrade and ran an index or otherwise triggered a save of the branch config.xml at 19:11 3. Upgraded again. If we look at the branch api 2.0.x tracking file: scm-revision-hash.xml we can see some more details: <?xml version= '1.0' encoding= 'UTF-8' ?> <jenkins.plugins.git.AbstractGitSCMSource_-SCMRevisionImpl plugin= "git@3.0.2-beta-2" > <head class= "org.jenkinsci.plugins.github_branch_source.BranchSCMHead" plugin= "github-branch-source@2.0.1-beta-6" > <name> Laurinburg </name> </head> <hash> d1ecb00aeb4d9dd482d7574c7e2c02116022caf6 </hash> </jenkins.plugins.git.AbstractGitSCMSource_-SCMRevisionImpl> So at 18:08 we know that the Git 3.0.2-beta-2 plugin and github branch source 2.0.1-beta-6 plugins were installed and that the head was of type org.jenkinsci.plugins.github_branch_source.BranchSCMHead That type does not exist in the 1.x versions of GitHub Branch Source, so when downgrading to the pre-upgrade version the head will be resolved as null (because it is an unknown type) and then even when we upgrade again we will now have the null It is for the above exact reason that we set compatibleSinceVersion == 2.0.0 which means that if you upgrade to this version you cannot downgrade without restoring from a backup as there are changes made to the configuration files on disk that will result in data loss during a downgrade below the indicated version
          Hide
          stephenconnolly Stephen Connolly added a comment - - edited

          I could only reproduce this using the following procedure:

          1. Using a Jenkins with the upgrade versions of the scm-api family of plugins, set up a known good set of GitHub multibranch pipeline jobs with lots of active branches inside.
          2. Downgrade to the pre-upgrade versions of the scm-api family of plugins, trigger a scan / index.
          3. Upgrade the plugins again
          4. Relaunch Jenkins.

          Given that the Downgrade without restoring from backup is expressly not a supported configuration I am closing this issue as cannot reproduce especially in light of the time-stamp evidence that the Jenkins home contained remnants of a previously upgraded system

          Show
          stephenconnolly Stephen Connolly added a comment - - edited I could only reproduce this using the following procedure: Using a Jenkins with the upgrade versions of the scm-api family of plugins, set up a known good set of GitHub multibranch pipeline jobs with lots of active branches inside. Downgrade to the pre-upgrade versions of the scm-api family of plugins, trigger a scan / index. Upgrade the plugins again Relaunch Jenkins. Given that the Downgrade without restoring from backup is expressly not a supported configuration I am closing this issue as cannot reproduce especially in light of the time-stamp evidence that the Jenkins home contained remnants of a previously upgraded system
          Hide
          kshultz Karl Shultz added a comment -

          FTR: I'm in agreement that this looks like a rescan was performed inadvertently / without my knowledge. Possibly a periodic rescan was performed, and in the midst of applying upgrades and looking at other things, I didn't see it when it happened.

          Show
          kshultz Karl Shultz added a comment - FTR: I'm in agreement that this looks like a rescan was performed inadvertently / without my knowledge. Possibly a periodic rescan was performed, and in the midst of applying upgrades and looking at other things, I didn't see it when it happened.

            People

            • Assignee:
              stephenconnolly Stephen Connolly
              Reporter:
              kshultz Karl Shultz
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: