Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53004

checkout fails due to issue with JGit 5.0.2 (in 3.0.0.beta-5)

    Details

    • Similar Issues:

      Description

      when running checkout scm I get the error:

      java.lang.NoSuchMethodError: org.eclipse.jgit.lib.Repository.getRef(Ljava/lang/String;)Lorg/eclipse/jgit/lib/Ref;
       at jenkins.plugins.git.GitSCMFileSystem$1.invoke(GitSCMFileSystem.java:117)
       at jenkins.plugins.git.GitSCMFileSystem$1.invoke(GitSCMFileSystem.java:114)
       at jenkins.plugins.git.GitSCMFileSystem$3.invoke(GitSCMFileSystem.java:193)
       at org.jenkinsci.plugins.gitclient.AbstractGitAPIImpl.withRepository(AbstractGitAPIImpl.java:28)
       at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.withRepository(CliGitAPIImpl.java:75)
       at jenkins.plugins.git.GitSCMFileSystem.invoke(GitSCMFileSystem.java:189)
       at jenkins.plugins.git.GitSCMFileSystem.<init>(GitSCMFileSystem.java:114)
       at jenkins.plugins.git.GitSCMFileSystem$BuilderImpl.build(GitSCMFileSystem.java:353)
       at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:196)
       at jenkins.scm.api.SCMFileSystem.of(SCMFileSystem.java:172)
       at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:108)
       at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:67)
       at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:303)
       at hudson.model.ResourceController.execute(ResourceController.java:97)
       at hudson.model.Executor.run(Executor.java:429)
      Finished: FAILURE
      

      I suspect this is related to introducing JGit5.02 in 3.0.0.beta-5 When I step back to 2.7.3 it all works fine.

      I'd like to use the new beta because shallow clone of submodules is supported.

        Attachments

          Activity

          Hide
          markewaite Mark Waite added a comment -

          David Rutherford, the problem you're seeing is a problem in the artifactory plugin, not a problem in the git client plugin.

          The artifactory plugin depends on JGit version 3. Git client plugin 2.x delivered JGit version 4 which was largely compatible with JGit version 3 but deprecates some API's that were part of JGit 3. Git client plugin 3.x will deliver JGit 5 which removes at least one of the deprecated APIs.

          The artifactory plugin developers have created a pre-release of their plugin that uses the JGit 4 and JGit 5 API instead of the JGit 3 API.

          Your best near term solution is to downgrade to git client plugin 2.7.x. The git client plugin 3.0.0-rc and git plugin 4.0.0-rc were intended to be release candidate builds for further testing in the experimental update center. They were not intended to be production releases for general availability. I made a serious mistake when I chose the "-rc" suffix for the release.

          The update center will soon hide the git client plugin 3.0.0-rc and git plugin 4.0.0-rc release

          Show
          markewaite Mark Waite added a comment - David Rutherford, the problem you're seeing is a problem in the artifactory plugin, not a problem in the git client plugin. The artifactory plugin depends on JGit version 3. Git client plugin 2.x delivered JGit version 4 which was largely compatible with JGit version 3 but deprecates some API's that were part of JGit 3. Git client plugin 3.x will deliver JGit 5 which removes at least one of the deprecated APIs. The artifactory plugin developers have created a pre-release of their plugin that uses the JGit 4 and JGit 5 API instead of the JGit 3 API. Your best near term solution is to downgrade to git client plugin 2.7.x. The git client plugin 3.0.0-rc and git plugin 4.0.0-rc were intended to be release candidate builds for further testing in the experimental update center. They were not intended to be production releases for general availability. I made a serious mistake when I chose the "-rc" suffix for the release. The update center will soon hide the git client plugin 3.0.0-rc and git plugin 4.0.0-rc release
          Hide
          drutherford David Rutherford added a comment -

          My apologies Mark Waite, I naturally should have thought about the problem from the "opposite direction".  I made the mistake of updating my test Jenkins instance plugins wholesale without taking a backup of the plugins dir first and went into panic-mode once my pipeline job started hanging without any sort of logs indicating errors.  Then I managed to crater my entire install & had to restart from a completely clean install making it even harder to figure out where the problem plugin was.  I was certain the issue was somewhere in the recently updated Pipeline: step api plugins until I encountered this error in the debugger.  On the plus side, this was just a test instance running on my local machine & it forced me to learn how to debug Jenkins plugins but I was definitely pulling my hair out for a bit there. 

          Show
          drutherford David Rutherford added a comment - My apologies Mark Waite , I naturally should have thought about the problem from the "opposite direction".  I made the mistake of updating my test Jenkins instance plugins wholesale without taking a backup of the plugins dir first and went into panic-mode once my pipeline job started hanging without any sort of logs indicating errors.  Then I managed to crater my entire install & had to restart from a completely clean install making it even harder to figure out where the problem plugin was.  I was certain the issue was somewhere in the recently updated Pipeline: step api plugins until I encountered this error in the debugger.  On the plus side, this was just a test instance running on my local machine & it forced me to learn how to debug Jenkins plugins but I was definitely pulling my hair out for a bit there. 
          Hide
          olivergondza Oliver Gondža added a comment - - edited

          Mark Waite, we have a customer running into this with git-client 3.0.0 and git-plugin 3.12.0. The way I understand your analysis here is, an older jgit is leaking from some other plugin so git-(client-)plugin built against jgit5 is invoking signatures that was not present in the older version that is leaking in - so the fix is to get all those plugins to versions using more or less same (major) version of jgit. I tend to agree this might be the case here, as my Jenkins has loaded 2 older versions of jgit:

          $ ls -l /proc/9972/fd/ | grep jgit
          lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 333 -> /var/lib/jenkins/plugins/git-client/WEB-INF/lib/org.eclipse.jgit.lfs-5.5.1.201910021850-r.jar
          lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 334 -> /var/lib/jenkins/plugins/git-client/WEB-INF/lib/org.eclipse.jgit.http.apache-5.5.1.201910021850-r.jar
          lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 337 -> /var/lib/jenkins/plugins/git-client/WEB-INF/lib/org.eclipse.jgit.http.server-5.5.1.201910021850-r.jar
          lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 338 -> /var/lib/jenkins/plugins/git-client/WEB-INF/lib/org.eclipse.jgit-5.5.1.201910021850-r.jar
          lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 548 -> /var/lib/jenkins/plugins/blueocean-events/WEB-INF/lib/org.eclipse.jgit-4.5.4.201711221230-r.jar
          lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 549 -> /var/lib/jenkins/plugins/blueocean-events/WEB-INF/lib/org.eclipse.jgit.http.server-4.5.4.201711221230-r.jar
          lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 553 -> /var/lib/jenkins/plugins/blueocean-events/WEB-INF/lib/org.eclipse.jgit.http.apache-4.5.4.201711221230-r.jar
          lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 764 -> /var/lib/jenkins/plugins/gitlab-plugin/WEB-INF/lib/org.eclipse.jgit-3.5.2.201411120430-r.jar
          

          However, I was under assumption that Jenkins hierarchical classloading is here to prevent dependencies to leak from plugin to plugin (so in this case, git-plugin would get the version it requires and so will the other plugins). What makes this situation special?

          Thanks for any pointers!

          Show
          olivergondza Oliver Gondža added a comment - - edited Mark Waite , we have a customer running into this with git-client 3.0.0 and git-plugin 3.12.0. The way I understand your analysis here is, an older jgit is leaking from some other plugin so git-(client-)plugin built against jgit5 is invoking signatures that was not present in the older version that is leaking in - so the fix is to get all those plugins to versions using more or less same (major) version of jgit. I tend to agree this might be the case here, as my Jenkins has loaded 2 older versions of jgit: $ ls -l /proc/9972/fd/ | grep jgit lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 333 -> /var/lib/jenkins/plugins/git-client/WEB-INF/lib/org.eclipse.jgit.lfs-5.5.1.201910021850-r.jar lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 334 -> /var/lib/jenkins/plugins/git-client/WEB-INF/lib/org.eclipse.jgit.http.apache-5.5.1.201910021850-r.jar lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 337 -> /var/lib/jenkins/plugins/git-client/WEB-INF/lib/org.eclipse.jgit.http.server-5.5.1.201910021850-r.jar lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 338 -> /var/lib/jenkins/plugins/git-client/WEB-INF/lib/org.eclipse.jgit-5.5.1.201910021850-r.jar lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 548 -> /var/lib/jenkins/plugins/blueocean-events/WEB-INF/lib/org.eclipse.jgit-4.5.4.201711221230-r.jar lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 549 -> /var/lib/jenkins/plugins/blueocean-events/WEB-INF/lib/org.eclipse.jgit.http.server-4.5.4.201711221230-r.jar lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 553 -> /var/lib/jenkins/plugins/blueocean-events/WEB-INF/lib/org.eclipse.jgit.http.apache-4.5.4.201711221230-r.jar lr-x------. 1 jenkins jenkins 64 Mar 24 13:30 764 -> /var/lib/jenkins/plugins/gitlab-plugin/WEB-INF/lib/org.eclipse.jgit-3.5.2.201411120430-r.jar However, I was under assumption that Jenkins hierarchical classloading is here to prevent dependencies to leak from plugin to plugin (so in this case, git-plugin would get the version it requires and so will the other plugins). What makes this situation special? Thanks for any pointers!
          Hide
          markewaite Mark Waite added a comment - - edited

          Oliver Gondža sorry for the confusion caused by not having the same major version number on git client plugin and git plugin releases.

          Git client 3.x bundles JGit 5.x and is intended to be used by git plugin 4.x and later. Git client 2.x bundles JGit 4.x and is intended to be used by git plugin 3.x. In the case for your customer, the preferred change is probably to switch them either from git client plugin 3.0.0 to git client plugin 2.9.0 or from git plugin 3.12.0 to git plugin 4.2.2.

          You're correct that the JGit project made API breaking changes between JGit 4.x and JGit 5.x. Those API breaking changes required that we "lock-step" release the git plugin 4.x release and the git client plugin 3.x release.

          I'm not expert enough with the Jenkins class loader or with Java class loading to know if it is possible to have git client plugin 3.x loaded in memory with its bundled copy of JGit 5.x and still have git plugin 3.x cause the loading of JGit 4.x into memory. The git plugin depends on git client and does not declare a dependency on JGit. The git plugin relies on the git client plugin's commitment to always provide a JGit implementation.

          I think in your case that you want to either downgrade from git client plugin 3.0.0 to git client plugin 2.9.0 or upgrade from git plugin 3.12.0 to git plugin 4.2.2. In terms of functionality change, the downgrade from git client plugin 3.0.0 to git client plugin 2.9.0 is smaller than the change from git plugin 3.12.0 to git plugin 4.2.2.

          Show
          markewaite Mark Waite added a comment - - edited Oliver Gondža sorry for the confusion caused by not having the same major version number on git client plugin and git plugin releases. Git client 3.x bundles JGit 5.x and is intended to be used by git plugin 4.x and later. Git client 2.x bundles JGit 4.x and is intended to be used by git plugin 3.x. In the case for your customer, the preferred change is probably to switch them either from git client plugin 3.0.0 to git client plugin 2.9.0 or from git plugin 3.12.0 to git plugin 4.2.2. You're correct that the JGit project made API breaking changes between JGit 4.x and JGit 5.x. Those API breaking changes required that we "lock-step" release the git plugin 4.x release and the git client plugin 3.x release. I'm not expert enough with the Jenkins class loader or with Java class loading to know if it is possible to have git client plugin 3.x loaded in memory with its bundled copy of JGit 5.x and still have git plugin 3.x cause the loading of JGit 4.x into memory. The git plugin depends on git client and does not declare a dependency on JGit. The git plugin relies on the git client plugin's commitment to always provide a JGit implementation. I think in your case that you want to either downgrade from git client plugin 3.0.0 to git client plugin 2.9.0 or upgrade from git plugin 3.12.0 to git plugin 4.2.2. In terms of functionality change, the downgrade from git client plugin 3.0.0 to git client plugin 2.9.0 is smaller than the change from git plugin 3.12.0 to git plugin 4.2.2.
          Hide
          olivergondza Oliver Gondža added a comment -

          Thanks, Mark. Provided both git-plugin and git-client-plugin depend on jgit passing its objects between the codebases, I doubt any amount of classloader isolation is going to save us. I have not realized it is this 2 plugins that are clashing. Thanks for your recommendations.

          Show
          olivergondza Oliver Gondža added a comment - Thanks, Mark. Provided both git-plugin and git-client-plugin depend on jgit passing its objects between the codebases, I doubt any amount of classloader isolation is going to save us. I have not realized it is this 2 plugins that are clashing. Thanks for your recommendations.

            People

            • Assignee:
              Unassigned
              Reporter:
              sa_git sa_git Strukton
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: