Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-51871

Java 11 Docker Image: Invalid Manifest format when loading Git Client Plugin 2.7.2, JDK 11ea+18 is required

    Details

    • Similar Issues:

      Description

      When running on Java 11, some plugins fail to startup with the following error:

      INFO: Winstone Servlet Engine v4.0 running: controlPort=disabled
      Jun 11, 2018 10:01:07 PM jenkins.InitReactorRunner$1 onAttained
      INFO: Started initialization
      Jun 11, 2018 10:01:07 PM jenkins.InitReactorRunner$1 onTaskFailed
      SEVERE: Failed Inspecting plugin /Users/nenashev/Documents/jenkins/test/java10/work/plugins/git-client.jpi
      java.io.IOException: invalid manifest format
      	at java.base/java.util.jar.Manifest.read(Manifest.java:223)
      	at java.base/java.util.jar.Manifest.<init>(Manifest.java:69)
      	at java.base/java.util.jar.JarFile.getManifestFromReference(JarFile.java:416)
      	at java.base/java.util.jar.JarFile.getManifest(JarFile.java:402)
      	at jenkins.util.AntClassLoader.addPathFile(AntClassLoader.java:504)
      	at jenkins.util.AntWithFindResourceClassLoader.addPathFiles(AntWithFindResourceClassLoader.java:35)
      	at hudson.ClassicPluginStrategy.createClassLoader(ClassicPluginStrategy.java:317)
      	at hudson.ClassicPluginStrategy.createPluginWrapper(ClassicPluginStrategy.java:258)
      	at hudson.PluginManager$1$3$1.run(PluginManager.java:403)
      	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
      	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
      	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1068)
      	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
      	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:832)
      
      
      

        Attachments

          Activity

          oleg_nenashev Oleg Nenashev created issue -
          oleg_nenashev Oleg Nenashev made changes -
          Field Original Value New Value
          Epic Link JENKINS-51805 [ 191255 ]
          oleg_nenashev Oleg Nenashev made changes -
          Component/s git-plugin [ 15543 ]
          oleg_nenashev Oleg Nenashev made changes -
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          It seems to be a Git-specific issue. Many other plugins install and work successfully

          Show
          oleg_nenashev Oleg Nenashev added a comment - It seems to be a Git-specific issue. Many other plugins install and work successfully
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          CC Mark Waite just in case he knows the cause

          Show
          oleg_nenashev Oleg Nenashev added a comment - CC Mark Waite just in case he knows the cause
          Hide
          markewaite Mark Waite added a comment -

          I don't know the cause. The git client plugin being used in your test might be git client plugin 2.7.x. That version of git client plugin intentionally bundled JGit 4.5.4 in order to retain Java 7 compatibility.

          You might try git client plugin 3.0.0-beta3 to see if it shows the same failure mode. It bundles JGit 4.11.0 and requires Jenkins 2.60.3 or later.

          Show
          markewaite Mark Waite added a comment - I don't know the cause. The git client plugin being used in your test might be git client plugin 2.7.x. That version of git client plugin intentionally bundled JGit 4.5.4 in order to retain Java 7 compatibility. You might try git client plugin 3.0.0-beta3 to see if it shows the same failure mode. It bundles JGit 4.11.0 and requires Jenkins 2.60.3 or later.
          markewaite Mark Waite made changes -
          Component/s git-client-plugin [ 17423 ]
          Component/s git-plugin [ 15543 ]
          Hide
          markewaite Mark Waite added a comment -

          Can you provide the command line you're using to start Jenkins with JDK 11?

          I see in "end of the road for Java EE" and in Java JEP 320 that the java.xml.bind module is no longer available, but don't immediately see how to handle that in Jenkins.

          Show
          markewaite Mark Waite added a comment - Can you provide the command line you're using to start Jenkins with JDK 11? I see in " end of the road for Java EE " and in Java JEP 320 that the java.xml.bind module is no longer available, but don't immediately see how to handle that in Jenkins.
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Mark Waite I just dropped 3 libs into the exploded WAR in JENKINS_HOME/war/WEB-INF/lib , it was enough to get it running

          Show
          oleg_nenashev Oleg Nenashev added a comment - Mark Waite I just dropped 3 libs into the exploded WAR in JENKINS_HOME/war/WEB-INF/lib , it was enough to get it running
          oleg_nenashev Oleg Nenashev made changes -
          Summary Invalid Manifest format on Java 11 (Git Plugin) Invalid Manifest format on Java 11 (Git Client Plugin)
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          I am working on the Run guidelines now. BTW, Git Client Plugin leads to failure of some other plugins in the default setup flow

          Show
          oleg_nenashev Oleg Nenashev added a comment - I am working on the Run guidelines now. BTW, Git Client Plugin leads to failure of some other plugins in the default setup flow
          oleg_nenashev Oleg Nenashev made changes -
          oleg_nenashev Oleg Nenashev made changes -
          Priority Minor [ 4 ] Critical [ 2 ]
          Hide
          markewaite Mark Waite added a comment - - edited

          Since the failure is happening during install, I'm confident you're using git client plugin 2.7.2. Use 3.0.0-beta3 to see if it gives better results. It is built with Java 8 and JGit 4.11.0. The older version is JDK 7 compatible and is built with JGit 4.5.4

          If the git client plugin fails to load, the git plugin which critically depends on it will also fail to load. If the git plugin fails to load, pandemonium ensues.

          Show
          markewaite Mark Waite added a comment - - edited Since the failure is happening during install, I'm confident you're using git client plugin 2.7.2. Use 3.0.0-beta3 to see if it gives better results. It is built with Java 8 and JGit 4.11.0. The older version is JDK 7 compatible and is built with JGit 4.5.4 If the git client plugin fails to load, the git plugin which critically depends on it will also fail to load. If the git plugin fails to load, pandemonium ensues.
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Confirmed, Git Client (https://github.com/jenkinsci/git-client-plugin/releases/tag/git-client-3.0.0-beta3) installs well from Experimental UC.
          What would be the release ETA? If it is not going to happen soon, I will update guidelines

          Java 10 and 11 run guidelines BTW: https://github.com/jenkins-infra/jenkins.io/pull/1590

          Show
          oleg_nenashev Oleg Nenashev added a comment - Confirmed, Git Client ( https://github.com/jenkinsci/git-client-plugin/releases/tag/git-client-3.0.0-beta3 ) installs well from Experimental UC. What would be the release ETA? If it is not going to happen soon, I will update guidelines Java 10 and 11 run guidelines BTW: https://github.com/jenkins-infra/jenkins.io/pull/1590
          Hide
          markewaite Mark Waite added a comment -

          Update the guidelines. Release is a month or more away

          Show
          markewaite Mark Waite added a comment - Update the guidelines. Release is a month or more away
          oleg_nenashev Oleg Nenashev made changes -
          Summary Invalid Manifest format on Java 11 (Git Client Plugin) Invalid Manifest format on Java 11 (Git Client Plugin 2.7.2)
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          done, ready for review

          Show
          oleg_nenashev Oleg Nenashev added a comment - done, ready for review
          dnusbaum Devin Nusbaum made changes -
          Assignee Devin Nusbaum [ dnusbaum ]
          dnusbaum Devin Nusbaum made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          dnusbaum Devin Nusbaum added a comment -

          I am looking into the root cause to see what other plugins might be affected. A diff of the git-client MANIFEST.MF between the version shows only version number differences, so my guess is that this is a problem in one of the dependencies of git-client:2.7.2.

          Show
          dnusbaum Devin Nusbaum added a comment - I am looking into the root cause to see what other plugins might be affected. A diff of the git-client MANIFEST.MF between the version shows only version number differences, so my guess is that this is a problem in one of the dependencies of git-client:2.7.2.
          Hide
          dnusbaum Devin Nusbaum added a comment -

          I was unable to reproduce this locally running the java11-support branch locally with Java build 11-ea+18. My best guess is that there was an error in a SHA (maybe a corrupted download?), but I would have expected that to be caught earlier in the installation process, so I'm not sure. Oleg Nenashev are you still able to reproduce the issue?

          Show
          dnusbaum Devin Nusbaum added a comment - I was unable to reproduce this locally running the java11-support branch locally with Java build 11-ea+18. My best guess is that there was an error in a SHA (maybe a corrupted download?), but I would have expected that to be caught earlier in the installation process, so I'm not sure. Oleg Nenashev are you still able to reproduce the issue?
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Devin Nusbaum yes, I am still able to reproduce it in Docker

          1) Run docker run -p 8080:8080 -p 50000:50000 jenkins/jenkins-experimental:latest-jdk11
          2) Select "Git plugin" in the installation wizard

          Show
          oleg_nenashev Oleg Nenashev added a comment - Devin Nusbaum yes, I am still able to reproduce it in Docker 1) Run docker run -p 8080:8080 -p 50000:50000 jenkins/jenkins-experimental:latest-jdk11 2) Select "Git plugin" in the installation wizard
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          https://github.com/jenkinsci/jenkins/pull/3505 will change the image tho. Maybe it makes sense to try again once it's merged

          Show
          oleg_nenashev Oleg Nenashev added a comment - https://github.com/jenkinsci/jenkins/pull/3505 will change the image tho. Maybe it makes sense to try again once it's merged
          Hide
          dnusbaum Devin Nusbaum added a comment -

          Oleg Nenashev Ok, my local version is based on `jdk-11-ea+18`, but the docker images look like they are based on `jdk-11-ea+17`, so maybe that is related. I will investigate further.

          Show
          dnusbaum Devin Nusbaum added a comment - Oleg Nenashev Ok, my local version is based on `jdk-11-ea+18`, but the docker images look like they are based on `jdk-11-ea+17`, so maybe that is related. I will investigate further.
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Thanks! If it's related, I will rebuild the image tomorrow

          Show
          oleg_nenashev Oleg Nenashev added a comment - Thanks! If it's related, I will rebuild the image tomorrow
          oleg_nenashev Oleg Nenashev made changes -
          Labels java10_hackathon
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Devin Nusbaum jdk-11-ea+18 has not been pushed to Docker yet, so I cannot verify the theory.
          According to http://jdk.java.net/11/release-notes the build just adds GoDaddy certificates so there should be no relation

          Show
          oleg_nenashev Oleg Nenashev added a comment - Devin Nusbaum jdk-11-ea+18 has not been pushed to Docker yet, so I cannot verify the theory. According to http://jdk.java.net/11/release-notes the build just adds GoDaddy certificates so there should be no relation
          Hide
          dnusbaum Devin Nusbaum added a comment -

          Looks like the release notes are only notable changes. I looked through the history of Manifest.java, and saw this change in 11 build 17: https://bugs.openjdk.java.net/browse/JDK-8200530, which caused a regression which was fixed in 11 build 18: https://bugs.openjdk.java.net/browse/JDK-8204494. I will install build 17 locally to confirm that I can reproduce.

          Show
          dnusbaum Devin Nusbaum added a comment - Looks like the release notes are only notable changes. I looked through the history of Manifest.java, and saw this change in 11 build 17: https://bugs.openjdk.java.net/browse/JDK-8200530 , which caused a regression which was fixed in 11 build 18: https://bugs.openjdk.java.net/browse/JDK-8204494 . I will install build 17 locally to confirm that I can reproduce.
          Hide
          dnusbaum Devin Nusbaum added a comment -

          Confirmed locally as specific to JDK 11ea+17 (likely caused by JDK-8200530) and fixed in JDK 11ea+18 (likely fixed by JDK-8204494). There is nothing we need to do other than update our Docker images once the JDK 11ea+18 Docker image is available.

          Show
          dnusbaum Devin Nusbaum added a comment - Confirmed locally as specific to JDK 11ea+17 (likely caused by JDK-8200530 ) and fixed in JDK 11ea+18 (likely fixed by JDK-8204494 ). There is nothing we need to do other than update our Docker images once the JDK 11ea+18 Docker image is available.
          dnusbaum Devin Nusbaum made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolution Not A Defect [ 7 ]
          Hide
          oleg_nenashev Oleg Nenashev added a comment -

          Thanks Devin Nusbaum! I will reopen it and recategorize as a Docker image defect

          Show
          oleg_nenashev Oleg Nenashev added a comment - Thanks Devin Nusbaum ! I will reopen it and recategorize as a Docker image defect
          oleg_nenashev Oleg Nenashev made changes -
          Resolution Not A Defect [ 7 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Assignee Devin Nusbaum [ dnusbaum ] Oleg Nenashev [ oleg_nenashev ]
          oleg_nenashev Oleg Nenashev made changes -
          Component/s docker [ 20834 ]
          Component/s core [ 15593 ]
          Component/s git-client-plugin [ 17423 ]
          oleg_nenashev Oleg Nenashev made changes -
          Summary Invalid Manifest format on Java 11 (Git Client Plugin 2.7.2) Java 11 Docker Image: Invalid Manifest format when loading Git Client Plugin 2.7.2, JDK 11ea+18 is required
          Hide
          oleg_nenashev Oleg Nenashev added a comment - - edited

          It was fixed in new Docker image versions

          Show
          oleg_nenashev Oleg Nenashev added a comment - - edited It was fixed in new Docker image versions
          oleg_nenashev Oleg Nenashev made changes -
          Status Reopened [ 4 ] Fixed but Unreleased [ 10203 ]
          Resolution Fixed [ 1 ]
          oleg_nenashev Oleg Nenashev made changes -
          Epic Link JENKINS-51805 [ 191255 ] JENKINS-52012 [ 191514 ]
          oleg_nenashev Oleg Nenashev made changes -
          Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]

            People

            • Assignee:
              oleg_nenashev Oleg Nenashev
              Reporter:
              oleg_nenashev Oleg Nenashev
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: