Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49060

Jenkins fails to start

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
      Jenkins 2.102
      OpenJDK 1.8.0_151
      Amazon Linux
    • Similar Issues:

      Description

      I have Jenkins 2.101 running perfectly well. When yum upgrades to 2.102 it fails to start with a "java.nio.charset.MalformedInputException: Input length = 1" error.
      I can downgrade back to 101 and it starts fine.

      I'd guess at something in a config file somewhere it doesn't like, but 101 is happy with it, and it isn't telling me which config file. We're unlikely to be using any international characters beyond ASCII, but we could be, somewhere...
      (I've included the error messages about slaves but I expect they're just a consequence of the main app failing to start. It does run a web server, and I can get the same error about charset in my browser, framed by a normal Jenkins page.)

      Jenkins home directory: /var/lib/jenkins found at: SystemProperties.getProperty("JENKINS_HOME")
      Jan 21, 2018 12:43:40 PM org.eclipse.jetty.server.handler.ContextHandler doStart
      INFO: Started w.@798162bc{/,file:///var/cache/jenkins/war/,AVAILABLE}{/var/cache/jenkins/war}
      Jan 21, 2018 12:43:40 PM org.eclipse.jetty.server.AbstractConnector doStart
      INFO: Started ServerConnector@13526e59{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
      Jan 21, 2018 12:43:40 PM org.eclipse.jetty.server.Server doStart
      INFO: Started @2973ms
      Jan 21, 2018 12:43:40 PM winstone.Logger logInternal
      INFO: Winstone Servlet Engine v4.0 running: controlPort=disabled
      Jan 21, 2018 12:43:40 PM hudson.util.BootFailure publish
      SEVERE: Failed to initialize Jenkins
      hudson.util.HudsonFailedToLoad: java.nio.charset.MalformedInputException: Input length = 1
      at hudson.WebAppMain$3.run(WebAppMain.java:250)
      Caused by: java.nio.charset.MalformedInputException: Input length = 1
      at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
      at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
      at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
      at java.io.InputStreamReader.read(InputStreamReader.java:184)
      at java.io.BufferedReader.fill(BufferedReader.java:161)
      at java.io.BufferedReader.readLine(BufferedReader.java:324)
      at java.io.BufferedReader.readLine(BufferedReader.java:389)
      at hudson.util.TextFile.read(TextFile.java:76)
      at hudson.util.TextFile.readTrim(TextFile.java:204)
      at jenkins.model.Jenkins.<init>(Jenkins.java:869)
      at hudson.model.Hudson.<init>(Hudson.java:85)
      at hudson.model.Hudson.<init>(Hudson.java:81)
      at hudson.WebAppMain$3.run(WebAppMain.java:233)
      
      Jan 21, 2018 12:43:40 PM jenkins.model.Jenkins cleanUp
      INFO: Stopping Jenkins
      Jan 21, 2018 12:43:40 PM jenkins.model.Jenkins$19 onAttained
      INFO: Started termination
      Jan 21, 2018 12:43:40 PM jenkins.model.Jenkins$19 onTaskFailed
      SEVERE: Failed IOHubProvider.cleanUp
      java.lang.IllegalArgumentException: Unable to inject class jenkins.slaves.IOHubProvider
      at hudson.init.TaskMethodFinder.lookUp(TaskMethodFinder.java:125)
      at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:105)
      at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
      at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
      at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
      at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      at jenkins.model.Jenkins$18.execute(Jenkins.java:3274)
      at org.jvnet.hudson.reactor.Reactor$Node.runIfPossible(Reactor.java:139)
      at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:128)
      at jenkins.model.Jenkins$18.execute(Jenkins.java:3274)
      at org.jvnet.hudson.reactor.Reactor$Node.runIfPossible(Reactor.java:139)
      at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:263)
      at jenkins.model.Jenkins._cleanUpRunTerminators(Jenkins.java:3271)
      at jenkins.model.Jenkins.cleanUp(Jenkins.java:3192)
      at hudson.WebAppMain$3.run(WebAppMain.java:254)
      
      Jan 21, 2018 12:43:40 PM jenkins.model.Jenkins$19 onTaskFailed
      SEVERE: Failed NioChannelSelector.cleanUp
      java.lang.IllegalArgumentException: Unable to inject class jenkins.slaves.NioChannelSelector
      at hudson.init.TaskMethodFinder.lookUp(TaskMethodFinder.java:125)
      at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:105)
      at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
      at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
      at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
      at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
      at jenkins.model.Jenkins$18.execute(Jenkins.java:3274)
      at org.jvnet.hudson.reactor.Reactor$Node.runIfPossible(Reactor.java:139)
      at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:128)
      at jenkins.model.Jenkins$18.execute(Jenkins.java:3274)
      at org.jvnet.hudson.reactor.Reactor$Node.runIfPossible(Reactor.java:139)
      at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:263)
      at jenkins.model.Jenkins._cleanUpRunTerminators(Jenkins.java:3271)
      at jenkins.model.Jenkins.cleanUp(Jenkins.java:3192)
      at hudson.WebAppMain$3.run(WebAppMain.java:254)
      
      Jan 21, 2018 12:43:40 PM jenkins.model.Jenkins$19 onAttained
      INFO: Completed termination
      Jan 21, 2018 12:43:40 PM jenkins.model.Jenkins _cleanUpDisconnectComputers
      INFO: Starting node disconnection
      Jan 21, 2018 12:43:40 PM jenkins.model.Jenkins _cleanUpPersistQueue
      INFO: Persisting build queue
      Jan 21, 2018 12:43:41 PM jenkins.model.Jenkins cleanUp
      INFO: Jenkins stopped
      
      

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The line points to "secretKey = secretFile.readTrim();" which tries to read a secret from the disk (Jenkins:869). The code itself has not changed since 2012, but in 2.102 there was a change in the TextFile implementation: https://github.com/jenkinsci/jenkins/commit/67076834a24f03b832ab03b1bf9b96e05a5da81e#diff-7bec82f0a647b1592b682a66e7cc6bcf

            CC Daniel Trebbien

            Show
            oleg_nenashev Oleg Nenashev added a comment - The line points to "secretKey = secretFile.readTrim();" which tries to read a secret from the disk (Jenkins:869). The code itself has not changed since 2012, but in 2.102 there was a change in the TextFile implementation: https://github.com/jenkinsci/jenkins/commit/67076834a24f03b832ab03b1bf9b96e05a5da81e#diff-7bec82f0a647b1592b682a66e7cc6bcf CC Daniel Trebbien
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            I would assume that the file is corrupted somehow, but it needs to be diagnosed. And I doubt I can ask to attach the secret file to JIRA

            Show
            oleg_nenashev Oleg Nenashev added a comment - I would assume that the file is corrupted somehow, but it needs to be diagnosed. And I doubt I can ask to attach the secret file to JIRA
            Hide
            max_allan max allan added a comment -

            Thanks Oleg Nenashev
            Looking way back in time I can find a copy of the secret on an old server :

             

            ls -la secret.key 
            -rw-r--r-- 1 jenkins jenkins 64 Apr 27  2012 secret.key
            [root@jenkins jenkins]# more secret.key 
            d86b15...REDACTED...12de09
            

             

             

            Looking on the current server :

            ls -la secret.key
            -rw-r--r-- 1 jenkins jenkins 61440 May 18 2017 secret.key
            [root@jenkins]# more secret.key
            secret.key.not-so-secret
            #
            # This file is for Jenkins core developers to list plugin callables which we believe are safe.
            #
            # To whitelist other names, place *.conf files by other names into this folder.
            # This file gets overwritten every time Jenkins starts.
            #
            # See https://jenkins.io/redirect/security-144 for more details.
            etc.......
            more text, config files, random non-ASCII, etc...

             

            It's been like this since May last year!! And nothing broke.

            Show
            max_allan max allan added a comment - Thanks Oleg Nenashev Looking way back in time I can find a copy of the secret on an old server :   ls -la secret.key -rw-r--r-- 1 jenkins jenkins 64 Apr 27  2012 secret.key [root@jenkins jenkins]# more secret.key d86b15...REDACTED...12de09     Looking on the current server : ls -la secret.key -rw-r--r-- 1 jenkins jenkins 61440 May 18 2017 secret.key [root@jenkins]# more secret.key secret.key.not-so-secret # # This file is for Jenkins core developers to list plugin callables which we believe are safe. # # To whitelist other names, place *.conf files by other names into this folder. # This file gets overwritten every time Jenkins starts. # # See https: //jenkins.io/redirect/security-144 for more details. etc....... more text, config files, random non-ASCII, etc...   It's been like this since May last year!! And nothing broke.
            Hide
            dtrebbien Daniel Trebbien added a comment -

            The file probably has non-UTF-8 content.

            Investigating, I found that there is a slight difference between:

            BufferedReader in = new BufferedReader(new InputStreamReader(Files.newInputStream(file.toPath()), "UTF-8")
            

            and

            BufferedReader in = Files.newBufferedReader(file.toPath(), StandardCharsets.UTF_8)
            

            Whereas the former uses the InputStreamReader constructor taking a charset name, the latter uses the InputStreamReader taking a CharsetDecoder. CharsetDecoder is apparently more picky about not accepting malformed bytes.

            Show
            dtrebbien Daniel Trebbien added a comment - The file probably has non-UTF-8 content. Investigating, I found that there is a slight difference between: BufferedReader in = new BufferedReader( new InputStreamReader(Files.newInputStream(file.toPath()), "UTF-8" ) and BufferedReader in = Files.newBufferedReader(file.toPath(), StandardCharsets.UTF_8) Whereas the former uses the InputStreamReader constructor taking a charset name, the latter uses the InputStreamReader taking a CharsetDecoder . CharsetDecoder is apparently more picky about not accepting malformed bytes.
            Hide
            dtrebbien Daniel Trebbien added a comment -

            Here is a test I wrote showing the difference:

            import java.io.*;
            import java.nio.charset.*;
            
            public class CharsetsTest {
                public static void main(String[] args) throws Exception {
                    String str = "\uD83D\uDE0E";
                    byte[] bytes = str.getBytes(StandardCharsets.UTF_8);
                    for (byte b : bytes) {
                        System.out.printf("%02x ", b);
                    }
                    System.out.printf("(%d bytes)\n", bytes.length);
                    byte[] malformedBytes = new byte[bytes.length + 1];
                    System.arraycopy(bytes, 0, malformedBytes, 1, bytes.length);
                    malformedBytes[0] = malformedBytes[1];
                    for (byte b : malformedBytes) {
                        System.out.printf("%02x ", b);
                    }
                    System.out.printf("(%d bytes)\n", malformedBytes.length);
            
                    {
                        System.out.println("Reading via `new InputStreamReader(is, \"UTF-8\")'");
                        ByteArrayInputStream is = new ByteArrayInputStream(malformedBytes);
                        BufferedReader br = new BufferedReader(new InputStreamReader(is, "UTF-8"));
                        br.readLine();
                    }
                    {
                        System.out.println("Reading via `new InputStreamReader(is, StandardCharsets.UTF_8)'");
                        ByteArrayInputStream is = new ByteArrayInputStream(malformedBytes);
                        BufferedReader br = new BufferedReader(new InputStreamReader(is, StandardCharsets.UTF_8));
                        br.readLine();
                    }
                    {
                        System.out.println("Reading via `new InputStreamReader(is, d)'");
                        ByteArrayInputStream is = new ByteArrayInputStream(malformedBytes);
                        CharsetDecoder d = StandardCharsets.UTF_8.newDecoder();
                        BufferedReader br = new BufferedReader(new InputStreamReader(is, d));
                        br.readLine();
                    }
                }
            }
            

            This outputs:

            f0 9f 98 8e (4 bytes)
            f0 f0 9f 98 8e (5 bytes)
            Reading via `new InputStreamReader(is, "UTF-8")'
            Reading via `new InputStreamReader(is, StandardCharsets.UTF_8)'
            Reading via `new InputStreamReader(is, d)'
            Exception in thread "main" java.nio.charset.MalformedInputException: Input length = 1
            	at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
            	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
            	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
            	at java.io.InputStreamReader.read(InputStreamReader.java:184)
            	at java.io.BufferedReader.fill(BufferedReader.java:161)
            	at java.io.BufferedReader.readLine(BufferedReader.java:324)
            	at java.io.BufferedReader.readLine(BufferedReader.java:389)
            	at CharsetsTest.main(CharsetsTest.java:121)
            
            Show
            dtrebbien Daniel Trebbien added a comment - Here is a test I wrote showing the difference: import java.io.*; import java.nio.charset.*; public class CharsetsTest { public static void main( String [] args) throws Exception { String str = "\uD83D\uDE0E" ; byte [] bytes = str.getBytes(StandardCharsets.UTF_8); for ( byte b : bytes) { System .out.printf( "%02x " , b); } System .out.printf( "(%d bytes)\n" , bytes.length); byte [] malformedBytes = new byte [bytes.length + 1]; System .arraycopy(bytes, 0, malformedBytes, 1, bytes.length); malformedBytes[0] = malformedBytes[1]; for ( byte b : malformedBytes) { System .out.printf( "%02x " , b); } System .out.printf( "(%d bytes)\n" , malformedBytes.length); { System .out.println( "Reading via ` new InputStreamReader(is, \" UTF-8\ ")'" ); ByteArrayInputStream is = new ByteArrayInputStream(malformedBytes); BufferedReader br = new BufferedReader( new InputStreamReader(is, "UTF-8" )); br.readLine(); } { System .out.println( "Reading via ` new InputStreamReader(is, StandardCharsets.UTF_8)'" ); ByteArrayInputStream is = new ByteArrayInputStream(malformedBytes); BufferedReader br = new BufferedReader( new InputStreamReader(is, StandardCharsets.UTF_8)); br.readLine(); } { System .out.println( "Reading via ` new InputStreamReader(is, d)'" ); ByteArrayInputStream is = new ByteArrayInputStream(malformedBytes); CharsetDecoder d = StandardCharsets.UTF_8.newDecoder(); BufferedReader br = new BufferedReader( new InputStreamReader(is, d)); br.readLine(); } } } This outputs: f0 9f 98 8e (4 bytes) f0 f0 9f 98 8e (5 bytes) Reading via `new InputStreamReader(is, "UTF-8")' Reading via `new InputStreamReader(is, StandardCharsets.UTF_8)' Reading via `new InputStreamReader(is, d)' Exception in thread "main" java.nio.charset.MalformedInputException: Input length = 1 at java.nio.charset.CoderResult.throwException(CoderResult.java:281) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.readLine(BufferedReader.java:324) at java.io.BufferedReader.readLine(BufferedReader.java:389) at CharsetsTest.main(CharsetsTest.java:121)
            Hide
            dtrebbien Daniel Trebbien added a comment -

            Is the secret.key file on the current server a default file? Wouldn't that be a security configuration issue?

            Show
            dtrebbien Daniel Trebbien added a comment - Is the secret.key file on the current server a default file? Wouldn't that be a security configuration issue?
            Hide
            max_allan max allan added a comment -

            It's totally corrupted. Bits of Jenkins config, bits of binary from god knows where, bits of other files.... So, I removed it. Restarted Jenkins and all seems well. No idea how the file got corrupted so badly... Depending on how worried the project are about bad files causing weird  problems this can probably be closed. Hopefully anyone else googling the same error will find this solution!

            Show
            max_allan max allan added a comment - It's totally corrupted. Bits of Jenkins config, bits of binary from god knows where, bits of other files.... So, I removed it. Restarted Jenkins and all seems well. No idea how the file got corrupted so badly... Depending on how worried the project are about bad files causing weird  problems this can probably be closed. Hopefully anyone else googling the same error will find this solution!
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Trebbien
            Path:
            core/src/main/java/hudson/util/TextFile.java
            http://jenkins-ci.org/commit/jenkins/0f1c35291d38a7b5700a4cfa47eb04d2cee3a0a2
            Log:
            JENKINS-49060 Include the filename of the unreadable file in the exception message

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Trebbien Path: core/src/main/java/hudson/util/TextFile.java http://jenkins-ci.org/commit/jenkins/0f1c35291d38a7b5700a4cfa47eb04d2cee3a0a2 Log: JENKINS-49060 Include the filename of the unreadable file in the exception message
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/main/java/hudson/util/TextFile.java
            http://jenkins-ci.org/commit/jenkins/66bb3dc6fd2a7ef456b3b7dba732ac2336943236
            Log:
            Merge pull request #3255 from dtrebbien/JENKINS-49060-include-filename-of-unreadable-file

            JENKINS-49060 Include the filename of the unreadable file in the exception message

            Compare: https://github.com/jenkinsci/jenkins/compare/839b003261e8...66bb3dc6fd2a

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/hudson/util/TextFile.java http://jenkins-ci.org/commit/jenkins/66bb3dc6fd2a7ef456b3b7dba732ac2336943236 Log: Merge pull request #3255 from dtrebbien/ JENKINS-49060 -include-filename-of-unreadable-file JENKINS-49060 Include the filename of the unreadable file in the exception message Compare: https://github.com/jenkinsci/jenkins/compare/839b003261e8...66bb3dc6fd2a
            Hide
            danielbeck Daniel Beck added a comment -

            Resolved towards 2.104.

            Show
            danielbeck Daniel Beck added a comment - Resolved towards 2.104.

              People

              • Assignee:
                dtrebbien Daniel Trebbien
                Reporter:
                max_allan max allan
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: