Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10547

Windows service can't start if Windows Application Event log is full

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      Windows XP
    • Similar Issues:

      Description

      If the windows event log is full and jenkins in installed as a service the service will start and then straight away stop.
      Nothing appears in any log files.

      When I changed the event log setting from over write if older 7 day to over write as necessary the service started as expected.

        Attachments

          Issue Links

            Activity

            Hide
            danielbeck Daniel Beck added a comment -

            Could someone please explain why this is a Jenkins bug?

            Does it still occur on recent Jenkins versions and Windows versions?

            Show
            danielbeck Daniel Beck added a comment - Could someone please explain why this is a Jenkins bug? Does it still occur on recent Jenkins versions and Windows versions?
            Hide
            evernat evernat added a comment - - edited

            In my opinion, it was a Jenkins bug, because it caused Jenkins to not start and Jenkins did not say why, causing much mystery quite often.
            When Jenkins starts ok, it adds following events:

            • envar JENKINS_HOME=...
            • Starting ...\bin\java -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "...\jenkins.war"
            • Service started successfully.

            That said, it seems that Microsoft has understood the problem. Since Vista, the maximum event log size is bigger.
            And I suppose that XP is not supported any more.

            Show
            evernat evernat added a comment - - edited In my opinion, it was a Jenkins bug, because it caused Jenkins to not start and Jenkins did not say why, causing much mystery quite often. When Jenkins starts ok, it adds following events: envar JENKINS_HOME=... Starting ...\bin\java -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "...\jenkins.war" Service started successfully. That said, it seems that Microsoft has understood the problem. Since Vista, the maximum event log size is bigger. And I suppose that XP is not supported any more.
            Hide
            danielbeck Daniel Beck added a comment -

            I checked the EventLogger class in .NET, it doesn't look like failure to log (assuming this isn't considered out of memory or otherwise fatal) should affect anything.

            Is there a way to easily reproduce this? Write a tool that spams event log until it crashes, then try to start Jenkins installed as service?

            Show
            danielbeck Daniel Beck added a comment - I checked the EventLogger class in .NET, it doesn't look like failure to log (assuming this isn't considered out of memory or otherwise fatal) should affect anything. Is there a way to easily reproduce this? Write a tool that spams event log until it crashes, then try to start Jenkins installed as service?
            Hide
            evernat evernat added a comment -

            I have reproduced this on XP with Jenkins v1.574.
            After changing the properties of application events log to not overwrite (the default is to overwrite only events older than 7 days), the Jenkins service does not want to start and it does not say anything on why.
            There is nothing written in jenkins.wrapper.log, jenkins.out.log or jenkins.err.log when this happens.

            Show
            evernat evernat added a comment - I have reproduced this on XP with Jenkins v1.574. After changing the properties of application events log to not overwrite (the default is to overwrite only events older than 7 days), the Jenkins service does not want to start and it does not say anything on why. There is nothing written in jenkins.wrapper.log, jenkins.out.log or jenkins.err.log when this happens.
            Hide
            danielbeck Daniel Beck added a comment -

            I'd like to reproduce this on a more recent Windows version so make sure this is still relevant in some way (even if much less likely to occur), is that the same strategy?

            As I'm not regularly using Windows and would need to start a special Windows testing VM I doubt I have enough events to hit that limit (unless it's a real joke). Hence the question how to generate events.

            Show
            danielbeck Daniel Beck added a comment - I'd like to reproduce this on a more recent Windows version so make sure this is still relevant in some way (even if much less likely to occur), is that the same strategy? As I'm not regularly using Windows and would need to start a special Windows testing VM I doubt I have enough events to hit that limit (unless it's a real joke). Hence the question how to generate events.
            Hide
            evernat evernat added a comment - - edited

            I do not know how to generate windows applications events, but you can set a very low log size and no overwrite in the control panel:
            http://technet.microsoft.com/fr-fr/library/cc722385%28v=ws.10%29.aspx

            or create events:
            http://stackoverflow.com/questions/446691/how-to-create-windows-eventlog-source-from-command-line

            Show
            evernat evernat added a comment - - edited I do not know how to generate windows applications events, but you can set a very low log size and no overwrite in the control panel: http://technet.microsoft.com/fr-fr/library/cc722385%28v=ws.10%29.aspx or create events: http://stackoverflow.com/questions/446691/how-to-create-windows-eventlog-source-from-command-line
            Hide
            danielbeck Daniel Beck added a comment -

            I reproduced the issue as described on Windows 8.1 generating the ~3000 events necessary for the 1MB log to stop accepting new events, and it caused a Jenkins service restart to fail.

            Opened JENKINS_HOME, ordered files by modification date, and found in jenkins.wrapper.log:

            2014-09-06 16:49:50 - Stop exception
            Message:The event log file is full
            Stacktrace:   at System.Diagnostics.EventLogInternal.InternalWriteEvent(UInt32 eventID, UInt16 category, EventLogEntryType type, String[] strings, Byte[] rawData, String currentMachineName)
               at System.Diagnostics.EventLogInternal.WriteEntry(String message, EventLogEntryType type, Int32 eventID, Int16 category, Byte[] rawData)
               at System.Diagnostics.EventLog.WriteEntry(String message)
               at winsw.WrapperService.LogEvent(String message)
               at winsw.WrapperService.StopIt()
               at winsw.WrapperService.OnStop()
            

            Unfortunately it seems this is only added when stopping the service fails, not during the start.

            Show
            danielbeck Daniel Beck added a comment - I reproduced the issue as described on Windows 8.1 generating the ~3000 events necessary for the 1MB log to stop accepting new events, and it caused a Jenkins service restart to fail. Opened JENKINS_HOME, ordered files by modification date, and found in jenkins.wrapper.log : 2014-09-06 16:49:50 - Stop exception Message:The event log file is full Stacktrace: at System.Diagnostics.EventLogInternal.InternalWriteEvent(UInt32 eventID, UInt16 category, EventLogEntryType type, String[] strings, Byte[] rawData, String currentMachineName) at System.Diagnostics.EventLogInternal.WriteEntry(String message, EventLogEntryType type, Int32 eventID, Int16 category, Byte[] rawData) at System.Diagnostics.EventLog.WriteEntry(String message) at winsw.WrapperService.LogEvent(String message) at winsw.WrapperService.StopIt() at winsw.WrapperService.OnStop() Unfortunately it seems this is only added when stopping the service fails, not during the start.
            Hide
            danielbeck Daniel Beck added a comment -

            Created pull request that logs failures to write to the Event Log to the `jenkins.wrapper.log`.

            Show
            danielbeck Daniel Beck added a comment - Created pull request that logs failures to write to the Event Log to the `jenkins.wrapper.log`.
            Hide
            danielbeck Daniel Beck added a comment -

            PR was merged, but given how rare releases of winsw are, who knows when that will happen.

            Show
            danielbeck Daniel Beck added a comment - PR was merged, but given how rare releases of winsw are, who knows when that will happen.
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/windows-slave-installer-module/95206829480d66058fedf75d781358166374142a Log: Update to winsw-1.18 Changes: https://github.com/kohsuke/winsw/issues?q=milestone%3Awinsw-1.17 and https://github.com/kohsuke/winsw/commit/2bf9e858a94c7caca41812d644543e61ba1c179e Jenkins issues: JENKINS-10547 , JENKINS-22685
            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/22d9db09d56c1caa2a8a3e008e6bd29f5c9ed395 Log: Update windows-slave-installer dependency to 1.5.1 Jenkins issues: JENKINS-10547 , JENKINS-22685 Changes in winsw: https://github.com/kohsuke/winsw/issues?q=milestone%3Awinsw-1.17 kohsuke/winsw@2bf9e85 Changes in the module: https://github.com/jenkinsci/windows-slave-installer-module/commit/a4d2121812fe9fe1b38bbe9d2cb57b9cd5435a0d https://github.com/jenkinsci/windows-slave-installer-module/commit/3fbc8434aba05d5d36f52f345d9d0176e98459be
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            war/pom.xml
            http://jenkins-ci.org/commit/jenkins/ddfe4d207750352f123bfb510db70c0ea0820ebe
            Log:
            Merge pull request #1832 from oleg-nenashev/windows-slave-installer-1.5

            [JENKINS-10547, JENKINS-22685] Update windows-slave-installer dependency to 1.5

            Compare: https://github.com/jenkinsci/jenkins/compare/d86a88ab042c...ddfe4d207750

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: war/pom.xml http://jenkins-ci.org/commit/jenkins/ddfe4d207750352f123bfb510db70c0ea0820ebe Log: Merge pull request #1832 from oleg-nenashev/windows-slave-installer-1.5 [JENKINS-10547, JENKINS-22685] Update windows-slave-installer dependency to 1.5 Compare: https://github.com/jenkinsci/jenkins/compare/d86a88ab042c...ddfe4d207750
            Hide
            danielbeck Daniel Beck added a comment -

            Should be fixed in Jenkins 1.630+

            Show
            danielbeck Daniel Beck added a comment - Should be fixed in Jenkins 1.630+

              People

              • Assignee:
                danielbeck Daniel Beck
                Reporter:
                millergarym millergarym
              • Votes:
                3 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: